Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,448 Commits 326 Branches 371 Tags
Commit Graph

21448 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
sguggilam
970b9a283b
HADOOP-17159. Make UGI support forceful relogin from keytab ignoring the last login time (#2249) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-26 23:58:34 -07:00
Mingliang Liu
acec431377
Revert "HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197)" 
This reverts commit d06f0de3affbd5e8232a6fcdb9a3c396934b6a05.
 2020-08-26 11:23:26 -07:00
Surendra Singh Lilhore
3bacea2e5e HDFS-14096. [SPS] : Add Support for Storage Policy Satisfier in ViewFs. Contributed by Ayush Saxena. 
(cherry picked from commit 788e7473a404fa074b3af522416ee3d2fae865a0)
 2020-08-26 10:31:39 -07:00
Uma Maheswara Rao G
0512b27172 HDFS-8631. WebHDFS : Support setQuota. Contributed by Chao Sun. (Backported) 2020-08-26 09:33:51 -07:00
Steve Loughran
134539f054
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport #2210) 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!
 2020-08-25 22:47:43 +01:00
Uma Maheswara Rao G
978ce5a1ee HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e27add0b9070c829cd828d41a150e8c)
 2020-08-25 12:05:31 -07:00
S O'Donnell
a3f12f7f20 HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bffbe9bad1c719fc3a3b734f0060df70a)
 2020-08-25 15:49:54 +01:00
sguggilam
d06f0de3af
HADOOP-17159 Ability for forceful relogin in UserGroupInformation class (#2197) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-24 23:58:36 -07:00
Joey
956c097f25 HADOOP-16925. MetricsConfig incorrectly loads the configuration whose value is String list in the properties file (#1896) 
Contributed by Jiayi Liu
 2020-08-24 14:06:14 +01:00
Mukund Thakur
28db29d32f HADOOP-17130. Configuration.getValByRegex() shouldn't be updating the results while fetching. (#2142) 
Contributed by Mukund Thakur

Change-Id: I35fe671d4026e8d3c04fc52012c3edcd4495e14a
 2020-08-24 14:05:03 +01:00
S O'Donnell
4163b43a83 HADOOP-17209. Erasure Coding: Native library memory leak. Contributed by Sean Chow 
(cherry picked from commit 17cd8a1b1627f4d87ddc5dcc2ec7c738ffdc576b)
 2020-08-24 13:06:15 +01:00
Chen Liang
4ab14692e6 HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit f734455e5d76c75b7d3a0b751023f5bd02ba38d2)
 2020-08-20 15:26:44 -07:00
Sunil G
94723bff64 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 2a40a33dfecb17eba42f67c0151be9b1e86740aa.
 2020-08-20 19:15:10 +05:30
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext. 
Contributed by Steve Loughran.

This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.

This is intended to allow applications to determine

* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.

As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
 fs. + schema + .

The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.

Note

* The remote store is not expected to be checked for the feature;
  It is more a check of client API and the client's configuration/knowledge
  of the state of the remote system.
* Permissions are not checked.
 2020-08-19 17:15:06 +01:00
Sunil G
2a40a33dfe YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f4421c162a505ba5e5b329e4be199878a7)
 2020-08-19 12:00:33 +05:30
Chao Sun
5858c74a38
HADOOP-17205. Move personality file from Yetus to Hadoop repository (#2226) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fefacf2578e58cf7bf96d920f5b01dba66909ca2)
 2020-08-18 12:11:57 +09:00
Akira Ajisaka
9aa78fe0d2
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584bd253f0831d39557f54627f26cef7f7)
 2020-08-12 16:09:35 +09:00
Gautham B A
ff907b310d
HADOOP-17196. Fix C/C++ standard warnings (#2208) 
* Passing C/C++ standard flags -std is
  not cross-compiler friendly as not all
  compilers support all values.
* Thus, we need to make use of the
  appropriate flags provided by CMake in
  order to specify the C/C++ standards.

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 909f1e82d3e956bdd01b9d24ed2108f93bb94953)
 2020-08-11 16:47:26 +09:00
Sahil Takiar
eae1387f34
HDFS-14394: Add -std=c99 / -std=gnu99 to libhdfs compile flags 
Signed-off-by: Todd Lipcon <todd@apache.org>
(cherry picked from commit 3b0c5016b2931e3b87153c1cf368314d229e79ff)
 2020-08-11 16:45:27 +09:00
Jonathan Hung
17d18a2a3a YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:52 -07:00
Ayush Saxena
d6a9ed8140 HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui. 2020-08-07 22:23:02 +05:30
Wanqiang Ji
ea7e41e29e
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183) 
(cherry picked from commit dc5470ae86b96d6f84abd7354c3c056d3627871b)
 2020-08-06 09:37:03 +09:00
Eric Badger
9a1db93b1b YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 5edd8b925ef22b83350a21abed6ecc551adb92ee)
 2020-08-05 19:03:48 +00:00
Mingliang Liu
41c211d762
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:01:04 -07:00
Mingliang Liu
e094521298
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f782dfcf2deaac428a927f0de6c71def)
 2020-08-05 10:52:07 +09:00
sguggilam
e484f5529c
HADOOP-17164. UGI loginUserFromKeytab doesn't set the last login time (#2178) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-04 10:47:31 -07:00
Uma Maheswara Rao G
cd5efe91d9 HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. (#2160). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit ac9a07b51aefd0fd3b4602adc844ab0f172835e3)
 2020-07-31 01:39:41 -07:00
Uma Maheswara Rao G
65778cdd47 HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links (#2132). Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 3e700066394fb9f516e23537d8abb4661409cae1)
 2020-07-31 01:39:06 -07:00
Uma Maheswara Rao G
1369e41c65 HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit dc0626b5f2f2ba0bd3919650ea231cedd424f77a)
 2020-07-31 01:38:45 -07:00
Uma Maheswara Rao G
512d1d6d27 HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 55a2ae80dc9b45413febd33840b8a653e3e29440)
 2020-07-31 01:38:18 -07:00
Uma Maheswara Rao G
655b39cc30 HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 1f2a80b5e5024aeb7fb1f8c31b8fdd0fdb88bb66)
 2020-07-31 01:37:14 -07:00
Yuan
737bbab90a
HDFS-14950. fix missing libhdfspp lib in dist-package (#1947) 
libhdfspp.{a,so} are missed in dist-package.
This patch fixed this by copying these libs to the right directory

Signed-off-by: Yuan Zhou <yuan.zhou@intel.com>
(cherry picked from commit e756fe3590906bfd8ffe4ab5cc8b9b24a9b2b4b2)
 2020-07-31 16:52:14 +09:00
Eric E Payne
863689ff9a YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166df48a47267dbc81d124ab78267e039)
 2020-07-30 17:08:02 +00:00
ywheel
3a5f48acab
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338) 
(cherry picked from commit cf4eb75608527a8a56ad9d13a3009ccc38b29c8e)
 2020-07-30 13:30:46 +09:00
Jonathan Hung
ffb920de2a YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:44:17 -07:00
Eric Badger
7350773b69 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef0f0ad4210e4027f53877760fa606a5)
 2020-07-24 22:55:08 +00:00
Akira Ajisaka
0fb7c48acb
HADOOP-17147. Dead link in hadoop-kms/index.md.vm. Contributed by Xieming Li. 
(cherry picked from commit d5b476615820a7fa75b41e323db5deb5c2ed3bd5)
 2020-07-23 00:41:00 +09:00
Ayush Saxena
27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Chen Liang
370efc6d95 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e087518196bcb77ac2479c66a0b188d8)
 2020-07-20 14:07:10 -07:00
Akira Ajisaka
0d949d375e HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
 2020-07-20 13:20:34 -07:00
He Xiaoqiao
6d003f67dd HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-16 23:59:05 +08:00
Ahmed Hussein
8fd3dcc9ce HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
 2020-07-15 12:05:49 -05:00
Erik Krogen
bd37b72ea9 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:34:21 -07:00
Ahmed Hussein
43a865dc07 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
 2020-07-15 10:18:47 -05:00
Shashikant Banerjee
acaee85a69 HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 
(cherry picked from commit 82343790eebc3ebe7ef81f6b89260e5bbf121d83)
 2020-07-15 13:38:29 +01:00
Eric Badger
09f1547697 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.
 2020-07-14 18:26:15 +00:00
Eric E Payne
52f2303b5a YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b7543d412f4fafe631b7ace289662d28c)
 2020-07-13 21:34:21 +00:00
Eric Badger
37032bcaf6 Revert "HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell." 
This reverts commit edb45cac8216648fa2a121944ae4499397efd26f.
 2020-07-13 21:08:12 +00:00
He Xiaoqiao
edb45cac82 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:40:21 +08:00
Siyao Meng
0bd0cb1b06 HDFS-15462. Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xml (#2131) 
(cherry picked from commit 0e694b20b9d59cc46882df506dcea386020b1e4d)
 2020-07-09 16:31:43 -07:00