Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
23,683 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

23683 Commits

Author SHA1 Message Date
Wanqiang Ji 215a62e99f
MAPREDUCE-7268. Fix TestMapreduceConfigFields (#1935) 
(cherry picked from commit f8e598f278)
 2020-04-03 11:00:02 +09:00
Akira Ajisaka a2f4dcb470
YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen Zhao. 
(cherry picked from commit c162648aff)
 2020-04-01 16:06:16 +09:00
Inigo Goiri f7a94ec0a4 HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina. 2020-03-29 08:17:02 -07:00
Ayush Saxena 696a663cd0 HDFS-15193. Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`. Contributed by Ctest. 2020-03-29 00:01:22 +05:30
Chen Liang f531a4a487 HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand. 2020-03-27 12:55:38 -07:00
Ayush Saxena cbe71eaddf HDFS-15238. RBF: NamenodeHeartbeatService caused memory to grow rapidly. Contributed by xuzq. 2020-03-27 13:42:06 +05:30
Steve Loughran eaaaba12b1
HADOOP-16939 fs.s3a.authoritative.path should support multiple FS URIs (#1914) 
add unit test, new ITest and then fix the issue: different schema, bucket == skip

factored out the underlying logic for unit testing; also moved
maybeAddTrailingSlash to S3AUtils (while retaining/forwarnding existing method
in S3AFS).

tested: london, sole failure is
testListingDelete[auth=true](org.apache.hadoop.fs.s3a.ITestS3GuardOutOfBandOperations)

filed HADOOP-16853

Change-Id: I4b8d0024469551eda0ec70b4968cba4abed405ed
 2020-03-26 12:59:11 -06:00
Steve Loughran 745a6c1e69
Revert "HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob" 
This reverts commit 3612317038.

Change-Id: Ie0d36f25de0b55a937894f4d9963c495bae0576a
 2020-03-26 15:24:37 +00:00
Inigo Goiri 679631b188 HDFS-13470. RBF: Add Browse the Filesystem button to the UI. 2020-03-26 08:23:18 -07:00
RogPodge 50f7f6dfd1
YARN-10210. Add a RMFailoverProxyProvider that does DNS resolution on failover. 2020-03-26 08:14:30 -07:00
Szilard Nemeth 3d5ade1839 YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R 2020-03-26 13:23:43 +01:00
Szilard Nemeth 9a297ff31d YARN-10197. FS-CS converter: fix emitted ordering policy string and max-am-resource percent value. Contributed by Peter Bacsko 2020-03-26 12:35:10 +01:00
Szilard Nemeth 348685dcb9 YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-03-26 12:18:15 +01:00
Wei-Chiu Chuang 0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. (#1909) 2020-03-25 16:03:26 -07:00
Inigo Goiri cdcb77a2c5 HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 2020-03-25 11:30:54 -07:00
Jonathan Hung 6ce189c621 YARN-10200. Add number of containers to RMAppManager summary 2020-03-25 10:41:59 -07:00
Arpit Agarwal a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle. 
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
 2020-03-25 10:28:30 -07:00
Chao Sun e3fbdcbc14 HADOOP-16912. Emit per priority RPC queue time and processing time from DecayRpcScheduler. Contributed by Fengnan Li. 2020-03-25 10:21:20 -07:00
RogPodge 2d294bd575
HADOOP-16938. Make non-HA proxy providers pluggable 2020-03-25 08:06:58 -07:00
Sunil G cdb2107066 YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
Nicholas Chammas 25a03bfece
HADOOP-16930. Add hadoop-aws documentation for ProfileCredentialsProvider 
Contributed by Nicholas Chammas.
 2020-03-25 10:39:35 +00:00
Toshihiro Suzuki d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong 2020-03-24 14:50:15 -07:00
Isa Hekmatizadeh d9c4f1129c
HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 2020-03-24 23:17:22 +05:30
Ayush Saxena ea87d60493 HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun. 2020-03-24 22:56:09 +05:30
Akira Ajisaka 4454c6d14b
Revert "HADOOP-16054. Update Dockerfile to use Bionic." (#1911) 
This reverts commit 367833cf41.
 2020-03-24 22:05:33 +09:00
Steve Loughran 28afdce009
Revert ""HADOOP-16910. ABFS Streams to update FileSystem.Statistics counters on IO." 
This reverts commit e2c7ac71b5.

Change-Id: I5b5a93f5a36cdb0c3d56d1b3f747c318f089de20
 2020-03-24 12:11:18 +00:00
Karthik Palanisamy 5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit (#1870) 2020-03-24 15:15:46 +05:30
He Xiaoqiao 5eddc82fb8 HDFS-15088. RBF: Correct annotation typo of RouterPermissionChecker#checkPermission. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 22:41:31 -07:00
Akira Ajisaka f59f6891c8
HDFS-15232. Fix libhdfspp test failures with GCC 7. (#1906) 2020-03-24 11:36:14 +09:00
He Xiaoqiao af64ce2f4a HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-03-23 12:43:51 -07:00
Mehakmeet Singh e2c7ac71b5
ABFS Streams to update FileSystem.Statistics counters on IO. 
Contributed by Mehakmeet Singh
 2020-03-23 13:50:18 +00:00
Prabhu Joseph d312991ead YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue 
Contributed by Peter Bacsko. Reviewed by Manikandan R.
 2020-03-23 18:01:14 +05:30
Ayush Saxena 62c8015683 HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena. 2020-03-22 11:44:18 +05:30
Takanobu Asanuma b74e47e31a HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina. 2020-03-22 01:44:55 +09:00
ishaniahuja 3612317038
HADOOP-16818. ABFS: Combine append+flush calls for blockblob & appendblob 
Contributed by Ishani Ahuja.
 2020-03-20 10:27:41 +00:00
Vinayakumar B f02d5abacd
HADOOP-16927. Update hadoop-thirdparty dependency version to 1.0.0 (#1900). Contributed by Vinayakumar B. 2020-03-20 15:13:07 +05:30
Ayush Saxena f9bb2a8cc5 HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-03-19 22:26:34 +05:30
Ayush Saxena bb41ddaf1e HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena. 2020-03-19 21:23:13 +05:30
Prabhu Joseph f2d3ac2a3f YARN-10034. Remove Allocation Tags from released container from Decommission node 
Contributed by Kyungwan Nam. Reviewed by Adam Antal.
 2020-03-19 12:55:14 +05:30
Ayush Saxena 654db35fa2 HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun. 2020-03-18 21:00:45 +05:30
bilaharith 6ce5f8734f
HADOOP-16920 ABFS: Make list page size configurable. 
Contributed by Bilahari T H.

The page limit is set in "fs.azure.list.max.results"; default value is 500. 

There's currently a limit of 5000 in the store -there are no range checks
in the client code so that limit can be changed on the server without
any need to update the abfs connector.
 2020-03-18 14:14:18 +00:00
Akira Ajisaka 367833cf41
HADOOP-16054. Update Dockerfile to use Bionic. 
Contributed by Akira Ajisaka.
 2020-03-18 13:44:44 +00:00
Wei-Chiu Chuang 096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
 2020-03-18 13:27:13 +00:00
Gabor Bota c91ff8c18f
HADOOP-16858. S3Guard fsck: Add option to remove orphaned entries (#1851). Contributed by Gabor Bota. 
Adding a new feature to S3GuardTool's fsck: -fix. 

Change-Id: I2cdb6601fea1d859b54370046b827ef06eb1107d
 2020-03-18 12:48:52 +01:00
Steve Loughran 8d6373483e
HADOOP-16319. S3A Etag tests fail with default encryption enabled on bucket. 
Contributed by Ben Roling.

ETag values are unpredictable with some S3 encryption algorithms.

Skip ITestS3AMiscOperations tests which make assertions about etags
when default encryption on a bucket is enabled.

When testing with an AWS an account which lacks the privilege
for a call to getBucketEncryption(), we don't skip the tests.
In the event of failure, developers get to expand the
permissions of the account or relax default encryption settings.
 2020-03-17 13:31:48 +00:00
Eric Badger 1975479285 YARN-2710. RM HA tests failed intermittently on trunk. Contributed by Ahmed 
Hussein.
 2020-03-16 22:24:02 +00:00
Wei-Chiu Chuang f197f05cff
HADOOP-16661. Support TLS 1.3 (#1880) 2020-03-16 10:56:30 -07:00
Wanqiang Ji ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters (#1397) 2020-03-16 11:28:36 +09:00
Surendra Singh Lilhore 1d772dc542 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:44:32 +05:30
Ayush Saxena 74780c22eb HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:30:39 +05:30