Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,602 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26602 Commits

Author SHA1 Message Date
GuoPhilipse fe0541b58d
HDFS-16913. Fix flaky some unit tests since they offen timeout (#5377) 
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-14 15:29:38 +08:00
Ayush Saxena 1def35d802
HADOOP-18524. Addendum: Deploy Hadoop trunk version website. (#5389). Contributed by Ayush Saxena. 
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
 2023-02-14 11:05:41 +05:30
Viraj Jasani 90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Ayush Saxena 30f560554d
HADOOP-18524. Deploy Hadoop trunk version website. (#5386). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2023-02-14 00:03:02 +05:30
Tamas Domok e4b5314991
MAPREDUCE-7433. Remove unused mapred/LoggingHttpResponseEncoder.java. (#5388) 2023-02-13 16:21:27 +01:00
Steve Vaughan f42c89dffb
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:31:32 +00:00
hfutatzhanghb f3c4277576
HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (#5276) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-11 08:00:43 +08:00
Owen O'Malley 26fba8701c
HDFS-18324. Fix race condition in closing IPC connections. (#5371) 2023-02-10 17:51:03 +00:00
Tamas Domok 151b71d7af
MAPREDUCE-7431. ShuffleHandler refactor and fix after Netty4 upgrade. (#5311) 2023-02-10 17:40:21 +01:00
Viraj Jasani 17c8cdf63c
HDFS-16907. ADDENDUM: Remove unused variables from testDataNodeMXBeanLastHeartbeats. (#5373) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-10 00:33:20 +05:30
Mehakmeet Singh 9e4f50d8a0
HADOOP-18596. Distcp -update to use modification time while checking for file skip. (#5308) 
Adding toggleable support for modification time during distcp -update between two stores with incompatible checksum comparison.

Contributed by: Mehakmeet Singh <mehakmeet.singh.behl@gmail.com>
 2023-02-09 21:31:09 +05:30
huhaiyang 113a9e40cb
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301) 
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
 2023-02-09 13:28:34 +00:00
huhaiyang d5c046518e
HDFS-16910. Fix incorrectly initializing RandomAccessFile caused flush performance decreased for JN (#5359) 2023-02-09 10:47:57 +08:00
Viraj Jasani 4fcceff535
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) 2023-02-09 10:45:07 +08:00
slfan1989 af20841fb1
YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router. (#5272) 2023-02-08 11:48:38 -08:00
Steve Vaughan 08f58ecf07
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-08 17:15:42 +00:00
He Xiaoqiao 3ba058a894
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 11:19:07 +08:00
He Xiaoqiao 7e919212c4
Revert "HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB." 
This reverts commit eb04ecd29d.
 2023-02-08 11:15:22 +08:00
hfutatzhanghb eb04ecd29d
HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 10:33:28 +08:00
hchaverri d310642626
HDFS-16895. [RBF] NamenodeHeartbeatService should use credentials of logged in user 2023-02-07 18:45:05 +00:00
gardenia 8714403dc7
HADOOP-18621. Resource leak in CryptoOutputStream.close() (#5347) 
When closing we need to wrap the flush() in a try .. finally, otherwise
when flush throws it will stop completion of the remainder of the
close activities and in particular the close of the underlying wrapped
stream object resulting in a resource leak.

Contributed by Colm Dougan
 2023-02-07 12:01:57 +00:00
Viraj Jasani f02c452cf1
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-07 09:54:40 +08:00
slfan1989 a6a9fe17e0
YARN-3657. Federation maintenance mechanisms (simple CLI and command propagation). (#5348) 2023-02-06 11:47:07 -08:00
Steve Vaughan 5f5157ac53
HADOOP-18612. Avoid mixing canonical and non-canonical when performing comparisons (#5339) 
Contributed by Steve Vaughan Jr
 2023-02-06 18:28:29 +00:00
Steve Vaughan aed6fcee5b
HADOOP-18576. Java 11 JavaDoc fails due to missing package comments (#5344) 
Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
 2023-02-06 18:17:57 +00:00
hfutatzhanghb be564f5c20
[HDFS-16903]. Fix javadoc of LightWeightResizableGSet class (#5338) 2023-02-06 13:21:28 +09:00
sunhao 0ae075a2af
HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance (#5147) 2023-02-05 09:33:05 +08:00
jokercurry dad73b76c0
YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc (#5309) 
Co-authored-by: smallzhongfeng <982458633@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-04 10:29:19 +08:00
Viraj Jasani bce388fd3f
HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 09:11:02 +08:00
Ankit Saurabh 22f6d55b71
HADOOP-18246. Reduce lower limit on fs.s3a.prefetch.block.size to 1 byte. (#5120) 
The minimum value of fs.s3a.prefetch.block.size is now 1

Contributed by Ankit Saurabh
 2023-02-02 18:45:21 +00:00
Viraj Jasani ad0cff2f97
HADOOP-18592. Sasl connection failure should log remote address. (#5294) 
Contributed by Viraj Jasani <vjasani@apache.org>

Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2023-02-01 10:15:20 -08:00
Masatake Iwasaki 6d325d9d09 HADOOP-18598. maven site generation doesn't include javadocs. (#5319) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 004121f9cc)
 2023-01-31 10:50:41 +00:00
Masatake Iwasaki a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
huhaiyang 88c8ac750d
HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-01-31 17:46:38 +09:00
Wei-Chiu Chuang 9d47108b50
HADOOP-18584. [NFS GW] Fix regression after netty4 migration. (#5252) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
 2023-01-31 01:17:04 +08:00
Ayush Saxena 952d707240
HADOOP-18604. Add compile platform in the hadoop version output. (#5327). Contributed by Ayush Saxena. 
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-28 14:19:19 +05:30
Szilard Nemeth b677d40ab5 HADOOP-18602. Remove netty3 dependency 2023-01-27 16:32:50 +01:00
Steve Loughran 970ebaeded
HADOOP-17717. Update wildfly openssl to 1.1.3.Final. (#5310) 
Contributed by Wei-Chiu Chuang
 2023-01-27 11:50:17 +00:00
slfan1989 468135a4d9
YARN-11218. [Federation] Add getActivities, getBulkActivities REST APIs for Router. (#5284) 2023-01-26 11:14:05 -08:00
Szilard Nemeth cf1b3711cb YARN-10965. Centralize queue resource calculation based on CapacityVectors. Contributed by Andras Gyori 2023-01-26 19:45:54 +01:00
Szilard Nemeth 815cde9810 YARN-6971. Clean up different ways to create resources. Contributed by Riya Khandelwal 2023-01-25 17:28:29 +01:00
Szilard Nemeth 29f2230cb6 YARN-5607. Document TestContainerResourceUsage#waitForContainerCompletion. Contributed by Susheel Gupta 2023-01-25 15:13:24 +01:00
Szilard Nemeth 8eda456d37 YARN-11404. Add junit5 dependency to hadoop-mapreduce-client-app to fix few unit test failure. Contributed by Susheel Gupta 2023-01-25 15:06:20 +01:00
kevin wan 3b7b79b37a
HADOOP-18582. skip unnecessary cleanup logic in distcp (#5251) 
Co-authored-by: 万康 <mingge@xiaohongshu.com>
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-24 15:49:32 -08:00
slfan1989 3f767a61b1
YARN-8900. [Follow Up] Fix FederationInterceptorREST#invokeConcurrent Inaccurate Order of Subclusters. (#5260) 2023-01-19 17:13:55 -08:00
zhtttylz 72b760130a
HDFS-16893. Standardize the usage of DFSClient debug log (#5303) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-01-19 07:56:41 +08:00
Viraj Jasani 04f3573f6a
HDFS-16891 Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel (#5300) 
Reviewed-by: Stephen O'Donnell <sodonnell@apache.org>
Signed-off-by: Chris Nauroth <cnauroth@apache.org>
 2023-01-18 13:13:41 -08:00
slfan1989 442a5fb285
YARN-11320. [Federation] Add getSchedulerInfo REST APIs for Router. (#5217) 2023-01-17 09:36:19 -08:00
Nikita Eshkeev 4de31123ce
Fix "the the" and friends typos (#5267) 
Signed-off-by: Nikita Eshkeev <neshkeev@yandex.ru>
 2023-01-17 03:33:59 +08:00
PJ Fanning d81d98388c
HADOOP-18575: followup: try to avoid repeatedly hitting exceptions when transformer factories do not support attributes (#5253) 
Part of HADOOP-18469 and the hardening of XML/XSL parsers.
Followup to the main HADOOP-18575 patch, to improve performance when
working with xml/xsl engines which don't support the relevant attributes.

Include this change when backporting.

Contributed by PJ Fanning.
 2023-01-16 13:15:37 +00:00