Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,467 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

26467 Commits

Author SHA1 Message Date
xihuanbanku 60ed024200
[YARN-11443]"State" column in YARN RM page bug 
"State" column in YARN RM web page shows "undefined" while using fairscheduler.
 2023-02-19 20:00:29 +08:00
zhtttylz a3b500d046
HDFS-16916. Improve the use of JUnit Test in DFSClient (#5404) 
Co-authored-by: Zhtttylz <hualong.z@hotmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:17:37 +08:00
slfan1989 7e486038ea
YARN-11439. Fix Typo of hadoop-yarn-ui README.md. (#5405) 
Co-authored-by: Shilun Fan <slfan1989@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-18 13:14:01 +08:00
Arnout Engelen 02fd87a4d8
HADOOP-18627. Add stronger wording in 'secure mode' introduction (#5406) 
Make it more clear that when deploying Hadoop 'secure mode' is generally not optional.

Contributed by Arnout Engelen
 2023-02-17 16:30:41 +00:00
Steve Loughran 10e7ca481c
YARN-11441. Revert YARN-10495. 
This reverts commit 7d3c8ef606.
 2023-02-17 15:05:06 +00:00
Bryan Beaudreault 7e19bc31b6
HADOOP-18215. Enhance WritableName to be able to return aliases for classes that use serializers (#4215) 2023-02-16 18:13:25 +00:00
Mehakmeet Singh 7a0903b743
HADOOP-18633. fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip (#5401) 
Contributed by: Mehakmeet Singh
 2023-02-16 10:09:06 +05:30
hfutatzhanghb 723535b788
HDFS-16914. Add some logs for updateBlockForPipeline RPC. (#5381) 
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-16 09:25:50 +08:00
slfan1989 a5f48eacca
YARN-11425. [Federation] Router Supports SubClusterCleaner. (#5326) 2023-02-15 14:40:34 -08:00
slfan1989 c3706597a3
YARN-11349. [Federation] Router Support DelegationToken With SQL. (#5244) 2023-02-15 14:38:41 -08:00
Ankit Saurabh f4f2793f3b
HADOOP-18351. Reduce excess logging of errors during S3A prefetching reads (#5274) 
Contributed by Ankit Saurabh
 2023-02-15 18:28:42 +00:00
Zita Dombi 4cbe19f3a2
HDFS-16761. Namenode UI for Datanodes page not loading if any data node is down (#5390) 2023-02-15 16:16:04 +00:00
Steve Loughran d56977e909
HADOOP-18470. More in the 3.3.5 index.html about security (#5383) 
Expands on the comments in cluster config to tell people
they shouldn't be running a cluster without a private VLAN
in cloud, that Knox is good here, and unsecured clusters
without a VLAN are just computation-as-a-service to crypto miners

Contributed by Steve Loughran
 2023-02-14 17:22:59 +00:00
SimhadriGovindappa e2ab35084a
HADOOP-18630. Add gh-pages in asf.yaml to deploy the current trunk doc (#5393). Contributed by Simhadri Govindappa. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-14 18:13:29 +05:30
Viraj Jasani 021fcc6c5e
HADOOP-18628. IPC Server Connection should log host name before returning VersionMismatch error (#5385) 
Contributed by Viraj Jasani
 2023-02-14 11:48:48 +00:00
GuoPhilipse fe0541b58d
HDFS-16913. Fix flaky some unit tests since they offen timeout (#5377) 
Co-authored-by: gf13871 <gf13871@ly.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-14 15:29:38 +08:00
Ayush Saxena 1def35d802
HADOOP-18524. Addendum: Deploy Hadoop trunk version website. (#5389). Contributed by Ayush Saxena. 
Reviewed-by: Vinayakumar B <vinayakumarb@apache.org>
 2023-02-14 11:05:41 +05:30
Viraj Jasani 90de1ff151
HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future (#5315) 2023-02-14 03:24:06 +08:00
Ayush Saxena 30f560554d
HADOOP-18524. Deploy Hadoop trunk version website. (#5386). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2023-02-14 00:03:02 +05:30
Tamas Domok e4b5314991
MAPREDUCE-7433. Remove unused mapred/LoggingHttpResponseEncoder.java. (#5388) 2023-02-13 16:21:27 +01:00
Steve Vaughan f42c89dffb
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5372) 
This is a followup to the original patch, 08f58ecf07, which it supercedes
* Switch to org.apache.hadoop.io.IOUtils and closeStream.
* Use cleanupWithLogger to include error logging

Contributed by Steve Vaughan Jr
 2023-02-13 14:31:32 +00:00
hfutatzhanghb f3c4277576
HDFS-16882. RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath (#5276) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-11 08:00:43 +08:00
Owen O'Malley 26fba8701c
HDFS-18324. Fix race condition in closing IPC connections. (#5371) 2023-02-10 17:51:03 +00:00
Tamas Domok 151b71d7af
MAPREDUCE-7431. ShuffleHandler refactor and fix after Netty4 upgrade. (#5311) 2023-02-10 17:40:21 +01:00
Viraj Jasani 17c8cdf63c
HDFS-16907. ADDENDUM: Remove unused variables from testDataNodeMXBeanLastHeartbeats. (#5373) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2023-02-10 00:33:20 +05:30
Mehakmeet Singh 9e4f50d8a0
HADOOP-18596. Distcp -update to use modification time while checking for file skip. (#5308) 
Adding toggleable support for modification time during distcp -update between two stores with incompatible checksum comparison.

Contributed by: Mehakmeet Singh <mehakmeet.singh.behl@gmail.com>
 2023-02-09 21:31:09 +05:30
huhaiyang 113a9e40cb
HADOOP-18625. Fix method name of RPC.Builder#setnumReaders (#5301) 
Changes method name of RPC.Builder#setnumReaders to setNumReaders()

The original method is still there, just marked deprecated.
It is the one which should be used when working with older branches.

Contributed by Haiyang Hu
 2023-02-09 13:28:34 +00:00
huhaiyang d5c046518e
HDFS-16910. Fix incorrectly initializing RandomAccessFile caused flush performance decreased for JN (#5359) 2023-02-09 10:47:57 +08:00
Viraj Jasani 4fcceff535
HADOOP-18620 Avoid using grizzly-http-* APIs (#5356) 2023-02-09 10:45:07 +08:00
slfan1989 af20841fb1
YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router. (#5272) 2023-02-08 11:48:38 -08:00
Steve Vaughan 08f58ecf07
HDFS-16904. Close webhdfs during TestSymlinkHdfs teardown (#5342) 
Contributed by Steve Vaughan Jr
 2023-02-08 17:15:42 +00:00
He Xiaoqiao 3ba058a894
HDFS-16898. Remove write lock for processCommandFromActor of DataNode to reduce impact on heartbeat (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 11:19:07 +08:00
He Xiaoqiao 7e919212c4
Revert "HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB." 
This reverts commit eb04ecd29d.
 2023-02-08 11:15:22 +08:00
hfutatzhanghb eb04ecd29d
HDFS-16898. Make write lock fine-grain in method processCommandFromActor (#5330). Contributed by ZhangHB. 
Reviewed-by: zhangshuyan <zqingchai@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2023-02-08 10:33:28 +08:00
hchaverri d310642626
HDFS-16895. [RBF] NamenodeHeartbeatService should use credentials of logged in user 2023-02-07 18:45:05 +00:00
gardenia 8714403dc7
HADOOP-18621. Resource leak in CryptoOutputStream.close() (#5347) 
When closing we need to wrap the flush() in a try .. finally, otherwise
when flush throws it will stop completion of the remainder of the
close activities and in particular the close of the underlying wrapped
stream object resulting in a resource leak.

Contributed by Colm Dougan
 2023-02-07 12:01:57 +00:00
Viraj Jasani f02c452cf1
HDFS-16907. Add LastHeartbeatResponseTime for BP service actor (#5349) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-07 09:54:40 +08:00
slfan1989 a6a9fe17e0
YARN-3657. Federation maintenance mechanisms (simple CLI and command propagation). (#5348) 2023-02-06 11:47:07 -08:00
Steve Vaughan 5f5157ac53
HADOOP-18612. Avoid mixing canonical and non-canonical when performing comparisons (#5339) 
Contributed by Steve Vaughan Jr
 2023-02-06 18:28:29 +00:00
Steve Vaughan aed6fcee5b
HADOOP-18576. Java 11 JavaDoc fails due to missing package comments (#5344) 
Add JavaDoc comments to package-info.java to avoid errors resulting from the use of Hadoop annotations.

Contributed by Steve Vaughan Jr
 2023-02-06 18:17:57 +00:00
hfutatzhanghb be564f5c20
[HDFS-16903]. Fix javadoc of LightWeightResizableGSet class (#5338) 2023-02-06 13:21:28 +09:00
sunhao 0ae075a2af
HDFS-16848. RBF: Improve StateStoreZooKeeperImpl performance (#5147) 2023-02-05 09:33:05 +08:00
jokercurry dad73b76c0
YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc (#5309) 
Co-authored-by: smallzhongfeng <982458633@qq.com>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Shilun Fan <slfan1989@apache.org>
 2023-02-04 10:29:19 +08:00
Viraj Jasani bce388fd3f
HDFS-16902 Add Namenode status to BPServiceActor metrics and improve logging in offerservice (#5334) 
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Shilun Fan <slfan1989@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
 2023-02-03 09:11:02 +08:00
Ankit Saurabh 22f6d55b71
HADOOP-18246. Reduce lower limit on fs.s3a.prefetch.block.size to 1 byte. (#5120) 
The minimum value of fs.s3a.prefetch.block.size is now 1

Contributed by Ankit Saurabh
 2023-02-02 18:45:21 +00:00
Viraj Jasani ad0cff2f97
HADOOP-18592. Sasl connection failure should log remote address. (#5294) 
Contributed by Viraj Jasani <vjasani@apache.org>

Signed-off-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2023-02-01 10:15:20 -08:00
Masatake Iwasaki 6d325d9d09 HADOOP-18598. maven site generation doesn't include javadocs. (#5319) 
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit 004121f9cc)
 2023-01-31 10:50:41 +00:00
Masatake Iwasaki a70f84098f
HADOOP-18601. Fix build failure with docs profile. (#5331) 
Reviewed-by: Steve Loughran <stevel@apache.org>
 2023-01-31 19:44:19 +09:00
huhaiyang 88c8ac750d
HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile (#5296) 
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2023-01-31 17:46:38 +09:00
Wei-Chiu Chuang 9d47108b50
HADOOP-18584. [NFS GW] Fix regression after netty4 migration. (#5252) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
 2023-01-31 01:17:04 +08:00