Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,793 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24793 Commits

Author SHA1 Message Date
jianghuazhu 82cf0aa296 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2022-01-04 14:37:52 -08:00
Szilard Nemeth d3fded12dc YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2022-01-04 14:37:47 -08:00
Dhananjay Badaya 4201cdb44e HADOOP-13500. Synchronizing iteration of Configuration properties object (#3775) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607a4e)
 2022-01-04 14:37:43 -08:00
litao c0ae38c6b4 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2022-01-04 14:37:40 -08:00
litao 24917bae9f HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2022-01-04 14:37:37 -08:00
Ayush Saxena 57c9a8509f HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-04 14:37:26 -08:00
litao f16d9dfc31 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b)
 2022-01-04 14:37:22 -08:00
Viraj Jasani a45df81822 HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3764) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 14:37:19 -08:00
GuoPhilipse 45e6d22462 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2022-01-04 14:37:15 -08:00
Anoop Sam John de27fa0097 HADOOP-17643 WASB : Make metadata checks case insensitive (#3103) 2022-01-04 14:37:12 -08:00
Akira Ajisaka c257abca21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2022-01-04 14:37:08 -08:00
qinyuren 04f0025ad2 HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2022-01-04 14:37:04 -08:00
Szilard Nemeth 52c5bbc993 YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori 2022-01-04 14:37:01 -08:00
Takanobu Asanuma 1e3f94fa3c Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd.
 2022-01-04 14:36:57 -08:00
qinyuren 5214140123 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2022-01-04 14:36:54 -08:00
lzx404243 47436346b4 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c1235)
 2022-01-04 14:36:50 -08:00
Ashutosh Gupta b410289745 YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900)
 2022-01-04 14:36:46 -08:00
LeonGao d02bf2dcce HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2022-01-04 14:36:42 -08:00
Prabhu Joseph 77c1c482ef YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 
(cherry picked from commit 6b4987f09c)
 2022-01-04 14:36:39 -08:00
Takanobu Asanuma 906eb831c6 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2022-01-04 14:36:36 -08:00
huhaiyang e50c9224fe HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2022-01-04 14:36:32 -08:00
huhaiyang 069426f5eb HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411e)
 2022-01-04 14:36:28 -08:00
Steve Loughran 19b99c1ecc HADOOP-17979. Add Interface EtagSource to allow FileStatus subclasses to provide etags (#3633) 
Contributed by Steve Loughran

Change-Id: I596205d788f623114c12962941445432e2036c34
 2022-01-04 14:36:10 -08:00
litao ce0bc7b473 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4)
 2022-01-04 14:35:59 -08:00
litao 30f0629d6e HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2022-01-04 14:35:56 -08:00
GuoPhilipse e449daccf4 YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c9d64bad37)
 2022-01-04 14:35:53 -08:00
Akira Ajisaka 5c18979782 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2022-01-04 14:35:49 -08:00
Viraj Jasani 544dffd179 HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2022-01-04 14:35:43 -08:00
Viraj Jasani c6914b1cb6 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2022-01-04 14:35:40 -08:00
Ayush Saxena 328d3b84df HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2022-01-04 14:35:36 -08:00
Viraj Jasani 3ae8d4ccb9 HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2022-01-04 14:35:33 -08:00
Istvan Fajth 15d3448e25 HADOOP-17975. Fallback to simple auth does not work for a secondary DistributedFileSystem instance. (#3579) 
(cherry picked from commit ae3ba45db5)
 2022-01-04 14:35:29 -08:00
Ashutosh Gupta dd50261219 YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 08f3df3ea2)
 2022-01-04 14:35:11 -08:00
Mukul Kumar Singh 7729505f8a HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2022-01-04 14:34:39 -08:00
litao 8ae7691216 HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2022-01-04 14:34:15 -08:00
litao 486a57fb77 HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2022-01-04 14:30:52 -08:00
GuoPhilipse abc48be5b4 HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2022-01-04 14:30:18 -08:00
litao c236d34b3b HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2022-01-04 14:29:56 -08:00
GuoPhilipse 1e7a46b14d HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2022-01-04 14:29:19 -08:00
Viraj Jasani bf37c64d6f HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2022-01-04 14:28:54 -08:00
litao e11dd859ff HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2022-01-04 14:28:28 -08:00
Shubham Gupta 8d9d3ffd47 YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2022-01-04 14:28:12 -08:00
Viraj Jasani 1e6e8a19ab HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2022-01-04 14:27:52 -08:00
Steve Loughran 5314f2e7a3 HADOOP-18002. ABFS rename idempotency broken -remove recovery (#3641) 
Cut modtime-based rename recovery as object modification time
is not updated during rename operation.
Applications will have to use etag API of HADOOP-17979
and implement it themselves.

Why not do the HEAD and etag recovery in ABFS client?
Cuts the IO capacity in half so kills job commit performance.
The manifest committer of MAPREDUCE-7341 will do this recovery
and act as the reference implementation of the algorithm.

Contributed by: Steve Loughran

Change-Id: I810054c9fd05041dac552f13d31fb15d7524721b
 2022-01-04 14:27:22 -08:00
Abhishek Das 41f38610f7 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 
(cherry picked from commit 54a1d78e16)
 2022-01-04 14:26:58 -08:00
litao 3cbfef9130 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2022-01-04 14:26:17 -08:00
GuoPhilipse dd9fd3b01d HADOOP-18026. Fix default value of Magic committer (#3723) 
Contributed by guophilipse

Change-Id: If915623c76619dd3d3b3bdf989688fa13e56fec1
 2022-01-04 14:21:00 -08:00
bitterfox ef462b21bf HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2022-01-04 14:20:25 -08:00
Cheng Pan bbf4f9a5d2 YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4db27acf06)
(cherry picked from commit 7b51e6824d)
 2022-01-04 13:13:49 +09:00
Ayush Saxena d8b7a52362 HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53249a40db)
 2022-01-04 13:12:29 +09:00