Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,732 Commits 326 Branches 371 Tags
Commit Graph

24732 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
qinyuren
04f0025ad2 HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2022-01-04 14:37:04 -08:00
Szilard Nemeth
52c5bbc993 YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori 2022-01-04 14:37:01 -08:00
Takanobu Asanuma
1e3f94fa3c Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd533f0319e31357f037939339a6c156.
 2022-01-04 14:36:57 -08:00
qinyuren
5214140123 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043c063f548b81ad431d9452f010b18f7)
 2022-01-04 14:36:54 -08:00
lzx404243
47436346b4 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c123544e1c1cbee759a11879af3b19260b3)
 2022-01-04 14:36:50 -08:00
Ashutosh Gupta
b410289745 YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900aa143f6db281eb22928965f3b2754f)
 2022-01-04 14:36:46 -08:00
LeonGao
d02bf2dcce HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661d694fb0775f4f84333c1317f4e6899)
 2022-01-04 14:36:42 -08:00
Prabhu Joseph
77c1c482ef YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 
(cherry picked from commit 6b4987f09cbc53e00e85cb82ad97505858cada11)
 2022-01-04 14:36:39 -08:00
Takanobu Asanuma
906eb831c6 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106c03d041a0d280ea727e3f252460163)
 2022-01-04 14:36:36 -08:00
huhaiyang
e50c9224fe HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b402b4bdf5b7771f5124790fc866965)
 2022-01-04 14:36:32 -08:00
huhaiyang
069426f5eb HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411ebee11372314e82d7ea0606c348d014)
 2022-01-04 14:36:28 -08:00
Steve Loughran
19b99c1ecc HADOOP-17979. Add Interface EtagSource to allow FileStatus subclasses to provide etags (#3633) 
Contributed by Steve Loughran

Change-Id: I596205d788f623114c12962941445432e2036c34
 2022-01-04 14:36:10 -08:00
litao
ce0bc7b473 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4240347675e20df0ce047245f465c7b)
 2022-01-04 14:35:59 -08:00
litao
30f0629d6e HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82208b6b04551e637414d0d9d64a4173)
 2022-01-04 14:35:56 -08:00
GuoPhilipse
e449daccf4 YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c9d64bad37fbfbaa77481f67338b39a029d7dc51)
 2022-01-04 14:35:53 -08:00
Akira Ajisaka
5c18979782 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794657a77550318c83d3d7303d745740b.
 2022-01-04 14:35:49 -08:00
Viraj Jasani
544dffd179 HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523941622a140fd877f06e9b59f48c48b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2022-01-04 14:35:43 -08:00
Viraj Jasani
c6914b1cb6 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b619159dc4bf185ad53b950bdf5de52a)
 2022-01-04 14:35:40 -08:00
Ayush Saxena
328d3b84df HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2022-01-04 14:35:36 -08:00
Viraj Jasani
3ae8d4ccb9 HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b38129bcef687506742b3d7d35e8e4947a9)
 2022-01-04 14:35:33 -08:00
Istvan Fajth
15d3448e25 HADOOP-17975. Fallback to simple auth does not work for a secondary DistributedFileSystem instance. (#3579) 
(cherry picked from commit ae3ba45db58467ce57b0a440e236fd80f6be9ec6)
 2022-01-04 14:35:29 -08:00
Ashutosh Gupta
dd50261219 YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 08f3df3ea2afcce3973749199269a032df3732e0)
 2022-01-04 14:35:11 -08:00
Mukul Kumar Singh
7729505f8a HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1f708b7c3a0701886fa63068327e9b3)
 2022-01-04 14:34:39 -08:00
litao
8ae7691216 HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babbcaa0560c11e2065cff3fff0bf328c5)
 2022-01-04 14:34:15 -08:00
litao
486a57fb77 HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f978dc74f03a0dd5551904f8ec40ccf82)
 2022-01-04 14:30:52 -08:00
GuoPhilipse
abc48be5b4 HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc1799e95feced466411743e9f9fddd4)
 2022-01-04 14:30:18 -08:00
litao
c236d34b3b HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e90f7ca7cf6d1418893340338eb8760f)
 2022-01-04 14:29:56 -08:00
GuoPhilipse
1e7a46b14d HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b1e0c8bdc3f4170295771d75c5ed776)
 2022-01-04 14:29:19 -08:00
Viraj Jasani
bf37c64d6f HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e118711852c58e9f0ebc3493fcb5ebeb)
 2022-01-04 14:28:54 -08:00
litao
e11dd859ff HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b4ab2c9c26a1efe7479c31fd1de80d5)
 2022-01-04 14:28:28 -08:00
Shubham Gupta
8d9d3ffd47 YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069c7d35b4638af3f2299a248eeca3923)
 2022-01-04 14:28:12 -08:00
Viraj Jasani
1e6e8a19ab HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5de956e570a81dcd4890f028965be33a)
 2022-01-04 14:27:52 -08:00
Steve Loughran
5314f2e7a3 HADOOP-18002. ABFS rename idempotency broken -remove recovery (#3641) 
Cut modtime-based rename recovery as object modification time
is not updated during rename operation.
Applications will have to use etag API of HADOOP-17979
and implement it themselves.

Why not do the HEAD and etag recovery in ABFS client?
Cuts the IO capacity in half so kills job commit performance.
The manifest committer of MAPREDUCE-7341 will do this recovery
and act as the reference implementation of the algorithm.

Contributed by: Steve Loughran

Change-Id: I810054c9fd05041dac552f13d31fb15d7524721b
 2022-01-04 14:27:22 -08:00
Abhishek Das
41f38610f7 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 
(cherry picked from commit 54a1d78e16533e286455de62a545ee75cbc1eff5)
 2022-01-04 14:26:58 -08:00
litao
3cbfef9130 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2022-01-04 14:26:17 -08:00
GuoPhilipse
dd9fd3b01d HADOOP-18026. Fix default value of Magic committer (#3723) 
Contributed by guophilipse

Change-Id: If915623c76619dd3d3b3bdf989688fa13e56fec1
 2022-01-04 14:21:00 -08:00
bitterfox
ef462b21bf HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93e8319560b633360f30ac84fc48e403)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2022-01-04 14:20:25 -08:00
Cheng Pan
bbf4f9a5d2 YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4db27acf06cb8526e3aa84a18904baa7ddd2c7c8)
(cherry picked from commit 7b51e6824d632f56dfaafbce8501074da14691eb)
 2022-01-04 13:13:49 +09:00
Ayush Saxena
d8b7a52362 HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53249a40dbe7b4dae7da0a42d35c4601b043ebbe)
 2022-01-04 13:12:29 +09:00
Siyao Meng
d4794dd3b2 HADOOP-18048. [branch-3.3] Dockerfile_aarch64 build fails with fatal error: Python.h: No such file or directory (#3801) 2021-12-14 17:58:14 -08:00
smarthan
3e24bc669a HADOOP-18023. Allow cp command to run with multi threads. (#3721) 
(cherry picked from commit 932a78fe38b34a923f6852a1a19482075806ecba)
release-3.3.2-RC0 		2021-11-29 14:36:49 +00:00
smarthan
da57ee936b HADOOP-17998. Allow get command to run with multi threads. (#3645) 
(cherry picked from commit 63018dc73f4d29632e93be08d035ab9a7e73531c)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java

(cherry picked from commit cbb3ba135cfa3a9bc9f444e8b4a7875758721a3c)
 2021-11-22 12:21:01 +00:00
Chao Sun
b55edde707 Also update hadoop.version to 3.3.2 2021-11-16 16:14:22 -08:00
Chao Sun
ce74635dd4 Preparing for 3.3.1 release 2021-11-16 16:07:17 -08:00
Neil
c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36b9794f4b1af8b9a7d8d0055914a094)
 2021-11-16 22:59:25 +08:00
Renukaprasad C
3bb4a09295
HADOOP-17946. Upgrade commons-lang to 3.12.0 (#3575) 
(cherry picked from commit b923fa7a1c74ffd5859803e6e2571d367d066efb)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein
25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
litao
340dee4469
HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:12:13 +05:30
litao
421013825f
HADOOP-18005. Correct log format for LdapGroupsMapping (#3647). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:01:09 +05:30
GuoPhilipse
b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30