Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,777 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24777 Commits

Author SHA1 Message Date
Ashutosh Gupta b410289745 YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900)
 2022-01-04 14:36:46 -08:00
LeonGao d02bf2dcce HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2022-01-04 14:36:42 -08:00
Prabhu Joseph 77c1c482ef YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 
(cherry picked from commit 6b4987f09c)
 2022-01-04 14:36:39 -08:00
Takanobu Asanuma 906eb831c6 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2022-01-04 14:36:36 -08:00
huhaiyang e50c9224fe HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2022-01-04 14:36:32 -08:00
huhaiyang 069426f5eb HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411e)
 2022-01-04 14:36:28 -08:00
Steve Loughran 19b99c1ecc HADOOP-17979. Add Interface EtagSource to allow FileStatus subclasses to provide etags (#3633) 
Contributed by Steve Loughran

Change-Id: I596205d788f623114c12962941445432e2036c34
 2022-01-04 14:36:10 -08:00
litao ce0bc7b473 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4)
 2022-01-04 14:35:59 -08:00
litao 30f0629d6e HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2022-01-04 14:35:56 -08:00
GuoPhilipse e449daccf4 YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c9d64bad37)
 2022-01-04 14:35:53 -08:00
Akira Ajisaka 5c18979782 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2022-01-04 14:35:49 -08:00
Viraj Jasani 544dffd179 HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2022-01-04 14:35:43 -08:00
Viraj Jasani c6914b1cb6 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2022-01-04 14:35:40 -08:00
Ayush Saxena 328d3b84df HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2022-01-04 14:35:36 -08:00
Viraj Jasani 3ae8d4ccb9 HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2022-01-04 14:35:33 -08:00
Istvan Fajth 15d3448e25 HADOOP-17975. Fallback to simple auth does not work for a secondary DistributedFileSystem instance. (#3579) 
(cherry picked from commit ae3ba45db5)
 2022-01-04 14:35:29 -08:00
Ashutosh Gupta dd50261219 YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 08f3df3ea2)
 2022-01-04 14:35:11 -08:00
Mukul Kumar Singh 7729505f8a HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2022-01-04 14:34:39 -08:00
litao 8ae7691216 HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2022-01-04 14:34:15 -08:00
litao 486a57fb77 HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2022-01-04 14:30:52 -08:00
GuoPhilipse abc48be5b4 HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2022-01-04 14:30:18 -08:00
litao c236d34b3b HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2022-01-04 14:29:56 -08:00
GuoPhilipse 1e7a46b14d HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2022-01-04 14:29:19 -08:00
Viraj Jasani bf37c64d6f HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2022-01-04 14:28:54 -08:00
litao e11dd859ff HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2022-01-04 14:28:28 -08:00
Shubham Gupta 8d9d3ffd47 YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2022-01-04 14:28:12 -08:00
Viraj Jasani 1e6e8a19ab HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2022-01-04 14:27:52 -08:00
Steve Loughran 5314f2e7a3 HADOOP-18002. ABFS rename idempotency broken -remove recovery (#3641) 
Cut modtime-based rename recovery as object modification time
is not updated during rename operation.
Applications will have to use etag API of HADOOP-17979
and implement it themselves.

Why not do the HEAD and etag recovery in ABFS client?
Cuts the IO capacity in half so kills job commit performance.
The manifest committer of MAPREDUCE-7341 will do this recovery
and act as the reference implementation of the algorithm.

Contributed by: Steve Loughran

Change-Id: I810054c9fd05041dac552f13d31fb15d7524721b
 2022-01-04 14:27:22 -08:00
Abhishek Das 41f38610f7 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 
(cherry picked from commit 54a1d78e16)
 2022-01-04 14:26:58 -08:00
litao 3cbfef9130 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2022-01-04 14:26:17 -08:00
GuoPhilipse dd9fd3b01d HADOOP-18026. Fix default value of Magic committer (#3723) 
Contributed by guophilipse

Change-Id: If915623c76619dd3d3b3bdf989688fa13e56fec1
 2022-01-04 14:21:00 -08:00
bitterfox ef462b21bf HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2022-01-04 14:20:25 -08:00
Cheng Pan bbf4f9a5d2 YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4db27acf06)
(cherry picked from commit 7b51e6824d)
 2022-01-04 13:13:49 +09:00
Ayush Saxena d8b7a52362 HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53249a40db)
 2022-01-04 13:12:29 +09:00
Siyao Meng d4794dd3b2 HADOOP-18048. [branch-3.3] Dockerfile_aarch64 build fails with fatal error: Python.h: No such file or directory (#3801) 2021-12-14 17:58:14 -08:00
smarthan 3e24bc669a HADOOP-18023. Allow cp command to run with multi threads. (#3721) 
(cherry picked from commit 932a78fe38)
 2021-11-29 14:36:49 +00:00
smarthan da57ee936b HADOOP-17998. Allow get command to run with multi threads. (#3645) 
(cherry picked from commit 63018dc73f)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java

(cherry picked from commit cbb3ba135c)
 2021-11-22 12:21:01 +00:00
Chao Sun b55edde707 Also update hadoop.version to 3.3.2 2021-11-16 16:14:22 -08:00
Chao Sun ce74635dd4 Preparing for 3.3.1 release 2021-11-16 16:07:17 -08:00
Neil c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
 2021-11-16 22:59:25 +08:00
Renukaprasad C 3bb4a09295
HADOOP-17946. Upgrade commons-lang to 3.12.0 (#3575) 
(cherry picked from commit b923fa7a1c)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein 25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
litao 340dee4469
HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:12:13 +05:30
litao 421013825f
HADOOP-18005. Correct log format for LdapGroupsMapping (#3647). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:01:09 +05:30
GuoPhilipse b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30
Wei-Chiu Chuang 5333e872e2
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
 2021-11-12 14:32:02 +08:00
litao c4e8826194 HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:25:40 +09:00
litao 808da62cfa HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:23:58 +09:00
litao 0db04fab20 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:18:18 +09:00
jianghuazhu 46e69715b5 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:08 +09:00