Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,719 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24719 Commits

Author SHA1 Message Date
litao 30f0629d6e HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2022-01-04 14:35:56 -08:00
GuoPhilipse e449daccf4 YARN-11007. Correct words in YARN documents (#3680) 
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c9d64bad37)
 2022-01-04 14:35:53 -08:00
Akira Ajisaka 5c18979782 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2022-01-04 14:35:49 -08:00
Viraj Jasani 544dffd179 HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2022-01-04 14:35:43 -08:00
Viraj Jasani c6914b1cb6 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2022-01-04 14:35:40 -08:00
Ayush Saxena 328d3b84df HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2022-01-04 14:35:36 -08:00
Viraj Jasani 3ae8d4ccb9 HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2022-01-04 14:35:33 -08:00
Istvan Fajth 15d3448e25 HADOOP-17975. Fallback to simple auth does not work for a secondary DistributedFileSystem instance. (#3579) 
(cherry picked from commit ae3ba45db5)
 2022-01-04 14:35:29 -08:00
Ashutosh Gupta dd50261219 YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 08f3df3ea2)
 2022-01-04 14:35:11 -08:00
Mukul Kumar Singh 7729505f8a HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2022-01-04 14:34:39 -08:00
litao 8ae7691216 HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2022-01-04 14:34:15 -08:00
litao 486a57fb77 HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2022-01-04 14:30:52 -08:00
GuoPhilipse abc48be5b4 HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2022-01-04 14:30:18 -08:00
litao c236d34b3b HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2022-01-04 14:29:56 -08:00
GuoPhilipse 1e7a46b14d HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2022-01-04 14:29:19 -08:00
Viraj Jasani bf37c64d6f HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2022-01-04 14:28:54 -08:00
litao e11dd859ff HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2022-01-04 14:28:28 -08:00
Shubham Gupta 8d9d3ffd47 YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2022-01-04 14:28:12 -08:00
Viraj Jasani 1e6e8a19ab HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2022-01-04 14:27:52 -08:00
Steve Loughran 5314f2e7a3 HADOOP-18002. ABFS rename idempotency broken -remove recovery (#3641) 
Cut modtime-based rename recovery as object modification time
is not updated during rename operation.
Applications will have to use etag API of HADOOP-17979
and implement it themselves.

Why not do the HEAD and etag recovery in ABFS client?
Cuts the IO capacity in half so kills job commit performance.
The manifest committer of MAPREDUCE-7341 will do this recovery
and act as the reference implementation of the algorithm.

Contributed by: Steve Loughran

Change-Id: I810054c9fd05041dac552f13d31fb15d7524721b
 2022-01-04 14:27:22 -08:00
Abhishek Das 41f38610f7 HADOOP-17999. No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem. Contributed by Abhishek Das. (#3639) 
(cherry picked from commit 54a1d78e16)
 2022-01-04 14:26:58 -08:00
litao 3cbfef9130 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2022-01-04 14:26:17 -08:00
GuoPhilipse dd9fd3b01d HADOOP-18026. Fix default value of Magic committer (#3723) 
Contributed by guophilipse

Change-Id: If915623c76619dd3d3b3bdf989688fa13e56fec1
 2022-01-04 14:21:00 -08:00
bitterfox ef462b21bf HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2022-01-04 14:20:25 -08:00
Cheng Pan bbf4f9a5d2 YARN-11053. AuxService should not use class name as default system classes (#3816) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4db27acf06)
(cherry picked from commit 7b51e6824d)
 2022-01-04 13:13:49 +09:00
Ayush Saxena d8b7a52362 HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53249a40db)
 2022-01-04 13:12:29 +09:00
Siyao Meng d4794dd3b2 HADOOP-18048. [branch-3.3] Dockerfile_aarch64 build fails with fatal error: Python.h: No such file or directory (#3801) 2021-12-14 17:58:14 -08:00
smarthan 3e24bc669a HADOOP-18023. Allow cp command to run with multi threads. (#3721) 
(cherry picked from commit 932a78fe38)
 2021-11-29 14:36:49 +00:00
smarthan da57ee936b HADOOP-17998. Allow get command to run with multi threads. (#3645) 
(cherry picked from commit 63018dc73f)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java

(cherry picked from commit cbb3ba135c)
 2021-11-22 12:21:01 +00:00
Chao Sun b55edde707 Also update hadoop.version to 3.3.2 2021-11-16 16:14:22 -08:00
Chao Sun ce74635dd4 Preparing for 3.3.1 release 2021-11-16 16:07:17 -08:00
Neil c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
 2021-11-16 22:59:25 +08:00
Renukaprasad C 3bb4a09295
HADOOP-17946. Upgrade commons-lang to 3.12.0 (#3575) 
(cherry picked from commit b923fa7a1c)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein 25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
litao 340dee4469
HDFS-16319. Add metrics doc for ReadLockLongHoldCount and WriteLockLongHoldCount (#3653). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:12:13 +05:30
litao 421013825f
HADOOP-18005. Correct log format for LdapGroupsMapping (#3647). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:01:09 +05:30
GuoPhilipse b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30
Wei-Chiu Chuang 5333e872e2
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
 2021-11-12 14:32:02 +08:00
litao c4e8826194 HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:25:40 +09:00
litao 808da62cfa HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:23:58 +09:00
litao 0db04fab20 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:18:18 +09:00
jianghuazhu 46e69715b5 HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:08 +09:00
Gautham B A e392ce7a73
HADOOP-17988. Disable JIRA plugin for YETUS on Hadoop (#3610) 2021-11-09 22:05:10 +05:30
litao 26b7bc75ed HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:09:05 +09:00
Steve Loughran 7b632dd22b Revert "HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3341)" 
This reverts commit 0379aebafe.
 2021-11-05 14:22:07 +00:00
sumangala-patki 689dd7bf17
HADOOP-17863. ABFS: Fix compiler deprecation warning in TextFileBasedIdentityHandler (#3332) 
Closes #3332

Contributed by Sumangala Patki

Change-Id: I2abd33bd62bb734a431cccfc50a52bdeb2bf7db6
 2021-11-05 12:55:45 +00:00
Jinhu Wu 0557da6820 HADOOP-17374. support listObjectV2 (#3587) 
(cherry picked from commit a9c51ea57d)
 2021-11-04 21:45:04 -07:00
sumangala-patki 0379aebafe
HADOOP-17873. ABFS: Fix transient failures in ITestAbfsStreamStatistics and ITestAbfsRestOperationException (#3341) 
Addresses transient failures in the following test classes:

* ITestAbfsStreamStatistics: Uses a filesystem level static instance to record read/write statistics, which also tracks these operations in other tests running in parallel. Marked for sequential-only run to avoid transient failure

* ITestAbfsRestOperationException: The use of a static member to track retry count causes transient failures when two tests of this class happen to run together. Switch to non-static variable for assertions on retry count

closes #3341

Contributed by Sumangala Patki

Change-Id: Ied4dec35c81e94efe5f999acae4bb8fde278202e
 2021-11-04 15:57:42 +00:00
Steve Loughran a68671eaf7
HADOOP-17928. Syncable: S3A to warn and downgrade (#3585) 
This switches the default behavior of S3A output streams
to warning that Syncable.hsync() or hflush() have been
called; it's not considered an error unless the defaults
are overridden.

This avoids breaking applications which call the APIs,
at the risk of people trying to use S3 as a safe store
of streamed data (HBase WALs, audit logs etc).

Contributed by Steve Loughran.

Change-Id: I0a02ec1e622343619f147f94158c18928a73a885
 2021-11-04 14:41:42 +00:00
jianghuazhu 4d04efb900 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:50:31 +09:00