Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,533 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

24533 Commits

Author SHA1 Message Date
litao 7cb91db575 HDFS-16122. Fix DistCpContext#toString() (#3191). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-10 13:56:36 +05:30
Artem Smotrakov c18ad9a007 HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada)
 2021-07-10 13:43:31 +09:00
Akira Ajisaka 6cc9549499 HADOOP-17794. Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS (#3190) 
(cherry picked from commit 6ac5d8a1ed)
 2021-07-09 22:50:18 +09:00
S O'Donnell 705884b87a HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76)
 2021-07-09 10:19:47 +01:00
bshashikant ab939814a3 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9a)
 2021-07-09 09:17:10 +01:00
Rafal Wojdyla e3fb63f33f
HADOOP-17402. Add GCS config to the core-site (#2638) 
Contributed by Rafal Wojdyla
 2021-07-07 22:43:31 +01:00
Eric Payne e395711164 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 
(cherry picked from commit 7581413156)
 2021-07-07 20:57:32 +00:00
Masatake Iwasaki 99cff01e38
HADOOP-17775. Remove JavaScript package from Docker environment. (#3182) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-07 17:49:19 +09:00
liangxs 24b780820c HADOOP-17749. Remove lock contention in SelectorPool of SocketIOWithTimeout (#3080) 
(cherry picked from commit a5db6831bc)
 2021-07-07 09:41:11 +08:00
Mukund Thakur e8f9af6f2a
HADOOP-17250 Lot of short reads can be merged with readahead. (#3110) 
Introducing fs.azure.readahead.range parameter which can be set by the user.
Data will be populated in buffer for random reads as well which leads to fewer
remote calls.

This patch also changes the seek implementation to perform a lazy seek. The
actual seek is done when a read is initiated and data is not present in the buffer else
data is returned from the buffer thus reducing the number of remote storage calls.

Contributed By: Mukund Thakur

Change-Id: Ib920eedd0087caa150afa4d4c23e89df56b29e83
 2021-07-05 11:23:32 +01:00
Viraj Jasani 2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf)
 2021-07-05 13:42:51 +09:00
litao aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e4)
 2021-07-05 08:15:10 +09:00
Mehakmeet Singh f1a14df9e6
HADOOP-17774. S3A bytesRead FS statistic showing twice the correct value (#3144) 
Contributed by: Mehakmeet Singh

Change-Id: I3302654ca36474a5f399aa848f88bce4587022d8
 2021-07-02 14:13:26 +01:00
Viraj Jasani bf6ed21104 HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:59:25 -07:00
Wanqiang Ji 628b08db63
HADOOP-17777. Update clover-maven-plugin version from 3.3.0 to 4.4.1 (#3152) 
(cherry picked from commit c005195b78)
 2021-06-30 09:35:36 +08:00
litao 4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 
(cherry picked from commit 95454d821c)
 2021-06-29 18:15:56 +08:00
Viraj Jasani 8538881710
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073)
 2021-06-29 18:13:00 +08:00
Jim Brennan 47b3939009 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 
(cherry picked from commit dc6f456e95)
 2021-06-28 14:55:26 +00:00
Jim Brennan 75f8198aa8 YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 
(cherry picked from commit 7c7d02edbd)
 2021-06-25 20:36:41 +00:00
Zamil Majdy 80859d714d
HADOOP-17764. S3AInputStream read does not re-open the input stream on the second read retry attempt (#3109) 
Contributed by Zamil Majdy.

Change-Id: I680d9c425c920ff1a7cd4764d62e10e6ac78bee4
 2021-06-25 20:47:11 +01:00
Ahmed Hussein 22e7567475
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery (#3130). Contributed by Ahmed Hussein. (#3138) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 581f43dce1)
 2021-06-25 22:48:56 +05:30
Steve Loughran 39e6f2d191
HADOOP-17771. S3AFS creation fails "Unable to find a region via the region provider chain." (#3133) 
This addresses the regression in Hadoop 3.3.1 where if no S3 endpoint
is set in fs.s3a.endpoint, S3A filesystem creation may fail on
non-EC2 deployments, depending on the local host environment setup.

* If fs.s3a.endpoint is empty/null, and fs.s3a.endpoint.region
  is null, the region is set to "us-east-1".
* If fs.s3a.endpoint.region is explicitly set to "" then the client
  falls back to the SDK region resolution chain; this works on EC2
* Details in troubleshooting.md, including a workaround for Hadoop-3.3.1+
* Also contains some minor restructuring of troubleshooting.md

Contributed by Steve Loughran.

Change-Id: Ife482cff513307cd52d59eec56beac0a33e031f5
 2021-06-24 16:38:55 +01:00
litao 4ca49fcd44 HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 16:00:47 +05:30
litao 5fc61d2e56 HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 
(cherry picked from commit fdef2b4cca)
 2021-06-24 13:27:33 +09:00
Masatake Iwasaki b672edb64c YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 
(cherry picked from commit 11d144d228)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/yarn.lock
 2021-06-23 10:50:59 +00:00
Viraj Jasani 68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 
(cherry picked from commit d9fbb3c508)
 2021-06-23 11:24:00 +08:00
Viraj Jasani 4825c7c28d MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects (#3123) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa)
 2021-06-21 16:47:37 +09:00
Viraj Jasani b8a98e4f82 HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79)
 2021-06-21 10:28:05 +09:00
litao 7d202e8365 HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:24:03 +05:30
Viraj Jasani ef3a2a3b3b HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:52 +05:30
Ahmed Hussein 92ade1f6f9 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:49 +05:30
litao 8d241482bd HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:26:38 +05:30
Hui Fei e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3113) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 14:41:02 +08:00
Wei-Chiu Chuang 577b96ef42
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e548)
 2021-06-17 11:39:34 +08:00
Takanobu Asanuma 25138c98bf HADOOP-17760. Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml (#3099) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9e7c7ad129)
 2021-06-17 10:00:36 +09:00
Ayush Saxena 3e03ecd680 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:54:36 +05:30
Jim Brennan 57ee4238eb YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 
(cherry picked from commit 9a6a11c452)
 2021-06-15 19:30:16 +00:00
Takanobu Asanuma 546c66b812 HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682)
 2021-06-15 17:22:34 +09:00
Ahmed Hussein 840612b3b1 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 
(cherry picked from commit 747883af4d)
 2021-06-14 14:37:27 +00:00
Takanobu Asanuma 10000f162d HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours. (#3090) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit ae9630f580)
 2021-06-14 10:21:30 +09:00
Petre Bogdan Stolojan 254d943126
HADOOP-17547 Magic committer to downgrade abort in cleanup if list uploads fails with access denied (#3051) 
Contributed by Bogdan Stolojan

Change-Id: I32d6dc4f72087783a3ea12473d11690ac14fe3cb
 2021-06-12 17:46:11 +01:00
litao 52d055a7eb HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0)
 2021-06-11 17:33:06 +09:00
Mingliang Liu 91bcfbd72e
Revert "HDFS-16033 Fix issue of the StatisticsDataReferenceCleaner cleanUp (#3042)" 
This reverts commit 8c0f948054.
 2021-06-11 00:35:41 -07:00
Akira Ajisaka 19c969981d
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. (#3068) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9a17f97133)
 2021-06-10 14:33:41 +09:00
Viraj Jasani 0d6939ffc4
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ca27744770)
 2021-06-09 10:38:38 +08:00
Wei-Chiu Chuang 83d119fe94
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. (#3054) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c6539e3289)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-08 19:32:34 -07:00
Viraj Jasani 8f0ba9ee1b
HADOOP-17725. Improve error message for token providers in ABFS (#3041) 
Contributed by Viraj Jasani.
 2021-06-08 22:05:01 +01:00
Steve Loughran 4ac9123619
HADOOP-17631. Configuration ${env.VAR:-FALLBACK} to eval FALLBACK when restrictSystemProps=true (#2977) 
Contributed by Steve Loughran.

Change-Id: I9b82109eddeb659c01896152cf603d458e2a04cd
 2021-06-08 22:05:00 +01:00
litao 02249171b1 HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a)
 2021-06-08 16:59:59 +09:00
litao 46d4b51bff
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics for branch-3.3 (#3021) 2021-06-08 00:10:37 -07:00