Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,841 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21841 Commits

Author SHA1 Message Date
jianghuazhu ed83a278c2
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e712)
(cherry picked from commit 4aa9698ee1e9a10a0f882229036bbb4db1ede939)
 2022-04-19 11:13:29 +08:00
daimin 248c548616
Fix decommission UnsupportedOperationException (#4172) 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 10:40:12 +08:00
Takanobu Asanuma b8c6ba6046 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959)
 2022-04-14 11:52:12 +09:00
qinyuren cb14e8de81 HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a1)
 2022-04-13 11:53:58 +09:00
Viraj Jasani 2d2631afc0 HADOOP-18191. Log retry count while handling exceptions in RetryInvocationHandler (#4133) 
(cherry picked from commit b69ede7154)
 2022-04-11 15:24:24 +09:00
Hanisha Koneru 9cdf5ca105 HADOOP-17116. Skip Retry INFO logging on first failover from a proxy 
(cherry picked from commit e62d8f8412)
 2022-04-11 15:19:54 +09:00
Stephen O'Donnell 72e48ac84e HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf)
 2022-04-06 13:06:28 +01:00
wangzhaohui 545d3a6cb7 HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:31:42 +09:00
Masatake Iwasaki 2d770e49f1
HADOOP-18192. Fix multiple_bindings warning about slf4j-reload4j. (#4136) 2022-04-06 14:13:19 +09:00
litao 6b24b85e3f HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:04:04 -07:00
Kengo Seki 5c6547d794 MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ (#4120) 
(cherry picked from commit dc4a680da8)
 2022-03-30 22:51:40 +09:00
Masatake Iwasaki 228ad1ac25 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a244)
 2022-03-28 08:09:58 +00:00
Masatake Iwasaki 5c16db8b17 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 10876333ac)
 2022-03-28 08:04:47 +00:00
GuoPhilipse ff03212145 HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)
 2022-03-27 22:33:17 +09:00
Akira Ajisaka 73775fe277
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. (#4102) 
(cherry picked from commit a0deda1a77)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

Co-authored-by: Peter Bacsko <pbacsko@cloudera.com>
 2022-03-24 17:36:44 +09:00
qinyuren 66be0a713f HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-03-23 15:06:20 +09:00
Masatake Iwasaki e13408d9d2
HADOOP-18088. Replace log4j 1.x with reload4j. (#4084) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-23 12:57:27 +09:00
Surendra Singh Lilhore 58fe5b2f35 HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 
(cherry picked from commit 5ff76cb8bc)
 2022-03-18 18:39:20 +09:00
Wei-Chiu Chuang 826d13a59a HADOOP-18155. Refactor tests in TestFileUtil (#4063) 
(cherry picked from commit d0fa9b5775)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Co-authored-by: Gautham B A <gautham.bangalore@gmail.com>
(cherry picked from commit 743db6e7b4)
 2022-03-14 10:05:01 +09:00
belugabehr d09ca8bceb HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811). Contributed by David Mollitor. 
(cherry picked from commit 1afd54fbbb)
 2022-03-14 10:04:47 +09:00
Thinker313 6f815a546a
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 43153e80cb)
 2022-03-12 20:10:44 +08:00
Wei-Chiu Chuang c531613cc9 HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad564)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
(cherry picked from commit 5e09d38b25)

Conflicts:
	dev-support/docker/Dockerfile_aarch64
 2022-03-10 09:56:04 +00:00
Akira Ajisaka 523bb2efc3 YARN-9783. Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5. Contributed by Mate Szalay-Beko. 
(cherry picked from commit dc0acceabb)
 2022-03-10 07:41:14 +00:00
srinivasst c64eda080e YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
(cherry picked from commit 98565b6c60)
 2022-03-10 16:35:40 +09:00
Szilard Nemeth 9e2e1c795f YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2022-03-02 14:21:36 +01:00
Szilard Nemeth 4bd67d7516 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-03-01 18:06:39 +01:00
Akira Ajisaka 06a2e4ccde HADOOP-16717. Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter. Contributed by Xieming Li. 
(cherry picked from commit da9435182a)
 2022-02-28 10:54:06 +09:00
Steve Loughran adfd767225 HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.

(cherry picked from commit 88a550bc3a)

 Conflicts:
	hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java
 2022-02-28 10:41:47 +09:00
Ayush Saxena 551f8d3ddc
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
 2022-02-23 11:41:35 +08:00
Viraj Jasani 9d618bcc6b
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d4636)
(cherry picked from commit d763c99707)
 2022-02-22 11:01:51 +08:00
Steve Loughran a8512d6627
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: Id6f3f2b0eeeb186a757d8b6ee5a0584cb4114469
 2022-02-21 17:18:03 +00:00
Chentao Yu 5066722eb7 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:07:46 -08:00
GuoPhilipse ebf866cb6f
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b68964336d)
 2022-02-14 05:41:07 +09:00
daimin 62192478b2
Fix thread safety of EC decoding during concurrent preads (#3881) 
(cherry picked from commit 0e74f1e467)
(cherry picked from commit 9071c9646c)
 2022-02-11 10:21:00 +08:00
singer-bin 0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
(cherry picked from commit ce7cabb771)
 2022-02-06 13:37:16 +08:00
daimin 337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
Abhishek Das f72dc63d8b HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package 
Fixes #3950

Signed-off-by: Owen O'Malley <omalley@apache.org>

Cherry-picked from 3684c7f6 by Owen O'Malley
 2022-02-04 12:15:26 -08:00
Ananya Singh 82d56659b2
HADOOP-17796. Upgrade jetty version to 9.4.43 (#3757) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-02 20:41:17 +09:00
KevinWikant 3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant 543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:14:42 +09:00
litao fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:45:47 +09:00
Xing Lin 54de8eb956
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. (#3931) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ff)
 2022-01-26 21:53:50 +05:30
daimin 0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c)
 2022-01-25 10:01:09 +08:00
liubingxing 0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma 53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229.
 2022-01-20 22:07:41 +09:00
qinyuren 96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:27 +09:00
Gera Shegalov 76212079c9 YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a94e9fcbde)
 2022-01-17 16:21:35 +09:00
Jackson Wang 74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta 3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:14:52 +09:00