Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,878 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21878 Commits

Author SHA1 Message Date
GuoPhilipse ff03212145 HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)
 2022-03-27 22:33:17 +09:00
Akira Ajisaka 73775fe277
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. (#4102) 
(cherry picked from commit a0deda1a77)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java

Co-authored-by: Peter Bacsko <pbacsko@cloudera.com>
 2022-03-24 17:36:44 +09:00
qinyuren 66be0a713f HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-03-23 15:06:20 +09:00
Masatake Iwasaki e13408d9d2
HADOOP-18088. Replace log4j 1.x with reload4j. (#4084) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-23 12:57:27 +09:00
Surendra Singh Lilhore 58fe5b2f35 HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 
(cherry picked from commit 5ff76cb8bc)
 2022-03-18 18:39:20 +09:00
Wei-Chiu Chuang 826d13a59a HADOOP-18155. Refactor tests in TestFileUtil (#4063) 
(cherry picked from commit d0fa9b5775)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileUtil.java
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileUtil.java

Co-authored-by: Gautham B A <gautham.bangalore@gmail.com>
(cherry picked from commit 743db6e7b4)
 2022-03-14 10:05:01 +09:00
belugabehr d09ca8bceb HADOOP-16811: Use JUnit TemporaryFolder Rule in TestFileUtils (#1811). Contributed by David Mollitor. 
(cherry picked from commit 1afd54fbbb)
 2022-03-14 10:04:47 +09:00
Thinker313 6f815a546a
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 43153e80cb)
 2022-03-12 20:10:44 +08:00
Wei-Chiu Chuang c531613cc9 HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad564)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
(cherry picked from commit 5e09d38b25)

Conflicts:
	dev-support/docker/Dockerfile_aarch64
 2022-03-10 09:56:04 +00:00
Akira Ajisaka 523bb2efc3 YARN-9783. Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5. Contributed by Mate Szalay-Beko. 
(cherry picked from commit dc0acceabb)
 2022-03-10 07:41:14 +00:00
srinivasst c64eda080e YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
(cherry picked from commit 98565b6c60)
 2022-03-10 16:35:40 +09:00
Szilard Nemeth 9e2e1c795f YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2022-03-02 14:21:36 +01:00
Szilard Nemeth 4bd67d7516 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-03-01 18:06:39 +01:00
Akira Ajisaka 06a2e4ccde HADOOP-16717. Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter. Contributed by Xieming Li. 
(cherry picked from commit da9435182a)
 2022-02-28 10:54:06 +09:00
Steve Loughran adfd767225 HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.

(cherry picked from commit 88a550bc3a)

 Conflicts:
	hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java
 2022-02-28 10:41:47 +09:00
Ayush Saxena 551f8d3ddc
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
 2022-02-23 11:41:35 +08:00
Viraj Jasani 9d618bcc6b
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d4636)
(cherry picked from commit d763c99707)
 2022-02-22 11:01:51 +08:00
Steve Loughran a8512d6627
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: Id6f3f2b0eeeb186a757d8b6ee5a0584cb4114469
 2022-02-21 17:18:03 +00:00
Chentao Yu 5066722eb7 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:07:46 -08:00
GuoPhilipse ebf866cb6f
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b68964336d)
 2022-02-14 05:41:07 +09:00
daimin 62192478b2
Fix thread safety of EC decoding during concurrent preads (#3881) 
(cherry picked from commit 0e74f1e467)
(cherry picked from commit 9071c9646c)
 2022-02-11 10:21:00 +08:00
singer-bin 0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
(cherry picked from commit ce7cabb771)
 2022-02-06 13:37:16 +08:00
daimin 337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
Abhishek Das f72dc63d8b HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package 
Fixes #3950

Signed-off-by: Owen O'Malley <omalley@apache.org>

Cherry-picked from 3684c7f6 by Owen O'Malley
 2022-02-04 12:15:26 -08:00
Ananya Singh 82d56659b2
HADOOP-17796. Upgrade jetty version to 9.4.43 (#3757) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-02 20:41:17 +09:00
KevinWikant 3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant 543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:14:42 +09:00
litao fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:45:47 +09:00
Xing Lin 54de8eb956
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. (#3931) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ff)
 2022-01-26 21:53:50 +05:30
daimin 0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c)
 2022-01-25 10:01:09 +08:00
liubingxing 0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma 53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229.
 2022-01-20 22:07:41 +09:00
qinyuren 96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:27 +09:00
Gera Shegalov 76212079c9 YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a94e9fcbde)
 2022-01-17 16:21:35 +09:00
Jackson Wang 74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta 3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:14:52 +09:00
liever18 1e1e4a2f51 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 16:02:49 +00:00
Ashutosh Gupta 278fc00871 HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:26:09 +09:00
Ayush Saxena 6eb60353db
HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 08:01:00 +05:30
jianghuazhu 6854f7502f HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7398a0f1b2)
 2022-01-04 11:27:35 +09:00
Igor Dvorzhak e88fe6a2ad HADOOP-13464. Upgrade Gson dependency to version 2.8.9 (#2524) 
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml

(cherry picked from commit 795054882a)
(cherry picked from commit 5d72fdfcb2)
Change-Id: I7bcea58613340bf97a5038da40a3643971f99364
 2021-12-30 22:06:01 +00:00
Ashutosh Gupta 5aeaa368ef HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec88)
(cherry picked from commit 6535a183b2)
 2021-12-28 22:03:10 +09:00
jianghuazhu 20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:28:00 +09:00
Ashutosh Gupta e0e84431f0 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
 2021-12-23 17:40:59 +09:00
Eric Payne 3924efb595 YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 
(cherry picked from commit e2d6fd075d)
 2021-12-21 19:34:29 +00:00
wangzhaohui c66bae3fc3 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2021-12-21 12:02:25 +00:00
Szilard Nemeth 020acb8d39 YARN-10503. Support queue capacity in terms of absolute resources with custom resourceType. Contributed by Qi Zhu. 2021-12-20 17:59:06 +01:00
jianghuazhu b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 12:08:30 +00:00
Szilard Nemeth 01de8fd9d6 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-12-17 14:53:06 +01:00