Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,820 Commits 326 Branches 371 Tags
Commit Graph

21820 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
c531613cc9 HADOOP-17718. Explicitly set locale in the Dockerfile. (#3034) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ad923ad5642b2b11357fbee4277f3435300a19c5)

Conflicts:
	dev-support/docker/Dockerfile_centos_7
	dev-support/docker/Dockerfile_centos_8
(cherry picked from commit 5e09d38b2505db70c712b8b6c93635220325156f)

Conflicts:
	dev-support/docker/Dockerfile_aarch64
 2022-03-10 09:56:04 +00:00
Akira Ajisaka
523bb2efc3 YARN-9783. Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5. Contributed by Mate Szalay-Beko. 
(cherry picked from commit dc0acceabb6a5189974109cfea09f598c2a99d14)
 2022-03-10 07:41:14 +00:00
srinivasst
c64eda080e YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a8a45ff8c7d25d52cb014d0ce858cc2)
(cherry picked from commit 98565b6c60c4c1cefaabe4a1fa5fe92158b19b97)
 2022-03-10 16:35:40 +09:00
Szilard Nemeth
9e2e1c795f YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2022-03-02 14:21:36 +01:00
Szilard Nemeth
4bd67d7516 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-03-01 18:06:39 +01:00
Akira Ajisaka
06a2e4ccde HADOOP-16717. Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter. Contributed by Xieming Li. 
(cherry picked from commit da9435182a8fccec743406e089325655138d9de8)
 2022-02-28 10:54:06 +09:00
Steve Loughran
adfd767225 HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.

(cherry picked from commit 88a550bc3a73dbbbfb35e2ef29d9c34ad059d1a8)

 Conflicts:
	hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java
 2022-02-28 10:41:47 +09:00
Ayush Saxena
551f8d3ddc
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409bc1d62f23efef485d1c6f1ff21e86c)
(cherry picked from commit fa30224e95d920adaea0ec65c360359044e3f754)
 2022-02-23 11:41:35 +08:00
Viraj Jasani
9d618bcc6b
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation (#3991) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d463640a7107a622262eb2d333d0458fd8b)
(cherry picked from commit d763c997071752c351ceb689716c8d44803f29b4)
 2022-02-22 11:01:51 +08:00
Steve Loughran
a8512d6627
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files. 
Contributed by Steve Loughran

Change-Id: Id6f3f2b0eeeb186a757d8b6ee5a0584cb4114469
 2022-02-21 17:18:03 +00:00
Chentao Yu
5066722eb7 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb28539f8c79bbb24f703301489825a6)
 2022-02-15 16:07:46 -08:00
GuoPhilipse
ebf866cb6f
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b68964336d0d0b390ffb68be45e48e0c754ad495)
 2022-02-14 05:41:07 +09:00
daimin
62192478b2
Fix thread safety of EC decoding during concurrent preads (#3881) 
(cherry picked from commit 0e74f1e467fde9622af4eb8f18312583d2354c0f)
(cherry picked from commit 9071c9646c7d86706581d523a084a8584f2cdcab)
 2022-02-11 10:21:00 +08:00
singer-bin
0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b616040b4f98956aa946cc51e99f7d596c2)
(cherry picked from commit ce7cabb771363ce460ac1c174c6380710e733ce8)
 2022-02-06 13:37:16 +08:00
daimin
337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994eec9c835906d28b0ae755dfd6698dcd)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
Abhishek Das
f72dc63d8b HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package 
Fixes #3950

Signed-off-by: Owen O'Malley <omalley@apache.org>

Cherry-picked from 3684c7f6 by Owen O'Malley
 2022-02-04 12:15:26 -08:00
Ananya Singh
82d56659b2
HADOOP-17796. Upgrade jetty version to 9.4.43 (#3757) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-02 20:41:17 +09:00
KevinWikant
3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de2135988c326328e544f04b5129c53517)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant
543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree
b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28fec1d0de7e5a80dc3399f0ad1fd3fce)
 2022-01-28 17:14:42 +09:00
litao
fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a351fa1a5dd668c318b20437e6aaa35b)
 2022-01-27 13:45:47 +09:00
Xing Lin
54de8eb956
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. (#3931) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ffee2f645f405ad46b0afa65224f87d5)
 2022-01-26 21:53:50 +05:30
daimin
0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1ed49e06cc8973412952e09ed08bb9c0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c9d224727580f9ed4f754af5ab9f8f1)
 2022-01-25 10:01:09 +08:00
liubingxing
0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52ada03e099fa7dbb3efe62345ca27588)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma
53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229968b715e5c117bf0b35474ee0f05ec.
 2022-01-20 22:07:41 +09:00
qinyuren
96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699bd1360c109c18df0261897afea05ba)
 2022-01-19 13:44:27 +09:00
Gera Shegalov
76212079c9 YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a94e9fcbde123edd96dd8b0009cf3618e2b578a7)
 2022-01-17 16:21:35 +09:00
Jackson Wang
74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d22dbba8facb94e80333cea5ffeed5ba)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta
3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a869ca5c2bdc457375461ae89cdf97315)
 2022-01-11 02:14:52 +09:00
liever18
1e1e4a2f51 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5851d3cdc3b125daebe5cc9305ad6f0)
 2022-01-05 16:02:49 +00:00
Ashutosh Gupta
278fc00871 HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2d79c3691bee29cb3fc4123062e5f7a)
 2022-01-04 13:26:09 +09:00
Ayush Saxena
6eb60353db
HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 08:01:00 +05:30
jianghuazhu
6854f7502f HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7398a0f1b2ca434f41d1414f884731637b9855bf)
 2022-01-04 11:27:35 +09:00
Igor Dvorzhak
e88fe6a2ad HADOOP-13464. Upgrade Gson dependency to version 2.8.9 (#2524) 
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml

(cherry picked from commit 795054882af72a9d5d4f4e96f775e96473d75c23)
(cherry picked from commit 5d72fdfcb2d8bc1415b3a603075048a231da18ca)
Change-Id: I7bcea58613340bf97a5038da40a3643971f99364
 2021-12-30 22:06:01 +00:00
Ashutosh Gupta
5aeaa368ef HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec889a0771191b58714c306439b2415d91)
(cherry picked from commit 6535a183b22dc8915cea9a93ab8c0370ef9bd2f0)
 2021-12-28 22:03:10 +09:00
jianghuazhu
20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f784f113489aee375ee1c2da8332d629)
 2021-12-24 17:28:00 +09:00
Ashutosh Gupta
e0e84431f0 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd598602fdccf31826948c198c053fa5)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
 2021-12-23 17:40:59 +09:00
Eric Payne
3924efb595 YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 
(cherry picked from commit e2d6fd075dff4e6ea290ec638f0a3f6688e76335)
 2021-12-21 19:34:29 +00:00
wangzhaohui
c66bae3fc3 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629c2db33f03c80e93c7e54fceedbfbe6)
 2021-12-21 12:02:25 +00:00
Szilard Nemeth
020acb8d39 YARN-10503. Support queue capacity in terms of absolute resources with custom resourceType. Contributed by Qi Zhu. 2021-12-20 17:59:06 +01:00
jianghuazhu
b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
 2021-12-20 12:08:30 +00:00
Szilard Nemeth
01de8fd9d6 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-12-17 14:53:06 +01:00
Dhananjay Badaya
253b217efa HADOOP-13500. Synchronizing iteration of Configuration properties object (#3775) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607a4edc0f13264f4ea37abd90aba97e1ef0)
 2021-12-17 16:13:25 +09:00
litao
3257646975 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d21c9db9ceecf611128508542ec0f11)
 2021-12-16 13:52:01 +09:00
Akira Ajisaka
07d3d5ea0b HADOOP-17534. Update Jackson to 2.10.5 and Jackson databind to 2.10.5.1 (#2708) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e119063a078f06523158dba3b9e16ca04768e920)
 2021-12-14 23:53:33 +09:00
Ayush Saxena
a57e83a47b HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:24:05 +09:00
GuoPhilipse
038759e444 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7430692d1f40f59e9cb24690cbda216)
 2021-12-10 16:36:40 +09:00
Akira Ajisaka
bc6874139f
HADOOP-16905. Update jackson-databind to 2.10.3 to relieve us from the endless CVE patches (#3748) 
(cherry picked from commit 69faaa1d58ad7de18a8dfa477531653a2c061568)

 Conflicts:
	hadoop-project/pom.xml
 2021-12-10 16:24:06 +09:00
Akira Ajisaka
94ca965e21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83beb4f68e1ff18907589b062a0467e8)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml
 2021-12-10 01:41:44 +09:00
Akira Ajisaka
3854b59bd1
HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:27:27 +09:00