Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,859 Commits 326 Branches 371 Tags
Commit Graph

21859 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
GuoPhilipse
ebf866cb6f
HDFS-16449. Fix hadoop web site release notes and changelog not available (#3967) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b68964336d0d0b390ffb68be45e48e0c754ad495)
 2022-02-14 05:41:07 +09:00
daimin
62192478b2
Fix thread safety of EC decoding during concurrent preads (#3881) 
(cherry picked from commit 0e74f1e467fde9622af4eb8f18312583d2354c0f)
(cherry picked from commit 9071c9646c7d86706581d523a084a8584f2cdcab)
 2022-02-11 10:21:00 +08:00
singer-bin
0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b616040b4f98956aa946cc51e99f7d596c2)
(cherry picked from commit ce7cabb771363ce460ac1c174c6380710e733ce8)
 2022-02-06 13:37:16 +08:00
daimin
337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994eec9c835906d28b0ae755dfd6698dcd)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
Abhishek Das
f72dc63d8b HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package 
Fixes #3950

Signed-off-by: Owen O'Malley <omalley@apache.org>

Cherry-picked from 3684c7f6 by Owen O'Malley
 2022-02-04 12:15:26 -08:00
Ananya Singh
82d56659b2
HADOOP-17796. Upgrade jetty version to 9.4.43 (#3757) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-02-02 20:41:17 +09:00
KevinWikant
3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de2135988c326328e544f04b5129c53517)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant
543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree
b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28fec1d0de7e5a80dc3399f0ad1fd3fce)
 2022-01-28 17:14:42 +09:00
litao
fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a351fa1a5dd668c318b20437e6aaa35b)
 2022-01-27 13:45:47 +09:00
Xing Lin
54de8eb956
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java (#3918). Contributed by Xing Lin. (#3931) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ffee2f645f405ad46b0afa65224f87d5)
 2022-01-26 21:53:50 +05:30
daimin
0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1ed49e06cc8973412952e09ed08bb9c0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c9d224727580f9ed4f754af5ab9f8f1)
 2022-01-25 10:01:09 +08:00
liubingxing
0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52ada03e099fa7dbb3efe62345ca27588)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma
53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229968b715e5c117bf0b35474ee0f05ec.
 2022-01-20 22:07:41 +09:00
qinyuren
96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699bd1360c109c18df0261897afea05ba)
 2022-01-19 13:44:27 +09:00
Gera Shegalov
76212079c9 YARN-11055. Add missing newline in cgroups-operations.c (#3851) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a94e9fcbde123edd96dd8b0009cf3618e2b578a7)
 2022-01-17 16:21:35 +09:00
Jackson Wang
74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d22dbba8facb94e80333cea5ffeed5ba)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta
3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a869ca5c2bdc457375461ae89cdf97315)
 2022-01-11 02:14:52 +09:00
liever18
1e1e4a2f51 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5851d3cdc3b125daebe5cc9305ad6f0)
 2022-01-05 16:02:49 +00:00
Ashutosh Gupta
278fc00871 HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2d79c3691bee29cb3fc4123062e5f7a)
 2022-01-04 13:26:09 +09:00
Ayush Saxena
6eb60353db
HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 08:01:00 +05:30
jianghuazhu
6854f7502f HADOOP-18063. Remove unused import AbstractJavaKeyStoreProvider in Shell class. (#3846) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7398a0f1b2ca434f41d1414f884731637b9855bf)
 2022-01-04 11:27:35 +09:00
Igor Dvorzhak
e88fe6a2ad HADOOP-13464. Upgrade Gson dependency to version 2.8.9 (#2524) 
Change-Id: Ifd3fb9ec6ebfc8874bb799bc198219511fe55a2f

Update pom.xml

Update pom.xml

(cherry picked from commit 795054882af72a9d5d4f4e96f775e96473d75c23)
(cherry picked from commit 5d72fdfcb2d8bc1415b3a603075048a231da18ca)
Change-Id: I7bcea58613340bf97a5038da40a3643971f99364
 2021-12-30 22:06:01 +00:00
Ashutosh Gupta
5aeaa368ef HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec889a0771191b58714c306439b2415d91)
(cherry picked from commit 6535a183b22dc8915cea9a93ab8c0370ef9bd2f0)
 2021-12-28 22:03:10 +09:00
jianghuazhu
20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f784f113489aee375ee1c2da8332d629)
 2021-12-24 17:28:00 +09:00
Ashutosh Gupta
e0e84431f0 YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00e2405fbd598602fdccf31826948c198c053fa5)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/TestSystemMetricsPublisher.java
 2021-12-23 17:40:59 +09:00
Eric Payne
3924efb595 YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 
(cherry picked from commit e2d6fd075dff4e6ea290ec638f0a3f6688e76335)
 2021-12-21 19:34:29 +00:00
wangzhaohui
c66bae3fc3 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629c2db33f03c80e93c7e54fceedbfbe6)
 2021-12-21 12:02:25 +00:00
Szilard Nemeth
020acb8d39 YARN-10503. Support queue capacity in terms of absolute resources with custom resourceType. Contributed by Qi Zhu. 2021-12-20 17:59:06 +01:00
jianghuazhu
b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
 2021-12-20 12:08:30 +00:00
Szilard Nemeth
01de8fd9d6 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-12-17 14:53:06 +01:00
Dhananjay Badaya
253b217efa HADOOP-13500. Synchronizing iteration of Configuration properties object (#3775) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607a4edc0f13264f4ea37abd90aba97e1ef0)
 2021-12-17 16:13:25 +09:00
litao
3257646975 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d21c9db9ceecf611128508542ec0f11)
 2021-12-16 13:52:01 +09:00
Akira Ajisaka
07d3d5ea0b HADOOP-17534. Update Jackson to 2.10.5 and Jackson databind to 2.10.5.1 (#2708) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e119063a078f06523158dba3b9e16ca04768e920)
 2021-12-14 23:53:33 +09:00
Ayush Saxena
a57e83a47b HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:24:05 +09:00
GuoPhilipse
038759e444 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7430692d1f40f59e9cb24690cbda216)
 2021-12-10 16:36:40 +09:00
Akira Ajisaka
bc6874139f
HADOOP-16905. Update jackson-databind to 2.10.3 to relieve us from the endless CVE patches (#3748) 
(cherry picked from commit 69faaa1d58ad7de18a8dfa477531653a2c061568)

 Conflicts:
	hadoop-project/pom.xml
 2021-12-10 16:24:06 +09:00
Akira Ajisaka
94ca965e21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83beb4f68e1ff18907589b062a0467e8)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml
 2021-12-10 01:41:44 +09:00
Akira Ajisaka
3854b59bd1
HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:27:27 +09:00
qinyuren
72ffbd956a HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:42:07 +09:00
Anoop Sam John
84cab2ee29
HADOOP-17643 WASB : Make metadata checks case insensitive (#3102) 2021-12-09 14:45:48 +05:30
Takanobu Asanuma
2315849b8f Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 2072a6a47654c140d4bc9e386a0bc333114d1c2c.
 2021-12-09 16:37:43 +09:00
qinyuren
2072a6a476 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043c063f548b81ad431d9452f010b18f7)
 2021-12-09 13:27:33 +09:00
lzx404243
6114816b03 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c123544e1c1cbee759a11879af3b19260b3)
 2021-12-08 20:49:27 +09:00
Ashutosh Gupta
c8cf7a9bdb YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900aa143f6db281eb22928965f3b2754f)
 2021-12-06 20:15:11 +09:00
LeonGao
b4fc326acf HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661d694fb0775f4f84333c1317f4e6899)
 2021-12-06 16:19:05 +09:00
Prabhu Joseph
292527b325 YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 
(cherry picked from commit 6b4987f09cbc53e00e85cb82ad97505858cada11)
 2021-12-06 14:46:34 +09:00
bitterfox
7bdb50c793 HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93e8319560b633360f30ac84fc48e403)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:43:37 +09:00
Ananya Singh
b8dbc24727
HDFS-16317 : Backport HDFS-14729 for branch-3.2 (#3692) 
Co-authored-by: Sunil G <sunilg@apache.org>,Ananya Singh <purpul90@gmail.com>
 2021-11-30 18:01:53 +05:30
smarthan
2f3e186978 HADOOP-18023. Allow cp command to run with multi threads. (#3721) 
(cherry picked from commit 932a78fe38b34a923f6852a1a19482075806ecba)
 2021-11-29 15:06:29 +00:00