Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,692 Commits 326 Branches 371 Tags
Commit Graph

21692 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
daimin
e1f11d5f63 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc53a0144311877362787ed9ece0cb118)
 2021-10-18 12:37:38 +01:00
Brahma Reddy Battula
b55002f6e6 HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1e48db05ab827d0cf09963a430615ed)
 2021-10-08 17:23:21 +05:30
Siyao Meng
b93fe96fad
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3aaac8a1f682ecda4d41221e8a836bc5401e3b1d)

 Conflicts:
	LICENSE-binary

(cherry picked from commit c8db6b1eb2a541af20c4c99c23b627cec2300411)
 2021-10-08 14:59:05 +09:00
Takanobu Asanuma
091f93cf09 HADOOP-17955. Bump netty to the latest 4.1.68. (#3528) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit cf4d2d826cad941ae3648ff9b6057856bb199fc4)
 2021-10-08 10:43:43 +09:00
Rohith Sharma K S
71ba9f1ccc YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. 
(cherry picked from commit f63e4e4f2eee6b75c3a6c3fb5926d44c7e846145)

Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 51d17f3cb399ae0ed66d3a2357ef091151d25646)
 2021-10-06 22:42:51 +09:00
AngersZhuuuu
2bd78b7eaa
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4112e7b4ee9e6b948ce46a2b152a2db)
(cherry picked from commit 1df3b3de463332b688299078a59265ce405af6ac)
 2021-09-29 18:36:18 +08:00
He Xiaoqiao
1d9a6ffae5
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe0677b266acf237e8458e93743424a6)
(cherry picked from commit cfd02659751d06f40a4c51aa3de2f8df78a42b66)
 2021-09-29 18:36:10 +08:00
Wei-Chiu Chuang
6ef0b3fc21
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
(cherry picked from commit 978ec57445f4178b31a6bfc383c6be032692c288)
 2021-09-28 16:26:55 +09:00
Takanobu Asanuma
9b016b401a HADOOP-17940. Upgrade Kafka to 2.8.1 (#3488) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2068b0041c05a7b53f32606cc24922cfac299ca8)
 2021-09-28 13:43:30 +09:00
Inigo Goiri
1f55483339
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dcbf1901ca6fad11b5c996e54e32ed6b)
(cherry picked from commit 86028d975c050c5d2a4f3fde22cba9b867bb3f03)
 2021-09-27 14:52:51 +09:00
Akira Ajisaka
91acb7b9fe
HADOOP-17917. Backport HADOOP-15993 to branch-3.2 which address CVE-2014-4611. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 85346db636f0dcfe7869e3f09f108014e857c0d5)
 2021-09-24 15:00:12 +09:00
He Xiaoqiao
7fea1dc006
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a72705c5b68ddfe4d29f0d208fd38e7)
(cherry picked from commit 4944dafa297e9aad93d39e861b5dc2835b7cbb5f)
 2021-09-22 18:14:51 +08:00
He Xiaoqiao
56f7e52be5
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff44396eb478f709307d647ca884de99)
(cherry picked from commit 50b0f0dc42d5cdcfbd381527ff774f20852888ba)
 2021-09-22 18:14:46 +08:00
He Xiaoqiao
c7213e4d2b
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
(cherry picked from commit fc5fe67b3f73e98d931d475c02efb52fb3c7a2b4)
 2021-09-22 18:14:41 +08:00
Inigo Goiri
79187b5a5e
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5ca99502d2ac2fbf803f22463eb1343)
(cherry picked from commit 5da46e197b63e4c847732a64e9694c9e485d5f2d)
 2021-09-22 18:14:35 +08:00
Ayush Saxena
cedb8fec9d
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab0fb385252d4eeb19141bd059303d59)
(cherry picked from commit f77965b29eb3f33713c39f1cb341fdc7c91b0220)
 2021-09-22 18:14:29 +08:00
Inigo Goiri
baa4190eab
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d95440de099831e0db38db037f16bdd)
(cherry picked from commit d5501c125fa0fffe846c8be8718e05f55e288959)
 2021-09-22 18:14:21 +08:00
Ahmed Hussein
0d0f344d65
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 14:26:32 +08:00
LeonGao
38973a52f0
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)
 2021-09-14 14:26:27 +08:00
Ahmed Hussein
0879f4ec0f
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)
 2021-09-14 14:26:22 +08:00
Stephen O'Donnell
84e16adab3
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)
 2021-09-14 14:26:09 +08:00
Stephen O'Donnell
3c715a272a
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 14:25:57 +08:00
Ahmed Hussein
a9dbe47a72
HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
(cherry picked from commit 601591c02144c9c0dfed2c4011187dc7bb329157)
 2021-09-03 10:17:41 +09:00
Szilard Nemeth
867acc2e6d
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f76e470527ad94b0894dacb28db01465)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java

(cherry picked from commit 7aea2e1b5c24cd6e2dffbe6942f0dadb6a45c48f)
 2021-09-01 13:16:05 +09:00
Xiaoyu Yao
3c79d65272
HADOOP-15939. Filter overlapping objenesis class in hadoop-client-minicluster. Contributed by Xiaoyu Yao. 
(cherry picked from commit 397f523e22a4f76b5484bed26ef4e6d40200611e)
(cherry picked from commit 3ea74033bf712ad63c4e43ffda50fe13c8522a20)
 2021-08-25 21:10:12 +09:00
Masatake Iwasaki
bcd81a7fb4 HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar. 
(cherry picked from commit 0d59500e8cf89e38cd0f0e45553e4557e5358a8b)
 2021-08-25 11:50:49 +09:00
Masatake Iwasaki
996cbe453a
HADOOP-17850. Upgrade ZooKeeper to 3.4.14 in branch-3.2. (#3308) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 037875ba798422b6b342797c85c66bcf9781824e)
 2021-08-17 13:37:18 +09:00
Masatake Iwasaki
462b808d42 HADOOP-17849. Exclude spotbugs-annotations from transitive dependencies on branch-3.2. (#3305) 
(cherry picked from commit 786888b8d32e35cad49eb1ba54a35f1508234ed5)
 2021-08-16 21:01:33 +00:00
Renukaprasad C
fd1d138a62 HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b90389ae9840ccfdcab411e286547895a647feed)

 Conflicts:
	LICENSE-binary

(cherry picked from commit d18c0a1d25a2fc6a048fd7123655d9d31e07d797)
 2021-08-14 20:28:26 +09:00
Brahma Reddy Battula
182167a9fb Preparing for 3.2.3 release. 2021-08-08 19:17:14 +05:30
Akira Ajisaka
840c6ef7fa
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 
(cherry picked from commit 3565c9477dffd104a3aa3603845f7a1305815ac3)

 Conflicts:
	LICENSE-binary
 2021-08-08 11:26:03 +09:00
Bryan Beaudreault
dab829063d
HADOOP-17840: Backport HADOOP-17837 to branch-3.2 (#3275) 
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
 2021-08-07 09:49:46 +05:30
Akira Ajisaka
78c5fdd90d
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246f4aab665fd04ce72d1848bc5da9d4e)
 2021-08-07 11:28:22 +09:00
wangzhaohui
b98a8d259b
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c4d1368e1195132bb32a815cb12934c)
(cherry picked from commit 7068e18842117f33b5caceee244e8b27618c121a)
 2021-08-06 18:25:28 +08:00
wangzhaohui
6b10154123 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b733177cf9b8b10088527e34099d73f)
 2021-08-06 09:52:51 +01:00
Viraj Jasani
2f5e527910 HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 
(cherry picked from commit 9fe1f24ec11c3e1c2ed5ee9db801d40a2f7a2ed6)
 2021-08-06 09:54:34 +08:00
Akira Ajisaka
c40abe10c0
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3265) 
* YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081)

Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ca2774477052429a9d8371d66af9b7f7c007acd4)

* Use mockito 1.x

Co-authored-by: Viraj Jasani <vjasani@apache.org>
 2021-08-05 10:05:10 +09:00
Abhishek Modi
c5e2ec465b
YARN-9990. Testcase fails with Insufficient configured threads: required=16 < max=10. Contributed by Prabhu Joseph. 
(cherry picked from commit a2dadac790ae3b4e3ab411be84d909d18af33f6e)
 2021-08-04 17:05:35 +09:00
Jim Brennan
1f9a315625
YARN-10337. Amendment to fix import as in HADOOP-17100 
(cherry picked from commit 53a1c7653fd3ff0206da8acf3fdbacbf932b932d)
 2021-08-04 16:38:52 +09:00
Prabhu Joseph
b9d0412fd5
YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.

(cherry picked from commit 2bbd00dff498027241a5d84713f4e3f13ac45e65)
 2021-08-04 16:37:47 +09:00
Akira Ajisaka
d830e84217
YARN-9875. Improve fair scheduler configuration store on HDFS. (#3262) 
Contributed by Prabhu Joseph

(cherry picked from commit 155864da006346a500ff35c2f6b69281093195b1)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestFSSchedulerConfigurationStore.java

Co-authored-by: Eric Yang <eyang@apache.org>
 2021-08-04 16:12:54 +09:00
Giovanni Matteo Fumarola
2f3c0b8dd1
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 
(cherry picked from commit 1a08302897bac18baf0f56b7d41ad54f63ae191b)
 2021-08-04 15:54:08 +09:00
Akira Ajisaka
b838647b16
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#3264) 
* HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9a146b2dd9857a17a0db6fcddb1de23a0d)

* Add assertj-core dependency to hadoop-common for test
* Bump assertj-core version to 3.12.2 to use assertThat.hasSizeGreaterThan
 2021-08-04 15:46:00 +09:00
Akira Ajisaka
e259d85a99
YARN-8992. Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue. (Contributed by Wilfred Spiegelenburg) (#3257) 
(cherry picked from commit a41b648e98b6a1c5a9cdb7393e73e576a97f56d4)

Co-authored-by: Haibo Chen <haibochen@apache.org>
 2021-08-04 10:17:51 +09:00
Siyao Meng
fe59fe6744 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00b5589fe610ff994d81feba86039297)
(cherry picked from commit 72508e6430b9fe5145120ac23d3cf714c16e7579)
 2021-08-03 20:45:17 +01:00
Akira Ajisaka
786a43e729
YARN-8990. Fix fair scheduler race condition in app submit and queue cleanup. (Contributed by Wilfred Spiegelenburg) (#3254) 
(cherry picked from commit 524a7523c427b55273133078898ae3535897bada)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java

Co-authored-by: Haibo Chen <haibochen@apache.org>
 2021-08-03 13:12:01 +09:00
Vrushali C
521c1085b4
YARN-9338 Timeline related testcases are failing. Contributed by Abhishek Modi. 
(cherry picked from commit 17a3e14d25877af90ef6655750ce2b035c2982b5)
 2021-08-03 11:42:28 +09:00
jenny
18ba3d6b1e
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup (#1808) 
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e31169c8641384fa4b45d8a27c4b7a152c1ea35b)
 2021-08-02 15:40:14 +09:00
Szilard Nemeth
e4df745c13 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:54:51 +02:00
Szilard Nemeth
7a8b6265c6 YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-07-29 19:22:57 +02:00