Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,712 Commits 326 Branches 371 Tags
Commit Graph

21712 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
srinivasst
e765f9f85e YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a8a45ff8c7d25d52cb014d0ce858cc2)
(cherry picked from commit 98565b6c60c4c1cefaabe4a1fa5fe92158b19b97)
(cherry picked from commit c64eda080e6ed0d47fbc91a22050612e1d4c80d3)
 2022-03-10 16:38:17 +09:00
Akira Ajisaka
072958324c HADOOP-16717. Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter. Contributed by Xieming Li. 
(cherry picked from commit da9435182a8fccec743406e089325655138d9de8)
(cherry picked from commit 06a2e4ccded279381ce65add79e926e6d2679fc0)
 2022-02-28 10:56:53 +09:00
Ashutosh Gupta
65783db095 HDFS-14099. Unknown frame descriptor when decompressing multiple frames (#3836) 
Co-authored-by: xuzq <xuzengqiang@kuaishou.com>
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit caab29ec889a0771191b58714c306439b2415d91)
(cherry picked from commit 6535a183b22dc8915cea9a93ab8c0370ef9bd2f0)
(cherry picked from commit 5aeaa368efb6a3b847013b411b352eb3fcbd3ac0)
 2022-02-28 10:47:19 +09:00
Eric Yang
662aaf576c YARN-7266. Fixed deadlock in Timeline Server thread initialization. 
Contributed by Prabhu Joseph

(cherry picked from commit 7b42e0e32ac7dfb60f25fa656a9bef69c2a62501)
(cherry picked from commit c35b3be700ab61ef86520265d2fd9ba03ba32ea3)
 2022-02-28 10:44:15 +09:00
Steve Loughran
a8f08410d4 HADOOP-17112. S3A committers can't handle whitespace in paths. (#2953) 
Contributed by Krzysztof Adamski.

(cherry picked from commit 88a550bc3a73dbbbfb35e2ef29d9c34ad059d1a8)

 Conflicts:
	hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/commit/ITestCommitOperations.java

(cherry picked from commit adfd767225b6d65e40d3511a32dbf72649041d94)
 2022-02-28 10:42:52 +09:00
Jackson Wang
46d09e4e8b HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d22dbba8facb94e80333cea5ffeed5ba)
(cherry picked from commit 74d3e83be012a3201d0a19124bee59d1e6d93b47)
 2022-01-14 23:02:05 +09:00
Ashutosh Gupta
322750e3c6
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a869ca5c2bdc457375461ae89cdf97315)
(cherry picked from commit 3992ba5d8ca3b8d8f04b447f79d774ab0e2c4c75)
 2022-01-11 02:15:23 +09:00
Ayush Saxena
c3d125e8a2 HADOOP-18061. Update the year to 2022. (#3845). Contributed by Ayush Saxena. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6eb60353db569da4c17bca5c53b0105f908f8d4f)
 2022-01-04 13:14:55 +09:00
Ananya Singh
88e3dafe9e
HDFS-16317 : Backport HDFS-14729 for branch-3.2.3 (#3780) 
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
 2021-12-22 13:32:30 +08:00
jianghuazhu
d93760c8c2 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
(cherry picked from commit b68084ab43d5a033ac33fd1017c502589a46a73c)
 2021-12-21 11:43:57 +00:00
Akira Ajisaka
2fac7630e2 HADOOP-17534. Update Jackson to 2.10.5 and Jackson databind to 2.10.5.1 (#2708) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e119063a078f06523158dba3b9e16ca04768e920)
(cherry picked from commit 07d3d5ea0b1e56c2f6027536d4c4667dbb9c8512)
 2021-12-18 02:27:11 +09:00
Dhananjay Badaya
f2d12b32b9 HADOOP-13500. Synchronizing iteration of Configuration properties object (#3775) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4483607a4edc0f13264f4ea37abd90aba97e1ef0)
(cherry picked from commit 253b217efa1650bbd6da35b0eeeb3d7504171ecb)
 2021-12-18 02:26:03 +09:00
Akira Ajisaka
a7a85292ad HADOOP-16905. Update jackson-databind to 2.10.3 to relieve us from the endless CVE patches (#3748) 
(cherry picked from commit 69faaa1d58ad7de18a8dfa477531653a2c061568)

 Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit bc6874139f534af81a83523cf10508d3d16a032f)
 2021-12-16 20:50:28 +05:30
smarthan
4e9b9adb96 HADOOP-18023. Allow cp command to run with multi threads. (#3721) 
(cherry picked from commit 932a78fe38b34a923f6852a1a19482075806ecba)
(cherry picked from commit 2f3e18697802cce81eb3c976088583ef365daea0)
 2021-11-29 15:26:17 +00:00
smarthan
d15afa3da8 HADOOP-17998. Allow get command to run with multi threads. (#3645) 
(cherry picked from commit 63018dc73f4d29632e93be08d035ab9a7e73531c)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/CopyCommands.java

(cherry picked from commit cbb3ba135cfa3a9bc9f444e8b4a7875758721a3c)
(cherry picked from commit 6e179d421d73fd1a5b0f4dac5f6decaecd424376)
 2021-11-22 16:14:37 +00:00
jianghuazhu
c5c158df73 HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 4c94831364e9258247029c22a222a665771ab4c0)
(cherry picked from commit 7c663043b2c1e207dd8c05e09e17811d68badfac)
(cherry picked from commit dac74b0e115fa466467aa1de2a71fb8704a04639)
 2021-11-22 15:39:39 +00:00
Shubham Gupta
fa737f7684
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069c7d35b4638af3f2299a248eeca3923)
(cherry picked from commit ccb711544b13d4f52846dc5d4ab2491c010a2b9f)
 2021-11-19 00:26:55 +09:00
daimin
cbf4a33549 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3644944fe04cf558d593b96da0263fd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java

(cherry picked from commit 29fd36e2f1e3b34b3cb4c047d93bf88a70dc8149)
 2021-11-03 21:36:47 +00:00
Ananya Singh
0a51b7fd11 HADOOP-17972. Backport HADOOP-17683 for branch-3.2 (#3573) 
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit df9c07fc042cb5bc6b8b6090b579ccb9b8020290)
 2021-10-28 11:47:02 +09:00
Nicholas Marion
6ecaa0ac8b
HADOOP-17971. Exclude IBM Java security classes from being shaded/relocated. (#3567) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3e1135469eec67a8bfbe60feeb41a25aea013596)
(cherry picked from commit 87869c6c266a8fb6b66eccffd81d3dabab909218)
 2021-10-20 14:30:55 +09:00
daimin
e1f11d5f63 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc53a0144311877362787ed9ece0cb118)
 2021-10-18 12:37:38 +01:00
Brahma Reddy Battula
b55002f6e6 HADOOP-17236. Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit eb84793af1e48db05ab827d0cf09963a430615ed)
 2021-10-08 17:23:21 +05:30
Siyao Meng
b93fe96fad
HADOOP-17834. Bump aliyun-sdk-oss to 3.13.0 (#3261) 
Change-Id: I335d4a2cb08c75dc24ef36bdfab51111f87e0762
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3aaac8a1f682ecda4d41221e8a836bc5401e3b1d)

 Conflicts:
	LICENSE-binary

(cherry picked from commit c8db6b1eb2a541af20c4c99c23b627cec2300411)
 2021-10-08 14:59:05 +09:00
Takanobu Asanuma
091f93cf09 HADOOP-17955. Bump netty to the latest 4.1.68. (#3528) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit cf4d2d826cad941ae3648ff9b6057856bb199fc4)
 2021-10-08 10:43:43 +09:00
Rohith Sharma K S
71ba9f1ccc YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. 
(cherry picked from commit f63e4e4f2eee6b75c3a6c3fb5926d44c7e846145)

Conflicts:
	hadoop-project/pom.xml

(cherry picked from commit 51d17f3cb399ae0ed66d3a2357ef091151d25646)
 2021-10-06 22:42:51 +09:00
AngersZhuuuu
2bd78b7eaa
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4112e7b4ee9e6b948ce46a2b152a2db)
(cherry picked from commit 1df3b3de463332b688299078a59265ce405af6ac)
 2021-09-29 18:36:18 +08:00
He Xiaoqiao
1d9a6ffae5
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe0677b266acf237e8458e93743424a6)
(cherry picked from commit cfd02659751d06f40a4c51aa3de2f8df78a42b66)
 2021-09-29 18:36:10 +08:00
Wei-Chiu Chuang
6ef0b3fc21
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
(cherry picked from commit 978ec57445f4178b31a6bfc383c6be032692c288)
 2021-09-28 16:26:55 +09:00
Takanobu Asanuma
9b016b401a HADOOP-17940. Upgrade Kafka to 2.8.1 (#3488) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2068b0041c05a7b53f32606cc24922cfac299ca8)
 2021-09-28 13:43:30 +09:00
Inigo Goiri
1f55483339
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dcbf1901ca6fad11b5c996e54e32ed6b)
(cherry picked from commit 86028d975c050c5d2a4f3fde22cba9b867bb3f03)
 2021-09-27 14:52:51 +09:00
Akira Ajisaka
91acb7b9fe
HADOOP-17917. Backport HADOOP-15993 to branch-3.2 which address CVE-2014-4611. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 85346db636f0dcfe7869e3f09f108014e857c0d5)
 2021-09-24 15:00:12 +09:00
He Xiaoqiao
7fea1dc006
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a72705c5b68ddfe4d29f0d208fd38e7)
(cherry picked from commit 4944dafa297e9aad93d39e861b5dc2835b7cbb5f)
 2021-09-22 18:14:51 +08:00
He Xiaoqiao
56f7e52be5
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff44396eb478f709307d647ca884de99)
(cherry picked from commit 50b0f0dc42d5cdcfbd381527ff774f20852888ba)
 2021-09-22 18:14:46 +08:00
He Xiaoqiao
c7213e4d2b
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
(cherry picked from commit fc5fe67b3f73e98d931d475c02efb52fb3c7a2b4)
 2021-09-22 18:14:41 +08:00
Inigo Goiri
79187b5a5e
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5ca99502d2ac2fbf803f22463eb1343)
(cherry picked from commit 5da46e197b63e4c847732a64e9694c9e485d5f2d)
 2021-09-22 18:14:35 +08:00
Ayush Saxena
cedb8fec9d
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab0fb385252d4eeb19141bd059303d59)
(cherry picked from commit f77965b29eb3f33713c39f1cb341fdc7c91b0220)
 2021-09-22 18:14:29 +08:00
Inigo Goiri
baa4190eab
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d95440de099831e0db38db037f16bdd)
(cherry picked from commit d5501c125fa0fffe846c8be8718e05f55e288959)
 2021-09-22 18:14:21 +08:00
Ahmed Hussein
0d0f344d65
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 14:26:32 +08:00
LeonGao
38973a52f0
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)
 2021-09-14 14:26:27 +08:00
Ahmed Hussein
0879f4ec0f
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)
 2021-09-14 14:26:22 +08:00
Stephen O'Donnell
84e16adab3
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)
 2021-09-14 14:26:09 +08:00
Stephen O'Donnell
3c715a272a
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 14:25:57 +08:00
Ahmed Hussein
a9dbe47a72
HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
(cherry picked from commit 601591c02144c9c0dfed2c4011187dc7bb329157)
 2021-09-03 10:17:41 +09:00
Szilard Nemeth
867acc2e6d
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f76e470527ad94b0894dacb28db01465)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java

(cherry picked from commit 7aea2e1b5c24cd6e2dffbe6942f0dadb6a45c48f)
 2021-09-01 13:16:05 +09:00
Xiaoyu Yao
3c79d65272
HADOOP-15939. Filter overlapping objenesis class in hadoop-client-minicluster. Contributed by Xiaoyu Yao. 
(cherry picked from commit 397f523e22a4f76b5484bed26ef4e6d40200611e)
(cherry picked from commit 3ea74033bf712ad63c4e43ffda50fe13c8522a20)
 2021-08-25 21:10:12 +09:00
Masatake Iwasaki
bcd81a7fb4 HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar. 
(cherry picked from commit 0d59500e8cf89e38cd0f0e45553e4557e5358a8b)
 2021-08-25 11:50:49 +09:00
Masatake Iwasaki
996cbe453a
HADOOP-17850. Upgrade ZooKeeper to 3.4.14 in branch-3.2. (#3308) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 037875ba798422b6b342797c85c66bcf9781824e)
 2021-08-17 13:37:18 +09:00
Masatake Iwasaki
462b808d42 HADOOP-17849. Exclude spotbugs-annotations from transitive dependencies on branch-3.2. (#3305) 
(cherry picked from commit 786888b8d32e35cad49eb1ba54a35f1508234ed5)
 2021-08-16 21:01:33 +00:00
Renukaprasad C
fd1d138a62 HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b90389ae9840ccfdcab411e286547895a647feed)

 Conflicts:
	LICENSE-binary

(cherry picked from commit d18c0a1d25a2fc6a048fd7123655d9d31e07d797)
 2021-08-14 20:28:26 +09:00
Brahma Reddy Battula
182167a9fb Preparing for 3.2.3 release. 2021-08-08 19:17:14 +05:30