Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,712 Commits 326 Branches 371 Tags
Commit Graph

21712 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Rohith Sharma K S
51d17f3cb3 YARN-8936. Bump up Atsv2 hbase versions. Contributed by Vrushali C. 
(cherry picked from commit f63e4e4f2eee6b75c3a6c3fb5926d44c7e846145)

Conflicts:
	hadoop-project/pom.xml
 2021-10-06 20:49:00 +09:00
Viraj Jasani
28d4affea0 HADOOP-17947. Additional element types for VisibleForTesting (ADDENDUM) (#3521) 
(cherry picked from commit 783e4805e7ef46da6095e0375fe8dd5e9cd26e85)
 2021-10-06 02:19:44 +09:00
Viraj Jasani
ce30f796f6 HADOOP-17947. Provide alternative to Guava VisibleForTesting (#3505) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 5b1d594005293e8de7e58cd70cde88c9041107cf)
 2021-10-05 10:02:08 +09:00
AngersZhuuuu
1df3b3de46
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4112e7b4ee9e6b948ce46a2b152a2db)
 2021-09-29 17:44:32 +08:00
He Xiaoqiao
cfd0265975
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe0677b266acf237e8458e93743424a6)
 2021-09-29 17:44:00 +08:00
Takanobu Asanuma
3851ace026 HADOOP-17940. Upgrade Kafka to 2.8.1 (#3488) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 2068b0041c05a7b53f32606cc24922cfac299ca8)
 2021-09-28 13:36:40 +09:00
Inigo Goiri
86028d975c
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dcbf1901ca6fad11b5c996e54e32ed6b)
 2021-09-27 14:52:27 +09:00
Wei-Chiu Chuang
978ec57445 HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
 2021-09-24 09:15:04 -07:00
Akira Ajisaka
85346db636
HADOOP-17917. Backport HADOOP-15993 to branch-3.2 which address CVE-2014-4611. Contributed by Brahma Reddy Battula. 2021-09-24 14:59:21 +09:00
Eric Badger
ce3def0aa7 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-22 20:29:12 +00:00
Rintaro Ikeda
5de18fb483 HADOOP-17926. Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself. (#3465) 
(cherry picked from commit 962068d2d8143b97db1ed5e1f5e6c998d0960ded)
 2021-09-22 13:09:38 +00:00
He Xiaoqiao
4944dafa29
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a72705c5b68ddfe4d29f0d208fd38e7)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
50b0f0dc42
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff44396eb478f709307d647ca884de99)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
fc5fe67b3f
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
5da46e197b
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5ca99502d2ac2fbf803f22463eb1343)
 2021-09-17 22:16:11 +08:00
Ayush Saxena
f77965b29e
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab0fb385252d4eeb19141bd059303d59)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
d5501c125f
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d95440de099831e0db38db037f16bdd)
 2021-09-17 22:16:07 +08:00
Rintaro Ikeda
a777264c7f HADOOP-17919. Fix command line example in Hadoop Cluster Setup documentation. (#3453) 
(cherry picked from commit 607c20c6122238ee579a3ad5da882128e7203fad)
 2021-09-17 13:35:54 +00:00
Steve Loughran
5094dede8e
HADOOP-17126. implement non-guava Precondition checkNotNull 
This adds a new class org.apache.hadoop.util.Preconditions which is

* @Private/@Unstable
* Intended to allow us to move off Google Guava
* Is designed to be trivially backportable
  (i.e contains no references to guava classes internally)

Please use this instead of the guava equivalents, where possible.

Contributed by: Ahmed Hussein

Change-Id: Ic392451bcfe7d446184b7c995734bcca8c07286e
 2021-09-17 11:37:29 +01:00
wangzhaohui
928dbac0a6 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14dde86a5c0b86f9272cf98084478994)
 2021-09-15 10:06:27 -07:00
Szilard Nemeth
4066cb0793 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-14 18:09:35 +02:00
Tamas Domok
478d2f2897
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir (#3410) 
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-14 17:35:07 +02:00
EungsopYoo
4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
(cherry picked from commit 51a4a23e373abe9b2f6469ffcd5bc8294307f025)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein
a3920e89de
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 12:31:49 +08:00
LeonGao
3c03ade285
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)
 2021-09-14 12:31:44 +08:00
Ahmed Hussein
33c60ba060
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)
 2021-09-14 12:31:34 +08:00
Stephen O'Donnell
ad55d5d1d2
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)
 2021-09-14 12:31:20 +08:00
Stephen O'Donnell
7039433146
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 12:31:00 +08:00
He Xiaoqiao
ca182b6e5f
Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69bc578db1ce95198aa7fa7bc7199320.
 2021-09-14 12:27:42 +08:00
Ahmed Hussein
318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Ahmed Hussein
1944e0d714 HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:28:29 +00:00
Masatake Iwasaki
34d443a2f0 HADOOP-17897. Allow nested blocks in switch case in checkstyle settings. (#3394) 
(cherry picked from commit e183ec8998d0272884b73df7eb1a6da5adf1040a)
 2021-09-08 14:41:56 +09:00
Chris Nauroth
736367e578 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348) 
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
 2021-09-03 19:19:44 +00:00
Viraj Jasani
25fe54b890
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99a157fa4ad174c4ff979414af2edb67a98eb9fe)
 2021-09-03 11:06:19 +09:00
Ahmed Hussein
601591c021 HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
 2021-09-02 16:18:51 -05:00
lzx404243
8f70d48d14
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter (#2500) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
(cherry picked from commit 6187f76f11e885023c99382f0fd18f8fea8bcbb6)
 2021-09-01 17:20:48 +09:00
Szilard Nemeth
7aea2e1b5c
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f76e470527ad94b0894dacb28db01465)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
 2021-09-01 10:58:41 +09:00
lzx404243
5c9ab0ce09
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() (#2968) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7b5be74228cb4a68d11f52ac061829b70a4f3144)
 2021-08-30 08:48:11 +09:00
jianghuazhu
dac74b0e11
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 4c94831364e9258247029c22a222a665771ab4c0)
(cherry picked from commit 7c663043b2c1e207dd8c05e09e17811d68badfac)
 2021-08-27 12:50:22 +08:00
Xiaoyu Yao
3ea74033bf
HADOOP-15939. Filter overlapping objenesis class in hadoop-client-minicluster. Contributed by Xiaoyu Yao. 
(cherry picked from commit 397f523e22a4f76b5484bed26ef4e6d40200611e)
 2021-08-25 21:09:44 +09:00
jianghuazhu
3b38d011a8
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit ad54f5195c8c01f333703c55cd70703109d75f29)
(cherry picked from commit 2b2f8f575bffb18df53a3cf70cc45ba6384798e2)
 2021-08-25 17:36:07 +08:00
Masatake Iwasaki
50c7fec545 HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar. 
(cherry picked from commit 0d59500e8cf89e38cd0f0e45553e4557e5358a8b)
 2021-08-25 10:57:22 +09:00
Viraj Jasani
f26a547f2b HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 
(cherry picked from commit fc566ad9b0e1336381c5f6b321cdcff81768f47a)
 2021-08-24 22:46:09 +09:00
litao
ce100b5807
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e2bf4c87f12b53e508edf8fee05856a)
(cherry picked from commit 0067c06606e8e3176148fa1856aeb3ae0930951c)
 2021-08-19 12:31:44 +08:00
Abhishek Modi
6a02596f42 YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph. 
(cherry picked from commit b4097b96a39bad6214b01989e7f2fb37dad70793)
 2021-08-18 17:48:17 -05:00
Masatake Iwasaki
037875ba79
HADOOP-17850. Upgrade ZooKeeper to 3.4.14 in branch-3.2. (#3308) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-17 13:36:25 +09:00
Masatake Iwasaki
786888b8d3
HADOOP-17849. Exclude spotbugs-annotations from transitive dependencies on branch-3.2. (#3305) 2021-08-16 21:46:19 +09:00
Renukaprasad C
d18c0a1d25 HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b90389ae9840ccfdcab411e286547895a647feed)

 Conflicts:
	LICENSE-binary
 2021-08-14 20:15:36 +09:00
Tamas Domok
3fdb056b54
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty. 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-10 01:35:18 +02:00
Brahma Reddy Battula
3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30