Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,684 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

21684 Commits

Author SHA1 Message Date
Hexiaoqiao 21fc30c1aa Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69.
 2021-09-14 12:36:32 +08:00
Ahmed Hussein 318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Ahmed Hussein 1944e0d714 HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:28:29 +00:00
Masatake Iwasaki 34d443a2f0 HADOOP-17897. Allow nested blocks in switch case in checkstyle settings. (#3394) 
(cherry picked from commit e183ec8998)
 2021-09-08 14:41:56 +09:00
Chris Nauroth 736367e578 HADOOP-15129. Datanode caches namenode DNS lookup failure and cannot startup (#3348) 
Co-authored-by:  Karthik Palaniappan

Change-Id: Id079a5319e5e83939d5dcce5fb9ebe3715ee864f
 2021-09-03 19:19:44 +00:00
Viraj Jasani 25fe54b890
HADOOP-17874. ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner (#3343) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 99a157fa4a)
 2021-09-03 11:06:19 +09:00
Ahmed Hussein 601591c021 HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 
(cherry picked from commit 051207375b)
 2021-09-02 16:18:51 -05:00
lzx404243 8f70d48d14
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter (#2500) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
(cherry picked from commit 6187f76f11)
 2021-09-01 17:20:48 +09:00
Szilard Nemeth 7aea2e1b5c
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f7)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java
 2021-09-01 10:58:41 +09:00
lzx404243 5c9ab0ce09
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() (#2968) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7b5be74228)
 2021-08-30 08:48:11 +09:00
jianghuazhu dac74b0e11
HDFS-16173.Improve CopyCommands#Put#executor queue configurability. (#3302) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 4c94831364)
(cherry picked from commit 7c663043b2)
 2021-08-27 12:50:22 +08:00
Xiaoyu Yao 3ea74033bf
HADOOP-15939. Filter overlapping objenesis class in hadoop-client-minicluster. Contributed by Xiaoyu Yao. 
(cherry picked from commit 397f523e22)
 2021-08-25 21:09:44 +09:00
jianghuazhu 3b38d011a8
HDFS-16175.Improve the configurable value of Server #PURGE_INTERVAL_NANOS. (#3307) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit ad54f5195c)
(cherry picked from commit 2b2f8f575b)
 2021-08-25 17:36:07 +08:00
Masatake Iwasaki 50c7fec545 HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar. 
(cherry picked from commit 0d59500e8c)
 2021-08-25 10:57:22 +09:00
Viraj Jasani f26a547f2b HADOOP-17858. Avoid possible class loading deadlock with VerifierNone initialization (#3321) 
(cherry picked from commit fc566ad9b0)
 2021-08-24 22:46:09 +09:00
litao ce100b5807
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e)
(cherry picked from commit 0067c06606e8e3176148fa1856aeb3ae0930951c)
 2021-08-19 12:31:44 +08:00
Abhishek Modi 6a02596f42 YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph. 
(cherry picked from commit b4097b96a3)
 2021-08-18 17:48:17 -05:00
Masatake Iwasaki 037875ba79
HADOOP-17850. Upgrade ZooKeeper to 3.4.14 in branch-3.2. (#3308) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-17 13:36:25 +09:00
Masatake Iwasaki 786888b8d3
HADOOP-17849. Exclude spotbugs-annotations from transitive dependencies on branch-3.2. (#3305) 2021-08-16 21:46:19 +09:00
Renukaprasad C d18c0a1d25 HADOOP-17844. Upgrade JSON smart to 2.4.7 (#3299) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b90389ae98)

 Conflicts:
	LICENSE-binary
 2021-08-14 20:15:36 +09:00
Tamas Domok 3fdb056b54
YARN-10814. Fallback to RandomSecretProvider if the secret file is empty. 
The rest endpoint would be unusable with an empty secret file
(throwing IllegalArgumentExceptions).

Any IO error would have resulted in the same fallback path.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-08-10 01:35:18 +02:00
Brahma Reddy Battula 3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30
Akira Ajisaka 840c6ef7fa
HADOOP-17370. Upgrade commons-compress to 1.21 (#3274) 
(cherry picked from commit 3565c9477d)

 Conflicts:
	LICENSE-binary
 2021-08-08 11:26:03 +09:00
Bryan Beaudreault dab829063d
HADOOP-17840: Backport HADOOP-17837 to branch-3.2 (#3275) 
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
 2021-08-07 09:49:46 +05:30
Akira Ajisaka 78c5fdd90d
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246)
 2021-08-07 11:28:22 +09:00
wangzhaohui b98a8d259b
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c)
(cherry picked from commit 7068e18842)
 2021-08-06 18:25:28 +08:00
wangzhaohui 6b10154123 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b)
 2021-08-06 09:52:51 +01:00
Viraj Jasani 2f5e527910 HADOOP-17808. Avoid excessive logging for interruption (ADDENDUM) (#3267) 
(cherry picked from commit 9fe1f24ec1)
 2021-08-06 09:54:34 +08:00
Akira Ajisaka c40abe10c0
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3265) 
* YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils (#3081)

Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit ca27744770)

* Use mockito 1.x

Co-authored-by: Viraj Jasani <vjasani@apache.org>
 2021-08-05 10:05:10 +09:00
Abhishek Modi c5e2ec465b
YARN-9990. Testcase fails with Insufficient configured threads: required=16 < max=10. Contributed by Prabhu Joseph. 
(cherry picked from commit a2dadac790)
 2021-08-04 17:05:35 +09:00
Jim Brennan 1f9a315625
YARN-10337. Amendment to fix import as in HADOOP-17100 
(cherry picked from commit 53a1c7653f)
 2021-08-04 16:38:52 +09:00
Prabhu Joseph b9d0412fd5
YARN-10337. Fix failing testcase TestRMHATimelineCollectors. 
Contributed by Bilwa S T.

(cherry picked from commit 2bbd00dff4)
 2021-08-04 16:37:47 +09:00
Akira Ajisaka d830e84217
YARN-9875. Improve fair scheduler configuration store on HDFS. (#3262) 
Contributed by Prabhu Joseph

(cherry picked from commit 155864da00)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/conf/TestFSSchedulerConfigurationStore.java

Co-authored-by: Eric Yang <eyang@apache.org>
 2021-08-04 16:12:54 +09:00
Giovanni Matteo Fumarola 2f3c0b8dd1
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 
(cherry picked from commit 1a08302897)
 2021-08-04 15:54:08 +09:00
Akira Ajisaka b838647b16
HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#3264) 
* HADOOP-16768. SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data. (#2003)

(cherry picked from commit 328eae9a14)

* Add assertj-core dependency to hadoop-common for test
* Bump assertj-core version to 3.12.2 to use assertThat.hasSizeGreaterThan
 2021-08-04 15:46:00 +09:00
Akira Ajisaka e259d85a99
YARN-8992. Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue. (Contributed by Wilfred Spiegelenburg) (#3257) 
(cherry picked from commit a41b648e98)

Co-authored-by: Haibo Chen <haibochen@apache.org>
 2021-08-04 10:17:51 +09:00
Siyao Meng fe59fe6744 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00)
(cherry picked from commit 72508e6430)
 2021-08-03 20:45:17 +01:00
Akira Ajisaka 786a43e729
YARN-8990. Fix fair scheduler race condition in app submit and queue cleanup. (Contributed by Wilfred Spiegelenburg) (#3254) 
(cherry picked from commit 524a7523c4)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java

Co-authored-by: Haibo Chen <haibochen@apache.org>
 2021-08-03 13:12:01 +09:00
Vrushali C 521c1085b4
YARN-9338 Timeline related testcases are failing. Contributed by Abhishek Modi. 
(cherry picked from commit 17a3e14d25)
 2021-08-03 11:42:28 +09:00
jenny 18ba3d6b1e
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup (#1808) 
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e31169c864)
 2021-08-02 15:40:14 +09:00
Szilard Nemeth e4df745c13 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:54:51 +02:00
Szilard Nemeth 7a8b6265c6 YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-07-29 19:22:57 +02:00
He Xiaoqiao f46a9e229e
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4)
 2021-07-29 11:26:09 +08:00
hchaverr 4f602cb17b HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681)
 2021-07-28 15:59:45 -07:00
9uapaw 1dc343726c YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a)
(cherry picked from commit f001595b47)
 2021-07-28 17:14:33 +02:00
Szilard Nemeth 6bf02a63e7 YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-07-28 14:54:36 +02:00
Akira Ajisaka 7fe4f3cb89 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60c.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488)
 2021-07-26 01:54:57 +09:00
Viraj Jasani 1df581a20c HADOOP-17808. ipc.Client to set interrupt flag after catching InterruptedException (#3219) 
(cherry picked from commit 3a52bfc5db)
 2021-07-23 12:57:36 +08:00
Viraj Jasani 1b1b3ceddc HADOOP-17813. Checkstyle - Allow line length: 100 
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 98412ce2e1)
 2021-07-22 15:09:12 -05:00
zhuqi-lucas cfe36ba762 YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-22 10:13:55 +08:00