Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,456 Commits 326 Branches 371 Tags
Commit Graph

15456 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Weiwei Yang
bdd396b26d YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang. 
(cherry picked from commit 5be9f4a5d05c9cb99348719fe35626b1de3055db)
 2021-10-08 16:08:45 +00:00
Weiwei Yang
dc03afc7df YARN-8127. Resource leak when async scheduling is enabled. Contributed by Tao Yang. 
(cherry picked from commit 7eb783e2634d8c11fb646f1f2fdf597336325312)
 2021-10-04 20:16:40 +00:00
Wei-Chiu Chuang
6d11f7ae8c HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
(cherry picked from commit 978ec57445f4178b31a6bfc383c6be032692c288)
(cherry picked from commit cd7c34f9b4005d27886f73e58bef88e706fcccf9)
 2021-09-24 09:15:48 -07:00
Eric Badger
008bd8afc3 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-23 17:12:45 +00:00
Rintaro Ikeda
25d0a97557 HADOOP-17926. Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself. (#3465) 
(cherry picked from commit 962068d2d8143b97db1ed5e1f5e6c998d0960ded)
 2021-09-22 13:10:07 +00:00
Kihwal Lee
30f48c4d06 HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
 2021-09-15 14:45:28 -05:00
wangzhaohui
79c1a93bd7 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14dde86a5c0b86f9272cf98084478994)
(cherry picked from commit 928dbac0a627c5198b3d5884ef5a7dee3d975855)
(cherry picked from commit 9b63f601d3285c3885b2941c5874f225bfa9ab31)
 2021-09-15 10:10:33 -07:00
Surendra Singh Lilhore
edc138186f HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f070bbfe65c736f6f3eca721b9d227f5)
(cherry picked from commit 2e939515dfbaf26ca466c8a755cedde0ce4e9c1a)
 2021-09-10 15:35:49 -07:00
Ahmed Hussein
7f32a31abf HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 21:34:04 +00:00
Masatake Iwasaki
ba86e01336 HADOOP-17897. Allow nested blocks in switch case in checkstyle settings. (#3394) 
(cherry picked from commit e183ec8998d0272884b73df7eb1a6da5adf1040a)
 2021-09-08 15:14:24 +09:00
Ahmed Hussein
42599f6c8f
HADOOP-17885. Upgrade JSON smart to 1.3.3 on branch-2.10 (#3370) 2021-09-02 16:40:06 -05:00
Ahmed Hussein
d636420e7c HADOOP-17886. Upgrade ant to 1.10.11 (#3371) 
(cherry picked from commit 051207375b30ffa126cf9c961ee97535c1dfabb1)
 2021-09-02 16:22:23 -05:00
Pranav Bheda
bc8069ac48
HADOOP-17223 update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13 (#2242) 
* update org.apache.httpcomponents:httpclient from 4.5.6 to 4.5.13
* update org.apache.httpcomponents:httpcore from 4.4.10 to 4.4.13

(cherry picked from commit be3edd05321a0cbe5e6d60536b4ab6ed34f2cef6)

This closes #3363
 2021-09-01 18:29:53 +09:00
Akira Ajisaka
126b314ae8
HADOOP-15584. Move httpcomponents versions in pom.xml. Contributed by Brandon Scheller. 
(cherry picked from commit 76d0365190af472f922a62aa24dd7288a743210c)
 2021-09-01 18:28:03 +09:00
Szilard Nemeth
b196130c29
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f76e470527ad94b0894dacb28db01465)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java

(cherry picked from commit 7aea2e1b5c24cd6e2dffbe6942f0dadb6a45c48f)
 2021-09-01 13:16:30 +09:00
Abhishek Modi
b79cd11fcd YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph. 
(cherry picked from commit b4097b96a39bad6214b01989e7f2fb37dad70793)
 2021-08-18 17:54:10 -05:00
Akira Ajisaka
bd7e20527e
HADOOP-17370. Upgrade commons-compress to 1.21 2021-08-08 11:37:11 +09:00
Akira Ajisaka
d07955ce50
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246f4aab665fd04ce72d1848bc5da9d4e)
 2021-08-07 11:46:30 +09:00
Chris Douglas
f856092bdb MAPREDUCE-7011. TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec 
(cherry picked from commit 782ba3bf9da52699b27405a3f147464975d1df99)
 2021-08-03 21:00:44 +00:00
hchaverr
2187c6d366 HADOOP-17819. Add extensions to ProtobufRpcEngine RequestHeaderProto. Contributed by Hector Sandoval Chaverri. (#3242) 
(cherry picked from commit 3c8a48e681babe4038e49205624e7be183b3466e)
 2021-07-28 16:04:46 -07:00
Viraj Jasani
5e65126e38 HADOOP-17813. Checkstyle - Allow line length: 100 
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 98412ce2e1453308655d4f43e7f8c922960e6955)
 2021-07-22 15:33:09 -05:00
zhuqi-lucas
34acf9d4c8 YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:35:45 +08:00
Abhishek Das
99b952fcb4 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260, #3218) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-20 18:26:24 -07:00
Jim Brennan
39c8297350 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 21:13:25 +00:00
Jim Brennan
577ed175f9 YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 15:21:02 +00:00
Akira Ajisaka
992d2c5f4e
HADOOP-12665. Document hadoop.security.token.service.use_ip. (#3187) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
(cherry picked from commit c81f82e21d137b6d45c20c481e7bbc6160d72f94)
 2021-07-12 10:19:06 +09:00
Artem Smotrakov
9241470dfd HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 15:06:37 +09:00
Eric Payne
cef04b40b6 MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST) 
(cherry picked from commit 7581413156da396db218e36a966c5749589b31a7)
 2021-07-07 21:40:09 +00:00
Masatake Iwasaki
6df4bc1a8c
HADOOP-17775. Remove JavaScript package from Docker environment. (#3184) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-07 17:49:59 +09:00
Jim Brennan
f7bcc58e0f YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-29 14:22:39 +00:00
Jim Brennan
51be95e234 YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 
(cherry picked from commit 7c7d02edbd6c17ee8ae2c4bf75e87adace059b76)
 2021-06-25 20:58:50 +00:00
Jim Brennan
7d231f4b98 HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2021-06-25 18:06:37 +00:00
Ahmed Hussein
b58be3a12d
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery (#3131). Contributed by Ahmed Hussein. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-23 20:05:31 +05:30
S O'Donnell
caa59d7a58 HDFS-15150. Introduce read write lock to Datanode. Contributed by Stephen O'Donnell and Ahmed Hussein 2021-06-22 16:24:19 +01:00
Kihwal Lee
ab6b5681e8 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 2021-06-16 11:38:30 -05:00
Kihwal Lee
e7c7fb2896 HDFS-15963. Unreleased volume references cause an infinite loop. (#2941) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fdcf1e61b39651a51f8223cfb8ce2e31)
 2021-06-15 19:57:21 -05:00
Takanobu Asanuma
a690e8ae59 HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682270bc6ba151875549d64d81917f425)
 2021-06-15 17:24:27 +09:00
Takanobu Asanuma
f22ff63a15 HADOOP-17756. Increase precommit job timeout from 20 hours to 24 hours. (#3090) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit ae9630f580d73e81042f3b8b2b337aba86130653)
 2021-06-14 10:23:00 +09:00
Hideyuki Furue
a07293f819
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9c7b8cf54ea88833d54fc71a9612c448dc0eb78d)
(cherry picked from commit f5568e15d0d6fdf9593ee869485aa6471b889d52)
 2021-06-06 09:14:18 +09:00
Konstantin V Shvachko
149325484f HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 20:19:44 -07:00
Konstantin V Shvachko
8715a18f95 HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f4f236674ce929164cc460037730abb)

# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogAsync.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NameNodeAdapter.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2021-05-26 16:00:18 -07:00
Akira Ajisaka
76eea30bdd
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3b025e4c0c844e8bfe8d9007af2be3b)
 2021-05-25 18:30:21 +09:00
lujiefsi
13a2e751e0
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790e5ad7d8e21fc3949cdd0f74d496b1b)
 2021-05-17 19:47:03 +09:00
hchaverri
7f6185b17c
HDFS-15912. Allow ProtobufRpcEngine to be extensible (#2999) 2021-05-17 00:37:07 -07:00
Hui Fei
77d9c6d0f7 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151cd4a922372835b18a9a031c9d3475e)
 2021-04-29 12:56:57 -07:00
Eric Badger
7b3a6e96d9 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6ac566a60336c0a28951f09ecda39f24)
 2021-04-23 23:10:50 +00:00
Eric Badger
58b33d6570 MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka. 
Conflicts only in imports

(cherry picked from commit da93cd962c8d19874d09360726628cdde8595a7d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/ReadaheadPool.java

(cherry picked from commit 7ad844ee9ed7eaab4fbb4d38576be10d497460cf)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
 2021-04-23 20:16:11 +00:00
bibinchundatt
94c1cf2ff2 YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 
(cherry picked from commit 6d80b9bc3ff3ba8073e3faf64551b9109d2aa2ad)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java

(cherry picked from commit 4a39165b41dc079dd2de140e610255f363982f59)
 2021-04-23 16:48:04 +00:00
Eric Badger
eb176bb2af YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2021-04-20 00:31:08 +00:00
Ahmed Hussein
55645d78e1
YARN-10733. TimelineService Hbase tests failing with timeouts (#2899) 2021-04-14 13:44:38 -05:00