Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,812 Commits 326 Branches 371 Tags
Commit Graph

19812 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Badger
9327039fb8 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b2bdf36589da1174d4d8ee29acc57f6)
 2021-02-19 23:44:28 +00:00
Jim Brennan
1b5de609ec [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit e6f5dbbe7f556e3525293814ad2239e4f04d88ad)
 2021-02-16 17:20:52 +00:00
Zehao Chen
e162ef84dc HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd27c69b86b837e8da320bad0031f7895)
 2021-02-15 19:41:30 +00:00
Jim Brennan
d8d8546678 YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 
(cherry picked from commit 2117ab6b7125bfcbdae907015001a7599e2f0c4c)
 2021-02-11 21:47:41 +00:00
Akira Ajisaka
3675f60354
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 933fbeffee1e9e2b9d78b5be31466d4d640768c5)
 2021-02-11 08:35:50 +09:00
Eric Badger
9c620c6302 MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.

(cherry picked from commit 462561654bb37dc1d453bfe15c3e75c274bbfb02)
 2021-02-09 19:23:23 +00:00
Jim Brennan
0661977821 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 23:01:50 +00:00
fermi
e4d5ac227c HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c8932c54ced1f0d4b892a54d22d7cfaf3)
 2021-02-05 11:18:09 +08:00
Hui Fei
6e5cc45a16 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc02af81e89e9fb96691f6cdf19ceff5)
 2021-02-04 10:01:55 +08:00
crossfire
69635e09d2 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e204d105fb05807d33387a048e9ddb762)
 2021-02-02 10:44:59 +00:00
Akira Ajisaka
5302731ac4
HADOOP-17354. Move Jenkinsfile outside of the root directory. (#2647) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 84b154ebc054843234e9a3c4c27748f87582807f)
 2021-02-01 18:53:42 +09:00
He Xiaoqiao
3a835753ee
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:00:41 +08:00
Jim Brennan
dfe23ab6ba [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939332c2b17a0f470ac650ab55c6c0f64)
 2021-01-28 18:41:09 +00:00
Jim Brennan
7943b31c29 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f670c181d1fda99e76729e920fff287e)
 2021-01-28 17:14:27 +00:00
Akira Ajisaka
fb5b97305c
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a2e9f48050e0c41440b36c759ec068f)
 2021-01-24 22:58:06 +09:00
Akira Ajisaka
0eee9bda61
HDFS-15731. Reduce threadCount for unit tests to reduce the memory usage (#2556) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9cf408d6baf2fa9071729cac57516d366e928a9f)
 2021-01-24 22:53:42 +09:00
Anton Kutuzov
397ff302b3 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:16:21 -08:00
He Xiaoqiao
d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00
Akira Ajisaka
33a71c8b9e
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a8470975e67cce0e3e4d8eb8c7b6e7b6992f)
 2021-01-21 10:12:03 +09:00
Masatake Iwasaki
9d5144e66d
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c72c27b02032ed3cf9dde15745ed605)
 2021-01-14 16:44:59 +09:00
Ahmed Hussein
640d8a1f75
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186cc0c5c1f90028a9383851d47791bf04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
 2021-01-14 10:15:58 +09:00
Eric Badger
d5b8b2b50d YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba3f82802e2ebf39d44fe6c30037583c)
 2021-01-14 00:01:32 +00:00
Eric Payne
a4dea9de14 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit a093bd859da0d1f2834f873d4dbe2bbd2f00f2d2)
 2021-01-13 19:54:14 +00:00
touchida
669c6a7cf4
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba462770c99f4fe3521f4448d5bc7b0a)
 2021-01-13 11:39:27 +09:00
lzx404243
286009b869
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad331e3942547b88fdc4c0ad59865a71a)
 2021-01-12 16:08:15 +09:00
Ahmed Hussein
194e0c3cc2
HADOOP-17438. Increase docker memory limit in Jenkins (#2560) 
Reviewed-by: Eric Badger <ebadger@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae894ecfcc46e40d9e2f535ac6acec50f8136fae)
 2021-01-12 15:36:49 +09:00
Szilard Nemeth
770a56315e YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:51:06 +01:00
Wei-Chiu Chuang
13768fe6f0 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf576e2da29ab86e2920302b82b47e435d)
(cherry picked from commit 94c126cc9e9aadaf9aa28ebd55a8712aca54e709)
(cherry picked from commit 8af0fcf628b0a8ab1b9b44762eb4a9341f9d7b6a)
 2021-01-04 21:00:29 -08:00
He Xiaoqiao
5d0eb4c6f0
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:35:36 +08:00
He Xiaoqiao
6e2493f48e
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:49:26 +08:00
Eric Badger
52791d2005 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230fa7adbb4a041c9a33bea2ef5a36fd3)
 2020-12-21 23:32:18 +00:00
Ahmed Hussein
a23faaec18 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234256e342fd9b5ef6fe4455471009693)
 2020-12-17 18:29:03 +00:00
He Xiaoqiao
1fe15113cf HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc001e28d9912e8d985d09bab75aedb87c)
(cherry picked from commit 3a860e876e799ef44fca34e3cb19a89dbd4f6744)
(cherry picked from commit 8d7600ae6b7fadba1b4695666cf9287be021d481)
 2020-12-15 14:02:15 -08:00
Ayush Saxena
9e50a1082b HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-12-15 11:05:17 +05:30
Ayush Saxena
a68ca4a292 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-12-15 10:35:01 +05:30
Ayush Saxena
a09d636e2c HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aaa628b750ea5ad2eb0d68e0fdbc55d5)
(cherry picked from commit ed78df0f7596ee613db5ee9a591c05f03e6b3fc4)
 2020-12-14 20:41:36 -08:00
S O'Donnell
e7df47c45c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 52adda89c422dbaa30ca48ea70c45693fdcb9df7)
 2020-12-14 11:51:44 +00:00
Jim Brennan
392f69471d HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb678496393b625f430084a6c64e24b804)
 2020-12-11 21:07:03 +00:00
Jim Brennan
777f426d6a HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 22:16:52 +00:00
Neil
6258405f10 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50672ebdd7f12b9a2235aec37b2a07cf)
(cherry picked from commit bc5458bbd4e6829bc02de57671c017fbdac06849)
(cherry picked from commit 9123f9245cc49764d93a5295544a17519191430f)
 2020-12-10 13:50:55 -08:00
Ankit Kumar
b89aff6db4
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519) 
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320a9d11d2fd8c86fb5fa4e2d1c45d9ef)
 2020-12-09 10:54:39 +09:00
Ahmed Hussein
8b8a685b65 HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a21722be706d7ab682323285e794f71cadf)
 2020-12-08 19:54:12 +00:00
Hui Fei
2fde1f8bbf HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 11:03:02 +08:00
crossfire
173d298c63 HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d5765c98d41c78736124b7b7f078aa2)
(cherry picked from commit edd9b659caf77d14caf6df88bac3cedf025e1f25)
(cherry picked from commit 31a6b45117c7af2b045cc19461e1c232ce805b02)
 2020-12-07 15:50:58 -08:00
Jim Brennan
8df2902a53 HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 
(cherry picked from commit 6a5864ee4ac387a390642f56aff8581fb5ed8ffd)
 2020-12-07 21:38:39 +00:00
Jim Brennan
486da59e1f HADOOP-17389. KMS should log full UGI principal. (#2476) Contributed by Ahmed Hussein 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
    Reviewed-by: Jim Brennan <jbrennan@apache.org>
 2020-12-07 17:57:28 +00:00
Ayush Saxena
cfb96240dc
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 
(cherry picked from commit b52fd05d42d9a76f6936a5d86c23fcd66244fe3d)
 2020-12-04 11:19:21 +09:00
Takanobu Asanuma
43f7ab2cf0
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 
(cherry picked from commit 559cb11551d7045015aa60afe7e7998fe0b2ef13)
 2020-12-04 11:19:14 +09:00
Akira Ajisaka
e17b7efd3b
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) 
(cherry picked from commit 3e7b65c3e5920ed78bd87b68de1075003d86e7a0)
 2020-12-04 10:43:32 +09:00
Ahmed Hussein
5f5d7ecf59 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040806c4c9687288de89e42cfacf0365e)
(cherry picked from commit a7dbd3b6f978d28ef7e046add2e3270ae3d9d9f6)
(cherry picked from commit f547cd43d140dc680fab1782cda071f4a41dddc5)
 2020-12-03 15:15:47 -08:00