Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,910 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19910 Commits

Author SHA1 Message Date
Zehao Chen e162ef84dc HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 19:41:30 +00:00
Jim Brennan d8d8546678 YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 
(cherry picked from commit 2117ab6b71)
 2021-02-11 21:47:41 +00:00
Akira Ajisaka 3675f60354
HADOOP-17516. Upgrade ant to 1.10.9. (#2683) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 933fbeffee)
 2021-02-11 08:35:50 +09:00
Eric Badger 9c620c6302 MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log. 
Contributed by Jim Brennan.

(cherry picked from commit 462561654b)
 2021-02-09 19:23:23 +00:00
Jim Brennan 0661977821 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 23:01:50 +00:00
fermi e4d5ac227c HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c89)
 2021-02-05 11:18:09 +08:00
Hui Fei 6e5cc45a16 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 10:01:55 +08:00
crossfire 69635e09d2 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 10:44:59 +00:00
Akira Ajisaka 5302731ac4
HADOOP-17354. Move Jenkinsfile outside of the root directory. (#2647) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 84b154ebc0)
 2021-02-01 18:53:42 +09:00
He Xiaoqiao 3a835753ee
HADOOP-17501. Fix logging typo in ShutdownHookManager. Contributed by Fengnan Li. 2021-01-31 20:00:41 +08:00
Jim Brennan dfe23ab6ba [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:41:09 +00:00
Jim Brennan 7943b31c29 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 17:14:27 +00:00
Akira Ajisaka fb5b97305c
HADOOP-17478. Improve the description of hadoop.http.authentication.signature.secret.file (#2628) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d82009599a)
 2021-01-24 22:58:06 +09:00
Akira Ajisaka 0eee9bda61
HDFS-15731. Reduce threadCount for unit tests to reduce the memory usage (#2556) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 9cf408d6ba)
 2021-01-24 22:53:42 +09:00
Anton Kutuzov 397ff302b3 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5)
 2021-01-22 18:16:21 -08:00
He Xiaoqiao d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00
Akira Ajisaka 33a71c8b9e
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:12:03 +09:00
Masatake Iwasaki 9d5144e66d
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c)
 2021-01-14 16:44:59 +09:00
Ahmed Hussein 640d8a1f75
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
 2021-01-14 10:15:58 +09:00
Eric Badger d5b8b2b50d YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba)
 2021-01-14 00:01:32 +00:00
Eric Payne a4dea9de14 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit a093bd859d)
 2021-01-13 19:54:14 +00:00
touchida 669c6a7cf4
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba)
 2021-01-13 11:39:27 +09:00
lzx404243 286009b869
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad3)
 2021-01-12 16:08:15 +09:00
Ahmed Hussein 194e0c3cc2
HADOOP-17438. Increase docker memory limit in Jenkins (#2560) 
Reviewed-by: Eric Badger <ebadger@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae894ecfcc)
 2021-01-12 15:36:49 +09:00
Szilard Nemeth 770a56315e YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:51:06 +01:00
Wei-Chiu Chuang 13768fe6f0 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
(cherry picked from commit 94c126cc9e)
(cherry picked from commit 8af0fcf628)
 2021-01-04 21:00:29 -08:00
He Xiaoqiao 5d0eb4c6f0
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:35:36 +08:00
He Xiaoqiao 6e2493f48e
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:49:26 +08:00
Eric Badger 52791d2005 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)
 2020-12-21 23:32:18 +00:00
Ahmed Hussein a23faaec18 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 18:29:03 +00:00
He Xiaoqiao 1fe15113cf HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc00)
(cherry picked from commit 3a860e876e)
(cherry picked from commit 8d7600ae6b)
 2020-12-15 14:02:15 -08:00
Ayush Saxena 9e50a1082b HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-12-15 11:05:17 +05:30
Ayush Saxena a68ca4a292 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-12-15 10:35:01 +05:30
Ayush Saxena a09d636e2c HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aa)
(cherry picked from commit ed78df0f75)
 2020-12-14 20:41:36 -08:00
S O'Donnell e7df47c45c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 52adda89c4)
 2020-12-14 11:51:44 +00:00
Jim Brennan 392f69471d HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb67)
 2020-12-11 21:07:03 +00:00
Jim Brennan 777f426d6a HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 22:16:52 +00:00
Neil 6258405f10 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
(cherry picked from commit bc5458bbd4)
(cherry picked from commit 9123f9245c)
 2020-12-10 13:50:55 -08:00
Ankit Kumar b89aff6db4
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519) 
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320)
 2020-12-09 10:54:39 +09:00
Ahmed Hussein 8b8a685b65 HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a2172)
 2020-12-08 19:54:12 +00:00
Hui Fei 2fde1f8bbf HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 11:03:02 +08:00
crossfire 173d298c63 HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d)
(cherry picked from commit edd9b659ca)
(cherry picked from commit 31a6b45117)
 2020-12-07 15:50:58 -08:00
Jim Brennan 8df2902a53 HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 
(cherry picked from commit 6a5864ee4a)
 2020-12-07 21:38:39 +00:00
Jim Brennan 486da59e1f HADOOP-17389. KMS should log full UGI principal. (#2476) Contributed by Ahmed Hussein 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
    Reviewed-by: Jim Brennan <jbrennan@apache.org>
 2020-12-07 17:57:28 +00:00
Ayush Saxena cfb96240dc
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 
(cherry picked from commit b52fd05d42)
 2020-12-04 11:19:21 +09:00
Takanobu Asanuma 43f7ab2cf0
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 
(cherry picked from commit 559cb11551)
 2020-12-04 11:19:14 +09:00
Akira Ajisaka e17b7efd3b
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) 
(cherry picked from commit 3e7b65c3e5)
 2020-12-04 10:43:32 +09:00
Ahmed Hussein 5f5d7ecf59 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040)
(cherry picked from commit a7dbd3b6f9)
(cherry picked from commit f547cd43d1)
 2020-12-03 15:15:47 -08:00
Jim Brennan c43eca6cdc HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 5bfb97bc7d)
 2020-12-03 19:47:46 +00:00
Andrea Scarpino c8178a9caa
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b)
 2020-12-04 00:18:12 +09:00