Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,796 Commits 326 Branches 371 Tags
Commit Graph

19796 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Anton Kutuzov
397ff302b3 HDFS-15632. AbstractContractDeleteTest should set recursive peremeter to true for recursive test cases. Contributed by Anton Kutuzov. 
(cherry picked from commit 91d4ba57c5b85379303ac8fb2a1a03ba10b07d4e)
 2021-01-22 18:16:21 -08:00
He Xiaoqiao
d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00
Akira Ajisaka
33a71c8b9e
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a8470975e67cce0e3e4d8eb8c7b6e7b6992f)
 2021-01-21 10:12:03 +09:00
Masatake Iwasaki
9d5144e66d
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c72c27b02032ed3cf9dde15745ed605)
 2021-01-14 16:44:59 +09:00
Ahmed Hussein
640d8a1f75
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186cc0c5c1f90028a9383851d47791bf04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
 2021-01-14 10:15:58 +09:00
Eric Badger
d5b8b2b50d YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba3f82802e2ebf39d44fe6c30037583c)
 2021-01-14 00:01:32 +00:00
Eric Payne
a4dea9de14 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit a093bd859da0d1f2834f873d4dbe2bbd2f00f2d2)
 2021-01-13 19:54:14 +00:00
touchida
669c6a7cf4
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba462770c99f4fe3521f4448d5bc7b0a)
 2021-01-13 11:39:27 +09:00
lzx404243
286009b869
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499) 
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad331e3942547b88fdc4c0ad59865a71a)
 2021-01-12 16:08:15 +09:00
Ahmed Hussein
194e0c3cc2
HADOOP-17438. Increase docker memory limit in Jenkins (#2560) 
Reviewed-by: Eric Badger <ebadger@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae894ecfcc46e40d9e2f535ac6acec50f8136fae)
 2021-01-12 15:36:49 +09:00
Szilard Nemeth
770a56315e YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:51:06 +01:00
Wei-Chiu Chuang
13768fe6f0 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf576e2da29ab86e2920302b82b47e435d)
(cherry picked from commit 94c126cc9e9aadaf9aa28ebd55a8712aca54e709)
(cherry picked from commit 8af0fcf628b0a8ab1b9b44762eb4a9341f9d7b6a)
 2021-01-04 21:00:29 -08:00
He Xiaoqiao
5d0eb4c6f0
HDFS-15751. Add documentation for msync() API to filesystem.md. Contributed by Konstantin V Shvachko. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Chao Sun <sunchao@apache.org>
 2021-01-03 16:35:36 +08:00
He Xiaoqiao
6e2493f48e
HADOOP-17445. Update the year to 2021. Contributed by He Xiaoqiao. 2020-12-24 20:49:26 +08:00
Eric Badger
52791d2005 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230fa7adbb4a041c9a33bea2ef5a36fd3)
 2020-12-21 23:32:18 +00:00
Ahmed Hussein
a23faaec18 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234256e342fd9b5ef6fe4455471009693)
 2020-12-17 18:29:03 +00:00
He Xiaoqiao
1fe15113cf HADOOP-17068. Client fails forever when namenode ipaddr changed. Contributed by Sean Chow. 
(cherry picked from commit fa14e4bc001e28d9912e8d985d09bab75aedb87c)
(cherry picked from commit 3a860e876e799ef44fca34e3cb19a89dbd4f6744)
(cherry picked from commit 8d7600ae6b7fadba1b4695666cf9287be021d481)
 2020-12-15 14:02:15 -08:00
Ayush Saxena
9e50a1082b HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-12-15 11:05:17 +05:30
Ayush Saxena
a68ca4a292 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-12-15 10:35:01 +05:30
Ayush Saxena
a09d636e2c HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aaa628b750ea5ad2eb0d68e0fdbc55d5)
(cherry picked from commit ed78df0f7596ee613db5ee9a591c05f03e6b3fc4)
 2020-12-14 20:41:36 -08:00
S O'Donnell
e7df47c45c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 52adda89c422dbaa30ca48ea70c45693fdcb9df7)
 2020-12-14 11:51:44 +00:00
Jim Brennan
392f69471d HADOOP-13571. ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0. Contributed by Eric Badger 
(cherry picked from commit 6de1a8eb678496393b625f430084a6c64e24b804)
 2020-12-11 21:07:03 +00:00
Jim Brennan
777f426d6a HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 22:16:52 +00:00
Neil
6258405f10 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50672ebdd7f12b9a2235aec37b2a07cf)
(cherry picked from commit bc5458bbd4e6829bc02de57671c017fbdac06849)
(cherry picked from commit 9123f9245cc49764d93a5295544a17519191430f)
 2020-12-10 13:50:55 -08:00
Ankit Kumar
b89aff6db4
YARN-10491. Fix deprecation warnings in SLSWebApp.java (#2519) 
Signed-off-by: Akira Ajisaka <ajisaka@apache.org>
(cherry picked from commit aaf9e3d320a9d11d2fd8c86fb5fa4e2d1c45d9ef)
 2020-12-09 10:54:39 +09:00
Ahmed Hussein
8b8a685b65 HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a21722be706d7ab682323285e794f71cadf)
 2020-12-08 19:54:12 +00:00
Hui Fei
2fde1f8bbf HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 11:03:02 +08:00
crossfire
173d298c63 HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d5765c98d41c78736124b7b7f078aa2)
(cherry picked from commit edd9b659caf77d14caf6df88bac3cedf025e1f25)
(cherry picked from commit 31a6b45117c7af2b045cc19461e1c232ce805b02)
 2020-12-07 15:50:58 -08:00
Jim Brennan
8df2902a53 HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 
(cherry picked from commit 6a5864ee4ac387a390642f56aff8581fb5ed8ffd)
 2020-12-07 21:38:39 +00:00
Jim Brennan
486da59e1f HADOOP-17389. KMS should log full UGI principal. (#2476) Contributed by Ahmed Hussein 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
    Reviewed-by: Jim Brennan <jbrennan@apache.org>
 2020-12-07 17:57:28 +00:00
Ayush Saxena
cfb96240dc
HDFS-13404. Addendum: RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fail. Contributed by Takanobu Asanuma. 
(cherry picked from commit b52fd05d42d9a76f6936a5d86c23fcd66244fe3d)
 2020-12-04 11:19:21 +09:00
Takanobu Asanuma
43f7ab2cf0
HDFS-13404. RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended fails. 
(cherry picked from commit 559cb11551d7045015aa60afe7e7998fe0b2ef13)
 2020-12-04 11:19:14 +09:00
Akira Ajisaka
e17b7efd3b
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136) 
(cherry picked from commit 3e7b65c3e5920ed78bd87b68de1075003d86e7a0)
 2020-12-04 10:43:32 +09:00
Ahmed Hussein
5f5d7ecf59 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040806c4c9687288de89e42cfacf0365e)
(cherry picked from commit a7dbd3b6f978d28ef7e046add2e3270ae3d9d9f6)
(cherry picked from commit f547cd43d140dc680fab1782cda071f4a41dddc5)
 2020-12-03 15:15:47 -08:00
Jim Brennan
c43eca6cdc HADOOP-17392. Remote exception messages should not include the exception class (#2486). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 5bfb97bc7d5ff5e083aff90e5a380a850d9e4530)
 2020-12-03 19:47:46 +00:00
Andrea Scarpino
c8178a9caa
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:18:12 +09:00
Jim Brennan
521d5b381a HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 22:25:12 +00:00
Ahmed Hussein
a63a30ff99 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7f17f095548b4a4485e7b7218af74be)
 2020-12-02 20:54:32 +00:00
Eric Payne
73cf0bdba1 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit 1184284bafacbc2cbb31050e829886e2e3d8aeea)
 2020-12-02 17:27:32 +00:00
Ahmed Hussein
2331f86dfb HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82fb0bc6f96301c7ca74a32018b60c49e)
 2020-12-01 21:43:37 +00:00
kevinzhao1661
a60aacc0c8
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398949acfd0b6406abdaeb17456a25c37)
 2020-11-30 11:19:03 +09:00
Peter Bacsko
7d60217754 MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan. 2020-11-25 11:46:00 +01:00
Akira Ajisaka
4638ed94db
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 
(cherry picked from commit c4ba0ab7dfcfe76f928d06fcb29ca0cc4d778737)
 2020-11-24 15:23:34 +09:00
Eric Payne
c7845dc574 HADOOP-17346. Fair call queue is defeated by abusive service principals. Contributed by Ahmed Hussein (ahussein). 2020-11-23 21:34:52 +00:00
Peter Bacsko
6062978768 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:16:48 +01:00
Jim Brennan
da87653b66 HADOOP-17367. Add InetAddress api to ProxyUsers.authorize (#2449). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 6b3d65b46ed6535c7754501287186c7128b32ed4)
 2020-11-19 21:59:57 +00:00
Ahmed Hussein
56e327b7ca HADOOP-17360. Log the remote address for authentication success (#2441) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit 1ea3f74246294c280a2ccb0ff3e90b5721c5f0e2)
 2020-11-16 22:08:14 +00:00
Ahmed Hussein
de9f778750 YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 
(cherry picked from commit 0b2510ee1fb2c92d3b7d2b2183e532b7637d93b9)
 2020-11-16 21:30:50 +00:00
Peter Bacsko
a86c91bc06 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-11-16 11:54:19 +01:00
Jim Brennan
118d8f5aa1 HADOOP-17362. reduce RPC calls doing ls on HAR file (#2444). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 4bdba35b9ffd57e249d2467d618ed63f1950b438)
 2020-11-13 21:52:10 +00:00