Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,472 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8395 Commits

Author SHA1 Message Date
He Xiaoqiao 03151aa78b
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:44:15 +08:00
Akira Ajisaka f1dff467e1
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:09:51 +09:00
Masatake Iwasaki 371955fb50
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c)
 2021-01-14 16:32:04 +09:00
touchida e2681dba22
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba)
 2021-01-13 11:32:41 +09:00
He Xiaoqiao a17793534c
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:46 +08:00
Wei-Chiu Chuang 8af0fcf628 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
(cherry picked from commit 94c126cc9e)
 2021-01-04 20:56:32 -08:00
Ahmed Hussein 0a3c6b218a HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 18:07:36 +00:00
Ayush Saxena 079d3ea6ac HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-12-15 10:53:26 +05:30
Ayush Saxena 40285be1d5 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-12-15 10:34:22 +05:30
Ayush Saxena ed78df0f75 HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aa)
 2020-12-14 20:41:21 -08:00
S O'Donnell 52adda89c4 HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 2020-12-14 11:37:08 +00:00
Jim Brennan eec10a087e HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 2020-12-11 18:10:24 +00:00
Neil 9123f9245c HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
(cherry picked from commit bc5458bbd4)
 2020-12-10 13:50:42 -08:00
Jim Brennan 76316c4bc0 HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 21:30:03 +00:00
Ahmed Hussein bb23d4f14b HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a2172)
 2020-12-08 19:41:54 +00:00
crossfire 31a6b45117 HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d)
(cherry picked from commit edd9b659ca)
 2020-12-07 15:50:47 -08:00
Jim Brennan d7c955b646 HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 
(cherry picked from commit 6a5864ee4a)
 2020-12-07 21:22:15 +00:00
Chao Sun 3b2a131885
HDFS-15708. TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2 (#2517) 2020-12-04 10:42:46 -08:00
Hui Fei 6e215953e6
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-04 17:11:21 +08:00
Ahmed Hussein f547cd43d1 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040)
(cherry picked from commit a7dbd3b6f9)
 2020-12-03 15:15:31 -08:00
Ahmed Hussein 91310102d7 HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 60201cbf69)
 2020-12-02 21:52:52 +00:00
Ahmed Hussein 6cc0824072 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7)
 2020-12-02 20:31:04 +00:00
Ahmed Hussein 625f85f40f HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82f)
 2020-12-01 21:31:18 +00:00
Takanobu Asanuma 3ee0a174dd HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d)
 2020-11-13 11:08:57 +09:00
Yizhong Zhang 52c2ce53c2 HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794)
(cherry picked from commit 12bfd9194a)
 2020-11-10 09:14:25 -08:00
Giovanni Matteo Fumarola 4af5f55623
HDFS-14395. Remove WARN Logging From Interrupts. Contributed by David Mollitor. 2020-11-10 14:44:34 +08:00
Ayush Saxena ce883fe1bb HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:53:28 +05:30
Konstantin V Shvachko 23dcd8edb6 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 12:04:38 -08:00
Ahmed Hussein dedf0d42bc
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4)
 2020-10-27 20:52:23 +09:00
Ahmed Hussein 244fd5eed2
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails (#2404) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit afaab3d332)
 2020-10-27 17:09:20 +09:00
Hui Fei b32926f110 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827)
 2020-10-26 22:34:06 +08:00
Wei-Chiu Chuang 87666e2438 HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002)
(cherry picked from commit c8eb86ee3d)
 2020-10-23 12:06:36 -07:00
Inigo Goiri 7119af2c6d HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a2)
 2020-10-23 09:49:46 -07:00
Kihwal Lee f363c3b315 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc2)
 2020-10-22 19:48:42 -05:00
Kihwal Lee f3b2d85690 HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 
(cherry picked from commit cf932a7e2d)
 2020-10-22 14:59:09 -05:00
Takanobu Asanuma 0d6958962c HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:13:20 +09:00
Kihwal Lee 7e4572e56a HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c4635)
 2020-10-16 11:17:34 -05:00
He Xiaoqiao 3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Kihwal Lee 392b26c23b HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d)
 2020-10-14 18:19:16 -05:00
Akira Ajisaka 69b60ead5b
Addendum to HADOOP-16990. Update Mockserver. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 800b1ed1c2)
 2020-10-13 18:09:14 +09:00
Konstantin V Shvachko 23b6ab67f7 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:42:58 -07:00
Doroszlai, Attila 61563885f8 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:33 -07:00
Stephen O'Donnell 71d006130e HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-09-28 12:45:28 -07:00
Kihwal Lee 25a51f2f39 HDFS-15581. Access Controlled HttpFS Server. Contributed by Richard Ross. 
(cherry picked from commit dfc2682213)
 2020-09-22 11:12:43 -05:00
S O'Donnell 5f34e3214e HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 2020-09-22 10:34:05 +01:00
hemanthboyina aa582ccc2a HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 2020-09-17 10:15:18 +05:30
Ayush Saxena 544d6da3ab HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena. (#2261) 2020-09-06 07:56:16 +05:30
Kitti Nanasi 9b8eaa0572 HDFS-14188. Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameter. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 14282e311b)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi 3ba40d0ccf HDFS-14125. Use parameterized log format in ECTopologyVerifier. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5f15a60e32)
 2020-08-30 10:21:53 +05:30
Kitti Nanasi 0dbcb81a41 HDFS-14061. Check if the cluster topology supports the EC policy before setting, enabling or adding it. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 951cdd7e4c)
 2020-08-30 10:21:53 +05:30