2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. ( #2677 )
2021-02-09 10:27:51 +05:30
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. ( #2625 )
2021-02-08 11:01:21 -08:00
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit ( #2682 )
2021-02-06 08:26:12 -08:00
c4918fb298
[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan
2021-02-05 21:20:30 +00:00
7ef2875c89
HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang
2021-02-05 11:01:49 +08:00
b99dba38b9
HDFS-15819. Fix import style #2681
2021-02-05 10:11:27 +08:00
15a1f7adfc
HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang
2021-02-04 09:19:51 +08:00
182623d2bc
HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.
2021-02-03 08:55:50 -06:00
66ecee333e
HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang
2021-02-03 20:00:45 +08:00
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type ( #2377 )
...
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied
Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-02-02 22:44:34 -08:00
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka ( #2657 )
2021-02-02 09:02:09 +00:00
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-02 13:50:54 +08:00
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-01 23:28:29 +08:00
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. ( #2652 )
2021-02-01 01:56:59 -08:00
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer ( #2645 )
2021-02-01 13:42:44 +09:00
f565476939
[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.
2021-01-28 18:17:15 +00:00
6184ab07f6
[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.
2021-01-28 16:47:52 +00:00
f8769e0f4b
HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.
2021-01-28 10:10:39 +08:00
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement ( #2626 )
2021-01-21 10:07:46 +09:00
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-16 14:06:56 +08:00
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount ( #2583 ). Contributed by Leon Gao.
2021-01-15 16:28:11 -08:00
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. ( #2475 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-14 16:19:35 +09:00
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently ( #2598 )
...
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-13 11:23:07 +09:00
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. ( #2507 )
2021-01-11 11:59:32 -08:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. ( #2601 ). Contributed by Ahmed Hussein and Daryn Sharp.
2021-01-08 13:10:09 -06:00
87bd4d2aca
HDFS-15754. Add DataNode packet metrics ( #2578 )
...
Contributed by Fengnan Li.
2021-01-07 23:46:23 -08:00
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. ( #2453 )
2021-01-04 09:43:58 -08:00
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. ( #2501 )
2021-01-04 14:00:57 +09:00
1cd96e8dd8
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.
2021-01-01 11:37:52 +05:30
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable ( #2572 )
2020-12-28 14:18:18 +08:00
16a20503ca
HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.
2020-12-22 17:27:31 +05:30
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType ( #2566 )
2020-12-22 09:25:11 +08:00
ba496f61f8
HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.
2020-12-22 02:16:00 +05:30
2aea43bf4f
HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.
2020-12-20 23:06:19 +05:30
3ecd3628d4
HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.
2020-12-20 15:18:25 +05:30
70789fc6b9
HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.
2020-12-20 15:12:32 +05:30
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. ( #2511 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-17 11:33:29 -06:00
0da00fc743
HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-12-14 20:42:23 -08:00
3363cc380b
HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.( #2534 )
2020-12-15 10:06:11 +05:30
9ed737001c
HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell
2020-12-11 20:53:10 +00:00
be35fa186c
HDFS-15717. Improve fsck logging. ( #2529 ) Contributed by Kihwal Lee and Ahmed Hussein
2020-12-11 11:02:42 -06:00
9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config ( #2532 )
2020-12-10 13:47:00 -08:00
01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy ( #2528 )
2020-12-08 13:25:24 -06:00
40f7543a6d
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. ( #2518 )
2020-12-07 15:49:45 -08:00
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. ( #2516 )
2020-12-07 14:22:06 -06:00
ad40715690
HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.
2020-12-07 15:00:19 +05:30
e2c1268ebd
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-04 09:20:09 +08:00
42a29199c0
HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.
2020-12-02 21:32:02 -08:00
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration ( #2483 ). Contributed by Leon Gao.
2020-12-02 15:53:09 -08:00
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op ( #2508 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 15:38:20 -06:00
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode ( #2489 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 13:59:00 -06:00
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. ( #2487 ) Contributed by Kuhu Shukla and Ahmed Hussein
2020-12-01 15:02:53 -06:00
68442b48c8
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 ( #2495 )
2020-11-28 12:19:57 +08:00
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check ( #2472 )
2020-11-25 11:01:04 -08:00
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2020-11-23 11:26:52 +08:00
d73029463c
HDFS-15690. Add lz4-java as test dependency ( #2481 )
2020-11-21 17:49:56 -08:00
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false ( #2443 ). Contributed by Ahmed Hussein.
2020-11-20 18:03:45 +05:30
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. ( #2467 )
2020-11-18 16:11:09 +09:00
f56cd88d7d
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
2020-11-13 11:02:26 +09:00
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… ( #2255 )
...
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
2020-11-12 14:57:26 -08:00
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. ( #2138 )
2020-11-10 16:34:17 -08:00
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
2020-11-10 09:11:31 -08:00
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete ( #2437 )
2020-11-10 13:01:10 +08:00
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount ( #2288 ). Contributed by Leon Gao.
2020-11-09 15:06:16 -08:00
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
2020-11-04 13:53:46 +08:00
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:28:44 +05:30
d07dc7afb4
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
2020-11-03 12:02:40 -08:00
324879127a
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently ( #2419 )
2020-10-28 16:24:34 -07:00
d0c786db4d
HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. ( #2406 )
2020-10-27 18:56:40 -07:00
98097b8f19
HDFS-15457. TestFsDatasetImpl fails intermittently ( #2407 )
2020-10-27 18:52:56 -07:00
8a6d5b9151
HDFS-15652. Make block size from NNThroughputBenchmark configurable ( #2416 )
2020-10-28 09:13:25 +08:00
ae74407ac4
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky ( #2410 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-10-27 20:45:00 +09:00
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails ( #2309 )
2020-10-27 19:18:08 +09:00
afaab3d332
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails ( #2404 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-10-27 16:56:30 +09:00
12c908c827
HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.
2020-10-26 22:12:27 +08:00
74634eb002
HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.
2020-10-23 12:05:53 -07:00
e7aa4da0a2
HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.
2020-10-23 09:47:10 -07:00
da1b6e3cc2
HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.
2020-10-22 16:59:49 -05:00
88a9f42f32
HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.
2020-10-21 00:59:35 -05:00
2e8cafac3b
HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.
2020-10-19 10:48:47 +05:30
173310e2f5
HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang.
2020-10-19 10:34:17 +05:30
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. ( #2342 ). Contributed by Ayush Saxena.
2020-10-17 12:01:18 +05:30
740a2c4635
HDFS-15627. Log delete audits before collecting blocks.
2020-10-16 11:13:41 -05:00
a308a1ec22
HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled ( #2370 )
2020-10-13 10:59:42 -07:00
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
...
Contributed by Gabor Bota.
2020-10-13 16:17:44 +01:00
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
Change-Id: I6cec41c20467920cf21f169653553535414b2680
2020-10-13 13:31:00 +01:00
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing ( #2380 )"
...
(reverting to change the JIRA number)
This reverts commit 3e1b1dfd77
2020-10-13 13:30:38 +01:00
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
2020-10-13 13:27:55 +01:00
b3786d6c3c
HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.
2020-10-12 17:26:49 -07:00
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup ( #2378 ). Contributed by Uma Maheswara Rao G.
2020-10-12 14:22:44 -07:00
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently ( #2299 )
2020-10-10 22:08:36 +05:30
735e85a1d2
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 ( #2365 )
2020-10-08 12:51:04 +05:30
df4006eb81
HDFS-15253 Default checkpoint transfer speed, 50mb per second ( #2366 )
2020-10-07 22:09:04 +05:30
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. ( #2355 )
2020-10-07 10:34:32 +05:30
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently ( #2356 ). Contributed by Hui Fei.
2020-10-05 22:11:24 +05:30
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir ( #2352 )
2020-10-05 05:02:00 -07:00
984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc ( #2345 )
2020-09-28 09:54:01 -07:00
d89af79507
HDFS-15577. Refactor TestTracing. ( #2302 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-09-28 18:27:41 +09:00
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng
2020-09-27 19:27:13 +05:30
00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip ( #2332 )
2020-09-25 09:47:54 -07:00
486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 ( #2330 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2020-09-24 21:15:55 +09:00
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS ( #2189 )
...
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
2020-09-24 14:27:04 +05:30
368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled ( #2326 )
2020-09-24 14:00:41 +05:30
ae089f2db7
HDFS-15557. Log the reason why a storage log file can't be deleted ( #2274 )
2020-09-22 13:23:20 -07:00
474fa80bfb
HADOOP-17277. Correct spelling errors for separator ( #2322 )
...
Contributed by Hui Fei.
2020-09-22 17:10:33 +01:00
2377df38ad
HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.
2020-09-18 17:28:11 +05:30
7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. ( #2312 ). Contributed by Uma Maheswara Rao G.
2020-09-18 15:18:10 +05:30
20a0e6278d
HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell
2020-09-17 14:11:42 +01:00
425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. ( #2296 )
2020-09-17 14:50:08 +05:30
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled ( #2305 ). Contributed by Uma Maheswara Rao G.
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-16 22:43:00 -07:00
5c5b2ed7c7
HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui.
2020-09-16 01:36:46 +09:00
f4ed9f3f91
HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.
2020-09-15 10:10:21 +05:30
924959088e
HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun.
2020-09-15 12:05:04 +08:00
6d3312de47
HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty ( #2258 )
2020-09-14 13:31:34 -07:00
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. ( #2298 ). Contributed by Uma Maheswara Rao G.
2020-09-12 17:06:39 -07:00
f59f7f2175
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell
2020-09-12 01:41:38 -07:00
2a6fdcedeb
HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.
2020-09-11 18:21:25 +05:30
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache ( #2241 )
...
Contributed by fanrui.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-09-10 22:30:52 -07:00
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree ( #2185 ). Contributed by Zhenzhao Wang.
...
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
2020-09-10 21:20:32 -07:00
43c52d6495
HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir ( #2295 )
...
Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc
2020-09-10 15:55:23 +05:30
1d6d0d8207
HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun.
...
Co-authored-by: Chen Zhang <chzhang1987@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2020-09-09 20:09:13 +08:00
ac7d4623ae
HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured ( #2275 ) Contributed by Uma Maheswara Rao G.
2020-09-07 11:36:13 -07:00
b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well ( #2234 ). Contributed by Uma Maheswara Rao G.
2020-09-03 11:06:20 -07:00
60de592a88
HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree.
2020-08-31 14:49:12 +08:00
2d03209a06
HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion ( #2251 )
2020-08-31 11:29:48 +05:30
c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing ( #2252 )
...
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
2020-08-28 15:39:09 +01:00
06793da100
HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory ( #2230 )
2020-08-27 11:36:30 -07:00
41182a9b6d
HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. ( #2233 )
2020-08-27 14:54:52 +05:30
2ffe00fc46
HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-08-26 23:05:32 -07:00
6a49bf9bff
HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.
2020-08-25 15:18:36 +01:00
a7830423c5
HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.
2020-08-24 21:16:00 +08:00
83fd15b412
HDFS-14504. Rename with Snapshots does not honor quota limit.
2020-08-23 16:10:47 +05:30
f734455e5d
HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.
2020-08-20 15:25:10 -07:00
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. ( #2229 ). Contributed by Uma Maheswara Rao G.
2020-08-19 09:30:41 -07:00
b65e43fe38
HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS ( #2220 )
2020-08-18 19:28:19 +09:00
092bfe7c8e
HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. ( #2172 )
...
* HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories.
* Addressed review comments.
* Rebased to latest trunk and added snapshotTrashRoot config check.
* Addressed review comments.
* Removede empty line added in SnapshotManager.Java.
* Addressed whitespace issues.
2020-08-17 12:56:13 +05:30
e3d1966f58
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
2020-08-15 15:07:58 +05:30
b93dd7c281
HDFS-15519. Check inaccessible INodes in FsImageValidation. ( #2224 )
2020-08-14 10:10:01 -07:00
15a76e8d65
HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. ( #2219 )
2020-08-14 20:46:48 +05:30
cb50e3fcf7
HDFS-15496. Add UI for deleted snapshots ( #2212 )
2020-08-13 22:36:15 +05:30
e592ec5f8b
HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. ( #2217 )
...
* HDFS-15518. Fixed String operationName = ListSnapshot.
* HDFS-15518. Changed ListSnapshot to operationName.
Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>
2020-08-13 00:09:38 +05:30
10716040a8
HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang
2020-08-12 09:02:47 +01:00
11cec9ab94
HDFS-15523. Fix findbugs warnings from HDFS-15520. ( #2218 )
2020-08-11 21:41:28 -07:00
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory ( #2176 )
2020-08-11 08:52:16 -07:00
8955a6ceb7
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false ( #2205 )
...
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false
* Used LambdaTestUtils#intercept in test
2020-08-11 00:01:58 -07:00
592127bdc2
HDFS-15520 Use visitor pattern to visit namespace tree ( #2203 )
2020-08-11 12:30:30 +05:30
7938ebfb9d
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.
2020-08-11 11:14:58 +09:00
c2a17659d1
HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.
2020-08-08 14:33:53 +05:30
975b6024dd
HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.
2020-08-07 22:19:17 +05:30
e072d33327
HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable ( #2175 )
2020-08-04 14:10:29 +05:30
82f3ffcd64
HDFS-15503. File and directory permissions are not able to be modified from WebUI.
2020-08-03 23:21:01 +05:30
c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors ( #2098 )
2020-08-03 10:46:51 +09:00
d8a2df25ad
HDFS-15498. Show snapshots deletion status in snapList cmd. ( #2181 )
2020-08-02 21:59:56 +05:30
528a799a78
HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree.
2020-08-02 12:02:37 +05:30
05b3337a46
HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots ( #2165 )
2020-07-30 23:06:51 +05:30
68287371cc
HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. ( #2166 )
2020-07-29 21:33:25 +05:30
026dce5334
HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. ( #2135 )
2020-07-27 09:55:11 -07:00
2d12496643
HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr ( #2163 )
2020-07-22 16:16:27 -07:00
ac9a07b51a
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. ( #2160 ). Contributed by Uma Maheswara Rao G.
2020-07-21 23:29:10 -07:00
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 )
2020-07-22 13:40:20 +09:00
8b7695bb26
HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.
2020-07-21 09:00:07 -07:00
d57462f2da
HDFS-15479. Ordered snapshot deletion: make it a configurable feature ( #2156 )
2020-07-21 11:36:24 +05:30
d9441f95c3
HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee.
2020-07-21 11:13:05 +05:30
3833c616e0
HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.
2020-07-20 12:49:58 -07:00
6cbd8854ee
HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi.
2020-07-20 21:49:05 +05:30
2cec50cf16
HDFS-15463. Add a tool to validate FsImage. ( #2140 )
2020-07-20 11:44:30 +05:30
6bcb24d269
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-18 14:33:43 +05:30
1f71c4ae71
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 11:39:06 -05:00
98fcffe93f
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 09:53:18 -05:00
85d4718ed7
HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee.
2020-07-15 10:32:25 +05:30
bdce75d737
HDFS-15371. Nonstandard characters exist in NameNode.java ( #2032 )
...
Contributed by zhaoyim
2020-07-14 14:42:12 +01:00
b97fea65e7
HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.
2020-07-13 14:12:48 +08:00
3e70006639
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links ( #2132 ). Contributed by Uma Maheswara Rao G.
2020-07-11 23:50:04 -07:00
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. ( #2085 )
2020-07-09 11:33:37 -07:00
f77bbc2123
HDFS-15312. Apply umask when creating directory by WebHDFS ( #2096 )
2020-07-06 19:01:46 -07:00
dc0626b5f2
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.
2020-07-06 18:50:03 -07:00
834372f404
HDFS-15451. Do not discard non-initial block report for provided storage. ( #2119 ). Contributed by Shanyu Zhao.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-07-06 23:43:34 +08:00
55a2ae80dc
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.
2020-07-04 13:45:49 -07:00
f86f15cf20
HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.
2020-07-04 13:00:43 +05:30
1f2a80b5e5
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
2020-07-04 00:12:10 -07:00
9ac498e300
HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu.
2020-07-01 12:38:07 +08:00
2a67e2b1a0
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-30 07:10:17 -07:00
c71ce7ac33
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
2020-06-28 16:02:47 +09:00
8db38c98a6
HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.
2020-06-27 19:13:01 +05:30
bed0a3a374
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable ( #2100 )
...
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable
* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests
* Address Uma's comments on PR#2100
* Sort lists in test to match without concern to order
* Address comments, fix checkstyle and fix failing tests
* Fix checkstyle
2020-06-26 13:19:16 -07:00
d5e1bb6155
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
2020-06-26 01:29:38 -07:00
7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
2020-06-23 01:42:25 -07:00
17ffcab5f6
HDFS-14546. Document block placement policies. Contributed by Amithsha.
2020-06-21 17:27:28 +05:30
b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.
2020-06-20 00:32:02 -07:00
d50e93ce7b
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-18 06:45:15 -07:00
edf716a5c3
Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell."
...
This reverts commit 730a39d138
2020-06-18 06:45:15 -07:00
123777823e
HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina
2020-06-18 12:26:22 +01:00
c8ed33cd2a
HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.
2020-06-17 13:55:40 +05:30
730a39d138
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-16 15:59:13 -07:00
f41a144077
HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.
2020-06-15 09:15:53 +09:00
719b53a79d
HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.
2020-06-13 09:35:05 -07:00
785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.
2020-06-12 14:32:19 -07:00
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:16:33 +05:30
b735a77717
HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.
2020-06-10 12:06:16 +05:30
8525874561
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
...
*Added missed test file.
2020-06-09 18:53:33 +05:30
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.
2020-06-06 08:11:57 -07:00
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena
2020-06-05 22:19:38 -07:00
76fa0222f0
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
2020-06-05 10:58:21 -07:00
2326123705
HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena.
2020-06-05 19:15:05 +05:30
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories ( #2052 )
...
Contributed by Toshihiro Suzuki.
2020-06-05 11:11:49 +01:00
e8cb2ae409
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang.
2020-06-03 12:47:15 +05:30
f6453244ab
HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan.
2020-06-03 12:16:36 +05:30
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.
2020-06-02 11:09:26 -07:00
ae13a5ccbe
HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang.
2020-05-31 17:10:59 +05:30
d9838f2d42
HADOOP-17055. Remove residual code of Ozone ( #2039 )
2020-05-29 16:49:18 +09:00
a838d871a7
HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He.
2020-05-28 20:52:58 +05:30
9b38be43c6
HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
2020-05-28 13:00:04 +05:30
2148a8fe64
HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina.
2020-05-27 11:06:13 -07:00
6c9f75cf16
HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena.
2020-05-26 16:26:57 +05:30
f43a152b97
HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun.
2020-05-24 18:03:41 +05:30
f4901d0778
HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun.
2020-05-24 17:59:16 +05:30
6e04b00df1
HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun.
2020-05-23 09:58:19 -07:00
e0ae232f66
HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena.
2020-05-23 18:38:32 +05:30
f99fcb26ab
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.
2020-05-23 18:29:31 +05:30
4d22d1c58f
HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina.
2020-05-23 17:28:38 +09:00
4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.
2020-05-21 21:34:58 -07:00
be374faf42
HDFS-13639. SlotReleaser is not fast enough ( #1885 )
2020-05-21 13:21:17 -07:00
ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.
2020-05-18 22:27:12 -07:00
7bb902bc0d
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang
2020-05-18 10:58:52 -07:00
c84e6beada
HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena.
2020-05-18 22:24:34 +05:30
2abcf7762a
HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )
2020-05-18 09:23:09 -07:00
4525292d41
Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )"
...
This reverts commit 86e6aa8eec
2020-05-18 09:22:05 -07:00
50caba1a92
HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun.
2020-05-18 08:40:38 -07:00
a3f44dacc1
HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-05-18 07:09:43 -07:00
86e6aa8eec
HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )
...
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.
2020-05-18 07:04:04 -07:00
6e416a83d1
HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena.
2020-05-17 03:06:27 +05:30
178336f8a8
HDFS-15356. Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys. Contributed by Xiaoqiao He.
2020-05-16 16:35:06 +05:30
ac4a2e11d9
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.
2020-05-14 17:29:35 -07:00
1958cb7c2b
YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng
2020-05-14 15:09:34 +05:30
0918433b4d
YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng.
2020-05-14 00:36:20 +05:30
3cacf1ce56
HDFS-15344. DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. ( #2004 )
2020-05-13 11:47:19 -07:00
743c2e9071
HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina
2020-05-13 15:01:07 +05:30
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) ( #2007 )
...
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
29dddb8a14
HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.
2020-05-12 15:07:51 +01:00
928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. ( #2008 )
2020-05-12 09:04:03 -05:00
8dad38c0be
HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.
2020-05-12 18:34:26 +05:30
0fe49036e5
HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu.
2020-05-12 13:11:31 +05:30
4c53fb9ce1
HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.
2020-05-11 16:48:34 +05:30
cb64e993c2
HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi.
2020-05-10 00:11:46 +05:30
d59de27c01
HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.
2020-05-08 00:53:49 +05:30
130f89e068
HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina.
2020-05-06 13:18:32 -07:00
7fddf4855e
HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.
2020-05-06 16:55:04 +05:30
adecdb8b53
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. ( #1998 )
...
Reviewed-by: Arpit Agarwal <arp@apache.org>
2020-05-05 16:04:54 -07:00
bshashikant
LeonGao
Jim Brennan
fermi
maobaolong
Hui Fei
Kihwal Lee
YaYun-Wang
crossfire
He Xiaoqiao
belugabehr
Narges Shadab
Siyao Meng
sunlisheng
Akira Ajisaka
Masatake Iwasaki
touchida
Ahmed Hussein
lfengnan
Wei-Chiu Chuang
Ayush Saxena
maobaolong
hemanthboyina
langlaile1221
liuyanpunk
S O'Donnell
Neil
Liang-Chi Hsieh
Takanobu Asanuma
Issac Buenrostro
jianghuazhu
Yizhong Zhang
Konstantin V Shvachko
Inigo Goiri
Gabor Bota
Mukund Thakur
Steve Loughran
Uma Maheswara Rao G
Karthik Palanisamy
Fei Hui
Vinayakumar B
Ye Ni
Aryan Gupta
huangtianhua
Mingliang Liu
1996fanrui
zz
Vivek Ratnavel Subramanian
Tsz-Wo Nicholas Sze
Stephen O'Donnell
Chen Liang
Ahmed Hussein
zhaoyim
Xiaoyu Yao
Shanyu Zhao
Virajith Jalaparti
Toshihiro Suzuki
Wanqiang Ji
leosunli
Abhishek Das
pustota2009
Surendra Singh Lilhore
Elixir Kook