cd2501e54b
HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun.
2021-03-28 16:13:59 +05:30
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks ( #2748 )
...
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-03-27 17:57:09 +09:00
ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.
2021-03-25 12:24:14 +08:00
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode ( #2585 )
2021-03-24 16:56:09 +08:00
4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result ( #2794 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 22:47:45 -07:00
569e407f64
HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.
2021-03-24 02:10:02 +05:30
9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof ( #2805 )
2021-03-23 08:51:35 -07:00
d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal ( #2790 ). Contributed by Narges Shadab.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 17:06:58 +05:30
343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete ( #2788 )
2021-03-22 20:29:17 +09:00
03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-22 18:37:00 +08:00
261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() ( #2785 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-19 20:37:44 -07:00
f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream ( #2736 ). Contributed by Narges Shadab.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-19 16:36:48 +08:00
9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles ( #2772 )
2021-03-17 15:12:36 +09:00
9ba60c33d8
HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.
2021-03-17 14:14:26 +09:00
25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically ( #2726 )
...
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-17 12:00:05 +09:00
0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails ( #2779 )
2021-03-17 09:11:30 +08:00
7025f39944
HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.
2021-03-15 11:34:13 +08:00
970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun.
2021-03-14 17:48:18 +08:00
6921ec8b40
HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options ( #2751 ). Contributed by Xie Lei.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-13 14:22:11 +08:00
7cf5969f0b
HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. ( #2764 )
2021-03-12 11:27:53 -08:00
ecd3335187
HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.
2021-03-12 15:42:15 +09:00
6a55baeee4
HDFS-15875. Check whether file is being truncated before truncate ( #2746 )
2021-03-10 14:11:29 +08:00
6a922f9896
HDFS-15873. Add namenode address in logs for block report ( #2743 )
2021-03-08 11:50:51 -05:00
9cb51bf106
HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. ( #2668 ) Contributed by tomscut.
2021-03-06 13:40:44 -08:00
cdba06e380
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min ( #2739 )
...
Co-authored-by: tomscut <litao@bigo.sg>
2021-03-03 22:34:02 +09:00
1f1a1ef52d
HDFS-15856: Make write pipeline retry times configurable. ( #2721 ). Contributed by Qi Zhu
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-02 13:16:11 +08:00
9501c698f4
HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.
2021-03-01 17:17:23 -08:00
32353eb38a
HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker ( #2718 )
...
Authored-by: tomscut <litao@bigo.sg>
2021-03-01 07:52:59 -08:00
c3b3b36dee
HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell
2021-03-01 11:36:41 +00:00
9c583f36d5
HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee
2021-02-24 10:29:36 +00:00
940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. ( #2704 )
2021-02-23 22:29:48 -08:00
647d23b611
HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.
2021-02-23 19:08:38 +05:30
a57282540f
HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. ( #2714 )
2021-02-23 15:12:06 +09:00
1e32ab7d2f
HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
2021-02-22 18:13:36 +08:00
f8059d0bdf
Revert "first commit ( #2559 )"
...
This reverts commit 1b94569d89
2021-02-22 18:13:06 +08:00
1b94569d89
first commit ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
2021-02-22 02:10:39 -08:00
98ffd65d8b
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. ( #2686 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
2021-02-21 23:22:32 -08:00
ff84a57483
HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.
2021-02-20 19:48:00 +05:30
25af8901a9
HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.
2021-02-19 14:44:58 +09:00
2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable ( #2694 )
2021-02-19 09:07:22 +08:00
e391844e8e
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.
2021-02-18 01:39:04 +05:30
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level ( #2696 )
2021-02-17 01:59:24 -05:00
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes ( #2690 ). Contributed by Zehao Chen.
2021-02-15 11:14:32 -06:00
c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics ( #2676 )
2021-02-14 00:12:24 -05:00
798df6d699
HADOOP-13327 Output Stream Specification. ( #2587 )
...
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.
The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.
Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao ( #2679 )
2021-02-09 10:49:28 +00:00
2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. ( #2677 )
2021-02-09 10:27:51 +05:30
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. ( #2625 )
2021-02-08 11:01:21 -08:00
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit ( #2682 )
2021-02-06 08:26:12 -08:00
c4918fb298
[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan
2021-02-05 21:20:30 +00:00
7ef2875c89
HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang
2021-02-05 11:01:49 +08:00
b99dba38b9
HDFS-15819. Fix import style #2681
2021-02-05 10:11:27 +08:00
15a1f7adfc
HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang
2021-02-04 09:19:51 +08:00
182623d2bc
HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.
2021-02-03 08:55:50 -06:00
66ecee333e
HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang
2021-02-03 20:00:45 +08:00
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type ( #2377 )
...
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied
Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-02-02 22:44:34 -08:00
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka ( #2657 )
2021-02-02 09:02:09 +00:00
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-02 13:50:54 +08:00
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-01 23:28:29 +08:00
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. ( #2652 )
2021-02-01 01:56:59 -08:00
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer ( #2645 )
2021-02-01 13:42:44 +09:00
f565476939
[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.
2021-01-28 18:17:15 +00:00
6184ab07f6
[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.
2021-01-28 16:47:52 +00:00
f8769e0f4b
HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.
2021-01-28 10:10:39 +08:00
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement ( #2626 )
2021-01-21 10:07:46 +09:00
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-16 14:06:56 +08:00
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount ( #2583 ). Contributed by Leon Gao.
2021-01-15 16:28:11 -08:00
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. ( #2475 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-14 16:19:35 +09:00
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently ( #2598 )
...
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-13 11:23:07 +09:00
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. ( #2507 )
2021-01-11 11:59:32 -08:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. ( #2601 ). Contributed by Ahmed Hussein and Daryn Sharp.
2021-01-08 13:10:09 -06:00
87bd4d2aca
HDFS-15754. Add DataNode packet metrics ( #2578 )
...
Contributed by Fengnan Li.
2021-01-07 23:46:23 -08:00
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. ( #2453 )
2021-01-04 09:43:58 -08:00
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. ( #2501 )
2021-01-04 14:00:57 +09:00
1cd96e8dd8
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.
2021-01-01 11:37:52 +05:30
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable ( #2572 )
2020-12-28 14:18:18 +08:00
16a20503ca
HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.
2020-12-22 17:27:31 +05:30
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType ( #2566 )
2020-12-22 09:25:11 +08:00
ba496f61f8
HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.
2020-12-22 02:16:00 +05:30
2aea43bf4f
HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.
2020-12-20 23:06:19 +05:30
3ecd3628d4
HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.
2020-12-20 15:18:25 +05:30
70789fc6b9
HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.
2020-12-20 15:12:32 +05:30
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. ( #2511 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-17 11:33:29 -06:00
0da00fc743
HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-12-14 20:42:23 -08:00
3363cc380b
HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.( #2534 )
2020-12-15 10:06:11 +05:30
9ed737001c
HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell
2020-12-11 20:53:10 +00:00
be35fa186c
HDFS-15717. Improve fsck logging. ( #2529 ) Contributed by Kihwal Lee and Ahmed Hussein
2020-12-11 11:02:42 -06:00
9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config ( #2532 )
2020-12-10 13:47:00 -08:00
01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy ( #2528 )
2020-12-08 13:25:24 -06:00
40f7543a6d
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. ( #2518 )
2020-12-07 15:49:45 -08:00
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. ( #2516 )
2020-12-07 14:22:06 -06:00
ad40715690
HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.
2020-12-07 15:00:19 +05:30
e2c1268ebd
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-04 09:20:09 +08:00
42a29199c0
HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.
2020-12-02 21:32:02 -08:00
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration ( #2483 ). Contributed by Leon Gao.
2020-12-02 15:53:09 -08:00
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op ( #2508 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 15:38:20 -06:00
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode ( #2489 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 13:59:00 -06:00
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. ( #2487 ) Contributed by Kuhu Shukla and Ahmed Hussein
2020-12-01 15:02:53 -06:00
68442b48c8
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 ( #2495 )
2020-11-28 12:19:57 +08:00
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check ( #2472 )
2020-11-25 11:01:04 -08:00
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2020-11-23 11:26:52 +08:00
d73029463c
HDFS-15690. Add lz4-java as test dependency ( #2481 )
2020-11-21 17:49:56 -08:00
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false ( #2443 ). Contributed by Ahmed Hussein.
2020-11-20 18:03:45 +05:30
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. ( #2467 )
2020-11-18 16:11:09 +09:00
f56cd88d7d
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
2020-11-13 11:02:26 +09:00
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… ( #2255 )
...
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
2020-11-12 14:57:26 -08:00
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. ( #2138 )
2020-11-10 16:34:17 -08:00
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
2020-11-10 09:11:31 -08:00
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete ( #2437 )
2020-11-10 13:01:10 +08:00
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount ( #2288 ). Contributed by Leon Gao.
2020-11-09 15:06:16 -08:00
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
2020-11-04 13:53:46 +08:00
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:28:44 +05:30
d07dc7afb4
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
2020-11-03 12:02:40 -08:00
324879127a
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently ( #2419 )
2020-10-28 16:24:34 -07:00
d0c786db4d
HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. ( #2406 )
2020-10-27 18:56:40 -07:00
98097b8f19
HDFS-15457. TestFsDatasetImpl fails intermittently ( #2407 )
2020-10-27 18:52:56 -07:00
8a6d5b9151
HDFS-15652. Make block size from NNThroughputBenchmark configurable ( #2416 )
2020-10-28 09:13:25 +08:00
ae74407ac4
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky ( #2410 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-10-27 20:45:00 +09:00
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails ( #2309 )
2020-10-27 19:18:08 +09:00
afaab3d332
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails ( #2404 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-10-27 16:56:30 +09:00
12c908c827
HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.
2020-10-26 22:12:27 +08:00
74634eb002
HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.
2020-10-23 12:05:53 -07:00
e7aa4da0a2
HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.
2020-10-23 09:47:10 -07:00
da1b6e3cc2
HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.
2020-10-22 16:59:49 -05:00
88a9f42f32
HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.
2020-10-21 00:59:35 -05:00
2e8cafac3b
HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.
2020-10-19 10:48:47 +05:30
173310e2f5
HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang.
2020-10-19 10:34:17 +05:30
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. ( #2342 ). Contributed by Ayush Saxena.
2020-10-17 12:01:18 +05:30
740a2c4635
HDFS-15627. Log delete audits before collecting blocks.
2020-10-16 11:13:41 -05:00
a308a1ec22
HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled ( #2370 )
2020-10-13 10:59:42 -07:00
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
...
Contributed by Gabor Bota.
2020-10-13 16:17:44 +01:00
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
Change-Id: I6cec41c20467920cf21f169653553535414b2680
2020-10-13 13:31:00 +01:00
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing ( #2380 )"
...
(reverting to change the JIRA number)
This reverts commit 3e1b1dfd77
2020-10-13 13:30:38 +01:00
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
2020-10-13 13:27:55 +01:00
b3786d6c3c
HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.
2020-10-12 17:26:49 -07:00
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup ( #2378 ). Contributed by Uma Maheswara Rao G.
2020-10-12 14:22:44 -07:00
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently ( #2299 )
2020-10-10 22:08:36 +05:30
735e85a1d2
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 ( #2365 )
2020-10-08 12:51:04 +05:30
df4006eb81
HDFS-15253 Default checkpoint transfer speed, 50mb per second ( #2366 )
2020-10-07 22:09:04 +05:30
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. ( #2355 )
2020-10-07 10:34:32 +05:30
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently ( #2356 ). Contributed by Hui Fei.
2020-10-05 22:11:24 +05:30
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir ( #2352 )
2020-10-05 05:02:00 -07:00
984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc ( #2345 )
2020-09-28 09:54:01 -07:00
d89af79507
HDFS-15577. Refactor TestTracing. ( #2302 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-09-28 18:27:41 +09:00
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng
2020-09-27 19:27:13 +05:30
00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip ( #2332 )
2020-09-25 09:47:54 -07:00
486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 ( #2330 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2020-09-24 21:15:55 +09:00
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS ( #2189 )
...
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
2020-09-24 14:27:04 +05:30
368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled ( #2326 )
2020-09-24 14:00:41 +05:30
ae089f2db7
HDFS-15557. Log the reason why a storage log file can't be deleted ( #2274 )
2020-09-22 13:23:20 -07:00
474fa80bfb
HADOOP-17277. Correct spelling errors for separator ( #2322 )
...
Contributed by Hui Fei.
2020-09-22 17:10:33 +01:00
2377df38ad
HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.
2020-09-18 17:28:11 +05:30
7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. ( #2312 ). Contributed by Uma Maheswara Rao G.
2020-09-18 15:18:10 +05:30
20a0e6278d
HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell
2020-09-17 14:11:42 +01:00
425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. ( #2296 )
2020-09-17 14:50:08 +05:30
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled ( #2305 ). Contributed by Uma Maheswara Rao G.
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-16 22:43:00 -07:00
5c5b2ed7c7
HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui.
2020-09-16 01:36:46 +09:00
f4ed9f3f91
HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.
2020-09-15 10:10:21 +05:30
924959088e
HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun.
2020-09-15 12:05:04 +08:00
6d3312de47
HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty ( #2258 )
2020-09-14 13:31:34 -07:00
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. ( #2298 ). Contributed by Uma Maheswara Rao G.
2020-09-12 17:06:39 -07:00
f59f7f2175
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell
2020-09-12 01:41:38 -07:00
2a6fdcedeb
HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.
2020-09-11 18:21:25 +05:30
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache ( #2241 )
...
Contributed by fanrui.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-09-10 22:30:52 -07:00
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree ( #2185 ). Contributed by Zhenzhao Wang.
...
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
2020-09-10 21:20:32 -07:00
43c52d6495
HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir ( #2295 )
...
Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc
2020-09-10 15:55:23 +05:30
1d6d0d8207
HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun.
...
Co-authored-by: Chen Zhang <chzhang1987@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2020-09-09 20:09:13 +08:00
ac7d4623ae
HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured ( #2275 ) Contributed by Uma Maheswara Rao G.
2020-09-07 11:36:13 -07:00
b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well ( #2234 ). Contributed by Uma Maheswara Rao G.
2020-09-03 11:06:20 -07:00
60de592a88
HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree.
2020-08-31 14:49:12 +08:00
2d03209a06
HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion ( #2251 )
2020-08-31 11:29:48 +05:30
c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing ( #2252 )
...
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
2020-08-28 15:39:09 +01:00
06793da100
HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory ( #2230 )
2020-08-27 11:36:30 -07:00
41182a9b6d
HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. ( #2233 )
2020-08-27 14:54:52 +05:30
2ffe00fc46
HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-08-26 23:05:32 -07:00
6a49bf9bff
HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.
2020-08-25 15:18:36 +01:00
a7830423c5
HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.
2020-08-24 21:16:00 +08:00
83fd15b412
HDFS-14504. Rename with Snapshots does not honor quota limit.
2020-08-23 16:10:47 +05:30
f734455e5d
HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.
2020-08-20 15:25:10 -07:00
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. ( #2229 ). Contributed by Uma Maheswara Rao G.
2020-08-19 09:30:41 -07:00
b65e43fe38
HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS ( #2220 )
2020-08-18 19:28:19 +09:00
092bfe7c8e
HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. ( #2172 )
...
* HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories.
* Addressed review comments.
* Rebased to latest trunk and added snapshotTrashRoot config check.
* Addressed review comments.
* Removede empty line added in SnapshotManager.Java.
* Addressed whitespace issues.
2020-08-17 12:56:13 +05:30
e3d1966f58
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
2020-08-15 15:07:58 +05:30
b93dd7c281
HDFS-15519. Check inaccessible INodes in FsImageValidation. ( #2224 )
2020-08-14 10:10:01 -07:00
15a76e8d65
HDFS-15524. Add edit log entry for Snapshot deletion GC thread snapshot deletion. ( #2219 )
2020-08-14 20:46:48 +05:30
cb50e3fcf7
HDFS-15496. Add UI for deleted snapshots ( #2212 )
2020-08-13 22:36:15 +05:30
e592ec5f8b
HDFS-15518. Wrong operation name in FsNamesystem for listSnapshots. ( #2217 )
...
* HDFS-15518. Fixed String operationName = ListSnapshot.
* HDFS-15518. Changed ListSnapshot to operationName.
Co-authored-by: Aryan Gupta <aryan.gupta@cloudera.com>
2020-08-13 00:09:38 +05:30
10716040a8
HDFS-15493. Update block map and name cache in parallel while loading fsimage. Contributed by Chengwei Wang
2020-08-12 09:02:47 +01:00
11cec9ab94
HDFS-15523. Fix findbugs warnings from HDFS-15520. ( #2218 )
2020-08-11 21:41:28 -07:00
3fd3aeb621
HDFS-15492. Make trash root inside each snapshottable directory ( #2176 )
2020-08-11 08:52:16 -07:00
8955a6ceb7
HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false ( #2205 )
...
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false
* Used LambdaTestUtils#intercept in test
2020-08-11 00:01:58 -07:00
592127bdc2
HDFS-15520 Use visitor pattern to visit namespace tree ( #2203 )
2020-08-11 12:30:30 +05:30
7938ebfb9d
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.
2020-08-11 11:14:58 +09:00
c2a17659d1
HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.
2020-08-08 14:33:53 +05:30
975b6024dd
HDFS-15514. Remove useless dfs.webhdfs.enabled. Contributed by Fei Hui.
2020-08-07 22:19:17 +05:30
e072d33327
HDFS-15497. Make snapshot limit on global as well per snapshot root directory configurable ( #2175 )
2020-08-04 14:10:29 +05:30
82f3ffcd64
HDFS-15503. File and directory permissions are not able to be modified from WebUI.
2020-08-03 23:21:01 +05:30
c40cbc57fa
HADOOP-17091. [JDK11] Fix Javadoc errors ( #2098 )
2020-08-03 10:46:51 +09:00
d8a2df25ad
HDFS-15498. Show snapshots deletion status in snapList cmd. ( #2181 )
2020-08-02 21:59:56 +05:30
528a799a78
HDFS-15229. Truncate info should be logged at INFO level. Contributed by Ravuri Sushma sree.
2020-08-02 12:02:37 +05:30
05b3337a46
HDFS-15481. Ordered snapshot deletion: garbage collect deleted snapshots ( #2165 )
2020-07-30 23:06:51 +05:30
68287371cc
HDFS-15488. Add a command to list all snapshots for a snaphottable root with snapshot Ids. ( #2166 )
2020-07-29 21:33:25 +05:30
026dce5334
HDFS-15465. Support WebHDFS accesses to the data stored in secure Datanode through insecure Namenode. ( #2135 )
2020-07-27 09:55:11 -07:00
2d12496643
HDFS-15480. Ordered snapshot deletion: record snapshot deletion in XAttr ( #2163 )
2020-07-22 16:16:27 -07:00
ac9a07b51a
HDFS-15478: When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs. ( #2160 ). Contributed by Uma Maheswara Rao G.
2020-07-21 23:29:10 -07:00
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 )
2020-07-22 13:40:20 +09:00
8b7695bb26
HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.
2020-07-21 09:00:07 -07:00
d57462f2da
HDFS-15479. Ordered snapshot deletion: make it a configurable feature ( #2156 )
2020-07-21 11:36:24 +05:30
d9441f95c3
HDFS-15470. Added more unit tests to validate rename behaviour across snapshots. Contributed by Shashikant Banerjee.
2020-07-21 11:13:05 +05:30
3833c616e0
HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang.
2020-07-20 12:49:58 -07:00
6cbd8854ee
HDFS-15381. Fix typos corrputBlocksFiles to corruptBlocksFiles. Contributed by bianqi.
2020-07-20 21:49:05 +05:30
2cec50cf16
HDFS-15463. Add a tool to validate FsImage. ( #2140 )
2020-07-20 11:44:30 +05:30
6bcb24d269
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-18 14:33:43 +05:30
1f71c4ae71
HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 11:39:06 -05:00
98fcffe93f
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 09:53:18 -05:00
85d4718ed7
HDFS-15319. Fix INode#isInLatestSnapshot() API. Contributed by Shashikant Banerjee.
2020-07-15 10:32:25 +05:30
bdce75d737
HDFS-15371. Nonstandard characters exist in NameNode.java ( #2032 )
...
Contributed by zhaoyim
2020-07-14 14:42:12 +01:00
b97fea65e7
HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell.
2020-07-13 14:12:48 +08:00
3e70006639
HDFS-15464: ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links ( #2132 ). Contributed by Uma Maheswara Rao G.
2020-07-11 23:50:04 -07:00
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. ( #2085 )
2020-07-09 11:33:37 -07:00
f77bbc2123
HDFS-15312. Apply umask when creating directory by WebHDFS ( #2096 )
2020-07-06 19:01:46 -07:00
dc0626b5f2
HDFS-15449. Optionally ignore port number in mount-table name when picking from initialized uri. Contributed by Uma Maheswara Rao G.
2020-07-06 18:50:03 -07:00
834372f404
HDFS-15451. Do not discard non-initial block report for provided storage. ( #2119 ). Contributed by Shanyu Zhao.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-07-06 23:43:34 +08:00
55a2ae80dc
HDFS-15450. Fix NN trash emptier to work if ViewFSOveroadScheme enabled. Contributed by Uma Maheswara Rao G.
2020-07-04 13:45:49 -07:00
f86f15cf20
HDFS-15446. CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/path. Contributed by Stephen O'Donnell.
2020-07-04 13:00:43 +05:30
1f2a80b5e5
HDFS-15430. create should work when parent dir is internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
2020-07-04 00:12:10 -07:00
9ac498e300
HDFS-15416. Improve DataStorage#addStorageLocations() for empty locations. Contibuted by jianghua zhu.
2020-07-01 12:38:07 +08:00
2a67e2b1a0
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-30 07:10:17 -07:00
c71ce7ac33
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
2020-06-28 16:02:47 +09:00
8db38c98a6
HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.
2020-06-27 19:13:01 +05:30
bed0a3a374
HDFS-15436. Default mount table name used by ViewFileSystem should be configurable ( #2100 )
...
* HDFS-15436. Default mount table name used by ViewFileSystem should be configurable
* Replace Constants.CONFIG_VIEWFS_DEFAULT_MOUNT_TABLE use in tests
* Address Uma's comments on PR#2100
* Sort lists in test to match without concern to order
* Address comments, fix checkstyle and fix failing tests
* Fix checkstyle
2020-06-26 13:19:16 -07:00
d5e1bb6155
HDFS-15429. mkdirs should work when parent dir is an internalDir and fallback configured. Contributed by Uma Maheswara Rao G.
2020-06-26 01:29:38 -07:00
7c02d1889b
HDFS-15427. Merged ListStatus with Fallback target filesystem and InternalDirViewFS. Contributed by Uma Maheswara Rao G.
2020-06-23 01:42:25 -07:00
17ffcab5f6
HDFS-14546. Document block placement policies. Contributed by Amithsha.
2020-06-21 17:27:28 +05:30
b27810aa60
HDFS-15418. ViewFileSystemOverloadScheme should represent mount links as non symlinks. Contributed by Uma Maheswara Rao G.
2020-06-20 00:32:02 -07:00
d50e93ce7b
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-18 06:45:15 -07:00
edf716a5c3
Revert "HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell."
...
This reverts commit 730a39d138
2020-06-18 06:45:15 -07:00
123777823e
HDFS-15406. Improve the speed of Datanode Block Scan. Contributed by hemanthboyina
2020-06-18 12:26:22 +01:00
c8ed33cd2a
HADOOP-9851. dfs -chown does not like "+" plus sign in user name. Contributed by Andras Bokor.
2020-06-17 13:55:40 +05:30
730a39d138
HDFS-15372. Files in snapshots no longer see attribute provider permissions. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-06-16 15:59:13 -07:00
f41a144077
HDFS-15403. NPE in FileIoProvider#transferToSocketFully. Contributed by hemanthboyina.
2020-06-15 09:15:53 +09:00
719b53a79d
HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.
2020-06-13 09:35:05 -07:00
785b1def95
HDFS-15387. FSUsage#DF should consider ViewFSOverloadScheme in processPath. Contributed by Uma Maheswara Rao G.
2020-06-12 14:32:19 -07:00
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:16:33 +05:30
b735a77717
HDFS-15398. EC: hdfs client hangs due to exception during addBlock. Contributed by Hongbing Wang.
2020-06-10 12:06:16 +05:30
8525874561
HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena.
...
*Added missed test file.
2020-06-09 18:53:33 +05:30
3ca15292c5
HDFS-15394. Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault. Contributed by Uma Maheswara Rao G.
2020-06-06 08:11:57 -07:00
cc671b16f7
HDFS-15389. DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadScheme. Contributed by Ayush Saxena
2020-06-05 22:19:38 -07:00
76fa0222f0
HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
2020-06-05 10:58:21 -07:00
2326123705
HDFS-15359. EC: Allow closing a file with committed blocks. Contributed by Ayush Saxena.
2020-06-05 19:15:05 +05:30
545a0a147c
HDFS-15386 ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories ( #2052 )
...
Contributed by Toshihiro Suzuki.
2020-06-05 11:11:49 +01:00
e8cb2ae409
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang.
2020-06-03 12:47:15 +05:30
f6453244ab
HDFS-14960. TestBalancerWithNodeGroup should not succeed with DFSNetworkTopology. Contributed by Jim Brennan.
2020-06-03 12:16:36 +05:30
ed83c865dd
HDFS-15321. Make DFSAdmin tool to work with ViewFileSystemOverloadScheme. Contributed by Uma Maheswara Rao G.
2020-06-02 11:09:26 -07:00
ae13a5ccbe
HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang.
2020-05-31 17:10:59 +05:30
d9838f2d42
HADOOP-17055. Remove residual code of Ozone ( #2039 )
2020-05-29 16:49:18 +09:00
a838d871a7
HDFS-15368. TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally. Contributed by Xiaoqiao He.
2020-05-28 20:52:58 +05:30
9b38be43c6
HDFS-13183. Addendum: Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
2020-05-28 13:00:04 +05:30
2148a8fe64
HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina.
2020-05-27 11:06:13 -07:00
6c9f75cf16
HDFS-15373. Fix number of threads in IPCLoggerChannel#createParallelExecutor. Contributed by Ayush Saxena.
2020-05-26 16:26:57 +05:30
f43a152b97
HDFS-15369. Refactor method VolumeScanner#runLoop(). Contributed by Yang Yun.
2020-05-24 18:03:41 +05:30
f4901d0778
HDFS-15355. Make the default block storage policy ID configurable. Contributed by Yang Yun.
2020-05-24 17:59:16 +05:30
6e04b00df1
HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun.
2020-05-23 09:58:19 -07:00
e0ae232f66
HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena.
2020-05-23 18:38:32 +05:30
f99fcb26ab
HDFS-15288. Add Available Space Rack Fault Tolerant BPP. Contributed by Ayush Saxena.
2020-05-23 18:29:31 +05:30
4d22d1c58f
HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina.
2020-05-23 17:28:38 +09:00
4734c77b4b
HDFS-15322. Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same. Contributed by Uma Maheswara Rao G.
2020-05-21 21:34:58 -07:00
be374faf42
HDFS-13639. SlotReleaser is not fast enough ( #1885 )
2020-05-21 13:21:17 -07:00
ce4ec74453
HADOOP-17024. ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root). Contributed by Abhishek Das.
2020-05-18 22:27:12 -07:00
7bb902bc0d
HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang
2020-05-18 10:58:52 -07:00
c84e6beada
HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena.
2020-05-18 22:24:34 +05:30
2abcf7762a
HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )
2020-05-18 09:23:09 -07:00
4525292d41
Revert "HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )"
...
This reverts commit 86e6aa8eec
2020-05-18 09:22:05 -07:00
50caba1a92
HDFS-15207. VolumeScanner skip to scan blocks accessed during recent scan peroid. Contributed by Yang Yun.
2020-05-18 08:40:38 -07:00
a3f44dacc1
HDFS-13183. Standby NameNode process getBlocks request to reduce Active load. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-05-18 07:09:43 -07:00
86e6aa8eec
HDFS-15202 Boost short circuit cache (rebase PR-1884) ( #2016 )
...
Added parameter dfs.client.short.circuit.num improving HDFS-client's massive reading performance by create few instances ShortCircuit caches instead of one. It helps avoid locks and lets CPU do job.
2020-05-18 07:04:04 -07:00
6e416a83d1
HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena.
2020-05-17 03:06:27 +05:30
178336f8a8
HDFS-15356. Unify configuration `dfs.ha.allow.stale.reads` to DFSConfigKeys. Contributed by Xiaoqiao He.
2020-05-16 16:35:06 +05:30
ac4a2e11d9
HDFS-15306. Make mount-table to read from central place ( Let's say from HDFS). Contributed by Uma Maheswara Rao G.
2020-05-14 17:29:35 -07:00
1958cb7c2b
YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng
2020-05-14 15:09:34 +05:30
0918433b4d
YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng.
2020-05-14 00:36:20 +05:30
3cacf1ce56
HDFS-15344. DataNode#checkSuperuserPrivilege should use UGI#getGroups after HADOOP-13442. ( #2004 )
2020-05-13 11:47:19 -07:00
743c2e9071
HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina
2020-05-13 15:01:07 +05:30
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) ( #2007 )
...
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
29dddb8a14
HDFS-15255. Consider StorageType when DatanodeManager#sortLocatedBlock(). Contributed by Lisheng Sun.
2020-05-12 15:07:51 +01:00
928b81a533
HDFS-15350. Set dfs.client.failover.random.order to true as default. ( #2008 )
2020-05-12 09:04:03 -05:00
8dad38c0be
HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei.
2020-05-12 18:34:26 +05:30
0fe49036e5
HDFS-15243. Add an option to prevent sub-directories of protected directories from deletion. Contributed by liuyanyu.
2020-05-12 13:11:31 +05:30
4c53fb9ce1
HDFS-15338. listOpenFiles() should throw InvalidPathException in case of invalid paths. Contributed by Jinglun.
2020-05-11 16:48:34 +05:30
cb64e993c2
HDFS-15347. Replace the deprecated method shaHex. Contributed by bianqi.
2020-05-10 00:11:46 +05:30
d59de27c01
HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole.
2020-05-08 00:53:49 +05:30
130f89e068
HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina.
2020-05-06 13:18:32 -07:00
7fddf4855e
HDFS-14283. DFSInputStream to prefer cached replica. Contributed by Lisheng Sun.
2020-05-06 16:55:04 +05:30
adecdb8b53
HDFS-15334. INodeAttributeProvider's new API checkPermissionWithContext not getting called in for authorization. ( #1998 )
...
Reviewed-by: Arpit Agarwal <arp@apache.org>
2020-05-05 16:04:54 -07:00
9c8236d04d
HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G.
2020-05-04 17:55:40 -07:00
ebb878bab9
HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko.
2020-05-04 10:29:50 -07:00
8dace8ff3a
HDFS-15328. Use DFSConfigKeys MONITOR_CLASS_DEFAULT constant. Contributed by bianqi.
2020-05-03 23:59:17 +05:30
d6fc482a54
Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida.
2020-05-03 19:09:14 +05:30
44de193bec
HDFS-15325. TestRefreshCallQueue is failing due to changed CallQueue constructor. Contributed by Fengnan Li
2020-05-02 20:56:29 -07:00
f40dacdc2e
HDFS-15320. StringIndexOutOfBoundsException in HostRestrictingAuthorizationFilter ( #1992 )
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-02 13:02:27 -07:00
82343790ee
HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee.
2020-05-01 12:16:02 +05:30
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2020-04-30 14:15:28 -07:00
9ca6298a9a
HDFS-15285. The same distance and load nodes don't shuffle when consider DataNode load. Contributed by Lisheng Sun.
2020-04-29 15:56:06 +01:00
5e0eda5d5f
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina.
2020-04-28 06:17:14 +09:00
62c26b91fd
HDFS-15298 Fix the findbugs warnings introduced in HDFS-15217 ( #1979 )
2020-04-27 17:20:47 +09:00
74d4a18735
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena
2020-04-25 15:43:02 -07:00
453c9c9a67
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address ( #1964 )
...
Contributed by Dhiraj Hegde.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
2020-04-25 13:04:32 -07:00
453771f170
HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun.
2020-04-24 11:53:24 -07:00
5958af4dbc
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina
2020-04-22 13:31:02 -07:00
1824aee9da
HDFS-15217 Add more information to longest write/read lock held log
2020-04-18 13:52:07 -07:00
37d6582223
HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena.
2020-04-17 13:32:02 +05:30
3481895f8a
HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-04-16 20:18:25 -07:00
cc5c1da7c1
HDFS-15277. Parent directory in the explorer does not support all path formats. Contributed by Jianfei Jiang.
2020-04-15 23:01:58 +05:30
061afcdf30
HDFS-15269. NameNode should check the authorization API version only … ( #1945 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-04-09 09:19:35 -07:00
1189af4746
HDFS-15263. Fix the logic of scope and excluded scope in Network Topology. Contributed by Ayush Saxena.
2020-04-08 15:55:14 +05:30
c12ddbd1de
HDFS-15249 ThrottledAsyncChecker is not thread-safe. ( #1922 )
2020-04-07 14:38:09 +09:00
0b855b9f35
HDFS-15256. Fix typo in DataXceiverServer#run(). Contributed by Lisheng Sun.
2020-04-06 13:05:33 -07:00
d3b5951572
HDFS-15242. Add metrics for operations hold lock times of FsDatasetImpl. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-04-01 16:36:46 -07:00
960c9ebaea
HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang.
2020-03-30 02:40:25 +05:30
8914cf9167
Preparing for 3.4.0 development
2020-03-29 23:24:25 +05:30
f7a94ec0a4
HDFS-15239. Add button to go to the parent directory in the explorer. Contributed by hemanthboyina.
2020-03-29 08:17:02 -07:00
696a663cd0
HDFS-15193. Improving the error message for missing `dfs.namenode.rpc-address.$NAMESERVICE`. Contributed by Ctest.
2020-03-29 00:01:22 +05:30
f531a4a487
HDFS-15191. EOF when reading legacy buffer in BlockTokenIdentifier. Contributed by Steven Rand.
2020-03-27 12:55:38 -07:00
0fa7bf47df
HDFS-15234. Add a default method body for the INodeAttributeProvider#checkPermissionWithContext API. ( #1909 )
2020-03-25 16:03:26 -07:00
cdcb77a2c5
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He.
2020-03-25 11:30:54 -07:00
a700803a18
HDFS-15154. Allow only hdfs superusers the ability to assign HDFS storage policies. Contributed by Siddharth Wagle.
...
Change-Id: I32d6dd2837945b8fc026a759aa367c55daefe348
2020-03-25 10:28:30 -07:00
d353b30baf
HDFS-15215. The Timestamp for longest write/read lock held log is wrong
2020-03-24 14:50:15 -07:00
ea87d60493
HDFS-13377. The owner of folder can set quota for his sub folder. Contributed by Yang Yun.
2020-03-24 22:56:09 +05:30
5250cd6db3
HDFS-15201 SnapshotCounter hits MaxSnapshotID limit ( #1870 )
2020-03-24 15:15:46 +05:30
af64ce2f4a
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-03-23 12:43:51 -07:00
62c8015683
HDFS-15227. NPE if the last block changes from COMMITTED to COMPLETE during FSCK. Contributed by Ayush Saxena.
2020-03-22 11:44:18 +05:30
b74e47e31a
HDFS-15214. WebHDFS: Add snapshot counts to Content Summary. Contributed by hemanthboyina.
2020-03-22 01:44:55 +09:00
f9bb2a8cc5
HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena.
2020-03-19 22:26:34 +05:30
bb41ddaf1e
HDFS-15223. FSCK fails if one namenode is not available. Contributed by Ayush Saxena.
2020-03-19 21:23:13 +05:30
654db35fa2
HDFS-14919. Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
2020-03-18 21:00:45 +05:30
096533c2dc
HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
...
Contributed by Wei-Chiu Chuang.
Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
2020-03-18 13:27:13 +00:00
8a7c54995a
HDFS-15159. Prevent adding same DN multiple times in PendingReconstructionBlocks. Contributed by hemanthboyina.
2020-03-15 16:16:27 +05:30
7b637a3341
HDFS-14820. The default 8KB buffer of BlockReaderRemote#newBlockReader#BufferedOutputStream is too big. Contributed by Lisheng Sun.
2020-03-13 19:01:23 -07:00
e9955bb8ff
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2020-03-13 18:55:06 -07:00
4b95c242ec
HDFS-14743. Enhance INodeAttributeProvider/ AccessControlEnforcer Interface in HDFS to support Authorization of mkdir, rm, rmdir, copy, move etc... ( #1829 )
...
Reviewed-by: Xiaoyu Yao <xyao@apache.org>
2020-03-13 11:29:44 -07:00
fb1d7635ae
HDFS-15155. writeIoRate of DataNodeVolumeMetrics is never used. Contributed by Haibin Huang.
2020-03-13 21:10:03 +05:30
25497799cb
HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree.
2020-03-13 15:24:19 +00:00
20903f72b4
HDFS-15039. Cache meta file length of FinalizedReplica to reduce call File.length(). Contributed by Yang Yun.
2020-03-12 19:23:12 -07:00
0695f7a538
HDFS-14612. SlowDiskReport won't update when SlowDisks is always empty in heartbeat. Contributed by Haibin Huang.
2020-03-12 17:22:11 -07:00
f736408a83
HDFS-14442. Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionId. Contributed by Ravuri Sushma sree.
2020-03-12 18:59:17 +05:30
c734d69a55
HADOOP-16898. Batch listing of multiple directories via an (unstable) interface
...
Contributed by Steve Loughran.
This moves the new API of HDFS-13616 into a interface which is implemented by
HDFS RPC filesystem client (not WebHDFS or any other connector)
This new interface, BatchListingOperations, is in hadoop-common,
so applications do not need to be compiled with HDFS on the classpath.
They must cast the FS into the interface.
instanceof can probe the client for having the new interface -the patch
also adds a new path capability to probe for this.
The FileSystem implementation is cut; tests updated as appropriate.
All new interfaces/classes/constants are marked as @unstable.
Change-Id: I5623c51f2c75804f58f915dd7e60cb2cffdac681
2020-03-09 14:51:16 +00:00
4062217189
HDFS-15212. TestEncryptionZones.testVersionAndSuiteNegotiation fails in trunk. Contributed by Ayush Saxena.
2020-03-06 10:14:53 -08:00
3afd4cbe89
HDFS-14977. Quota Usage and Content summary are not same in Truncate with Snapshot. Contributed by hemanthboyina.
2020-03-04 10:13:23 -08:00
1d4d0fcbe1
HDFS-15204. TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittently. Contributed by Ahmed Hussein.
2020-03-04 10:02:54 -08:00
bbd704bb82
HADOOP-16899. Update HdfsDesign.md to reduce ambiguity. ( #1871 )
2020-03-04 11:07:31 +09:00
1a636da041
HDFS-15033. Support to save replica cached files to other place and make expired time configurable. Contributed by Yang Yun.
2020-02-29 09:55:29 +05:30
97b797c314
HDFS-15149. TestDeadNodeDetection test cases time-out. Contributed by Lisheng Sun.
2020-02-28 18:47:22 -08:00
fccfb02672
HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena.
2020-02-28 18:33:06 +05:30
429da635ec
HDFS-15186. Erasure Coding: Decommission may generate the parity block's content with all 0 in some case. Contributed by Yao Guangdong.
2020-02-28 00:31:55 +05:30
cd2c6b1aac
HDFS-15124. Crashing bugs in NameNode when using a valid configuration for . Contributed by Ctest.
2020-02-27 22:19:35 +05:30
2059f255d3
HDFS-15167. Block Report Interval shouldn't be reset apart from first Block Report. Contributed by Ayush Saxena.
2020-02-27 14:08:42 +05:30
033a3d7ff9
HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li.
2020-02-27 04:02:26 +05:30
209630472a
HDFS-15120. Refresh BlockPlacementPolicy at runtime. Contributed by Jinglun.
2020-02-27 02:22:24 +05:30
352a4ec16d
HDFS-15147. LazyPersistTestCase wait logic is flawed. Contributed by Ahmed Hussein.
2020-02-26 09:33:29 -06:00
900430b990
HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-25 13:28:50 -08:00
1c5d2f1fdc
HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun.
2020-02-24 13:39:16 -08:00
7f8685f476
HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena.
2020-02-24 20:38:04 +05:30
93b8f453b9
HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li.
2020-02-24 20:17:01 +05:30
b5698e0c33
HDFS-15176. Enable GcTimePercentage Metric in NameNode's JvmMetrics. Contributed by Jinglun.
2020-02-24 00:07:18 +05:30
9eb7a8bdf8
HDFS-15041. Make MAX_LOCK_HOLD_MS and full queue size configurable. Contributed by zhuqi.
2020-02-23 14:25:39 +05:30
ba9025c7cd
HDFS-15182. TestBlockManager#testOneOfTwoRacksDecommissioned() fail in trunk. Contributed by Lisheng Sun.
2020-02-22 14:27:26 +05:30
ecbcb058b8
HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko.
2020-02-21 17:53:37 -08:00
6f84269bcd
HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko.
2020-02-21 10:51:14 -08:00
2338d25dc7
HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. ( #1758 )
2020-02-21 11:56:07 +09:00
ec7507162c
HDFS-15165. In Du missed calling getAttributesProvider. Contributed by Bharat Viswanadham.
2020-02-19 11:33:58 -08:00
ac4b556e2d
HDFS-13739. Add option to disable rack local write preference. Contributed by Ayush Saxena.
2020-02-19 08:20:59 +05:30
84f7638840
HADOOP-13666. Supporting rack exclusion in countNumOfAvailableNodes in NetworkTopology. Contributed by Inigo Goiri.
2020-02-18 00:43:33 +05:30
810783d443
HDFS-15135. EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped. Contributed by Ravuri Sushma sree.
2020-02-16 12:15:03 +05:30
c75756fe13
HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena.
2020-02-14 22:07:24 +05:30
a98352ced1
HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina.
2020-02-13 16:57:41 +05:30
0ddb5f0881
HDFS-13989. RBF: Add FSCK to the Router ( #1832 )
...
Co-authored-by: Inigo Goiri <inigoiri@apache.org>
2020-02-13 10:06:07 +09:00
9b8a78d97b
HDFS-14758. Make lease hard limit configurable and reduce the default.
...
Contributed by hemanthboyina.
2020-02-11 12:40:00 -06:00
d7c136b9ed
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-02-11 08:00:15 -08:00
6191d4b4a0
HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun.
2020-02-09 23:32:22 +05:30
d23317b102
HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang
2020-02-08 10:33:57 +05:30
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency ( #1635 ). Contributed by Vinayakumar B.
2020-02-07 14:51:24 +05:30
ce7b8b5634
HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang.
2020-02-04 12:12:35 -08:00
10a60fbe20
HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan.
2020-02-04 12:22:35 -06:00
1e3a0b0d93
HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
2020-01-31 16:13:02 -08:00
799d4c1cf4
HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails
...
intermittently. Contributed by Ahmed Hussein.
2020-01-29 11:00:27 -06:00
483397c7f7
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang
2020-01-28 15:20:36 -08:00
3e86807802
Revert "[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang."
...
This reverts commit ff8ff0f7e5
2020-01-28 15:19:47 -08:00
ff8ff0f7e5
[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang.
2020-01-28 15:14:58 -08:00
1839c467f6
HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein.
2020-01-28 10:10:35 -08:00
87c198468b
HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun.
2020-01-28 23:09:38 +05:30
f876dc228b
HDFS-15143. LocatedStripedBlock returns wrong block type. Contributed by Ayush Saxena.
2020-01-28 19:34:21 +05:30
6d008c0d39
HDFS-15128. Unit test failing to clean testing data and crashed future Maven test run due to failure in TestDataNodeVolumeFailureToleration. Contributed by Ctest
2020-01-25 11:47:29 +05:30
d10f77e3c9
HDFS-15119. Allow expiration of cached locations in DFSInputStream.
...
Contributed by Ahmed Hussein.
2020-01-24 09:15:27 -06:00
92c58901d7
HDFS-15117. EC: Add getECTopologyResultForPolicies to DistributedFileSystem. Contributed by Ayush Saxena
2020-01-23 18:18:34 +05:30
bd03053ea2
HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein.
2020-01-22 09:14:38 -06:00
8cfc3673dc
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui.
2020-01-21 13:29:20 -08:00
b657822b98
HDFS-15126. TestDatanodeRegistration#testForcedRegistration fails intermittently. Contributed by Ahmed Hussein.
2020-01-21 13:22:53 -08:00
1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li.
2020-01-21 10:58:32 +09:00
d7c4f8ab21
HDFS-13616. Batch listing of multiple directories ( #1725 )
2020-01-15 17:22:47 -08:00
52b360a928
HADOOP-16797. Add Dockerfile for ARM builds. Contributed by Vinayakumar B. ( #1801 )
2020-01-13 10:40:29 +05:30
cebce0a348
HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena.
2020-01-11 10:29:11 +05:30
26a969ec73
HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.
2020-01-10 17:06:59 -08:00
5fb901ac40
HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein.
2020-01-10 12:43:21 -06:00
b32757c616
HDFS-15107. dfs.client.server-defaults.validity.period.ms to support time units. Contributed by Ayush Saxena.
2020-01-10 08:14:56 +05:30
bf45f3b80a
HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina.
2020-01-09 12:04:05 +05:30
7030722e5d
HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He.
2020-01-08 14:25:17 +05:30
aba3f6c3e1
HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. ( #1797 )
2020-01-08 16:45:39 +09:00
a43c177f1d
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. ( #1775 )
2020-01-08 11:28:34 +09:00
f8644fbe9f
HDFS-15091. Cache Admin and Quota Commands Should Check SuperUser Before Taking Lock. Contributed by Ayush Saxena.
2020-01-04 19:02:59 +05:30
037ec8cfb1
HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2020-01-04 01:55:36 +09:00
d79cce20ab
HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.
2020-01-02 11:44:00 +05:30
074050ca59
HDFS-15063. HttpFS: getFileStatus doesn't return ecPolicy. Contributed by hemanthboyina.
2020-01-01 11:26:38 +09:00
ee51eadda0
HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina.
2019-12-29 11:15:54 +05:30
dc32f583af
HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena.
2019-12-28 21:32:15 +09:00
8730a7bf60
HDFS-15003. RBF: Make Router support storage type quota. Contributed by Jinglun.
2019-12-27 09:22:58 +05:30
80f91d14ab
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He.
2019-12-27 09:12:44 +05:30
300505c562
HDFS-15054. Delete Snapshot not updating new modification time. Contributed by hemanthboyina.
2019-12-25 12:42:50 +05:30
b98ac2a3af
HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko.
2019-12-24 12:08:34 -08:00
52d7b745c6
HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui.
2019-12-19 09:42:17 -08:00
b86895485d
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He.
2019-12-19 09:34:43 -08:00
fdd96e46d1
HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee.
2019-12-18 22:50:46 +05:30
24080666e5
HDFS-14908. LeaseManager should check parent-child relationship when filter open files. Contributed by Jinglun.
2019-12-16 18:41:45 -08:00
dc6cf17b34
HDFS-15048. Fix findbug in DirectoryScanner.
2019-12-16 19:00:57 +09:00
7a87007545
HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina.
2019-12-15 00:33:32 +05:30
65c4660bcd
HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang.
2019-12-12 10:22:05 -08:00
bdd00f10b4
HDFS-15047. Document the new decommission monitor (HDFS-14854). ( #1755 )
2019-12-13 00:35:41 +09:00
93bb368094
HDFS-14983. RBF: Add dfsrouteradmin -refreshSuperUserGroupsConfiguration command option. Contributed by Xieming Li
2019-12-12 10:56:19 +09:00
c174d50b30
HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen.
2019-12-11 09:39:37 -08:00
c93cb6790e
HDFS-14854. Create improved decommission monitor implementation. Contributed by Stephen O'Donnell.
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-12-10 17:16:43 -08:00
11cd5b6e39
HDFS-15028. Keep the capacity of volume and reduce a system call. Contributed by Yang Yun.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-12-07 08:46:45 +09:00
ecd461f940
HDFS-14751. Synchronize on diffs in DirectoryScanner. Contributed by Lisheng Sun.
2019-12-06 15:10:40 -08:00
313b76f8e9
HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow.
2019-12-06 15:10:40 -08:00
705b172b95
HDFS-14998. [SBN read] Update Observer Namenode doc for ZKFC after HDFS-14130. Contributed by Fei Hui.
2019-12-06 23:04:09 +05:30
83a14559e5
HDFS-15023. [SBN read] ZKFC should check the state before joining the election. Contributed by Fei Hui.
2019-12-05 18:52:01 +05:30
0c217feed8
HDFS-15026. TestPendingReconstruction#testPendingReconstruction() fail in trunk. Contributed by Lisheng Sun.
2019-12-03 09:18:09 +05:30
6b2d6d4aaf
HDFS-15009. FSCK -list-corruptfileblocks return Invalid Entries. Contributed by hemanthboyina.
2019-11-29 23:55:37 +05:30
44f7b9159d
HDFS-15013. Reduce NameNode overview tab response time. Contributed by HuangTao.
2019-11-29 00:42:31 +05:30
0384687811
HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore.
2019-11-28 23:49:35 +05:30
46166bd8d1
HDFS-14961. [SBN read] Prevent ZKFC changing Observer Namenode state. Contributed by Ayush Saxena.
2019-11-28 18:28:20 +05:30
c3659f8f94
HDFS-15019. Refactor the unit test of TestDeadNodeDetection. Contributed by Lisheng Sun.
2019-11-28 14:41:49 +08:00
2b452b4e60
HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago.
2019-11-28 10:43:35 +08:00
c8bef4d6a6
HDFS-14649. Add suspect probe for DeadNodeDetector. Contributed by Lisheng Sun.
2019-11-27 10:57:20 +08:00
049940e77b
HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar.
2019-11-23 02:30:02 +05:30
b25e94ce29
HDFS-14924. RenameSnapshot not updating new modification time. Contributed by hemanthboyina
2019-11-22 13:48:02 +09:00
9b6906fe91
HDFS-14651. DeadNodeDetector checks dead node periodically. Contributed by Lisheng Sun.
2019-11-22 10:53:55 +08:00
26270196a2
HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina.
2019-11-22 00:31:25 +05:30
98d249dcda
HDFS-14996. RBF: GetFileStatus fails for directory with EC policy set in case of multiple destinations. Contributed by Ayush Saxena.
2019-11-21 12:08:42 +05:30
fd264b8265
HDFS-14995. Use log variable directly instead of passing as argument in InvalidateBlocks#printBlockDeletionTime(). Contributed by Lisheng Sun.
2019-11-21 00:00:42 +05:30
Ayush Saxena
litao
He Xiaoqiao
touchida
Viraj Jasani
Jack Jiang
Narges Shadab
Takanobu Asanuma
Hui Fei
sunlisheng
lamberken
Xiaoyu Yao
tomscut
litao
zhuqi
Konstantin V Shvachko
S O'Donnell
LeonGao
Wang Yuxuan
Wei-Chiu Chuang
jianghuazhu
Akira Ajisaka
Zehao Chen
Steve Loughran
bshashikant
Jim Brennan
fermi
maobaolong
Kihwal Lee
YaYun-Wang
crossfire
belugabehr
Siyao Meng
Masatake Iwasaki
Ahmed Hussein
lfengnan
maobaolong
hemanthboyina
langlaile1221
liuyanpunk
Neil
Liang-Chi Hsieh
Issac Buenrostro
Yizhong Zhang
Inigo Goiri
Gabor Bota
Mukund Thakur
Uma Maheswara Rao G
Karthik Palanisamy
Fei Hui
Vinayakumar B
Ye Ni
Aryan Gupta
huangtianhua
Mingliang Liu
1996fanrui
zz
Vivek Ratnavel Subramanian
Tsz-Wo Nicholas Sze
Stephen O'Donnell
Chen Liang
Ahmed Hussein
zhaoyim
Shanyu Zhao
Virajith Jalaparti
Toshihiro Suzuki
Wanqiang Ji
leosunli
Abhishek Das
pustota2009
Surendra Singh Lilhore
Elixir Kook
Dhiraj
Brahma Reddy Battula
Chen Liang
Arpit Agarwal
Akshay Nehe
Chao Sun
Rakesh Radhakrishnan
Erik Krogen
Yiqun Lin