c821008836
HDFS-15850. Superuser actions should be reported to external enforcers ( #2784 )
2021-04-15 14:45:51 -07:00
56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) ( #2874 )
2021-04-09 11:10:18 +09:00
4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell ( #2838 )
2021-04-08 11:59:02 +01:00
459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell.
2021-04-06 21:22:48 +08:00
3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. ( #2862 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 13:39:10 +09:00
de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.( #2844 ). Contributed by Viraj Jasani
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 00:19:21 +05:30
96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. ( #2829 ). Contributed by Qi Zhu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-04-04 03:11:02 +05:30
3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker ( #2855 ). Contributed by tomscut.
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-02 10:37:04 -07:00
ed74479ea5
HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree.
2021-04-02 09:47:20 +05:30
7c83f140dc
HDFS-15938. Fix java doc in FSEditLog ( #2837 ). Contributed by tomscut.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-01 19:28:17 -07:00
4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync ( #2770 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-02 10:33:40 +09:00
4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management ( #2830 ). Contributed by Viraj Jasani
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-01 16:34:31 -07:00
0665ce9930
HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree.
2021-04-01 09:19:39 +05:30
5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel.
2021-04-01 11:32:03 +08:00
90fdd04f54
HDFS-15926 : Removed duplicate dependency of hadoop-annotations ( #2823 ). Contributed by Viraj Jasani.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-29 23:22:54 +05:30
cd2501e54b
HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun.
2021-03-28 16:13:59 +05:30
72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks ( #2748 )
...
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-03-27 17:57:09 +09:00
ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.
2021-03-25 12:24:14 +08:00
95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode ( #2585 )
2021-03-24 16:56:09 +08:00
4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result ( #2794 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 22:47:45 -07:00
569e407f64
HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.
2021-03-24 02:10:02 +05:30
9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof ( #2805 )
2021-03-23 08:51:35 -07:00
d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal ( #2790 ). Contributed by Narges Shadab.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 17:06:58 +05:30
343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete ( #2788 )
2021-03-22 20:29:17 +09:00
03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-22 18:37:00 +08:00
261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() ( #2785 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-19 20:37:44 -07:00
f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream ( #2736 ). Contributed by Narges Shadab.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-19 16:36:48 +08:00
9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles ( #2772 )
2021-03-17 15:12:36 +09:00
9ba60c33d8
HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.
2021-03-17 14:14:26 +09:00
25390babc7
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically ( #2726 )
...
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-17 12:00:05 +09:00
0d42104119
HDFS-15898. Test case TestOfflineImageViewer fails ( #2779 )
2021-03-17 09:11:30 +08:00
7025f39944
HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.
2021-03-15 11:34:13 +08:00
970455c917
HDFS-15816. Fix shouldAvoidStaleDataNodesForWrite returns when no stale node in cluster. Contributed by Yang Yun.
2021-03-14 17:48:18 +08:00
6921ec8b40
HDFS-15882. Fix incorrectly initializing RandomAccessFile based on configuration options ( #2751 ). Contributed by Xie Lei.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-13 14:22:11 +08:00
7cf5969f0b
HADOOP-17581. Fix reference to LOG is ambiguous after HADOOP-17482. ( #2764 )
2021-03-12 11:27:53 -08:00
ecd3335187
HDFS-15848. Snapshot Operations: Add debug logs at the entry point. Contributed by Bhavik Patel.
2021-03-12 15:42:15 +09:00
6a55baeee4
HDFS-15875. Check whether file is being truncated before truncate ( #2746 )
2021-03-10 14:11:29 +08:00
6a922f9896
HDFS-15873. Add namenode address in logs for block report ( #2743 )
2021-03-08 11:50:51 -05:00
9cb51bf106
HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. ( #2668 ) Contributed by tomscut.
2021-03-06 13:40:44 -08:00
cdba06e380
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min ( #2739 )
...
Co-authored-by: tomscut <litao@bigo.sg>
2021-03-03 22:34:02 +09:00
1f1a1ef52d
HDFS-15856: Make write pipeline retry times configurable. ( #2721 ). Contributed by Qi Zhu
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-02 13:16:11 +08:00
9501c698f4
HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.
2021-03-01 17:17:23 -08:00
32353eb38a
HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker ( #2718 )
...
Authored-by: tomscut <litao@bigo.sg>
2021-03-01 07:52:59 -08:00
c3b3b36dee
HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell
2021-03-01 11:36:41 +00:00
9c583f36d5
HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee
2021-02-24 10:29:36 +00:00
940c780feb
HDFS-15781. Add metrics for how blocks are moved in replaceBlock. ( #2704 )
2021-02-23 22:29:48 -08:00
647d23b611
HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.
2021-02-23 19:08:38 +05:30
a57282540f
HDFS-15845. RBF: Router fails to start due to NoClassDefFoundError for hadoop-federation-balance. ( #2714 )
2021-02-23 15:12:06 +09:00
1e32ab7d2f
HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
2021-02-22 18:13:36 +08:00
f8059d0bdf
Revert "first commit ( #2559 )"
...
This reverts commit 1b94569d89
2021-02-22 18:13:06 +08:00
1b94569d89
first commit ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
2021-02-22 02:10:39 -08:00
98ffd65d8b
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. ( #2686 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
2021-02-21 23:22:32 -08:00
ff84a57483
HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.
2021-02-20 19:48:00 +05:30
25af8901a9
HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.
2021-02-19 14:44:58 +09:00
2970bd93f3
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable ( #2694 )
2021-02-19 09:07:22 +08:00
e391844e8e
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun.
2021-02-18 01:39:04 +05:30
0ed9e36282
HDFS-15834. Remove the usage of org.apache.log4j.Level ( #2696 )
2021-02-17 01:59:24 -05:00
07a4220cd2
HDFS-15821. Add metrics for in-service datanodes ( #2690 ). Contributed by Zehao Chen.
2021-02-15 11:14:32 -06:00
c3134ab3a9
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics ( #2676 )
2021-02-14 00:12:24 -05:00
798df6d699
HADOOP-13327 Output Stream Specification. ( #2587 )
...
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.
The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.
Contributed by Steve Loughran.
2021-02-10 10:28:59 +00:00
9434c1eccc
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao ( #2679 )
2021-02-09 10:49:28 +00:00
2df2dfb9ed
HDFS-15817. Rename snapshots while marking them deleted. ( #2677 )
2021-02-09 10:27:51 +05:30
0e2b3086e3
HDFS-15683. Allow configuring DISK/ARCHIVE capacity for individual volumes. ( #2625 )
2021-02-08 11:01:21 -08:00
19ae0faacc
HDFS-15820. Ensure snapshot root trash provisioning happens only post safe mode exit ( #2682 )
2021-02-06 08:26:12 -08:00
c4918fb298
[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan
2021-02-05 21:20:30 +00:00
7ef2875c89
HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang
2021-02-05 11:01:49 +08:00
b99dba38b9
HDFS-15819. Fix import style #2681
2021-02-05 10:11:27 +08:00
15a1f7adfc
HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang
2021-02-04 09:19:51 +08:00
182623d2bc
HDFS-15799. Make DisallowedDatanodeException terse. Contributed by Richard Ross.
2021-02-03 08:55:50 -06:00
66ecee333e
HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang
2021-02-03 20:00:45 +08:00
394b9f7a5c
HDFS-15624. fix the function of setting quota by storage type ( #2377 )
...
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied
Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2021-02-02 22:44:34 -08:00
18978f2e20
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka ( #2657 )
2021-02-02 09:02:09 +00:00
1b893e10ae
HDFS-15792. Addendum: ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-02 13:50:54 +08:00
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
b54134661b
HDFS-15792. ClasscastException while loading FSImage. Contributed by Renukaprasad C.
2021-02-01 23:28:29 +08:00
115623a6ee
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. ( #2652 )
2021-02-01 01:56:59 -08:00
1a205cc3ad
HADOOP-17424. Replace HTrace with No-Op tracer ( #2645 )
2021-02-01 13:42:44 +09:00
f565476939
[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.
2021-01-28 18:17:15 +00:00
6184ab07f6
[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.
2021-01-28 16:47:52 +00:00
f8769e0f4b
HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.
2021-01-28 10:10:39 +08:00
45b3a84709
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement ( #2626 )
2021-01-21 10:07:46 +09:00
97f843de3a
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-16 14:06:56 +08:00
7743d40ac5
HDFS-15549. Use Hardlink to move replica between DISK and ARCHIVE storage if on same filesystem mount ( #2583 ). Contributed by Leon Gao.
2021-01-15 16:28:11 -08:00
630f8ddd2c
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. ( #2475 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-14 16:19:35 +09:00
8ec824f2ba
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently ( #2598 )
...
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-13 11:23:07 +09:00
32abc0af49
HDFS-15702. Fix intermittent falilure of TestDecommission#testAllocAndIBRWhileDecommission. ( #2507 )
2021-01-11 11:59:32 -08:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
77435a025e
HADOOP-17408. Optimize NetworkTopology sorting block locations. ( #2601 ). Contributed by Ahmed Hussein and Daryn Sharp.
2021-01-08 13:10:09 -06:00
87bd4d2aca
HDFS-15754. Add DataNode packet metrics ( #2578 )
...
Contributed by Fengnan Li.
2021-01-07 23:46:23 -08:00
66ee0a6df0
HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. ( #2453 )
2021-01-04 09:43:58 -08:00
2825d060cf
HDFS-15648. TestFileChecksum should be parameterized. ( #2501 )
2021-01-04 14:00:57 +09:00
1cd96e8dd8
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang.
2021-01-01 11:37:52 +05:30
3b77cf4446
HDFS-15749. Make size of editPendingQ can be configurable ( #2572 )
2020-12-28 14:18:18 +08:00
16a20503ca
HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.
2020-12-22 17:27:31 +05:30
4ae561bcb4
HDFS-15739. Add missing Javadoc for a param in method chooseRandomWithStorageType ( #2566 )
2020-12-22 09:25:11 +08:00
ba496f61f8
HDFS-15308. TestReconstructStripedFile#testNNSendsErasureCodingTasks fails intermittently. Contributed by Hemanth Boyina.
2020-12-22 02:16:00 +05:30
2aea43bf4f
HDFS-15655. Add option to make balancer prefer to get cold blocks. Contributed by Yang Yun.
2020-12-20 23:06:19 +05:30
3ecd3628d4
HDFS-15733. Add seqno in log when BlockReceiver receive packet. Contributed by Haibin Huang.
2020-12-20 15:18:25 +05:30
70789fc6b9
HDFS-15116. Correct spelling of comments for NNStorage.setRestoreFailedStorage. Contributed by Xudong Cao.
2020-12-20 15:12:32 +05:30
c2672bb234
HDFS-15704. Mitigate lease monitor's rapid infinite loop. ( #2511 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-17 11:33:29 -06:00
0da00fc743
HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-12-14 20:42:23 -08:00
3363cc380b
HDFS-15728. Update description of dfs.datanode.handler.count in hdfs-default.xml. Contributed by liuyan.( #2534 )
2020-12-15 10:06:11 +05:30
9ed737001c
HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell
2020-12-11 20:53:10 +00:00
be35fa186c
HDFS-15717. Improve fsck logging. ( #2529 ) Contributed by Kihwal Lee and Ahmed Hussein
2020-12-11 11:02:42 -06:00
9bd3c9bc50
HDFS-15720 namenode audit async logger should add some log4j config ( #2532 )
2020-12-10 13:47:00 -08:00
01383a2172
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy ( #2528 )
2020-12-08 13:25:24 -06:00
40f7543a6d
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. ( #2518 )
2020-12-07 15:49:45 -08:00
32099e36dd
HDFS-15707. NNTop counts don't add up as expected. ( #2516 )
2020-12-07 14:22:06 -06:00
ad40715690
HDFS-15221. Add checking of effective filesystem during initializing storage locations. Contributed by Yang Yun.
2020-12-07 15:00:19 +05:30
e2c1268ebd
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-04 09:20:09 +08:00
42a29199c0
HDFS-15705. Fix a typo in SecondaryNameNode.java. Contributed by Sixiang Ma.
2020-12-02 21:32:02 -08:00
6ff2409b31
HDFS-14904. Add Option to let Balancer prefer highly utilized nodes in each iteration ( #2483 ). Contributed by Leon Gao.
2020-12-02 15:53:09 -08:00
60201cbf69
HDFS-15703. Don't generate edits for set operations that are no-op ( #2508 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 15:38:20 -06:00
2b5b556dd7
HDFS-15695. NN should not let the balancer run in safemode ( #2489 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 13:59:00 -06:00
918ba9e82f
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. ( #2487 ) Contributed by Kuhu Shukla and Ahmed Hussein
2020-12-01 15:02:53 -06:00
68442b48c8
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 ( #2495 )
2020-11-28 12:19:57 +08:00
235947e282
HDFS-15689. allow/disallowSnapshot on EZ roots shouldn't fail due to trash provisioning/emptiness check ( #2472 )
2020-11-25 11:01:04 -08:00
641d8856d2
HDFS-15684. EC: Call recoverLease on DFSStripedOutputStream close exception. Contributed by Hongbing Wang.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2020-11-23 11:26:52 +08:00
d73029463c
HDFS-15690. Add lz4-java as test dependency ( #2481 )
2020-11-21 17:49:56 -08:00
747883af4d
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false ( #2443 ). Contributed by Ahmed Hussein.
2020-11-20 18:03:45 +05:30
425996eb4a
HDFS-15674. TestBPOfferService#testMissBlocksWhenReregister fails on trunk. ( #2467 )
2020-11-18 16:11:09 +09:00
f56cd88d7d
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
2020-11-13 11:02:26 +09:00
f5e6be337b
HDFS-15545 - Allow WebHdfsFileSystem to read a new delegation token f… ( #2255 )
...
Co-authored-by: Issac Buenrostro <ibuenros@linkedin.com>
2020-11-12 14:57:26 -08:00
e6d2dccbef
HDFS-15469. Dynamically configure the size of PacketReceiver#MAX_PACKET_SIZE. ( #2138 )
2020-11-10 16:34:17 -08:00
82de07c794
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
2020-11-10 09:11:31 -08:00
95c96605b3
HDFS-15667. Audit log record the unexpected allowed result when delete ( #2437 )
2020-11-10 13:01:10 +08:00
9a9ab5b48e
HDFS-15548. Allow configuring DISK/ARCHIVE storage types on same device mount ( #2288 ). Contributed by Leon Gao.
2020-11-09 15:06:16 -08:00
3067a25fa1
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
2020-11-04 13:53:46 +08:00
7ac7840fc3
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:28:44 +05:30
d07dc7afb4
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
2020-11-03 12:02:40 -08:00
324879127a
HDFS-15654. TestBPOfferService#testMissBlocksWhenReregister fails intermittently ( #2419 )
2020-10-28 16:24:34 -07:00
d0c786db4d
HDFS-15460. TestFileCreation#testServerDefaultsWithMinimalCaching fails intermittently. ( #2406 )
2020-10-27 18:56:40 -07:00
98097b8f19
HDFS-15457. TestFsDatasetImpl fails intermittently ( #2407 )
2020-10-27 18:52:56 -07:00
8a6d5b9151
HDFS-15652. Make block size from NNThroughputBenchmark configurable ( #2416 )
2020-10-28 09:13:25 +08:00
ae74407ac4
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky ( #2410 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-10-27 20:45:00 +09:00
3848b44b37
HDFS-15580. [JDK 12] DFSTestUtil#addDataNodeLayoutVersion fails ( #2309 )
2020-10-27 19:18:08 +09:00
afaab3d332
HDFS-15461. TestDFSClientRetries testGetFileChecksum fails ( #2404 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
2020-10-27 16:56:30 +09:00
12c908c827
HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.
2020-10-26 22:12:27 +08:00
74634eb002
HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein.
2020-10-23 12:05:53 -07:00
e7aa4da0a2
HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein.
2020-10-23 09:47:10 -07:00
da1b6e3cc2
HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein.
2020-10-22 16:59:49 -05:00
88a9f42f32
HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein.
2020-10-21 00:59:35 -05:00
2e8cafac3b
HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.
2020-10-19 10:48:47 +05:30
173310e2f5
HDFS-15629. Add seqno when warning slow mirror/disk in BlockReceiver. Contributed by Haibin Huang.
2020-10-19 10:34:17 +05:30
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. ( #2342 ). Contributed by Ayush Saxena.
2020-10-17 12:01:18 +05:30
740a2c4635
HDFS-15627. Log delete audits before collecting blocks.
2020-10-16 11:13:41 -05:00
a308a1ec22
HDFS-15614. Initialize snapshot trash root during NameNode startup if enabled ( #2370 )
2020-10-13 10:59:42 -07:00
59f01a548e
HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same
...
Contributed by Gabor Bota.
2020-10-13 16:17:44 +01:00
0507c4160f
HDFS-15626. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
Change-Id: I6cec41c20467920cf21f169653553535414b2680
2020-10-13 13:31:00 +01:00
d606ddc12c
Revert "HADOOP-17303. TestWebHDFS.testLargeDirectory failing ( #2380 )"
...
(reverting to change the JIRA number)
This reverts commit 3e1b1dfd77
2020-10-13 13:30:38 +01:00
3e1b1dfd77
HADOOP-17303. TestWebHDFS.testLargeDirectory failing ( #2380 )
...
Fixes the regression caused by HADOOP-17281, where the WebHDFS client
listStatusIterator (correctly) throws NoSuchElementException when next()
runs out of values.
Contributed by Mukund Thakur.
2020-10-13 13:27:55 +01:00
b3786d6c3c
HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko.
2020-10-12 17:26:49 -07:00
b76b36ebbc
HDFS-15625: Namenode trashEmptier should not init ViewFs on startup ( #2378 ). Contributed by Uma Maheswara Rao G.
2020-10-12 14:22:44 -07:00
de055ec8aa
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently ( #2299 )
2020-10-10 22:08:36 +05:30
735e85a1d2
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 ( #2365 )
2020-10-08 12:51:04 +05:30
df4006eb81
HDFS-15253 Default checkpoint transfer speed, 50mb per second ( #2366 )
2020-10-07 22:09:04 +05:30
16aea11c94
HDFS-15611. Add list Snapshot command in WebHDFS. ( #2355 )
2020-10-07 10:34:32 +05:30
6ece6407e1
HDFS-15458. TestNameNodeRetryCacheMetrics fails intermittently ( #2356 ). Contributed by Hui Fei.
2020-10-05 22:11:24 +05:30
43b0c0b054
HDFS-15607. Create trash dir when allowing snapshottable dir ( #2352 )
2020-10-05 05:02:00 -07:00
984875c624
HDFS-15604. Fix Typo for HdfsDataNodeAdminGuide doc ( #2345 )
2020-09-28 09:54:01 -07:00
d89af79507
HDFS-15577. Refactor TestTracing. ( #2302 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-09-28 18:27:41 +09:00
82b86e3754
HDFS-15098. Add SM4 encryption method for HDFS. Contributed by liusheng
2020-09-27 19:27:13 +05:30
00c4de63cf
HDFS-15594. Lazy calculate live datanodes in safe mode tip ( #2332 )
2020-09-25 09:47:54 -07:00
486ddb73f9
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 ( #2330 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2020-09-24 21:15:55 +09:00
ff59fbb8b0
HDFS-15025. Applying NVDIMM storage media to HDFS ( #2189 )
...
* HDFS-15025. Applying NVDIMM storage media to HDFS
Co-authored-by: YaYun-Wang <yywangyayun@163.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>
2020-09-24 14:27:04 +05:30
368f2f637e
HDFS-15590. namenode fails to start when ordered snapshot deletion feature is disabled ( #2326 )
2020-09-24 14:00:41 +05:30
ae089f2db7
HDFS-15557. Log the reason why a storage log file can't be deleted ( #2274 )
2020-09-22 13:23:20 -07:00
474fa80bfb
HADOOP-17277. Correct spelling errors for separator ( #2322 )
...
Contributed by Hui Fei.
2020-09-22 17:10:33 +01:00
2377df38ad
HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.
2020-09-18 17:28:11 +05:30
7bba4c609c
HDFS-15585: ViewDFS#getDelegationToken should not throw UnsupportedOperationException. ( #2312 ). Contributed by Uma Maheswara Rao G.
2020-09-18 15:18:10 +05:30
20a0e6278d
HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell
2020-09-17 14:11:42 +01:00
425f48799c
HDFS-15568. namenode start failed to start when dfs.namenode.max.snapshot.limit set. ( #2296 )
2020-09-17 14:50:08 +05:30
e4cb0d3514
HDFS-15578: Fix the rename issues with fallback fs enabled ( #2305 ). Contributed by Uma Maheswara Rao G.
...
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
2020-09-16 22:43:00 -07:00
5c5b2ed7c7
HDFS-15576. Erasure Coding: Add rs and rs-legacy codec test for addPolicies. Contributed by Fei Hui.
2020-09-16 01:36:46 +09:00
f4ed9f3f91
HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell.
2020-09-15 10:10:21 +05:30
924959088e
HDFS-15559. Complement initialize member variables in TestHdfsConfigFields#initializeMemberVariables. Contributed by Lisheng Sun.
2020-09-15 12:05:04 +08:00
6d3312de47
HDFS-15539. When disallowing snapshot on a dir, throw exception if its trash root is not empty ( #2258 )
2020-09-14 13:31:34 -07:00
d2779de3f5
HDFS-15532: listFiles on root/InternalDir will fail if fallback root has file. ( #2298 ). Contributed by Uma Maheswara Rao G.
2020-09-12 17:06:39 -07:00
f59f7f2175
HDFS-15573. Only log warning if considerLoad and considerStorageType are both true. Contributed by Stephen O'Donnell
2020-09-12 01:41:38 -07:00
2a6fdcedeb
HDFS-15564. Add Test annotation for TestPersistBlocks#testRestartDfsWithSync. Contributed by Fei Hui.
2020-09-11 18:21:25 +05:30
56ebabd426
HADOOP-17222. Create socket address leveraging URI cache ( #2241 )
...
Contributed by fanrui.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2020-09-10 22:30:52 -07:00
12a316cdf9
HADOOP-15891. provide Regex Based Mount Point In Inode Tree ( #2185 ). Contributed by Zhenzhao Wang.
...
Co-authored-by: Zhenzhao Wang <zhenzhaowang@gmail.com>
2020-09-10 21:20:32 -07:00
43c52d6495
HDFS-15563. Incorrect getTrashRoot return value when a non-snapshottable dir prefix matches the path of a snapshottable dir ( #2295 )
...
Change-Id: Iefcbc14ca939b61f0c1befc9a94d686be1684ddc
2020-09-10 15:55:23 +05:30
1d6d0d8207
HDFS-14694. Call recoverLease on DFSOutputStream close exception. Contributed by Lisheng Sun.
...
Co-authored-by: Chen Zhang <chzhang1987@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2020-09-09 20:09:13 +08:00
ac7d4623ae
HDFS-15558: ViewDistributedFileSystem#recoverLease should call super.recoverLease when there are no mounts configured ( #2275 ) Contributed by Uma Maheswara Rao G.
2020-09-07 11:36:13 -07:00
b3660d0147
HDFS-15529: getChildFilesystems should include fallback fs as well ( #2234 ). Contributed by Uma Maheswara Rao G.
2020-09-03 11:06:20 -07:00
60de592a88
HDFS-15550. Remove unused imports from TestFileTruncate.java. Contributed by Ravuri Sushma sree.
2020-08-31 14:49:12 +08:00
2d03209a06
HDFS-15542. Add identified snapshot corruption tests for ordered snapshot deletion ( #2251 )
2020-08-31 11:29:48 +05:30
c392d9022a
HDFS-15471. TestHDFSContractMultipartUploader failing ( #2252 )
...
Contributed by Steve Loughran
(Was: broken by Steve Loughran)
2020-08-28 15:39:09 +01:00
06793da100
HDFS-15531. Namenode UI: List snapshots in separate table for each snapshottable directory ( #2230 )
2020-08-27 11:36:30 -07:00
41182a9b6d
HDFS-15500. In-order deletion of snapshots: Diff lists must be update only in the last snapshot. ( #2233 )
2020-08-27 14:54:52 +05:30
2ffe00fc46
HDFS-15540. Directories protected from delete can still be moved to the trash. Contributed by Stephen O'Donnell.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-08-26 23:05:32 -07:00
6a49bf9bff
HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui.
2020-08-25 15:18:36 +01:00
a7830423c5
HDFS-15448. Remove duplicate BlockPoolManager starting when run DataNode. Contriubted by jianghua zhu.
2020-08-24 21:16:00 +08:00
83fd15b412
HDFS-14504. Rename with Snapshots does not honor quota limit.
2020-08-23 16:10:47 +05:30
f734455e5d
HDFS-15290. NPE in HttpServer during NameNode startup. Contributed by Simbarashe Dzinamarira.
2020-08-20 15:25:10 -07:00
dd013f2fdf
HDFS-15533: Provide DFS API compatible class, but use ViewFileSystemOverloadScheme inside. ( #2229 ). Contributed by Uma Maheswara Rao G.
2020-08-19 09:30:41 -07:00
b65e43fe38
HDFS-15525. Make trash root inside each snapshottable directory for WebHDFS ( #2220 )
2020-08-18 19:28:19 +09:00
092bfe7c8e
HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories. ( #2172 )
...
* HDFS-15483. Ordered snapshot deletion: Disallow rename between two snapshottable directories.
* Addressed review comments.
* Rebased to latest trunk and added snapshotTrashRoot config check.
* Addressed review comments.
* Removede empty line added in SnapshotManager.Java.
* Addressed whitespace issues.
2020-08-17 12:56:13 +05:30
e3d1966f58
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
2020-08-15 15:07:58 +05:30
Vivek Ratnavel Subramanian
Viraj Jasani
Stephen O'Donnell
He Xiaoqiao
zhuqi
litao
Brahma Reddy Battula
Ayush Saxena
touchida
Jack Jiang
Narges Shadab
Takanobu Asanuma
Hui Fei
sunlisheng
lamberken
Xiaoyu Yao
tomscut
litao
Konstantin V Shvachko
S O'Donnell
LeonGao
Wang Yuxuan
Wei-Chiu Chuang
jianghuazhu
Akira Ajisaka
Zehao Chen
Steve Loughran
bshashikant
Jim Brennan
fermi
maobaolong
Kihwal Lee
YaYun-Wang
crossfire
belugabehr
Siyao Meng
Masatake Iwasaki
Ahmed Hussein
lfengnan
maobaolong
hemanthboyina
langlaile1221
liuyanpunk
Neil
Liang-Chi Hsieh
Issac Buenrostro
Yizhong Zhang
Inigo Goiri
Gabor Bota
Mukund Thakur
Uma Maheswara Rao G
Karthik Palanisamy
Fei Hui
Vinayakumar B
Ye Ni
Aryan Gupta
huangtianhua
Mingliang Liu
1996fanrui
zz
Tsz-Wo Nicholas Sze
Stephen O'Donnell
Chen Liang