705884b87a
HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma
...
(cherry picked from commit b6c06c4b76
2021-07-09 10:19:47 +01:00
ab939814a3
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. ( #3188 )
...
(cherry picked from commit 6450c1ea9a
2021-07-09 09:17:10 +01:00
2d56335a3d
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer ( #3169 )
...
(cherry picked from commit da575a53cf
2021-07-05 13:42:51 +09:00
aec8c9188a
HDFS-16109. Fix flaky some unit tests since they offen timeout ( #3172 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b87bac13e4
2021-07-05 08:15:10 +09:00
bf6ed21104
HDFS-16090. Fine grained lock for datanodeNetworkCounts ( #3148 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack Chuang <stack@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-30 12:59:25 -07:00
4ee7bd345f
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor ( #3146 )
...
(cherry picked from commit 95454d821c
2021-06-29 18:15:56 +08:00
8538881710
HDFS-16092. Avoid creating LayoutFlags redundant objects ( #3150 )
...
(cherry picked from commit 93a1685073
2021-06-29 18:13:00 +08:00
4ca49fcd44
HDFS-16085. Move the getPermissionChecker out of the read lock ( #3134 ). Contributed by tomscut.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-24 16:00:47 +05:30
5fc61d2e56
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations ( #3117 )
...
(cherry picked from commit fdef2b4cca
2021-06-24 13:27:33 +09:00
68604b4cd0
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer ( #3127 )
...
(cherry picked from commit d9fbb3c508
2021-06-23 11:24:00 +08:00
b8a98e4f82
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects ( #3115 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit c488abbc79
2021-06-21 10:28:05 +09:00
7d202e8365
HDFS-16079. Improve the block state change log ( #3120 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:24:03 +05:30
ef3a2a3b3b
HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result ( #3121 ). Contributed by Viraj Jasani.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 17:23:52 +05:30
92ade1f6f9
HDFS-16061. DFTestUtil.waitReplication can produce false positives ( #3095 ). Contributed by Ahmed Hussein.
...
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:26:49 +05:30
8d241482bd
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() ( #3119 ). Contributed by tomscut.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-06-20 10:26:38 +05:30
e55d76e26c
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet ( #3113 )
...
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
2021-06-18 14:41:02 +08:00
577b96ef42
HDFS-16074. Remove an expensive debug string concatenation ( #3107 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e548
2021-06-17 11:39:34 +08:00
3e03ecd680
HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w.
2021-06-16 15:54:36 +05:30
546c66b812
HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS ( #3104 )
...
(cherry picked from commit 839fcf7682
2021-06-15 17:22:34 +09:00
840612b3b1
HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false ( #2443 ). Contributed by Ahmed Hussein.
...
(cherry picked from commit 747883af4d
2021-06-14 14:37:27 +00:00
52d055a7eb
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state ( #3084 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 51ebf7b2a0
2021-06-11 17:33:06 +09:00
83d119fe94
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. ( #2863 ). Contributed by Ayush Saxena. ( #3054 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c6539e3289
2021-06-08 19:32:34 -07:00
02249171b1
HDFS-16048. RBF: Print network topology on the router web ( #3062 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c748fce17a
2021-06-08 16:59:59 +09:00
46d4b51bff
HDFS-15814. Make some parameters configurable for DataNodeDiskMetrics for branch-3.3 ( #3021 )
2021-06-08 00:10:37 -07:00
ede03cc35c
HDFS-15998. Fix NullPointException In listOpenFiles ( #3036 )
...
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
(cherry picked from commit b38b00e528
2021-06-01 15:30:50 +08:00
c5535caf6e
HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira.
...
(cherry picked from commit 8ce30f51f9
2021-05-27 18:53:46 -07:00
5308d44be4
HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit 1abd03d68f
2021-05-26 15:09:47 -07:00
41025cd585
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap ( #3027 )
...
(cherry picked from commit 3fdeb7435a
2021-05-26 14:45:20 +09:00
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )"
...
This reverts commit 4ffe5eb1dd
2021-05-24 17:37:18 +08:00
ca52d9946b
HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu.
...
(cherry picked from commit da1ea2530f
2021-05-24 10:48:48 +08:00
fa4915fdbb
Preparing for 3.3.2 development
2021-05-19 21:52:37 +08:00
63392376c9
HDFS-16027. Replace abstract methods with default methods in JournalNodeMXBean. ( #3018 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-05-18 19:49:00 -07:00
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )
...
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb
* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661 ). Contributed by Vinayakumar B.
(cherry picked from commit c4c0683dff
2021-05-17 18:57:46 -07:00
2f56e2b54d
HDFS-15873. Add namenode address in logs for block report ( #2743 )
...
(cherry picked from commit 6a922f9896
2021-05-17 12:29:11 +08:00
28079e9c30
HADOOP-17283. Hadoop - Upgrade to jQuery 3.5.1 ( #2330 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 486ddb73f9
2021-05-13 12:16:17 +08:00
41b39ff292
HDFS-15245. Improve JournalNode web UI. Contributed by Jianfei Jiang.
...
(cherry picked from commit 960c9ebaea
2021-05-13 12:14:09 +08:00
3176ab028b
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException ( #2982 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b944084b32
2021-05-11 12:39:42 +09:00
43db94c7a3
HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w.
2021-05-10 17:11:09 +05:30
1326cad66a
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditsLogOpCodes (branch-3.3) ( #2989 ). Contributed by Akira Ajisaka.
2021-05-08 10:48:56 +08:00
00c3f3c42f
HDFS-15865. Interrupt DataStreamer thread if no ack ( #2728 )
...
(cherry picked from commit bd3da73a0f
2021-05-04 11:11:36 +01:00
25244c4eec
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. ( #2970 )
...
(cherry picked from commit 7c561e2a02
2021-05-04 00:26:34 +09:00
43fac739bb
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds ( #2910 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e525ab81c
2021-05-02 19:19:55 +09:00
670205c541
HADOOP-17653. Do not use guava's Files.createTempDir(). ( #2945 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit f1e1809029
2021-05-02 11:12:37 +09:00
9aa6106689
HDFS-15652. Make block size from NNThroughputBenchmark configurable ( #2416 )
...
(cherry picked from commit 8a6d5b9151
2021-04-29 12:10:04 -07:00
0e9042fed5
HDFS-15991. Add location into datanode info for NameNodeMXBean ( #2933 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit b968fa0957
2021-04-28 18:52:05 +09:00
842e05ba53
HDFS-15698. Fix the typo of dfshealth.html after HDFS-15358 ( #2495 )
...
(cherry picked from commit 68442b48c8
2021-04-28 18:51:50 +09:00
f809b863fe
HDFS-15358. RBF: Unify router datanode UI with namenode datanode UI. Contributed by Ayush Saxena.
...
(cherry picked from commit 6e416a83d1
2021-04-28 18:51:43 +09:00
3c7433f2a1
HDFS-15247. RBF: Provide Non DFS Used per DataNode in DataNode UI. Contributed by Lisheng Sun.
...
(cherry picked from commit 8d49229c37
2021-04-28 18:51:37 +09:00
187b4a63a4
HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina.
...
(cherry picked from commit 8db38c98a6
2021-04-27 23:18:12 +09:00
f45365f201
HDFS-15975. Use LongAdder instead of AtomicLong for branch-3.3 ( #2940 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-27 22:39:25 +09:00
f6efb58b07
HDFS-15621. Datanode DirectoryScanner uses excessive memory ( #2849 ). Contributed by Stephen O'Donnell
...
(cherry picked from commit 605ed85c29
2021-04-26 11:23:56 +01:00
a5f038b3f6
HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel.
...
(cherry picked from commit 01bad0e92a
2021-04-26 13:31:51 +09:00
99dfd3b2d0
HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2621d3f15b
2021-04-26 11:40:16 +08:00
96a1dfa313
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() ( #2942 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-23 12:57:23 +09:00
2752156e3e
HDFS-15974. RBF: Unable to display the datanode UI of the router ( #2915 )
...
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit dff95c5eca
2021-04-23 11:22:52 +09:00
9f2db2c9fd
HDFS-15963. Unreleased volume references cause an infinite loop. ( #2941 ) Contributed by Shuyan Zhang.
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-04-21 18:32:56 +08:00
29414871bd
HDFS-15879. Exclude slow nodes when choose targets for blocks ( #2928 )
2021-04-21 10:41:13 +09:00
90f63cb12c
HDFS-15569. Speed up the Storage#doRecover during datanode rolling upgrade. Contributed by Hemanth Boyina.
...
(cherry picked from commit 16a20503ca
2021-04-20 16:21:57 +01:00
c93a2f6789
HDFS-14999. Avoid Potential Infinite Loop in DFSNetworkTopology. Contributed by Ayush Saxena.
...
(cherry picked from commit c84e6beada
2021-04-20 12:13:39 +01:00
dc830cf277
HDFS-15970. Print network topology on the web ( #2896 )
...
(cherry picked from commit 85a3532849
2021-04-19 21:22:47 +09:00
4c0b833bab
HDFS-15898. Test case TestOfflineImageViewer fails ( #2779 )
...
(cherry picked from commit 0d42104119
2021-04-16 14:07:15 +09:00
a3405cb56a
HADOOP-17569. Building native code fails on Fedora 33. ( #2886 )
...
(cherry picked from commit 2717203f85
2021-04-15 12:39:07 +00:00
bfba6f1f3c
HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. ( #2882 )
...
(cherry picked from commit e391844e8e
2021-04-12 23:55:32 -07:00
0e6cd352b5
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. ( #2813 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0
2021-04-09 10:25:14 +01:00
4994b73eeb
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) ( #2874 )
...
(cherry picked from commit 56bd968fb4
2021-04-09 11:13:39 +09:00
df99ac0399
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell.
...
(cherry picked from commit 459df41997
2021-04-08 13:15:37 +01:00
ef95f7a963
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell ( #2838 )
...
(cherry picked from commit 4c567fcff7
2021-04-08 12:15:31 +01:00
dca2bf9dd5
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode ( #2585 )
...
(cherry picked from commit 95e6892675
2021-04-08 17:20:08 +08:00
d7b31acbba
HDFS-15949. Fix integer overflow ( #2857 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 36014b8282
2021-04-06 18:23:38 +09:00
8b4b3d6fe6
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. ( #2862 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 3f2682b92b
2021-04-06 14:18:31 +09:00
2b5fd341b9
HDFS-15940. Fixing and refactoring tests specific to Block recovery.( #2844 ). Contributed by Viraj Jasani
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-04-06 00:20:56 +05:30
7a7e194228
HDFS-15951. Remove unused parameters in NameNodeProxiesClient ( #2859 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9e2d5d6d05
2021-04-05 17:49:27 +09:00
e4d04a640b
HDFS-15950. Remove unused hdfs.proto import ( #2858 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca5e13fd74
2021-04-05 11:42:52 +09:00
d973f37054
HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks. Contributed by hemanthboyina.
...
(cherry picked from commit 2148a8fe64
2021-04-02 17:14:34 +08:00
887948d127
HDFS-15351. Blocks scheduled count was wrong on truncate. Contributed by hemanthboyina.
...
(cherry picked from commit 719b53a79d
2021-04-02 17:14:24 +08:00
02d37c7d22
HDFS-15316. Deletion failure should not remove directory from snapshottables. Contributed by hemanthboyina
...
(cherry picked from commit 743c2e9071
2021-04-02 17:14:14 +08:00
14fddba715
HDFS-15252. HttpFS: setWorkingDirectory should not accept invalid paths. Contributed by hemanthboyina.
...
(cherry picked from commit 736659e0e1
2021-04-02 17:13:59 +08:00
2da3356428
HDFS-15591. RBF: Fix webHdfs file display error. Contributed by wangzhaohui.
...
(cherry picked from commit fc8a6dd8a9
2021-04-02 17:13:00 +08:00
f5cc1540b4
HDFS-15510. RBF: Quota and Content Summary was not correct in Multiple Destinations. Contributed by Hemanth Boyina.
...
(cherry picked from commit ca8e7a7725
2021-04-02 17:11:33 +08:00
a549b4a82e
HDFS-15300. RBF: updateActiveNamenode() is invalid when RPC address is IP. Contributed by xuzq.
...
(cherry picked from commit 936bf09c37
2021-04-02 17:11:16 +08:00
5bfc3a4c3a
HDFS-15198. RBF: Add test for MountTableRefresherService failed to refresh other router MountTableEntries in secure mode. Contributed by zhengchenyu.
...
(cherry picked from commit 8a9a674ef1
2021-04-02 17:10:56 +08:00
327488d688
HDFS-15196. RBF: RouterRpcServer getListing cannot list large dirs correctly. Contributed by Fengnan Li.
...
(cherry picked from commit 80b877a72f
2021-04-02 14:12:17 +08:00
57d2fff84b
HDFS-15246. ArrayIndexOfboundsException in BlockManager CreateLocatedBlock. Contributed by Hemanth Boyina.
...
(cherry picked from commit 8b7695bb26
2021-04-02 14:06:52 +08:00
1148d3895f
HDFS-15332. Quota Space consumed was wrong in truncate with Snapshots. Contributed by hemanthboyina.
...
(cherry picked from commit 130f89e068
2021-04-02 14:06:52 +08:00
41fad5056c
HDFS-15265. HttpFS: validate content-type in HttpFSUtils. Contributed by hemanthboyina.
...
(cherry picked from commit 31b2f687ef
2021-04-02 14:06:52 +08:00
5187bd37ae
HDFS-15266. Add missing DFSOps Statistics in WebHDFS. Contributed by Ayush Saxena.
...
(cherry picked from commit 37d6582223
2021-04-02 14:06:52 +08:00
2b207ea402
HDFS-15667. Audit log record the unexpected allowed result when delete ( #2437 )
...
(cherry picked from commit 95c96605b3
2021-04-02 14:06:52 +08:00
62937d15bb
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync ( #2770 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6
2021-04-02 10:57:05 +09:00
f707d8407a
HDFS-15931 : Fix non-static inner classes for better memory management ( #2830 ). Contributed by Viraj Jasani
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-01 16:35:44 -07:00
e5d7b22b8b
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel.
2021-04-01 11:59:53 +08:00
13878fc06b
HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree.
...
(cherry picked from commit 0665ce9930
2021-04-01 09:22:18 +05:30
c365149e16
HADOOP-16524. Automatic keystore reloading for HttpServer2
...
Reapply of issue reverted first because it caused yarn failures.
Signed-off-by: stack <stack@apache.org>
2021-03-31 10:50:28 -07:00
ab451b8e42
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa1
2021-03-30 22:29:49 +08:00
53b024a2b7
HDFS-15508. [JDK 11] Fix javadoc errors in hadoop-hdfs-rbf module
...
(cherry picked from commit 9062814bab
2021-03-30 22:29:49 +08:00
a407f7de5a
HDFS-15507. [JDK 11] Fix javadoc errors in hadoop-hdfs-client module. Contributed by Xieming Li.
...
(cherry picked from commit 32895f4f7e
2021-03-30 22:29:49 +08:00
c4f7b8578b
HDFS-15506. [JDK 11] Fix javadoc errors in hadoop-hdfs module. Contributed by Xieming Li.
...
(cherry picked from commit 7938ebfb9d
2021-03-30 22:29:49 +08:00
769df1e964
HDFS-15253 Default checkpoint transfer speed, 50mb per second ( #2366 )
...
(cherry picked from commit df4006eb81
2021-03-30 22:29:49 +08:00
7f20fad419
HDFS-14383. Compute datanode load based on StoragePolicy. Contributed by Ayush Saxena.
...
(cherry picked from commit 2e8cafac3b
2021-03-30 22:29:48 +08:00
56ef16468a
HADOOP-17222. Create socket address leveraging URI cache ( #2817 )
...
Contributed by fanrui.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-03-30 11:59:44 +01:00
d6aaf516d9
HDFS-15926 : Removed duplicate dependency of hadoop-annotations ( #2823 ). Contributed by Viraj Jasani.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-29 23:25:14 +05:30
db8af483d1
HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. ( #2787 )
...
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
(cherry picked from commit ea6595d3b6
2021-03-29 11:47:16 +09:00
b3e968d21e
HDFS-13639. SlotReleaser is not fast enough ( #1885 )
...
(cherry picked from commit be374faf42
2021-03-25 13:25:17 +00:00
bf018e7237
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell.
2021-03-25 12:25:22 +08:00
4db5643ff9
HDFS-15456. TestExternalStoragePolicySatisfier fails intermittently ( #2299 )
...
(cherry picked from commit de055ec8aa
2021-03-25 05:09:11 +08:00
9a0c5d0ca2
HDFS-15275. HttpFS: Response of Create was not correct with noredirect and data are true. Contributed by hemanthboyina.
...
(cherry picked from commit 1fdfaebd98
2021-03-24 17:44:29 +08:00
813aafcee2
HDFS-12288. Fix DataNode's xceiver count calculation. Contributed by Lisheng Sun.
...
(cherry picked from commit 6e04b00df1
2021-03-24 17:44:29 +08:00
e4217be749
HDFS-15380. RBF: Could not fetch real remote IP in RouterWebHdfsMethods. Contributed by tomscut.
...
(cherry picked from commit 5ab1db9e21
2021-03-24 17:44:29 +08:00
580634882e
HDFS-14546. Document block placement policies. Contributed by Amithsha.
...
(cherry picked from commit 17ffcab5f6
2021-03-24 17:44:29 +08:00
c96821ece8
HDFS-15610 Reduced datanode upgrade/hardlink thread from 12 to 6 ( #2365 )
...
(cherry picked from commit 735e85a1d2
2021-03-24 17:44:29 +08:00
9e1e89b1b9
HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel.
...
(cherry picked from commit b4d97a8dc7
2021-03-24 17:53:19 +09:00
a8c0083b1b
HDFS-15809. DeadNodeDetector does not remove live nodes from dead node set. Contributed by Jinglun.
...
(cherry picked from commit 7025f39944
2021-03-24 14:04:47 +08:00
94766fdb13
HDFS-15806. DeadNodeDetector should close all the threads when it is closed. Contributed by Jinglun.
...
(cherry picked from commit ff84a57483
2021-03-24 14:04:47 +08:00
4f79df4da1
HDFS-15551. Tiny Improve for DeadNode detector ( #2265 )
...
Contributed by imbajin.
Reviewed-by: leosunli <lisheng.sun08@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 89428f142f
2021-03-24 14:04:47 +08:00
3ecd02ac78
HDFS-15661. The DeadNodeDetector should not be shared by different DFSClients. Contributed by Jinglun.
...
(cherry picked from commit f8769e0f4b
2021-03-24 14:04:47 +08:00
f07bde90c9
HDFS-15791. Possible Resource Leak in FSImageFormatProtobuf. ( #2652 )
...
(cherry picked from commit 115623a6ee
2021-03-24 14:04:47 +08:00
3a355282c9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream ( #2736 ). Contributed by Narges Shadab.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit f5f3fc6fe9
2021-03-24 14:04:47 +08:00
4abe6e6ac1
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal ( #2790 ). Contributed by Narges Shadab.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit d77c7ab4e0
2021-03-24 14:04:47 +08:00
a216ee3b85
HDFS-15910. Improve security with explicit_bzero ( #2793 )
...
(cherry picked from commit 714427c414
2021-03-24 14:04:47 +08:00
e9b06d942c
HDFS-15249 ThrottledAsyncChecker is not thread-safe. ( #1922 )
...
(cherry picked from commit c12ddbd1de
2021-03-24 14:04:47 +08:00
6d3f5e844b
HDFS-15911 : Provide blocks moved count in Balancer iteration result ( #2796 )
...
Contributed by Viraj Jasani.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-03-23 22:51:21 -07:00
76c40a52d7
HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell.
2021-03-24 02:13:52 +05:30
b26d75ce37
HDFS-15093. RENAME.TO_TRASH is ignored When RENAME.OVERWRITE is specified. Contributed by Ayush Saxena.
...
(cherry picked from commit e0ae232f66
2021-03-23 15:34:48 +00:00
968c95bfba
HDFS-10792. RedundantEditLogInputStream should log caught exceptions. Contributed by Wei-Chiu Chuang.
...
(cherry picked from commit ae13a5ccbe
2021-03-23 15:34:36 +00:00
d8314fffc0
HDFS-15443. Setting dfs.datanode.max.transfer.threads to a very small value can cause strange failure. Contributed by AMC-team.
...
(cherry picked from commit c2a17659d1
2021-03-22 12:38:09 +00:00
793ab9acf3
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
...
(cherry picked from commit e3d1966f58
2021-03-22 12:35:11 +00:00
bc2bdee76c
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete ( #2788 )
...
(cherry picked from commit 343ce8a8c9
2021-03-22 20:30:53 +09:00
0b43c65816
HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.
...
(cherry picked from commit 2377df38ad
2021-03-19 18:50:31 +00:00
9665a0be73
HDFS-15633. Avoid redundant RPC calls for getDiskStatus. ( #2386 ). Contributed by Ayush Saxena.
2021-03-19 23:14:18 +05:30
17307ee88d
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles ( #2772 )
...
(cherry picked from commit 9c43b60348
2021-03-17 15:13:27 +09:00
aec9d2cff7
HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.
...
(cherry picked from commit 9ba60c33d8
2021-03-17 14:15:53 +09:00
f4a9327002
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically ( #2726 )
...
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7
2021-03-17 12:01:15 +09:00
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1
2021-03-11 14:57:03 +09:00
34f7562abe
HDFS-15875. Check whether file is being truncated before truncate ( #2746 )
...
(cherry picked from commit 6a55baeee4
2021-03-11 10:07:48 +08:00
c498ef4b6d
HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. ( #2668 ) Contributed by tomscut.
...
(cherry picked from commit 9cb51bf106
2021-03-06 14:00:25 -08:00
852adff194
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min ( #2739 )
...
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380
2021-03-03 22:35:45 +09:00
56679e83bf
HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.
...
(cherry picked from commit 9501c698f4
2021-03-01 17:20:40 -08:00
24a0304159
HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell
...
(cherry picked from commit c3b3b36dee
2021-03-01 11:50:23 +00:00
a6f86af39f
HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore.
...
(cherry picked from commit db6252b6c3
2021-02-26 10:49:16 +08:00
e3ef326663
HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee
...
(cherry picked from commit 9c583f36d5
2021-02-24 11:04:52 +00:00
6ff5da931f
HDFS-15847. create client protocol: add ecPolicyName & storagePolicy param to debug statement string. Contributed by Bhavik Patel.
2021-02-23 19:12:42 +05:30
52a46f064a
HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f
2021-02-22 18:14:18 +08:00
ee0e32abdc
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. ( #2686 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b
2021-02-22 15:24:37 +08:00
1125e3f0c0
HDFS-15835. Erasure coding: Add/remove logs for the better readability/debugging. Contributed by Bhavik Patel.
...
(cherry picked from commit 25af8901a9
2021-02-19 14:47:16 +09:00
a9d3500894
HDFS-15830. Support to make dfs.image.parallel.load reconfigurable ( #2694 )
2021-02-19 09:14:14 +08:00
e45149bf70
HDFS-15836. RBF: Fix contract tests after HADOOP-13327 ( #2702 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 2b3c5b1733
2021-02-16 14:52:31 +09:00
1768c0096f
HDFS-15821. Add metrics for in-service datanodes ( #2690 ). Contributed by Zehao Chen.
...
(cherry picked from commit 07a4220cd2
2021-02-15 17:32:14 +00:00
98e4d516ea
HADOOP-13327 Output Stream Specification. ( #2587 )
...
This defines what output streams and especially those which implement
Syncable are meant to do, and documents where implementations (HDFS; S3)
don't. With tests.
The file:// FileSystem now supports Syncable if an application calls
FileSystem.setWriteChecksum(false) before creating a file -checksumming
and Syncable.hsync() are incompatible.
Contributed by Steve Loughran.
Change-Id: I892d768de6268f4dd6f175b3fe3b7e5bcaa91194
2021-02-10 10:31:22 +00:00
62389a5a04
[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan
2021-02-05 22:16:36 +00:00
ae80fc2477
HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang
...
(cherry picked from commit 7ef2875c89
2021-02-05 11:05:18 +08:00
164a4e63e0
HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang
...
(cherry picked from commit 15a1f7adfc
2021-02-04 09:58:53 +08:00
S O'Donnell
bshashikant
Viraj Jasani
litao
Ahmed Hussein
Hui Fei
Wei-Chiu Chuang
Ayush Saxena
Takanobu Asanuma
huhaiyang
Konstantin V Shvachko
Narges Shadab
Yiqun Lin
Aryan Gupta
Akira Ajisaka
Karthik Palanisamy
lfengnan
Stephen O'Donnell
Brahma Reddy Battula
zhuxiangyi
zhangshuyan0
hemanthboyina
Masatake Iwasaki
He Xiaoqiao
touchida
Gautham B A
Inigo Goiri
Surendra Singh Lilhore
maobaolong
Borislav Iordanov
Harunobu Daikoku
leosunli
LeonGao91
sunlisheng
imbajin
Toshihiro Suzuki
tomscut
litao
Wang Yuxuan
jianghuazhu
Zehao Chen
Steve Loughran
Jim Brennan
fermi