Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,124 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8237 Commits

Author SHA1 Message Date
Kihwal Lee 7034482222 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4)
 2020-01-29 11:01:12 -06:00
Chen Liang 9813ba2ebc [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Contributed by Chen Liang 2020-01-28 16:36:27 -08:00
Ayush Saxena 8a1b02f44c HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:12:44 +05:30
Kihwal Lee 1bb9667137 HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c9)
 2020-01-24 09:21:09 -06:00
Kihwal Lee 4842708320 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 
(cherry picked from commit bd03053ea2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java
 2020-01-22 09:26:47 -06:00
Akira Ajisaka d4f75e2798
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf40)
 2020-01-21 18:03:56 +09:00
Wei-Chiu Chuang aa3c96474f HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu. 
(cherry picked from commit 6b86a5110e)
 2020-01-13 10:49:26 -08:00
Ayush Saxena d3c4c8be6a HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:34:01 +05:30
Chen Liang 1ee90d6731 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 
(cherry picked from commit 26a969ec73)
 2020-01-10 17:17:11 -08:00
Kihwal Lee 59260a0e0e HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein. 
(cherry picked from commit 5fb901ac40)
 2020-01-10 12:46:49 -06:00
Surendra Singh Lilhore 6304f94bc0 HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:07:23 +05:30
Rakesh Radhakrishnan d7f9a90e2c HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He. 
(cherry picked from commit 7030722e5d)
 2020-01-08 14:29:05 +05:30
Masatake Iwasaki 3e17b471b6 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1)
 2020-01-08 17:05:20 +09:00
Masatake Iwasaki 3e32f5476a
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 
(cherry picked from commit a43c177f1d)
 2020-01-08 11:29:23 +09:00
Rakesh Radhakrishnan a27f9bd28e HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 
(cherry picked from commit d79cce20ab)
 2020-01-02 11:57:02 +05:30
Surendra Singh Lilhore 5d9cbca31e HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:21:33 +05:30
Takanobu Asanuma ad11b25df2 HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 
(cherry picked from commit dc32f583af)
 2019-12-28 21:34:19 +09:00
Chandra Sanivarapu 0d94e0b1a0
HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 
(cherry picked from commit d8cd7098b4)
 2019-12-25 11:12:03 +09:00
Konstantin V Shvachko 82957562ea HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b98ac2a3af)
 2019-12-24 12:50:57 -08:00
Eric Badger 11c5396b4a Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui." 
This reverts commit 59cdf18d5a.
 2019-12-20 19:04:45 +00:00
Inigo Goiri 59cdf18d5a HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 
(cherry picked from commit 52d7b745c6)
 2019-12-19 09:43:40 -08:00
Shashikant Banerjee afe00a1ca5 HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 
(cherry picked from commit fdd96e46d1)
 2019-12-18 23:10:00 +05:30
Chen Liang d4a6901c42 HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 
(cherry picked from commit 65c4660bcd)
 2019-12-12 15:57:15 -08:00
He Xiaoqiao 8ba4e13dcc HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
 2019-12-11 16:34:28 -08:00
Erik Krogen 3402c87353 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 
(cherry picked from c174d50b30)
 2019-12-11 09:40:36 -08:00
Surendra Singh Lilhore 298cda22a3 HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-28 23:58:46 +05:30
Ayush Saxena 32fdb3004a HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-11-23 02:36:38 +05:30
Surendra Singh Lilhore 39366b1370 HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina. 2019-11-22 00:33:42 +05:30
He Xiaoqiao 5fe9b81941 HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
(cherry picked from commit 0b50aa29fd)
 2019-11-19 12:22:30 -08:00
Ayush Saxena 051dae1f89 HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 
(cherry picked from commit 9e287054a8)
 2019-11-18 10:24:53 -08:00
Erik Krogen 60655bfe54 HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a)
 2019-11-15 10:11:56 -08:00
Erik Krogen dec765b329 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe711)
 2019-11-13 11:57:42 -08:00
Chao Sun 0c12d633b9 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 
(cherry picked from commit 97ec34e117)
 2019-11-12 11:40:56 -08:00
Siyao Meng e0cf1735e1 HADOOP-16676. Backport HADOOP-16152 to branch-3.2. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-11-12 11:38:42 -08:00
Surendra Singh Lilhore c4e22e0c80 HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 
(cherry picked from commit 320008bb7c)
 2019-11-09 22:28:15 +05:30
Ayush Saxena 2e843cb773 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:50:30 +05:30
Chen Liang c9ed0c3a37 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:12:51 -08:00
Chen Liang 431edfcbe4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:00:15 -08:00
Ayush Saxena 5941a91f64 HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-06 22:10:27 +05:30
Surendra Singh Lilhore aa1c795dc9 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c)
 2019-11-06 19:37:41 +05:30
Wei-Chiu Chuang 240b5ee380 HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 
(cherry picked from commit eb73ba6ed5)
 2019-11-04 09:40:57 -08:00
Zhao Junwang c9fc118991 HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
(cherry picked from commit de6b8b0c0b)
 2019-11-01 16:39:18 -07:00
Ayush Saxena b1821be1ec HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:51:50 +05:30
Ayush Saxena 239505a0af HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:06 +05:30
Jonathan Hung 5d2ffcc7aa Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
 2019-10-30 20:59:20 -07:00
Duo Zhang a37a4bc62f HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 20:26:54 -07:00
Eric Badger 3d9127dc97 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2019-10-28 16:38:28 +00:00
Rakesh Radhakrishnan a6cdcf676e HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d)
 2019-10-28 10:41:40 +05:30
Rakesh Radhakrishnan dc2fad4bd3 HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6f)
 2019-10-28 10:41:25 +05:30
Rakesh Radhakrishnan c6d59e6a47 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1)
 2019-10-28 10:41:12 +05:30