Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,615 Commits 326 Branches 371 Tags
Commit Graph

7539 Commits

Author SHA1 Message Date
Kihwal Lee
bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)
(cherry picked from commit f1840669cfc92c033521bf95989479523e2a649d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
 2020-02-11 14:57:49 -06:00
Ayush Saxena
547f1798aa HDFS-15157. Fix compilation failure for branch-3.1. Contributed by Ayush Saxena 2020-02-09 08:14:33 +05:30
Chen Liang
466f77a5cf HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 
(cherry picked from commit 226cd3977bfc5948afff6a2d38a41e341089d5fe)
 2020-02-04 12:36:46 -08:00
Kihwal Lee
a55a0a1f6d HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 10a60fbe20bb08cdd71076ea9bf2ebb3a2f6226e)
 2020-02-04 12:24:58 -06:00
Stephen O'Donnell
8ea4787f9f HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d931676b191cb4813ed1a283ebb24d4eb)
(cherry picked from commit e89436b1839aa87b4bedfc7e2d558c7541e73540)
 2020-01-31 16:13:53 -08:00
Inigo Goiri
ffaf0d3c82 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:25:36 -08:00
Chen Liang
5e1d89b74d [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-29 11:15:35 -08:00
Kihwal Lee
b8d87cfc32 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4e8fe78eb9ab607a0449cdd075041fb)
 2020-01-29 11:01:51 -06:00
Ayush Saxena
1293199532 HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:17:21 +05:30
Kihwal Lee
7dd43cec7f HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c91225f86ed9c0f0e6a9adf2e1434674)
(cherry picked from commit 1bb9667137bc80e63dea3e53bb2746be011a2a9a)
 2020-01-24 09:53:26 -06:00
Kihwal Lee
b4e9725955 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 
(cherry picked from commit bd03053ea2f32ef982e37fbf2ffd679cb7dda797)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java

(cherry picked from commit 484270832064c84122348e663aaf5927ed411ebb)
 2020-01-22 09:28:01 -06:00
Akira Ajisaka
2c84ea96e3
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:05:13 +09:00
Wei-Chiu Chuang
286c94f13d HDFS-15097. Purge log in KMS and HttpFS. Contributed by Doris Gu. 
(cherry picked from commit 6b86a5110e4b3f45bf55f97bd402680bf26cefb9)
(cherry picked from commit aa3c96474f52fba01d999fa75880e0948affbb01)
 2020-01-13 10:49:39 -08:00
Ayush Saxena
12b258115b HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:34:37 +05:30
Chen Liang
5490061885 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 
(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)
 2020-01-10 17:19:05 -08:00
Kihwal Lee
82fa8d88ef HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein. 
(cherry picked from commit 5fb901ac4017b4f13b089ecd920e864cd53ad3a6)
 2020-01-10 12:47:49 -06:00
Surendra Singh Lilhore
09d3707875 HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:18:58 +05:30
Rakesh Radhakrishnan
b1eb47865c HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He. 
(cherry picked from commit 7030722e5d9f376245a9ab0a6a883538b6c55f82)
 2020-01-08 14:32:45 +05:30
Masatake Iwasaki
c5580af1c4 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1fbb150ea7ff0411c41ffd3a2796208)
 2020-01-08 17:06:49 +09:00
Masatake Iwasaki
4c8b01abcb
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 
(cherry picked from commit a43c177f1d4c2b6149a2680dd23d91103eca3be0)
 2020-01-08 12:38:51 +09:00
Rakesh Radhakrishnan
8b748c0308 HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 
(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)
 2020-01-02 12:19:38 +05:30
Surendra Singh Lilhore
55510d32bf HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:25:28 +05:30
Takanobu Asanuma
454a0c2913 HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena. 
(cherry picked from commit dc32f583afffc372f78fb45211c3e7ce13f6a4be)
 2019-12-28 21:35:22 +09:00
Chandra Sanivarapu
e33651ace7
HDFS-15073. Replace curator-shaded guava import with the standard one (#1782) 
(cherry picked from commit d8cd7098b4bcfbfd76915b9ecefb2c7ea320e149)
 2019-12-25 11:12:29 +09:00
Konstantin V Shvachko
1a22e4036c HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b98ac2a3af50ccf2af07790ab0760d4c51820836)
 2019-12-24 12:50:13 -08:00
Eric Badger
ef00db4ce9 Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui." 
This reverts commit 27555620fae3791d9fd24168e346714de8467a1d.
 2019-12-20 19:16:22 +00:00
Inigo Goiri
27555620fa HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 
(cherry picked from commit 52d7b745c6d95e799542d6409dac30d0418ce8a8)
 2019-12-19 09:47:30 -08:00
Shashikant Banerjee
280cd6dd1e HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee. 
(cherry picked from commit fdd96e46d1f89f0ecdb9b1836dc7fca9fbb954fd)
 2019-12-18 23:11:18 +05:30
Chen Liang
ea0eeb8f1a HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang. 
(cherry picked from commit 65c4660bcd897e139fc175ca438cff75ec0c6be8)
(cherry picked from commit d4a6901c429a24dffdb5c884b1ba8e6492c233fe)
 2019-12-12 15:58:09 -08:00
He Xiaoqiao
974706c5fd HDFS-15050. Optimize log information when DFSInputStream meet CannotObtainBlockLengthException. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 99af49878cc47f5750b6c0edadc8a4c2fcd7a7c6)
(cherry picked from commit 8ba4e13dcc4ad16e8483d33a31ed52e1b8d4e686)
 2019-12-12 02:23:22 -08:00
Erik Krogen
b06965ae32 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen. 
(cherry picked from c174d50b30abc08a4642614fb35165e79792608b)
(cherry picked from 3402c87353c18730cb659ac1d8b43b4e21c634f6)
 2019-12-11 09:41:11 -08:00
Surendra Singh Lilhore
5eb21b83df HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-29 00:07:45 +05:30
Ayush Saxena
35f0478042 HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-11-23 02:47:59 +05:30
Surendra Singh Lilhore
8551c04726 HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina. 2019-11-22 00:45:55 +05:30
He Xiaoqiao
d101b74814 HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
(cherry picked from commit 0b50aa29fd5dc114b3e0fc54b5c393bbc9f3102e)
(cherry picked from commit 1d383436f09f8e490171c8f6af4605971ad2f904)
 2019-11-19 12:22:54 -08:00
Siyao Meng
df13f6d485 HADOOP-16703. Backport HADOOP-16152 to branch-3.1. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-11-19 12:14:27 -08:00
Ayush Saxena
b4f0c63a6a HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 
(cherry picked from commit 9e287054a8aa0725643bc5c90601645302fffade)
(cherry picked from commit d16441460215155b25fa649a7ca733e579d97929)
 2019-11-18 10:23:48 -08:00
Erik Krogen
17779adb32 HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a78f31cdd937dc4d1a2255f80c5881e)
(cherry picked from 60655bfe54e138957ef5bbf480a4541bd83152fd)
 2019-11-15 11:14:49 -08:00
Erik Krogen
5cf36aa2b1 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe7113ed246ed0275bb3833882a3d873d70)
(cherry picked from dec765b329d3947f30273c0e7f0c4eb607ec42c9)
 2019-11-13 11:57:59 -08:00
Chao Sun
7e0f1eabd8 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 
(cherry picked from commit 97ec34e117af71e1a9950b8002131c45754009c7)
(cherry picked from commit 0c12d633b9797f2cb14246fe020993b2e9123e82)
 2019-11-12 11:41:09 -08:00
Surendra Singh Lilhore
9179046368 HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 
(cherry picked from commit 320008bb7cc558b1300398178bd2f48cbf0b6c80)
 2019-11-09 22:36:48 +05:30
Ayush Saxena
9aa59c0629 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:42:07 +05:30
Chen Liang
76ca864406 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:21:41 -08:00
Chen Liang
c6cbf67b82 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:11:09 -08:00
Ayush Saxena
5841d0b2f9 HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-06 22:17:11 +05:30
Surendra Singh Lilhore
4690142e86 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c212b26d75268ee3659aa2cadcff349)
 2019-11-06 19:48:16 +05:30
Wei-Chiu Chuang
c9ff4b7a1b HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 
(cherry picked from commit eb73ba6ed5f7c5500cc0ef36ca22aae4e71046fa)
(cherry picked from commit 240b5ee3802550950707c91f22154386d915fffb)
 2019-11-04 09:41:05 -08:00
Zhao Junwang
cda8b2ac69 HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
(cherry picked from commit de6b8b0c0b1933aab2af3e8adc50a2091d428238)
(cherry picked from commit c9fc118991104acf9109aa769fe1a0e84f425a61)
 2019-11-01 16:39:29 -07:00
Ayush Saxena
9956d18129 HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:52:12 +05:30
Ayush Saxena
98e3c6af2c HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:37 +05:30