Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,037 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7083 Commits

Author SHA1 Message Date
Yiqun Lin b46ec61909 HDFS-15660. StorageTypeProto is not compatiable between 3.x and 2.6. Contributed by Ryan Wu. 
(cherry picked from commit da1ea2530f)
 2020-12-07 18:54:32 +08:00
Toshihiro Suzuki fe89969a9d HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2052) 
Contributed by Toshihiro Suzuki.

(cherry picked from commit 545a0a147c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit e5a02a7499)
 2020-06-05 12:51:40 +01:00
S O'Donnell b7b3164ce7 Revert "HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein." 
This reverts commit ed59412434.
 2020-06-05 12:48:07 +01:00
Inigo Goiri ed59412434 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f6)
 2020-01-31 11:26:32 -08:00
Inigo Goiri 380fefc40e Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui." 
This reverts commit 0b75c5238b.
 2019-12-20 11:28:17 -08:00
Inigo Goiri 0b75c5238b HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui. 
(cherry picked from commit 52d7b745c6)
 2019-12-19 09:48:02 -08:00
Duo Zhang 322cef7762 HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:57:26 -07:00
Eric Badger 5196b035f7 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2019-10-28 18:17:47 +00:00
Xiaoyu Yao 0b80308b78
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:52:50 -07:00
Xiaoyu Yao 121c6cb59e
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 5369a3bd94.
 2019-09-05 16:52:50 -07:00
Xiaoyu Yao 5369a3bd94
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:44:29 -07:00
Shashikant Banerjee 5b064381aa HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit ab7509efe7)
 2019-08-19 10:07:16 -07:00
Surendra Singh Lilhore 0cb3738567 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
(cherry picked from commit 6b01effd01)
 2019-08-18 18:18:02 -07:00
Chao Sun 4c06b2f96c HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
(cherry picked from d38b617baa)
(cherry picked from 90447baeb0)
 2019-08-08 10:47:33 -07:00
Erik Krogen 5657e45fb2 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
(cherry picked from f6ce2f4a50)
 2019-08-07 09:34:01 -07:00
Yiqun Lin ea5330643c HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 23:05:09 +08:00
Erik Krogen b0c5151959 HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a807)
(cherry picked from 69e3ac5cbb)
 2019-08-06 09:49:56 -07:00
Erik Krogen 055f824bcb HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263)
 2019-08-02 11:52:26 -07:00
Christopher Gregorian ec00431eaa HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
(cherry-picked from 7b8f08f59e)
 2019-07-30 14:46:11 -07:00
Inigo Goiri 998593e7f2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f)
(cherry picked from commit fe40fbbd4d)
 2019-07-29 14:39:20 -07:00
Wei-Chiu Chuang b5303012a0 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-07-25 10:07:06 -07:00
Erik Krogen 32d80d2201 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-25 10:07:06 -07:00
Erik Krogen 11fee2d4e1 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:06 -07:00
Erik Krogen 16a4043f01 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen 2598282293 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen 45cd309a40 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko 380668559f HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-07-25 10:07:05 -07:00
Chao Sun 9c1c73a0d5 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko 451b8b8f16 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-07-25 10:07:05 -07:00
Chao Sun 9aecfdcb52 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chao Sun 6e19a945bc HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Erik Krogen 8b8ec65e65 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Erik Krogen 7b1e3c49d1 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-25 10:07:05 -07:00
Chen Liang 82f68a47c4 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Erik Krogen b8df864ab0 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chen Liang 8769e6feda HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko 54a1c66e48 HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. 2019-07-25 10:07:05 -07:00
Chao Sun 6916ee71fd HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Chen Liang 96cdd13de5 HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Chen Liang 683daedc1f HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Chao Sun 4ce7f9f2e6 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00
Chao Sun 11897618b9 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko ca565efe2f HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko 7b425c8703 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko 28b11b03d7 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-07-25 10:07:05 -07:00
Chen Liang 56af83c961 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-07-25 10:07:05 -07:00
Erik Krogen 894f095219 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Erik Krogen 32737360a5 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-07-25 10:07:05 -07:00
Konstantin V Shvachko ea2d862320 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2019-07-25 10:07:05 -07:00
Chen Liang 613c9e4f7b HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-07-25 10:07:05 -07:00