Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,181 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7077 Commits

Author SHA1 Message Date
Erik Krogen f6ce2f4a50 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
 2019-08-07 09:33:36 -07:00
Lokesh Jain d06fca0104 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
(cherry picked from commit f6fe18a282)
 2019-08-06 15:31:15 -07:00
Stephen O'Donnell 99af4ce0ac HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 13:53:01 -07:00
Erik Krogen 69e3ac5cbb HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a807)
 2019-08-06 09:49:23 -07:00
Wei-Chiu Chuang a7a348446d Revert "HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell." 
This reverts commit 45ba41570b.
 2019-08-06 08:57:00 -07:00
Stephen O'Donnell 45ba41570b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
(cherry picked from commit 8414e6354b)
 2019-08-06 07:58:23 -07:00
Erik Krogen da548cf263 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
 2019-08-02 11:51:22 -07:00
Wei-Chiu Chuang 68d618acd2 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
(cherry picked from commit e1659fe009)
 2019-08-02 10:52:05 -07:00
Wei-Chiu Chuang d66ee4bf61 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98)
(cherry picked from commit 5c1f91fd97)
 2019-08-01 15:49:14 -07:00
Christopher Gregorian 7b8f08f59e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
 2019-07-30 14:45:00 -07:00
Wei-Chiu Chuang 608e7044a5 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388)
(cherry picked from commit d28e624958)
 2019-07-29 14:33:08 -07:00
Yiqun Lin 2e1afa9a86 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b)
(cherry picked from commit 69f1c8d77b)
 2019-07-29 06:50:41 -07:00
Inigo Goiri fe40fbbd4d HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f)
 2019-07-26 15:56:44 -07:00
Erik Krogen 42f10712a5 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit bf4a2258ce.
 2019-07-26 11:36:59 -07:00
Masatake Iwasaki bf4a2258ce HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
 2019-07-26 09:49:42 +09:00
Chen Liang 2a1451aa0e HDFS-13791. Amend fix LOG string format. 2019-07-12 12:58:08 -07:00
Wei-Chiu Chuang 3ba7826d41 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-07-12 12:26:23 -07:00
Erik Krogen cc22373606 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-12 12:25:19 -07:00
Erik Krogen 769b5a802b HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen 35200b3671 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen ee0416b456 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-07-12 12:24:43 -07:00
Chao Sun b028d62d1f HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-07-12 12:24:42 -07:00
Konstantin V Shvachko b8f9739024 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-07-12 12:24:42 -07:00
Chao Sun 8f36bdc3b9 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Chao Sun 7a650b018c HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Erik Krogen 6c1bf61628 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-12 12:24:40 -07:00
Erik Krogen 67c4768825 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-12 12:24:36 -07:00
Erik Krogen 265bd68464 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-07-12 12:23:17 -07:00
Chao Sun c4d944aeb3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-07-12 12:23:16 -07:00
Chen Liang 53a88bfdb4 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-07-12 12:22:25 -07:00
Chao Sun 8100dcbb34 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-07-12 12:11:18 -07:00
Konstantin V Shvachko 66db260cdc HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-07-12 12:11:18 -07:00
Konstantin V Shvachko e860882d60 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-07-12 12:11:17 -07:00
Konstantin V Shvachko a054c65185 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-07-12 12:11:14 -07:00
Chen Liang 91803c8216 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-07-12 11:50:51 -07:00
Erik Krogen d4adf921a3 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-07-12 11:50:05 -07:00
Erik Krogen 07558629db HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-07-12 11:50:04 -07:00
Konstantin V Shvachko e639297ba0 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2019-07-12 11:50:04 -07:00
Chen Liang 8ec17f6aea HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-07-12 11:50:03 -07:00
Erik Krogen 504cb70619 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 11:50:03 -07:00
Konstantin V Shvachko 1ad1c800a4 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-12 11:50:02 -07:00
Chen Liang ee9977cc69 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-07-12 11:50:01 -07:00
Chao Sun d96ddb3b87 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-07-12 11:50:01 -07:00
Erik Krogen 849a6c6f0d HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-07-12 11:49:58 -07:00
Erik Krogen ac2e79330c HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-07-12 11:05:50 -07:00
Chao Sun 9840dae72f HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 11:05:07 -07:00
Konstantin V Shvachko 405f6b6fbf HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-07-12 11:05:07 -07:00
Erik Krogen 8e551708d8 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:05:04 -07:00
Erik Krogen abb8614dcf HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:04:24 -07:00
Erik Krogen c40ddf9b5b HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-07-12 11:03:32 -07:00