Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,008 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

15008 Commits

Author SHA1 Message Date
Chen Liang c6267bb5ba HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chen Liang 17fac44040 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chao Sun 697207c487 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chao Sun 40d5cc15b2 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 37dfd294e2 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 63cf9801e8 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko ae559db6fd HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Chen Liang b4fc93d90b HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen 7146e67893 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Erik Krogen a7d551efd5 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko 7eacf5f8f1 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. Also fix a number of java7 incompatibilities from previous SbN read commits. 2019-08-09 15:49:56 -07:00
Chen Liang bf9d8ba66b HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-08-09 15:49:56 -07:00
Erik Krogen 7ccb2a48ff HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 
Also add a missing change from HDFS-13610
 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko c09665d14a HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko fcafa72958 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-08-09 15:49:56 -07:00
Chen Liang 9f90bcadf2 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-08-09 15:49:56 -07:00
Chao Sun f531301e7e HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen 33d8aae555 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 
Also fix a issue in IPCLoggerChannel
 2019-08-09 15:49:56 -07:00
Erik Krogen ea6d36763b HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov 9803d8db4c HDFS-13665. [SBN read] Move RPC response serialization into Server.doResponse(). Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Chao Sun 04eefe245a HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Konstantin V Shvachko 272a198175 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-08-09 15:49:55 -07:00
Erik Krogen 85f81fe458 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 
HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou.
 2019-08-09 15:49:55 -07:00
Erik Krogen 88d65af8a1 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Erik Krogen befe41074c HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov 3ed8d45660 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Erik Krogen c450e69551 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Erik Krogen d5da529acf HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Erik Krogen d0c85f83da HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov dd70e6d364 HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Chao Sun 61b510815c HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Siyao Meng 15062b6d28 HDFS-14696. Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class) (#1251) Contributed by Siyao Meng. 2019-08-08 14:55:41 -07:00
Erik Krogen ce12c8fc54 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 2019-08-08 10:18:00 -07:00
Erik Krogen 97600cd916 HDFS-14631. The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 2019-08-07 13:46:56 -07:00
Yiqun Lin 63531be505 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 23:01:51 +08:00
Haibo Chen 4a9fc45f6f YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d539)
(cherry picked from commit 8d357343c4)
(cherry picked from commit f943bff254)
(cherry picked from commit 6109ecf16d)
 2019-08-06 15:14:45 -07:00
Weiwei Yang 567e1178d8 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 2019-08-06 14:30:17 -07:00
Erik Krogen ee1547a807 HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-08-06 09:15:24 -07:00
Weiwei Yang 19af228957 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23)
(cherry picked from commit 70efe253f3)
(cherry picked from commit cd8e9fa80b2903935ca9c337631e60161a6053b3)
 2019-08-05 10:04:09 -07:00
Weiwei Yang 4e5ba315c6 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 78ea897b67)
(cherry picked from commit 7dd8eafe33)
(cherry picked from commit d095325234fe8fa8d2513c2f03c03347ef5f3606)
 2019-08-05 10:04:00 -07:00
Wangda Tan 75fc058290 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07)
(cherry picked from commit 46baafedf1)
(cherry picked from commit bf4da6aacee62be906e71eca6e33821f5ee79a2d)
 2019-08-05 10:03:56 -07:00
Eric Badger 9aae72af84 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 
(cherry picked from commit 144a55f0e3)
 2019-08-02 21:06:54 +00:00
Erik Krogen a503662d6c HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263)
(cherry picked from 055f824bcb)
 2019-08-02 11:53:34 -07:00
Steve Loughran e8ccc299f6
HADOOP-15237. In KMS docs there should be one space between KMS_LOG and NOTE. 
Contributed by Snigdhanjali Mishra.

Change-Id: I3abaa658c8786f8afa802ccbb629551313431b0e
 2019-08-01 10:37:28 +01:00
Eric E Payne 77d1aa9376 YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 
(cherry picked from commit c04ccf296d)
 2019-07-30 20:31:02 +00:00
Wei-Chiu Chuang 3941b3c708 HDFS-14464. Remove unnecessary log message from DFSInputStream. Contributed by Chao Sun. 2019-07-30 11:36:56 -07:00
Akira Ajisaka 46c00aca21
HADOOP-16435. RpcMetrics should not retained forever. Contributed by Zoltan Haindrich. 
(cherry picked from commit cbfa3f3e98)
 2019-07-29 17:40:14 -07:00
Erik Krogen 61581a86b2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-29 14:36:57 -07:00
Wei-Chiu Chuang 27ead06a13 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:34:31 -07:00
Erik Krogen edb46353c6 HADOOP-16245. Restrict the effect of LdapGroupsMapping SSL configurations to avoid interfering with other SSL connections. Contributed by Erik Krogen. 
(cherry-picked from 62efb63006)
(cherry-picked from 5ec34bf43a)
(cherry-picked from e2aab0efef9d504e4695a9c1df2835ed60f6c0c7)
(cherry-picked from f7e517ee24)
 2019-07-26 11:44:12 -07:00