Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,166 Commits 326 Branches 371 Tags
Commit Graph

4949 Commits

Author SHA1 Message Date
Kihwal Lee
d4038e7a39 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 1851d06eb3b70f39f3054a7c06f0ad2bc664aaec)
 2019-09-18 16:51:21 -05:00
He Xiaoqiao
6fe8261fa0 HDFS-14771. Backport HDFS-14617 to branch-2 (Improve fsimage load time by writing sub-sections to the fsimage index). Contributed by He Xiaoqiao and Stephen O'Donnel. 
Reviewed-by: Stephen O'Donnell <sodonnell@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-09-17 09:34:54 -07:00
Ayush Saxena
b9e832bf73 HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-09-14 10:30:46 +05:30
Xiaoyu Yao
c44e53e7a3
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:24:17 -07:00
Xiaoyu Yao
1c9255aebf
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit a524608d1e0a276fd822e9574f0a4a6d51298b13.
 2019-09-05 16:22:15 -07:00
Ayush Saxena
ad250a8cbb HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang 2019-09-05 22:05:47 +05:30
Chen Liang
1ff68d0e47 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang 2019-09-03 17:18:41 -07:00
Chen Liang
163cb4d5ae HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-09-03 17:11:29 -07:00
Chen Liang
e63619a6dd HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-09-03 17:11:29 -07:00
Chen Liang
a06e35a197 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-09-03 17:11:28 -07:00
Xiaoyu Yao
a524608d1e
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:27:23 -07:00
Chen Liang
f05d87b32e HDFS-14726. Fix JN incompatibility issue in branch-2 due to backport of HDFS-10519. Contributed by Chen Liang. 2019-08-30 15:55:35 -07:00
Chen Liang
e786147096 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen. 2019-08-26 14:32:27 -07:00
Wei-Chiu Chuang
69b80f1ee8 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8755de108d0e711cbfa784a20bd89e28fe79266b.
 2019-08-21 09:04:07 -07:00
Wei-Chiu Chuang
bdab03cf71 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:38:52 -07:00
Shashikant Banerjee
ab7509efe7 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 19:01:12 -07:00
Wei-Chiu Chuang
8755de108d HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:51:44 -07:00
He Xiaoqiao
b020c5ccfd HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-16 17:53:45 -07:00
Masatake Iwasaki
670de354b3 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe0473e353ee2d489156248a01aa982dfd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63d89e5c8377a035f93485a7c77c3430)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
 2019-08-14 12:35:14 +09:00
Christopher Gregorian
99cd181a78 HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a4e3b8f165806dfd2966f0951a5e575.

(cherry-picked from f96a2df38d889f29314c57f4d94227b2e419a11f)
(cherry-picked from d4492bdd9edec60c236aff85de50b963097e5a7f)
(cherry-picked from 7b8f08f59e5c8906930ccc67b967b7cfcbd41768)
(cherry picked from ec00431eaaa25eff5bb5e0cafb52de685187a159)
 2019-08-13 14:28:18 -07:00
Erik Krogen
719214ff30 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e24be294b40088a8aa46086ba8ca4ba8)
(cherry picked from 016aa139406d1a151fbec3fb8912eb08e0f638d7)
(cherry picked from f6ce2f4a50898b16556089fe18a6717a15570d85)
(cherry picked from 5657e45fb2b904bb31a4e3b70a360477ba288c15)
 2019-08-13 08:30:33 -07:00
Chen Liang
885b7f4c98 Fix JDK7 compatibility in branch-2. 2019-08-12 17:42:01 -07:00
Chen Liang
0d153dcb8c HDFS-13898 Addendum: add a missing test helper method 2019-08-12 10:31:37 -07:00
Wei-Chiu Chuang
585b6de637 Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit 567e1178d88ccfc258ce2ade4f8af66cc5a4daa7.
 2019-08-09 18:29:00 -07:00
Chen Liang
097710131a [SBN read] Addendum: Misc fix of issues, javadoc, whitespace unused backported fields etc 2019-08-09 15:49:58 -07:00
Wei-Chiu Chuang
12dc7ff566 HDFS-12836. startTxId could be greater than endTxId when tailing in-progress edit log. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Wei-Chiu Chuang
9599c91518 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-08-09 15:49:58 -07:00
Erik Krogen
c377540704 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
5190fd64be HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
c055e7812c HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen
3b2b97a6e6 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3f74968c92751f0972746b012e72810)
(cherry-picked from commit 6d076dd5e8be8fee5062a04d4cd2aa0c5da0ea56)
 2019-08-09 15:49:58 -07:00
Erik Krogen
a0327f58cb HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Chen Liang
b247a8b0de [SBN read] Addendum: Misc fix to be Java 7 compatible 2019-08-09 15:49:58 -07:00
Chao Sun
8cf1c0b91b HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko
3ec2ff756d HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-08-09 15:49:57 -07:00
Chao Sun
41e85ccf44 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chao Sun
705e1c8d96 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Erik Krogen
7b1a3b5619 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Erik Krogen
a86d955aa7 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Erik Krogen
0705e83bdb HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chao Sun
52d066a9d3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chen Liang
17fac44040 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chao Sun
40d5cc15b2 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko
37dfd294e2 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko
63cf9801e8 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko
ae559db6fd HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Chen Liang
b4fc93d90b HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen
7146e67893 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Erik Krogen
a7d551efd5 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko
7eacf5f8f1 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. Also fix a number of java7 incompatibilities from previous SbN read commits. 2019-08-09 15:49:56 -07:00