Commit Graph

5122 Commits

Author SHA1 Message Date
Xiaoyu Yao a524608d1e
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun.
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
2019-09-03 08:27:23 -07:00
Chen Liang f05d87b32e HDFS-14726. Fix JN incompatibility issue in branch-2 due to backport of HDFS-10519. Contributed by Chen Liang. 2019-08-30 15:55:35 -07:00
Chen Liang e786147096 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen. 2019-08-26 14:32:27 -07:00
Wei-Chiu Chuang 69b80f1ee8 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow."
This reverts commit 8755de108d.
2019-08-21 09:04:07 -07:00
Wei-Chiu Chuang bdab03cf71 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:38:52 -07:00
Shashikant Banerjee ab7509efe7 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-18 19:01:12 -07:00
Wei-Chiu Chuang 8755de108d HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:51:44 -07:00
He Xiaoqiao b020c5ccfd HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-16 17:53:45 -07:00
Masatake Iwasaki 61bd868e80 HDFS-14707. Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-08-15 11:21:17 +09:00
Masatake Iwasaki 670de354b3 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
2019-08-14 12:35:14 +09:00
Christopher Gregorian 99cd181a78 HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
(cherry-picked from 7b8f08f59e)
(cherry picked from ec00431eaa)
2019-08-13 14:28:18 -07:00
Erik Krogen 719214ff30 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen.
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
(cherry picked from f6ce2f4a50)
(cherry picked from 5657e45fb2)
2019-08-13 08:30:33 -07:00
Chen Liang 885b7f4c98 Fix JDK7 compatibility in branch-2. 2019-08-12 17:42:01 -07:00
Chen Liang 0d153dcb8c HDFS-13898 Addendum: add a missing test helper method 2019-08-12 10:31:37 -07:00
Wei-Chiu Chuang 585b6de637 Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao."
This reverts commit 567e1178d8.
2019-08-09 18:29:00 -07:00
Chen Liang 097710131a [SBN read] Addendum: Misc fix of issues, javadoc, whitespace unused backported fields etc 2019-08-09 15:49:58 -07:00
Wei-Chiu Chuang 12dc7ff566 HDFS-12836. startTxId could be greater than endTxId when tailing in-progress edit log. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Wei-Chiu Chuang 9599c91518 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-08-09 15:49:58 -07:00
Erik Krogen 2fe15053ab HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen c377540704 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen 5190fd64be HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen c055e7812c HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen 3b2b97a6e6 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam.
(cherry-picked from commit 1bc282e0b3)
(cherry-picked from commit 6d076dd5e8)
2019-08-09 15:49:58 -07:00
Erik Krogen a0327f58cb HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Chen Liang b247a8b0de [SBN read] Addendum: Misc fix to be Java 7 compatible 2019-08-09 15:49:58 -07:00
Konstantin V Shvachko 3804b8e7be HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-08-09 15:49:57 -07:00
Chao Sun 8cf1c0b91b HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 3ec2ff756d HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-08-09 15:49:57 -07:00
Chao Sun 41e85ccf44 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chao Sun 705e1c8d96 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Erik Krogen 7b1a3b5619 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Erik Krogen a86d955aa7 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Chen Liang 68da1e48f9 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Erik Krogen 0705e83bdb HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chen Liang 3c73819a63 HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 3518767ce7 HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. 2019-08-09 15:49:57 -07:00
Chao Sun 52d066a9d3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chen Liang c6267bb5ba HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chen Liang 17fac44040 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chao Sun 697207c487 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chao Sun 40d5cc15b2 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 37dfd294e2 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 63cf9801e8 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko ae559db6fd HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Chen Liang b4fc93d90b HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen 7146e67893 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Erik Krogen a7d551efd5 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko 7eacf5f8f1 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. Also fix a number of java7 incompatibilities from previous SbN read commits. 2019-08-09 15:49:56 -07:00
Chen Liang bf9d8ba66b HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-08-09 15:49:56 -07:00
Erik Krogen 7ccb2a48ff HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen.
Also add a missing change from HDFS-13610
2019-08-09 15:49:56 -07:00
Konstantin V Shvachko c09665d14a HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko fcafa72958 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-08-09 15:49:56 -07:00
Chen Liang 9f90bcadf2 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-08-09 15:49:56 -07:00
Chao Sun f531301e7e HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen 33d8aae555 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang.
Also fix a issue in IPCLoggerChannel
2019-08-09 15:49:56 -07:00
Erik Krogen ea6d36763b HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Chao Sun 04eefe245a HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Konstantin V Shvachko 272a198175 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-08-09 15:49:55 -07:00
Erik Krogen 85f81fe458 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen.
HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou.
2019-08-09 15:49:55 -07:00
Erik Krogen 88d65af8a1 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Erik Krogen befe41074c HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov 3ed8d45660 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Erik Krogen c450e69551 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Erik Krogen d5da529acf HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Erik Krogen d0c85f83da HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov dd70e6d364 HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Chao Sun 61b510815c HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Siyao Meng 15062b6d28 HDFS-14696. Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class) (#1251) Contributed by Siyao Meng. 2019-08-08 14:55:41 -07:00
Erik Krogen ce12c8fc54 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 2019-08-08 10:18:00 -07:00
Erik Krogen 97600cd916 HDFS-14631. The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 2019-08-07 13:46:56 -07:00
Yiqun Lin 63531be505 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 23:01:51 +08:00
Weiwei Yang 567e1178d8 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 2019-08-06 14:30:17 -07:00
Erik Krogen ee1547a807 HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-08-06 09:15:24 -07:00
Erik Krogen a503662d6c HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira.
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263)
(cherry picked from 055f824bcb)
2019-08-02 11:53:34 -07:00
Wei-Chiu Chuang 3941b3c708 HDFS-14464. Remove unnecessary log message from DFSInputStream. Contributed by Chao Sun. 2019-07-30 11:36:56 -07:00
Erik Krogen 61581a86b2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-29 14:36:57 -07:00
Wei-Chiu Chuang 27ead06a13 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:34:31 -07:00
Erik Krogen 653951e1f7 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena."
This reverts commit 3bfae20e94.
2019-07-26 08:27:36 -07:00
Masatake Iwasaki 3bfae20e94 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
2019-07-26 09:54:27 +09:00
Erik Krogen cf7b9a5bd5 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-16 07:22:45 -07:00
Weiwei Yang 3f112e7347 HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang.
(cherry picked from commit 6c8cfd135e)
2019-07-11 10:07:47 +08:00
Yiqun Lin 78aebee5c5 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao.
(cherry picked from commit 993dc8726b)
2019-07-10 10:58:59 +08:00
stack a7c6d25dc3 HDFS-14483 Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9""
This is a revert of a revert, i.e. a reapplication of the HDFS-14483
patch. Original commit was missing the JIRA number.

Revert "Revert "Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9""

This reverts commit b26ad8a4c3.
2019-07-09 08:58:16 -07:00
stack b26ad8a4c3 Revert "Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9"
This reverts commit 4f09a6bcfd.
2019-07-09 08:57:57 -07:00
sunlisheng 4f09a6bcfd Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9
Signed-off-by: stack <stack@apache.org>
2019-07-09 08:54:37 -07:00
leosunli 0ff5db746e HDFS-13694. Making md5 computing being in parallel with image loading.
(cherry picked from commit 9c90729486)
2019-07-07 20:04:08 -07:00
Masatake Iwasaki e61e52805f HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina.
(cherry picked from commit 1c254a8ec7)
2019-07-04 23:55:13 +09:00
stack bb7e143d34 HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9
Reapply but include JIRA # in the commit message this time.

Revert "Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9""

This reverts commit 1dd54163fc.
2019-07-01 11:08:05 -07:00
stack 1dd54163fc Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9"
This reverts commit bccb0fd846.
2019-07-01 11:07:43 -07:00
sunlisheng bccb0fd846 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-07-01 10:59:58 -07:00
Inigo Goiri 0272480e9f HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao.
(cherry picked from commit 80b77deb42)
(cherry picked from commit 9637124535)
(cherry picked from commit 90b88db35d)
(cherry picked from commit 9daa45f646)
2019-06-24 21:17:38 -07:00
Inigo Goiri 1373889ac2 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException.
(cherry picked from commit 0966407ad6)
2019-06-24 19:04:33 -07:00
Wei-Chiu Chuang bf63475e7a HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 21:06:06 -07:00
Wei-Chiu Chuang f876e82abc HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-20 19:36:44 -07:00
Zsolt Venczel a68de43957 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff)
(cherry picked from commit 4ea3b04bf3)
(cherry picked from commit 76faa41f18)
2019-06-18 14:38:42 -07:00
Kitti Nanasi abd11f9db1 HDFS-13770. dfsadmin -report does not always decrease "missing blocks (with replication factor 1)" metrics when file is deleted. Contributed by Kitti Nanasi.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-06-18 00:25:53 -07:00
Wei-Chiu Chuang ce49f8a5df HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu.
(cherry picked from commit 62ad9885ea)
(cherry picked from commit 45543ffd6a)
(cherry picked from commit fc8983a5b8)
(cherry picked from commit 79f8e4d5bc)
2019-06-17 20:20:34 -07:00
Akira Ajisaka 73a59b2c42 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827)
(cherry picked from commit f68099aa13)
(cherry picked from commit 3d1d8623a1)
(cherry picked from commit 010a99104b)
2019-06-17 17:04:38 -07:00
huzheng 8b90c69357 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read
(cherry picked from commit ea3b0a1844)
(cherry picked from commit 9b61ecfcbe)
(cherry picked from commit b082628e5a)
(cherry picked from commit 2b9c4fccf4)
2019-06-17 07:22:01 -07:00
Kihwal Lee 500dc05924 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee
(cherry picked from commit 9122b9b649)
(cherry picked from commit fd8b8c7494)
2019-05-30 17:37:38 -05:00