Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,023 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8208 Commits

Author SHA1 Message Date
Jonathan Hung d255efa5f8 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 
(cherry picked from commit 8ab7020e64)
 2019-08-27 10:30:55 -07:00
Arpit Agarwal 8b12381717
HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 
(cherry picked from commit a64a43b77fb1032dcb66730a6b6257a24726c256)
 2019-08-26 15:57:41 -07:00
Chen Liang c824653955 HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:28:13 -07:00
Erik Krogen 2408c2491f HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 
(cherry picked from d699022fce)
 2019-08-23 11:48:11 -07:00
Akira Ajisaka e8b93d04c4
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5)
 2019-08-23 15:28:54 +09:00
Akira Ajisaka d3e5898712
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4ca)
 2019-08-23 14:45:57 +09:00
Ayush Saxena b7a66471ca HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:35:48 +05:30
Wei-Chiu Chuang 933ea6aa4d Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8b802d6b44.
 2019-08-21 09:00:23 -07:00
Surendra Singh Lilhore 03c62c7989 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
 2019-08-20 15:54:46 -07:00
Wei-Chiu Chuang 77fd690c05 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
 2019-08-20 10:37:05 -07:00
Wei-Chiu Chuang b004d305e9 HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde2)
 2019-08-20 10:33:59 -07:00
Wei-Chiu Chuang 6d93886231 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342)
 2019-08-19 13:11:11 -07:00
HUAN-PING SU abda503040 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2)
 2019-08-19 11:15:37 -07:00
Wei-Chiu Chuang 8b802d6b44 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-19 08:38:06 -07:00
Surendra Singh Lilhore 6b01effd01 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
 2019-08-18 18:17:41 -07:00
Shashikant Banerjee e89413da88 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959c)
 2019-08-16 19:09:34 -07:00
Siyao Meng c6e4906ffc HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b9)
 2019-08-16 19:08:05 -07:00
Chen Liang d8a6098a96 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang f507bc059d HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang ca15968d09 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-16 10:50:00 -07:00
Chen Liang 22d81f7fd0 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-16 10:49:59 -07:00
Wei-Chiu Chuang f5661b630a HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
 2019-08-13 17:15:19 -07:00
Masatake Iwasaki d7ca016d63 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
 2019-08-14 08:45:10 +09:00
Siyao Meng 88aece4083 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1267) Contributed by Siyao Meng. 
(cherry picked from commit 6ae8bc3a4a)
 2019-08-13 16:36:52 -07:00
Siyao Meng c5f433b11f HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a)
 2019-08-12 17:30:18 -07:00
Yiqun Lin a7bc09c596 HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c7)
 2019-08-09 18:41:03 -07:00
Wei-Chiu Chuang 2f2fa3dad6 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494)
 2019-08-08 13:51:14 -07:00
Stephen O'Donnell a00a327594 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf)
 2019-08-08 13:37:10 -07:00
Chao Sun d38b617baa HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
 2019-08-08 10:15:48 -07:00
Erik Krogen 016aa13940 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
 2019-08-07 09:32:32 -07:00
Lokesh Jain f6fe18a282 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
 2019-08-06 15:31:05 -07:00
Inigo Goiri 3e8e1570f6 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from 5740eea081)
 2019-08-06 09:26:01 -07:00
Stephen O'Donnell 8414e6354b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
 2019-08-06 07:58:12 -07:00
Erik Krogen 2634aaaf9a HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
 2019-08-02 12:00:48 -07:00
Wei-Chiu Chuang e1659fe009 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
 2019-08-02 10:51:55 -07:00
Wei-Chiu Chuang 5c1f91fd97 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98)
 2019-08-01 15:49:05 -07:00
Christopher Gregorian d4492bdd9e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
 2019-07-30 14:41:36 -07:00
Wei-Chiu Chuang d28e624958 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388)
 2019-07-29 14:32:46 -07:00
Yiqun Lin 69f1c8d77b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b)
 2019-07-29 06:50:28 -07:00
Erik Krogen bd5033e6f9 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit 2478d333a6.
 2019-07-26 11:36:06 -07:00
Masatake Iwasaki 2478d333a6 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
 2019-07-26 09:48:00 +09:00
Masatake Iwasaki 9ed2c22d57
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00dd843a1a)
 2019-07-12 10:08:18 +09:00
Inigo Goiri 950aa74d5f HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
 2019-07-10 11:07:15 -07:00
Erik Krogen 491dc7cc9d HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148)
 2019-07-08 15:15:41 -07:00
Wei-Chiu Chuang 29b9049bde HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 
(cherry picked from commit ec851e4db2)
 2019-07-08 15:25:10 +08:00
leosunli adec307251 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c90729486)
 2019-07-07 20:02:08 -07:00
Masatake Iwasaki ee81a64da7 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec7)
 2019-07-04 23:53:30 +09:00
Weiwei Yang fa8b8edbac HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 
(cherry picked from commit 729cb3aefe)
 2019-07-04 18:06:19 +08:00
Wei-Chiu Chuang 66920e6d37 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-28 14:28:35 -07:00
Erik Krogen e53cee0c14 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-06-28 14:28:28 -07:00
Erik Krogen 3c6b74a6a4 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:28:17 -07:00
Erik Krogen e1e4056353 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-06-28 14:28:10 -07:00
Erik Krogen 98434bed5c HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:28:03 -07:00
Erik Krogen 73d56b9d2a HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-06-28 14:27:54 -07:00
Konstantin V Shvachko c936562c72 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-06-28 14:23:14 -07:00
Chao Sun 720eed9ef9 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-06-28 14:23:14 -07:00
Konstantin V Shvachko 2be93bd54b HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-06-28 14:23:14 -07:00
Chao Sun eb7d7fa5e2 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-06-28 14:23:13 -07:00
Chao Sun 25913f1c34 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-06-28 14:23:13 -07:00
Erik Krogen e917ac2854 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-06-28 14:23:12 -07:00
Erik Krogen c1c061d767 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-06-28 14:23:12 -07:00
Chen Liang 589d2c5922 HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2019-06-28 14:23:11 -07:00
Erik Krogen f1c5b247b9 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-06-28 14:23:11 -07:00
Chen Liang f7f7cac2e4 HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2019-06-28 14:23:11 -07:00
Konstantin V Shvachko f0b73502c5 HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. 2019-06-28 14:23:10 -07:00
Chao Sun 45d6834d4c HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-06-28 14:23:10 -07:00
Chen Liang 03a2c60a4b HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. 2019-06-28 14:23:10 -07:00
Chen Liang 3d3686a619 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-06-28 14:23:09 -07:00
Chao Sun fc54850509 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. 2019-06-28 14:23:09 -07:00
Chao Sun cc70e81931 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-06-28 14:23:09 -07:00
Konstantin V Shvachko 8eef380af7 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-06-28 14:23:08 -07:00
Konstantin V Shvachko dea7e1a239 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-06-28 14:23:08 -07:00
Konstantin V Shvachko c2bfaf4d27 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-06-28 14:23:07 -07:00
Chen Liang 3998bab3da HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-06-28 14:23:05 -07:00
Erik Krogen a856eea78a HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-06-28 14:22:12 -07:00
Erik Krogen 556b7a840b HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-06-28 14:22:12 -07:00
Konstantin V Shvachko 0de06c010a HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2019-06-28 14:22:12 -07:00
Chen Liang 4c4ca86286 HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-06-28 14:22:11 -07:00
Erik Krogen afeaeadd4f HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:22:11 -07:00
Konstantin V Shvachko 0b56a5300b HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-06-28 14:22:11 -07:00
Konstantin V Shvachko 0c9d846c60 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-06-28 14:22:10 -07:00
Chen Liang e2337bfc57 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-06-28 14:22:10 -07:00
Chao Sun 6fe755f0c7 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-06-28 14:22:09 -07:00
Erik Krogen e3f3fb1c18 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-06-28 14:22:09 -07:00
Erik Krogen ea402854aa HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-06-28 14:22:09 -07:00
Chao Sun a97f00e1c6 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-06-28 14:22:08 -07:00
Konstantin V Shvachko 1f5852fb12 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-06-28 14:22:08 -07:00
Erik Krogen ac3eeeeb3c HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-06-28 14:22:07 -07:00
Erik Krogen b13110a6f0 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-06-28 14:22:07 -07:00
Erik Krogen 174c41c68f HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-06-28 14:22:07 -07:00
Plamen Jeliazkov a3521c53fe HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-06-28 14:22:03 -07:00
Erik Krogen dbf777e4a5 HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-06-28 14:20:02 -07:00
Erik Krogen 49d7bb6a92 HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-06-28 14:20:01 -07:00
Erik Krogen c0ca2bb853 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-06-28 14:20:01 -07:00
Plamen Jeliazkov 1eeca2d9fb HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-06-28 14:20:00 -07:00
Chao Sun 3932ac4ef7 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-06-28 14:20:00 -07:00
Daniel Templeton 252c3a158d HDFS-14047. [libhdfs++] Fix hdfsGetLastExceptionRootCause bug in test_libhdfs_threaded.c 
(Contributed by Antoli Shein via Daniel Templeton)

Change-Id: Ia5546fd5f8270f285c373b9023a9bd9835059845
 2019-06-27 15:02:59 -07:00
Anu Engineer e7fce2104f HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee)
 2019-06-24 21:19:50 -07:00
Inigo Goiri 9637124535 HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42)
 2019-06-24 21:16:46 -07:00
Inigo Goiri 0966407ad6 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 2019-06-24 19:02:41 -07:00
Wei-Chiu Chuang d5ecca1d4c HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f)
 2019-06-21 18:27:38 -07:00
Wei-Chiu Chuang 845d7611ba HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c5)
 2019-06-21 18:18:08 -07:00
Wei-Chiu Chuang a5d76a4965 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 20:54:16 -07:00
Lokesh Jain 461c624cc0 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 272b96d243)
 2019-06-20 20:23:31 -07:00
Takanobu Asanuma 6dffad028e HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643)
 2019-06-20 20:17:45 -07:00
Wei-Chiu Chuang 7e9b9e2a2e Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 36dc0810bf.
 2019-06-19 06:55:08 -07:00
Virajith Jalaparti 36dc0810bf HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
 2019-06-18 22:16:56 -07:00
Zsolt Venczel 7c00756aff HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
 2019-06-18 14:37:16 -07:00
Elek, Marton 1a99f97945 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
 2019-06-18 12:00:36 -07:00
Wei-Chiu Chuang 223a582eb8 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
 2019-06-18 00:34:38 -07:00
Wei-Chiu Chuang 0b5bd8e451 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78)
 2019-06-17 22:47:55 -07:00
Weiwei Yang f36cdc4d7f HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
 2019-06-17 21:46:58 -07:00
Wei-Chiu Chuang 45543ffd6a HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea)
 2019-06-17 20:19:59 -07:00
Akira Ajisaka f68099aa13 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827)
 2019-06-17 17:03:51 -07:00
Santosh Marella 1ee18ebd26 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:26:39 -07:00
Wei-Chiu Chuang 412fbf12cb Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit fe46d1c79f.
 2019-06-17 16:25:49 -07:00
Wei-Chiu Chuang 0eb42e1e05 HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
 2019-06-17 15:18:00 -07:00
huzheng 9b61ecfcbe HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a1844)
 2019-06-17 07:20:34 -07:00
Santosh Marella fe46d1c79f HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 15:15:58 -07:00
Inigo Goiri 335aebb9c4 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 
(cherry picked from commit 4f455290b1)
 2019-06-13 18:27:26 -07:00
Sean Mackrory e0b3cbd221 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:53:40 -06:00
Stephen O'Donnell 71159f9540 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0)
 2019-06-12 10:31:03 -07:00
He Xiaoqiao d7560c866e HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2019-06-10 17:26:42 -07:00
Hanisha Koneru 210480a23c HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40)
 2019-06-10 13:46:53 -07:00
Kihwal Lee 056bb773ce HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b649)
 2019-05-30 17:30:48 -05:00
Ayush Saxena 2cd7577a5a HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f2)
 2019-05-29 20:58:11 -07:00
Takanobu Asanuma a9a3450560 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:34:16 +09:00
Akira Ajisaka 855dc997d6
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:27:33 +09:00
Erik Krogen c26cf22d6b HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f0)
 2019-05-24 13:18:49 -07:00
Konstantin V Shvachko 9836bac35c HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:49:06 -07:00
Siyao Meng a4f425d3ae HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9dff6eff81)
 2019-05-22 13:24:58 +09:00
Srinivasu Majeti 85f3664bb8 HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3e5e5b028a)
 2019-05-17 19:21:42 +02:00
Siyao Meng bba8971897 HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a)
 2019-05-01 16:49:45 -07:00
Arpit Agarwal db4c0b357d
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f8)
 2019-04-29 14:54:30 -07:00
Puleya7 d724f74f6b HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479)
 2019-04-17 21:55:45 +09:00
Akira Ajisaka f020ac2c06
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:30:46 +09:00
Stephen O'Donnell 388f445dde HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
 2019-04-04 04:06:15 -07:00
Wei-Chiu Chuang c8703dda07 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070)
 2019-04-03 11:01:44 -07:00
Gabor Bota 67cdf807a2
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e)
 2019-04-01 13:52:19 +09:00
Takanobu Asanuma 162e9999c7 HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb)
 2019-03-27 03:28:54 +09:00
Inigo Goiri 4eb0497091 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 13:58:03 -07:00
Erik Krogen fec7c5f3eb HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
 2019-03-13 13:19:18 -07:00
Shweta Yakkali 1ceefa726e HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020)
 2019-03-12 16:05:55 -07:00
Stephen O'Donnell a21e2e4dbc HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3)
 2019-03-12 10:18:56 -07:00
Erik Krogen 6d076dd5e8 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
 2019-03-07 08:42:41 -08:00
Wei-Chiu Chuang e58ccca3ce HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
 2019-03-04 10:45:31 -08:00
Erik Krogen af16db86d4 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:44 -08:00
Surendra Singh Lilhore 2e939515df HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f0)
 2019-02-21 20:47:50 +05:30
Surendra Singh Lilhore b93b127956 HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 
(cherry picked from commit 41e18feda3)
 2019-02-20 17:12:58 +05:30
Surendra Singh Lilhore f4b9ba230b HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 
(cherry picked from commit 29b411d5f0)
 2019-02-13 23:51:36 +05:30