Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,218 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7335 Commits

Author SHA1 Message Date
Siyao Meng e0a5fee7f5 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1291) Contributed by Siyao Meng. 2019-08-20 10:49:51 -07:00
Wei-Chiu Chuang 7a1f508f58 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
(cherry picked from commit f78dcbaae5535e3e05c9f29986b8963ba262baa9)
 2019-08-20 10:36:55 -07:00
Wei-Chiu Chuang d8c60f829f HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde2)
(cherry picked from commit b004d305e9)
 2019-08-20 10:34:25 -07:00
Wei-Chiu Chuang fc062b6fef HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342)
(cherry picked from commit 6d93886231)
 2019-08-19 13:11:21 -07:00
HUAN-PING SU 18f98491e7 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2)
(cherry picked from commit abda503040)
 2019-08-19 11:15:48 -07:00
Wei-Chiu Chuang e978c6c9ed HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 
(cherry picked from commit 8b802d6b44)
 2019-08-19 08:39:00 -07:00
Surendra Singh Lilhore 92c3a581ec HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
(cherry picked from commit 6b01effd01)
 2019-08-18 18:17:50 -07:00
Shashikant Banerjee d18b720c70 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959c)
(cherry picked from commit e89413da88)
 2019-08-17 03:07:03 -07:00
Siyao Meng 224643a58c HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b9)
(cherry picked from commit 136a97a74dbc12f05b88d0abda101690e7c727d9)
 2019-08-14 07:40:48 -07:00
Wei-Chiu Chuang 6966b76230 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
(cherry picked from commit 4784165bb24228d13f4e738e0093ab0dade0bff1)
 2019-08-13 17:15:10 -07:00
Masatake Iwasaki e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)
 2019-08-14 09:07:00 +09:00
Siyao Meng b040eb91c7 HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a)
(cherry picked from commit c5f433b11f)
 2019-08-12 17:30:32 -07:00
Yiqun Lin f2007818fc HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c7)
(cherry picked from commit a7bc09c596)
 2019-08-09 18:41:18 -07:00
Wei-Chiu Chuang 0aa847e158 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494)
(cherry picked from commit 2f2fa3dad6)
 2019-08-08 13:51:23 -07:00
Stephen O'Donnell 163fb887a6 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf)
(cherry picked from commit a00a327594)
 2019-08-08 13:37:22 -07:00
Chao Sun 90447baeb0 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
(cherry picked from d38b617baa)
 2019-08-08 10:42:27 -07:00
Erik Krogen f6ce2f4a50 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
 2019-08-07 09:33:36 -07:00
Lokesh Jain d06fca0104 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
(cherry picked from commit f6fe18a282)
 2019-08-06 15:31:15 -07:00
Stephen O'Donnell 99af4ce0ac HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 13:53:01 -07:00
Erik Krogen 69e3ac5cbb HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a807)
 2019-08-06 09:49:23 -07:00
Wei-Chiu Chuang a7a348446d Revert "HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell." 
This reverts commit 45ba41570b.
 2019-08-06 08:57:00 -07:00
Stephen O'Donnell 45ba41570b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
(cherry picked from commit 8414e6354b)
 2019-08-06 07:58:23 -07:00
Erik Krogen da548cf263 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
 2019-08-02 11:51:22 -07:00
Wei-Chiu Chuang 68d618acd2 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
(cherry picked from commit e1659fe009)
 2019-08-02 10:52:05 -07:00
Wei-Chiu Chuang d66ee4bf61 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98)
(cherry picked from commit 5c1f91fd97)
 2019-08-01 15:49:14 -07:00
Christopher Gregorian 7b8f08f59e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
 2019-07-30 14:45:00 -07:00
Wei-Chiu Chuang 608e7044a5 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388)
(cherry picked from commit d28e624958)
 2019-07-29 14:33:08 -07:00
Yiqun Lin 2e1afa9a86 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b)
(cherry picked from commit 69f1c8d77b)
 2019-07-29 06:50:41 -07:00
Inigo Goiri fe40fbbd4d HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f)
 2019-07-26 15:56:44 -07:00
Erik Krogen 42f10712a5 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit bf4a2258ce.
 2019-07-26 11:36:59 -07:00
Masatake Iwasaki bf4a2258ce HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
 2019-07-26 09:49:42 +09:00
Chen Liang 2a1451aa0e HDFS-13791. Amend fix LOG string format. 2019-07-12 12:58:08 -07:00
Wei-Chiu Chuang 3ba7826d41 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-07-12 12:26:23 -07:00
Erik Krogen cc22373606 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-12 12:25:19 -07:00
Erik Krogen 769b5a802b HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen 35200b3671 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen 98684fb821 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 12:24:43 -07:00
Erik Krogen ee0416b456 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-07-12 12:24:43 -07:00
Konstantin V Shvachko 321e26ced1 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-07-12 12:24:43 -07:00
Chao Sun b028d62d1f HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-07-12 12:24:42 -07:00
Konstantin V Shvachko b8f9739024 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-07-12 12:24:42 -07:00
Chao Sun 8f36bdc3b9 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Chao Sun 7a650b018c HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Erik Krogen 6c1bf61628 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-12 12:24:40 -07:00
Erik Krogen 67c4768825 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-12 12:24:36 -07:00
Chen Liang 8e77349bee HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2019-07-12 12:23:56 -07:00
Erik Krogen 265bd68464 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-07-12 12:23:17 -07:00
Chen Liang 6e54d97458 HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2019-07-12 12:23:17 -07:00
Konstantin V Shvachko 6cb1b5aa2f HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. 2019-07-12 12:23:16 -07:00
Chao Sun c4d944aeb3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-07-12 12:23:16 -07:00