Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,155 Commits 326 Branches 371 Tags
Commit Graph

8254 Commits

Author SHA1 Message Date
Akira Ajisaka
e8b93d04c4
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5d4ff351645e34c0ef09b7ba82f4285)
 2019-08-23 15:28:54 +09:00
Akira Ajisaka
d3e5898712
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4caea0e197568ee28392c46f1ce0d42e1)
 2019-08-23 14:45:57 +09:00
Ayush Saxena
b7a66471ca HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:35:48 +05:30
Wei-Chiu Chuang
933ea6aa4d Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8b802d6b444b95791ba156c7bb307dd584cc9dba.
 2019-08-21 09:00:23 -07:00
Surendra Singh Lilhore
03c62c7989 HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
 2019-08-20 15:54:46 -07:00
Wei-Chiu Chuang
77fd690c05 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
 2019-08-20 10:37:05 -07:00
Wei-Chiu Chuang
b004d305e9 HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde289719e0b300e11568c5e68c36b5d05)
 2019-08-20 10:33:59 -07:00
Wei-Chiu Chuang
6d93886231 HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342f3c8cb8246f011abb9bcb0b6ef3eaa)
 2019-08-19 13:11:11 -07:00
HUAN-PING SU
abda503040 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2760500b7e7d4414a43069ed4a45930)
 2019-08-19 11:15:37 -07:00
Wei-Chiu Chuang
8b802d6b44 HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-19 08:38:06 -07:00
Surendra Singh Lilhore
6b01effd01 HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a9812023754ed1b3e5b428e161f0add50)
 2019-08-18 18:17:41 -07:00
Shashikant Banerjee
e89413da88 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959ce505f0659e5c69d0ca83a5dce0a7c2)
 2019-08-16 19:09:34 -07:00
Siyao Meng
c6e4906ffc HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b933b7acfe55081f5bad46f9fe05a14b)
 2019-08-16 19:08:05 -07:00
Chen Liang
d8a6098a96 HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang
f507bc059d HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-16 10:50:00 -07:00
Chen Liang
ca15968d09 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-16 10:50:00 -07:00
Chen Liang
22d81f7fd0 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-16 10:49:59 -07:00
Wei-Chiu Chuang
f5661b630a HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
 2019-08-13 17:15:19 -07:00
Masatake Iwasaki
d7ca016d63 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe0473e353ee2d489156248a01aa982dfd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
 2019-08-14 08:45:10 +09:00
Siyao Meng
88aece4083 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1267) Contributed by Siyao Meng. 
(cherry picked from commit 6ae8bc3a4a07c6b4e7060362b749be8c7afe0560)
 2019-08-13 16:36:52 -07:00
Siyao Meng
c5f433b11f HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a078ce7fb4e2a852e315de5b6410082)
 2019-08-12 17:30:18 -07:00
Yiqun Lin
a7bc09c596 HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c734bea0eb490f30c908748458c190c3)
 2019-08-09 18:41:03 -07:00
Wei-Chiu Chuang
2f2fa3dad6 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494c3aea146d7741bf0ad52ce16ad08e6)
 2019-08-08 13:51:14 -07:00
Stephen O'Donnell
a00a327594 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf6e92be540f5665bb571418b916d789)
 2019-08-08 13:37:10 -07:00
Chao Sun
d38b617baa HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d74029b6ae82263739f598ceb25c597dcd)
 2019-08-08 10:15:48 -07:00
Erik Krogen
016aa13940 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e24be294b40088a8aa46086ba8ca4ba8)
 2019-08-07 09:32:32 -07:00
Lokesh Jain
f6fe18a282 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360e083010f395cb9d3bd46701417e7b7)
 2019-08-06 15:31:05 -07:00
Inigo Goiri
3e8e1570f6 HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from 5740eea0818f2ba334b99eed0f085300e6b16905)
 2019-08-06 09:26:01 -07:00
Stephen O'Donnell
8414e6354b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7bc3149bf8b941fd85d7807d9906d27b)
 2019-08-06 07:58:12 -07:00
Erik Krogen
2634aaaf9a HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83c1fb933d409c514d2155e986e4e25b)
 2019-08-02 12:00:48 -07:00
Wei-Chiu Chuang
e1659fe009 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
 2019-08-02 10:51:55 -07:00
Wei-Chiu Chuang
5c1f91fd97 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98a7ab70741a2efc98eff548c1e431c1)
 2019-08-01 15:49:05 -07:00
Christopher Gregorian
d4492bdd9e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a4e3b8f165806dfd2966f0951a5e575.

(cherry-picked from f96a2df38d889f29314c57f4d94227b2e419a11f)
 2019-07-30 14:41:36 -07:00
Wei-Chiu Chuang
d28e624958 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388fec17a40856f36821c142be32aa364)
 2019-07-29 14:32:46 -07:00
Yiqun Lin
69f1c8d77b HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b7d40ac832ae3e23b64e8541b62c4bd)
 2019-07-29 06:50:28 -07:00
Erik Krogen
bd5033e6f9 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit 2478d333a6dc3123f4e84d4bbb6991f1cbda1ea6.
 2019-07-26 11:36:06 -07:00
Masatake Iwasaki
2478d333a6 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97251267253fa045ba03c4749f95f530)
 2019-07-26 09:48:00 +09:00
Masatake Iwasaki
9ed2c22d57
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00dd843a1a6c9d8b616631cdcf24c00e82498dab)
 2019-07-12 10:08:18 +09:00
Inigo Goiri
950aa74d5f HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
 2019-07-10 11:07:15 -07:00
Erik Krogen
491dc7cc9d HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148ed7dec75810feddb22ad98e4670483)
 2019-07-08 15:15:41 -07:00
Wei-Chiu Chuang
29b9049bde HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 
(cherry picked from commit ec851e4db24fad68c1d70a981b4253c0207abc45)
 2019-07-08 15:25:10 +08:00
leosunli
adec307251 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c907294860a48f0d1676a31bda15795b7a6771a)
 2019-07-07 20:02:08 -07:00
Masatake Iwasaki
ee81a64da7 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec78c8dab9111f3ba2709cef20794b73a)
 2019-07-04 23:53:30 +09:00
Weiwei Yang
fa8b8edbac HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 
(cherry picked from commit 729cb3aefe71d7f728c7edea78ce7f268a1fdecb)
 2019-07-04 18:06:19 +08:00
Wei-Chiu Chuang
66920e6d37 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-28 14:28:35 -07:00
Erik Krogen
e53cee0c14 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-06-28 14:28:28 -07:00
Erik Krogen
3c6b74a6a4 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:28:17 -07:00
Erik Krogen
e1e4056353 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-06-28 14:28:10 -07:00
Erik Krogen
98434bed5c HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:28:03 -07:00
Erik Krogen
73d56b9d2a HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-06-28 14:27:54 -07:00