Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,257 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7357 Commits

Author SHA1 Message Date
Xiaoyu Yao fbdfb36fdf
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 29e6a97faa.
 2019-09-05 16:52:09 -07:00
Zhankun Tang ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Xiaoyu Yao 29e6a97faa
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:58:59 -07:00
Stephen O'Donnell 63b18d764a HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0)
(cherry picked from commit 2c7f49743fba587e32398c81412db9eac426cf98)
 2019-09-02 09:58:48 -07:00
Wei-Chiu Chuang 8ffaef78da Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit d42b48d4d0.
 2019-09-02 09:58:32 -07:00
Istvan Fajth e3c01e174c HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-30 13:13:48 -07:00
Ayush Saxena e44693953d HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:59:52 +05:30
leosunli 0011136e66 Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit 915271245b)
(cherry picked from commit b516d2ea7b)
 2019-08-29 18:46:25 -07:00
Stephen O'Donnell d42b48d4d0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9)
(cherry picked from commit 9c0d6e1657)
 2019-08-29 18:26:14 -07:00
Jonathan Hung 79364a9c53 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 
(cherry picked from commit 8ab7020e64)
(cherry picked from commit d255efa5f8)
 2019-08-27 10:31:21 -07:00
Chen Liang d2bb764ffa HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:29:23 -07:00
Erik Krogen 9dc921f5e5 HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 
(cherry picked from d699022fce)
(cherry picked from 2408c2491f)
 2019-08-23 11:48:51 -07:00
Chen Liang 0839cf1d4e HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang ef9b0a1d9a HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang 3135cc35e3 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-23 10:11:22 -07:00
Chen Liang 3835a6c4e8 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Akira Ajisaka 83f67cca60
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5)
 2019-08-23 15:29:25 +09:00
Akira Ajisaka b40aa71cf0
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4ca)
 2019-08-23 15:08:34 +09:00
Xiao Chen a5690b29a7
HDFS-13642. Creating a file with block size smaller than EC policy's cell size should fail. 
(cherry picked from commit cf4108313d)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
 2019-08-23 15:03:57 +09:00
Ayush Saxena 900384cf57 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:54:50 +05:30
Wei-Chiu Chuang 2ce0914176 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit e978c6c9ed.
 2019-08-21 09:00:59 -07:00
Surendra Singh Lilhore c2aeeb01fa HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
(cherry picked from commit 03c62c7989)
 2019-08-21 09:00:52 -07:00
Siyao Meng e0a5fee7f5 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1291) Contributed by Siyao Meng. 2019-08-20 10:49:51 -07:00
Wei-Chiu Chuang 7a1f508f58 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
(cherry picked from commit f78dcbaae5535e3e05c9f29986b8963ba262baa9)
 2019-08-20 10:36:55 -07:00
Wei-Chiu Chuang d8c60f829f HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde2)
(cherry picked from commit b004d305e9)
 2019-08-20 10:34:25 -07:00
Wei-Chiu Chuang fc062b6fef HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342)
(cherry picked from commit 6d93886231)
 2019-08-19 13:11:21 -07:00
HUAN-PING SU 18f98491e7 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2)
(cherry picked from commit abda503040)
 2019-08-19 11:15:48 -07:00
Wei-Chiu Chuang e978c6c9ed HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 
(cherry picked from commit 8b802d6b44)
 2019-08-19 08:39:00 -07:00
Surendra Singh Lilhore 92c3a581ec HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
(cherry picked from commit 6b01effd01)
 2019-08-18 18:17:50 -07:00
Shashikant Banerjee d18b720c70 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959c)
(cherry picked from commit e89413da88)
 2019-08-17 03:07:03 -07:00
Siyao Meng 224643a58c HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b9)
(cherry picked from commit 136a97a74dbc12f05b88d0abda101690e7c727d9)
 2019-08-14 07:40:48 -07:00
Wei-Chiu Chuang 6966b76230 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
(cherry picked from commit 4784165bb24228d13f4e738e0093ab0dade0bff1)
 2019-08-13 17:15:10 -07:00
Masatake Iwasaki e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)
 2019-08-14 09:07:00 +09:00
Siyao Meng b040eb91c7 HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a)
(cherry picked from commit c5f433b11f)
 2019-08-12 17:30:32 -07:00
Yiqun Lin f2007818fc HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c7)
(cherry picked from commit a7bc09c596)
 2019-08-09 18:41:18 -07:00
Wei-Chiu Chuang 0aa847e158 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494)
(cherry picked from commit 2f2fa3dad6)
 2019-08-08 13:51:23 -07:00
Stephen O'Donnell 163fb887a6 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf)
(cherry picked from commit a00a327594)
 2019-08-08 13:37:22 -07:00
Chao Sun 90447baeb0 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
(cherry picked from d38b617baa)
 2019-08-08 10:42:27 -07:00
Erik Krogen f6ce2f4a50 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
 2019-08-07 09:33:36 -07:00
Lokesh Jain d06fca0104 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
(cherry picked from commit f6fe18a282)
 2019-08-06 15:31:15 -07:00
Stephen O'Donnell 99af4ce0ac HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 13:53:01 -07:00
Erik Krogen 69e3ac5cbb HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a807)
 2019-08-06 09:49:23 -07:00
Wei-Chiu Chuang a7a348446d Revert "HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell." 
This reverts commit 45ba41570b.
 2019-08-06 08:57:00 -07:00
Stephen O'Donnell 45ba41570b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
(cherry picked from commit 8414e6354b)
 2019-08-06 07:58:23 -07:00
Erik Krogen da548cf263 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
 2019-08-02 11:51:22 -07:00
Wei-Chiu Chuang 68d618acd2 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
(cherry picked from commit e1659fe009)
 2019-08-02 10:52:05 -07:00
Wei-Chiu Chuang d66ee4bf61 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98)
(cherry picked from commit 5c1f91fd97)
 2019-08-01 15:49:14 -07:00
Christopher Gregorian 7b8f08f59e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
 2019-07-30 14:45:00 -07:00
Wei-Chiu Chuang 608e7044a5 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388)
(cherry picked from commit d28e624958)
 2019-07-29 14:33:08 -07:00
Yiqun Lin 2e1afa9a86 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b)
(cherry picked from commit 69f1c8d77b)
 2019-07-29 06:50:41 -07:00