Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,726 Commits 326 Branches 371 Tags
Commit Graph

4154 Commits

Author SHA1 Message Date
Yongjun Zhang
18f7628664 HDFS-10625. VolumeScanner to report why a block is found bad. Contributed by Rushabh S Shah and Yiqun Lin. 
(cherry picked from commit 5d1609ddf275e4907bd224bf618e2aad4b262888)
 2016-08-29 14:15:23 -07:00
Mingliang Liu
f84ae6f091 HDFS-10807. Doc about upgrading to a version of HDFS with snapshots may be confusing. (Contributed by Mingliang Liu) 
(cherry picked from commit 6742fb6e68d349055f985eb640d845e689d75384)
 2016-08-29 10:19:53 -07:00
Yongjun Zhang
6f10a0a87b HDFS-10652. Add a unit test for HDFS-4660. Contributed by Vinayakumar B., Wei-Chiu Chuang, Yongjun Zhang. 
(cherry picked from commit c25817159af17753b398956cfe6ff14984801b01)
 2016-08-27 23:11:57 -07:00
Andrew Wang
42e03d7bfe HDFS-10793. Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. Contributed by Manoj Govindassamy. 
(cherry picked from commit a445b82baaa58dcaaa3831e724a9915e9dde57c5)
(cherry picked from commit 1c47a356d8a29d95f9f991379d3bec92d28a728c)
 2016-08-26 15:40:13 -07:00
Zhe Zhang
96cf681b5d HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen. 
(cherry picked from commit 407b519fb14f79f19ebc4fbdf08204336a7acf77)
(cherry picked from commit f3a4cbcc60572c73e0d15580bae42a36c92fe247)
 2016-08-26 14:23:51 -07:00
Kihwal Lee
1c5ecb6cc4 HDFS-8915. TestFSNamesystem.testFSLockGetWaiterCount fails intermittently. Contributed by Masatake Iwasaki. 
(cherry picked from commit 13fb1b50e608558b2970184908ee5b9fcd7eb7b6)
 2016-08-26 10:01:19 -05:00
Xiaoyu Yao
db415ea166 HDFS-10748. TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout. Contributed by Yiqun Lin. 
(cherry picked from commit 4da5000dd33cf013e7212848ed2c44f1e60e860e)
 2016-08-25 09:22:16 -07:00
Wei-Chiu Chuang
bd729af8a8 HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-24 08:53:43 -07:00
Zhe Zhang
8bc33bf343 HADOOP-12765. HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections. Contributed by Min Shen. Branch-2 patch contributed by Wei-Chiu Chuang. 
(cherry picked from commit dfcbc12026b591745a7d7279f2b840152cb53a91)
 2016-08-23 14:46:47 -07:00
Jason Lowe
f57a26378e HDFS-9745. TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts. Contributed by Xiao Chen 
(cherry picked from commit 126d165efd80e266a8309241f3cf059e358f5019)
 2016-08-23 14:49:27 +00:00
Akira Ajisaka
dde3eaed62 HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin. 
(cherry picked from commit e90f3359de299ef5e3a54ca71070e3dfe1dbb98c)
(cherry picked from commit 94a844b4f61d4a47b2a5885e3db065a07ed29d97)
 2016-08-23 19:59:07 +09:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
c0c4cc8ee8 HDFS-10692. Update JDiff report's base version for HDFS from 2.6.0 to 2.7.2. Contributed by Wangda Tan. 
(cherry picked from commit dc7a1c54f4447fb01980ae61d5d67e90c4f52f00)
 2016-08-22 11:37:14 -07:00
Kihwal Lee
c9199095f8 HDFS-10738. Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure. Contributed by Rakesh R. 
(cherry picked from commit 0f701f433dd3be233bf53e856864c82349e8274e)
(cherry picked from commit 7f1879abe642219fce0328a8e96c983601d202b0)
 2016-08-22 11:04:03 -05:00
Jing Zhao
8628ad0790 HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal. 
(cherry picked from commit 0faee62a0c8c1b8fd83227babfd00fbc2b26bddf)
(cherry picked from commit fff0418458b23c3096384082f8f387b451efad09)
 2016-08-19 22:27:24 -07:00
Kihwal Lee
4f3df8e22a HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 
(cherry picked from commit ae90d4dd908cf3f9e9ff26fa8e92f028057a9ca1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-19 15:32:11 -05:00
Arpit Agarwal
1fe08c919a HDFS-10682. Replace FsDatasetImpl object lock with a separate lock object. (Contributed by Chen Liang) 2016-08-17 16:22:07 -07:00
Xiao Chen
b89d79ca1d HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin. 
(cherry picked from commit 85aacaadb5a3f8c78b191867c0bde09b3c4b3c3c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java

(cherry picked from commit a20b943cf951cf38eb4950177c826bbcf424aade)
 2016-08-17 16:00:33 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
d65024eddc Revert "HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla." 
Reverting this on branch-2.* as it's an incompatible change.

This reverts commit 83f7f62be379045ad6933689b21b76c7086f919d.

(cherry picked from commit 2486c4c63a35fcef7338ea63f0d8aafa778cd05d)
 2016-08-17 13:34:35 -07:00
Kihwal Lee
b76f6e5de3 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 
(cherry picked from commit b10c936020e2616609dcb3b2126e8c34328c10ca)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 65a41696b2a22834f4592bcb49e40fffd2dbb96e)
 2016-08-16 09:39:33 -05:00
Mingliang Liu
54c9743263 HDFS-10641. TestBlockManager#testBlockReportQueueing fails intermittently. (Contributed by Daryn Sharp) 2016-08-15 20:46:48 -07:00
Mingliang Liu
e9eaad0e67 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:46:34 -07:00
Mingliang Liu
0b934c375e HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:46:28 -07:00
Kihwal Lee
1a8280edde HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 
(cherry picked from commit 864f878d5912c82f3204f1582cfb7eb7c9f1a1da)
(cherry picked from commit e78db7d2a430983807750666fb72ebd5c97ce867)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2016-08-15 17:37:39 -05:00
Kihwal Lee
579709fc22 HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 
(cherry picked from commit 83e57e083f2cf6c0de8a46966c5492faeabd8f2a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java

(cherry picked from commit 4766a3153dd517ac832d4761c884ed88a83a6c09)
 2016-08-15 13:13:49 -05:00
Wei-Chiu Chuang
53f097a474 HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9019606b69bfb7019c8642b6cbcbb93645cc19e3)
(cherry picked from commit 20a8ff0a8d17c05305f712da887ad6665eb654b5)
 2016-08-12 06:11:22 -07:00
Zhe Zhang
9c1c48c0d4 HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 
(cherry picked from commit 3e8099a45a4cfd4c5c0e3dce4370514cb2c90da9)
(cherry picked from commit 00ff3d737c71e1a228ede64a6b9abd5227ef2117)
 2016-08-11 12:42:17 -07:00
Akira Ajisaka
f29d0739d7 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 
(cherry picked from commit 932aed64d77edcc8483a95c1ce31a4c9ae679446)
(cherry picked from commit 0529da723966e03c7dfa8201a515320c904d32a7)
 2016-08-11 12:42:17 -07:00
Zhe Zhang
b8b4ea67d8 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
(cherry picked from commit ccff6035f50769eb69701128ae61efc69e82609d)
(cherry picked from commit 7d402692b4b06af0459ea81129e93a5182291a27)
 2016-08-11 12:42:17 -07:00
Wei-Chiu Chuang
65e3657149 HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 
(cherry picked from commit f1d5a95cf31f593f362a4e9c0afa7587f6e14957)
 2016-08-10 22:30:06 -07:00
Xiaoyu Yao
7a1a3f7754 HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit ec289bbeceff064ad24e189db20a3e0a296822c1)
 2016-08-10 15:56:56 -07:00
Xiaoyu Yao
8808df09c6 Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca74c6e6c43ecfc8770228c518620c9cb.

(cherry picked from commit fdafd0cb4f1626ee41539f4b80711c4e78e3e16c)
 2016-08-10 15:56:43 -07:00
Xiaoyu Yao
c0e331935d Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit 9f816d7ca74c6e6c43ecfc8770228c518620c9cb)
 2016-08-10 15:19:10 -07:00
Wei-Chiu Chuang
753edc4931 HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 
(cherry picked from commit 8efd4959f3fd48fe281aa26a46668071461dee8b)
 2016-08-10 11:39:12 -07:00
Yuanbo Liu
d46f5ca7e2 HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 
(cherry picked from commit 10e84c6a6e831fe2bea061fb21bd0dfe32bc9953)
 2016-08-10 11:12:21 -07:00
Konstantin V Shvachko
6b3114a2a5 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 
(cherry picked from commit 4d3af47f2765f6f57936d316ef2a4150b787cc97)
 2016-08-07 14:36:24 -07:00
Kihwal Lee
5b3deac2e3 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit f715f141856cb6a4c6574893f40f9865653b631e)
 2016-08-05 11:43:43 -05:00
Akira Ajisaka
5758026833 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 
(cherry picked from commit ef432579a7763cc0e482fe049027c6e5325eb034)
(cherry picked from commit 68638661271e6e69c220b19fec7cbc7ae5e6208e)
 2016-08-05 17:33:07 +09:00
Kihwal Lee
e8cd1439d3 HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit 57369a678c4c51627fe9a654e697a906a6bef123)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 690ec789f0448d9d2b908562f54a1731ed9e97aa)
 2016-08-04 15:27:51 -05:00
Kihwal Lee
eb9af6ca34 HDFS-10717. Fix findbugs warnings of hadoop-hdfs-client in branch-2. Contributed by Akira Ajisaka. 
(cherry picked from commit 6ead6f3bb83b5c27d884401fb950b5b629fec7ec)
 2016-08-04 12:36:51 -05:00
Arpit Agarwal
f3424d662a HDFS-10722. Fix build break. 2016-08-04 10:28:43 -07:00
Kihwal Lee
3a85aea42d HDFS-10722. Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs. Contributed by Daryn Sharp. 
(cherry picked from commit 4a8e6dc02fb1d0dc89c3981b444575494511444a)
 2016-08-04 11:56:26 -05:00
Tsz-Wo Nicholas Sze
cefa21e98a In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:54:15 -07:00
Kihwal Lee
58df27b873 HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 
(cherry picked from commit 6ae39199dac6ac7be6802b31452552c76da16e24)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

(cherry picked from commit 77b61d1f4e34de9fed3dd6167fa93bb35e85e038)
 2016-08-04 09:45:03 -05:00
Kihwal Lee
804db5b277 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 
(cherry picked from commit c0166b074382576da2aaa0eb76965abe91ed4115)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-04 09:35:16 -05:00
Kihwal Lee
ceac628378 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 
(cherry picked from commit 6f63566694f8cec64a469448a8fa00ce921ce367)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java

(cherry picked from commit 1ca2ab4013268ccab72e88a311a37262dcd867b9)
 2016-08-03 15:28:13 -05:00
Jing Zhao
5bdee64bde HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 
(cherry picked from commit f4ba5ff1d70ef92d59851c09c4bd4b43d6c04971)
(cherry picked from commit 180a26d9fb2b3f4a0632e1379f867020326c7702)
 2016-08-03 12:44:49 -07:00
Kihwal Lee
33a87ffe10 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 
(cherry picked from commit 58db263e93daf08280e6a586a10cebd6122cf72a)
 2016-08-03 12:19:05 -05:00
Kihwal Lee
288f9ccde2 HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 
(cherry picked from commit 5e5b8793fba8e25aeba7a74878da4cf8e806f061)
 2016-08-02 15:43:11 -05:00
Konstantin V Shvachko
9b3b7703b1 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 9ab153d7ae47c9998816950745b04824b3332608.
 2016-08-01 22:48:09 -07:00
Daryn Sharp
ecb143ed21 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 
(cherry picked from commit 9f473cf903e586c556154abd56b3a3d820c6b028)
 2016-08-01 10:18:45 -05:00