Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,876 Commits 326 Branches 371 Tags
Commit Graph

4122 Commits

Author SHA1 Message Date
Kihwal Lee
1ef8d7a638 HDFS-10744. Internally optimize path component resolution. Contributed by Daryn Sharp. 
(cherry picked from commit 03dea65e0b17ca2f9460bb6110f6ab3a321b8bf2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
 2016-08-15 17:01:40 -05:00
Kihwal Lee
4766a3153d HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 
(cherry picked from commit 83e57e083f2cf6c0de8a46966c5492faeabd8f2a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
 2016-08-15 13:09:50 -05:00
Wei-Chiu Chuang
20a8ff0a8d HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9019606b69bfb7019c8642b6cbcbb93645cc19e3)
 2016-08-12 06:02:47 -07:00
Zhe Zhang
00ff3d737c HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 
(cherry picked from commit 3e8099a45a4cfd4c5c0e3dce4370514cb2c90da9)
 2016-08-11 11:32:54 -07:00
Akira Ajisaka
0529da7239 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 
(cherry picked from commit 932aed64d77edcc8483a95c1ce31a4c9ae679446)
 2016-08-11 10:55:53 -07:00
Zhe Zhang
7d402692b4 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
(cherry picked from commit ccff6035f50769eb69701128ae61efc69e82609d)
 2016-08-11 10:53:16 -07:00
Wei-Chiu Chuang
f1d5a95cf3 HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 2016-08-10 22:24:21 -07:00
Xiaoyu Yao
de6eafc696 HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit ec289bbeceff064ad24e189db20a3e0a296822c1)
 2016-08-10 15:55:12 -07:00
Xiaoyu Yao
c341280129 Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca74c6e6c43ecfc8770228c518620c9cb.

(cherry picked from commit fdafd0cb4f1626ee41539f4b80711c4e78e3e16c)
 2016-08-10 15:54:59 -07:00
Xiaoyu Yao
d583a357f9 Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit 9f816d7ca74c6e6c43ecfc8770228c518620c9cb)
 2016-08-10 15:15:51 -07:00
Wei-Chiu Chuang
8efd4959f3 HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 2016-08-10 11:34:59 -07:00
Yuanbo Liu
44162e555a HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 
(cherry picked from commit 10e84c6a6e831fe2bea061fb21bd0dfe32bc9953)
 2016-08-10 11:10:00 -07:00
Kihwal Lee
7f1879abe6 HDFS-10738. Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure. Contributed by Rakesh R. 
(cherry picked from commit 0f701f433dd3be233bf53e856864c82349e8274e)
 2016-08-09 14:55:46 -05:00
Lei Xu
e7c701586d HDFS-4176. EditLogTailer should call rollEdits with a timeout. (lei) 2016-08-08 16:32:01 -07:00
Lei Xu
9cb6d291ea HDFS-10457. DataNode should not auto-format block pool directory if VERSION is missing. (Wei-Chiu Chuang via lei) 
(cherry picked from commit bb3bcb9397593fc8a2fa63a48eba126609f72c42)
 2016-08-08 16:19:58 -07:00
Konstantin V Shvachko
ab130f97ca HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 
(cherry picked from commit 4d3af47f2765f6f57936d316ef2a4150b787cc97)
 2016-08-07 14:35:35 -07:00
Akira Ajisaka
6863866127 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 
(cherry picked from commit ef432579a7763cc0e482fe049027c6e5325eb034)
 2016-08-05 17:32:23 +09:00
Kihwal Lee
09b06a6e9e HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 
(cherry picked from commit 438a9f047eb6af2a4b916a4f6ef6f68adeab8068)
 2016-08-04 16:17:41 -05:00
Kihwal Lee
690ec789f0 HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit 57369a678c4c51627fe9a654e697a906a6bef123)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 2016-08-04 15:12:21 -05:00
Kihwal Lee
6ead6f3bb8 HDFS-10717. Fix findbugs warnings of hadoop-hdfs-client in branch-2. Contributed by Akira Ajisaka. 2016-08-04 12:34:54 -05:00
Kihwal Lee
3b2554f3b0 HDFS-10722. Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs. Contributed by Daryn Sharp. 
(cherry picked from commit 4a8e6dc02fb1d0dc89c3981b444575494511444a)
 2016-08-04 11:55:52 -05:00
Tsz-Wo Nicholas Sze
5004b0571e In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:53:15 -07:00
Kihwal Lee
77b61d1f4e HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 
(cherry picked from commit 6ae39199dac6ac7be6802b31452552c76da16e24)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
 2016-08-04 09:43:56 -05:00
Kihwal Lee
c0166b0743 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 2016-08-04 09:28:30 -05:00
Kihwal Lee
1ca2ab4013 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 
(cherry picked from commit 6f63566694f8cec64a469448a8fa00ce921ce367)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
 2016-08-03 15:27:03 -05:00
Jing Zhao
180a26d9fb HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 
(cherry picked from commit f4ba5ff1d70ef92d59851c09c4bd4b43d6c04971)
 2016-08-03 12:42:44 -07:00
Kihwal Lee
a5d12d9c1f HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 
(cherry picked from commit 22ef5286bc8511ddee9594b7cecc598bf41a850b)
 2016-08-03 13:15:28 -05:00
Kihwal Lee
c15959fa60 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 
(cherry picked from commit 58db263e93daf08280e6a586a10cebd6122cf72a)
 2016-08-03 12:18:15 -05:00
Kihwal Lee
ab7072838f HDFS-10656. Optimize conversion of byte arrays back to path string. Contributed by Daryn Sharp. 
(cherry picked from commit bebf10d2455cad1aa8985553417d4d74a61150ee)
 2016-08-03 11:54:51 -05:00
Kihwal Lee
dc0a17e828 HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 
(cherry picked from commit 5e5b8793fba8e25aeba7a74878da4cf8e806f061)
 2016-08-02 15:39:03 -05:00
Konstantin V Shvachko
24574f0e16 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 0dde4035c9303ea1531c8d61f81ad9de2d630aa3.
 2016-08-01 22:44:32 -07:00
Daryn Sharp
4ad2a73df6 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 
(cherry picked from commit 9f473cf903e586c556154abd56b3a3d820c6b028)
 2016-08-01 10:17:24 -05:00
Yongjun Zhang
f5bc6dd478 HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 8d32bd884d53948728b77de14531b3be196f4bc7)
 2016-07-29 23:39:38 -07:00
Akira Ajisaka
0cff416c35 HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 
(cherry picked from commit 204a2055b1b9270ae13ea03b7aeac62b65166efd)
 2016-07-29 15:40:59 +09:00
Xiao Chen
24d464a150 HDFS-9276. Failed to Update HDFS Delegation Token for long running application in HA mode. Contributed by Liangliang Gu and John Zhuge 
(cherry picked from commit d9aae22fdf2ab22ae8ce4a9d32ac71b3dde084d3)
 2016-07-28 16:41:31 -07:00
Xiaoyu Yao
f03ad5677a HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:16:21 -07:00
Akira Ajisaka
ab49c2b399 HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 
(cherry picked from commit 414fbfab41470923eb82a21628709e51fd3a3f6e)
 2016-07-28 16:59:25 +09:00
Yongjun Zhang
bd84cf3036 HDFS-10667. Report more accurate info about data corruption location. (Yuanbo Liu via Yongjun Zhang) 
(cherry picked from commit eb7ff0c9927131f4a797148b970a95a1abf7d847)
 2016-07-27 15:23:38 -07:00
Brahma Reddy Battula
096d3fa25c HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 2016-07-27 10:28:31 +05:30
Arpit Agarwal
c08c7b7a94 HDFS-10642. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. (Contributed by Mingliang Liu) 2016-07-26 12:44:36 -07:00
Wei-Chiu Chuang
0ee1644e39 HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. Contributed by Kai Sasaki. 
(cherry picked from commit 25f0a9b3ab136d44f70eb2805598e2df8325612a)
 2016-07-26 08:05:14 -07:00
Steve Loughran
e5a4b6cb94 Revert "HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu." 
This reverts commit b3ce778268a441b742b625594d1a55b83960f355.
 2016-07-26 14:30:31 +01:00
Masatake Iwasaki
f4207d0336 HDFS-10671. Fix typo in HdfsRollingUpgrade.md. Contributed by Yiqun Lin. 
(cherry picked from commit 59466b8c180716dda7aa670728580a88e54eb4d2)
 2016-07-26 19:50:39 +09:00
Brahma Reddy Battula
b3ce778268 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 7cac7655fd84ac394250705b31e3927fe548e34a)
 2016-07-26 11:02:23 +05:30
Akira Ajisaka
a04b193374 Revert "HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg." 
This reverts commit 2685e16f381532d53aea413ce1c0a14181ee7c89.
 2016-07-26 11:04:05 +09:00
Vinitha Reddy Gankidi
0dde4035c9 HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:55:00 -07:00
Jing Zhao
30c772fc59 HDFS-10688. BPServiceActor may run into a tight loop for sending block report when hitting IOException. Contributed by Chen Liang. 
(cherry picked from commit 0cde9e12a7175e4d8bc4ccd5c36055b280d1fbd6)
 2016-07-25 18:43:25 -07:00
Xiao Chen
1a115ea317 HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 132deb4cacc413a85a6af2f390ec79a76c91961c)
 2016-07-22 15:04:17 -07:00
Arpit Agarwal
2f1e352912 HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-22 12:12:29 -07:00
Akira Ajisaka
1222889f1a HDFS-10287. MiniDFSCluster should implement AutoCloseable. Contributed by Andras Bokor. 
(cherry picked from commit fcde6940e0cbdedb1105007e4857137ecdfa1284)
 2016-07-21 11:28:19 -07:00