Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,028 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4273 Commits

Author SHA1 Message Date
Arpit Agarwal 162ee0f0a4 HDFS-9038. DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) 2016-09-06 13:52:05 -07:00
Xiao Chen e84b5c5e3e HDFS-10841. Remove duplicate or unused variable in appendFile(). Contributed by Kihwal Lee. 
(cherry picked from commit f6c0b7543f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java

(cherry picked from commit 3025992bd3)
 2016-09-06 11:54:02 -07:00
Andrew Wang 9b7e079d4b HDFS-10833. Fix JSON errors in WebHDFS.md examples. 
(cherry picked from commit cbd909ce2a)
(cherry picked from commit 40eab77562)
 2016-09-02 14:51:12 -07:00
Kihwal Lee 16c166249c HDFS-10772. Reduce byte/string conversions for get listing. Contributed by Daryn Sharp. 
(cherry picked from commit a1f3293762)
 2016-09-02 12:32:34 -05:00
Kihwal Lee 1c946a2665 HDFS-10768. Optimize mkdir ops. Contributed by Daryn Sharp. 
(cherry picked from commit 442dfd1bbe)
 2016-09-02 12:28:11 -05:00
Kihwal Lee bf350e4480 HDFS-10762. Pass IIP for file status related methods 
(cherry picked from commit a30f6a68fa)
 2016-09-02 12:23:21 -05:00
Kihwal Lee c2708bc5fb HDFS-9621. Consolidate FSDirStatAndListingOp#createFileStatus to let its INodesInPath parameter always include the target INode. Contributed by Jing Zhao. 
(cherry picked from commit 313f03bfda)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
 2016-09-02 12:16:08 -05:00
Kihwal Lee 69bdcd9b1e HDFS-10745. Directly resolve paths into INodesInPath. Contributed by Daryn Sharp. 
(cherry picked from commit 922d045e11)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
 2016-09-02 11:44:01 -05:00
Kihwal Lee 3dac73080a HDFS-10744. Internally optimize path component resolution. Contributed by Daryn Sharp. 
(cherry picked from commit 1ef8d7a638)
 2016-09-02 11:31:48 -05:00
Kihwal Lee 5a779cce8e HDFS-10711. Optimize FSPermissionChecker group membership check. Contributed by Daryn Sharp. 
(cherry picked from commit 43d1279df0)
 2016-09-02 11:24:20 -05:00
Kihwal Lee f09b04b3f5 HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 
(cherry picked from commit 438a9f047e)
(cherry picked from commit 09b06a6e9e)
 2016-09-01 17:03:48 -07:00
Zhe Zhang 08b6808489 Addendum fix for HDFS-10817 to fix failure of the added testFSReadLockLongHoldingReport in branch-2. 
(cherry picked from commit 6c600360ca)
(cherry picked from commit 776eccd873)
 2016-08-31 23:45:45 -07:00
Zhe Zhang 7adbd569a6 HDFS-10817. Add Logging for Long-held NN Read Locks. Contributed by Erik Krogen. 
(cherry picked from commit 6f4b0d33ca)
(cherry picked from commit f87e2f5d05)
 2016-08-31 15:54:46 -07:00
Kihwal Lee e53f6fde46 HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 
(cherry picked from commit 22ef5286bc)
(cherry picked from commit a5d12d9c1f)
 2016-08-31 15:16:07 -07:00
Kihwal Lee 7b5e12229d HDFS-10656. Optimize conversion of byte arrays back to path string. Contributed by Daryn Sharp. 
(cherry picked from commit bebf10d245)
(cherry picked from commit ab7072838f)
 2016-08-31 15:11:27 -07:00
Kihwal Lee ce902fc80a HDFS-10729. Improve log message for edit loading failures caused by FS limit checks. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 01721dd88e)
 2016-08-31 14:12:02 -05:00
Wei-Chiu Chuang 80bb9ffb67 HDFS-10760. DataXceiver#run() should not log InvalidToken exception as an error. Contributed by Pan Yuxuan. 
(cherry picked from commit c4ee6915a1)
 2016-08-30 10:44:48 -07:00
Zhe Zhang ee28034a8a HDFS-10814. Add assertion for getNumEncryptionZones when no EZ is created. Contributed by Vinitha Reddy Gankidi. 
(cherry picked from commit 4bd45f54ee)
 2016-08-29 23:43:04 -07:00
Xiao Chen a291306510 HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge. 
(cherry picked from commit 046f6bf0f944311fc70de3a96fc6ab8211cbbbd8)
 2016-08-29 17:42:30 -07:00
Yongjun Zhang 18f7628664 HDFS-10625. VolumeScanner to report why a block is found bad. Contributed by Rushabh S Shah and Yiqun Lin. 
(cherry picked from commit 5d1609ddf2)
 2016-08-29 14:15:23 -07:00
Mingliang Liu f84ae6f091 HDFS-10807. Doc about upgrading to a version of HDFS with snapshots may be confusing. (Contributed by Mingliang Liu) 
(cherry picked from commit 6742fb6e68)
 2016-08-29 10:19:53 -07:00
Yongjun Zhang 6f10a0a87b HDFS-10652. Add a unit test for HDFS-4660. Contributed by Vinayakumar B., Wei-Chiu Chuang, Yongjun Zhang. 
(cherry picked from commit c25817159a)
 2016-08-27 23:11:57 -07:00
Andrew Wang 42e03d7bfe HDFS-10793. Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. Contributed by Manoj Govindassamy. 
(cherry picked from commit a445b82baa)
(cherry picked from commit 1c47a356d8)
 2016-08-26 15:40:13 -07:00
Zhe Zhang 96cf681b5d HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen. 
(cherry picked from commit 407b519fb1)
(cherry picked from commit f3a4cbcc60)
 2016-08-26 14:23:51 -07:00
Kihwal Lee 1c5ecb6cc4 HDFS-8915. TestFSNamesystem.testFSLockGetWaiterCount fails intermittently. Contributed by Masatake Iwasaki. 
(cherry picked from commit 13fb1b50e6)
 2016-08-26 10:01:19 -05:00
Xiaoyu Yao db415ea166 HDFS-10748. TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout. Contributed by Yiqun Lin. 
(cherry picked from commit 4da5000dd3)
 2016-08-25 09:22:16 -07:00
Wei-Chiu Chuang bd729af8a8 HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-24 08:53:43 -07:00
Zhe Zhang 8bc33bf343 HADOOP-12765. HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections. Contributed by Min Shen. Branch-2 patch contributed by Wei-Chiu Chuang. 
(cherry picked from commit dfcbc12026)
 2016-08-23 14:46:47 -07:00
Jason Lowe f57a26378e HDFS-9745. TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts. Contributed by Xiao Chen 
(cherry picked from commit 126d165efd)
 2016-08-23 14:49:27 +00:00
Akira Ajisaka dde3eaed62 HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin. 
(cherry picked from commit e90f3359de)
(cherry picked from commit 94a844b4f6)
 2016-08-23 19:59:07 +09:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) c0c4cc8ee8 HDFS-10692. Update JDiff report's base version for HDFS from 2.6.0 to 2.7.2. Contributed by Wangda Tan. 
(cherry picked from commit dc7a1c54f4)
 2016-08-22 11:37:14 -07:00
Kihwal Lee c9199095f8 HDFS-10738. Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure. Contributed by Rakesh R. 
(cherry picked from commit 0f701f433d)
(cherry picked from commit 7f1879abe6)
 2016-08-22 11:04:03 -05:00
Jing Zhao 8628ad0790 HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal. 
(cherry picked from commit 0faee62a0c)
(cherry picked from commit fff0418458)
 2016-08-19 22:27:24 -07:00
Kihwal Lee 4f3df8e22a HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 
(cherry picked from commit ae90d4dd90)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-19 15:32:11 -05:00
Arpit Agarwal 1fe08c919a HDFS-10682. Replace FsDatasetImpl object lock with a separate lock object. (Contributed by Chen Liang) 2016-08-17 16:22:07 -07:00
Xiao Chen b89d79ca1d HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin. 
(cherry picked from commit 85aacaadb5a3f8c78b191867c0bde09b3c4b3c3c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java

(cherry picked from commit a20b943cf9)
 2016-08-17 16:00:33 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) d65024eddc Revert "HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla." 
Reverting this on branch-2.* as it's an incompatible change.

This reverts commit 83f7f62be3.

(cherry picked from commit 2486c4c63a)
 2016-08-17 13:34:35 -07:00
Kihwal Lee b76f6e5de3 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 
(cherry picked from commit b10c936020)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 65a41696b2)
 2016-08-16 09:39:33 -05:00
Mingliang Liu 54c9743263 HDFS-10641. TestBlockManager#testBlockReportQueueing fails intermittently. (Contributed by Daryn Sharp) 2016-08-15 20:46:48 -07:00
Mingliang Liu e9eaad0e67 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:46:34 -07:00
Mingliang Liu 0b934c375e HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:46:28 -07:00
Kihwal Lee 1a8280edde HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 
(cherry picked from commit 864f878d59)
(cherry picked from commit e78db7d2a4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2016-08-15 17:37:39 -05:00
Kihwal Lee 579709fc22 HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 
(cherry picked from commit 83e57e083f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java

(cherry picked from commit 4766a3153d)
 2016-08-15 13:13:49 -05:00
Wei-Chiu Chuang 53f097a474 HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9019606b69)
(cherry picked from commit 20a8ff0a8d)
 2016-08-12 06:11:22 -07:00
Zhe Zhang 9c1c48c0d4 HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 
(cherry picked from commit 3e8099a45a)
(cherry picked from commit 00ff3d737c)
 2016-08-11 12:42:17 -07:00
Akira Ajisaka f29d0739d7 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 
(cherry picked from commit 932aed64d7)
(cherry picked from commit 0529da7239)
 2016-08-11 12:42:17 -07:00
Zhe Zhang b8b4ea67d8 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
(cherry picked from commit ccff6035f5)
(cherry picked from commit 7d402692b4)
 2016-08-11 12:42:17 -07:00
Wei-Chiu Chuang 65e3657149 HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 
(cherry picked from commit f1d5a95cf3)
 2016-08-10 22:30:06 -07:00
Xiaoyu Yao 7a1a3f7754 HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit ec289bbece)
 2016-08-10 15:56:56 -07:00
Xiaoyu Yao 8808df09c6 Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca7.

(cherry picked from commit fdafd0cb4f)
 2016-08-10 15:56:43 -07:00
Xiaoyu Yao c0e331935d Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit 9f816d7ca7)
 2016-08-10 15:19:10 -07:00
Wei-Chiu Chuang 753edc4931 HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 
(cherry picked from commit 8efd4959f3)
 2016-08-10 11:39:12 -07:00
Yuanbo Liu d46f5ca7e2 HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 
(cherry picked from commit 10e84c6a6e)
 2016-08-10 11:12:21 -07:00
Konstantin V Shvachko 6b3114a2a5 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 
(cherry picked from commit 4d3af47f27)
 2016-08-07 14:36:24 -07:00
Kihwal Lee 5b3deac2e3 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit f715f14185)
 2016-08-05 11:43:43 -05:00
Akira Ajisaka 5758026833 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 
(cherry picked from commit ef432579a7)
(cherry picked from commit 6863866127)
 2016-08-05 17:33:07 +09:00
Kihwal Lee e8cd1439d3 HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit 57369a678c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 690ec789f0)
 2016-08-04 15:27:51 -05:00
Kihwal Lee eb9af6ca34 HDFS-10717. Fix findbugs warnings of hadoop-hdfs-client in branch-2. Contributed by Akira Ajisaka. 
(cherry picked from commit 6ead6f3bb8)
 2016-08-04 12:36:51 -05:00
Arpit Agarwal f3424d662a HDFS-10722. Fix build break. 2016-08-04 10:28:43 -07:00
Kihwal Lee 3a85aea42d HDFS-10722. Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs. Contributed by Daryn Sharp. 
(cherry picked from commit 4a8e6dc02f)
 2016-08-04 11:56:26 -05:00
Tsz-Wo Nicholas Sze cefa21e98a In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:54:15 -07:00
Kihwal Lee 58df27b873 HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 
(cherry picked from commit 6ae39199da)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

(cherry picked from commit 77b61d1f4e)
 2016-08-04 09:45:03 -05:00
Kihwal Lee 804db5b277 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 
(cherry picked from commit c0166b0743)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-04 09:35:16 -05:00
Kihwal Lee ceac628378 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 
(cherry picked from commit 6f63566694)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java

(cherry picked from commit 1ca2ab4013)
 2016-08-03 15:28:13 -05:00
Jing Zhao 5bdee64bde HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 
(cherry picked from commit f4ba5ff1d7)
(cherry picked from commit 180a26d9fb)
 2016-08-03 12:44:49 -07:00
Kihwal Lee 33a87ffe10 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 
(cherry picked from commit 58db263e93)
 2016-08-03 12:19:05 -05:00
Kihwal Lee 288f9ccde2 HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 
(cherry picked from commit 5e5b8793fb)
 2016-08-02 15:43:11 -05:00
Konstantin V Shvachko 9b3b7703b1 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 9ab153d7ae.
 2016-08-01 22:48:09 -07:00
Daryn Sharp ecb143ed21 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 
(cherry picked from commit 9f473cf903)
 2016-08-01 10:18:45 -05:00
Yongjun Zhang cc066de22c HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 8d32bd884d)
 2016-07-29 23:47:46 -07:00
Akira Ajisaka e467f838bb HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 
(cherry picked from commit 204a2055b1)
(cherry picked from commit 0cff416c35)
 2016-07-29 15:41:17 +09:00
Xiaoyu Yao ac510e5cd0 HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:16:44 -07:00
Akira Ajisaka 960caf624d HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 
(cherry picked from commit 414fbfab41)
(cherry picked from commit ab49c2b399)
 2016-07-28 16:59:49 +09:00
Yongjun Zhang 71c49b1786 HDFS-10667. Report more accurate info about data corruption location. (Yuanbo Liu via Yongjun Zhang) 
(cherry picked from commit eb7ff0c992)
 2016-07-27 15:49:56 -07:00
Brahma Reddy Battula 5dc29ab0f2 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 096d3fa25c)
 2016-07-27 10:31:29 +05:30
Arpit Agarwal 45a6ee97b2 HDFS-10642. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. (Contributed by Mingliang Liu) 2016-07-26 12:45:06 -07:00
Wei-Chiu Chuang 53bf632702 HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. Contributed by Kai Sasaki. 
(cherry picked from commit 25f0a9b3ab)
 2016-07-26 08:31:00 -07:00
Steve Loughran 7921ad949f Revert "HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu." 
This reverts commit 32468b1ba5.
 2016-07-26 14:31:02 +01:00
Masatake Iwasaki 6b5d546014 HDFS-10671. Fix typo in HdfsRollingUpgrade.md. Contributed by Yiqun Lin. 
(cherry picked from commit 59466b8c18)
 2016-07-26 19:52:23 +09:00
Brahma Reddy Battula 32468b1ba5 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 7cac7655fd)
 2016-07-26 11:00:11 +05:30
Akira Ajisaka e377e94f64 Revert "HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg." 
This reverts commit 91f47ab5f9.
 2016-07-26 11:06:10 +09:00
Vinitha Reddy Gankidi 9ab153d7ae HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:59:42 -07:00
Jing Zhao 62800dc8b6 HDFS-10688. BPServiceActor may run into a tight loop for sending block report when hitting IOException. Contributed by Chen Liang. 
(cherry picked from commit 0cde9e12a7)
(cherry picked from commit 30c772fc59)
 2016-07-25 18:44:06 -07:00
Arpit Agarwal 00cbd3136e HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-25 09:51:59 -07:00
Jing Zhao 54be36a6d6 HDFS-9894. Add unsetStoragePolicy API to FileContext/AbstractFileSystem and derivatives. Contributed by Xiaobing Zhou. 
(cherry picked from commit 7149cdb3c2d9dd390cd8668883cbe5db94090e0a)
 2016-07-25 09:51:59 -07:00
Arpit Agarwal bbf711c030 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-07-25 09:51:59 -07:00
Xiao Chen 8559442ed1 HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 132deb4cac)
 2016-07-22 15:04:21 -07:00
Jing Zhao 6a7fe835f1 HDFS-10653. Optimize conversion from path string to components. Contributed by Daryn Sharp. 
(cherry picked from commit bd3dcf46e2)
 2016-07-21 11:29:27 -07:00
Akira Ajisaka 2f14d585c4 HDFS-10287. MiniDFSCluster should implement AutoCloseable. Contributed by Andras Bokor. 
(cherry picked from commit fcde6940e0)
(cherry picked from commit 1222889f1a)
 2016-07-21 11:28:46 -07:00
Akira Ajisaka 91f47ab5f9 HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 
(cherry picked from commit f7dabe3add)
(cherry picked from commit 2685e16f38)
 2016-07-19 15:19:02 -07:00
Akira Ajisaka d8181a8cb8 HDFS-10603. Fix flaky tests in org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot. Contributed by Yiqun Lin. 
(cherry picked from commit fc570b55b9)
(cherry picked from commit c92601e981)
 2016-07-19 13:52:17 -07:00
Jitendra Pandey a363277be5 HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:32:20 -07:00
Akira Ajisaka 83126ac90d HDFS-10628. Log HDFS Balancer exit message to its own log. Contributed by Jiayi Zhou. 
(cherry picked from commit c48e9d608c)
(cherry picked from commit 6171498375)
 2016-07-15 11:43:54 -07:00
Akira Ajisaka 6a04d9ab63 HADOOP-13351. TestDFSClientSocketSize buffer size tests are flaky. Contributed by Aaron Fabbri and Mingliang Liu. 
(cherry picked from commit 5537c6b234)
(cherry picked from commit c9f5348bbf)
 2016-07-14 14:42:55 -07:00
Zhe Zhang af94dfaa53 HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. 
(cherry picked from commit 087290e6b1)
(cherry picked from commit 5b4e916b3c)
 2016-07-12 23:28:16 -07:00
Jitendra Pandey 35108025e7 HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS protection. Contributed by Anu Engineer. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
 2016-07-12 14:38:34 -07:00
Yongjun Zhang 96e68e722f HDFS-10512. VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. Contributed by Wei-Chiu Chuang and Yiqun Lin. 
(cherry picked from commit da6f1b88dd)
 2016-07-08 22:20:27 -07:00
Jitendra Pandey 9c7ce389c7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:04:47 -07:00
Uma Maheswara Rao G 097022e172 HDFS-10592:Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning. Contributed by Rakesh R 
(cherry picked from commit ab092c56c2)
 2016-07-07 14:02:25 -07:00
Andrew Wang f826d92909 HDFS-10488. Update WebHDFS documentation regarding CREATE and MKDIR default permissions. Contributed by Wellington Chevreuil. 
(cherry picked from commit 647b084738b26dcf0c0c4afa4ec3039e6aca2d61)
 2016-07-07 10:37:16 -07:00
Akira Ajisaka 4fba09d7e8 HDFS-10582. Change deprecated configuration fs.checkpoint.dir to dfs.namenode.checkpoint.dir in HDFS Commands Doc. Contributed by Pan Yuxuan. 
(cherry picked from commit a3f93beaa1)
(cherry picked from commit 8a9440298d)
 2016-07-06 17:54:44 -07:00
Akira Ajisaka 4b263b30c2 HDFS-10589. Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params. Contributed by shenyinjie. 
(cherry picked from commit 8b4b5259d5)
(cherry picked from commit 0401adcaca)
 2016-07-04 07:15:54 -07:00
Jitendra Pandey bc7fd76a1f HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:37:32 -07:00
Vinayakumar B f44d7782f6 HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) 
(cherry picked from commit 2a0082c51d)
(cherry picked from commit 1e34763181)
 2016-06-28 16:55:57 +05:30
Akira Ajisaka 700dacf8c4 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 
(cherry picked from commit d433b16ce6)
(cherry picked from commit ad50a64ded)
 2016-06-22 11:02:48 +09:00
Brahma Reddy Battula aa8f4cc48c HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 
(cherry picked from commit f2ac132d6a)
(cherry picked from commit ee0f389ecd)
 2016-06-21 15:55:57 +05:30
Colin Patrick Mccabe 9b51420ea5 HDFS-10448. CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1 (Yiqun Lin via cmccabe) 
(cherry picked from commit 46f1602e89)
(cherry picked from commit 4e11f33ccc)
 2016-06-20 18:32:00 -07:00
Jitendra Pandey 29c3ee9744 HDFS-10538. Remove AsyncDistributedFileSystem. Contributed by Xiaobing Zhou and Jitendra Pandey. 2016-06-20 18:05:32 -07:00
Jitendra Pandey 2f95d22b17 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:31:54 -07:00
Ravi Prakash 588ea6d431 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 
(cherry picked from commit ae047655f4)
(cherry picked from commit 9c5f7f290e)
 2016-06-20 14:38:05 -07:00
Brahma Reddy Battula 9d27530c36 HDFS-10474. hftp copy fails when file name with Chinese+special char in branch-2 (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 3f27f40503)
 2016-06-20 12:40:11 +05:30
Andrew Wang 3f2d2cd0a1 Fix hadoop-hdfs-native-client compilation on Windows. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 3ff0510ffd)
(cherry picked from commit 94759470b8)
 2016-06-17 15:55:43 -07:00
Kihwal Lee edd5373095 HDFS-10493. Add links to datanode web UI in namenode datanodes page. Contributed by Weiwei Yang. 
(cherry picked from commit 280069510b)
 2016-06-17 11:13:12 -05:00
Akira Ajisaka 242263c56d HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 
(cherry picked from commit 09e82acaf9)
(cherry picked from commit a36aa920f4)
 2016-06-17 16:21:31 +09:00
Colin Patrick Mccabe cc53ca8d69 HDFS-9466. TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit c7921c9bdd)
(cherry picked from commit 49c0d9b325)
 2016-06-16 13:53:55 -07:00
Arpit Agarwal 394dae17ac HDFS-10532. Typo in RollingUpgrade docs. (Contributed by Yiqun Lin) 2016-06-16 10:11:05 -07:00
Colin Patrick Mccabe ea2e7321d6 HDFS-10525. Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap (Xiao Chen via cmccabe) 
(cherry picked from commit 2ca73445f5)
(cherry picked from commit 8280e3bcc2)
 2016-06-15 22:51:27 -07:00
Colin Patrick Mccabe ef08eb9031 HDFS-10505. OIV's ReverseXML processor should support ACLs (Surendra Singh Lilhore via cmccabe) 
(cherry picked from commit 2449db507d)
(cherry picked from commit 17301ab395)
 2016-06-15 22:44:41 -07:00
Akira Ajisaka b35c7cdd14 HADOOP-12892. fix/rewrite create-release. Contributed by Allen Wittenauer. 2016-06-16 14:06:08 +09:00
Ming Ma 03ed3f9b66 Revert "HDFS-9016. Display upgrade domain information in fsck. (mingma)" 
This reverts commit 68ab0aa9d0.
 2016-06-14 20:14:00 -07:00
Ming Ma 68ab0aa9d0 HDFS-9016. Display upgrade domain information in fsck. (mingma) 
(cherry picked from commit b7436f4468)
 2016-06-14 20:08:29 -07:00
Andrew Wang 56e29d2711 HDFS-10516. Fix bug when warming up EDEK cache of more than one encryption zone. Contributed by Xiao Chen. 
(cherry picked from commit 709a814fe0)
(cherry picked from commit ff4466fe15)
 2016-06-13 11:50:27 -07:00
Andrew Wang f583fa5f4e Remove redundant TestMiniDFSCluster.testDualClusters. Contributed by Jiayi Zhou. 
(cherry picked from commit 1ee9ea0026)
(cherry picked from commit 9319665461)
 2016-06-08 12:59:08 -07:00
Zhe Zhang 72addd8d5c HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster. 
(cherry picked from commit 6de9213df1)
(cherry picked from commit 06e38c835d)
(cherry picked from commit 0499907466)
 2016-06-07 09:09:18 -07:00
Jason Lowe 738b67e9d3 HDFS-9580. TestComputeInvalidateWork#testDatanodeReRegistration failed due to unexpected number of invalidate blocks. Contributed by Wei-Chiu Chuang. 
Change-Id: I9411dfb1dd4605cbcc515ca3361218d35d7c5fe3
(cherry picked from commit 0087734cc1)

Conflicts:

	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-06-07 15:36:53 +00:00
Wangda Tan e91eba2f81 Revert "HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster." 
This reverts commit 0274636529.
 2016-06-06 22:13:02 -07:00
Zhe Zhang 0274636529 HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster. 
(cherry picked from commit 6de9213df1)
(cherry picked from commit 06e38c835d)
 2016-06-06 15:54:36 -07:00
Tsz-Wo Nicholas Sze 49535c0683 Revert "Revert "HADOOP-13226 Support async call retry and failover."" 
This reverts commit 886e239606.
 2016-06-06 16:54:02 +08:00
Tsz-Wo Nicholas Sze 359a316736 Revert "Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou."" 
This reverts commit f176311366.
 2016-06-06 16:53:56 +08:00
Tsz-Wo Nicholas Sze 025f62afcc Revert "Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou"" 
This reverts commit 0af02cfd37.
 2016-06-06 16:53:51 +08:00
Tsz-Wo Nicholas Sze 79beac49ec Revert "Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit 1de712f22a.
 2016-06-06 16:53:46 +08:00
Tsz-Wo Nicholas Sze c949001eee Revert "Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls."" 
This reverts commit 308d28640d.
 2016-06-06 16:53:41 +08:00
Tsz-Wo Nicholas Sze 5b054b1bfc Revert "Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit ffa85f28a2.
 2016-06-06 16:53:36 +08:00
Tsz-Wo Nicholas Sze 8ccc8aeb6c Revert "Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou"" 
This reverts commit 8a07026e77.
 2016-06-06 16:53:31 +08:00
Tsz-Wo Nicholas Sze 2cf663bb59 Revert "Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit d740a90260.
 2016-06-06 16:53:24 +08:00
Andrew Wang d740a90260 Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit bdc45bef64.
 2016-06-03 18:17:33 -07:00
Andrew Wang 8a07026e77 Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou" 
This reverts commit a1ba6eee38.
 2016-06-03 18:17:33 -07:00
Andrew Wang ffa85f28a2 Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit ac04900450.
 2016-06-03 18:12:31 -07:00
Andrew Wang 308d28640d Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls." 
This reverts commit fa7c7f2510.
 2016-06-03 18:12:31 -07:00
Andrew Wang 1de712f22a Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 339b803a52.
 2016-06-03 18:12:31 -07:00
Andrew Wang 0af02cfd37 Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou" 
This reverts commit 2e755a7f0e.
 2016-06-03 18:12:31 -07:00
Andrew Wang f176311366 Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou." 
This reverts commit de69b1da34.
 2016-06-03 18:12:30 -07:00
Andrew Wang 886e239606 Revert "HADOOP-13226 Support async call retry and failover." 
This reverts commit a8941d7790.
 2016-06-03 18:12:30 -07:00
Andrew Wang 6f69113417 HDFS-10481. HTTPFS server should correctly impersonate as end user to open file. Contributed by Xiao Chen. 
(cherry picked from commit 47e0321ee9)
(cherry picked from commit d5609e3499)
 2016-06-03 17:21:34 -07:00
Andrew Wang d712b2ee3b HADOOP-13155. Implement TokenRenewer to renew and cancel delegation tokens in KMS. Contributed by Xiao Chen. 
(cherry picked from commit 713cb71820)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderDelegationTokenExtension.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java

(cherry picked from commit 03c4724c88)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
 2016-06-03 17:03:40 -07:00
Akira Ajisaka d0dc5aaa2d HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (aajisaka) 
(cherry picked from commit b6d5546e24)
 2016-06-03 16:04:40 +09:00
Akira Ajisaka a8a2f4b500 HDFS-10471. DFSAdmin#SetQuotaCommand's help msg is not correct. Contributed by Yiqun Lin. 
(cherry picked from commit 1df6f5735c)
(cherry picked from commit cddf6b4fc7)
 2016-06-03 04:11:13 +09:00
Masatake Iwasaki 5b3d9ca0cd HDFS-10367. TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit aadb77e412)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

(cherry picked from commit 3740974413)
 2016-06-02 21:04:48 +09:00
Akira Ajisaka fe4ad4a39c HDFS-9476. TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail. Contributed by Masatake Iwasaki. 
(cherry picked from commit 69555fca06)
(cherry picked from commit 3dd557e4b6)
 2016-06-02 18:53:42 +09:00
Colin Patrick Mccabe 6fe1c67fa1 HADOOP-13137. TraceAdmin should support Kerberized cluster (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit 8ceb06e239)
(cherry picked from commit a64e1425ed)
 2016-05-31 18:26:37 -07:00
Colin Patrick Mccabe 0d6a65a40a HDFS-10415. TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called (Mingliang Liu via cmccabe) 
(cherry picked from commit 14907d9b8c)
 2016-05-31 17:27:14 -07:00
Tsz-Wo Nicholas Sze a8941d7790 HADOOP-13226 Support async call retry and failover. 2016-05-31 17:16:44 -07:00
Chris Nauroth de69b1da34 HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou. 
(cherry picked from commit 21890c4239)
(cherry picked from commit 193f822b9a)
 2016-05-27 14:20:50 -07:00
Yongjun Zhang 3e4c7906c2 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 
(cherry picked from commit 5ea6fd85c7)
 2016-05-27 10:18:47 -07:00
Tsz-Wo Nicholas Sze 2e755a7f0e HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:24:21 -07:00
Tsz-Wo Nicholas Sze 339b803a52 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-26 12:23:05 -07:00
Tsz-Wo Nicholas Sze fa7c7f2510 HADOOP-13168. Support Future.get with timeout in ipc async calls. 2016-05-26 12:22:57 -07:00
Tsz-Wo Nicholas Sze ac04900450 HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-26 12:20:31 -07:00
Tsz-Wo Nicholas Sze f6dfb71762 HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-05-25 12:26:25 -07:00
Vinayakumar B 00011e6a98 HDFS-10217. show 'blockScheduled' tooltip in datanodes table. 
(cherry picked from commit 2dab64da2f)
(cherry picked from commit e28a51e988)
 2016-05-25 12:35:42 +05:30
Tsz-Wo Nicholas Sze 2ab789b19e HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 13:11:32 -07:00
Masatake Iwasaki 1ab79e255c HDFS-10439. Update setOwner doc in HdfsPermissionsGuide. Contributed by John Zhuge. 
(cherry picked from commit 12eb5a7b11)
 2016-05-22 19:43:47 +09:00
Chris Nauroth 5b907a17ed HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 
(cherry picked from commit 500e946729)
(cherry picked from commit e7b56cf93c)
 2016-05-20 13:02:06 -07:00
Chris Nauroth 877af3ec7e HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 
(cherry picked from commit bcde1562d2)
(cherry picked from commit d38f2090fa)
 2016-05-20 12:50:23 -07:00
Kihwal Lee a81150ae9e HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 204de430bf)
 2016-05-19 15:54:18 -05:00
Arpit Agarwal 01a0730996 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:25:33 -07:00
Xiaoyu Yao ccb1cade5b HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 18:00:18 -07:00
Yongjun Zhang 2bcf1eb36a HDFS-10381, DataStreamer DataNode exclusion log message should be warning. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 16c07cc68a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java

(cherry picked from commit 456fe08a6d)
 2016-05-17 16:00:49 -07:00
Kihwal Lee 01de843a31 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit cb5a1edd1c.
 2016-05-17 15:21:41 -05:00
Kihwal Lee cb5a1edd1c HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 0942954e8a)
 2016-05-17 15:11:40 -05:00
Kihwal Lee 362a778bf5 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4a5819dae2)
 2016-05-17 09:00:59 -05:00
Akira Ajisaka 388b2010ae HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 
(cherry picked from commit 9fe5828f05)
(cherry picked from commit 7f03c1c90e)
 2016-05-17 15:56:36 +09:00
Andrew Wang 29ed65ff9b HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 
(cherry picked from commit 45788204ae)
(cherry picked from commit 413f3dabbd)
 2016-05-15 22:15:18 -07:00
Colin Patrick Mccabe 5f04795aed HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 
(cherry picked from commit 687233f20d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

(cherry picked from commit 15bfcde238)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2016-05-11 14:24:51 -07:00
Colin Patrick Mccabe 154332bb54 HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO level (Yiqun Lin via cmccabe) 
(cherry picked from commit f0252ad252)
(cherry picked from commit 43f32d3f4a)
 2016-05-10 11:21:16 -07:00
Akira Ajisaka a84c4d7d4b HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. Contributed by Gergely Novák. 
(cherry picked from commit 901588542b)
 2016-05-10 09:33:55 -07:00
Akira Ajisaka 1622fc1373 Revert "HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated." 
This reverts commit f25dcb935e.
 2016-05-10 09:33:18 -07:00
Akira Ajisaka f25dcb935e HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. 
(cherry picked from commit c297df5995)
 2016-05-10 09:29:47 -07:00
Kihwal Lee d7cda6d031 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah. 
(cherry picked from commit b9e5a32fa1)
 2016-05-09 15:52:12 -05:00
Masatake Iwasaki c80b47e73f HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. 
(cherry picked from commit 2835f146b5)
 2016-05-06 15:50:48 +09:00
Xiaoyu Yao 59eeee7501 HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit dacd1f50fe)
(cherry picked from commit 35859872b7)
 2016-05-05 18:23:00 -07:00
Ming Ma d6e95ae47b HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 
(cherry picked from commit 1268cf5fbe)
 2016-05-04 17:07:36 -07:00
Arpit Agarwal ce94203566 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:53:07 -07:00
Xiaoyu Yao 5dfcfb2d2d HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao. 
(cherry picked from commit 45a753ccf7)
 2016-05-02 19:50:10 -07:00
Xiaoyu Yao 643d2fd3c1 HADOOP-12985. Support MetricsSource interface for DecayRpcScheduler Metrics. Contributed by Xiaoyu Yao. 
(cherry picked from commit 5bd7b592e5)
(cherry picked from commit a2a65922dd)
 2016-05-02 19:38:03 -07:00
Harsh J cf6efccebb HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) 
(cherry picked from commit 54e612bfb9)
 2016-05-02 17:53:10 -07:00
Tsz-Wo Nicholas Sze a1ba6eee38 HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:34:59 -07:00
Tsz-Wo Nicholas Sze bdc45bef64 HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-02 11:34:22 -07:00
Lei Xu 5477c31c5c HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 0a152103f1)
(cherry picked from commit 4289cb8b36)
 2016-05-02 10:21:59 -07:00
Kihwal Lee e9cd88d484 HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah. 
(cherry picked from commit 7da540d03e)
 2016-04-29 15:58:45 -05:00
Tsz-Wo Nicholas Sze 31ed34c982 HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:27:14 -07:00
Kihwal Lee a4db3729be HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 
(cherry picked from commit af9b000535)
 2016-04-29 10:05:58 -05:00
Andrew Wang bdb0e98483 HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge. 2016-04-27 17:03:36 -07:00
Kihwal Lee dd7c9f5b8f HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 
(cherry picked from commit e181092b86)
 2016-04-27 08:30:29 -05:00
Kihwal Lee 621be60592 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun. 
(cherry picked from commit c6a2430b47)
 2016-04-26 11:47:15 -05:00
Kihwal Lee 973c73aa68 HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 
(cherry picked from commit 10f0f7851a)
 2016-04-25 11:42:17 -05:00
Masatake Iwasaki c4d8090b72 HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 
(cherry picked from commit 6fcde2e38d)
 2016-04-23 23:55:27 +09:00
Kihwal Lee 3d3ed9b68e HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang. 
(cherry picked from commit 5d4255a801)
 2016-04-22 11:10:36 -05:00
Tsz-Wo Nicholas Sze 60c68c65c1 HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:45:36 -07:00
Tsz-Wo Nicholas Sze 704c57e3eb HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:25:49 -07:00
Chris Nauroth 7fb1cfc591 HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 
(cherry picked from commit 63ac2db59a)
(cherry picked from commit cc56bed686)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
 2016-04-20 14:00:06 -07:00
Colin Patrick Mccabe e2968f5da2 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe) 
(cherry picked from commit ad36fa6f42)
(cherry picked from commit 76d963acb4)
 2016-04-20 11:52:49 -07:00
Arpit Agarwal a13628fe4a HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:27:46 -07:00
Walter Su 1860b8ceb5 HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun. 
(cherry picked from commit 90d1af1416)
 2016-04-19 18:56:23 +08:00
Colin Patrick Mccabe e1d9e5ab82 HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 
(cherry picked from commit cb3ca460ef)
(cherry picked from commit a69b6b1e8b)
 2016-04-18 13:49:05 -07:00
Kihwal Lee 7ac2e5ec7b HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun. 
(cherry picked from commit d8b729e16f)
 2016-04-18 08:01:17 -05:00
Walter Su 09ffc65d94 HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun. 
(cherry picked from commit ab903029a9)
 2016-04-18 20:32:36 +08:00
Steve Loughran e9699aa443 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 2016-04-18 10:26:35 +01:00
Walter Su 7f47d984fa HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 
(cherry picked from commit 67523ffcf4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java

(cherry picked from commit a6d0e9e56a)
 2016-04-18 09:39:46 +08:00
Kihwal Lee 8b1e7842e3 HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu. 
(cherry picked from commit b9c9d03591)
 2016-04-14 15:30:59 -05:00
Kihwal Lee 7dd8798933 HDFS-10292. Add block id when client got Unable to close file exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 2c155afe27)
 2016-04-14 14:26:59 -05:00
Kihwal Lee 019ca8f7c1 HDFS-10280. Document new dfsadmin command -evictWriters. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit c970f1d005)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
 2016-04-14 12:50:36 -05:00
Andrew Wang 1dee3e1200 HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun. 
(cherry picked from commit 314aa21a89)
(cherry picked from commit 4c18f54c1d)
 2016-04-13 16:40:08 -07:00
Kihwal Lee ccaf746eea HDFS-10270. TestJMXGet:testNameNode() fails. Contributed by Gergely Novák 
(cherry picked from commit d2f3bbc290)
 2016-04-13 11:24:45 -05:00
Walter Su f3dbabb13a HDFS-9772. TestBlockReplacement#testThrottler doesn't work as expected. Contributed by Lin Yiqun. 
(cherry picked from commit 903428bf94)
 2016-04-13 17:50:47 +08:00
Colin Patrick Mccabe 59b0661c7f HADOOP-12973. Make DU pluggable. (Elliott Clark via cmccabe) 
(cherry picked from commit 35f0770555)
(cherry picked from commit 2b0b332e2f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2016-04-12 16:30:58 -07:00
Vinayakumar B b804b20843 HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula) 
(cherry picked from commit a9a607f8fc)
(cherry picked from commit 04621537c9)
 2016-04-12 11:47:01 +05:30
Akira Ajisaka e2b8bdbe0a HDFS-10277. PositionedReadable test testReadFullyZeroByteFile failing in HDFS. Contributed by Steve Loughran. 
(cherry picked from commit a409508b3f)
(cherry picked from commit 8f968171bd)
 2016-04-12 14:55:42 +09:00
Chris Nauroth 1b5b85e560 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d)
(cherry picked from commit 26a23eff82)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 2016-04-08 13:44:27 -07:00
Tsz-Wo Nicholas Sze f872d519db HDFS-10186. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. Contributed by Rakesh R 2016-04-07 12:42:27 +08:00
Colin Patrick Mccabe 354801fe61 HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose 
(cherry picked from commit 4bd7cbc29d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

(cherry picked from commit b5e6ad457f)
 2016-04-06 21:11:48 -07:00
Eric Payne 497c65ad0c HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal Lee 
(cherry picked from commit aede8c10ec)
 2016-04-06 20:33:40 +00:00
Colin Patrick Mccabe 8514ab8cb5 HDFS-6520. hdfs fsck passes invalid length value when creating BlockReader (Xiao Chen via cmccabe) 
(cherry picked from commit 188f65287d)
(cherry picked from commit 387d3f25c1)
 2016-04-06 11:38:19 -07:00
Akira Ajisaka e426d1bb3f HADOOP-12022. fix site -Pdocs -Pdist in hadoop-project-dist; cleanout remaining forrest bits. Contributed by Allen Wittenauer. 
(cherry picked from commit 1371f993a9)
 2016-04-06 17:44:54 +09:00
Kihwal Lee 8bb4655649 HDFS-10261. TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports. Contributed by Eric Badger. 
(cherry picked from commit 9ba1e5af06)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java

(cherry picked from commit 6e37c5fe80)
 2016-04-05 16:54:46 -05:00
Ravi Prakash 7286c435c4 HDFS-10235. Last contact for Live Nodes should be relative time. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0cd320a846)
 2016-04-05 13:43:08 -07:00
Masatake Iwasaki 886273e123 HADOOP-12672. RPC timeout should not override IPC ping interval (iwasakims) 
(cherry picked from commit 85ec5573eb)
 2016-04-06 03:29:49 +09:00
Kihwal Lee f9764d073f HDFS-10239. Fsshell mv fails if port usage doesn't match in src and destination paths. Contributed by Kuhu Shukla. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

(cherry picked from commit ef3da82357)
 2016-04-05 09:16:51 -05:00
Vinayakumar B 3deeb9a15f HDFS-9917. IBR accumulate more objects when SNN was down for sometime. (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 818d6b799e)
(cherry picked from commit ad08114b92)
 2016-04-05 09:54:50 +08:00
Colin Patrick Mccabe 63fe2ae72c HDFS-8496. Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe) 
(cherry picked from commit f6b1a81812)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

(cherry picked from commit d55af8b5a3)
 2016-04-04 18:07:21 -07:00
Kihwal Lee 7d671cad3f HDFS-10178. Permanent write failures can happen if pipeline recoveries occur for the first packet. Contributed by Kihwal Lee. 
(cherry picked from commit a7d1fb0cd2)
 2016-04-04 16:41:23 -05:00
Masatake Iwasaki 97cda4077a HDFS-9599. TestDecommissioningStatus.testDecommissionStatus occasionally fails (Lin Yiqun via iwasakims) 
(cherry picked from commit 154d2532cf)
 2016-04-05 03:44:12 +09:00
Vinayakumar B bafcb584a8 HDFS-10253. Fix TestRefreshCallQueue failure (Contributed by Xiaoyu Yao) 
(cherry picked from commit 54b2e78fd2)
(cherry picked from commit aa38748997)
 2016-04-03 13:28:52 +05:30
Akira Ajisaka 3892efd3fb HDFS-10221. Add .json to the rat exclusions. Contributed by Ming Ma. 
(cherry picked from commit 32c0c3ecdf)
(cherry picked from commit 0795ee687b)
 2016-03-31 09:06:48 +09:00
Colin Patrick Mccabe 5cf55c5fb6 HDFS-10223. peerFromSocketAndKey performs SASL exchange before setting connection timeouts (cmccabe) 
(cherry picked from commit 37e23ce45c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java

(cherry picked from commit be5894c12d)
 2016-03-30 13:45:05 -07:00
Vinayakumar B def143e1e1 HDFS-5177. blocksScheduled count should be decremented for abandoned blocks (Contributed by Vinayakumar B) 
(cherry picked from commit 09d63d5a19)
(cherry picked from commit 0852e44110)
 2016-03-30 14:23:56 +08:00
Arpit Agarwal bfe657e83a HDFS-10228. TestHDFSCLI fails. (Contributed by Akira AJISAKA) 2016-03-29 13:56:12 -07:00
Andrew Wang 026b84bf70 HDFS-10197. TestFsDatasetCache failing intermittently due to timeout. Contributed by Lin Yiqun. 
(cherry picked from commit f2aec4eb82)
(cherry picked from commit 1d79b57c14)
 2016-03-29 11:55:04 -07:00
Vinayakumar B 88b56cdfaa HDFS-9871. "Bytes Being Moved" -ve(-1 B) when cluster was already balanced. (Contributed by Brahma Reddy Battulla) 
(cherry picked from commit 1f004b3367)
(cherry picked from commit f74364e8b9)
 2016-03-29 11:31:18 +08:00
Walter Su d71843558b HDFS-10182. Hedged read might overwrite user's buf. Contributed by zhouyingchao. 
(cherry picked from commit d8383c687c)
 2016-03-28 15:45:40 +08:00
Andrew Wang d4aeddee31 HDFS-10200. Docs for WebHDFS still describe GETDELEGATIONTOKENS operation. Contributed by Wellington Chevreuil. 
(cherry picked from commit 8f85e5d212)
(cherry picked from commit c658deabb6)
 2016-03-23 12:56:17 -07:00
Colin Patrick Mccabe 4ca13e7b98 HDFS-10193. fuse_dfs segfaults if uid cannot be resolved to a username (John Thiltges via cmccabe) 
(cherry picked from commit 0d19a0ce98)
(cherry picked from commit 7ad3162df4)
 2016-03-23 11:20:12 -07:00
Colin Patrick Mccabe 31843df277 HDFS-10189. PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE (Joe Pallas via cmccabe) 
(cherry picked from commit a7d8f2b396)
(cherry picked from commit a68a9eb2b1)
 2016-03-22 17:22:15 -07:00
Colin Patrick Mccabe 29872506bc HDFS-9951. Use string constants for XML tags in OfflineImageReconstructor (Lin Yiqun via cmccabe) 
(cherry picked from commit 680716f31e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java

(cherry picked from commit 2ce24bf218)
 2016-03-21 11:49:19 -07:00
Andrew Wang ea44650a64 HDFS-9405. Warmup NameNode EDEK caches in background thread. Contributed by Xiao Chen. 
(cherry picked from commit e3bb38d625)
(cherry picked from commit e826d860a7)
 2016-03-21 11:40:26 -07:00
Haohui Mai 57fed702f1 HDFS-7166. SbNN Web UI shows #Under replicated blocks and #pending deletion blocks. Contributed by Wei-Chiu Chuang. 2016-03-20 17:22:09 -07:00
Akira Ajisaka 2fe4225051 HDFS-3677. dfs.namenode.edits.dir.required is missing from hdfs-default.xml. Contributed by Mark Yang. 
(cherry picked from commit 414f33126ac178a44146666cc7c00059cfc9cabf)
(cherry picked from commit 41621a994d)
 2016-03-19 01:11:06 +09:00
Kihwal Lee 242c7f1fee HDFS-9874. Long living DataXceiver threads cause volume shutdown to block. Contributed by Rushabh Shah. 
(cherry picked from commit 63c966a3fb)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2016-03-18 10:33:13 -05:00
Colin Patrick Mccabe 73b5a44b0e HDFS-9949. Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared (Harsh J via cmccabe) 
(cherry picked from commit dc951e606f)
(cherry picked from commit f69a6c3632)
 2016-03-17 10:43:07 -07:00
Kihwal Lee fce39557d3 HDFS-9904. testCheckpointCancellationDuringUpload occasionally fails. Contributed by Lin Yiqun. 
(cherry picked from commit d457401784)
 2016-03-15 10:55:10 -05:00