Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,194 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4321 Commits

Author SHA1 Message Date
Chris Nauroth e36b665f90 HADOOP-13150. Avoid use of toString() in output of HDFS ACL shell commands. Contributed by Chris Nauroth. 
(cherry picked from commit 1d330fbaf6)
(cherry picked from commit a28ffd0fde)
 2016-10-06 13:24:54 -07:00
Brahma Reddy Battula a4ee4f9e51 HDFS-10963. Reduce log level when network topology cannot find enough datanodes. Contributed by Xiao chen 
(cherry picked from commit b90fc70d67)
 2016-10-06 17:49:48 +05:30
Andrew Wang a2cdd2215e HDFS-10962. TestRequestHedgingProxyProvider is flaky. 
(cherry picked from commit e68c7b96c7)
(cherry picked from commit 6b2a853dd7)
 2016-10-05 15:09:13 -07:00
Kihwal Lee d03240e0bf HDFS-10956. Remove rename/delete performance penalty when not using snapshots. Contributed by Daryn Sharp. 
(cherry picked from commit 44f48ee96e)
 2016-10-04 15:29:39 -05:00
Kihwal Lee 33cafc9fcd HDFS-10878. TestDFSClientRetries#testIdempotentAllocateBlockAndClose throws ConcurrentModificationException. Contributed by Rushabh S Shah. 
(cherry picked from commit 382307cbdd)
 2016-10-04 11:13:39 -05:00
Brahma Reddy Battula 253e2442e4 HDFS-10947. Correct the API name for truncate in webhdfs document. Contributed by Surendra Singh Lilhore 
(cherry picked from commit ef7f06f7d1)
 2016-10-04 18:53:12 +05:30
Brahma Reddy Battula 1ae35b6fff HDFS-10944. Correct the javadoc of dfsadmin#disallowSnapshot. Contributed by Jagadesh Kiran N 
(cherry picked from commit 5ea0210c7f)
 2016-10-04 18:39:31 +05:30
Mingliang Liu 438402a710 HDFS-10934. TestDFSShell#testStat fails intermittently. Contributed by Eric Badger 
(cherry picked from commit f61e3d13e9)
 2016-10-03 22:44:22 -04:00
Mingliang Liu 11ea5a59c2 HDFS-10810. setreplication removing block from underconstrcution temporarily. Contributed by Brahma Reddy Battula 
(cherry picked from commit d963ecb918)
 2016-10-03 22:04:49 -04:00
Xiao Chen bf043e63f9 HDFS-10918. Add a tool to get FileEncryptionInfo from CLI. Contributed by Xiao Chen. 
(cherry picked from commit 853d65a157)
 2016-10-03 16:02:11 -07:00
Xiaoyu Yao b0b29de8ab HDFS-10690. Optimize insertion/removal of replica in ShortCircuitCache. Contributed by Fenghua Hu. 
(cherry picked from commit 607705c488)
 2016-10-03 11:05:58 -07:00
Kihwal Lee 52c534cd0f HDFS-10940. Reduce performance penalty of block caching when not used. Contributed by Daryn Sharp. 
(cherry picked from commit 03b797a6ac)
 2016-10-03 11:42:06 -05:00
Kihwal Lee 84b20d1c18 HDFS-10945. Fix the Findbugwaring FSNamesystem#renameTo() in branch-2. Contributed by Brahma Reddy Battula. 2016-10-03 11:09:40 -05:00
Kihwal Lee 3a89a8814f HDFS-10619. Cache path in InodesInPath. Contributed by Daryn Sharp. 
(cherry picked from commit 90020624b0)
 2016-10-03 09:26:41 -05:00
Arpit Agarwal c31d23db35 Revert "HDFS-10923. Make InstrumentedLock require ReentrantLock." 
This reverts commit 9d4e0168d5.
 2016-09-30 23:12:16 -07:00
Arpit Agarwal 9d4e0168d5 HDFS-10923. Make InstrumentedLock require ReentrantLock. 2016-09-30 23:04:30 -07:00
Kihwal Lee bfc7f0630b HDFS-10851. FSDirStatAndListingOp: stop passing path as string. Contributed by Daryn Sharp. 
(cherry picked from commit 2551ff80b7)
 2016-09-30 16:02:26 -05:00
Zhe Zhang 345debf22e HDFS-10896. Move lock logging logic from FSNamesystem into FSNamesystemLock. Contributed by Erik Krogen. 
(cherry-picked from commit 434c5ea75d)
 2016-09-30 13:43:49 -07:00
Kihwal Lee f5b193b482 HDFS-10850. getEZForPath should NOT throw FNF. Contributed by Andrew Wang. 
(cherry picked from commit 0670149c88)
 2016-09-30 09:16:13 -05:00
Mingliang Liu 7467628d43 HDFS-10892. Add unit tests for HDFS command 'dfs -tail' and 'dfs -stat'. Contributed by Mingliang Liu 
(cherry picked from commit 84c6264079)
 2016-09-28 16:09:42 -07:00
Kihwal Lee 5294eec729 HDFS-10779. Rename does not need to re-solve destination. Contributed by Daryn Sharp. 
(cherry picked from commit 5f34402ada)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2016-09-28 18:03:58 -05:00
Andrew Wang c4f81f199f HDFS-10914. Move remnants of oah.hdfs.client to hadoop-hdfs-client. 
(cherry picked from commit 92e5e91598)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java

(cherry picked from commit 9ddaa29301ea8ae2e3cb0f1e031343d5b91f92ee)
 2016-09-28 16:03:02 -07:00
Arpit Agarwal 76f92277c3 HDFS-10824. MiniDFSCluster#storageCapacities has no effects on real capacity. Contributed by Xiaobing Zhou. 2016-09-28 15:56:39 -07:00
Brahma Reddy Battula 5f754e8638 HDFS-9444. Add utility to find set of available ephemeral ports to ServerSocketUtil. Contributed by Masatake Iwasaki 
(cherry picked from commit e9a34ae29c)
 2016-09-28 11:16:40 +05:30
Mingliang Liu 6da9a3c603 HDFS-10915. Fix time measurement bug in TestDatanodeRestart. Contributed by Xiaobing Zhou 
(cherry picked from commit 6437ba18c5)
 2016-09-27 20:47:04 -07:00
Wangda Tan 69f91d8c48 HADOOP-13544. JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs. (vinodkv via wangda) 
(cherry picked from commit 875062b5bc)
(cherry picked from commit 269401dc83)
 2016-09-27 11:46:49 -07:00
Arpit Agarwal 3f5482bb69 HDFS-10828. Fix usage of FsDatasetImpl object lock in ReplicaMap. (Arpit Agarwal) 2016-09-27 10:02:27 -07:00
Masatake Iwasaki 756dbc505e HDFS-10426. TestPendingInvalidateBlock failed in trunk. Contributed by Yiqun Lin. 
(cherry picked from commit f55eb981dd)
 2016-09-28 01:47:31 +09:00
Arpit Agarwal a349c54603 HDFS-10713. Throttle FsNameSystem lock warnings. Contributed by Hanisha Koneru. 2016-09-27 09:05:53 -07:00
Brahma Reddy Battula 5bc297c58f HDFS-10889. Remove outdated Fault Injection Framework documentaion. Contributed by Brahma Reddy Battula 
(cherry picked from commit df1d0f5d56)
 2016-09-27 19:35:57 +05:30
Wei-Chiu Chuang f0f3b6a66a HDFS-10609. Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 3ae652f821)
(cherry picked from commit bde787db23)
 2016-09-26 14:46:02 -07:00
Brahma Reddy Battula 17b66c894b HDFS-10869. Remove the unused method InodeId#checkId(). Contributed by Jagadesh Kiran N 
(cherry picked from commit 3e37e243ee)
 2016-09-25 11:07:21 +05:30
Wei-Chiu Chuang 367d5b2a7e HDFS-10876. Dispatcher#dispatch should log IOException stacktrace. Contributed by Manoj Govindassamy. 
(cherry picked from commit 74b3dd514c)
(cherry picked from commit 6c98455728)
 2016-09-23 13:28:59 -07:00
Konstantin V Shvachko 20540df80d HDFS-10843. Update space quota when a UC block is completed rather than committed. Contributed by Erik Krogen. 
(cherry picked from commit a5bb88c8e0)
 2016-09-23 10:49:26 -07:00
Brahma Reddy Battula aa967869a5 HDFS-10886. Replace fs.default.name with fs.defaultFS in viewfs document. 
(cherry picked from commit bbdf350ff9)
 2016-09-23 19:35:34 +05:30
Masatake Iwasaki 1524e63206 HDFS-9333. Some tests using MiniDFSCluster errored complaining port in use. (iwasakims) 
(cherry picked from commit 964e546ab1)

 Conflicts:
  hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockTokenWithDFSStriped.java

(cherry picked from commit 1b91ebb714)
 2016-09-21 10:52:00 +09:00
Xiao Chen 11ed4f5d40 HDFS-10879. TestEncryptionZonesWithKMS#testReadWrite fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit d31bef575478fe78c0600fb602e117e4b81c9887)
 2016-09-20 16:57:44 -07:00
Xiao Chen e7daeea711 HDFS-10875. Optimize du -x to cache intermediate result. Contributed by Xiao Chen. 
(cherry picked from commit e52d6e7a46)
 2016-09-19 21:45:09 -07:00
Xiao Chen 900dc70a01 HDFS-10489. Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones. Contributed by Xiao Chen. 
(cherry picked from commit ea839bd48e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 20585ade1d9d7fbd8b9df8b1e3db5a74fd4441a6)
 2016-09-19 16:38:00 -07:00
Andrew Wang 8c24388af0 HDFS-10832. Propagate ACL bit and isEncrypted bit in HttpFS FileStatus permissions. 
(cherry picked from commit cba973f036)
(cherry picked from commit f448ce2a89)
 2016-09-19 16:38:00 -07:00
Akira Ajisaka ab57c869e2 HDFS-10862. Typos in 4 log messages. Contributed by Mehran Hassani. 
(cherry picked from commit b09a03cd7d)
(cherry picked from commit 214b6d16d5)
 2016-09-16 16:13:00 +09:00
Arpit Agarwal bdc71bd608 HDFS-10805. Reduce runtime for append test. Contributed by Gergely Novak. 2016-09-14 09:31:35 -07:00
Akira Ajisaka c06cf0f89c HDFS-10856. Update the comment of BPServiceActor$Scheduler#scheduleNextBlockReport. Contributed by Yiqun Lin. 
(cherry picked from commit f0876b8b60)
(cherry picked from commit 764cf431ca)
 2016-09-13 14:51:29 +09:00
Yongjun Zhang b96e2cbd20 HDFS-10657. testAclCLI.xml setfacl test should expect mask r-x. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit d2466ec3e0)
 2016-09-12 22:13:17 -07:00
Xiao Chen 76a18915bf HDFS-9781. FsDatasetImpl#getBlockReports can occasionally throw NullPointerException. Contributed by Manoj Govindassamy. 
(cherry picked from commit bb6d866207)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2016-09-10 23:14:22 -07:00
Arpit Agarwal ed35f99045 HDFS-10830. FsDatasetImpl#removeVolumes crashes with IllegalMonitorStateException when vol being removed is in use. (Arpit Agarwal and Manoj Govindassamy) 2016-09-10 18:34:28 -07:00
Arpit Agarwal 04f620c4d0 HDFS-10742. Measure lock time in FsDatasetImpl. Contributed by Chen Liang. 2016-09-10 18:01:37 -07:00
Andrew Wang 912631a226 HDFS-10831. Add log when URLConnectionFactory.openConnection failed. Contributed by yunjiong zhao. 
(cherry picked from commit b07c266dca)
(cherry picked from commit 72ea641468)
 2016-09-08 18:30:36 -07:00
Arpit Agarwal 162ee0f0a4 HDFS-9038. DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) 2016-09-06 13:52:05 -07:00
Xiao Chen e84b5c5e3e HDFS-10841. Remove duplicate or unused variable in appendFile(). Contributed by Kihwal Lee. 
(cherry picked from commit f6c0b7543f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirAppendOp.java

(cherry picked from commit 3025992bd3)
 2016-09-06 11:54:02 -07:00
Andrew Wang 9b7e079d4b HDFS-10833. Fix JSON errors in WebHDFS.md examples. 
(cherry picked from commit cbd909ce2a)
(cherry picked from commit 40eab77562)
 2016-09-02 14:51:12 -07:00
Kihwal Lee 16c166249c HDFS-10772. Reduce byte/string conversions for get listing. Contributed by Daryn Sharp. 
(cherry picked from commit a1f3293762)
 2016-09-02 12:32:34 -05:00
Kihwal Lee 1c946a2665 HDFS-10768. Optimize mkdir ops. Contributed by Daryn Sharp. 
(cherry picked from commit 442dfd1bbe)
 2016-09-02 12:28:11 -05:00
Kihwal Lee bf350e4480 HDFS-10762. Pass IIP for file status related methods 
(cherry picked from commit a30f6a68fa)
 2016-09-02 12:23:21 -05:00
Kihwal Lee c2708bc5fb HDFS-9621. Consolidate FSDirStatAndListingOp#createFileStatus to let its INodesInPath parameter always include the target INode. Contributed by Jing Zhao. 
(cherry picked from commit 313f03bfda)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
 2016-09-02 12:16:08 -05:00
Kihwal Lee 69bdcd9b1e HDFS-10745. Directly resolve paths into INodesInPath. Contributed by Daryn Sharp. 
(cherry picked from commit 922d045e11)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
 2016-09-02 11:44:01 -05:00
Kihwal Lee 3dac73080a HDFS-10744. Internally optimize path component resolution. Contributed by Daryn Sharp. 
(cherry picked from commit 1ef8d7a638)
 2016-09-02 11:31:48 -05:00
Kihwal Lee 5a779cce8e HDFS-10711. Optimize FSPermissionChecker group membership check. Contributed by Daryn Sharp. 
(cherry picked from commit 43d1279df0)
 2016-09-02 11:24:20 -05:00
Kihwal Lee f09b04b3f5 HDFS-10673. Optimize FSPermissionChecker's internal path usage. Contributed by Daryn Sharp. 
(cherry picked from commit 438a9f047e)
(cherry picked from commit 09b06a6e9e)
 2016-09-01 17:03:48 -07:00
Zhe Zhang 08b6808489 Addendum fix for HDFS-10817 to fix failure of the added testFSReadLockLongHoldingReport in branch-2. 
(cherry picked from commit 6c600360ca)
(cherry picked from commit 776eccd873)
 2016-08-31 23:45:45 -07:00
Zhe Zhang 7adbd569a6 HDFS-10817. Add Logging for Long-held NN Read Locks. Contributed by Erik Krogen. 
(cherry picked from commit 6f4b0d33ca)
(cherry picked from commit f87e2f5d05)
 2016-08-31 15:54:46 -07:00
Kihwal Lee e53f6fde46 HDFS-10674. Optimize creating a full path from an inode. Contributed by Daryn Sharp. 
(cherry picked from commit 22ef5286bc)
(cherry picked from commit a5d12d9c1f)
 2016-08-31 15:16:07 -07:00
Kihwal Lee 7b5e12229d HDFS-10656. Optimize conversion of byte arrays back to path string. Contributed by Daryn Sharp. 
(cherry picked from commit bebf10d245)
(cherry picked from commit ab7072838f)
 2016-08-31 15:11:27 -07:00
Kihwal Lee ce902fc80a HDFS-10729. Improve log message for edit loading failures caused by FS limit checks. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 01721dd88e)
 2016-08-31 14:12:02 -05:00
Wei-Chiu Chuang 80bb9ffb67 HDFS-10760. DataXceiver#run() should not log InvalidToken exception as an error. Contributed by Pan Yuxuan. 
(cherry picked from commit c4ee6915a1)
 2016-08-30 10:44:48 -07:00
Zhe Zhang ee28034a8a HDFS-10814. Add assertion for getNumEncryptionZones when no EZ is created. Contributed by Vinitha Reddy Gankidi. 
(cherry picked from commit 4bd45f54ee)
 2016-08-29 23:43:04 -07:00
Xiao Chen a291306510 HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot be resolved. Contributed by Charles Lamb and John Zhuge. 
(cherry picked from commit 046f6bf0f944311fc70de3a96fc6ab8211cbbbd8)
 2016-08-29 17:42:30 -07:00
Yongjun Zhang 18f7628664 HDFS-10625. VolumeScanner to report why a block is found bad. Contributed by Rushabh S Shah and Yiqun Lin. 
(cherry picked from commit 5d1609ddf2)
 2016-08-29 14:15:23 -07:00
Mingliang Liu f84ae6f091 HDFS-10807. Doc about upgrading to a version of HDFS with snapshots may be confusing. (Contributed by Mingliang Liu) 
(cherry picked from commit 6742fb6e68)
 2016-08-29 10:19:53 -07:00
Yongjun Zhang 6f10a0a87b HDFS-10652. Add a unit test for HDFS-4660. Contributed by Vinayakumar B., Wei-Chiu Chuang, Yongjun Zhang. 
(cherry picked from commit c25817159a)
 2016-08-27 23:11:57 -07:00
Andrew Wang 42e03d7bfe HDFS-10793. Fix HdfsAuditLogger binary incompatibility introduced by HDFS-9184. Contributed by Manoj Govindassamy. 
(cherry picked from commit a445b82baa)
(cherry picked from commit 1c47a356d8)
 2016-08-26 15:40:13 -07:00
Zhe Zhang 96cf681b5d HDFS-10798. Make the threshold of reporting FSNamesystem lock contention configurable. Contributed by Erik Krogen. 
(cherry picked from commit 407b519fb1)
(cherry picked from commit f3a4cbcc60)
 2016-08-26 14:23:51 -07:00
Kihwal Lee 1c5ecb6cc4 HDFS-8915. TestFSNamesystem.testFSLockGetWaiterCount fails intermittently. Contributed by Masatake Iwasaki. 
(cherry picked from commit 13fb1b50e6)
 2016-08-26 10:01:19 -05:00
Xiaoyu Yao db415ea166 HDFS-10748. TestFileTruncate#testTruncateWithDataNodesRestart runs sometimes timeout. Contributed by Yiqun Lin. 
(cherry picked from commit 4da5000dd3)
 2016-08-25 09:22:16 -07:00
Wei-Chiu Chuang bd729af8a8 HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-24 08:53:43 -07:00
Zhe Zhang 8bc33bf343 HADOOP-12765. HttpServer2 should switch to using the non-blocking SslSelectChannelConnector to prevent performance degradation when handling SSL connections. Contributed by Min Shen. Branch-2 patch contributed by Wei-Chiu Chuang. 
(cherry picked from commit dfcbc12026)
 2016-08-23 14:46:47 -07:00
Jason Lowe f57a26378e HDFS-9745. TestSecureNNWithQJM#testSecureMode sometimes fails with timeouts. Contributed by Xiao Chen 
(cherry picked from commit 126d165efd)
 2016-08-23 14:49:27 +00:00
Akira Ajisaka dde3eaed62 HDFS-10783. The option '-maxSize' and '-step' fail in OfflineImageViewer. Contributed by Yiqun Lin. 
(cherry picked from commit e90f3359de)
(cherry picked from commit 94a844b4f6)
 2016-08-23 19:59:07 +09:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) c0c4cc8ee8 HDFS-10692. Update JDiff report's base version for HDFS from 2.6.0 to 2.7.2. Contributed by Wangda Tan. 
(cherry picked from commit dc7a1c54f4)
 2016-08-22 11:37:14 -07:00
Kihwal Lee c9199095f8 HDFS-10738. Fix TestRefreshUserMappings.testRefreshSuperUserGroupsConfiguration test failure. Contributed by Rakesh R. 
(cherry picked from commit 0f701f433d)
(cherry picked from commit 7f1879abe6)
 2016-08-22 11:04:03 -05:00
Jing Zhao 8628ad0790 HDFS-10764. Fix INodeFile#getBlocks to not return null. Contributed by Arpit Agarwal. 
(cherry picked from commit 0faee62a0c)
(cherry picked from commit fff0418458)
 2016-08-19 22:27:24 -07:00
Kihwal Lee 4f3df8e22a HDFS-7959. WebHdfs logging is missing on Datanode (Kihwal Lee via sjlee) 
(cherry picked from commit ae90d4dd90)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-19 15:32:11 -05:00
Arpit Agarwal 1fe08c919a HDFS-10682. Replace FsDatasetImpl object lock with a separate lock object. (Contributed by Chen Liang) 2016-08-17 16:22:07 -07:00
Xiao Chen b89d79ca1d HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin. 
(cherry picked from commit 85aacaadb5a3f8c78b191867c0bde09b3c4b3c3c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java

(cherry picked from commit a20b943cf9)
 2016-08-17 16:00:33 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) d65024eddc Revert "HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla." 
Reverting this on branch-2.* as it's an incompatible change.

This reverts commit 83f7f62be3.

(cherry picked from commit 2486c4c63a)
 2016-08-17 13:34:35 -07:00
Kihwal Lee b76f6e5de3 HDFS-10342. BlockManager#createLocatedBlocks should not check corrupt replicas if none are corrupt. Contributed by Kuhu Shukla. 
(cherry picked from commit b10c936020)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 65a41696b2)
 2016-08-16 09:39:33 -05:00
Mingliang Liu 54c9743263 HDFS-10641. TestBlockManager#testBlockReportQueueing fails intermittently. (Contributed by Daryn Sharp) 2016-08-15 20:46:48 -07:00
Mingliang Liu e9eaad0e67 HDFS-10747. o.a.h.hdfs.tools.DebugAdmin usage message is misleading. (Contributed by Mingliang Liu) 2016-08-15 20:46:34 -07:00
Mingliang Liu 0b934c375e HDFS-10678. Documenting NNThroughputBenchmark tool. (Contributed by Mingliang Liu) 2016-08-15 20:46:28 -07:00
Kihwal Lee 1a8280edde HDFS-10763. Open files can leak permanently due to inconsistent lease update. Contributed by Kihwal Lee. 
(cherry picked from commit 864f878d59)
(cherry picked from commit e78db7d2a4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2016-08-15 17:37:39 -05:00
Kihwal Lee 579709fc22 HDFS-9696. Garbage snapshot records linger forever. Contributed by Kihwal Lee 
(cherry picked from commit 83e57e083f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java

(cherry picked from commit 4766a3153d)
 2016-08-15 13:13:49 -05:00
Wei-Chiu Chuang 53f097a474 HDFS-10731. FSDirectory#verifyMaxDirItems does not log path name. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9019606b69)
(cherry picked from commit 20a8ff0a8d)
 2016-08-12 06:11:22 -07:00
Zhe Zhang 9c1c48c0d4 HDFS-9888. Allow reseting KerberosName in unit tests. Contributed by Xiao Chen. 
(cherry picked from commit 3e8099a45a)
(cherry picked from commit 00ff3d737c)
 2016-08-11 12:42:17 -07:00
Akira Ajisaka f29d0739d7 HDFS-10336. TestBalancer failing intermittently because of not reseting UserGroupInformation completely. Contributed by Yiqun Lin. 
(cherry picked from commit 932aed64d7)
(cherry picked from commit 0529da7239)
 2016-08-11 12:42:17 -07:00
Zhe Zhang b8b4ea67d8 HDFS-9804. Allow long-running Balancer to login with keytab. Contributed by Xiao Chen. 
(cherry picked from commit ccff6035f5)
(cherry picked from commit 7d402692b4)
 2016-08-11 12:42:17 -07:00
Wei-Chiu Chuang 65e3657149 HDFS-8897. Balancer should handle fs.defaultFS trailing slash in HA. Contributed by John Zhuge. 
(cherry picked from commit f1d5a95cf3)
 2016-08-10 22:30:06 -07:00
Xiaoyu Yao 7a1a3f7754 HDFS-10643. Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit ec289bbece)
 2016-08-10 15:56:56 -07:00
Xiaoyu Yao 8808df09c6 Revert "Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao." 
This reverts commit 9f816d7ca7.

(cherry picked from commit fdafd0cb4f)
 2016-08-10 15:56:43 -07:00
Xiaoyu Yao c0e331935d Namenode should use loginUser(hdfs) to generateEncryptedKey. Contributed by Xiaoyu Yao. 
(cherry picked from commit 9f816d7ca7)
 2016-08-10 15:19:10 -07:00
Wei-Chiu Chuang 753edc4931 HDFS-8224. Schedule a block for scanning if its metadata file is corrupt. Contributed by Rushabh S Shah. 
(cherry picked from commit 8efd4959f3)
 2016-08-10 11:39:12 -07:00
Yuanbo Liu d46f5ca7e2 HDFS-10694. processReport() should print blockReportId in each log message. Contributed by Yuanbo Liu. 
(cherry picked from commit 10e84c6a6e)
 2016-08-10 11:12:21 -07:00
Konstantin V Shvachko 6b3114a2a5 HDFS-10693. metaSave should print blocks, not LightWeightHashSet. Contributed by Yuanbo Liu. 
(cherry picked from commit 4d3af47f27)
 2016-08-07 14:36:24 -07:00
Kihwal Lee 5b3deac2e3 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit f715f14185)
 2016-08-05 11:43:43 -05:00
Akira Ajisaka 5758026833 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 
(cherry picked from commit ef432579a7)
(cherry picked from commit 6863866127)
 2016-08-05 17:33:07 +09:00
Kihwal Lee e8cd1439d3 HDFS-10343. BlockManager#createLocatedBlocks may return blocks on failed storages. Contributed by Kuhu Shukla. 
(cherry picked from commit 57369a678c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 690ec789f0)
 2016-08-04 15:27:51 -05:00
Kihwal Lee eb9af6ca34 HDFS-10717. Fix findbugs warnings of hadoop-hdfs-client in branch-2. Contributed by Akira Ajisaka. 
(cherry picked from commit 6ead6f3bb8)
 2016-08-04 12:36:51 -05:00
Arpit Agarwal f3424d662a HDFS-10722. Fix build break. 2016-08-04 10:28:43 -07:00
Kihwal Lee 3a85aea42d HDFS-10722. Fix race condition in TestEditLog#testBatchedSyncWithClosedLogs. Contributed by Daryn Sharp. 
(cherry picked from commit 4a8e6dc02f)
 2016-08-04 11:56:26 -05:00
Tsz-Wo Nicholas Sze cefa21e98a In Balancer, the target task should be removed when its size < 0. Contributed by Yiqun Lin 2016-08-04 09:54:15 -07:00
Kihwal Lee 58df27b873 HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp. 
(cherry picked from commit 6ae39199da)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

(cherry picked from commit 77b61d1f4e)
 2016-08-04 09:45:03 -05:00
Kihwal Lee 804db5b277 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 
(cherry picked from commit c0166b0743)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
 2016-08-04 09:35:16 -05:00
Kihwal Lee ceac628378 HDFS-10569. A bug causes OutOfIndex error in BlockListAsLongs. Contributed by Weiwei Yang. 
(cherry picked from commit 6f63566694)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java

(cherry picked from commit 1ca2ab4013)
 2016-08-03 15:28:13 -05:00
Jing Zhao 5bdee64bde HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 
(cherry picked from commit f4ba5ff1d7)
(cherry picked from commit 180a26d9fb)
 2016-08-03 12:44:49 -07:00
Kihwal Lee 33a87ffe10 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 
(cherry picked from commit 58db263e93)
 2016-08-03 12:19:05 -05:00
Kihwal Lee 288f9ccde2 HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 
(cherry picked from commit 5e5b8793fb)
 2016-08-02 15:43:11 -05:00
Konstantin V Shvachko 9b3b7703b1 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 9ab153d7ae.
 2016-08-01 22:48:09 -07:00
Daryn Sharp ecb143ed21 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 
(cherry picked from commit 9f473cf903)
 2016-08-01 10:18:45 -05:00
Yongjun Zhang cc066de22c HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 8d32bd884d)
 2016-07-29 23:47:46 -07:00
Akira Ajisaka e467f838bb HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 
(cherry picked from commit 204a2055b1)
(cherry picked from commit 0cff416c35)
 2016-07-29 15:41:17 +09:00
Xiaoyu Yao ac510e5cd0 HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:16:44 -07:00
Akira Ajisaka 960caf624d HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 
(cherry picked from commit 414fbfab41)
(cherry picked from commit ab49c2b399)
 2016-07-28 16:59:49 +09:00
Yongjun Zhang 71c49b1786 HDFS-10667. Report more accurate info about data corruption location. (Yuanbo Liu via Yongjun Zhang) 
(cherry picked from commit eb7ff0c992)
 2016-07-27 15:49:56 -07:00
Brahma Reddy Battula 5dc29ab0f2 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 096d3fa25c)
 2016-07-27 10:31:29 +05:30
Arpit Agarwal 45a6ee97b2 HDFS-10642. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. (Contributed by Mingliang Liu) 2016-07-26 12:45:06 -07:00
Wei-Chiu Chuang 53bf632702 HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. Contributed by Kai Sasaki. 
(cherry picked from commit 25f0a9b3ab)
 2016-07-26 08:31:00 -07:00
Steve Loughran 7921ad949f Revert "HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu." 
This reverts commit 32468b1ba5.
 2016-07-26 14:31:02 +01:00
Masatake Iwasaki 6b5d546014 HDFS-10671. Fix typo in HdfsRollingUpgrade.md. Contributed by Yiqun Lin. 
(cherry picked from commit 59466b8c18)
 2016-07-26 19:52:23 +09:00
Brahma Reddy Battula 32468b1ba5 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 7cac7655fd)
 2016-07-26 11:00:11 +05:30
Akira Ajisaka e377e94f64 Revert "HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg." 
This reverts commit 91f47ab5f9.
 2016-07-26 11:06:10 +09:00
Vinitha Reddy Gankidi 9ab153d7ae HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:59:42 -07:00
Jing Zhao 62800dc8b6 HDFS-10688. BPServiceActor may run into a tight loop for sending block report when hitting IOException. Contributed by Chen Liang. 
(cherry picked from commit 0cde9e12a7)
(cherry picked from commit 30c772fc59)
 2016-07-25 18:44:06 -07:00
Arpit Agarwal 00cbd3136e HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-25 09:51:59 -07:00
Jing Zhao 54be36a6d6 HDFS-9894. Add unsetStoragePolicy API to FileContext/AbstractFileSystem and derivatives. Contributed by Xiaobing Zhou. 
(cherry picked from commit 7149cdb3c2d9dd390cd8668883cbe5db94090e0a)
 2016-07-25 09:51:59 -07:00
Arpit Agarwal bbf711c030 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-07-25 09:51:59 -07:00
Xiao Chen 8559442ed1 HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 132deb4cac)
 2016-07-22 15:04:21 -07:00
Jing Zhao 6a7fe835f1 HDFS-10653. Optimize conversion from path string to components. Contributed by Daryn Sharp. 
(cherry picked from commit bd3dcf46e2)
 2016-07-21 11:29:27 -07:00
Akira Ajisaka 2f14d585c4 HDFS-10287. MiniDFSCluster should implement AutoCloseable. Contributed by Andras Bokor. 
(cherry picked from commit fcde6940e0)
(cherry picked from commit 1222889f1a)
 2016-07-21 11:28:46 -07:00
Akira Ajisaka 91f47ab5f9 HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 
(cherry picked from commit f7dabe3add)
(cherry picked from commit 2685e16f38)
 2016-07-19 15:19:02 -07:00
Akira Ajisaka d8181a8cb8 HDFS-10603. Fix flaky tests in org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot. Contributed by Yiqun Lin. 
(cherry picked from commit fc570b55b9)
(cherry picked from commit c92601e981)
 2016-07-19 13:52:17 -07:00
Jitendra Pandey a363277be5 HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:32:20 -07:00
Akira Ajisaka 83126ac90d HDFS-10628. Log HDFS Balancer exit message to its own log. Contributed by Jiayi Zhou. 
(cherry picked from commit c48e9d608c)
(cherry picked from commit 6171498375)
 2016-07-15 11:43:54 -07:00
Akira Ajisaka 6a04d9ab63 HADOOP-13351. TestDFSClientSocketSize buffer size tests are flaky. Contributed by Aaron Fabbri and Mingliang Liu. 
(cherry picked from commit 5537c6b234)
(cherry picked from commit c9f5348bbf)
 2016-07-14 14:42:55 -07:00
Zhe Zhang af94dfaa53 HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. 
(cherry picked from commit 087290e6b1)
(cherry picked from commit 5b4e916b3c)
 2016-07-12 23:28:16 -07:00
Jitendra Pandey 35108025e7 HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS protection. Contributed by Anu Engineer. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
 2016-07-12 14:38:34 -07:00
Yongjun Zhang 96e68e722f HDFS-10512. VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. Contributed by Wei-Chiu Chuang and Yiqun Lin. 
(cherry picked from commit da6f1b88dd)
 2016-07-08 22:20:27 -07:00
Jitendra Pandey 9c7ce389c7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:04:47 -07:00
Uma Maheswara Rao G 097022e172 HDFS-10592:Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning. Contributed by Rakesh R 
(cherry picked from commit ab092c56c2)
 2016-07-07 14:02:25 -07:00
Andrew Wang f826d92909 HDFS-10488. Update WebHDFS documentation regarding CREATE and MKDIR default permissions. Contributed by Wellington Chevreuil. 
(cherry picked from commit 647b084738b26dcf0c0c4afa4ec3039e6aca2d61)
 2016-07-07 10:37:16 -07:00
Akira Ajisaka 4fba09d7e8 HDFS-10582. Change deprecated configuration fs.checkpoint.dir to dfs.namenode.checkpoint.dir in HDFS Commands Doc. Contributed by Pan Yuxuan. 
(cherry picked from commit a3f93beaa1)
(cherry picked from commit 8a9440298d)
 2016-07-06 17:54:44 -07:00
Akira Ajisaka 4b263b30c2 HDFS-10589. Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params. Contributed by shenyinjie. 
(cherry picked from commit 8b4b5259d5)
(cherry picked from commit 0401adcaca)
 2016-07-04 07:15:54 -07:00
Jitendra Pandey bc7fd76a1f HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:37:32 -07:00
Vinayakumar B f44d7782f6 HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) 
(cherry picked from commit 2a0082c51d)
(cherry picked from commit 1e34763181)
 2016-06-28 16:55:57 +05:30
Akira Ajisaka 700dacf8c4 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 
(cherry picked from commit d433b16ce6)
(cherry picked from commit ad50a64ded)
 2016-06-22 11:02:48 +09:00
Brahma Reddy Battula aa8f4cc48c HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 
(cherry picked from commit f2ac132d6a)
(cherry picked from commit ee0f389ecd)
 2016-06-21 15:55:57 +05:30
Colin Patrick Mccabe 9b51420ea5 HDFS-10448. CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1 (Yiqun Lin via cmccabe) 
(cherry picked from commit 46f1602e89)
(cherry picked from commit 4e11f33ccc)
 2016-06-20 18:32:00 -07:00
Jitendra Pandey 29c3ee9744 HDFS-10538. Remove AsyncDistributedFileSystem. Contributed by Xiaobing Zhou and Jitendra Pandey. 2016-06-20 18:05:32 -07:00
Jitendra Pandey 2f95d22b17 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:31:54 -07:00
Ravi Prakash 588ea6d431 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 
(cherry picked from commit ae047655f4)
(cherry picked from commit 9c5f7f290e)
 2016-06-20 14:38:05 -07:00
Brahma Reddy Battula 9d27530c36 HDFS-10474. hftp copy fails when file name with Chinese+special char in branch-2 (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 3f27f40503)
 2016-06-20 12:40:11 +05:30
Andrew Wang 3f2d2cd0a1 Fix hadoop-hdfs-native-client compilation on Windows. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 3ff0510ffd)
(cherry picked from commit 94759470b8)
 2016-06-17 15:55:43 -07:00
Kihwal Lee edd5373095 HDFS-10493. Add links to datanode web UI in namenode datanodes page. Contributed by Weiwei Yang. 
(cherry picked from commit 280069510b)
 2016-06-17 11:13:12 -05:00
Akira Ajisaka 242263c56d HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 
(cherry picked from commit 09e82acaf9)
(cherry picked from commit a36aa920f4)
 2016-06-17 16:21:31 +09:00
Colin Patrick Mccabe cc53ca8d69 HDFS-9466. TestShortCircuitCache#testDataXceiverCleansUpSlotsOnFailure is flaky (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit c7921c9bdd)
(cherry picked from commit 49c0d9b325)
 2016-06-16 13:53:55 -07:00
Arpit Agarwal 394dae17ac HDFS-10532. Typo in RollingUpgrade docs. (Contributed by Yiqun Lin) 2016-06-16 10:11:05 -07:00
Colin Patrick Mccabe ea2e7321d6 HDFS-10525. Fix NPE in CacheReplicationMonitor#rescanCachedBlockMap (Xiao Chen via cmccabe) 
(cherry picked from commit 2ca73445f5)
(cherry picked from commit 8280e3bcc2)
 2016-06-15 22:51:27 -07:00
Colin Patrick Mccabe ef08eb9031 HDFS-10505. OIV's ReverseXML processor should support ACLs (Surendra Singh Lilhore via cmccabe) 
(cherry picked from commit 2449db507d)
(cherry picked from commit 17301ab395)
 2016-06-15 22:44:41 -07:00
Akira Ajisaka b35c7cdd14 HADOOP-12892. fix/rewrite create-release. Contributed by Allen Wittenauer. 2016-06-16 14:06:08 +09:00
Ming Ma 03ed3f9b66 Revert "HDFS-9016. Display upgrade domain information in fsck. (mingma)" 
This reverts commit 68ab0aa9d0.
 2016-06-14 20:14:00 -07:00
Ming Ma 68ab0aa9d0 HDFS-9016. Display upgrade domain information in fsck. (mingma) 
(cherry picked from commit b7436f4468)
 2016-06-14 20:08:29 -07:00
Andrew Wang 56e29d2711 HDFS-10516. Fix bug when warming up EDEK cache of more than one encryption zone. Contributed by Xiao Chen. 
(cherry picked from commit 709a814fe0)
(cherry picked from commit ff4466fe15)
 2016-06-13 11:50:27 -07:00
Andrew Wang f583fa5f4e Remove redundant TestMiniDFSCluster.testDualClusters. Contributed by Jiayi Zhou. 
(cherry picked from commit 1ee9ea0026)
(cherry picked from commit 9319665461)
 2016-06-08 12:59:08 -07:00
Zhe Zhang 72addd8d5c HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster. 
(cherry picked from commit 6de9213df1)
(cherry picked from commit 06e38c835d)
(cherry picked from commit 0499907466)
 2016-06-07 09:09:18 -07:00
Jason Lowe 738b67e9d3 HDFS-9580. TestComputeInvalidateWork#testDatanodeReRegistration failed due to unexpected number of invalidate blocks. Contributed by Wei-Chiu Chuang. 
Change-Id: I9411dfb1dd4605cbcc515ca3361218d35d7c5fe3
(cherry picked from commit 0087734cc1)

Conflicts:

	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-06-07 15:36:53 +00:00
Wangda Tan e91eba2f81 Revert "HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster." 
This reverts commit 0274636529.
 2016-06-06 22:13:02 -07:00
Zhe Zhang 0274636529 HDFS-10458. getFileEncryptionInfo should return quickly for non-encrypted cluster. 
(cherry picked from commit 6de9213df1)
(cherry picked from commit 06e38c835d)
 2016-06-06 15:54:36 -07:00
Tsz-Wo Nicholas Sze 49535c0683 Revert "Revert "HADOOP-13226 Support async call retry and failover."" 
This reverts commit 886e239606.
 2016-06-06 16:54:02 +08:00
Tsz-Wo Nicholas Sze 359a316736 Revert "Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou."" 
This reverts commit f176311366.
 2016-06-06 16:53:56 +08:00
Tsz-Wo Nicholas Sze 025f62afcc Revert "Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou"" 
This reverts commit 0af02cfd37.
 2016-06-06 16:53:51 +08:00
Tsz-Wo Nicholas Sze 79beac49ec Revert "Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit 1de712f22a.
 2016-06-06 16:53:46 +08:00
Tsz-Wo Nicholas Sze c949001eee Revert "Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls."" 
This reverts commit 308d28640d.
 2016-06-06 16:53:41 +08:00
Tsz-Wo Nicholas Sze 5b054b1bfc Revert "Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit ffa85f28a2.
 2016-06-06 16:53:36 +08:00
Tsz-Wo Nicholas Sze 8ccc8aeb6c Revert "Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou"" 
This reverts commit 8a07026e77.
 2016-06-06 16:53:31 +08:00
Tsz-Wo Nicholas Sze 2cf663bb59 Revert "Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou"" 
This reverts commit d740a90260.
 2016-06-06 16:53:24 +08:00
Andrew Wang d740a90260 Revert "HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit bdc45bef64.
 2016-06-03 18:17:33 -07:00
Andrew Wang 8a07026e77 Revert "HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou" 
This reverts commit a1ba6eee38.
 2016-06-03 18:17:33 -07:00
Andrew Wang ffa85f28a2 Revert "HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit ac04900450.
 2016-06-03 18:12:31 -07:00
Andrew Wang 308d28640d Revert "HADOOP-13168. Support Future.get with timeout in ipc async calls." 
This reverts commit fa7c7f2510.
 2016-06-03 18:12:31 -07:00
Andrew Wang 1de712f22a Revert "HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou" 
This reverts commit 339b803a52.
 2016-06-03 18:12:31 -07:00
Andrew Wang 0af02cfd37 Revert "HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou" 
This reverts commit 2e755a7f0e.
 2016-06-03 18:12:31 -07:00
Andrew Wang f176311366 Revert "HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou." 
This reverts commit de69b1da34.
 2016-06-03 18:12:30 -07:00
Andrew Wang 886e239606 Revert "HADOOP-13226 Support async call retry and failover." 
This reverts commit a8941d7790.
 2016-06-03 18:12:30 -07:00
Andrew Wang 6f69113417 HDFS-10481. HTTPFS server should correctly impersonate as end user to open file. Contributed by Xiao Chen. 
(cherry picked from commit 47e0321ee9)
(cherry picked from commit d5609e3499)
 2016-06-03 17:21:34 -07:00
Andrew Wang d712b2ee3b HADOOP-13155. Implement TokenRenewer to renew and cancel delegation tokens in KMS. Contributed by Xiao Chen. 
(cherry picked from commit 713cb71820)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/KeyProviderDelegationTokenExtension.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java

(cherry picked from commit 03c4724c88)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
 2016-06-03 17:03:40 -07:00
Akira Ajisaka d0dc5aaa2d HDFS-10341. Add a metric to expose the timeout number of pending replication blocks. (aajisaka) 
(cherry picked from commit b6d5546e24)
 2016-06-03 16:04:40 +09:00
Akira Ajisaka a8a2f4b500 HDFS-10471. DFSAdmin#SetQuotaCommand's help msg is not correct. Contributed by Yiqun Lin. 
(cherry picked from commit 1df6f5735c)
(cherry picked from commit cddf6b4fc7)
 2016-06-03 04:11:13 +09:00
Masatake Iwasaki 5b3d9ca0cd HDFS-10367. TestDFSShell.testMoveWithTargetPortEmpty fails with Address bind exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit aadb77e412)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

(cherry picked from commit 3740974413)
 2016-06-02 21:04:48 +09:00
Akira Ajisaka fe4ad4a39c HDFS-9476. TestDFSUpgradeFromImage#testUpgradeFromRel1BBWImage occasionally fail. Contributed by Masatake Iwasaki. 
(cherry picked from commit 69555fca06)
(cherry picked from commit 3dd557e4b6)
 2016-06-02 18:53:42 +09:00
Colin Patrick Mccabe 6fe1c67fa1 HADOOP-13137. TraceAdmin should support Kerberized cluster (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit 8ceb06e239)
(cherry picked from commit a64e1425ed)
 2016-05-31 18:26:37 -07:00
Colin Patrick Mccabe 0d6a65a40a HDFS-10415. TestDistributedFileSystem#MyDistributedFileSystem attempts to set up statistics before initialize() is called (Mingliang Liu via cmccabe) 
(cherry picked from commit 14907d9b8c)
 2016-05-31 17:27:14 -07:00
Tsz-Wo Nicholas Sze a8941d7790 HADOOP-13226 Support async call retry and failover. 2016-05-31 17:16:44 -07:00
Chris Nauroth de69b1da34 HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou. 
(cherry picked from commit 21890c4239)
(cherry picked from commit 193f822b9a)
 2016-05-27 14:20:50 -07:00
Yongjun Zhang 3e4c7906c2 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang) 
(cherry picked from commit 5ea6fd85c7)
 2016-05-27 10:18:47 -07:00
Tsz-Wo Nicholas Sze 2e755a7f0e HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:24:21 -07:00
Tsz-Wo Nicholas Sze 339b803a52 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-26 12:23:05 -07:00
Tsz-Wo Nicholas Sze fa7c7f2510 HADOOP-13168. Support Future.get with timeout in ipc async calls. 2016-05-26 12:22:57 -07:00
Tsz-Wo Nicholas Sze ac04900450 HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-26 12:20:31 -07:00
Tsz-Wo Nicholas Sze f6dfb71762 HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-05-25 12:26:25 -07:00
Vinayakumar B 00011e6a98 HDFS-10217. show 'blockScheduled' tooltip in datanodes table. 
(cherry picked from commit 2dab64da2f)
(cherry picked from commit e28a51e988)
 2016-05-25 12:35:42 +05:30
Tsz-Wo Nicholas Sze 2ab789b19e HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 13:11:32 -07:00
Masatake Iwasaki 1ab79e255c HDFS-10439. Update setOwner doc in HdfsPermissionsGuide. Contributed by John Zhuge. 
(cherry picked from commit 12eb5a7b11)
 2016-05-22 19:43:47 +09:00
Chris Nauroth 5b907a17ed HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth. 
(cherry picked from commit 500e946729)
(cherry picked from commit e7b56cf93c)
 2016-05-20 13:02:06 -07:00
Chris Nauroth 877af3ec7e HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth. 
(cherry picked from commit bcde1562d2)
(cherry picked from commit d38f2090fa)
 2016-05-20 12:50:23 -07:00
Kihwal Lee a81150ae9e HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 204de430bf)
 2016-05-19 15:54:18 -05:00
Arpit Agarwal 01a0730996 HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:25:33 -07:00
Xiaoyu Yao ccb1cade5b HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 18:00:18 -07:00
Yongjun Zhang 2bcf1eb36a HDFS-10381, DataStreamer DataNode exclusion log message should be warning. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 16c07cc68a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java

(cherry picked from commit 456fe08a6d)
 2016-05-17 16:00:49 -07:00
Kihwal Lee 01de843a31 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit cb5a1edd1c.
 2016-05-17 15:21:41 -05:00
Kihwal Lee cb5a1edd1c HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 0942954e8a)
 2016-05-17 15:11:40 -05:00
Kihwal Lee 362a778bf5 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4a5819dae2)
 2016-05-17 09:00:59 -05:00
Akira Ajisaka 388b2010ae HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 
(cherry picked from commit 9fe5828f05)
(cherry picked from commit 7f03c1c90e)
 2016-05-17 15:56:36 +09:00
Andrew Wang 29ed65ff9b HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 
(cherry picked from commit 45788204ae)
(cherry picked from commit 413f3dabbd)
 2016-05-15 22:15:18 -07:00
Colin Patrick Mccabe 5f04795aed HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 
(cherry picked from commit 687233f20d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

(cherry picked from commit 15bfcde238)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2016-05-11 14:24:51 -07:00
Colin Patrick Mccabe 154332bb54 HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO level (Yiqun Lin via cmccabe) 
(cherry picked from commit f0252ad252)
(cherry picked from commit 43f32d3f4a)
 2016-05-10 11:21:16 -07:00
Akira Ajisaka a84c4d7d4b HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. Contributed by Gergely Novák. 
(cherry picked from commit 901588542b)
 2016-05-10 09:33:55 -07:00
Akira Ajisaka 1622fc1373 Revert "HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated." 
This reverts commit f25dcb935e.
 2016-05-10 09:33:18 -07:00
Akira Ajisaka f25dcb935e HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. 
(cherry picked from commit c297df5995)
 2016-05-10 09:29:47 -07:00
Kihwal Lee d7cda6d031 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah. 
(cherry picked from commit b9e5a32fa1)
 2016-05-09 15:52:12 -05:00
Masatake Iwasaki c80b47e73f HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. 
(cherry picked from commit 2835f146b5)
 2016-05-06 15:50:48 +09:00
Xiaoyu Yao 59eeee7501 HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit dacd1f50fe)
(cherry picked from commit 35859872b7)
 2016-05-05 18:23:00 -07:00
Ming Ma d6e95ae47b HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 
(cherry picked from commit 1268cf5fbe)
 2016-05-04 17:07:36 -07:00
Arpit Agarwal ce94203566 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:53:07 -07:00
Xiaoyu Yao 5dfcfb2d2d HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao. 
(cherry picked from commit 45a753ccf7)
 2016-05-02 19:50:10 -07:00
Xiaoyu Yao 643d2fd3c1 HADOOP-12985. Support MetricsSource interface for DecayRpcScheduler Metrics. Contributed by Xiaoyu Yao. 
(cherry picked from commit 5bd7b592e5)
(cherry picked from commit a2a65922dd)
 2016-05-02 19:38:03 -07:00
Harsh J cf6efccebb HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) 
(cherry picked from commit 54e612bfb9)
 2016-05-02 17:53:10 -07:00
Tsz-Wo Nicholas Sze a1ba6eee38 HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:34:59 -07:00
Tsz-Wo Nicholas Sze bdc45bef64 HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-02 11:34:22 -07:00
Lei Xu 5477c31c5c HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 0a152103f1)
(cherry picked from commit 4289cb8b36)
 2016-05-02 10:21:59 -07:00
Kihwal Lee e9cd88d484 HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah. 
(cherry picked from commit 7da540d03e)
 2016-04-29 15:58:45 -05:00
Tsz-Wo Nicholas Sze 31ed34c982 HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:27:14 -07:00
Kihwal Lee a4db3729be HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 
(cherry picked from commit af9b000535)
 2016-04-29 10:05:58 -05:00
Andrew Wang bdb0e98483 HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge. 2016-04-27 17:03:36 -07:00
Kihwal Lee dd7c9f5b8f HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 
(cherry picked from commit e181092b86)
 2016-04-27 08:30:29 -05:00
Kihwal Lee 621be60592 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun. 
(cherry picked from commit c6a2430b47)
 2016-04-26 11:47:15 -05:00
Kihwal Lee 973c73aa68 HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 
(cherry picked from commit 10f0f7851a)
 2016-04-25 11:42:17 -05:00
Masatake Iwasaki c4d8090b72 HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 
(cherry picked from commit 6fcde2e38d)
 2016-04-23 23:55:27 +09:00
Kihwal Lee 3d3ed9b68e HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang. 
(cherry picked from commit 5d4255a801)
 2016-04-22 11:10:36 -05:00
Tsz-Wo Nicholas Sze 60c68c65c1 HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:45:36 -07:00
Tsz-Wo Nicholas Sze 704c57e3eb HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:25:49 -07:00
Chris Nauroth 7fb1cfc591 HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 
(cherry picked from commit 63ac2db59a)
(cherry picked from commit cc56bed686)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
 2016-04-20 14:00:06 -07:00
Colin Patrick Mccabe e2968f5da2 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe) 
(cherry picked from commit ad36fa6f42)
(cherry picked from commit 76d963acb4)
 2016-04-20 11:52:49 -07:00