Commit Graph

4045 Commits

Author SHA1 Message Date
Tsz-Wo Nicholas Sze f7189d6a1c HADOOP-13226 Support async call retry and failover. 2016-05-31 17:01:43 -07:00
Chris Nauroth 193f822b9a HDFS-10430. Reuse FileSystem#access in TestAsyncDFS. Contributed by Xiaobing Zhou.
(cherry picked from commit 21890c4239)
2016-05-27 14:20:30 -07:00
Yongjun Zhang 2dfaedeb43 HDFS-10276. HDFS should not expose path info that user has no permission to see. (Yuanbo Liu via Yongjun Zhang)
(cherry picked from commit 5ea6fd85c7)
2016-05-27 10:14:30 -07:00
Karthik Kambatla b4c8729cf9 HDFS-10463. TestRollingFileSystemSinkWithHdfs needs some cleanup. (Daniel Templeton via kasha) 2016-05-26 14:39:07 -07:00
Tsz-Wo Nicholas Sze 72773f8ea8 HDFS-10431 Refactor and speedup TestAsyncDFSRename. Contributed by Xiaobing Zhou 2016-05-26 12:16:40 -07:00
Tsz-Wo Nicholas Sze e5dab68066 HDFS-8057 Move BlockReader implementation to the client implementation package. Contributed by Takanobu Asanuma 2016-05-25 12:12:27 -07:00
Vinayakumar B e28a51e988 HDFS-10217. show 'blockScheduled' tooltip in datanodes table.
(cherry picked from commit 2dab64da2f)
2016-05-25 12:35:15 +05:30
Karthik Kambatla afe603d973 HDFS-9782. RollingFileSystemSink should have configurable roll interval. (Daniel Templeton via kasha)
(cherry picked from commit 57c31a3fef)
2016-05-24 16:34:45 -07:00
Tsz-Wo Nicholas Sze 56745a0cf4 HDFS-10390. Implement asynchronous setAcl/getAclStatus for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-24 14:52:50 -07:00
Tsz-Wo Nicholas Sze 076a73f207 HDFS-9365. Balaner does not work with the HDFS-6376 HA setup. 2016-05-24 13:08:04 -07:00
Masatake Iwasaki 094bc727c5 HDFS-10439. Update setOwner doc in HdfsPermissionsGuide. Contributed by John Zhuge.
(cherry picked from commit 12eb5a7b11)
2016-05-22 19:15:28 +09:00
Chris Nauroth e7b56cf93c HDFS-10438. When NameNode HA is configured to use the lifeline RPC server, it should log the address of that server. Contributed by Chris Nauroth.
(cherry picked from commit 500e946729)
2016-05-20 13:01:59 -07:00
Chris Nauroth d38f2090fa HDFS-10424. DatanodeLifelineProtocol not able to use under security cluster. Contributed by Chris Nauroth.
(cherry picked from commit bcde1562d2)
2016-05-20 12:47:52 -07:00
Tsz-Wo Nicholas Sze 658072d62e HADOOP-13168. Support Future.get with timeout in ipc async calls. 2016-05-19 15:58:15 -07:00
Kihwal Lee bae151801d HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu.
(cherry picked from commit 204de430bf)
2016-05-19 15:53:41 -05:00
Andrew Wang a44c87c2d0 HDFS-2173. saveNamespace should not throw IOE when only one storage directory fails to write VERSION file. Contributed by Andras Bokor.
(cherry picked from commit 010e6ac328)
2016-05-18 14:13:47 -07:00
Xiaoyu Yao 1470af8c95 HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu.
(cherry picked from commit ccb1cade5b)
2016-05-18 11:02:31 -07:00
Arpit Agarwal 24bed7768d HDFS-9926. MiniDFSCluster leaks dependency Mockito via DataNodeTestUtils. (Contributed by Josh Elser) 2016-05-18 09:21:41 -07:00
Lei Xu 2d1d2dde17 HDFS-10360. DataNode may format directory and lose blocks if current/VERSION is missing. (Wei-Chiu Chuang via lei)
(cherry picked from commit bae11b95d492e0be7c5576097424879c2539b474)
2016-05-17 16:37:36 -07:00
Yongjun Zhang 456fe08a6d HDFS-10381, DataStreamer DataNode exclusion log message should be warning. (John Zhuge via Yongjun Zhang)
(cherry picked from commit 16c07cc68a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/StripedDataStreamer.java
2016-05-17 15:59:00 -07:00
Junping Du 7e36b9159e MAPREDUCE-6657. Job history server can fail on startup when NameNode is in start phase. Contributed by Haibo Chen.
(cherry picked from commit f6ef876fe1)
2016-05-17 14:42:11 -07:00
Kihwal Lee 1e7eb2d2ac Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu."
This reverts commit 70ee477e5f.
2016-05-17 15:20:59 -05:00
Kihwal Lee 70ee477e5f HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu.
(cherry picked from commit 0942954e8a)
2016-05-17 15:10:47 -05:00
Colin Patrick Mccabe 6ebb92c9c1 HDFS-10404. Fix formatting of CacheAdmin command usage help text (Yiqun Lin via cmccabe)
(cherry picked from commit 7cd5ae62f6)
2016-05-17 11:59:10 -07:00
Kihwal Lee 48be0ff67a HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 4a5819dae2)
2016-05-17 08:58:53 -05:00
Akira Ajisaka 7f03c1c90e HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi.
(cherry picked from commit 9fe5828f05)
2016-05-17 15:56:21 +09:00
Sangjin Lee 9330a7b4de HDFS-10208. Addendum for HDFS-9579: to handle the case when client machine can't resolve network path (Ming Ma via sjlee)
(cherry picked from commit 61f46be071)
2016-05-16 18:59:19 -07:00
Lei Xu 09a613b023 HDFS-10410. RedundantEditLogInputStream.LOG is set to wrong class. (John Zhuge via lei)
(cherry picked from commit 6a6e74acf5)
2016-05-16 17:07:03 -07:00
Andrew Wang 413f3dabbd HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin.
(cherry picked from commit 45788204ae)
2016-05-15 22:15:10 -07:00
Lei Xu 782441d47e HDFS-9389. Add maintenance states to AdminStates. (Ming Ma via lei)
(cherry picked from commit a2774debf7)
2016-05-12 15:44:07 -07:00
Tsz-Wo Nicholas Sze 69768bf9e0 HDFS-10346. Implement asynchronous setPermission/setOwner for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-11 17:57:20 -07:00
Colin Patrick Mccabe 15bfcde238 HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe)
(cherry picked from commit 687233f20d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
2016-05-11 14:02:03 -07:00
Colin Patrick Mccabe 43f32d3f4a HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO level (Yiqun Lin via cmccabe)
(cherry picked from commit f0252ad252)
2016-05-10 11:20:46 -07:00
Akira Ajisaka 901588542b HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. Contributed by Gergely Novák. 2016-05-10 09:32:33 -07:00
Akira Ajisaka a5bfb2dd25 Revert "HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated."
This reverts commit c297df5995.
2016-05-10 09:32:20 -07:00
Akira Ajisaka c297df5995 HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. 2016-05-10 09:29:30 -07:00
Kihwal Lee 2c782fdb17 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah.
(cherry picked from commit b9e5a32fa1)
2016-05-09 15:51:42 -05:00
Masatake Iwasaki 4ecbe2e9bf HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun.
(cherry picked from commit 2835f146b5)
2016-05-06 15:47:51 +09:00
Xiaoyu Yao 35859872b7 HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang.
(cherry picked from commit dacd1f50fe)
2016-05-05 18:15:32 -07:00
Ming Ma 5b53833171 HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma)
(cherry picked from commit 1268cf5fbe)
2016-05-04 17:04:44 -07:00
Arpit Agarwal 313b4dbf28 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:52:56 -07:00
Xiaoyu Yao ceb9a49d03 HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao.
(cherry picked from commit 45a753ccf7)
2016-05-02 19:37:48 -07:00
Harsh J 89c00da611 HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh)
(cherry picked from commit 54e612bfb9)
2016-05-02 17:54:31 -07:00
Tsz-Wo Nicholas Sze 7e0824faa9 HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:20:10 -07:00
Kihwal Lee 74938b537d HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah.
(cherry picked from commit 7da540d03e)
2016-04-29 15:58:07 -05:00
Tsz-Wo Nicholas Sze fe3c2886be HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:24:32 -07:00
Kihwal Lee 0b9057e1d3 HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah.
(cherry picked from commit af9b000535)
2016-04-29 09:59:58 -05:00
Kihwal Lee f715f14185 HDFS-9958. BlockManager#createLocatedBlocks can throw NPE for corruptBlocks on failed storages. Contributed by Kuhu Shukla. 2016-04-28 16:39:48 -05:00
Lei Xu 4289cb8b36 HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 0a152103f1)
2016-04-27 15:06:35 -07:00
Kihwal Lee e181092b86 HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla.
(cherry picked from commit 919a1d824a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
2016-04-27 08:29:30 -05:00
Tsz-Wo Nicholas Sze d316f44059 HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-04-26 17:21:20 -07:00
Andrew Wang 088a78241b HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge.
(cherry picked from commit 6be22ddbf1)
2016-04-26 13:04:21 -07:00
Kihwal Lee c991beac64 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun.
(cherry picked from commit c6a2430b47)
2016-04-26 11:46:46 -05:00
Kihwal Lee f6bd9e07d2 HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor.
(cherry picked from commit 10f0f7851a)
2016-04-25 11:41:40 -05:00
Ravi Prakash 88e1427a03 HDFS-9525. hadoop utilities need to support provided delegation tokens. Contributed by HeeSoo Kim 2016-04-23 20:50:56 -07:00
Masatake Iwasaki 96111caca9 HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims)
(cherry picked from commit 6fcde2e38d)
2016-04-23 23:50:01 +09:00
Kihwal Lee 43cfe5943b HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang.
(cherry picked from commit 5d4255a801)
2016-04-22 11:10:06 -05:00
Jing Zhao 5353289c8c HDFS-9894. Add unsetStoragePolicy API to FileContext/AbstractFileSystem and derivatives. Contributed by Xiaobing Zhou.
(cherry picked from commit 7149cdb3c2d9dd390cd8668883cbe5db94090e0a)
2016-04-21 20:16:02 -07:00
Tsz-Wo Nicholas Sze a96ea528cc HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:44:01 -07:00
Tsz-Wo Nicholas Sze 5245254a46 HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:21:45 -07:00
Xiaoyu Yao 432531dcf7 HDFS-10207. Support enable Hadoop IPC backoff without namenode restart. Contributed by Xiaobing Zhou.
(cherry picked from commit b4be288c5d)
2016-04-21 10:19:35 -07:00
Xiaoyu Yao a2a65922dd HADOOP-12985. Support MetricsSource interface for DecayRpcScheduler Metrics. Contributed by Xiaoyu Yao.
(cherry picked from commit 5bd7b592e5)
2016-04-20 23:11:42 -07:00
Chris Nauroth cc56bed686 HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth.
(cherry picked from commit 63ac2db59a)
2016-04-20 13:55:03 -07:00
Colin Patrick Mccabe 76d963acb4 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe)
(cherry picked from commit ad36fa6f42)
2016-04-20 11:52:26 -07:00
Arpit Agarwal b776db36f7 HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:27:36 -07:00
Walter Su 09a508beee HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun.
(cherry picked from commit 90d1af1416)
2016-04-19 18:56:08 +08:00
Walter Su d31a0d7443 HDFS-10284. o.a.h.hdfs.server.blockmanagement.TestBlockManagerSafeMode.testCheckSafeMode fails intermittently. Contributed by Mingliang Liu.
(cherry picked from commit 4b4df2cd48)
2016-04-19 18:08:52 +08:00
Jing Zhao bf4403e034 HDFS-10306. SafeModeMonitor should not leave safe mode if name system is starting active service. Contributed by Mingliang Liu.
(cherry picked from commit be0bce1b71)
2016-04-18 18:06:10 -07:00
Colin Patrick Mccabe a69b6b1e8b HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe)
(cherry picked from commit cb3ca460ef)
2016-04-18 13:48:24 -07:00
Kihwal Lee aa846bd492 HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun.
(cherry picked from commit d8b729e16f)
2016-04-18 08:00:44 -05:00
Walter Su a06ee5a71d HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun.
(cherry picked from commit ab903029a9)
2016-04-18 20:31:55 +08:00
Steve Loughran dd64769699 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 2016-04-18 10:26:58 +01:00
Walter Su a6d0e9e56a HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi.
(cherry picked from commit 67523ffcf4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
2016-04-18 09:34:01 +08:00
Jing Zhao eba7ecc993 HDFS-10283. o.a.h.hdfs.server.namenode.TestFSImageWithSnapshot#testSaveLoadImageWithAppending fails intermittently. Contributed by Mingliang Liu.
(cherry picked from commit 89a838769f)
2016-04-15 10:50:58 -07:00
Kihwal Lee 9b3f1139ba HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu.
(cherry picked from commit b9c9d03591)
2016-04-14 15:27:40 -05:00
Kihwal Lee 0bc8c327e9 HDFS-10292. Add block id when client got Unable to close file exception. Contributed by Brahma Reddy Battula.
(cherry picked from commit 2c155afe27)
2016-04-14 14:26:23 -05:00
Xiaoyu Yao 79f7c03fe1 HDFS-10286. Fix TestDFSAdmin#testNameNodeGetReconfigurableProperties. Contributed by Xiaobing Zhou.
(cherry picked from commit 809226752d)
2016-04-14 10:58:57 -07:00
Kihwal Lee 5e82431374 HDFS-10280. Document new dfsadmin command -evictWriters. Contributed by Wei-Chiu Chuang.
(cherry picked from commit c970f1d005)
2016-04-14 12:47:25 -05:00
Kihwal Lee 51fc96f5cf HDFS-10282. The VolumeScanner should warn about replica files which are misplaced. Contributed by Colin Patrick McCabe.
(cherry picked from commit 0d1c1152f1)
2016-04-14 07:59:41 -05:00
Xiaoyu Yao 1f75e62b6d HDFS-10209. Support enable caller context in HDFS namenode audit log without restart namenode. Contributed by Xiaobing Zhou.
(cherry picked from commit 5566177c9a)
2016-04-13 16:52:22 -07:00
Andrew Wang 4c18f54c1d HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun.
(cherry picked from commit 314aa21a89)
2016-04-13 16:39:59 -07:00
Kihwal Lee 4a7e52d302 HDFS-10270. TestJMXGet:testNameNode() fails. Contributed by Gergely Novák
(cherry picked from commit d2f3bbc290)
2016-04-13 11:24:06 -05:00
Walter Su c195b1c9fe HDFS-9772. TestBlockReplacement#testThrottler doesn't work as expected. Contributed by Lin Yiqun.
(cherry picked from commit 903428bf94)
2016-04-13 17:50:30 +08:00
Colin Patrick Mccabe 2b0b332e2f HADOOP-12973. Make DU pluggable. (Elliott Clark via cmccabe)
(cherry picked from commit 35f0770555)
2016-04-12 16:29:15 -07:00
Colin Patrick Mccabe 6e37e40dd2 HDFS-10273. Remove duplicate logSync() and log message in FSN#enterSafemode() (Vinayakumar B via cmccabe)
(cherry picked from commit 600d129bb8)
2016-04-12 10:55:43 -07:00
Akira Ajisaka bb1ed426d1 HDFS-8356. Document missing properties in hdfs-default.xml. Contributed by Ray Chiang. 2016-04-12 15:28:05 +09:00
Vinayakumar B 04621537c9 HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula)
(cherry picked from commit a9a607f8fc)
2016-04-12 11:45:44 +05:30
Akira Ajisaka 8f968171bd HDFS-10277. PositionedReadable test testReadFullyZeroByteFile failing in HDFS. Contributed by Steve Loughran.
(cherry picked from commit a409508b3f)
2016-04-12 14:55:25 +09:00
Chris Nauroth 26a23eff82 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran.
(cherry picked from commit 843ee8d59d)
2016-04-08 13:37:46 -07:00
Tsz-Wo Nicholas Sze bccc28914b HDFS-10186. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. Contributed by Rakesh R 2016-04-07 12:39:26 +08:00
Colin Patrick Mccabe b5e6ad457f HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose
(cherry picked from commit 4bd7cbc29d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java
2016-04-06 21:08:42 -07:00
Eric Payne caef78afe6 HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal Lee
(cherry picked from commit aede8c10ec)
2016-04-06 20:31:20 +00:00
Colin Patrick Mccabe 387d3f25c1 HDFS-6520. hdfs fsck passes invalid length value when creating BlockReader (Xiao Chen via cmccabe)
(cherry picked from commit 188f65287d)
2016-04-06 11:37:57 -07:00
Jing Zhao 7cac807eb3 HDFS-10192. Namenode safemode not coming out during failover. Contributed by Brahma Reddy Battula.
(cherry picked from commit 221b3a8722)
2016-04-06 10:43:38 -07:00
Akira Ajisaka 1371f993a9 HADOOP-12022. fix site -Pdocs -Pdist in hadoop-project-dist; cleanout remaining forrest bits. Contributed by Allen Wittenauer. 2016-04-06 17:43:31 +09:00
Kihwal Lee 6e37c5fe80 HDFS-10261. TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports. Contributed by Eric Badger.
(cherry picked from commit 9ba1e5af06)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java
2016-04-05 16:49:59 -05:00
Ravi Prakash 2ca6251a6a HDFS-10235. Last contact for Live Nodes should be relative time. Contributed by Brahma Reddy Battula.
(cherry picked from commit 0cd320a846)
2016-04-05 13:43:42 -07:00
Masatake Iwasaki 2542e9bccf HADOOP-12672. RPC timeout should not override IPC ping interval (iwasakims)
(cherry picked from commit 85ec5573eb)
2016-04-06 03:26:33 +09:00
Kihwal Lee ef3da82357 HDFS-10239. Fsshell mv fails if port usage doesn't match in src and destination paths. Contributed by Kuhu Shukla.
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java
2016-04-05 09:16:05 -05:00
Vinayakumar B ad08114b92 HDFS-9917. IBR accumulate more objects when SNN was down for sometime. (Contributed by Brahma Reddy Battula)
(cherry picked from commit 818d6b799e)
2016-04-05 09:53:57 +08:00