Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,441 Commits 326 Branches 371 Tags
Commit Graph

3972 Commits

Author SHA1 Message Date
Xiaoyu Yao
ccb1cade5b HDFS-10383. Safely close resources in DFSTestUtil. Contributed by Mingliang Liu. 2016-05-17 18:00:18 -07:00
Kihwal Lee
01de843a31 Revert "HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu." 
This reverts commit cb5a1edd1cfebec389110a192aa0e5359abeff0e.
 2016-05-17 15:21:41 -05:00
Kihwal Lee
cb5a1edd1c HDFS-10417. Improve error message from checkBlockLocalPathAccess. Contributed by Tianyin Xu. 
(cherry picked from commit 0942954e8ab6bef636c2648d0d5a0227a50f799c)
 2016-05-17 15:11:40 -05:00
Kihwal Lee
362a778bf5 HDFS-10303. DataStreamer#ResponseProcessor calculates packet ack latency incorrectly. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4a5819dae2b0ca8f8b6d94ef464882d079d86593)
 2016-05-17 09:00:59 -05:00
Akira Ajisaka
388b2010ae HDFS-10242. Cannot create space quota of zero. Contributed by Takashi Ohnishi. 
(cherry picked from commit 9fe5828f05accc6746cb08a43916f7557dac533a)
(cherry picked from commit 7f03c1c90e3816b3957dbf121c6d7bc3ce07cabd)
 2016-05-17 15:56:36 +09:00
Andrew Wang
29ed65ff9b HDFS-10333. Intermittent org.apache.hadoop.hdfs.TestFileAppend failure in trunk. Contributed by Yiqun Lin. 
(cherry picked from commit 45788204ae2ac82ccb3b4fe2fd22aead1dd79f0d)
(cherry picked from commit 413f3dabbd398b7d837893963556d9e6dd9e10e2)
 2016-05-15 22:15:18 -07:00
Colin Patrick Mccabe
5f04795aed HADOOP-13065. Add a new interface for retrieving FS and FC Statistics (Mingliang Liu via cmccabe) 
(cherry picked from commit 687233f20d24c29041929dd0a99d963cec54b6df)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java

(cherry picked from commit 15bfcde23832189accce8293a053c8f0ef1aeaae)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2016-05-11 14:24:51 -07:00
Colin Patrick Mccabe
154332bb54 HDFS-10377. CacheReplicationMonitor shutdown log message should use INFO level (Yiqun Lin via cmccabe) 
(cherry picked from commit f0252ad2525c90b2b89ae2ff1de12482da2c8ddd)
(cherry picked from commit 43f32d3f4a86d8f2a302530831003496c78fd561)
 2016-05-10 11:21:16 -07:00
Akira Ajisaka
a84c4d7d4b HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. Contributed by Gergely Novák. 
(cherry picked from commit 901588542b573cbefa91f7e0e6032f7163160a85)
 2016-05-10 09:33:55 -07:00
Akira Ajisaka
1622fc1373 Revert "HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated." 
This reverts commit f25dcb935e9dbc0ad2dfe4466302d8b65868e172.
 2016-05-10 09:33:18 -07:00
Akira Ajisaka
f25dcb935e HDFS-6187. Update the document of hftp/hsftp in branch-2 to mention that they are deprecated. 
(cherry picked from commit c297df59951dcde44474fed341fa495bb6308936)
 2016-05-10 09:29:47 -07:00
Kihwal Lee
d7cda6d031 HDFS-10372. Fix for failing TestFsDatasetImpl#testCleanShutdownOfVolume. Contributed by Rushabh Shah. 
(cherry picked from commit b9e5a32fa14b727b44118ec7f43fb95de05a7c2c)
 2016-05-09 15:52:12 -05:00
Masatake Iwasaki
c80b47e73f HDFS-2043. TestHFlush failing intermittently. Contributed by Lin Yiqun. 
(cherry picked from commit 2835f146b551f911ea13307e7762759813898757)
 2016-05-06 15:50:48 +09:00
Xiaoyu Yao
59eeee7501 HDFS-10324. Trash directory in an encryption zone should be pre-created with correct permissions. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit dacd1f50feb24ccdf6155b2b7a6126fe21a47ad0)
(cherry picked from commit 35859872b7e339ac0b57f8001c349afe027ee4d2)
 2016-05-05 18:23:00 -07:00
Ming Ma
d6e95ae47b HDFS-10320. Rack failures may result in NN terminate. (Xiao Chen via mingma) 
(cherry picked from commit 1268cf5fbe4458fa75ad0662512d352f9e8d3470)
 2016-05-04 17:07:36 -07:00
Arpit Agarwal
ce94203566 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. (Contributed by Brahma Reddy Battula) 2016-05-03 16:53:07 -07:00
Xiaoyu Yao
5dfcfb2d2d HDFS-10344. DistributedFileSystem#getTrashRoots should skip encryption zone that does not have .Trash. Contributed by Xiaoyu Yao. 
(cherry picked from commit 45a753ccf79d334513c7bc8f2b81c89a4697075d)
 2016-05-02 19:50:10 -07:00
Xiaoyu Yao
643d2fd3c1 HADOOP-12985. Support MetricsSource interface for DecayRpcScheduler Metrics. Contributed by Xiaoyu Yao. 
(cherry picked from commit 5bd7b592e5fbe4d448fd127c15d29f3121b8a833)
(cherry picked from commit a2a65922dd9783b76d9abacc07ba8cb32e81caea)
 2016-05-02 19:38:03 -07:00
Harsh J
cf6efccebb HDFS-2580. NameNode#main(...) can make use of GenericOptionsParser. Contributed by harsh. (harsh) 
(cherry picked from commit 54e612bfb9f877e58f7f153c43cb4147876826d3)
 2016-05-02 17:53:10 -07:00
Tsz-Wo Nicholas Sze
a1ba6eee38 HADOOP-12957. Limit the number of outstanding async calls. Contributed by Xiaobing Zhou 2016-05-02 11:34:59 -07:00
Tsz-Wo Nicholas Sze
bdc45bef64 HDFS-10224. Implement asynchronous rename for DistributedFileSystem. Contributed by Xiaobing Zhou 2016-05-02 11:34:22 -07:00
Lei Xu
5477c31c5c HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 0a152103f19a3e8e1b7f33aeb9dd115ba231d7b7)
(cherry picked from commit 4289cb8b36bcb96510b9e63e3e966e306c6e3893)
 2016-05-02 10:21:59 -07:00
Kihwal Lee
e9cd88d484 HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah. 
(cherry picked from commit 7da540d03eccc2f97950bf47e8b35ce8c889d1e0)
 2016-04-29 15:58:45 -05:00
Tsz-Wo Nicholas Sze
31ed34c982 HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:27:14 -07:00
Kihwal Lee
a4db3729be HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 
(cherry picked from commit af9b000535cc987f66327e2b2bfe08923ba24c13)
 2016-04-29 10:05:58 -05:00
Andrew Wang
bdb0e98483 HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge. 2016-04-27 17:03:36 -07:00
Kihwal Lee
dd7c9f5b8f HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 
(cherry picked from commit e181092b86a8822e0f252b6ffa77baad4f711e77)
 2016-04-27 08:30:29 -05:00
Kihwal Lee
621be60592 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun. 
(cherry picked from commit c6a2430b47692a72c500373d6fc1bce78652675b)
 2016-04-26 11:47:15 -05:00
Kihwal Lee
973c73aa68 HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 
(cherry picked from commit 10f0f7851a3255caab775777e8fb6c2781d97062)
 2016-04-25 11:42:17 -05:00
Masatake Iwasaki
c4d8090b72 HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 
(cherry picked from commit 6fcde2e38da04cae3aad6b13cf442af211f71506)
 2016-04-23 23:55:27 +09:00
Kihwal Lee
3d3ed9b68e HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang. 
(cherry picked from commit 5d4255a80156d2cacfea8184b41805070223d3a9)
 2016-04-22 11:10:36 -05:00
Tsz-Wo Nicholas Sze
60c68c65c1 HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:45:36 -07:00
Tsz-Wo Nicholas Sze
704c57e3eb HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:25:49 -07:00
Chris Nauroth
7fb1cfc591 HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 
(cherry picked from commit 63ac2db59af2b50e74dc892cae1dbc4d2e061423)
(cherry picked from commit cc56bed6867f844810f5fda0cba3cbc9a9367269)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
 2016-04-20 14:00:06 -07:00
Colin Patrick Mccabe
e2968f5da2 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe) 
(cherry picked from commit ad36fa6f42870c517526618a30204b443bfc6b5a)
(cherry picked from commit 76d963acb422f7b7e8e83768ab4be538ea5ad94e)
 2016-04-20 11:52:49 -07:00
Arpit Agarwal
a13628fe4a HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:27:46 -07:00
Walter Su
1860b8ceb5 HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun. 
(cherry picked from commit 90d1af141666ba12f18d44464cb6169c445d0902)
 2016-04-19 18:56:23 +08:00
Colin Patrick Mccabe
e1d9e5ab82 HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 
(cherry picked from commit cb3ca460efb97be8c031bdb14bb7705cc25f2117)
(cherry picked from commit a69b6b1e8b1e734ba9a106061cc6d22583d1d2d1)
 2016-04-18 13:49:05 -07:00
Kihwal Lee
7ac2e5ec7b HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun. 
(cherry picked from commit d8b729e16fb253e6c84f414d419b5663d9219a43)
 2016-04-18 08:01:17 -05:00
Walter Su
09ffc65d94 HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun. 
(cherry picked from commit ab903029a9d353677184ff5602966b11ffb408b9)
 2016-04-18 20:32:36 +08:00
Steve Loughran
e9699aa443 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 2016-04-18 10:26:35 +01:00
Walter Su
7f47d984fa HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 
(cherry picked from commit 67523ffcf491f4f2db5335899c00a174d0caaa9b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java

(cherry picked from commit a6d0e9e56a7eef226c8a891cf741ab22d8205e1c)
 2016-04-18 09:39:46 +08:00
Kihwal Lee
8b1e7842e3 HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu. 
(cherry picked from commit b9c9d03591a49be31f3fbc738d01a31700bfdbc4)
 2016-04-14 15:30:59 -05:00
Kihwal Lee
019ca8f7c1 HDFS-10280. Document new dfsadmin command -evictWriters. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit c970f1d00525e4273075cff7406dcbd71305abd5)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
 2016-04-14 12:50:36 -05:00
Andrew Wang
1dee3e1200 HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun. 
(cherry picked from commit 314aa21a89134fac68ac3cb95efdeb56bd3d7b05)
(cherry picked from commit 4c18f54c1d98e7ad61966af3384904e33a38daae)
 2016-04-13 16:40:08 -07:00
Kihwal Lee
ccaf746eea HDFS-10270. TestJMXGet:testNameNode() fails. Contributed by Gergely Novák 
(cherry picked from commit d2f3bbc29046435904ad9418073795439c71b441)
 2016-04-13 11:24:45 -05:00
Walter Su
f3dbabb13a HDFS-9772. TestBlockReplacement#testThrottler doesn't work as expected. Contributed by Lin Yiqun. 
(cherry picked from commit 903428bf946827b4d58c7c577ed0c574a7cff029)
 2016-04-13 17:50:47 +08:00
Colin Patrick Mccabe
59b0661c7f HADOOP-12973. Make DU pluggable. (Elliott Clark via cmccabe) 
(cherry picked from commit 35f07705552ef5636f8b8d2599a6af5ec1426203)
(cherry picked from commit 2b0b332e2faedcee34db7b0c640aa1aba54b2f14)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2016-04-12 16:30:58 -07:00
Vinayakumar B
b804b20843 HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula) 
(cherry picked from commit a9a607f8fc0d996af3fb37f7efa7591d6655900d)
(cherry picked from commit 04621537c94c8e406337e82800d2119ea25d8146)
 2016-04-12 11:47:01 +05:30
Tsz-Wo Nicholas Sze
f872d519db HDFS-10186. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. Contributed by Rakesh R 2016-04-07 12:42:27 +08:00