Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,592 Commits 326 Branches 371 Tags
Commit Graph

4085 Commits

Author SHA1 Message Date
Lei Xu
5477c31c5c HDFS-3702. Add an option for NOT writing the blocks locally if there is a datanode on the same box as the client. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 0a152103f19a3e8e1b7f33aeb9dd115ba231d7b7)
(cherry picked from commit 4289cb8b36bcb96510b9e63e3e966e306c6e3893)
 2016-05-02 10:21:59 -07:00
Kihwal Lee
e9cd88d484 HDFS-10347. Namenode report bad block method doesn't log the bad block or datanode. Contributed by Rushabh Shah. 
(cherry picked from commit 7da540d03eccc2f97950bf47e8b35ce8c889d1e0)
 2016-04-29 15:58:45 -05:00
Tsz-Wo Nicholas Sze
31ed34c982 HDFS-10335 Mover$Processor#chooseTarget() always chooses the first matching target storage group. Contributed by Mingliang Liu 2016-04-29 11:27:14 -07:00
Kihwal Lee
a4db3729be HDFS-10260. TestFsDatasetImpl#testCleanShutdownOfVolume often fails. Contributed by Rushabh Shah. 
(cherry picked from commit af9b000535cc987f66327e2b2bfe08923ba24c13)
 2016-04-29 10:05:58 -05:00
Andrew Wang
bdb0e98483 HDFS-10297. Increase default balance bandwidth and concurrent moves. Contributed by John Zhuge. 2016-04-27 17:03:36 -07:00
Kihwal Lee
dd7c9f5b8f HDFS-10330. Add Corrupt Blocks Information in Metasave output. Contributed by Kuhu Shukla. 
(cherry picked from commit e181092b86a8822e0f252b6ffa77baad4f711e77)
 2016-04-27 08:30:29 -05:00
Kihwal Lee
621be60592 HDFS-10329. Bad initialisation of StringBuffer in RequestHedgingProxyProvider. Contributed by Lin Yiqun. 
(cherry picked from commit c6a2430b47692a72c500373d6fc1bce78652675b)
 2016-04-26 11:47:15 -05:00
Kihwal Lee
973c73aa68 HDFS-10318. TestJMXGet hides the real error in case of test failure. Contributed by Andras Bokor. 
(cherry picked from commit 10f0f7851a3255caab775777e8fb6c2781d97062)
 2016-04-25 11:42:17 -05:00
Masatake Iwasaki
c4d8090b72 HDFS-9905. WebHdfsFileSystem#runWithRetry should display original stack trace on error. (Wei-Chiu Chuang via iwasakims) 
(cherry picked from commit 6fcde2e38da04cae3aad6b13cf442af211f71506)
 2016-04-23 23:55:27 +09:00
Kihwal Lee
3d3ed9b68e HDFS-9555. LazyPersistFileScrubber should still sleep if there are errors in the clear progress. Contributed by Phil Yang. 
(cherry picked from commit 5d4255a80156d2cacfea8184b41805070223d3a9)
 2016-04-22 11:10:36 -05:00
Tsz-Wo Nicholas Sze
60c68c65c1 HDFS-10309 Balancer doesn't honor dfs.blocksize value defined with suffix k(kilo), m(mega), g(giga). Contributed by Amit Anand 2016-04-21 15:45:36 -07:00
Tsz-Wo Nicholas Sze
704c57e3eb HDFS-10319. Balancer should not try to pair storages with different types. 2016-04-21 15:25:49 -07:00
Chris Nauroth
7fb1cfc591 HDFS-10312. Large block reports may fail to decode at NameNode due to 64 MB protobuf maximum length restriction. Contributed by Chris Nauroth. 
(cherry picked from commit 63ac2db59af2b50e74dc892cae1dbc4d2e061423)
(cherry picked from commit cc56bed6867f844810f5fda0cba3cbc9a9367269)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImplTestUtils.java
 2016-04-20 14:00:06 -07:00
Colin Patrick Mccabe
e2968f5da2 HDFS-10308. TestRetryCacheWithHA#testRetryCacheOnStandbyNN failing (Rakesh R via cmccabe) 
(cherry picked from commit ad36fa6f42870c517526618a30204b443bfc6b5a)
(cherry picked from commit 76d963acb422f7b7e8e83768ab4be538ea5ad94e)
 2016-04-20 11:52:49 -07:00
Arpit Agarwal
a13628fe4a HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:27:46 -07:00
Walter Su
1860b8ceb5 HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun. 
(cherry picked from commit 90d1af141666ba12f18d44464cb6169c445d0902)
 2016-04-19 18:56:23 +08:00
Colin Patrick Mccabe
e1d9e5ab82 HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 
(cherry picked from commit cb3ca460efb97be8c031bdb14bb7705cc25f2117)
(cherry picked from commit a69b6b1e8b1e734ba9a106061cc6d22583d1d2d1)
 2016-04-18 13:49:05 -07:00
Kihwal Lee
7ac2e5ec7b HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun. 
(cherry picked from commit d8b729e16fb253e6c84f414d419b5663d9219a43)
 2016-04-18 08:01:17 -05:00
Walter Su
09ffc65d94 HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun. 
(cherry picked from commit ab903029a9d353677184ff5602966b11ffb408b9)
 2016-04-18 20:32:36 +08:00
Steve Loughran
e9699aa443 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 2016-04-18 10:26:35 +01:00
Walter Su
7f47d984fa HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 
(cherry picked from commit 67523ffcf491f4f2db5335899c00a174d0caaa9b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java

(cherry picked from commit a6d0e9e56a7eef226c8a891cf741ab22d8205e1c)
 2016-04-18 09:39:46 +08:00
Kihwal Lee
8b1e7842e3 HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu. 
(cherry picked from commit b9c9d03591a49be31f3fbc738d01a31700bfdbc4)
 2016-04-14 15:30:59 -05:00
Kihwal Lee
7dd8798933 HDFS-10292. Add block id when client got Unable to close file exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 2c155afe2736a5571bbb3bdfb2fe6f9709227229)
 2016-04-14 14:26:59 -05:00
Kihwal Lee
019ca8f7c1 HDFS-10280. Document new dfsadmin command -evictWriters. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit c970f1d00525e4273075cff7406dcbd71305abd5)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
 2016-04-14 12:50:36 -05:00
Andrew Wang
1dee3e1200 HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun. 
(cherry picked from commit 314aa21a89134fac68ac3cb95efdeb56bd3d7b05)
(cherry picked from commit 4c18f54c1d98e7ad61966af3384904e33a38daae)
 2016-04-13 16:40:08 -07:00
Kihwal Lee
ccaf746eea HDFS-10270. TestJMXGet:testNameNode() fails. Contributed by Gergely Novák 
(cherry picked from commit d2f3bbc29046435904ad9418073795439c71b441)
 2016-04-13 11:24:45 -05:00
Walter Su
f3dbabb13a HDFS-9772. TestBlockReplacement#testThrottler doesn't work as expected. Contributed by Lin Yiqun. 
(cherry picked from commit 903428bf946827b4d58c7c577ed0c574a7cff029)
 2016-04-13 17:50:47 +08:00
Colin Patrick Mccabe
59b0661c7f HADOOP-12973. Make DU pluggable. (Elliott Clark via cmccabe) 
(cherry picked from commit 35f07705552ef5636f8b8d2599a6af5ec1426203)
(cherry picked from commit 2b0b332e2faedcee34db7b0c640aa1aba54b2f14)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2016-04-12 16:30:58 -07:00
Vinayakumar B
b804b20843 HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula) 
(cherry picked from commit a9a607f8fc0d996af3fb37f7efa7591d6655900d)
(cherry picked from commit 04621537c94c8e406337e82800d2119ea25d8146)
 2016-04-12 11:47:01 +05:30
Akira Ajisaka
e2b8bdbe0a HDFS-10277. PositionedReadable test testReadFullyZeroByteFile failing in HDFS. Contributed by Steve Loughran. 
(cherry picked from commit a409508b3f4c46b419c41b9cdff83429d9d025ce)
(cherry picked from commit 8f968171bd5d672c8078275a243e3d590feef6e4)
 2016-04-12 14:55:42 +09:00
Chris Nauroth
1b5b85e560 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d8bacbca0d87ccf0790772e39d16138)
(cherry picked from commit 26a23eff828a3f4fcbd4362761c329657603277d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 2016-04-08 13:44:27 -07:00
Tsz-Wo Nicholas Sze
f872d519db HDFS-10186. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. Contributed by Rakesh R 2016-04-07 12:42:27 +08:00
Colin Patrick Mccabe
354801fe61 HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose 
(cherry picked from commit 4bd7cbc29d142fc56324156333b9a8a7d7b68042)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

(cherry picked from commit b5e6ad457f39dd5f2176de700612e2757b3c1812)
 2016-04-06 21:11:48 -07:00
Eric Payne
497c65ad0c HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal Lee 
(cherry picked from commit aede8c10ecad4f2a8802a834e4bd0b8286cebade)
 2016-04-06 20:33:40 +00:00
Colin Patrick Mccabe
8514ab8cb5 HDFS-6520. hdfs fsck passes invalid length value when creating BlockReader (Xiao Chen via cmccabe) 
(cherry picked from commit 188f65287d5b2f26a8862c88198f83ac59035016)
(cherry picked from commit 387d3f25c192dc89484c3372bbc401a57175ff8e)
 2016-04-06 11:38:19 -07:00
Akira Ajisaka
e426d1bb3f HADOOP-12022. fix site -Pdocs -Pdist in hadoop-project-dist; cleanout remaining forrest bits. Contributed by Allen Wittenauer. 
(cherry picked from commit 1371f993a9fa59d7e91321f022eab9974319d641)
 2016-04-06 17:44:54 +09:00
Kihwal Lee
8bb4655649 HDFS-10261. TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports. Contributed by Eric Badger. 
(cherry picked from commit 9ba1e5af06070ba01dcf46e1a4c66713a1d43352)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java

(cherry picked from commit 6e37c5fe80ee9ce46afc26838391cc27ca6b1e6f)
 2016-04-05 16:54:46 -05:00
Ravi Prakash
7286c435c4 HDFS-10235. Last contact for Live Nodes should be relative time. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0cd320a8463efe19a6228f9fe14693aa37ac8a10)
 2016-04-05 13:43:08 -07:00
Masatake Iwasaki
886273e123 HADOOP-12672. RPC timeout should not override IPC ping interval (iwasakims) 
(cherry picked from commit 85ec5573eb9fd746a9295ecc6fe1ae683073aaf5)
 2016-04-06 03:29:49 +09:00
Kihwal Lee
f9764d073f HDFS-10239. Fsshell mv fails if port usage doesn't match in src and destination paths. Contributed by Kuhu Shukla. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

(cherry picked from commit ef3da823573cbf16fd1d84479330dd457f95e0ff)
 2016-04-05 09:16:51 -05:00
Vinayakumar B
3deeb9a15f HDFS-9917. IBR accumulate more objects when SNN was down for sometime. (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 818d6b799eead13a17a0214172df60a269b046fb)
(cherry picked from commit ad08114b9285fecd2c41a4ac3e073662b1e212d3)
 2016-04-05 09:54:50 +08:00
Colin Patrick Mccabe
63fe2ae72c HDFS-8496. Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe) 
(cherry picked from commit f6b1a818124cc42688c4c5acaf537d96cf00e43b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

(cherry picked from commit d55af8b5a356e88a12f0b343b3ed3a974140f81d)
 2016-04-04 18:07:21 -07:00
Kihwal Lee
7d671cad3f HDFS-10178. Permanent write failures can happen if pipeline recoveries occur for the first packet. Contributed by Kihwal Lee. 
(cherry picked from commit a7d1fb0cd2fdbf830602eb4dbbd9bbe62f4d5584)
 2016-04-04 16:41:23 -05:00
Masatake Iwasaki
97cda4077a HDFS-9599. TestDecommissioningStatus.testDecommissionStatus occasionally fails (Lin Yiqun via iwasakims) 
(cherry picked from commit 154d2532cf015e9ab9141864bd3ab0d6100ef597)
 2016-04-05 03:44:12 +09:00
Vinayakumar B
bafcb584a8 HDFS-10253. Fix TestRefreshCallQueue failure (Contributed by Xiaoyu Yao) 
(cherry picked from commit 54b2e78fd28c9def42bec7f0418833bad352686c)
(cherry picked from commit aa38748997ae93d0ba91ecfbad77f7dfea25c7da)
 2016-04-03 13:28:52 +05:30
Akira Ajisaka
3892efd3fb HDFS-10221. Add .json to the rat exclusions. Contributed by Ming Ma. 
(cherry picked from commit 32c0c3ecdf72e89a63f4aee5e75d1c5a12714b89)
(cherry picked from commit 0795ee687b8f0cde251ba80faff4900be8d72c10)
 2016-03-31 09:06:48 +09:00
Colin Patrick Mccabe
5cf55c5fb6 HDFS-10223. peerFromSocketAndKey performs SASL exchange before setting connection timeouts (cmccabe) 
(cherry picked from commit 37e23ce45c592f3c9c48a08a52a5f46787f6c0e9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java

(cherry picked from commit be5894c12dd3af20474d74c444ad8ffd798ff417)
 2016-03-30 13:45:05 -07:00
Vinayakumar B
def143e1e1 HDFS-5177. blocksScheduled count should be decremented for abandoned blocks (Contributed by Vinayakumar B) 
(cherry picked from commit 09d63d5a192b5d6b172f94ff6c94da348fd49ea6)
(cherry picked from commit 0852e441108fd279d87a167a442eba046832779e)
 2016-03-30 14:23:56 +08:00
Arpit Agarwal
bfe657e83a HDFS-10228. TestHDFSCLI fails. (Contributed by Akira AJISAKA) 2016-03-29 13:56:12 -07:00
Andrew Wang
026b84bf70 HDFS-10197. TestFsDatasetCache failing intermittently due to timeout. Contributed by Lin Yiqun. 
(cherry picked from commit f2aec4eb824647a01e14b4eede03af0babe65fb6)
(cherry picked from commit 1d79b57c14410417b1ecccabfe2c695a1c6cd6a2)
 2016-03-29 11:55:04 -07:00