Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,557 Commits 326 Branches 371 Tags
Commit Graph

4071 Commits

Author SHA1 Message Date
Arpit Agarwal
a13628fe4a HDFS-10264. Logging improvements in FSImageFormatProtobuf.Saver. (Contributed by Xiaobing Zhou) 2016-04-19 11:27:46 -07:00
Walter Su
1860b8ceb5 HDFS-9744. TestDirectoryScanner#testThrottling occasionally time out after 300 seconds. Contributed by Lin Yiqun. 
(cherry picked from commit 90d1af141666ba12f18d44464cb6169c445d0902)
 2016-04-19 18:56:23 +08:00
Colin Patrick Mccabe
e1d9e5ab82 HDFS-10265. OEV tool fails to read edit xml file if OP_UPDATE_BLOCKS has no BLOCK tag (Wan Chang via cmccabe) 
(cherry picked from commit cb3ca460efb97be8c031bdb14bb7705cc25f2117)
(cherry picked from commit a69b6b1e8b1e734ba9a106061cc6d22583d1d2d1)
 2016-04-18 13:49:05 -07:00
Kihwal Lee
7ac2e5ec7b HDFS-10302. BlockPlacementPolicyDefault should use default replication considerload value. Contributed by Lin Yiqun. 
(cherry picked from commit d8b729e16fb253e6c84f414d419b5663d9219a43)
 2016-04-18 08:01:17 -05:00
Walter Su
09ffc65d94 HDFS-10275. TestDataNodeMetrics failing intermittently due to TotalWriteTime counted incorrectly. Contributed by Lin Yiqun. 
(cherry picked from commit ab903029a9d353677184ff5602966b11ffb408b9)
 2016-04-18 20:32:36 +08:00
Steve Loughran
e9699aa443 HDFS-10291 TestShortCircuitLocalRead failing (stevel) 2016-04-18 10:26:35 +01:00
Walter Su
7f47d984fa HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. 
(cherry picked from commit 67523ffcf491f4f2db5335899c00a174d0caaa9b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java

(cherry picked from commit a6d0e9e56a7eef226c8a891cf741ab22d8205e1c)
 2016-04-18 09:39:46 +08:00
Kihwal Lee
8b1e7842e3 HDFS-10281. TestPendingCorruptDnMessages fails intermittently. Contributed by Mingliang Liu. 
(cherry picked from commit b9c9d03591a49be31f3fbc738d01a31700bfdbc4)
 2016-04-14 15:30:59 -05:00
Kihwal Lee
7dd8798933 HDFS-10292. Add block id when client got Unable to close file exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 2c155afe2736a5571bbb3bdfb2fe6f9709227229)
 2016-04-14 14:26:59 -05:00
Kihwal Lee
019ca8f7c1 HDFS-10280. Document new dfsadmin command -evictWriters. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit c970f1d00525e4273075cff7406dcbd71305abd5)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
 2016-04-14 12:50:36 -05:00
Andrew Wang
1dee3e1200 HDFS-10279. Improve validation of the configured number of tolerated failed volumes. Contributed by Lin Yiqun. 
(cherry picked from commit 314aa21a89134fac68ac3cb95efdeb56bd3d7b05)
(cherry picked from commit 4c18f54c1d98e7ad61966af3384904e33a38daae)
 2016-04-13 16:40:08 -07:00
Kihwal Lee
ccaf746eea HDFS-10270. TestJMXGet:testNameNode() fails. Contributed by Gergely Novák 
(cherry picked from commit d2f3bbc29046435904ad9418073795439c71b441)
 2016-04-13 11:24:45 -05:00
Walter Su
f3dbabb13a HDFS-9772. TestBlockReplacement#testThrottler doesn't work as expected. Contributed by Lin Yiqun. 
(cherry picked from commit 903428bf946827b4d58c7c577ed0c574a7cff029)
 2016-04-13 17:50:47 +08:00
Colin Patrick Mccabe
59b0661c7f HADOOP-12973. Make DU pluggable. (Elliott Clark via cmccabe) 
(cherry picked from commit 35f07705552ef5636f8b8d2599a6af5ec1426203)
(cherry picked from commit 2b0b332e2faedcee34db7b0c640aa1aba54b2f14)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2016-04-12 16:30:58 -07:00
Vinayakumar B
b804b20843 HDFS-10271. Extra bytes are getting released from reservedSpace for append (Contributed by Brahma Reddy Battula) 
(cherry picked from commit a9a607f8fc0d996af3fb37f7efa7591d6655900d)
(cherry picked from commit 04621537c94c8e406337e82800d2119ea25d8146)
 2016-04-12 11:47:01 +05:30
Akira Ajisaka
e2b8bdbe0a HDFS-10277. PositionedReadable test testReadFullyZeroByteFile failing in HDFS. Contributed by Steve Loughran. 
(cherry picked from commit a409508b3f4c46b419c41b9cdff83429d9d025ce)
(cherry picked from commit 8f968171bd5d672c8078275a243e3d590feef6e4)
 2016-04-12 14:55:42 +09:00
Chris Nauroth
1b5b85e560 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d8bacbca0d87ccf0790772e39d16138)
(cherry picked from commit 26a23eff828a3f4fcbd4362761c329657603277d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 2016-04-08 13:44:27 -07:00
Tsz-Wo Nicholas Sze
f872d519db HDFS-10186. DirectoryScanner: Improve logs by adding full path of both actual and expected block directories. Contributed by Rakesh R 2016-04-07 12:42:27 +08:00
Colin Patrick Mccabe
354801fe61 HDFS-10267. Extra "synchronized" on FsDatasetImpl#recoverAppend and FsDatasetImpl#recoverClose 
(cherry picked from commit 4bd7cbc29d142fc56324156333b9a8a7d7b68042)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

(cherry picked from commit b5e6ad457f39dd5f2176de700612e2757b3c1812)
 2016-04-06 21:11:48 -07:00
Eric Payne
497c65ad0c HDFS-9945. Datanode command for evicting writers. Contributed by Kihwal Lee 
(cherry picked from commit aede8c10ecad4f2a8802a834e4bd0b8286cebade)
 2016-04-06 20:33:40 +00:00
Colin Patrick Mccabe
8514ab8cb5 HDFS-6520. hdfs fsck passes invalid length value when creating BlockReader (Xiao Chen via cmccabe) 
(cherry picked from commit 188f65287d5b2f26a8862c88198f83ac59035016)
(cherry picked from commit 387d3f25c192dc89484c3372bbc401a57175ff8e)
 2016-04-06 11:38:19 -07:00
Akira Ajisaka
e426d1bb3f HADOOP-12022. fix site -Pdocs -Pdist in hadoop-project-dist; cleanout remaining forrest bits. Contributed by Allen Wittenauer. 
(cherry picked from commit 1371f993a9fa59d7e91321f022eab9974319d641)
 2016-04-06 17:44:54 +09:00
Kihwal Lee
8bb4655649 HDFS-10261. TestBookKeeperHACheckpoints doesn't handle ephemeral HTTP ports. Contributed by Eric Badger. 
(cherry picked from commit 9ba1e5af06070ba01dcf46e1a4c66713a1d43352)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/test/java/org/apache/hadoop/contrib/bkjournal/TestBookKeeperHACheckpoints.java

(cherry picked from commit 6e37c5fe80ee9ce46afc26838391cc27ca6b1e6f)
 2016-04-05 16:54:46 -05:00
Ravi Prakash
7286c435c4 HDFS-10235. Last contact for Live Nodes should be relative time. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0cd320a8463efe19a6228f9fe14693aa37ac8a10)
 2016-04-05 13:43:08 -07:00
Masatake Iwasaki
886273e123 HADOOP-12672. RPC timeout should not override IPC ping interval (iwasakims) 
(cherry picked from commit 85ec5573eb9fd746a9295ecc6fe1ae683073aaf5)
 2016-04-06 03:29:49 +09:00
Kihwal Lee
f9764d073f HDFS-10239. Fsshell mv fails if port usage doesn't match in src and destination paths. Contributed by Kuhu Shukla. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java

(cherry picked from commit ef3da823573cbf16fd1d84479330dd457f95e0ff)
 2016-04-05 09:16:51 -05:00
Vinayakumar B
3deeb9a15f HDFS-9917. IBR accumulate more objects when SNN was down for sometime. (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 818d6b799eead13a17a0214172df60a269b046fb)
(cherry picked from commit ad08114b9285fecd2c41a4ac3e073662b1e212d3)
 2016-04-05 09:54:50 +08:00
Colin Patrick Mccabe
63fe2ae72c HDFS-8496. Calling stopWriter() with FSDatasetImpl lock held may block other threads (cmccabe) 
(cherry picked from commit f6b1a818124cc42688c4c5acaf537d96cf00e43b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

(cherry picked from commit d55af8b5a356e88a12f0b343b3ed3a974140f81d)
 2016-04-04 18:07:21 -07:00
Kihwal Lee
7d671cad3f HDFS-10178. Permanent write failures can happen if pipeline recoveries occur for the first packet. Contributed by Kihwal Lee. 
(cherry picked from commit a7d1fb0cd2fdbf830602eb4dbbd9bbe62f4d5584)
 2016-04-04 16:41:23 -05:00
Masatake Iwasaki
97cda4077a HDFS-9599. TestDecommissioningStatus.testDecommissionStatus occasionally fails (Lin Yiqun via iwasakims) 
(cherry picked from commit 154d2532cf015e9ab9141864bd3ab0d6100ef597)
 2016-04-05 03:44:12 +09:00
Vinayakumar B
bafcb584a8 HDFS-10253. Fix TestRefreshCallQueue failure (Contributed by Xiaoyu Yao) 
(cherry picked from commit 54b2e78fd28c9def42bec7f0418833bad352686c)
(cherry picked from commit aa38748997ae93d0ba91ecfbad77f7dfea25c7da)
 2016-04-03 13:28:52 +05:30
Akira Ajisaka
3892efd3fb HDFS-10221. Add .json to the rat exclusions. Contributed by Ming Ma. 
(cherry picked from commit 32c0c3ecdf72e89a63f4aee5e75d1c5a12714b89)
(cherry picked from commit 0795ee687b8f0cde251ba80faff4900be8d72c10)
 2016-03-31 09:06:48 +09:00
Colin Patrick Mccabe
5cf55c5fb6 HDFS-10223. peerFromSocketAndKey performs SASL exchange before setting connection timeouts (cmccabe) 
(cherry picked from commit 37e23ce45c592f3c9c48a08a52a5f46787f6c0e9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java

(cherry picked from commit be5894c12dd3af20474d74c444ad8ffd798ff417)
 2016-03-30 13:45:05 -07:00
Vinayakumar B
def143e1e1 HDFS-5177. blocksScheduled count should be decremented for abandoned blocks (Contributed by Vinayakumar B) 
(cherry picked from commit 09d63d5a192b5d6b172f94ff6c94da348fd49ea6)
(cherry picked from commit 0852e441108fd279d87a167a442eba046832779e)
 2016-03-30 14:23:56 +08:00
Arpit Agarwal
bfe657e83a HDFS-10228. TestHDFSCLI fails. (Contributed by Akira AJISAKA) 2016-03-29 13:56:12 -07:00
Andrew Wang
026b84bf70 HDFS-10197. TestFsDatasetCache failing intermittently due to timeout. Contributed by Lin Yiqun. 
(cherry picked from commit f2aec4eb824647a01e14b4eede03af0babe65fb6)
(cherry picked from commit 1d79b57c14410417b1ecccabfe2c695a1c6cd6a2)
 2016-03-29 11:55:04 -07:00
Vinayakumar B
88b56cdfaa HDFS-9871. "Bytes Being Moved" -ve(-1 B) when cluster was already balanced. (Contributed by Brahma Reddy Battulla) 
(cherry picked from commit 1f004b3367c57de9e8a67040a57efc31c9ba8ee2)
(cherry picked from commit f74364e8b9423f2cd923097a3b30fbd925c70d19)
 2016-03-29 11:31:18 +08:00
Walter Su
d71843558b HDFS-10182. Hedged read might overwrite user's buf. Contributed by zhouyingchao. 
(cherry picked from commit d8383c687c95dbb37effa307ab2d41497da1cfc2)
 2016-03-28 15:45:40 +08:00
Andrew Wang
d4aeddee31 HDFS-10200. Docs for WebHDFS still describe GETDELEGATIONTOKENS operation. Contributed by Wellington Chevreuil. 
(cherry picked from commit 8f85e5d2128c54c47d2a0098f6f4d4e04d53d74b)
(cherry picked from commit c658deabb66ce2e02bf462ccdf603e21eb65e100)
 2016-03-23 12:56:17 -07:00
Colin Patrick Mccabe
4ca13e7b98 HDFS-10193. fuse_dfs segfaults if uid cannot be resolved to a username (John Thiltges via cmccabe) 
(cherry picked from commit 0d19a0ce98053572447bdadf88687ec55f2f1f46)
(cherry picked from commit 7ad3162df421ee1104c90de08235af467d08e76b)
 2016-03-23 11:20:12 -07:00
Colin Patrick Mccabe
31843df277 HDFS-10189. PacketResponder#toString should include the downstreams for PacketResponderType.HAS_DOWNSTREAM_IN_PIPELINE (Joe Pallas via cmccabe) 
(cherry picked from commit a7d8f2b3960d27c74abb17ce2aa4bcd999706ad2)
(cherry picked from commit a68a9eb2b131e426ad16854e83ef81418442e989)
 2016-03-22 17:22:15 -07:00
Colin Patrick Mccabe
29872506bc HDFS-9951. Use string constants for XML tags in OfflineImageReconstructor (Lin Yiqun via cmccabe) 
(cherry picked from commit 680716f31e120f4d3ee70b095e4db46c05b891d9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/OfflineImageReconstructor.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineImageViewer/PBImageXmlWriter.java

(cherry picked from commit 2ce24bf21826fcb81cbb27765807ca9d12bedf6a)
 2016-03-21 11:49:19 -07:00
Andrew Wang
ea44650a64 HDFS-9405. Warmup NameNode EDEK caches in background thread. Contributed by Xiao Chen. 
(cherry picked from commit e3bb38d62567eafe57d16b78deeba1b71c58e41c)
(cherry picked from commit e826d860a7f587fac95e0abf257750d0a40ec65d)
 2016-03-21 11:40:26 -07:00
Haohui Mai
57fed702f1 HDFS-7166. SbNN Web UI shows #Under replicated blocks and #pending deletion blocks. Contributed by Wei-Chiu Chuang. 2016-03-20 17:22:09 -07:00
Akira Ajisaka
2fe4225051 HDFS-3677. dfs.namenode.edits.dir.required is missing from hdfs-default.xml. Contributed by Mark Yang. 
(cherry picked from commit 414f33126ac178a44146666cc7c00059cfc9cabf)
(cherry picked from commit 41621a994dd3add51ec95a3136070c148b133726)
 2016-03-19 01:11:06 +09:00
Kihwal Lee
242c7f1fee HDFS-9874. Long living DataXceiver threads cause volume shutdown to block. Contributed by Rushabh Shah. 
(cherry picked from commit 63c966a3fbeb675959fc4101e65de9f57aecd17d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2016-03-18 10:33:13 -05:00
Colin Patrick Mccabe
73b5a44b0e HDFS-9949. Add a test case to ensure that the DataNode does not regenerate its UUID when a storage directory is cleared (Harsh J via cmccabe) 
(cherry picked from commit dc951e606f40bb779632a8a3e3a46aeccc4a446a)
(cherry picked from commit f69a6c36325f53d5a1b201c8357d94e4139b9a93)
 2016-03-17 10:43:07 -07:00
Kihwal Lee
fce39557d3 HDFS-9904. testCheckpointCancellationDuringUpload occasionally fails. Contributed by Lin Yiqun. 
(cherry picked from commit d4574017845cfa7521e703f80efd404afd09b8c4)
 2016-03-15 10:55:10 -05:00
Ravi Prakash
9e804d14c9 HDFS-9953. Download File from UI broken after pagination. Contributed by Brahma Reddy Battula 
(cherry picked from commit 51b7d8935af7487547d46693946b386409edc6c8)
 2016-03-14 14:33:29 -07:00
Chris Nauroth
e6c6f4ddac HDFS-9941. Do not log StandbyException on NN, other minor logging fixes. Contributed by Arpit Agarwal. 
(cherry picked from commit 5644137adad30c84e40d2c4719627b3aabc73628)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockUnderConstructionFeature.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

(cherry picked from commit dd462bc32f77799827c1bd438ee8fa54dacb20eb)
 2016-03-14 09:55:34 -07:00