Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,406 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4514 Commits

Author SHA1 Message Date
Jing Zhao 30b43a2d4e HDFS-11476. Fix NPE in FsDatasetImpl#checkAndUpdate. Contributed by Xiaobing Zhou. 
(cherry picked from commit ac5ae0065a)
 2017-03-03 13:35:13 -08:00
Andrew Wang d737a26c5a Revert "HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru." 
This reverts commit 5e760f8dc4.
 2017-03-02 16:21:32 -08:00
Arpit Agarwal 5e760f8dc4 HDFS-11477. Simplify file IO profiling configuration. Contributed by Hanisha Koneru. 2017-03-02 15:38:32 -08:00
Ming Ma 8ce4863103 HDFS-11412. Maintenance minimum replication config value allowable range should be [0, DefaultReplication]. (Manoj Govindassamy via mingma) 2017-03-02 11:06:48 -08:00
Brahma Reddy Battula a545ee4d7f HDFS-11432. Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands. Contributed by Brahma Reddy Battula. 
(cherry picked from commit dcd03df9f9)
 2017-03-01 10:47:47 +05:30
Zhe Zhang 5fb785e154 HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms. Contributed by Andrew Wang. 
(cherry picked from commit d269b488a7)
 2017-02-28 09:02:59 -08:00
Wei-Chiu Chuang f3cdf29af4 HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing Zhao. 2017-02-25 21:13:51 -08:00
Arpit Agarwal 98af813098 HDFS-11462. Fix occasional BindException in TestNameNodeMetricsLogger. 2017-02-24 19:53:45 -08:00
Yiqun Lin 31f9ef2203 HDFS-11433. Document missing usages of OfflineEditsViewer processors. Contributed by Yiqun Lin. 
(cherry picked from commit 4a58870a04)
 2017-02-25 10:03:02 +08:00
Arpit Agarwal fa86ec99a1 HDFS-11295. Check storage remaining instead of node remaining in BlockPlacementPolicyDefault.chooseReplicaToDelete(). Contributed by Marton Elek. 2017-02-24 15:44:19 -08:00
Akira Ajisaka bd3c2a2ee9 HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 2017-02-25 04:17:17 +09:00
Andrew Wang 6c5fd82773 HDFS-11438. Fix typo in error message of StoragePolicyAdmin tool. Contributed by Alison Yu. 
(cherry picked from commit d150f061f4)
 2017-02-22 15:16:12 -08:00
Ming Ma 0add328c8c HDFS-11411. Avoid OutOfMemoryError in TestMaintenanceState test runs. (Manoj Govindassamy via mingma) 
(cherry picked from commit cfcd527323)
 2017-02-22 09:42:47 -08:00
Andrew Wang 4428c3da69 HDFS-11406. Remove unused getStartInstance and getFinalizeInstance in FSEditLogOp. Contributed by Alison Yu. 
(cherry picked from commit 8ef7ebbc71)
 2017-02-21 14:54:32 -08:00
Eric Payne 41d31588d1 HDFS-11404. Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpc. Contributed by Eric Badger 
(cherry picked from commit a07ddef101)
 2017-02-21 12:09:29 -06:00
Vinayakumar B 00ca9f1792 HDFS-11177. 'storagepolicies -getStoragePolicy' command should accept URI based path. (Contributed by Surendra Singh Lilhore) 
(cherry picked from commit 4804050630)
 2017-02-21 18:15:03 +05:30
Xiao Chen 06386b7e78 HDFS-11410. Use the cached instance when edit logging SetAclOp, SetXAttrOp and RemoveXAttrOp. 
(cherry picked from commit 02c549484a)
 2017-02-16 18:08:16 -08:00
Ming Ma 0f8b99fd34 HDFS-11265. Extend visualization for Maintenance Mode under Datanode tab in the NameNode UI. (Marton Elek via mingma) 
(cherry picked from commit a136936d01)
 2017-02-15 20:25:04 -08:00
Mingliang Liu 96be70513a HDFS-11375. Display the volume storage type in datanode UI. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 0741dd3b9a)
 2017-02-15 12:34:43 -08:00
Wei-Chiu Chuang a8531d5d52 HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 859bd159ae)
 2017-02-15 02:59:45 -08:00
Yongjun Zhang eb5500f0e7 HDFS-11391. Numeric usernames do no work with WebHDFS FS write access. (Pierre Villard via Yongjun Zhang) 
(cherry picked from commit 8e53f2b9b0)
 2017-02-14 14:11:45 -08:00
Wei-Chiu Chuang 8e291e1015 HDFS-11084. Add a regression test for sticky bit support of OIV ReverseXML processor. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 89cadb42111e4ffbd3f4bde8250013bba23eb51e)
 2017-02-14 09:08:37 -08:00
Kihwal Lee 33c62d2d19 HDFS-11379. DFSInputStream may infinite loop requesting block locations. Contributed by Daryn Sharp. 2017-02-10 12:52:14 -06:00
Yiqun Lin 26bed5ce53 HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao. 
(cherry picked from commit 9cbbd1eae8)
 2017-02-06 13:18:14 +08:00
Arpit Agarwal b64d9093e1 HDFS-11112. Journal Nodes should refuse to format non-empty directories. Contributed by Yiqun Lin. 2017-02-01 16:52:07 -08:00
Lei Xu d258077bc1 HDFS-11335. Remove HdfsClientConfigKeys.DFS_CLIENT_SLOW_IO_WARNING_THRESHOLD_KEY usage from DNConf. (Manoj Govindassamy via lei) 
(cherry picked from commit bec9b7aa1d)
 2017-02-01 14:44:08 +08:00
Karthik Kambatla 8b01d5cbba HDFS-11378. Addendum to fix branch-2 compilation. Verify multiple DataNodes can be decommissioned/maintenance at the same time. (Manoj Govindassamy via kasha) 2017-01-29 11:18:54 -08:00
Ming Ma 34f9ceab4a HDFS-11378. Verify multiple DataNodes can be decommissioned/maintenance at the same time. (Manoj Govindassamy via mingma) 
(cherry picked from commit 312b36d113)
 2017-01-27 16:17:51 -08:00
Masatake Iwasaki 528bff9c42 HDFS-9884. Use doxia macro to generate in-page TOC of HDFS site documentation. (iwasakims) 2017-01-28 08:11:40 +09:00
Anu Engineer d20dc8691e HDFS-9911. TestDataNodeLifeline Fails intermittently. Contributed by Yiqun Lin 
(cherry picked from commit a95639068c)
 2017-01-27 20:55:59 +00:00
Arpit Agarwal 17d4ba26fc HDFS-11374. Skip FSync in CreateEditsLog to speed up edit log generation. Contributed by Hanisha Koneru. 2017-01-26 16:21:38 -08:00
Xiao Chen 67d2875f8f Revert "HDFS-8377. Support HTTP/2 in datanode. Contributed by Duo Zhang." 
This reverts commit f70b9d9241.
 2017-01-26 14:21:17 -08:00
Xiao Chen 827cd30923 HDFS-11364. Add a test to verify Audit log entries for setfacl/getfacl commands over FS shell. Contributed by Manoj Govindassamy. 
(cherry picked from commit 44606aa850)
 2017-01-26 10:58:17 -08:00
Arpit Agarwal deb3ef4587 HDFS-11369. Change exception message in StorageLocationChecker. 2017-01-26 09:12:46 -08:00
Zhe Zhang 558fee285a HDFS-10534. NameNode WebUI should display DataNode usage histogram. Contributed by Kai Sasaki. 
(cherry picked from commit 18e1d68209)
 2017-01-25 10:39:11 -08:00
Arpit Agarwal b4078e1d0e HDFS-11194. Maintain aggregated peer performance metrics on NameNode. 2017-01-24 19:34:33 -08:00
Ming Ma bed700e98f HDFS-11296. Maintenance state expiry should be an epoch time and not jvm monotonic. (Manoj Govindassamy via mingma) 
(cherry picked from commit f3fb94be05)
 2017-01-19 22:33:43 -08:00
Lei Xu 1cc5f460ed HDFS-11259. Update fsck to display maintenance state info. (Manoj Govindassamy via lei) 2017-01-19 16:24:58 +08:00
Lei Xu 541efe18c7 HDFS-11287. Storage class member storageDirs should be private to avoid unprotected access by derived classes. (Manoj Govindassamy via lei) 
(cherry picked from commit 63320d1daa)
 2017-01-19 15:38:40 +08:00
Vinitha Reddy Gankidi e2b3eff641 HDFS-10733. NameNode terminated after full GC thinking QJM is unresponsive. Contributed by Vinitha Gankidi. 2017-01-18 12:52:58 -08:00
Akira Ajisaka db47bd15dd HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen. 
(cherry picked from commit b1a9ec856b)
 2017-01-18 17:56:22 +09:00
Arpit Agarwal 42a14b6db9 HDFS-10917. Collect peer performance statistics on DataNode. Contributed by Xiaobing Zhou. 2017-01-17 15:31:18 -08:00
Xiaoyu Yao 420e4db98a HDFS-11209. SNN can't checkpoint when rolling upgrade is not finalized. Contributed by Xiaoyu Yao. 
(cherry picked from commit b1fce2b8b1)
 2017-01-17 14:36:03 -08:00
Xiaoyu Yao eae5bde007 HDFS-10913. Introduce fault injectors to simulate slow mirrors. Contributed by Xiaobing Zhou. 2017-01-17 10:30:34 -08:00
Arpit Agarwal be8e001f1f HDFS-11339. Support File IO sampling for Datanode IO profiling hooks. Contributed by Hanisha Koneru. 2017-01-16 22:49:14 -08:00
Arpit Agarwal 52fdd387ec HDFS-11299. Support multiple Datanode File IO hooks. Contributed by Hanisha Koneru. 2017-01-16 22:42:13 -08:00
Arpit Agarwal 8ec9dca2e1 HDFS-11279. Cleanup unused DataNode#checkDiskErrorAsync(). Contributed by Hanisha Koneru. 2017-01-16 22:14:06 -08:00
Xiaoyu Yao eafaddca1a HDFS-11274. Datanode should only check the failed volume upon IO errors. Contributed by Xiaoyu Yao. 2017-01-16 22:14:04 -08:00
Arpit Agarwal ec80de3ccc HDFS-11182. Update DataNode to use DatasetVolumeChecker. 2017-01-16 22:14:00 -08:00
Arpit Agarwal 413dccaf78 HDFS-10959. Update DataNode to use DatasetVolumeChecker. Contributed by Xiaoyu Yao. 2017-01-16 22:13:58 -08:00
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit e407449ddb)
 2017-01-17 10:22:29 +09:00
Arpit Agarwal 954dae26cd HDFS-11337. (HDFS-10958 backport). Add instrumentation hooks around Datanode disk IO. 2017-01-16 13:53:30 -08:00
Wei-Chiu Chuang 914eeb997b HDFS-11306. Print remaining edit logs from buffer if edit log can't be rolled. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 1cde954a4f)
 2017-01-13 11:46:57 -08:00
Ming Ma b6258e2b15 HDFS-9391. Update webUI/JMX to display maintenance state info. (Manoj Govindassamy via mingma) 2017-01-10 20:20:13 -08:00
Mingliang Liu ce5ad0e88f HDFS-11253. FileInputStream leak on failure path in BlockSender. Contributed by Arpit Agarwal 2017-01-10 14:48:28 -08:00
Xiaoyu Yao 48fb796fcb HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 2017-01-10 14:48:23 -08:00
Yongjun Zhang 137547579b HDFS-11292. log lastWrittenTxId etc info in logSyncAll. Contributed by Yongjun Zhang. 
(cherry picked from commit 603cbcd513)
 2017-01-09 16:16:42 -08:00
Arpit Agarwal 6f35700f04 HDFS-11149. Support for parallel checking of FsVolumes. 
Change-Id: I6e7ada81562f380a260f8f561d0fa6a4cb102560
 2017-01-06 07:55:19 -08:00
Arpit Agarwal fd3b1ca26c HDFS-11148. Update DataNode to use StorageLocationChecker at startup. 
Change-Id: I61dfae840e767f9a33c0bb45a058f2b866cbffa4
 2017-01-05 20:49:34 -08:00
Arpit Agarwal 76f1ab524c HDFS-11119. Support for parallel checking of StorageLocations on DataNode startup. 
Change-Id: Iddedbeefd056af165be3557ccad199fe2878b591
 2017-01-05 20:49:30 -08:00
Chris Nauroth 9c61858ee7 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e1302)
 2017-01-05 15:05:01 -08:00
Andrew Wang d4ca1c5226 HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 2017-01-04 15:10:15 -08:00
Andrew Wang 2b5b62f9cc Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang." 
This reverts commit 80e6640848.
 2017-01-03 11:12:53 -08:00
Andrew Wang 80e6640848 HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
(cherry picked from commit 7fcc73fc0d)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
 2017-01-03 09:58:45 -08:00
Lei Xu db81cc3add HDFS-11267. Avoid redefinition of storageDirs in NNStorage and cleanup its accessors in Storage. (Manoj Govindassamy via lei) 
(cherry picked from commit a4f66655ec)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNStorage.java
 2016-12-29 17:06:21 +08:00
Lei Xu f349c7692d HDFS-11251. ConcurrentModificationException during DataNode#refreshVolumes. (Manoj Govindassamy via lei) 
(cherry picked from commit e9f1396834)
 2016-12-29 15:35:17 +08:00
Daniel Templeton af266c8c85 HDFS-11278. Add missing @Test annotation for TestSafeMode.testSafeModeUtils() (Contributed by Lukas Majercak via Daniel Templeton) 
(cherry picked from commit ce3613c965)
 2016-12-28 20:11:09 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Brahma Reddy Battula 5d46d5d4c6 HDFS-11252. TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException. Contributed by Yiqun Lin. 
(cherry picked from commit 0ddb8defad)
 2016-12-28 10:37:21 +05:30
Mingliang Liu 74ee21a7b1 HDFS-11270. Document the missing options of NameNode bootstrap command. Contributed by Yiqun Lin 
(cherry picked from commit c0e0ef2969)
 2016-12-27 11:38:31 -08:00
Akira Ajisaka cd331556f6 HDFS-11272. Refine the assert messages in TestFSDirAttrOp. Contributed by Jimmy Xiang. 
(cherry picked from commit ea547529cb)
 2016-12-26 17:17:14 +09:00
Akira Ajisaka 1b39268233 HDFS-11250. Fix a typo in ReplicaUnderRecovery#setRecoveryID. Contributed by Yiqun Lin. 
(cherry picked from commit 8f218ea284)
 2016-12-26 16:52:43 +09:00
Akira Ajisaka a779b4ab8f HDFS-11271. Typo in NameNode UI. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 483cd06ad4)
 2016-12-26 16:37:52 +09:00
Akira Ajisaka 16c4dcad85 HDFS-11258. Addendum patch to fix compile issue. Contributed by Jimmy Xiang. 2016-12-23 03:56:12 +09:00
Haohui Mai de926f93c2 HDFS-11258. File mtime change could not save to editlog. Contributed by Jimmy Xiang. 2016-12-21 14:22:09 -08:00
Xiao Chen 1ab279c311 HDFS-11195. Return error when appending files by webhdfs rest api fails. Contributed by Yuanbo Liu. 
(cherry picked from commit 5b7acdd206)
 2016-12-20 12:27:40 -08:00
Brahma Reddy Battula 7a92345f16 HDFS-11263. ClassCastException when we use Bzipcodec for Fsimage compression. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 1b401f6a73)
 2016-12-20 21:02:16 +05:30
Akira Ajisaka a95acbcb6e HDFS-11262. Remove unused variables in FSImage.java. Contributed by Jagadesh Kiran N. 
(cherry picked from commit d2026ae9e6)
 2016-12-20 14:36:03 +09:00
Wei-Chiu Chuang 0cb99db9d9 HDFS-11160. VolumeScanner reports write-in-progress replicas as corrupt incorrectly. Contributed by Wei-Chiu Chuang and Yongjun Zhang. 2016-12-15 16:35:37 -08:00
Mingliang Liu 012d28a1fa HDFS-11094. Send back HAState along with NamespaceInfo during a versionRequest as an optional parameter. Contributed by Eric Badger 2016-12-15 14:54:40 -08:00
Akira Ajisaka eaa50a1087 HDFS-11249. Redundant toString() in DFSConfigKeys.java. Contributed by Jagadesh Kiran N. 
(cherry picked from commit 36947f79ba)
 2016-12-16 00:34:43 +09:00
Akira Ajisaka 190b2dca62 HDFS-11204. Document the missing options of hdfs zkfc command. Contributed by Yiqun Lin. 
(cherry picked from commit 72bff192cd)
 2016-12-14 18:53:24 +09:00
Andrew Wang 6b795c34d0 HDFS-10684. WebHDFS DataNode calls fail without parameter createparent. Contributed by John Zhuge. 
(cherry picked from commit fbdbbd57cd)
 2016-12-13 18:01:35 -08:00
Mingliang Liu 282a562437 HDFS-11226. cacheadmin, cryptoadmin and storagepolicyadmin should support generic options. Contributed by Brahma Reddy Battula 
(cherry picked from commit e91494c27a0b9a4a456d92fe7d632a608c4f11f3)
 2016-12-12 18:17:54 -08:00
Xiaoyu Yao 0bdaa23334 Revert "HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao." 
This reverts commit 4c9ca47386.
 2016-12-09 21:37:13 -08:00
Xiaoyu Yao 4c9ca47386 HDFS-10930. Refactor: Wrap Datanode IO related operations. Contributed by Xiaoyu Yao. 
(cherry picked from commit df983b524a)
 2016-12-09 21:31:45 -08:00
Wei-Chiu Chuang 292bd78b44 HDFS-11229. HDFS-11056 failed to close meta file. Contributed by Wei-Chiu Chuang. 2016-12-09 16:07:56 -08:00
Andrew Wang 215abbdf5f HDFS-8630. WebHDFS : Support get/set/unset StoragePolicy. Contributed by Surendra Singh Lilhore. 2016-12-09 14:29:22 -08:00
Andrew Wang 85adc6677b HDFS-11217. Annotate NameNode and DataNode MXBean interfaces as Private/Stable. Contributed by Jagadesh Kiran N. 
(cherry picked from commit 5bd7dece92)
 2016-12-09 12:05:45 -08:00
Vinayakumar B 199d8b5594 HDFS-11224. Lifeline message should be ignored for dead nodes (Contributed by Vinayakumar B) 
(cherry picked from commit d1d4aba71b)
 2016-12-09 14:54:37 +05:30
Xiao Chen e51f32f74c HDFS-11197. Listing encryption zones fails when deleting a EZ that is on a snapshotted directory. Contributed by Wellington Chevreuil. 
(cherry picked from commit 401c731872)
 2016-12-08 12:41:17 -08:00
Akira Ajisaka e00a124cb5 HDFS-11140. Directory Scanner should log startup message time correctly. Contributed by Yiqun Lin. 
(cherry picked from commit 9ef89ede2f)
 2016-12-08 13:39:58 +09:00
Vinayakumar B 4b21b47f56 HDFS-9057. allow/disallow snapshots via webhdfs (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 6d2332ae37)
 2016-12-07 15:58:15 -08:00
Ming Ma d2656dc5a6 HDFS-10206. Datanodes not sorted properly by distance when the reader isn't a datanode. (Nandakumar via mingma) 2016-12-07 08:27:17 -08:00
Karthik Kambatla 8773cd5f76 HADOOP-13859. TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties. (Haibo Chen via kasha) 
(cherry picked from commit c8d0a049b0)
 2016-12-06 14:16:38 -08:00
Mingliang Liu eb9e7820a9 HDFS-11211. Add a time unit to the DataNode client trace format. Contributed by Jagadesh Kiran N 
(cherry picked from commit 43cb1678cc)
 2016-12-06 09:13:38 -08:00
Kihwal Lee f74dad7414 HDFS-11198. NN UI should link DN web address using hostnames. Contributed by Weiwei Yang. 2016-12-06 08:07:54 -06:00
Andrew Wang 6a2f239d9e Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang" 
This reverts commit be969e5918.
 2016-12-05 23:09:14 -08:00
Andrew Wang 2fdae7324b HDFS-10581. Hide redundant table on NameNode WebUI when no nodes are decomissioning. Contributed by Weiwei Yang. 
(cherry picked from commit 8e63fa98ea)
 2016-12-05 18:14:06 -08:00
Mingliang Liu be969e5918 HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang 
(cherry picked from commit c7ff34f8dc)
 2016-12-02 11:36:11 -08:00
Akira Ajisaka 5a7941a4fc HDFS-11180. Intermittent deadlock in NameNode when failover happens. 2016-12-02 11:34:05 +09:00
Kihwal Lee 0d8a35bd6d HDFS-8674. Improve performance of postponed block scans. Contributed by Daryn Sharp. 2016-12-01 12:15:15 -06:00
Ravi Prakash fdf5394b2a HDFS-8678. Bring back the feature to view chunks of files in the HDFS file browser. Contributed by Ivo Udelsmann. 
(cherry picked from commit 625df87c7b)
 2016-11-30 09:14:36 -08:00
Robert Kanter 285edf691d HADOOP-13838. KMSTokenRenewer should close providers (xiaochen via rkanter) 
(cherry picked from commit 47ca9e26fb)
 2016-11-28 18:12:53 -08:00
Akira Ajisaka 0df7f5cdf6 HDFS-11175. Document uppercase key names are not supported in TransparentEncryption.md. Contributed by Yiqun Lin. 
(cherry picked from commit 636f7547b7)
 2016-11-25 17:50:14 +09:00
Brahma Reddy Battula 2207a84a7d HADOOP-13646. Remove outdated overview.html. Contributed By Brahma Reddy Battula. 
(cherry picked from commit afcf8d38e7)
 2016-11-22 20:32:30 +05:30
Kihwal Lee 774b0cd845 HDFS-10966. Enhance Dispatcher logic on deciding when to give up a source DataNode. Contributed by Mark Wagner and Zhe Zhang. 2016-11-21 10:39:04 -06:00
Brahma Reddy Battula aa76fe4132 HDFS-11101. TestDFSShell#testMoveWithTargetPortEmpty fails intermittently. Contributed By Brahma Reddy Battula. 
(cherry picked from commit f922067748)
 2016-11-21 19:36:10 +05:30
Brahma Reddy Battula f3649deca5 HDFS-11144. TestFileCreationDelete#testFileCreationDeleteParent fails wind bind exception. Contributed By Brahma Reddy Battula. 
(cherry picked from commit c68dad18ab)
 2016-11-21 10:59:23 +05:30
Brahma Reddy Battula 3e5a85f35e HDFS-11134. Fix bind exception threw in TestRenameWhileOpen. Contributed By Yiqun Lin. 
(cherry picked from commit b2d4b7b1b9)
 2016-11-17 10:24:21 +05:30
Arpit Agarwal da8c24bbad HDFS-11114. Support for running async disk checks in DataNode. 2016-11-16 13:38:15 -08:00
Mingliang Liu 57a0f84860 HDFS-11105. TestRBWBlockInvalidation#testRWRInvalidation fails intermittently. Contributed by Yiqun Lin 
(cherry picked from commit c90891e7b3)
 2016-11-16 13:06:25 -08:00
Kihwal Lee f08e227671 HDFS-8870. Lease is leaked on write failure. Contributed by Kuhu Shukla. 
(cherry picked from commit 4fcea8a0c8)
 2016-11-15 14:51:34 -06:00
Zhe Zhang 818915aabb HDFS-10872. Add MutableRate metrics for FSNamesystemLock operations. Contributed by Erik Krogen. 2016-11-14 16:33:15 -08:00
Brahma Reddy Battula 399abe7c74 HDFS-11135. The tests in TestBalancer run fails due to NPE. Contributed By Yiqun Lin 
(cherry picked from commit 6efb8c9c60)
 2016-11-14 20:41:55 +05:30
Erik Krogen 98d9e9c4c8 HDFS-11087. NamenodeFsck should check if the output writer is still writable. Contributed by Erik Krogen. 2016-11-13 14:50:33 -08:00
Mingliang Liu fb3a5085e8 HADOOP-13813. TestDelegationTokenFetcher#testDelegationTokenWithoutRenewer is failing. Contributed by Mingliang Liu 2016-11-12 10:36:14 -08:00
Mingliang Liu 685d753d07 HDFS-11129. TestAppendSnapshotTruncate fails with bind exception. Contributed by Brahma Reddy Battula 
(cherry picked from commit 2ee18fc15e)
 2016-11-11 22:16:05 -08:00
Xiaoyu Yao 5613b0818c HDFS-10941. Improve BlockManager#processMisReplicatesAsync log. Contributed by Chen Liang. 
(cherry picked from commit 4484b48498)
 2016-11-11 14:31:54 -08:00
Mingliang Liu 289f03e49e HDFS-11122. TestDFSAdmin#testReportCommand fails due to timed out. Contributed by Yiqun Lin 
(cherry picked from commit aa6010ccca)
 2016-11-11 12:30:22 -08:00
Brahma Reddy Battula b51806b45f HDFS-11128. CreateEditsLog throws NullPointerException. Contributed by Hanisha Koneru. 
(cherry picked from commit 1ae57f0f75)
 2016-11-11 20:09:49 +05:30
Xiao Chen a6146a1026 HDFS-11120. TestEncryptionZones should waitActive. Contributed by John Zhuge. 
(cherry picked from commit 907b1431c0ead934e26cf7dbc51bfab5d7cc5ddc)
 2016-11-09 17:16:31 -08:00
Wei-Chiu Chuang 16430c64b7 HDFS-11056. Concurrent append and read operations lead to checksum error. Contributed by Wei-Chiu Chuang. 2016-11-09 13:28:16 -08:00
Brahma Reddy Battula 53ff3c9e03 HDFS-9482. Replace DatanodeInfo constructors with a builder pattern. Contributed by Brahma Reddy Battula. 2016-11-09 09:58:36 +05:30
Mingliang Liu f5c9f9f1b2 HDFS-11083. Add unit test for DFSAdmin -report command. Contributed by Xiaobing Zhou 
(cherry picked from commit 62d8c17dfd)
 2016-11-08 17:05:34 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa)
 2016-11-08 16:33:58 -08:00
Arpit Agarwal b77239b46d Revert "HDFS-11114. Support for running async disk checks in DataNode." 
This reverts commit 732eaadddb.
 2016-11-08 09:04:58 -08:00
Arpit Agarwal 732eaadddb HDFS-11114. Support for running async disk checks in DataNode. 
This closes #153.
 2016-11-07 18:46:37 -08:00
Xiao Chen f30d338766 HDFS-10756. Expose getTrashRoot to HTTPFS and WebHDFS. Contributed by Yuanbo Liu. 
(cherry picked from commit d8bab3dcb6)
 2016-11-04 18:08:46 -07:00
Mingliang Liu 4cee535aa5 HDFS-11085. Add unit test for NameNode failing to start when name dir is unwritable. Contributed by Xiaobing Zhou 
(cherry picked from commit 0c0ab102ab)
 2016-11-04 14:39:54 -07:00
Andrew Wang 9ede5ccfdd HADOOP-11088. Quash unnecessary safemode WARN message during NameNode startup. Contributed by Yiqun Lin. 
(cherry picked from commit 5cad93d5c7)
 2016-11-03 15:31:21 -07:00
Mingliang Liu 6d0cea4a5a HDFS-11076. Add unit test for extended Acls. Contributed by Chen Liang 
(cherry picked from commit 7534aee09a)
 2016-11-03 13:52:18 -07:00
Arpit Agarwal 458b636a39 HDFS-11097. Fix warnings for deprecated StorageReceivedDeletedBlocks constructor. Contributed by Yiqun Lin. 2016-11-03 10:32:11 -07:00
Mingliang Liu 7a662bea89 HDFS-11095. BlockManagerSafeMode should respect extension period default config value (30s). Contributed by Mingliang Liu 
(cherry picked from commit 9b0c17f852)
 2016-11-02 15:02:08 -07:00
Brahma Reddy Battula 35e6081020 HDFS-11031. Add additional unit test for DataNode startup behavior when volumes fail. Contributed by Mingliang Liu. 2016-11-02 10:39:12 +05:30
Mingliang Liu 1dd59042ac HDFS-11065. Add space quota tests for heterogenous storages. Contributed by Xiaobing Zhou 
(cherry picked from commit fa1512a740)
 2016-11-01 14:46:24 -07:00
Kihwal Lee 7c1a1834e4 HDFS-10997. Reduce number of path resolving methods. Contributed by Daryn Sharp. 2016-11-01 08:04:54 -05:00
Brahma Reddy Battula 1c5708f6de HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Rakesh R. 2016-11-01 10:49:43 +05:30
Mingliang Liu 6bb12ecc3f HDFS-11030. TestDataNodeVolumeFailure#testVolumeFailure is flaky (though passing). Contributed by Mingliang Liu 2016-10-31 16:03:02 -07:00
Arpit Agarwal 56092e351b HDFS-11074. Remove unused method FsDatasetSpi#getFinalizedBlocksOnPersistentStorage. Contributed by Hanisha Koneru. 2016-10-31 12:39:54 -07:00
Mingliang Liu 2848a27700 HDFS-11047. Remove deep copies of FinalizedReplica to alleviate heap consumption on DataNode. Contributed by Xiaobing Zhou 2016-10-28 11:11:31 -07:00
Mingliang Liu d0da47ad9f HDFS-11048. Audit Log should escape control characters. Contributed by Eric Badger 
(cherry picked from commit 8a9388e5f6)
 2016-10-28 10:57:20 -07:00
Erik Krogen ce8ace372c HDFS-9500. Fix software version counts for DataNodes during rolling upgrade. Contributed by Erik Krogen. 
(cherry picked from commit f3ac1f41b8)
 2016-10-27 16:00:39 -07:00
Kihwal Lee 5057abc390 HDFS-11069. Tighten the authorization of datanode RPC. Contributed by Kihwal Lee 
(cherry picked from commit ae48c496dc)
 2016-10-27 14:18:10 -05:00
Kihwal Lee 6c11a11918 Revert "HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu" 
This reverts commit f4300ea12a.
 2016-10-27 14:10:07 -05:00
Brahma Reddy Battula f4300ea12a HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Tianyin Xiu 
(cherry picked from commit ac35ee9393)
 2016-10-27 20:22:08 +05:30
Vinayakumar B 144f7a9e0e HDFS-8492. DN should notify NN when client requests a missing block (Contributed by Walter Su) 
(cherry picked from commit 1cf6ec4ad4)
 2016-10-27 16:46:09 +05:30
Akira Ajisaka 53ed19efec HDFS-11049. The description of dfs.block.replicator.classname is not clear. Contributed by Yiqun Lin. 
(cherry picked from commit 9c680d0b05)
 2016-10-27 16:37:32 +09:00
Vinayakumar B 72ce627533 HDFS-9929. Duplicate keys in NAMENODE_SPECIFIC_KEYS (Contributed by Akira Ajisaka) 
(cherry picked from commit f0642b02996ae90d85499ddef2e20bdf88d98c7b)
 2016-10-27 12:19:08 +05:30
Mingliang Liu 2aed61d887 HDFS-10921. TestDiskspaceQuotaUpdate doesn't wait for NN to get out of safe mode. Contributed by Eric Badger 
(cherry picked from commit 55e1fb8e32)
 2016-10-26 14:12:08 -07:00
Kihwal Lee 0cd43dd2b7 HDFS-11053. Unnecessary superuser check in versionRequest(). Contributed by Kihwal Lee. 
(cherry picked from commit e90af4a89b)
 2016-10-26 09:17:29 -05:00
Arpit Agarwal a071e33a91 HDFS-11054. Suppress verbose log message in BlockPlacementPolicyDefault. Contributed by Chen Liang. 2016-10-25 19:51:32 -07:00
Xiaoyu Yao 590d19c0a8 HDFS-11011. Add unit tests for HDFS command 'dfsadmin -set/clrSpaceQuota'. Contributed by Xiaobing Zhou. 
(cherry picked from commit 85f6fec0370b3cd94d6c2f19920c0b6d33f127c6)
 2016-10-25 11:24:27 -07:00
Zhe Zhang ff806cbfc7 HDFS-11015. Enforce timeout in balancer. Contributed by Kihwal Lee. 
(cherry picked from commit f6367c5f44)
 2016-10-25 10:45:31 -07:00
Kihwal Lee 086577c67a HDFS-11042. Add missing cleanupSSLConfig() call for tests that use setupSSLConfig(). Contributed by Kuhu Shukla. 
(cherry picked from commit b18f35f097)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
 2016-10-24 08:56:14 -05:00
Chris Douglas 25f4327f0b HADOOP-13702. Add instrumented ReadWriteLock. Contributed by Jingcheng Du 
(cherry picked from commit ae8bccd509)
 2016-10-21 13:01:08 -07:00
Brahma Reddy Battula 2ab149f0e4 HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula 
(cherry picked from commit f872c6bc03)
 2016-10-21 16:19:18 +05:30
Mingliang Liu c37c00b061 HDFS-10998. Add unit tests for HDFS command 'dfsadmin -fetchImage' in HA. Contributed by Xiaobing Zhou 
(cherry picked from commit d7d87deece)
 2016-10-20 19:54:35 -07:00
Wei-Chiu Chuang 6f4192d77d HDFS-11018. Incorrect check and message in FsDatasetImpl#invalidate. Contributed by Yiqun Lin. 
(cherry picked from commit 6d2da38d16)

Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
 2016-10-20 11:58:46 -07:00
Steve Loughran ab36519b6f Revert "HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula" 
This reverts commit bf6379fdd2.
 2016-10-20 19:32:10 +01:00
Brahma Reddy Battula bf6379fdd2 HADOOP-13236. truncate will fail when we use viewfilesystem. Contributed by Brahma Reddy Battula 
(cherry picked from commit f872c6bc03)
 2016-10-20 21:48:23 +05:30
Kihwal Lee cacf83b6c3 HDFS-10627. Volume Scanner marks a block as "suspect" even if the exception is network-related. Contributed by Rushabh S Shah. 
(cherry picked from commit 5c0bffddc0)
 2016-10-20 11:02:02 -05:00
Brahma Reddy Battula 69a4b0c7d2 HDFS-10699. Log object instance get incorrectly in TestDFSAdmin. Contributed by Yiqun Lin 
(cherry picked from commit f5e401de3e7ad6e73b07fef1cb5915753b08fbdb)
 2016-10-20 21:08:39 +05:30
Brahma Reddy Battula df5671db74 HDFS-9480. Expose nonDfsUsed via StorageTypeStats. Contributed by Brahma Reddy Battula 
(cherry picked from commit 4c73be135c)
 2016-10-20 20:47:37 +05:30
Brahma Reddy Battula 5a37668d1b HDFS-11025. TestDiskspaceQuotaUpdate fails in trunk due to Bind exception. Contributed by Yiqun Lin 
(cherry picked from commit 73504b1bdc)
 2016-10-20 10:42:34 +05:30
Arpit Agarwal a3cbaf0c04 HDFS-10752. Several log refactoring/improvement suggestion in HDFS. Contributed by Hanisha Koneru. 2016-10-19 17:20:15 -07:00
Xiao Chen ad7d3c4db8 HDFS-11009. Add a tool to reconstruct block meta file from CLI. 
(cherry picked from commit bc4a32aea50e86819730312e89315c0244ce64bf)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java
 2016-10-18 22:40:58 -07:00
Xiaoyu Yao 74f1c49d76 HDFS-10906. Add unit tests for Trash with HDFS encryption zones. Contributed by Hanisha Koneru. 
(cherry picked from commit 0c79d5af74944363f3d04f7e1702411db564bbfd)
 2016-10-18 14:07:57 -07:00
Kai Zheng 3972bb3bba HDFS-10920. TestStorageMover#testNoSpaceDisk is failing intermittently. Contributed by Rakesh R 2016-10-18 15:08:15 +06:00
Akira Ajisaka b2618685fb HADOOP-13522. Add %A and %a formats for fs -stat command to print permissions. Contributed by Alex Garbarini. 
(cherry picked from commit bedfec0c10)
 2016-10-18 15:03:33 +09:00
Vinitha Reddy Gankidi 278eeb1056 HDFS-10712. TestDataNodeVolumeFailure should pass not null BlockReportContext. Contributed by Vinitha Gankidi. 2016-10-17 18:55:26 -07:00
Ming Ma d55a7f8935 HDFS-9390. Block management for maintenance states. 2016-10-17 17:46:29 -07:00
Xiaoyu Yao 61aceb87df HDFS-10922. Adding additional unit tests for Trash (II). Contributed by Weiwei Yang. 
(cherry picked from commit f1802d0be05ecc0b3248690b6f9efedbc7784112)
 2016-10-17 14:21:51 -07:00
Brahma Reddy Battula 64e532393b HDFS-11003. Expose XmitsInProgress through DataNodeMXBean. Contributed By Brahma Reddy Battula 
(cherry picked from commit 5f4ae85bd8)
 2016-10-15 22:29:55 +05:30
Vinitha Reddy Gankidi 863e1020c0 HDFS-10301. Remove FBR tracking state to fix false zombie storage detection for interleaving block reports. Contributed by Vinitha Gankidi. 
(cherry picked from commit 391ce535a7)
 2016-10-14 18:25:52 -07:00
Lei Xu 78cb79fa91 HDFS-10960. TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove. (Manoj Govindassamy via lei) 2016-10-14 15:00:27 -07:00
Mingliang Liu bbcbf78f0f HDFS-11008. Change unit test for testing parsing "-source" parameter in Balancer CLI. Contributed by Mingliang Liu 
(cherry picked from commit 76cc84e6d4)
 2016-10-14 14:43:58 -07:00
Kihwal Lee 1c0f2ef82f Revert "HDFS-10960. TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove. (Manoj Govindassamy via lei)" 
This reverts commit aa8593f923.
 2016-10-14 16:38:25 -05:00
Lei Xu aa8593f923 HDFS-10960. TestDataNodeHotSwapVolumes#testRemoveVolumeBeingWritten fails at disk error verification after volume remove. (Manoj Govindassamy via lei) 
(cherry picked from commit 8c520a27cb)
 2016-10-14 13:43:26 -07:00
Andrew Wang 86325e4346 HDFS-10883. 's behavior is not consistent in DFS after enabling EZ. Contributed by Yuanbo Liu. 
(cherry picked from commit 0007360c33)
 2016-10-14 11:41:36 -07:00
Kihwal Lee 6e0b55fe79 HDFS-10990. TestPendingInvalidateBlock should wait for IBRs. Contributed by Yiqun Lin. 
(cherry picked from commit 8a9f6635a3)
 2016-10-14 11:40:11 -05:00
Andrew Wang dc78e0e7fc Revert "HDFS-10990. TestPendingInvalidateBlock should wait for IBRs. Contributed by Yiqun Lin." 
This reverts commit 2e153bc8ab.
 2016-10-13 13:23:04 -07:00
Kihwal Lee c5a1303704 HDFS-10987. Make Decommission less expensive when lot of blocks present. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 332a61fd74)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java
 2016-10-13 14:55:22 -05:00
Andrew Wang 2e153bc8ab HDFS-10990. TestPendingInvalidateBlock should wait for IBRs. Contributed by Yiqun Lin. 
(cherry picked from commit fdce515091)
 2016-10-13 11:42:03 -07:00
Brahma Reddy Battula b60e545a08 10986. DFSAdmin should log detailed error message if any. Contributed by MingLiang Liu 
(cherry picked from commit 29aa11b1a252e007ed62fad362096ca43aa408af)
 2016-10-13 21:41:25 +05:30
Kihwal Lee a6197c1fa8 HDFS-11000. webhdfs PUT does not work if requests are routed to call queue. Contributed by Kihwal Lee. 
(cherry picked from commit 9454dc5e80)
 2016-10-13 08:48:33 -05:00
Akira Ajisaka 43cf0b2973 HDFS-11002. Fix broken attr/getfattr/setfattr links in ExtendedAttributes.md. Contributed by Mingliang Liu. 
(cherry picked from commit 901eca004d)
 2016-10-13 14:30:45 +09:00
Mingliang Liu b36aaa913c HDFS-10903. Replace config key literal strings with config key names II: hadoop hdfs. Contributed by Chen Liang 2016-10-12 17:26:11 -07:00
Kihwal Lee 5305a392c3 HDFS-10789. Route webhdfs through the RPC call queue. Contributed by Daryn Sharp and Rushabh S Shah. 
(cherry picked from commit 85cd06f663)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2016-10-12 15:22:51 -05:00
Mingliang Liu e341e5151a HDFS-10965. Add unit test for HDFS command 'dfsadmin -printTopology'. Contributed by Xiaobing Zhou 
(cherry picked from commit f340d286d441d78c847722b529939b78abf9d2f3)
 2016-10-11 19:59:36 -07:00
Xiaoyu Yao 2c77aca782 HDFS-10984. Expose nntop output as metrics. Contributed by Siddharth Wagle. 
(cherry picked from commit 61f0490a73)
 2016-10-11 15:56:12 -07:00
Wei-Chiu Chuang 402856a8a7 HDFS-10933. Refactor TestFsck. Contributed by Takanobu Asanuma. 2016-10-11 13:18:48 -07:00
Mingliang Liu 11e748394d HDFS-10988. Refactor TestBalancerBandwidth. Contributed by Brahma Reddy Battula 
(cherry picked from commit b963818621)
 2016-10-10 13:23:10 -07:00
Mingliang Liu 6c137ea94e HDFS-10972. Add unit test for HDFS command 'dfsadmin -getDatanodeInfo'. Contributed by Xiaobing Zhou 
(cherry picked from commit 3441c746b5)
 2016-10-10 11:45:43 -07:00
Xiao Chen 5c96ef365d HDFS-10797. Disk usage summary of snapshots causes renamed blocks to get counted twice. Contributed by Sean Mackrory. 
(cherry picked from commit 3ea027e970ec7c95a10e581435cdb6c3a6b64abb)
 2016-10-07 17:37:23 -07:00
Kihwal Lee 7f05306895 HDFS-10980. Optimize check for existence of parent directory. Contributed by Daryn Sharp. 
(cherry picked from commit e57fa81d95)
 2016-10-07 17:21:28 -05:00
Kihwal Lee 5ea87e4aeb HDFS-10979. Pass IIP for FSDirDeleteOp methods. Contributed by Daryn Sharp. 
(cherry picked from commit 3565c9af17)
 2016-10-07 14:35:28 -05:00
Kihwal Lee 76c03acc6d HDFS-10939. Reduce performance penalty of encryption zones. Contributed by Daryn Sharp. 2016-10-07 13:20:05 -05:00
Brahma Reddy Battula cddd35e1f0 HDFS-10969. Fix typos in hdfs-default.xml Contributed by Yiqun Lin 
(cherry picked from commit be3cb10f53)
 2016-10-07 22:19:21 +05:30
Kihwal Lee e3a9666d28 HDFS-10955. Pass IIP for FSDirAttr methods. Contributed by Daryn Sharp. 2016-10-06 16:32:12 -05:00