Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,420 Commits 326 Branches 371 Tags
Commit Graph

4009 Commits

Author SHA1 Message Date
Jing Zhao
5bdee64bde HDFS-10710. In BlockManager#rescanPostponedMisreplicatedBlocks(), postponed misreplicated block counts should be retrieved within the NN lock protection.Contributed by GAO Rui. 
(cherry picked from commit f4ba5ff1d70ef92d59851c09c4bd4b43d6c04971)
(cherry picked from commit 180a26d9fb2b3f4a0632e1379f867020326c7702)
 2016-08-03 12:44:49 -07:00
Kihwal Lee
33a87ffe10 HDFS-742. A down DataNode makes Balancer to hang on repeatingly asking NameNode its partial block list. Contributed by Mit Desai. 
(cherry picked from commit 58db263e93daf08280e6a586a10cebd6122cf72a)
 2016-08-03 12:19:05 -05:00
Kihwal Lee
288f9ccde2 HDFS-5805. TestCheckpoint.testCheckpoint fails intermittently on branch-2. Contributed by Eric Badger. 
(cherry picked from commit 5e5b8793fba8e25aeba7a74878da4cf8e806f061)
 2016-08-02 15:43:11 -05:00
Konstantin V Shvachko
9b3b7703b1 Revert "HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi." 
This reverts commit 9ab153d7ae47c9998816950745b04824b3332608.
 2016-08-01 22:48:09 -07:00
Daryn Sharp
ecb143ed21 HDFS-10655. Fix path related byte array conversion bugs. (daryn) 
(cherry picked from commit 9f473cf903e586c556154abd56b3a3d820c6b028)
 2016-08-01 10:18:45 -05:00
Yongjun Zhang
cc066de22c HDFS-10703. HA NameNode Web UI should show last checkpoint time. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit 8d32bd884d53948728b77de14531b3be196f4bc7)
 2016-07-29 23:47:46 -07:00
Akira Ajisaka
e467f838bb HDFS-10691. FileDistribution fails in hdfs oiv command due to ArrayIndexOutOfBoundsException. Contributed by Yiqun Lin. 
(cherry picked from commit 204a2055b1b9270ae13ea03b7aeac62b65166efd)
(cherry picked from commit 0cff416c35528ed33e87f8401a49503a44f992b0)
 2016-07-29 15:41:17 +09:00
Xiaoyu Yao
ac510e5cd0 HDFS-10676. Add namenode metric to measure time spent in generating EDEKs. Contributed by Hanisha Koneru. 2016-07-28 16:16:44 -07:00
Akira Ajisaka
960caf624d HDFS-10696. TestHDFSCLI fails. Contributed by Kai Sasaki. 
(cherry picked from commit 414fbfab41470923eb82a21628709e51fd3a3f6e)
(cherry picked from commit ab49c2b39943eb5a3510d22c09ae5630232bdddc)
 2016-07-28 16:59:49 +09:00
Yongjun Zhang
71c49b1786 HDFS-10667. Report more accurate info about data corruption location. (Yuanbo Liu via Yongjun Zhang) 
(cherry picked from commit eb7ff0c9927131f4a797148b970a95a1abf7d847)
 2016-07-27 15:49:56 -07:00
Brahma Reddy Battula
5dc29ab0f2 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 096d3fa25cbbdda4fa2ad30ea5a76f33e2631c7d)
 2016-07-27 10:31:29 +05:30
Arpit Agarwal
45a6ee97b2 HDFS-10642. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. (Contributed by Mingliang Liu) 2016-07-26 12:45:06 -07:00
Wei-Chiu Chuang
53bf632702 HDFS-9937. Update dfsadmin command line help and HdfsQuotaAdminGuide. Contributed by Kai Sasaki. 
(cherry picked from commit 25f0a9b3ab136d44f70eb2805598e2df8325612a)
 2016-07-26 08:31:00 -07:00
Steve Loughran
7921ad949f Revert "HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu." 
This reverts commit 32468b1ba5fe5e00e92656a8bf1591114383d7b3.
 2016-07-26 14:31:02 +01:00
Masatake Iwasaki
6b5d546014 HDFS-10671. Fix typo in HdfsRollingUpgrade.md. Contributed by Yiqun Lin. 
(cherry picked from commit 59466b8c180716dda7aa670728580a88e54eb4d2)
 2016-07-26 19:52:23 +09:00
Brahma Reddy Battula
32468b1ba5 HDFS-10668. Fix intermittently failing UT TestDataNodeMXBean#testDataNodeMXBeanBlockCount. Contributed by Mingliang Liu. 
(cherry picked from commit 7cac7655fd84ac394250705b31e3927fe548e34a)
 2016-07-26 11:00:11 +05:30
Akira Ajisaka
e377e94f64 Revert "HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg." 
This reverts commit 91f47ab5f97989872339a934ea351bf68a077553.
 2016-07-26 11:06:10 +09:00
Vinitha Reddy Gankidi
9ab153d7ae HDFS-10301. Interleaving processing of storages from repeated block reports causes false zombie storage detection, removes valid blocks. Contributed by Vinitha Gankidi. 2016-07-25 18:59:42 -07:00
Jing Zhao
62800dc8b6 HDFS-10688. BPServiceActor may run into a tight loop for sending block report when hitting IOException. Contributed by Chen Liang. 
(cherry picked from commit 0cde9e12a7175e4d8bc4ccd5c36055b280d1fbd6)
(cherry picked from commit 30c772fc595b71d3b26a13bd0473570a84b5e807)
 2016-07-25 18:44:06 -07:00
Arpit Agarwal
00cbd3136e HDFS-10660. Expose storage policy apis via HDFSAdmin interface. (Contributed by Rakesh R) 2016-07-25 09:51:59 -07:00
Jing Zhao
54be36a6d6 HDFS-9894. Add unsetStoragePolicy API to FileContext/AbstractFileSystem and derivatives. Contributed by Xiaobing Zhou. 
(cherry picked from commit 7149cdb3c2d9dd390cd8668883cbe5db94090e0a)
 2016-07-25 09:51:59 -07:00
Arpit Agarwal
bbf711c030 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-07-25 09:51:59 -07:00
Xiao Chen
8559442ed1 HDFS-10225. DataNode hot swap drives should disallow storage type changes. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 132deb4cacc413a85a6af2f390ec79a76c91961c)
 2016-07-22 15:04:21 -07:00
Jing Zhao
6a7fe835f1 HDFS-10653. Optimize conversion from path string to components. Contributed by Daryn Sharp. 
(cherry picked from commit bd3dcf46e263b6e6aa3fca6a5d9936cc49e3280f)
 2016-07-21 11:29:27 -07:00
Akira Ajisaka
2f14d585c4 HDFS-10287. MiniDFSCluster should implement AutoCloseable. Contributed by Andras Bokor. 
(cherry picked from commit fcde6940e0cbdedb1105007e4857137ecdfa1284)
(cherry picked from commit 1222889f1a3b45df30c48a80d8c2bec7ad45ee43)
 2016-07-21 11:28:46 -07:00
Akira Ajisaka
91f47ab5f9 HDFS-10620. StringBuilder created and appended even if logging is disabled. Contributed by Staffan Friberg. 
(cherry picked from commit f7dabe3addf3f6eb32ea9b8ec1354fb442ce4222)
(cherry picked from commit 2685e16f381532d53aea413ce1c0a14181ee7c89)
 2016-07-19 15:19:02 -07:00
Akira Ajisaka
d8181a8cb8 HDFS-10603. Fix flaky tests in org.apache.hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot. Contributed by Yiqun Lin. 
(cherry picked from commit fc570b55b9d1d1d57329eca223f443e2c86f34b7)
(cherry picked from commit c92601e9814a041800e466fabdcc3c44309364c4)
 2016-07-19 13:52:17 -07:00
Jitendra Pandey
a363277be5 HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:32:20 -07:00
Akira Ajisaka
83126ac90d HDFS-10628. Log HDFS Balancer exit message to its own log. Contributed by Jiayi Zhou. 
(cherry picked from commit c48e9d608c1daf7826f2e738c0a67d8eab79d8de)
(cherry picked from commit 617149837531062e4e1ee9eeb50a324766b14c60)
 2016-07-15 11:43:54 -07:00
Akira Ajisaka
6a04d9ab63 HADOOP-13351. TestDFSClientSocketSize buffer size tests are flaky. Contributed by Aaron Fabbri and Mingliang Liu. 
(cherry picked from commit 5537c6b23430285ebee33c6d9b69d3ec1e9b17b1)
(cherry picked from commit c9f5348bbfca664c6ab4c13f40cae0a2c5754da1)
 2016-07-14 14:42:55 -07:00
Zhe Zhang
af94dfaa53 HDFS-10544. Balancer doesn't work with IPFailoverProxyProvider. 
(cherry picked from commit 087290e6b1cb1082646d966b65494082712ebe3e)
(cherry picked from commit 5b4e916b3cae2540267d48716fd9240dfc055288)
 2016-07-12 23:28:16 -07:00
Jitendra Pandey
35108025e7 HDFS-10579. HDFS web interfaces lack configs for X-FRAME-OPTIONS protection. Contributed by Anu Engineer. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
 2016-07-12 14:38:34 -07:00
Yongjun Zhang
96e68e722f HDFS-10512. VolumeScanner may terminate due to NPE in DataNode.reportBadBlocks. Contributed by Wei-Chiu Chuang and Yiqun Lin. 
(cherry picked from commit da6f1b88dd47e22b24d44f6fc8bbee73e85746f7)
 2016-07-08 22:20:27 -07:00
Jitendra Pandey
9c7ce389c7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:04:47 -07:00
Uma Maheswara Rao G
097022e172 HDFS-10592:Fix intermittent test failure of TestNameNodeResourceChecker#testCheckThatNameNodeResourceMonitorIsRunning. Contributed by Rakesh R 
(cherry picked from commit ab092c56c251f27a9426e3c69a784417fdf9a26e)
 2016-07-07 14:02:25 -07:00
Andrew Wang
f826d92909 HDFS-10488. Update WebHDFS documentation regarding CREATE and MKDIR default permissions. Contributed by Wellington Chevreuil. 
(cherry picked from commit 647b084738b26dcf0c0c4afa4ec3039e6aca2d61)
 2016-07-07 10:37:16 -07:00
Akira Ajisaka
4fba09d7e8 HDFS-10582. Change deprecated configuration fs.checkpoint.dir to dfs.namenode.checkpoint.dir in HDFS Commands Doc. Contributed by Pan Yuxuan. 
(cherry picked from commit a3f93beaa19d9acb7b1db2958102d0004b0eee46)
(cherry picked from commit 8a9440298db2c3943cbe399c68e247adeefa28b0)
 2016-07-06 17:54:44 -07:00
Akira Ajisaka
4b263b30c2 HDFS-10589. Javadoc for HAState#HAState and HAState#setStateInternal contains non-existent params. Contributed by shenyinjie. 
(cherry picked from commit 8b4b5259d5088360476b4582c4cf7b0e7973eff3)
(cherry picked from commit 0401adcaca5f2e9680bc2dc3602af8696d204f21)
 2016-07-04 07:15:54 -07:00
Jitendra Pandey
bc7fd76a1f HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:37:32 -07:00
Vinayakumar B
f44d7782f6 HDFS-10440. Improve DataNode web UI (Contributed by Weiwei Yang) 
(cherry picked from commit 2a0082c51da7cbe2770eddb5f72cd7f8d72fa5f6)
(cherry picked from commit 1e347631817d882353bfb91d68f109cb8232e8c4)
 2016-06-28 16:55:57 +05:30
Akira Ajisaka
700dacf8c4 HDFS-7597. DelegationTokenIdentifier should cache the TokenIdentifier to UGI mapping. Contributed by Daryn Sharp, Bob Hansen, and Xiao Chen. 
(cherry picked from commit d433b16ce6d74f1a44bc29446c74b1cb5f8a10fa)
(cherry picked from commit ad50a64ded1ba0c32d7bc89e5100f3b4d86a790c)
 2016-06-22 11:02:48 +09:00
Brahma Reddy Battula
aa8f4cc48c HDFS-9530. ReservedSpace is not cleared for abandoned Blocks (Contributed by Brahma Reddy Battula) 
(cherry picked from commit f2ac132d6a21c215093b7f87acf2843ac8123716)
(cherry picked from commit ee0f389ecd0c479aa38dae7ee6fd02ed66a64e4e)
 2016-06-21 15:55:57 +05:30
Colin Patrick Mccabe
9b51420ea5 HDFS-10448. CacheManager#addInternal tracks bytesNeeded incorrectly when dealing with replication factors other than 1 (Yiqun Lin via cmccabe) 
(cherry picked from commit 46f1602e896273b308fbd5df6c75f6c142828227)
(cherry picked from commit 4e11f33ccc849355eaeb0694dbee702e34340a74)
 2016-06-20 18:32:00 -07:00
Jitendra Pandey
29c3ee9744 HDFS-10538. Remove AsyncDistributedFileSystem. Contributed by Xiaobing Zhou and Jitendra Pandey. 2016-06-20 18:05:32 -07:00
Jitendra Pandey
2f95d22b17 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:31:54 -07:00
Ravi Prakash
588ea6d431 HDFS-10220. A large number of expired leases can make namenode unresponsive and cause failover (Nicolas Fraison via raviprak) 
(cherry picked from commit ae047655f4355288406cd5396fb4e3ea7c307b14)
(cherry picked from commit 9c5f7f290eb05808fe89835a15bee0947b91d1a0)
 2016-06-20 14:38:05 -07:00
Brahma Reddy Battula
9d27530c36 HDFS-10474. hftp copy fails when file name with Chinese+special char in branch-2 (Contributed by Brahma Reddy Battula) 
(cherry picked from commit 3f27f405039926bbee4b36a60d6cecdb6322fff5)
 2016-06-20 12:40:11 +05:30
Andrew Wang
3f2d2cd0a1 Fix hadoop-hdfs-native-client compilation on Windows. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 3ff0510ffdc20e44f06aec591782f8875b686327)
(cherry picked from commit 94759470b8a922d410797b8b76e055fd2fc6dd69)
 2016-06-17 15:55:43 -07:00
Kihwal Lee
edd5373095 HDFS-10493. Add links to datanode web UI in namenode datanodes page. Contributed by Weiwei Yang. 
(cherry picked from commit 280069510b71d81a150b2b28d9fa987891d82774)
 2016-06-17 11:13:12 -05:00
Akira Ajisaka
242263c56d HADOOP-12943. Add -w -r options in dfs -test command. Contributed by Weiwei Yang. 
(cherry picked from commit 09e82acaf9a6d7663bc51bbca0cdeca4b582b535)
(cherry picked from commit a36aa920f437f9e21772e8067e5db86fbffe7895)
 2016-06-17 16:21:31 +09:00