Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,557 Commits 326 Branches 371 Tags
Commit Graph

3246 Commits

Author SHA1 Message Date
Steve Loughran
f225b5514f HADOOP-13026 Should not wrap IOExceptions into a AuthenticationException in KerberosAuthenticator. Xuan Gong via stevel 2017-05-24 19:00:18 -07:00
Vinayakumar B
b6afb4a951 HDFS-6291. FSImage may be left unclosed in BootstrapStandby#doRun() ( Contributed by Sanghyun Yun) 
(cherry picked from commit e5e492a9631ff78302fccedcb64d7b64b9407991)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2017-05-24 18:47:31 -07:00
Konstantin V Shvachko
2b067895a1 Commit missing file TestBatchIbr for HDFS-9710 backport. 2017-05-24 12:21:48 -07:00
Konstantin V Shvachko
ac1b8ff78f HDFS-9710. DN can be configured to send block receipt IBRs in batches. Contributed by Tsz-Wo Nicholas Sze. 
Backport HDFS-11837 by Vinitha Reddy Gankidi

(cherry picked from commit 5b95971f8a6dee09d1143c6cf121afa22fa6c16e)
 2017-05-24 11:42:52 -07:00
Konstantin V Shvachko
6d657054eb Update CHANGES.txt for HDFS-11377. 2017-05-23 15:02:30 -07:00
Brahma Reddy Battula
7ae1bc3c20 HDFS-11864. Document Metrics to track usage of memory for writes. Contributed by Yiqun Lin. 
(cherry picked from commit 52661e0912a79d1e851afc2b46c941ce952ca63f)
 2017-05-24 00:04:25 +08:00
Brahma Reddy Battula
0b53e9f9af Update CHANGES.txt for HDFS-11849 2017-05-23 23:31:03 +08:00
Yiqun Lin
f46781261f HDFS-11377. Balancer hung due to no available mover threads. Contributed by yunjiong zhao. 
(cherry picked from commit 9cbbd1eae893b21212c9bc9e6745c6859317a667)
 2017-05-22 17:44:12 -07:00
Brahma Reddy Battula
cecedcf558 HDFS-11849. JournalNode startup failure exception should be logged in log file. Contributed By Surendra Singh Lilhore. 
(cherry picked from commit 9cab42cc797986081fef184748044f1790a4f039)
 2017-05-23 02:52:41 +08:00
Akira Ajisaka
8a0e5e3b70 HDFS-10715. NPE when applying AvailableSpaceBlockPlacementPolicy. Contributed by Guangbin Zhu. 
(cherry picked from commit ef432579a7763cc0e482fe049027c6e5325eb034)
(cherry picked from commit 68638661271e6e69c220b19fec7cbc7ae5e6208e)

# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/AvailableSpaceBlockPlacementPolicy.java
 2017-05-21 17:47:51 -07:00
Kihwal Lee
9440ec2771 HDFS-8131. Implement a space balanced block placement policy. Contributed by Liu Shaohui. 
(cherry picked from commit de30d66b2673d0344346fb985e786247ca682317)

# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 2017-05-21 17:19:54 -07:00
Konstantin V Shvachko
321f730ca8 HDFS-9412. getBlocks occupies FSLock and takes too long to complete. Contributed by He Tianyi. Backport HDFS-11855 by Vinitha Reddy Gankidi. 
(cherry picked from commit 67523ffcf491f4f2db5335899c00a174d0caaa9b)
(cherry picked from commit a6d0e9e56a7eef226c8a891cf741ab22d8205e1c)
 2017-05-21 13:56:44 -07:00
Konstantin V Shvachko
eecc68d4d8 HDFS-9726. Refactor IBR code to a new class. Contributed by Tsz-Wo Nicholas Sze. Backport HDFS-11839 by Vinitha Reddy Gankidi. 
(cherry picked from commit 494b6c7c4b5e0a861fcbdcfb89cfc4869247e710)
 2017-05-18 18:09:36 -07:00
Konstantin V Shvachko
2c3f6aedf1 HDFS-7990. IBR delete ack should not be delayed. Contributed by Daryn Sharp. Backport HDFS-11838 by Vinitha Gankidi. 
(cherry picked from commit 60882ab26d49f05cbf0686944af6559f86b3417d)
 2017-05-17 23:58:27 -07:00
Konstantin V Shvachko
0823fb72a9 HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing Zhao. Backport HDFS-11732 by Zhe Zhang. 2017-05-16 11:03:12 -07:00
Brahma Reddy Battula
3004358639 HDFS-8312. Trash does not descent into child directories to check for permissions. Contributed By Weiwei Yang via Eric Yang. 2017-05-16 23:27:57 +05:30
Brahma Reddy Battula
b95d63cdac HDFS-10987. Make Decommission less expensive when lot of blocks present. Contributed By Brahma Reddy Battula. 2017-05-16 10:50:51 +05:30
Vinayakumar B
b0b6de38c9 HDFS-11674. reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered (Contributed by Vinayakumar B) 2017-05-12 14:26:28 +05:30
Brahma Reddy Battula
764fd3ed49 HDFS-9902. Support different values of dfs.datanode.du.reserved per storage type. Contributed By Brahma Reddy Battula. 2017-05-11 10:22:59 +05:30
Akira Ajisaka
f052976ff1
HDFS-11795. Fix ASF License warnings in branch-2.7. Contributed by Yiqun Lin. 2017-05-10 14:17:00 -05:00
Brahma Reddy Battula
48c5e37abf HDFS-11373. Backport HDFS-11258 and HDFS-11272(File mtime change could not save to editlog). (Contributed by Jimmy Xiang, backported by Akira Ajisaka) 2017-05-10 11:06:57 +05:30
Akira Ajisaka
6599e57a79
HDFS-11766. Fix findbugs warning in branch-2.7. Contributed by Chen Liang. 2017-05-09 11:22:57 -05:00
Konstantin V Shvachko
c7c410898e Update CHANGES.txt for HDFS-11717 branch-2.7. 2017-05-02 22:30:18 -07:00
Erik Krogen
a00cef6079 HDFS-11717. Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 
(cherry picked from commit d9014bda93760f223789d2ec9f5e35f40de157d4)
(cherry picked from commit 9d6b873d3c249aa76420ab03ceefc6180295915f)
 2017-05-02 18:46:58 -07:00
Kihwal Lee
130a0e2156 HDFS-11714. Newly added NN storage directory won't get initialized and cause space exhaustion. Contributed by Kihwal Lee. 
Added a CHANGES.txt entry
(cherry picked from commit f77d138725662197ca0be874e9ec4dc430db2a4f)
 2017-05-01 17:36:58 -05:00
Kihwal Lee
139327d34b HDFS-11609. Some blocks can be permanently lost if nodes are decommissioned while dead. Contributed by Kihwal Lee. 2017-05-01 14:31:59 -05:00
Zhe Zhang
8ad9efbe14 HDFS-8873. Allow the directoryScanner to be rate-limited. Contributed by Daniel Templeton. 2017-04-27 15:04:10 -07:00
Konstantin V Shvachko
5e67eb7560 Update CHANGES.txt for HDFS-11384 branch-2.7. 2017-04-27 12:23:41 -07:00
Konstantin V Shvachko
cf5abf8a7c HDFS-11384. Balancer disperses getBlocks calls to avoid NameNode's rpc queue saturation. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 28eb2aabebd15c15a357d86e23ca407d3c85211c)
# Conflicts:
#	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Dispatcher.java
#	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
 2017-04-27 11:57:36 -07:00
Zhe Zhang
bec23593b5 HDFS-11709. StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully. Contributed by Erik Krogen. 
(cherry picked from commit d8a33098309f17dfb0e3a000934f68394de44bf7)
(cherry picked from commit fc46f398d9192e83c7197115589eca7805c49943)
(cherry picked from commit 70f18911c71c61f9b3918e195bfb9a99d22d230c)
 2017-04-27 09:56:14 -07:00
Brahma Reddy Battula
ac12063eeb HDFS-10455. Logging the username when deny the setOwner operation. Contributed by Rakesh R. 
(cherry picked from commit 1c5708f6ded6ba9fc007eed93c1f65ba44451241)
(cherry picked from commit c895f0893621fd6196c42ed46b97cb99cc2e9bee)
 2017-04-26 15:59:21 -07:00
Konstantin V Shvachko
54bbdb4c02 Update CHANGES.txt for HDFS-11634. 2017-04-18 13:20:21 -07:00
Konstantin V Shvachko
5a714fee43 HDFS-11634. Optimize BlockIterator when interating starts in the middle. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 8dfcd95d580bb090af7f40af0a57061518c18c8c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestGetBlocks.java
 2017-04-17 17:39:29 -07:00
Zhe Zhang
ccd24f0917 HDFS-11615. FSNamesystemLock metrics can be inaccurate due to millisecond precision. Contributed by Erik Krogen. 
(cherry picked from commit ad49098eb324e238d97db68d7239ed2c4d84afa0)
(cherry picked from commit 051ab778065d3659c5284b5d95d04156c4ee7ffa)
(cherry picked from commit 07047335a3bc12b719ca608742bef1e565151035)
 2017-04-17 16:55:24 -07:00
Arpit Agarwal
9f4585c95f HDFS-11608. HDFS write crashed with block size greater than 2 GB. Contributed by Xiaobing Zhou. 2017-04-11 11:43:25 -07:00
Arpit Agarwal
f2d1d3d159 Revert "HDFS-11551. Handle SlowDiskReport from DataNode at the NameNode. Contributed by Xiaobing Zhou." 
This reverts commit ac76dc10dde45ed00885bb96e89ed78015435790.
 2017-04-11 11:42:57 -07:00
Xiaobing Zhou
ac76dc10dd HDFS-11551. Handle SlowDiskReport from DataNode at the NameNode. Contributed by Xiaobing Zhou. 2017-04-11 11:16:36 -07:00
Xiaobing Zhou
ea38aa727a HDFS-11628. Clarify the behavior of HDFS Mover in documentation. Contributed by Xiaobing Zhou. 2017-04-05 15:34:40 -07:00
Masatake Iwasaki
0ece3c8dd9 HDFS-11486. Client close() should not fail fast if the last block is being decommissioned. Contributed by Wei-Chiu Chuang and Yiqun Lin. 
(cherry picked from commit ca3ede5fda60f317c24be8a378e7450ae581f0c5)
 2017-03-28 18:17:45 +09:00
Zhe Zhang
4cc53b51f6 HADOOP-9631. ViewFs should use underlying FileSystem's server side defaults. Contributed by Lohit Vijayarenu and Erik Krogen. 
(cherry picked from commit 59d69257a888347f0fb9c51bb000afc986b64f98)
(cherry picked from commit f98f4bb6ca45e6f807ca747d673690232e39da52)
(cherry picked from commit 8ca9915d0f833bff6dae75fdcd071b6ae79b6320)
 2017-03-22 22:48:04 -07:00
Wei-Chiu Chuang
830a602375 HDFS-11499. Decommissioning stuck because of failing recovery. Contributed by Lukas Majercak and Manoj Govindassamy. 
(cherry picked from commit 385d2cb777a0272ac20c62336c944fad295d5d12)

 Conflicts:
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java

(cherry picked from commit 60be2e5d8a1a6a8921c68f8b0f428b55152d05db)

 Conflicts:
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
 2017-03-13 13:45:12 -07:00
Zhe Zhang
33f4263345 HDFS-11466. Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms. Contributed by Andrew Wang. 
(cherry picked from commit d269b488a71a158d3ddcbdea96992abe29725c69)
(cherry picked from commit 5fb785e154b429beb0b2e392c39be91dae65bb5c)
(cherry picked from commit 265ddb20c3aa99c2d91e2a85e82106d4960f5a7b)
 2017-02-28 09:07:23 -08:00
Xiao Chen
c2a4ce7717 HDFS-11379. DFSInputStream may infinite loop requesting block locations. Contributed by Daryn Sharp. 2017-02-17 15:38:55 -08:00
Wei-Chiu Chuang
f133e0161c HDFS-11333. Print a user friendly error message when plugins are not found. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 859bd159ae554174200334b5eb1d7e8dbef958ad)
(cherry picked from commit a8531d5d52fb219c2adb5b6025cfc3fad264e8aa)
(cherry picked from commit 4c47cb68e88505d5fdc6830a4789e78a974cb9cf)
 2017-02-15 03:05:01 -08:00
Zhe Zhang
03a67e1e62 HDFS-10534. NameNode WebUI should display DataNode usage histogram. Contributed by Kai Sasaki. 2017-01-26 10:34:00 -08:00
Akira Ajisaka
ab62484240 HDFS-11352. Add missing entry in CHANGES.txt. 2017-01-24 14:57:21 +09:00
Akira Ajisaka
3acc96edbf HDFS-11352. Potential deadlock in NN when failing over. Contributed by Erik Krogen. 2017-01-24 14:55:27 +09:00
Akira Ajisaka
b0b3966851 HDFS-11290. TestFSNameSystemMBean should wait until JMX cache is cleared. Contributed by Erik Krogen. 
(cherry picked from commit b1a9ec856b572894e769f052aea2340fc3f23c78)
(cherry picked from commit db47bd15dd80a4a5727607832c16635cb0ec7726)
 2017-01-24 14:19:21 +09:00
Vinitha Reddy Gankidi
a6625e69a2 HDFS-10733. NameNode terminated after full GC thinking QJM is unresponsive. Contributed by Vinitha Gankidi. 2017-01-18 12:59:37 -08:00
Haohui Mai
977fb0c524 HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao. 2017-01-04 21:16:20 -08:00