Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,092 Commits 326 Branches 371 Tags
Commit Graph

7797 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
a5d76a4965 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 20:54:16 -07:00
Lokesh Jain
461c624cc0 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 272b96d243383d9f50241d48cb070f638243bc9c)
 2019-06-20 20:23:31 -07:00
Takanobu Asanuma
6dffad028e HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:17:45 -07:00
Wei-Chiu Chuang
7e9b9e2a2e Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 36dc0810bf8032e4741e27a8f2b00e64dbdaef2a.
 2019-06-19 06:55:08 -07:00
Virajith Jalaparti
36dc0810bf HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
 2019-06-18 22:16:56 -07:00
Zsolt Venczel
7c00756aff HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
 2019-06-18 14:37:16 -07:00
Elek, Marton
1a99f97945 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
 2019-06-18 12:00:36 -07:00
Wei-Chiu Chuang
223a582eb8 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
 2019-06-18 00:34:38 -07:00
Weiwei Yang
f36cdc4d7f HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
 2019-06-17 21:46:58 -07:00
Santosh Marella
1ee18ebd26 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:26:39 -07:00
Wei-Chiu Chuang
412fbf12cb Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit fe46d1c79f46997711cded4b604243624665adfa.
 2019-06-17 16:25:49 -07:00
Wei-Chiu Chuang
0eb42e1e05 HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
 2019-06-17 15:18:00 -07:00
Santosh Marella
fe46d1c79f HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 15:15:58 -07:00
Inigo Goiri
335aebb9c4 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 
(cherry picked from commit 4f455290b15902e7e44c4b1a762bf915414b2bb6)
 2019-06-13 18:27:26 -07:00
Sean Mackrory
e0b3cbd221 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:53:40 -06:00
Stephen O'Donnell
71159f9540 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0f14526bfb7052f2a2e64543dc0fcbc)
 2019-06-12 10:31:03 -07:00
He Xiaoqiao
d7560c866e HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9244f3f3474d316255ac98717ed5719)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2019-06-10 17:26:42 -07:00
Hanisha Koneru
210480a23c HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40dd99068a3f3b155261b1dac6c67828)
 2019-06-10 13:46:53 -07:00
Kihwal Lee
056bb773ce HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
 2019-05-30 17:30:48 -05:00
Ayush Saxena
2cd7577a5a HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
 2019-05-29 20:58:11 -07:00
Takanobu Asanuma
a9a3450560 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:34:16 +09:00
Akira Ajisaka
855dc997d6
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:27:33 +09:00
Erik Krogen
c26cf22d6b HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f002f74cb4a0360b6682a1b6796d1598)
 2019-05-24 13:18:49 -07:00
Konstantin V Shvachko
9836bac35c HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:49:06 -07:00
Siyao Meng
a4f425d3ae HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9dff6eff819a832a5918d786675f42aa243a13e8)
 2019-05-22 13:24:58 +09:00
Siyao Meng
bba8971897 HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a92056783bad23a9abc6a264d71285d)
 2019-05-01 16:49:45 -07:00
Arpit Agarwal
db4c0b357d
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f87ad40d9c19ba160f706ffd0470a8d4)
 2019-04-29 14:54:30 -07:00
Puleya7
d724f74f6b HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d847986aa14a51cc2a3cc83151f22c43781)
 2019-04-17 21:55:45 +09:00
Akira Ajisaka
f020ac2c06
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f09502a4f07342dee457e47bb52b03441ae)
 2019-04-05 12:30:46 +09:00
Stephen O'Donnell
388f445dde HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
 2019-04-04 04:06:15 -07:00
Wei-Chiu Chuang
c8703dda07 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070a124234c77f16193ee925d32ca9a20)
 2019-04-03 11:01:44 -07:00
Gabor Bota
67cdf807a2
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8ecb83b666d4ed223fc270e1a46642c1)
 2019-04-01 13:52:19 +09:00
Takanobu Asanuma
162e9999c7 HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb48ca26a629d4d5f3f07e2858d09594)
 2019-03-27 03:28:54 +09:00
Inigo Goiri
4eb0497091 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631cb8a0f67bbc39b5b5a1b0a81ca3b3c)
 2019-03-15 13:58:03 -07:00
Erik Krogen
fec7c5f3eb HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae1da09ced735da36bf7d80a40c3fa1b)
 2019-03-13 13:19:18 -07:00
Shweta Yakkali
1ceefa726e HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020a538115b3e08f41ff88167033d2775)
 2019-03-12 16:05:55 -07:00
Stephen O'Donnell
a21e2e4dbc HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b38dccab25058dff1b8743d8a3f82734)
 2019-03-12 10:18:56 -07:00
Erik Krogen
6d076dd5e8 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3f74968c92751f0972746b012e72810)
 2019-03-07 08:42:41 -08:00
Wei-Chiu Chuang
e58ccca3ce HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587aa66ac99ec5f5b50827ec3e0a327613)
 2019-03-04 10:45:31 -08:00
Erik Krogen
af16db86d4 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:44 -08:00
Surendra Singh Lilhore
2e939515df HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f070bbfe65c736f6f3eca721b9d227f5)
 2019-02-21 20:47:50 +05:30
Surendra Singh Lilhore
b93b127956 HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 
(cherry picked from commit 41e18feda3f5ff924c87c4bed5b5cbbaecb19ae1)
 2019-02-20 17:12:58 +05:30
Surendra Singh Lilhore
f4b9ba230b HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 
(cherry picked from commit 29b411d5f065f177de05a69d2822836209f813c6)
 2019-02-13 23:51:36 +05:30
Surendra Singh Lilhore
e163bf2258 HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 35d4f32b32a3ba05332811caf3d245d9c3dcf1a5)
 2019-02-13 23:21:37 +05:30
Vinayakumar B
890736828b HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 21:58:59 +05:30
Surendra Singh Lilhore
2501fcd26b HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 4be87353e35a30d95d8847b09a1890b014bfc6bb)
 2019-02-07 16:52:53 -08:00
Kitti Nanasi
38541b2375 HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 49ddd8a6ed5b40d12defb0771b4c8b53d4ffde3f)
 2019-02-05 15:15:04 -08:00
Wangda Tan
649da5af04 Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa221a336ef0dbb1a17358b1ad6f9356)
 2019-02-05 14:05:51 -08:00
Steve Loughran
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f848345f6d34a2ac2d570d49e9787a0df6a)
 2019-02-04 17:10:19 +00:00
Masatake Iwasaki
a86893f875 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7bdf437713a8c6ca3c535c4b6d6992e)
 2019-01-29 02:28:18 +09:00