Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,050 Commits 326 Branches 371 Tags
Commit Graph

5086 Commits

Author SHA1 Message Date
Masatake Iwasaki
e61e52805f HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec78c8dab9111f3ba2709cef20794b73a)
 2019-07-04 23:55:13 +09:00
stack
bb7e143d34 HDFS-14585 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch2.9 
Reapply but include JIRA # in the commit message this time.

Revert "Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9""

This reverts commit 1dd54163fcb40b60efc7ca153d443129e424df7d.
 2019-07-01 11:08:05 -07:00
stack
1dd54163fc Revert "Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9" 
This reverts commit bccb0fd846b7cc69e0d83140193eaa7fc0ac081d.
 2019-07-01 11:07:43 -07:00
sunlisheng
bccb0fd846 Backport HDFS-8901 Use ByteBuffer in DFSInputStream#read to branch-2.9 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-07-01 10:59:58 -07:00
Inigo Goiri
0272480e9f HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42a3ef94d6bef160bc58d807f2faa104)
(cherry picked from commit 96371245357bda63b3ede10f37a37f5333a85d69)
(cherry picked from commit 90b88db35d42f2eab4da7f192a5fb99d9c834abb)
(cherry picked from commit 9daa45f646d9eddda21c087c23e6d1498f98c055)
 2019-06-24 21:17:38 -07:00
Inigo Goiri
1373889ac2 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6fddb7adf60dba846d1f7c8a13b8ec1)
 2019-06-24 19:04:33 -07:00
Wei-Chiu Chuang
bf63475e7a HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 21:06:06 -07:00
Wei-Chiu Chuang
f876e82abc HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-20 19:36:44 -07:00
Zsolt Venczel
a68de43957 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)
(cherry picked from commit 4ea3b04bf36af56612c6b47d9ff8aa49cbbbcc9e)
(cherry picked from commit 76faa41f18f1cb20a6469f53b475f52b662f8107)
 2019-06-18 14:38:42 -07:00
Kitti Nanasi
abd11f9db1 HDFS-13770. dfsadmin -report does not always decrease "missing blocks (with replication factor 1)" metrics when file is deleted. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 00:25:53 -07:00
Wei-Chiu Chuang
ce49f8a5df HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
(cherry picked from commit fc8983a5b8c94746d3e58cc203974aa09c1b640b)
(cherry picked from commit 79f8e4d5bc3afaff2f352b918e96a6ef3c2abcce)
 2019-06-17 20:20:34 -07:00
Akira Ajisaka
73a59b2c42 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)
(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)
(cherry picked from commit 3d1d8623a11b72c7939d6500537be5fd6565d4d6)
(cherry picked from commit 010a99104b53a23826f8f1f73d0125cb26f2adca)
 2019-06-17 17:04:38 -07:00
huzheng
8b90c69357 HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
(cherry picked from commit b082628e5a39fc1bc9d6094beb9c253cc096c701)
(cherry picked from commit 2b9c4fccf4a8930bfff99ab4928b9b3b4980d3b2)
 2019-06-17 07:22:01 -07:00
Kihwal Lee
500dc05924 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
(cherry picked from commit fd8b8c7494d19467485423ce2f3f7459a4fddc29)
 2019-05-30 17:37:38 -05:00
Ayush Saxena
c63049dfd4 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)
(cherry picked from commit 90cc5ddd4957fcb27ef5a1e7ee1a67c4eca9a3f5)
(cherry picked from commit 942c5b04f2f7bc33595c4df5c66104ce3c4c2c69)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2019-05-29 21:11:15 -07:00
Siyao Meng
cecba551aa HDFS-14514. Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2. Contributed by Siyao Meng, Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Stephen O'Donnell <sodonnell@cloudera.com>
 2019-05-29 17:28:49 -07:00
Takanobu Asanuma
e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka
cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Erik Krogen
f024d27cd9 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f002f74cb4a0360b6682a1b6796d1598)
(cherry-picked from c26cf22d6b3949a4798d8133669b451660a0f050)
(cherry-picked from 9295e95174ad8627874b81a9d43183384a81fb16)
 2019-05-24 13:39:52 -07:00
Konstantin V Shvachko
2afc770443 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:43:12 -07:00
Wei-Chiu Chuang
380289a167 HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge, Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Siyao Meng <smeng@cloudera.com>
 2019-05-24 18:36:53 +02:00
Puleya7
8579ab13cb HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d847986aa14a51cc2a3cc83151f22c43781)
 2019-04-17 21:58:43 +09:00
Wei-Chiu Chuang
8af767b069 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-10 14:36:07 -07:00
Dinesh Chitlangia
66ac003128 HDFS-14414. Clean up findbugs warning in branch-2. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-08 10:47:13 -07:00
Chen Liang
dce76e0355 HDFS-14415. Backport HDFS-13799 to branch-2. Contributed by Chao Sun. 2019-04-05 11:27:37 -07:00
Akira Ajisaka
2abc7927d1
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f09502a4f07342dee457e47bb52b03441ae)
 2019-04-05 12:32:07 +09:00
Chen Liang
a5bca767f7 HDFS-14397. Backport HADOOP-15684 to branch-2. Contributed by Chao Sun. 2019-04-03 17:32:56 -07:00
Chen Liang
d6083e9920 HDFS-14399. Backport HDFS-10536 to branch-2. Contributed by Chao Sun. 2019-04-01 12:47:02 -07:00
Chen Liang
812e6d5c81 HDFS-14392. Backport HDFS-9787 to branch-2. Contributed by Chao Sun. 2019-03-28 19:07:58 -07:00
Chen Liang
99e41293ee HDFS-14391. Backport HDFS-9659 to branch-2. Contributed by Chao Sun. 2019-03-28 10:39:45 -07:00
Chen Liang
7935485eec HDFS-14205. Backport HDFS-6440 to branch-2. Contributed by Chao Sun. 2019-03-26 12:12:14 -07:00
Inigo Goiri
525f4e52f3 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631cb8a0f67bbc39b5b5a1b0a81ca3b3c)
 2019-03-15 14:02:20 -07:00
Wei-Chiu Chuang
d71cfe1461 HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587aa66ac99ec5f5b50827ec3e0a327613)
(cherry picked from commit e58ccca3ce131c955ceb115cd0b75e452eea828b)
(cherry picked from commit d951497f57cf6556b0916cad08576481dfe2ae06)
(cherry picked from commit e23a448e0e32ce5139b76d47c73fce621ccb66bd)
 2019-03-04 11:30:56 -08:00
Masatake Iwasaki
9688ed3b48 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7bdf437713a8c6ca3c535c4b6d6992e)
 2019-01-29 01:45:30 +09:00
Wei-Chiu Chuang
10cc02f643 HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang. 
(cherry picked from commit e9a005dfcf4d2249235aeff4da85b321dddc1ca9)
(cherry picked from commit c48640ffbd3852b75f898325ae0b3ea0ba864578)
(cherry picked from commit f24684ca9cd70326e24376a33cda937c98295af0)
(cherry picked from commit 054594c6145711c60f3cb848f08e515c4179f0c4)
 2018-12-28 10:46:38 -08:00
Yiqun Lin
50e11dc807 HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng. 
(cherry picked from commit 0e36e935d909862401890d0a5410204504f48b31)
 2018-11-30 11:34:28 +08:00
Akira Ajisaka
6cae964f6c
Add 2.9.2 release notes and changes documents. 
(cherry picked from commit 1a00b4e325146988375c9ce5b11016c45f059a4e)
 2018-11-20 14:30:09 +09:00
Vinayakumar B
c3f0681f99 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 
(cherry picked from commit 886b935630bf2d46f4c6773de013c093d2e44c38)
 2018-11-16 18:02:10 +05:30
Vinayakumar B
bc4098eb07 HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena. 
(cherry picked from commit 8571507efa3af3e4a5213a7e24c65e26162eac4d)
 2018-11-16 17:43:49 +05:30
Inigo Goiri
d9b3b58389 HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 2018-11-06 12:16:46 -08:00
Inigo Goiri
98075d9224 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 
(cherry picked from commit f3296501e09fa7f1e81548dfcefa56f20fe337ca)
(cherry picked from commit 9bf4f3d61403b06f3d6a092dacab382c7c131e19)
 2018-11-05 16:56:37 -08:00
Akira Ajisaka
c19baf19da
Addendum: Fix potential FSImage corruption. Contributed by Xiao Chen. 
(cherry picked from commit dd70b1f53836961029767660398819a3b04eec58)
 2018-10-31 17:04:06 +09:00
Vinayakumar B
e212d7d81f Fix potential FSImage corruption. Contributed by Daryn Sharp. 2018-10-15 16:04:34 +05:30
Inigo Goiri
0ba621029c HDFS-13802. RBF: Remove FSCK from Router Web UI. Contributed by Fei Hui. 
(cherry picked from commit f63ee083db7c261207b726f0f44be2e5a92124f3)
 2018-10-12 10:12:40 -07:00
Yiqun Lin
c632c6e6e9 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 2018-10-09 10:20:37 +08:00
Inigo Goiri
5539dd97d2 HDFS-13964. RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice. Contributed by Ayush Saxena. 
(cherry picked from commit 3bfff0f79e7246cbb2989c720c14cfa2566a9d1a)
 2018-10-06 10:43:36 -07:00
Brahma Reddy Battula
3d67080a45 HDFS-13790. RBF: Move ClientProtocol APIs to its own module Contributed by Chao Sun. 2018-09-26 22:46:43 +05:30
Chen Liang
3a6ad9cd3d HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-20 10:00:13 -07:00
Arpit Agarwal
0dd686114f HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b1511f8f46866abe4eeec820f4191cc295)
 2018-09-19 13:49:16 -07:00
Inigo Goiri
3e476dcae8 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0f49772d001c1fdb385110c705a622a)
 2018-09-19 09:32:35 -07:00