Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,086 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7250 Commits

Author SHA1 Message Date
Anu Engineer 1bae5c7024 HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee)
(cherry picked from commit e7fce2104f)
 2019-06-24 21:20:00 -07:00
Inigo Goiri 90b88db35d HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42)
(cherry picked from commit 9637124535)
 2019-06-24 21:17:04 -07:00
Inigo Goiri aaf74ea5d7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6)
 2019-06-24 19:03:26 -07:00
Santosh Marella 026cc2647d HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-23 20:27:01 -07:00
Wei-Chiu Chuang 529d0955f5 HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f)
(cherry picked from commit d5ecca1d4c)
 2019-06-21 18:27:50 -07:00
Wei-Chiu Chuang 4507c2f3a5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c5)
(cherry picked from commit 845d7611ba)
 2019-06-21 18:18:36 -07:00
Wei-Chiu Chuang f33bcbb687 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965)
 2019-06-20 20:54:43 -07:00
Takanobu Asanuma 2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643)
 2019-06-20 20:18:00 -07:00
Wei-Chiu Chuang 58c1920472 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 49587e4df3.
 2019-06-19 06:55:25 -07:00
Virajith Jalaparti 49587e4df3 HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
(cherry picked from commit 36dc0810bf)
 2019-06-18 22:17:08 -07:00
Zsolt Venczel 4ea3b04bf3 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff)
 2019-06-18 14:38:05 -07:00
Elek, Marton 293f8bbd84 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
(cherry picked from commit 1a99f97945)
 2019-06-18 12:00:48 -07:00
Wei-Chiu Chuang 29c4aded34 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
(cherry picked from commit 223a582eb8)
 2019-06-18 00:34:55 -07:00
Wei-Chiu Chuang 4362fcb629 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78)
(cherry picked from commit f5428d4062c0a5e98461ea3b8420dc68555681f9)
 2019-06-17 22:48:07 -07:00
Weiwei Yang 3bdb661f35 HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
(cherry picked from commit f36cdc4d7f)
 2019-06-17 21:47:07 -07:00
Wei-Chiu Chuang fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea)
(cherry picked from commit 45543ffd6a)
 2019-06-17 20:20:08 -07:00
Akira Ajisaka 3d1d8623a1 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827)
(cherry picked from commit f68099aa13)
 2019-06-17 17:04:07 -07:00
Wei-Chiu Chuang b3e0430c3b HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05)
 2019-06-17 15:18:39 -07:00
huzheng b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a1844)
(cherry picked from commit 9b61ecfcbe)
 2019-06-17 07:21:07 -07:00
Inigo Goiri 76a66998fe HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:28:38 -07:00
Sean Mackrory fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Stephen O'Donnell a9b8310584 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0)
(cherry picked from commit 71159f9540)
 2019-06-12 10:31:21 -07:00
He Xiaoqiao 03ec4c4c7c HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit d7560c866e)
 2019-06-10 17:27:48 -07:00
Hanisha Koneru dd795edcc7 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40)
(cherry picked from commit 210480a23c)
 2019-06-10 13:47:17 -07:00
Kihwal Lee a935258e8c HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b649)
 2019-05-30 17:31:30 -05:00
Ayush Saxena 90cc5ddd49 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f2)
(cherry picked from commit 2cd7577a5a)
 2019-05-29 20:58:34 -07:00
Takanobu Asanuma 8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Erik Krogen 9295e95174 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f0)
(cherry-picked from c26cf22d6b)
 2019-05-24 13:19:39 -07:00
Konstantin V Shvachko 799ca572f7 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:35:32 -07:00
Siyao Meng 9196c84c9f HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9dff6eff81)
 2019-05-22 13:25:42 +09:00
Srinivasu Majeti e599e3b546 HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3e5e5b028a)
(cherry picked from commit 85f3664bb8)
 2019-05-17 19:22:03 +02:00
Siyao Meng 86fb5e1c8e HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a)
(cherry picked from commit 0550aa32f1a4b1809c234f8ed785db08b2e33d13)
 2019-05-01 16:49:36 -07:00
Arpit Agarwal 7b51edbab9
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f8)
 2019-04-29 14:54:11 -07:00
Puleya7 546ed4937c HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479)
 2019-04-17 21:57:06 +09:00
Akira Ajisaka a70076b3a5
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:31:14 +09:00
Stephen O'Donnell d9899015eb HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
(cherry picked from commit 388f445dde)
 2019-04-04 04:06:33 -07:00
Wei-Chiu Chuang 084fb9de22 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070)
(cherry picked from commit c8703dda07)
 2019-04-03 12:41:07 -07:00
Gabor Bota f01d592f3b
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e)
 2019-04-01 13:52:44 +09:00
Takanobu Asanuma 1fc74178eb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb)
 2019-03-27 03:29:42 +09:00
Inigo Goiri 80732146c6 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 13:58:39 -07:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
 2019-03-13 13:20:51 -07:00
Shweta Yakkali a324db2cce HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020)
(cherry picked from commit 1ceefa726e)
 2019-03-12 16:06:16 -07:00
Stephen O'Donnell 327e57ae6e HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3)
(cherry picked from commit a21e2e4dbc)
 2019-03-12 10:19:37 -07:00
Erik Krogen b0b8000240 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
(cherry-picked from commit 6d076dd5e8)
 2019-03-07 08:43:39 -08:00
Wei-Chiu Chuang d951497f57 HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
(cherry picked from commit e58ccca3ce)
 2019-03-04 10:46:00 -08:00
Erik Krogen b7eb9116eb HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:54 -08:00
Surendra Singh Lilhore bd9e2826fe HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 
(cherry picked from commit 29b411d5f0)
 2019-02-13 23:56:34 +05:30
Surendra Singh Lilhore 3bcdff8fac HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 35d4f32b32)
 2019-02-13 23:26:14 +05:30
Vinayakumar B 1f3e737407 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 22:35:55 +05:30