Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,188 Commits 326 Branches 371 Tags
Commit Graph

7322 Commits

Author SHA1 Message Date
Konstantin V Shvachko
1be8e187b1 HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-12 11:50:02 -07:00
Konstantin V Shvachko
1ad1c800a4 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-12 11:50:02 -07:00
Chen Liang
ee9977cc69 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-07-12 11:50:01 -07:00
Chao Sun
d96ddb3b87 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-07-12 11:50:01 -07:00
Erik Krogen
849a6c6f0d HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-07-12 11:49:58 -07:00
Erik Krogen
ac2e79330c HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-07-12 11:05:50 -07:00
Chao Sun
9840dae72f HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 11:05:07 -07:00
Konstantin V Shvachko
405f6b6fbf HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-07-12 11:05:07 -07:00
Erik Krogen
8e551708d8 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:05:04 -07:00
Erik Krogen
abb8614dcf HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:04:24 -07:00
Erik Krogen
c40ddf9b5b HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-07-12 11:03:32 -07:00
Plamen Jeliazkov
7d0c3ba700 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:31 -07:00
Erik Krogen
18e545f0be HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen
07a6510e6a HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen
b999e96c9a HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:30 -07:00
Plamen Jeliazkov
21a886a4ef HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:27 -07:00
Chao Sun
4cf63905d0 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-07-12 10:58:43 -07:00
Masatake Iwasaki
c6ac7f511c
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00dd843a1a6c9d8b616631cdcf24c00e82498dab)
 2019-07-12 10:14:49 +09:00
Weiwei Yang
9c3806cf1b HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-11 09:46:27 +08:00
Erik Krogen
a59553b5e6 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148ed7dec75810feddb22ad98e4670483)
(cherry-picked from 491dc7cc9d6836f34d2647b12dcefa86cc97a927)
 2019-07-08 15:20:29 -07:00
leosunli
7c7907f756 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c907294860a48f0d1676a31bda15795b7a6771a)
 2019-07-07 20:02:51 -07:00
Masatake Iwasaki
02ae1d6071 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec78c8dab9111f3ba2709cef20794b73a)
 2019-07-04 23:54:18 +09:00
Anu Engineer
1bae5c7024 HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee4a17714f31952dc6364c77a65b1b73)
(cherry picked from commit e7fce2104f28c48c817822bed9349d44b0694699)
 2019-06-24 21:20:00 -07:00
Inigo Goiri
90b88db35d HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42a3ef94d6bef160bc58d807f2faa104)
(cherry picked from commit 96371245357bda63b3ede10f37a37f5333a85d69)
 2019-06-24 21:17:04 -07:00
Inigo Goiri
aaf74ea5d7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6fddb7adf60dba846d1f7c8a13b8ec1)
 2019-06-24 19:03:26 -07:00
Santosh Marella
026cc2647d HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-23 20:27:01 -07:00
Wei-Chiu Chuang
529d0955f5 HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f4e29d10ef810386c89e6a7c8c61862)
(cherry picked from commit d5ecca1d4ca2b567167402bf8745fb2417bbf624)
 2019-06-21 18:27:50 -07:00
Wei-Chiu Chuang
4507c2f3a5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c52aba966cbbf1d8025ba165688ab9bb)
(cherry picked from commit 845d7611babac68a2b0caa6e4b8963f0c8332540)
 2019-06-21 18:18:36 -07:00
Wei-Chiu Chuang
f33bcbb687 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965b686ec5532add511c64d5f87c88865)
 2019-06-20 20:54:43 -07:00
Takanobu Asanuma
2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d20656433cdec76c2108d24ff3b935657c1e80)
 2019-06-20 20:18:00 -07:00
Wei-Chiu Chuang
58c1920472 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 49587e4df3e955466670384de867f690b557fe6e.
 2019-06-19 06:55:25 -07:00
Virajith Jalaparti
49587e4df3 HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
(cherry picked from commit 36dc0810bf8032e4741e27a8f2b00e64dbdaef2a)
 2019-06-18 22:17:08 -07:00
Zsolt Venczel
4ea3b04bf3 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff6bf8d5e1d05748ba32b679e397b53f)
 2019-06-18 14:38:05 -07:00
Elek, Marton
293f8bbd84 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
(cherry picked from commit 1a99f9794519faee5a196ab3d9cb91d55bfd0002)
 2019-06-18 12:00:48 -07:00
Wei-Chiu Chuang
29c4aded34 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
(cherry picked from commit 223a582eb839bb6e4298a2fe0243bc27d73fc977)
 2019-06-18 00:34:55 -07:00
Wei-Chiu Chuang
4362fcb629 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78dce2697ec94a22a4a3981fa5a3849e)
(cherry picked from commit f5428d4062c0a5e98461ea3b8420dc68555681f9)
 2019-06-17 22:48:07 -07:00
Weiwei Yang
3bdb661f35 HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
(cherry picked from commit f36cdc4d7f45d1a3f8edd2d804c38b0b39990410)
 2019-06-17 21:47:07 -07:00
Wei-Chiu Chuang
fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea8c75c134de43a3a925c76b253658e1)
(cherry picked from commit 45543ffd6af723bb83bd44a99f12165363c70abb)
 2019-06-17 20:20:08 -07:00
Akira Ajisaka
3d1d8623a1 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827381a7f87c01f0befc2475f9478e63b)
(cherry picked from commit f68099aa1316f34591c757bcf4198a3a3c7f691a)
 2019-06-17 17:04:07 -07:00
Wei-Chiu Chuang
b3e0430c3b HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05b780dab718bb3adfdd55f527dd5545)
 2019-06-17 15:18:39 -07:00
huzheng
b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a184405c3feca024a560807ea215b6858b9)
(cherry picked from commit 9b61ecfcbeeb1d3c320f775b43c68d3e39a271a3)
 2019-06-17 07:21:07 -07:00
Inigo Goiri
76a66998fe HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:28:38 -07:00
Sean Mackrory
fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Stephen O'Donnell
a9b8310584 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0f14526bfb7052f2a2e64543dc0fcbc)
(cherry picked from commit 71159f9540d5a2aac6fb904649639f75e457e94d)
 2019-06-12 10:31:21 -07:00
He Xiaoqiao
03ec4c4c7c HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9244f3f3474d316255ac98717ed5719)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit d7560c866ecbfcd533a43890522ccb8b732a5382)
 2019-06-10 17:27:48 -07:00
Hanisha Koneru
dd795edcc7 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40dd99068a3f3b155261b1dac6c67828)
(cherry picked from commit 210480a23cab052c5ea538fbfd9e50d888b8d3ff)
 2019-06-10 13:47:17 -07:00
Kihwal Lee
a935258e8c HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b6493d315c8910fbc06eaba96273ee335c)
 2019-05-30 17:31:30 -05:00
Ayush Saxena
90cc5ddd49 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f27e3e4f58ff1b5ef3e93efc56bbecbe)
(cherry picked from commit 2cd7577a5a58db42a14a659b85fbd475091d0cb2)
 2019-05-29 20:58:34 -07:00
Takanobu Asanuma
8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00