Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,085 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7795 Commits

Author SHA1 Message Date
Erik Krogen 491dc7cc9d HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148)
 2019-07-08 15:15:41 -07:00
Wei-Chiu Chuang 29b9049bde HDFS-12862. CacheDirective becomes invalid when NN restart or failover. Contributed by Wang XL. 
(cherry picked from commit ec851e4db2)
 2019-07-08 15:25:10 +08:00
leosunli adec307251 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c90729486)
 2019-07-07 20:02:08 -07:00
Masatake Iwasaki ee81a64da7 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec7)
 2019-07-04 23:53:30 +09:00
Weiwei Yang fa8b8edbac HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 
(cherry picked from commit 729cb3aefe)
 2019-07-04 18:06:19 +08:00
Wei-Chiu Chuang 66920e6d37 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-06-28 14:28:35 -07:00
Erik Krogen e53cee0c14 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-06-28 14:28:28 -07:00
Erik Krogen 3c6b74a6a4 HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:28:17 -07:00
Erik Krogen e1e4056353 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-06-28 14:28:10 -07:00
Erik Krogen 73d56b9d2a HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-06-28 14:27:54 -07:00
Chao Sun 720eed9ef9 HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-06-28 14:23:14 -07:00
Konstantin V Shvachko 2be93bd54b HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-06-28 14:23:14 -07:00
Chao Sun eb7d7fa5e2 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-06-28 14:23:13 -07:00
Chao Sun 25913f1c34 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-06-28 14:23:13 -07:00
Erik Krogen e917ac2854 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-06-28 14:23:12 -07:00
Erik Krogen c1c061d767 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-06-28 14:23:12 -07:00
Erik Krogen f1c5b247b9 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-06-28 14:23:11 -07:00
Chao Sun 45d6834d4c HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-06-28 14:23:10 -07:00
Chen Liang 3d3686a619 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-06-28 14:23:09 -07:00
Chao Sun cc70e81931 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-06-28 14:23:09 -07:00
Konstantin V Shvachko 8eef380af7 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-06-28 14:23:08 -07:00
Konstantin V Shvachko dea7e1a239 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-06-28 14:23:08 -07:00
Konstantin V Shvachko c2bfaf4d27 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-06-28 14:23:07 -07:00
Chen Liang 3998bab3da HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-06-28 14:23:05 -07:00
Erik Krogen a856eea78a HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-06-28 14:22:12 -07:00
Erik Krogen 556b7a840b HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-06-28 14:22:12 -07:00
Konstantin V Shvachko 0de06c010a HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2019-06-28 14:22:12 -07:00
Chen Liang 4c4ca86286 HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-06-28 14:22:11 -07:00
Erik Krogen afeaeadd4f HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-06-28 14:22:11 -07:00
Konstantin V Shvachko 0c9d846c60 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-06-28 14:22:10 -07:00
Chen Liang e2337bfc57 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-06-28 14:22:10 -07:00
Chao Sun 6fe755f0c7 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-06-28 14:22:09 -07:00
Erik Krogen e3f3fb1c18 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-06-28 14:22:09 -07:00
Erik Krogen ea402854aa HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-06-28 14:22:09 -07:00
Chao Sun a97f00e1c6 HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-06-28 14:22:08 -07:00
Konstantin V Shvachko 1f5852fb12 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-06-28 14:22:08 -07:00
Erik Krogen ac3eeeeb3c HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-06-28 14:22:07 -07:00
Erik Krogen b13110a6f0 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-06-28 14:22:07 -07:00
Erik Krogen 174c41c68f HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-06-28 14:22:07 -07:00
Plamen Jeliazkov a3521c53fe HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-06-28 14:22:03 -07:00
Erik Krogen 49d7bb6a92 HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-06-28 14:20:01 -07:00
Erik Krogen c0ca2bb853 HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-06-28 14:20:01 -07:00
Plamen Jeliazkov 1eeca2d9fb HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-06-28 14:20:00 -07:00
Chao Sun 3932ac4ef7 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-06-28 14:20:00 -07:00
Anu Engineer e7fce2104f HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee)
 2019-06-24 21:19:50 -07:00
Inigo Goiri 9637124535 HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42)
 2019-06-24 21:16:46 -07:00
Wei-Chiu Chuang d5ecca1d4c HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f)
 2019-06-21 18:27:38 -07:00
Wei-Chiu Chuang 845d7611ba HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c5)
 2019-06-21 18:18:08 -07:00
Wei-Chiu Chuang a5d76a4965 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 20:54:16 -07:00
Lokesh Jain 461c624cc0 HDFS-13893. DiskBalancer: no validations for Disk balancer commands. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 272b96d243)
 2019-06-20 20:23:31 -07:00
Takanobu Asanuma 6dffad028e HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643)
 2019-06-20 20:17:45 -07:00
Wei-Chiu Chuang 7e9b9e2a2e Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 36dc0810bf.
 2019-06-19 06:55:08 -07:00
Virajith Jalaparti 36dc0810bf HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
 2019-06-18 22:16:56 -07:00
Zsolt Venczel 7c00756aff HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
 2019-06-18 14:37:16 -07:00
Elek, Marton 1a99f97945 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
 2019-06-18 12:00:36 -07:00
Wei-Chiu Chuang 223a582eb8 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
 2019-06-18 00:34:38 -07:00
Weiwei Yang f36cdc4d7f HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
 2019-06-17 21:46:58 -07:00
Santosh Marella 1ee18ebd26 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-17 16:26:39 -07:00
Wei-Chiu Chuang 412fbf12cb Revert "HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao." 
This reverts commit fe46d1c79f.
 2019-06-17 16:25:49 -07:00
Wei-Chiu Chuang 0eb42e1e05 HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
 2019-06-17 15:18:00 -07:00
Santosh Marella fe46d1c79f HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-14 15:15:58 -07:00
Inigo Goiri 335aebb9c4 HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 
(cherry picked from commit 4f455290b1)
 2019-06-13 18:27:26 -07:00
Sean Mackrory e0b3cbd221 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:53:40 -06:00
Stephen O'Donnell 71159f9540 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0)
 2019-06-12 10:31:03 -07:00
He Xiaoqiao d7560c866e HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2019-06-10 17:26:42 -07:00
Hanisha Koneru 210480a23c HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40)
 2019-06-10 13:46:53 -07:00
Kihwal Lee 056bb773ce HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b649)
 2019-05-30 17:30:48 -05:00
Ayush Saxena 2cd7577a5a HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f2)
 2019-05-29 20:58:11 -07:00
Takanobu Asanuma a9a3450560 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:34:16 +09:00
Akira Ajisaka 855dc997d6
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:27:33 +09:00
Erik Krogen c26cf22d6b HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f0)
 2019-05-24 13:18:49 -07:00
Konstantin V Shvachko 9836bac35c HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:49:06 -07:00
Siyao Meng a4f425d3ae HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9dff6eff81)
 2019-05-22 13:24:58 +09:00
Siyao Meng bba8971897 HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a)
 2019-05-01 16:49:45 -07:00
Arpit Agarwal db4c0b357d
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f8)
 2019-04-29 14:54:30 -07:00
Puleya7 d724f74f6b HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479)
 2019-04-17 21:55:45 +09:00
Akira Ajisaka f020ac2c06
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:30:46 +09:00
Stephen O'Donnell 388f445dde HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
 2019-04-04 04:06:15 -07:00
Wei-Chiu Chuang c8703dda07 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070)
 2019-04-03 11:01:44 -07:00
Gabor Bota 67cdf807a2
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e)
 2019-04-01 13:52:19 +09:00
Takanobu Asanuma 162e9999c7 HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb)
 2019-03-27 03:28:54 +09:00
Inigo Goiri 4eb0497091 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 13:58:03 -07:00
Erik Krogen fec7c5f3eb HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
 2019-03-13 13:19:18 -07:00
Shweta Yakkali 1ceefa726e HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020)
 2019-03-12 16:05:55 -07:00
Stephen O'Donnell a21e2e4dbc HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3)
 2019-03-12 10:18:56 -07:00
Erik Krogen 6d076dd5e8 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
 2019-03-07 08:42:41 -08:00
Wei-Chiu Chuang e58ccca3ce HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
 2019-03-04 10:45:31 -08:00
Erik Krogen af16db86d4 HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:44 -08:00
Surendra Singh Lilhore 2e939515df HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f0)
 2019-02-21 20:47:50 +05:30
Surendra Singh Lilhore b93b127956 HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 
(cherry picked from commit 41e18feda3)
 2019-02-20 17:12:58 +05:30
Surendra Singh Lilhore f4b9ba230b HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 
(cherry picked from commit 29b411d5f0)
 2019-02-13 23:51:36 +05:30
Surendra Singh Lilhore e163bf2258 HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 35d4f32b32)
 2019-02-13 23:21:37 +05:30
Vinayakumar B 890736828b HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 21:58:59 +05:30
Surendra Singh Lilhore 2501fcd26b HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 4be87353e3)
 2019-02-07 16:52:53 -08:00
Kitti Nanasi 38541b2375 HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 49ddd8a6ed)
 2019-02-05 15:15:04 -08:00
Wangda Tan 649da5af04 Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
 2019-02-05 14:05:51 -08:00
Steve Loughran bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f8483)
 2019-02-04 17:10:19 +00:00
Masatake Iwasaki a86893f875 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7)
 2019-01-29 02:28:18 +09:00
Dinesh Chitlangia eb77e60163 HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cf)
 2019-01-25 07:37:43 -08:00
Takanobu Asanuma e88db02753 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 
(cherry picked from commit 7d6792e5d2)
 2019-01-23 10:04:50 +09:00
Arpit Agarwal 8dd8121f7d HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal. 
(cherry picked from commit 1ff658b2ef)
 2019-01-21 20:46:50 -08:00
Arpit Agarwal cd5e7f152f HDFS-14222. Make ThrottledAsyncChecker constructor public. Contributed by Arpit Agarwal. 
(cherry picked from commit 6f0756fc0e)
 2019-01-21 20:46:50 -08:00
Weiwei Yang 2253a86a3f HDFS-14207. ZKFC should catch exception when ha configuration missing. Contributed by Fei Hui. 
(cherry picked from commit de34fc148c)
 2019-01-22 12:08:03 +08:00
Surendra Singh Lilhore 5d2d0e74d5 HDFS-14198. Upload and Create button doesn't get enabled after getting reset. Contributed by Ayush Saxena. 
(cherry picked from commit 9aeaaa0479)
 2019-01-11 14:55:18 +05:30
Akira Ajisaka 6bc2781b39
HDFS-14183. [SPS] Remove the -w parameter from the -satisfystoragepolicy usage. Contributed by Ayush Saxena. 
(cherry picked from commit f30473f2ef)
 2018-12-31 03:17:27 +09:00
Wei-Chiu Chuang c48640ffbd HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang. 
(cherry picked from commit e9a005dfcf)
 2018-12-28 10:38:55 -08:00
Surendra Singh Lilhore d2ec1ff020 HDFS-14046. In-Maintenance ICON is missing in datanode info page. Contributed by Ranith Sardar. 
(cherry picked from commit 686fcd4db3)
 2018-12-24 11:09:07 +05:30
Vinayakumar B abe14d32d4 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 
(cherry picked from commit 39dc7345b8)
 2018-12-11 18:10:51 +05:30
Tsz Wo Nicholas Sze 9d508f719b HDFS-14112. Avoid recursive call to external authorizer for getContentSummary. 2018-11-29 13:56:43 -08:00
Brahma Reddy Battula e2fa9e8cdd HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 
(cherry picked from commit f534736867)
 2018-11-30 00:28:04 +05:30
Vinayakumar B 538757ed49 HDFS-13816. dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException. Contributed by Vinayakumar B. 
(cherry picked from commit b098281454)
 2018-11-26 18:23:09 +05:30
Brahma Reddy Battula 7e10dd03b5 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 15d8f592b51d08a658e88c6a7a7596e0edf5793b)
 2018-11-22 00:07:38 +05:30
Akira Ajisaka f6227367fb
Add 2.9.2 release notes and changes documents. 
(cherry picked from commit 1a00b4e325)
 2018-11-20 14:25:48 +09:00
Vinayakumar B 7a2a806213 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 
(cherry picked from commit 886b935630bf2d46f4c6773de013c093d2e44c38)
 2018-11-16 18:01:43 +05:30
Vinayakumar B 71a6ec6058 HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena. 
(cherry picked from commit 8571507efa)
 2018-11-16 17:29:25 +05:30
Xiao Chen 5afd7efe24 Revert "HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel." 
This reverts commit 7dc79a8b5b.
 2018-11-13 12:45:55 -08:00
Brahma Reddy Battula 5e86641265 HDFS-14070. Refactor NameNodeWebHdfsMethods to allow better extensibility. Contributed by CR Hota 
(cherry picked from commit e7b63baca1)
 2018-11-13 13:09:03 +05:30
Arpit Agarwal e6532f7eb2 HDFS-14065. Failed Storage Locations shows nothing in the Datanode Volume Failures. Contributed by Ayush Saxena. 
(cherry picked from commit b6d4e19f34)
 2018-11-12 15:32:14 -08:00
Kitti Nanasi 57409874bb HDFS-14039. ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3). Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 8d99648c20)
 2018-11-08 10:08:26 -08:00
Inigo Goiri e7fba62e6a HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 
(cherry picked from commit 887244de4a)
 2018-11-06 11:20:06 -08:00
Inigo Goiri 378f189c4f HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 
(cherry picked from commit f3296501e0)
 2018-11-05 16:49:12 -08:00
Giovanni Matteo Fumarola 6e1fad299a HDFS-14042. Fix NPE when PROVIDED storage is missing. Contributed by Virajith Jalaparti. 2018-11-05 11:33:32 -08:00
Xiao Chen 8788489df4 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 
(cherry picked from commit db7e636824)
 2018-10-29 19:13:43 -07:00
Wei-Chiu Chuang 5f8dd40647 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit fd02c501c0.
 2018-10-28 17:05:45 -07:00
Anu Engineer 3cbb028dd2 HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.

(cherry picked from commit 4f10d7e23f)
 2018-10-26 16:47:53 -07:00
Arpit Agarwal 642b613a7d HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 
(cherry picked from commit 9b899f1ebd)
 2018-10-25 15:14:58 -07:00
Wei-Chiu Chuang fd02c501c0 HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
 2018-10-25 09:36:28 -07:00
Inigo Goiri 5b995f85f5 HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 
(cherry picked from commit 0e56c883cd)
 2018-10-24 16:14:06 -07:00
Inigo Goiri b6937e8e70 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit c1874046e2)
 2018-10-24 09:38:32 -07:00
Arpit Agarwal 28a1cabb72 HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 
(cherry picked from commit c58811c77d)
 2018-10-22 14:45:11 -07:00
Inigo Goiri c0f618f3d5 HDFS-14002. TestLayoutVersion#testNameNodeFeatureMinimumCompatibleLayoutVersions fails. Contributed by Takanobu Asanuma. 
(cherry picked from commit be1cffb085)
 2018-10-18 17:14:56 -07:00
Sunil G bde4fd5ed9 Preparing for 3.2.0 release 2018-10-18 17:07:45 +05:30
Xiao Chen 08b415d170 HDFS-13662. TestBlockReaderLocal#testStatisticsForErasureCodingRead is flaky 
(cherry picked from commit 533138718c)
 2018-10-16 19:34:41 -07:00
Xiao Chen ced2596094 HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a025)
 2018-10-15 11:00:21 -07:00
Vinayakumar B 8a41edb089 Fix potential FSImage corruption. Contributed by Daryn Sharp. 
(cherry picked from commit b60ca37914)
 2018-10-15 15:49:56 +05:30
Vinayakumar B b185964b15 HDFS-13945. TestDataNodeVolumeFailure is Flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 6e0e6daaf3)
 2018-10-12 17:34:28 +05:30
Vinayakumar B d838d39a2c HDFS-13156. HDFS Block Placement Policy - Client Local Rack. Contributed by Ayush Saxena. 
(cherry picked from commit de44e1064f)
 2018-10-12 17:28:48 +05:30
Hrishikesh Gadre 90a9837c9d HDFS-13956. iNotify should include information to identify a file as either replicated or erasure coded. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bf3d591f0c)
 2018-10-10 10:24:23 -07:00
Weiwei Yang 966d94ea2f HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71)
 2018-10-10 10:14:57 -07:00
Inigo Goiri bc400c8a33 HDFS-11396. TestNameNodeMetadataConsistency#testGenerationStampInFuture timed out. Contributed by Ayush Saxena. 2018-10-09 17:04:17 -07:00
Hrishikesh Gadre a99658cd85 HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 08bb6c49a5)
 2018-10-08 20:39:48 -07:00
Yiqun Lin b6698e2a82 HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 1043795f7f)
 2018-10-09 10:35:08 +08:00
Sunil G 2c392da8aa HADOOP-15791. Remove Ozone related sources from the 3.2 branch. Contributed by Elek, Marton. 2018-10-05 16:10:47 +05:30
Yiqun Lin 62d02eecd0 HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. 
(cherry picked from commit 619e490333)
 2018-10-05 10:09:22 +08:00
Yiqun Lin 5689355783 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 2018-10-02 09:43:14 +08:00
Takanobu Asanuma e2113500df HDFS-8196. Post enabled Erasure Coding Policies on NameNode UI. Contributed by Kitti Nanasi and Kai Sasaki. 2018-09-29 00:53:35 +09:00
Akira Ajisaka f9c0221623
HDFS-13932. [JDK 11] Casts to BlockStoragePolicy[] in unit tests raise ClassCastExceptions. Contributed by Andrew Purtell. 2018-09-27 11:44:42 +09:00
Brahma Reddy Battula e0ff8e2c10 HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 2018-09-26 23:44:16 +05:30
Inigo Goiri 6275b42870 HDFS-13927. Addendum: Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena. 2018-09-26 09:50:09 -07:00
Inigo Goiri 4758b4b6da HDFS-13927. Improve TestDataNodeMultipleRegistrations#testDNWithInvalidStorageWithHA wait. Contributed by Ayush Saxena. 2018-09-21 15:32:28 -07:00
Anu Engineer 6fc293fece HDFS-13892. Disk Balancer: Make execute command documentation better. 
Contributed by  Ranith Sardar.
 2018-09-19 20:48:41 -07:00
Arpit Agarwal 2eb597b151 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 2018-09-19 13:48:27 -07:00
Chen Liang a30b4f9e71 HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 13:22:37 -07:00
Inigo Goiri 090272d7de HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 2018-09-19 13:00:30 -07:00
Inigo Goiri 56e0d635e0 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 2018-09-19 09:30:25 -07:00
Wei-Chiu Chuang 28ceb34a72 HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 2018-09-19 03:12:20 -07:00
Shweta 27978bcb66 HDFS-13833. Improve BlockPlacementPolicyDefault's consider load logic. Contributed by Shweta. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-18 20:23:50 -07:00
Mingliang Liu 6ff509c32a HDFS-11719. Arrays.fill() wrong index in BlockSender.readChecksum() exception handling. Contributed by Tao Zhang 2018-09-18 10:25:28 -07:00
Inigo Goiri d1541932db HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 2018-09-17 14:42:03 -07:00
Anu Engineer fdf5a3fd63 HDFS-13919. Documentation: Improper formatting in Disk Balancer for Settings. 
Contributed by Ayush Saxena.
 2018-09-17 10:08:23 -07:00
Wei-Chiu Chuang a1de8cbac5 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-09-14 05:22:56 +08:00
Inigo Goiri e1b242a984 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 2018-09-13 09:16:58 -07:00
Daniel Templeton 78bd3b1db9 HDFS-13846. Safe blocks counter is not decremented correctly if the block is striped 
(Contributed by Kitti Nanasi via Daniel Templeton)

Change-Id: Id41747a67dc946fdf0dbde90643bb1ea7e9e0f70
 2018-09-12 11:18:55 -07:00
Brahma Reddy Battula b2432d254c HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 2018-09-12 06:30:57 +05:30
Xiao Chen 8e54da1511 HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 2018-09-10 22:34:23 -07:00
Yiqun Lin 0da49642fc HDFS-13884. Improve the description of the setting dfs.image.compress. Contributed by Ryan Wu. 2018-09-10 13:57:36 +08:00
Vinayakumar B eef3bafae8 HDFS-13895. EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy. Contributed by Ayush Saxena. 2018-09-10 09:22:59 +05:30
Vinayakumar B 30eceec342 HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 2018-09-10 09:10:51 +05:30
Hrishikesh Gadre 335a8139f5 HDFS-13820. Add an ability to disable CacheReplicationMonitor. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-07 14:59:06 -07:00
Sean Mackrory 410dd3faa5 HDFS-13744. OIV tool should better handle control characters present in file or directory names. Contributed by Zsolt Venczel. 2018-09-07 12:39:33 -06:00
Giovanni Matteo Fumarola eca1a4bfe9 HDFS-13695. Move logging to slf4j in HDFS package. Contributed by Ian Pickering. 2018-09-06 14:48:00 -07:00
Giovanni Matteo Fumarola 9af96d4ed4 HADOOP-15707. Add IsActiveServlet to be used for Load Balancers. Contributed by Lukas Majercak. 2018-09-05 10:50:25 -07:00
Hrishikesh Gadre 6ccb809c2d HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-09-04 21:53:42 -07:00
Vinayakumar B 3801436e49 HDFS-13774. EC: 'hdfs ec -getPolicy' is not retrieving policy details when the special REPLICATION policy set on the directory. Contributed by Ayush Saxena. 2018-09-03 14:37:57 +05:30
Wei-Chiu Chuang 8aa6c4f079 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit 26c2a97c56.
 2018-08-30 11:44:20 -07:00
Vinayakumar B c36d69a7b3 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 
(cherry picked from commit 65977e5d81)
 2018-08-30 22:15:51 +05:30
Yiqun Lin 582cb10ec7 HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui. 2018-08-30 11:21:13 +08:00
Zsolt Venczel 3e18b957eb HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel. 2018-08-28 15:13:43 -07:00
Shweta 33f42efc94 HDFS-13837. Enable debug log for LeaseRenewer in TestDistributedFileSystem. Contributed by Shweta. 2018-08-28 13:56:32 -07:00
Wei-Chiu Chuang 26c2a97c56 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 2018-08-27 16:02:35 -07:00
Yiqun Lin b9b964d253 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 2018-08-27 14:55:46 +08:00
Surendra Singh Lilhore 96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Vinayakumar B 770d9d9bb6 HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 2018-08-21 09:33:19 +05:30
Giovanni Matteo Fumarola 79c97f6a0b HADOOP-14624. Add GenericTestUtils.DelayAnswer that accept slf4j logger API. Contributed by Ian Pickering and Wenxin He. 2018-08-17 14:40:00 -07:00
Márton Elek 77b015000a HADOOP-8807. Update README and website to reflect HADOOP-8662. Contributed by Andras Bokor. 2018-08-17 11:10:29 +02:00
Xiao Chen c67b0650ea HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 2018-08-16 23:14:21 -07:00
Wei-Chiu Chuang 1290e3c647 HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 2018-08-16 16:29:38 -07:00
Daniel Templeton 8512e1a91b HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
 2018-08-16 15:00:45 -07:00
Yiqun Lin 6df606f1b4 HDFS-13829. Remove redundant condition judgement in DirectoryScanner#scan. Contributed by liaoyuxiangqin. 2018-08-16 18:44:18 +08:00
Xiao Chen 7dc79a8b5b HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel. 2018-08-15 13:53:47 -07:00
Xiao Chen ef02f9b233 HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin. 2018-08-15 09:23:05 -07:00
Akira Ajisaka 3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering. 2018-08-16 00:31:59 +09:00
Daniel Templeton 4a5006b1d0 HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky 
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
 2018-08-14 17:22:15 -07:00
Wei-Chiu Chuang 363bd16e31 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-14 17:19:00 -07:00
Xiao Chen cede33997f HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. 2018-08-14 11:57:22 -07:00
Wei-Chiu Chuang 61a9b4f58b HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan. 2018-08-14 11:51:27 -07:00
Wei-Chiu Chuang 4023eeba05 HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh. 2018-08-13 17:36:13 -07:00
Wei-Chiu Chuang 23854443ef HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng. 2018-08-13 16:12:37 -07:00
Arpit Agarwal f760a544a7 HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar. 2018-08-13 11:35:19 -07:00
drankye 475bff6e8e HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 2018-08-13 17:32:56 +08:00