Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,445 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5023 Commits

Author SHA1 Message Date
Wei-Chiu Chuang 9599c91518 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-08-09 15:49:58 -07:00
Erik Krogen c377540704 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen 5190fd64be HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen c055e7812c HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-08-09 15:49:58 -07:00
Erik Krogen 3b2b97a6e6 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
(cherry-picked from commit 6d076dd5e8)
 2019-08-09 15:49:58 -07:00
Erik Krogen a0327f58cb HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-08-09 15:49:58 -07:00
Chen Liang b247a8b0de [SBN read] Addendum: Misc fix to be Java 7 compatible 2019-08-09 15:49:58 -07:00
Chao Sun 8cf1c0b91b HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 3ec2ff756d HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-08-09 15:49:57 -07:00
Chao Sun 41e85ccf44 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chao Sun 705e1c8d96 HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Erik Krogen 7b1a3b5619 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Erik Krogen a86d955aa7 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Erik Krogen 0705e83bdb HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chao Sun 52d066a9d3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Chen Liang 17fac44040 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-08-09 15:49:57 -07:00
Chao Sun 40d5cc15b2 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 37dfd294e2 HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko 63cf9801e8 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Konstantin V Shvachko ae559db6fd HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-08-09 15:49:57 -07:00
Chen Liang b4fc93d90b HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen 7146e67893 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Erik Krogen a7d551efd5 HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko 7eacf5f8f1 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. Also fix a number of java7 incompatibilities from previous SbN read commits. 2019-08-09 15:49:56 -07:00
Chen Liang bf9d8ba66b HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-08-09 15:49:56 -07:00
Erik Krogen 7ccb2a48ff HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 
Also add a missing change from HDFS-13610
 2019-08-09 15:49:56 -07:00
Konstantin V Shvachko fcafa72958 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-08-09 15:49:56 -07:00
Chen Liang 9f90bcadf2 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-08-09 15:49:56 -07:00
Chao Sun f531301e7e HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-08-09 15:49:56 -07:00
Erik Krogen 33d8aae555 HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 
Also fix a issue in IPCLoggerChannel
 2019-08-09 15:49:56 -07:00
Erik Krogen ea6d36763b HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Chao Sun 04eefe245a HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Konstantin V Shvachko 272a198175 HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-08-09 15:49:55 -07:00
Erik Krogen 85f81fe458 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 
HDFS-10519. Add a configuration option to enable in-progress edit log tailing. Contributed by Jiayi Zhou.
 2019-08-09 15:49:55 -07:00
Erik Krogen 88d65af8a1 HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Erik Krogen befe41074c HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov 3ed8d45660 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Erik Krogen d5da529acf HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Erik Krogen d0c85f83da HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Plamen Jeliazkov dd70e6d364 HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-08-09 15:49:55 -07:00
Chao Sun 61b510815c HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Siyao Meng 15062b6d28 HDFS-14696. Backport HDFS-11273 to branch-2 (Move TransferFsImage#doGetUrl function to a Util class) (#1251) Contributed by Siyao Meng. 2019-08-08 14:55:41 -07:00
Erik Krogen ce12c8fc54 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 2019-08-08 10:18:00 -07:00
Erik Krogen 97600cd916 HDFS-14631. The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 2019-08-07 13:46:56 -07:00
Yiqun Lin 63531be505 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 2019-08-07 23:01:51 +08:00
Weiwei Yang 567e1178d8 HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 2019-08-06 14:30:17 -07:00
Erik Krogen ee1547a807 HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 2019-08-06 09:15:24 -07:00
Erik Krogen a503662d6c HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
(cherry picked from da548cf263)
(cherry picked from 055f824bcb)
 2019-08-02 11:53:34 -07:00
Erik Krogen 61581a86b2 HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 2019-07-29 14:36:57 -07:00
Wei-Chiu Chuang 27ead06a13 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 2019-07-29 14:34:31 -07:00
Erik Krogen 653951e1f7 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit 3bfae20e94.
 2019-07-26 08:27:36 -07:00
Masatake Iwasaki 3bfae20e94 HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
 2019-07-26 09:54:27 +09:00
Erik Krogen cf7b9a5bd5 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 2019-07-16 07:22:45 -07:00
Weiwei Yang 3f112e7347 HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 
(cherry picked from commit 6c8cfd135e)
 2019-07-11 10:07:47 +08:00
Yiqun Lin 78aebee5c5 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b)
 2019-07-10 10:58:59 +08:00
stack a7c6d25dc3 HDFS-14483 Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9"" 
This is a revert of a revert, i.e. a reapplication of the HDFS-14483
patch. Original commit was missing the JIRA number.

Revert "Revert "Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9""

This reverts commit b26ad8a4c3.
 2019-07-09 08:58:16 -07:00
stack b26ad8a4c3 Revert "Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9" 
This reverts commit 4f09a6bcfd.
 2019-07-09 08:57:57 -07:00
sunlisheng 4f09a6bcfd Backport HDFS-3246,HDFS-14111 ByteBuffer pread interface to branch-2.9 
Signed-off-by: stack <stack@apache.org>
 2019-07-09 08:54:37 -07:00
leosunli 0ff5db746e HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c90729486)
 2019-07-07 20:04:08 -07:00
Masatake Iwasaki e61e52805f HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec7)
 2019-07-04 23:55:13 +09:00
Inigo Goiri 0272480e9f HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42)
(cherry picked from commit 9637124535)
(cherry picked from commit 90b88db35d)
(cherry picked from commit 9daa45f646)
 2019-06-24 21:17:38 -07:00
Wei-Chiu Chuang bf63475e7a HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-06-20 21:06:06 -07:00
Wei-Chiu Chuang f876e82abc HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 2019-06-20 19:36:44 -07:00
Zsolt Venczel a68de43957 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff)
(cherry picked from commit 4ea3b04bf3)
(cherry picked from commit 76faa41f18)
 2019-06-18 14:38:42 -07:00
Kitti Nanasi abd11f9db1 HDFS-13770. dfsadmin -report does not always decrease "missing blocks (with replication factor 1)" metrics when file is deleted. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-06-18 00:25:53 -07:00
Kihwal Lee 500dc05924 HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b649)
(cherry picked from commit fd8b8c7494)
 2019-05-30 17:37:38 -05:00
Ayush Saxena c63049dfd4 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f2)
(cherry picked from commit 2cd7577a5a)
(cherry picked from commit 90cc5ddd49)
(cherry picked from commit 942c5b04f2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
 2019-05-29 21:11:15 -07:00
Siyao Meng cecba551aa HDFS-14514. Actual read size of open file in encryption zone still larger than listing size even after enabling HDFS-11402 in Hadoop 2. Contributed by Siyao Meng, Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Stephen O'Donnell <sodonnell@cloudera.com>
 2019-05-29 17:28:49 -07:00
Takanobu Asanuma e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Erik Krogen f024d27cd9 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f0)
(cherry-picked from c26cf22d6b)
(cherry-picked from 9295e95174)
 2019-05-24 13:39:52 -07:00
Konstantin V Shvachko 2afc770443 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:43:12 -07:00
Wei-Chiu Chuang 380289a167 HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge, Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Siyao Meng <smeng@cloudera.com>
 2019-05-24 18:36:53 +02:00
Puleya7 8579ab13cb HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479)
 2019-04-17 21:58:43 +09:00
Wei-Chiu Chuang 8af767b069 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 2019-04-10 14:36:07 -07:00
Dinesh Chitlangia 66ac003128 HDFS-14414. Clean up findbugs warning in branch-2. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-08 10:47:13 -07:00
Chen Liang dce76e0355 HDFS-14415. Backport HDFS-13799 to branch-2. Contributed by Chao Sun. 2019-04-05 11:27:37 -07:00
Akira Ajisaka 2abc7927d1
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:32:07 +09:00
Chen Liang a5bca767f7 HDFS-14397. Backport HADOOP-15684 to branch-2. Contributed by Chao Sun. 2019-04-03 17:32:56 -07:00
Chen Liang d6083e9920 HDFS-14399. Backport HDFS-10536 to branch-2. Contributed by Chao Sun. 2019-04-01 12:47:02 -07:00
Chen Liang 812e6d5c81 HDFS-14392. Backport HDFS-9787 to branch-2. Contributed by Chao Sun. 2019-03-28 19:07:58 -07:00
Chen Liang 99e41293ee HDFS-14391. Backport HDFS-9659 to branch-2. Contributed by Chao Sun. 2019-03-28 10:39:45 -07:00
Chen Liang 7935485eec HDFS-14205. Backport HDFS-6440 to branch-2. Contributed by Chao Sun. 2019-03-26 12:12:14 -07:00
Inigo Goiri 525f4e52f3 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 14:02:20 -07:00
Wei-Chiu Chuang d71cfe1461 HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
(cherry picked from commit e58ccca3ce)
(cherry picked from commit d951497f57)
(cherry picked from commit e23a448e0e)
 2019-03-04 11:30:56 -08:00
Masatake Iwasaki 9688ed3b48 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7)
 2019-01-29 01:45:30 +09:00
Wei-Chiu Chuang 10cc02f643 HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang. 
(cherry picked from commit e9a005dfcf)
(cherry picked from commit c48640ffbd)
(cherry picked from commit f24684ca9c)
(cherry picked from commit 054594c614)
 2018-12-28 10:46:38 -08:00
Yiqun Lin 50e11dc807 HDFS-13870. WebHDFS: Document ALLOWSNAPSHOT and DISALLOWSNAPSHOT API doc. Contributed by Siyao Meng. 
(cherry picked from commit 0e36e935d9)
 2018-11-30 11:34:28 +08:00
Akira Ajisaka 6cae964f6c
Add 2.9.2 release notes and changes documents. 
(cherry picked from commit 1a00b4e325)
 2018-11-20 14:30:09 +09:00
Vinayakumar B c3f0681f99 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 
(cherry picked from commit 886b935630bf2d46f4c6773de013c093d2e44c38)
 2018-11-16 18:02:10 +05:30
Vinayakumar B bc4098eb07 HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena. 
(cherry picked from commit 8571507efa)
 2018-11-16 17:43:49 +05:30
Inigo Goiri d9b3b58389 HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 2018-11-06 12:16:46 -08:00
Inigo Goiri 98075d9224 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 
(cherry picked from commit f3296501e0)
(cherry picked from commit 9bf4f3d614)
 2018-11-05 16:56:37 -08:00
Akira Ajisaka c19baf19da
Addendum: Fix potential FSImage corruption. Contributed by Xiao Chen. 
(cherry picked from commit dd70b1f538)
 2018-10-31 17:04:06 +09:00
Vinayakumar B e212d7d81f Fix potential FSImage corruption. Contributed by Daryn Sharp. 2018-10-15 16:04:34 +05:30
Yiqun Lin c632c6e6e9 HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 2018-10-09 10:20:37 +08:00
Chen Liang 3a6ad9cd3d HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-20 10:00:13 -07:00
Arpit Agarwal 0dd686114f HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b151)
 2018-09-19 13:49:16 -07:00
Inigo Goiri 3e476dcae8 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0)
 2018-09-19 09:32:35 -07:00
Inigo Goiri 9bd2616138 HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 
(cherry picked from commit a9c1c55cafc0338bb35d2991f217d4157463f1d3)
 2018-09-17 14:40:00 -07:00
Inigo Goiri bbd09877d0 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 
(cherry picked from commit e1b242a984)
 2018-09-13 09:18:40 -07:00
Brahma Reddy Battula 9befa5ed95 HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 
(cherry picked from commit b2432d254c)
 2018-09-12 06:37:25 +05:30
Xiao Chen a7b9e7d858 HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 
(cherry picked from commit 8e54da1511)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java

(cherry picked from commit 2dd27c999b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java

(cherry picked from commit a5da4df73c)
 2018-09-10 23:13:14 -07:00
Brahma Reddy Battula e56f183696 HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar 2018-09-10 07:11:26 +05:30
Hrishikesh Gadre 0468b6e736 HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 6ccb809c2d)
 2018-09-04 21:55:09 -07:00
Yiqun Lin 361d0e45aa HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 
(cherry picked from commit b9b964d253)
 2018-08-27 15:03:33 +08:00
Wei-Chiu Chuang 04c8b5dbb6 HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan. 2018-08-14 11:57:30 -07:00
Wei-Chiu Chuang ad5c3300ff HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng. 
(cherry picked from commit 23854443ef)
(cherry picked from commit b38649c59a)
(cherry picked from commit 4b606eb9fd)
 2018-08-13 16:15:36 -07:00
Chao Sun 305b9a652a Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun. 
(cherry picked from commit a2a8c48699)
 2018-08-10 16:17:04 -07:00
Wei-Chiu Chuang 5ef8fd5119 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin. 
(cherry picked from commit e95c5e9f62)
(cherry picked from commit 964f3454d1)
(cherry picked from commit 0af04ad89f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
 2018-07-25 11:26:19 -07:00
Konstantin V Shvachko fccb3c02f0 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal. 
(cherry picked from commit 0a1e922f3d)
 2018-07-16 18:44:01 -07:00
Akira Ajisaka ff46db2ea6
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota. 2018-07-12 13:53:08 -04:00
Andrew Wang 913417bbea HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi. 2018-07-09 15:17:26 +02:00
Inigo Goiri 3e177a1d62 HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 7296b644f7)
 2018-07-02 16:38:13 -07:00
Inigo Goiri 9e148aa0bc HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 8762e9cf10)
 2018-06-15 16:51:03 -07:00
Inigo Goiri b52316d796 HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak. 
(cherry picked from commit d31a3ce767)
 2018-06-15 16:00:45 -07:00
Sean Mackrory b122961bc0 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel. 2018-06-15 16:28:36 -06:00
Inigo Goiri fe831cb5b2 HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang. 2018-06-15 13:15:20 -07:00
Inigo Goiri 55b50eecc4 HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit 43d994e4a6)
 2018-06-15 10:35:05 -07:00
Inigo Goiri fcd7ae2745 HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak. 2018-06-14 14:12:54 -07:00
Inigo Goiri 06c87edf51 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 8d4926f38b)
 2018-06-14 10:00:26 -07:00
Inigo Goiri 8059390798 HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 2018-06-11 13:47:08 -07:00
Yiqun Lin 1465830f7e HDFS-13644. Backport HDFS-10376 to branch-2. Contributed by Zsolt Venczel. 2018-06-07 18:10:52 +08:00
Inigo Goiri 65d6943b4f HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu. 
(cherry picked from commit a28848104f)
 2018-06-04 18:14:34 -07:00
Rushabh Shah 0caf40efe5 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah 
(cherry picked from commit e2289c8d14)
(cherry picked from commit 4ecb7aaf77)
 2018-06-04 10:07:24 -05:00
Chao Sun 87e805ba4d HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun. 
(cherry picked from commit ff013d2c95)
 2018-05-31 17:44:39 -07:00
Inigo Goiri 6b9ef2d5b7 HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu. 2018-05-30 10:10:25 -07:00
Yiqun Lin cb962139ee HDFS-13626. Fix incorrect username when deny the setOwner operation. Contributed by Zsolt Venczel. 2018-05-30 16:55:39 +08:00
Inigo Goiri 86736cb279 HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9dbf4f0166)
 2018-05-28 16:36:06 -07:00
Inigo Goiri 27091d2d16 HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu. 2018-05-25 19:48:16 -07:00
Inigo Goiri 1542484161 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
 2018-05-25 09:12:44 -07:00
Inigo Goiri c5ff9553fb HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c13dea87d9)
 2018-05-23 11:37:55 -07:00
Inigo Goiri 411a2f609b HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang. 2018-05-23 09:48:35 -07:00
Inigo Goiri 8f45d119a1 HDFS-13590. Backport HDFS-12378 to branch-2. Contributed by Lukas Majercak. 2018-05-18 11:07:12 -07:00
Inigo Goiri 1af41dadc2 HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9775ecb235)
 2018-05-18 09:47:50 -07:00
Inigo Goiri b0e0ec1648 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu. 
(cherry picked from commit 57b893de3d)
 2018-05-18 09:40:10 -07:00
Inigo Goiri a5ec44c655 HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 53b807a6a8)
 2018-05-17 17:04:59 -07:00
Inigo Goiri 6df8fa1bc9 HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit a97a2042f2)
 2018-05-17 16:55:01 -07:00
Inigo Goiri 315ffd1ada HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 65476458fa)
 2018-05-17 14:49:37 -07:00
Inigo Goiri b87cd6354b HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 26f1e22fc9)
 2018-05-17 13:36:53 -07:00
Inigo Goiri 33dfbed8d2 HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c0ec061e28)
 2018-05-17 09:23:11 -07:00
Kihwal Lee 033b25cbe9 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0ce6290de6)
 2018-05-17 09:29:08 -05:00
Inigo Goiri d2a3bd1430 HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu. 
(cherry picked from commit 454de3b543)
 2018-05-16 19:24:36 -07:00
Inigo Goiri 9c96822096 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 55d5549087)
 2018-05-16 10:10:37 -07:00
Inigo Goiri d163f72310 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu. 2018-05-16 09:33:16 -07:00
Inigo Goiri 3d48a895a6 HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu. 
(cherry picked from commit 63480976a0)
 2018-05-15 15:36:58 -07:00
Inigo Goiri cdb80d6dcb HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 2d6195c0af)
 2018-05-15 11:22:41 -07:00
Inigo Goiri cc9203615a HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu. 
(cherry picked from commit eaca7980ad)
 2018-05-15 10:29:44 -07:00
Inigo Goiri 4ae1ed5b17 HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 92ebd466c7)
 2018-05-15 10:23:28 -07:00
Hanisha Koneru c998c126b8 HDFS-13544. Improve logging for JournalNode in federated cluster. 
(cherry picked from commit 6beb25ab7e)
 2018-05-14 10:21:59 -07:00
Sammi Chen eabc588d22 Add 2.9.1 release notes and changes documents 2018-05-14 15:36:30 +08:00
Inigo Goiri d88d9f2874 HDFS-13542. TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu. 2018-05-11 09:50:40 -07:00
Chris Douglas b2acaa52d2 HDFS-13272. DataNodeHttpServer to have configurable HttpServer2 threads. Contributed by Erik Krogen 2018-05-10 14:53:39 -07:00
Xiao Chen 5679920793 HDFS-12981. renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi. 
(cherry picked from commit 696a4be0da)
 2018-05-07 15:51:33 -07:00
Xiao Chen d7f0af0b65 Revert "HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445." 
This reverts commit 2444d70a40.
 2018-05-07 14:11:13 -07:00
Inigo Goiri 9fd93ee533 HDFS-13503. Fix TestFsck test failures on Windows. Contributed by Xiao Liang. 2018-05-01 08:15:07 -07:00
Inigo Goiri cb3414a279 HDFS-13283. Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak. 2018-04-30 15:24:21 -07:00
Inigo Goiri c0c788aafc HDFS-13509. Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang. 2018-04-28 09:07:56 -07:00
Inigo Goiri 1c3d746fc0 HDFS-13326. RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li. 
(cherry picked from commit c394051a3d4d9d531f418503cb519606ae2b2e69)
 2018-04-26 13:14:08 -07:00
Yiqun Lin 33ffc96085 HDFS-13484. RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 30fef0bf1e5c8c0ca073df99ad9b33cb0e4431a5)
 2018-04-25 15:25:16 +08:00
Inigo Goiri db9da432b9 HDFS-13336. Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang. 
(cherry picked from commit df92a17e02)
 2018-04-23 19:14:03 -07:00
Chris Douglas 99e82e2c2c HDFS-13408. MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang 
(cherry picked from commit f411de6a79)
(cherry picked from commit cf272c5179)
(cherry picked from commit 956ab12ede)
 2018-04-23 11:15:28 -07:00
Xiao Chen 2444d70a40 HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445. 
(cherry picked from commit 6503593711)
 2018-04-18 22:43:24 -05:00
Erik Krogen 8f341c673f
HDFS-12828. OIV ReverseXML Processor fails with escaped characters 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 2d9e791a90)
 2018-04-18 14:40:45 +09:00
Inigo Goiri b4b10786c3 HDFS-13462. Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak. 2018-04-17 15:18:01 -07:00
Yiqun Lin a7721082dc HDFS-13418. NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie. 2018-04-13 17:59:35 +08:00
Akira Ajisaka f055a53b43
HDFS-7101. Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu. 
(cherry picked from commit 113af12cfb)
 2018-04-12 17:48:45 +09:00
Akira Ajisaka be627ccfe3
HDFS-13427. Fix the section titles of transparent encryption document. 
(cherry picked from commit c7cd362afd)
 2018-04-12 10:49:07 +09:00
Sean Mackrory b412958f70 HDFS-13176. WebHdfs file path gets truncated when having semicolon inside. Contributed by Zsolt Venczel. 2018-04-06 07:41:58 -06:00
Inigo Goiri 4276a858f8 HDFS-13337. Backport HDFS-4275 to branch-2.9. Contributed by Xiao Liang. 2018-04-03 13:55:44 -07:00
Xiao Chen 251fc1fb40 HADOOP-15317. Improve NetworkTopology chooseRandom's loop. 
(cherry picked from commit 57374c4737)
 2018-04-02 22:55:03 -07:00
Konstantin V Shvachko de419e2777 HADOOP-15253. Should update maxQueueSize when refresh call queue. Contributed by Tao Jie. 
(cherry picked from commit acfd764fcc)
 2018-03-30 19:08:12 -07:00
Arpit Agarwal 7e5c8faeb7 HDFS-13314. NameNode should optionally exit if it detects FsImage corruption. Contributed by Arpit Agarwal. 2018-03-28 13:08:04 -07:00
Wei Yan 69c08f3dd3 HDFS-12512. RBF: Add WebHDFS. 
(cherry picked from commit 6e31a09084)
 2018-03-23 10:39:08 -07:00
Kihwal Lee d2c3a56ad8 HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong. 
(cherry picked from commit 66461ed98b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
 2018-03-22 12:31:52 -05:00
Konstantin V Shvachko 52752840f8 HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan. 
(cherry picked from commit 8d898ab25f)
 2018-03-21 17:21:28 -07:00
weiy 4aa34324b2 HDFS-13215. RBF: Move Router to its own module. 2018-03-19 23:23:16 -07:00
Inigo Goiri f278494a5f HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak. 2018-03-16 14:59:01 -07:00
Inigo Goiri 5e917a1501 Revert "HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak." 
This reverts commit 972c322301.
 2018-03-16 14:26:09 -07:00
Inigo Goiri 972c322301 HDFS-12886. Ignore minReplication for block recovery. Contributed by Lukas Majercak. 
(cherry picked from commit 08ff1586d5)
 2018-03-16 10:31:02 -07:00
Inigo Goiri 4615c0bba2 HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit e71bc00a47)
 2018-03-16 09:15:23 -07:00
Inigo Goiri 3a5f50fa6e Revert "HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri." 
This reverts commit ccf07fdafd.
 2018-03-16 09:12:30 -07:00
Inigo Goiri b1b10007a4 HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. Contributed by Inigo Goiri. 2018-03-15 11:30:52 -07:00
Inigo Goiri ccf07fdafd HDFS-13224. RBF: Resolvers to support mount points across multiple subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit e71bc00a47)
 2018-03-15 10:51:04 -07:00
Yiqun Lin 3fad968624 HDFS-13261. Fix incorrect null value check. Contributed by Jianfei Jiang. 2018-03-15 11:12:25 +08:00
Inigo Goiri 299b40ff8b HDFS-13198. RBF: RouterHeartbeatService throws out CachedStateStore related exceptions when starting router. Contributed by Wei Yan. 
(cherry picked from commit 41fc7f80be)
 2018-03-14 13:16:32 -07:00
Yiqun Lin b3d56cb835 HDFS-12773. RBF: Improve State Store FS implementation. Contributed by Inigo Goiri. 
(cherry picked from commit 76be6cbf6c)
 2018-03-14 11:25:08 +08:00
Chris Douglas 340cd5f1b1 HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc)
(cherry picked from commit 09940b1eb3)
(cherry picked from commit a03c8ea61f)
 2018-03-12 20:08:46 -07:00
Yiqun Lin 38c21ec651 HDFS-13226. RBF: Throw the exception if mount table entry validated failed. Contributed by maobaolong. 
(cherry picked from commit 19292bc264)
 2018-03-13 11:06:53 +08:00
Yiqun Lin 5edbd1a258 HDFS-13253. RBF: Quota management incorrect parent-child relationship judgement. Contributed by Yiqun Lin. 
(cherry picked from commit 7fab787de7)
 2018-03-13 10:33:12 +08:00
Akira Ajisaka b0c80f1c81
HDFS-12156. TestFSImage fails without -Pnative 
(cherry picked from commit 319defafc1)
 2018-03-13 11:29:44 +09:00
Yiqun Lin 2ca6cf3170 HDFS-10803. TestBalancerWithMultipleNameNodes#testBalancing2OutOf3Blockpools fails intermittently due to no free space available. Contributed by Yiqun Lin. 
(cherry picked from commit 4afd50b106)
 2018-03-13 10:18:23 +08:00
Inigo Goiri 7fded36e4c HDFS-13241. RBF: TestRouterSafemode failed if the port 8888 is in use. Contributed by maobaolong. 
(cherry picked from commit 91c82c90f0)
 2018-03-12 17:30:45 -07:00
Inigo Goiri 737dca1959 HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin. 2018-03-12 12:54:42 -07:00
Anu Engineer 88c749a903 HDFS-10618. TestPendingReconstruction#testPendingAndInvalidate is flaky due to race condition. Contributed by Eric Badger. 2018-03-12 12:14:21 -07:00
Inigo Goiri 457895d50f Revert "HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin." 
This reverts commit ab0f127084.
 2018-03-12 11:44:06 -07:00
Inigo Goiri ab0f127084 HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin. 
(cherry picked from commit ac627f561f)
 2018-03-12 10:04:51 -07:00
Yiqun Lin 3a34924896 HDFS-13240. RBF: Update some inaccurate document descriptions. Contributed by Yiqun Lin. 
(cherry picked from commit 4743d4a2c7)
 2018-03-10 11:33:44 +08:00
Inigo Goiri 0dadcf9ea3 HDFS-13232. RBF: ConnectionPool should return first usable connection. Contributed by Ekanth S. 
(cherry picked from commit 8133cd5305)
 2018-03-09 18:28:59 -08:00
Inigo Goiri 4b6b07690e HDFS-13212. RBF: Fix router location cache issue. Contributed by Weiwei Wu. 
(cherry picked from commit afe1a3ccd5)
 2018-03-09 17:21:10 -08:00