Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,670 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7569 Commits

Author SHA1 Message Date
Chao Sun d96ddb3b87 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-07-12 11:50:01 -07:00
Erik Krogen 849a6c6f0d HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-07-12 11:49:58 -07:00
Erik Krogen ac2e79330c HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-07-12 11:05:50 -07:00
Chao Sun 9840dae72f HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 11:05:07 -07:00
Konstantin V Shvachko 405f6b6fbf HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-07-12 11:05:07 -07:00
Erik Krogen 8e551708d8 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:05:04 -07:00
Erik Krogen abb8614dcf HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:04:24 -07:00
Erik Krogen c40ddf9b5b HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-07-12 11:03:32 -07:00
Plamen Jeliazkov 7d0c3ba700 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:31 -07:00
Erik Krogen 18e545f0be HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen 07a6510e6a HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen b999e96c9a HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:30 -07:00
Plamen Jeliazkov 21a886a4ef HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:27 -07:00
Chao Sun 4cf63905d0 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-07-12 10:58:43 -07:00
Masatake Iwasaki c6ac7f511c
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00dd843a1a)
 2019-07-12 10:14:49 +09:00
Weiwei Yang 9c3806cf1b HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-11 09:46:27 +08:00
Erik Krogen a59553b5e6 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148)
(cherry-picked from 491dc7cc9d)
 2019-07-08 15:20:29 -07:00
leosunli 7c7907f756 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c90729486)
 2019-07-07 20:02:51 -07:00
Masatake Iwasaki 02ae1d6071 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec7)
 2019-07-04 23:54:18 +09:00
Anu Engineer 1bae5c7024 HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee)
(cherry picked from commit e7fce2104f)
 2019-06-24 21:20:00 -07:00
Inigo Goiri 90b88db35d HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42)
(cherry picked from commit 9637124535)
 2019-06-24 21:17:04 -07:00
Inigo Goiri aaf74ea5d7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6)
 2019-06-24 19:03:26 -07:00
Santosh Marella 026cc2647d HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-23 20:27:01 -07:00
Wei-Chiu Chuang 529d0955f5 HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f)
(cherry picked from commit d5ecca1d4c)
 2019-06-21 18:27:50 -07:00
Wei-Chiu Chuang 4507c2f3a5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c5)
(cherry picked from commit 845d7611ba)
 2019-06-21 18:18:36 -07:00
Wei-Chiu Chuang f33bcbb687 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965)
 2019-06-20 20:54:43 -07:00
Takanobu Asanuma 2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643)
 2019-06-20 20:18:00 -07:00
Wei-Chiu Chuang 58c1920472 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 49587e4df3.
 2019-06-19 06:55:25 -07:00
Virajith Jalaparti 49587e4df3 HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
(cherry picked from commit 36dc0810bf)
 2019-06-18 22:17:08 -07:00
Zsolt Venczel 4ea3b04bf3 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff)
 2019-06-18 14:38:05 -07:00
Elek, Marton 293f8bbd84 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
(cherry picked from commit 1a99f97945)
 2019-06-18 12:00:48 -07:00
Wei-Chiu Chuang 29c4aded34 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
(cherry picked from commit 223a582eb8)
 2019-06-18 00:34:55 -07:00
Wei-Chiu Chuang 4362fcb629 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78)
(cherry picked from commit f5428d4062c0a5e98461ea3b8420dc68555681f9)
 2019-06-17 22:48:07 -07:00
Weiwei Yang 3bdb661f35 HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
(cherry picked from commit f36cdc4d7f)
 2019-06-17 21:47:07 -07:00
Wei-Chiu Chuang fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea)
(cherry picked from commit 45543ffd6a)
 2019-06-17 20:20:08 -07:00
Akira Ajisaka 3d1d8623a1 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827)
(cherry picked from commit f68099aa13)
 2019-06-17 17:04:07 -07:00
Wei-Chiu Chuang b3e0430c3b HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05)
 2019-06-17 15:18:39 -07:00
huzheng b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a1844)
(cherry picked from commit 9b61ecfcbe)
 2019-06-17 07:21:07 -07:00
Inigo Goiri 76a66998fe HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:28:38 -07:00
Sean Mackrory fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Stephen O'Donnell a9b8310584 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0)
(cherry picked from commit 71159f9540)
 2019-06-12 10:31:21 -07:00
He Xiaoqiao 03ec4c4c7c HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit d7560c866e)
 2019-06-10 17:27:48 -07:00
Hanisha Koneru dd795edcc7 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40)
(cherry picked from commit 210480a23c)
 2019-06-10 13:47:17 -07:00
Kihwal Lee a935258e8c HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b649)
 2019-05-30 17:31:30 -05:00
Ayush Saxena 90cc5ddd49 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f2)
(cherry picked from commit 2cd7577a5a)
 2019-05-29 20:58:34 -07:00
Takanobu Asanuma 8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Erik Krogen 9295e95174 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f0)
(cherry-picked from c26cf22d6b)
 2019-05-24 13:19:39 -07:00
Konstantin V Shvachko 799ca572f7 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:35:32 -07:00
Siyao Meng 9196c84c9f HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9dff6eff81)
 2019-05-22 13:25:42 +09:00
Srinivasu Majeti e599e3b546 HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3e5e5b028a)
(cherry picked from commit 85f3664bb8)
 2019-05-17 19:22:03 +02:00
Siyao Meng 86fb5e1c8e HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a)
(cherry picked from commit 0550aa32f1a4b1809c234f8ed785db08b2e33d13)
 2019-05-01 16:49:36 -07:00
Arpit Agarwal 7b51edbab9
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f8)
 2019-04-29 14:54:11 -07:00
Puleya7 546ed4937c HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479)
 2019-04-17 21:57:06 +09:00
Akira Ajisaka a70076b3a5
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:31:14 +09:00
Stephen O'Donnell d9899015eb HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
(cherry picked from commit 388f445dde)
 2019-04-04 04:06:33 -07:00
Wei-Chiu Chuang 084fb9de22 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070)
(cherry picked from commit c8703dda07)
 2019-04-03 12:41:07 -07:00
Gabor Bota f01d592f3b
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e)
 2019-04-01 13:52:44 +09:00
Takanobu Asanuma 1fc74178eb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb)
 2019-03-27 03:29:42 +09:00
Inigo Goiri 80732146c6 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 13:58:39 -07:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
 2019-03-13 13:20:51 -07:00
Shweta Yakkali a324db2cce HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020)
(cherry picked from commit 1ceefa726e)
 2019-03-12 16:06:16 -07:00
Stephen O'Donnell 327e57ae6e HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3)
(cherry picked from commit a21e2e4dbc)
 2019-03-12 10:19:37 -07:00
Erik Krogen b0b8000240 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
(cherry-picked from commit 6d076dd5e8)
 2019-03-07 08:43:39 -08:00
Wei-Chiu Chuang d951497f57 HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
(cherry picked from commit e58ccca3ce)
 2019-03-04 10:46:00 -08:00
Erik Krogen b7eb9116eb HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:54 -08:00
Surendra Singh Lilhore bd9e2826fe HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 
(cherry picked from commit 29b411d5f0)
 2019-02-13 23:56:34 +05:30
Surendra Singh Lilhore 3bcdff8fac HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 35d4f32b32)
 2019-02-13 23:26:14 +05:30
Vinayakumar B 1f3e737407 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 22:35:55 +05:30
Surendra Singh Lilhore 1f4be45ef6 HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 4be87353e3)
(cherry picked from commit 2501fcd26b)
 2019-02-07 16:53:12 -08:00
Kitti Nanasi c25b641cec HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 49ddd8a6ed)
(cherry picked from commit 38541b2375)
 2019-02-05 15:15:31 -08:00
Wangda Tan a1e09b4c0c Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
(cherry picked from commit 649da5af04)
 2019-02-05 14:09:33 -08:00
Masatake Iwasaki dd3e7fc0c2 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7)
 2019-01-29 02:29:39 +09:00
Dinesh Chitlangia b63941516f HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cf)
(cherry picked from commit eb77e60163)
 2019-01-25 07:38:11 -08:00
Takanobu Asanuma 38e4bed224 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 
(cherry picked from commit 7d6792e5d2)
 2019-01-23 10:05:39 +09:00
Arpit Agarwal f5bfeac62e HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal. 
(cherry picked from commit 1ff658b2ef)
 2019-01-21 20:48:05 -08:00
Arpit Agarwal 78b45977b1 HDFS-14222. Make ThrottledAsyncChecker constructor public. Contributed by Arpit Agarwal. 
(cherry picked from commit 6f0756fc0e)
 2019-01-21 20:47:02 -08:00
Weiwei Yang cf520b7d64 HDFS-14207. ZKFC should catch exception when ha configuration missing. Contributed by Fei Hui. 
(cherry picked from commit de34fc148c)
 2019-01-22 12:17:33 +08:00
Surendra Singh Lilhore 6c846cfbf2 HDFS-14198. Upload and Create button doesn't get enabled after getting reset. Contributed by Ayush Saxena. 
(cherry picked from commit 9aeaaa0479)
 2019-01-11 14:59:17 +05:30
Wangda Tan 31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Wei-Chiu Chuang f24684ca9c HDFS-14171. Performance improvement in Tailing EditLog. Contributed by Kenneth Yang. 
(cherry picked from commit e9a005dfcf)
(cherry picked from commit c48640ffbd)
 2018-12-28 10:39:14 -08:00
Surendra Singh Lilhore de2b04a74b HDFS-14046. In-Maintenance ICON is missing in datanode info page. Contributed by Ranith Sardar. 
(cherry picked from commit 686fcd4db3)
 2018-12-24 11:11:21 +05:30
Brahma Reddy Battula 0b83c95ff4 HDFS-14095. EC: Track Erasure Coding commands in DFS statistics. Contributed by Ayush Saxena. 
(cherry picked from commit f534736867)
 2018-11-30 00:27:37 +05:30
Vinayakumar B 0254100074 HDFS-13816. dfs.getQuotaUsage() throws NPE on non-existent dir instead of FileNotFoundException. Contributed by Vinayakumar B. 
(cherry picked from commit b098281454)
 2018-11-26 18:22:58 +05:30
Vinayakumar B e5e4dc42a3 HDFS-14056. Fix error messages in HDFS-12716. Contributed by Ayush Saxena. 
(cherry picked from commit 886b935630bf2d46f4c6773de013c093d2e44c38)
 2018-11-16 18:01:54 +05:30
Vinayakumar B 53b6290949 HDFS-13963. NN UI is broken with IE11. Contributed by Ayush Saxena. 
(cherry picked from commit 8571507efa)
 2018-11-16 17:31:25 +05:30
Xiao Chen 8ab6aa1b42 Revert "HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel." 
This reverts commit 0424715207.
 2018-11-13 12:46:03 -08:00
Brahma Reddy Battula d72c134889 HDFS-14070. Refactor NameNodeWebHdfsMethods to allow better extensibility. Contributed by CR Hota 
(cherry picked from commit e7b63baca1)
 2018-11-13 13:11:39 +05:30
Arpit Agarwal b6ef824e76 HDFS-14065. Failed Storage Locations shows nothing in the Datanode Volume Failures. Contributed by Ayush Saxena. 
(cherry picked from commit b6d4e19f34)
 2018-11-12 15:32:05 -08:00
Kitti Nanasi 01b8197893 HDFS-14039. ec -listPolicies doesn't show correct state for the default policy when the default is not RS(6,3). Contributed by Kitti Nanasi. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 8d99648c20)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java
 2018-11-08 10:08:34 -08:00
Inigo Goiri f65324aa47 HDFS-14048. DFSOutputStream close() throws exception on subsequent call after DataNode restart. Contributed by Erik Krogen. 
(cherry picked from commit 887244de4a)
 2018-11-06 11:20:33 -08:00
Inigo Goiri 9bf4f3d614 HDFS-14043. Tolerate corrupted seen_txid file. Contributed by Lukas Majercak. 
(cherry picked from commit f3296501e0)
 2018-11-05 16:52:43 -08:00
Giovanni Matteo Fumarola a1321d020a HDFS-14042. Fix NPE when PROVIDED storage is missing. Contributed by Virajith Jalaparti. 2018-11-05 11:39:15 -08:00
Akira Ajisaka dd70b1f538
Addendum: Fix potential FSImage corruption. Contributed by Xiao Chen. 2018-10-31 17:02:15 +09:00
Xiao Chen 399645ebc1 HDFS-14027. DFSStripedOutputStream should implement both hsync methods. 
(cherry picked from commit db7e636824)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
 2018-10-29 19:13:51 -07:00
Wei-Chiu Chuang 35e09fe594 Revert "HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao." 
This reverts commit ef9f8ca13d.
 2018-10-28 17:06:40 -07:00
Anu Engineer 4a1be70099 HDFS-14028. HDFS OIV temporary dir deletes folder. 
Contributed by Adam Antal.

(cherry picked from commit 4f10d7e23f)
 2018-10-26 16:51:52 -07:00
Arpit Agarwal 8b4f9b3e2c HDFS-14029. Sleep in TestLazyPersistFiles should be put into a loop. Contributed by Adam Antal. 
(cherry picked from commit 9b899f1ebd)
 2018-10-25 15:14:48 -07:00
Wei-Chiu Chuang ef9f8ca13d HADOOP-15864. Job submitter / executor fail when SBN domain name can not resolved. Contributed by He Xiaoqiao. 
(cherry picked from commit fb2b72e6fc)
(cherry picked from commit fd02c501c0)
 2018-10-25 09:37:31 -07:00
Inigo Goiri 4b1f2ecd4c HDFS-14003. Fix findbugs warning in trunk for FSImageFormatPBINode. Contributed by Yiqun Lin. 
(cherry picked from commit 0e56c883cd)
 2018-10-24 16:14:14 -07:00
Xiao Chen 96cedb87b9 Fix potential FSImage corruption. Contributed by Daryn Sharp and Vinayakumar B. 
(cherry picked from commit f1996ccbaee734d423caa9d47a571cfff98ef42c)
 2018-10-24 16:10:25 -07:00
Inigo Goiri 2e43966b56 HDFS-14021. TestReconstructStripedBlocksWithRackAwareness#testReconstructForNotEnoughRacks fails intermittently. Contributed by Xiao Chen. 
(cherry picked from commit c1874046e2)
 2018-10-24 09:38:40 -07:00
Arpit Agarwal 8548864ccb HDFS-13941. make storageId in BlockPoolTokenSecretManager.checkAccess optional. Contributed by Ajay Kumar. 
(cherry picked from commit c58811c77d)
 2018-10-22 14:44:58 -07:00
Giovanni Matteo Fumarola 92c6bde6be HDFS-14005. RBF: Web UI update to bootstrap-3.3.7. Contributed by Inigo Goiri. 2018-10-18 14:28:19 -07:00
Inigo Goiri 16e1ab4c99 HDFS-14000. RBF: Documentation should reflect right scripts for v3.0 and above. Contributed by CR Hota. 
(cherry picked from commit 7347fa2df3)
 2018-10-17 16:42:32 -07:00
Xiao Chen d1749fbaae HADOOP-14445. Addendum: Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
(cherry picked from commit b6fc72a025)
 2018-10-15 11:00:30 -07:00
Xiao Chen 6a1ce74fb1 HADOOP-14445. Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances. 
Contributed by Daryn Sharp, Xiao Chen, Rushabh S Shah.

(cherry picked from commit 5ec86b445c)
(cherry picked from commit e93e401f492c7fc112da3d0c63b3b186c1b196ec)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/key/kms/KMSClientProvider.java
 2018-10-12 12:08:26 -07:00
Inigo Goiri 6342a7cb98 HDFS-13802. RBF: Remove FSCK from Router Web UI. Contributed by Fei Hui. 
(cherry picked from commit f63ee083db)
 2018-10-12 10:11:24 -07:00
Vinayakumar B bed2f407a9 HDFS-13945. TestDataNodeVolumeFailure is Flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 6e0e6daaf3)
 2018-10-12 17:36:03 +05:30
Hrishikesh Gadre 323b76bccf HDFS-13926. ThreadLocal aggregations for FileSystem.Statistics are incorrect with striped reads. 
Contributed by Xiao Chen, Hrishikesh Gadre.

Signed-off-by: Xiao Chen <xiao@apache.org>
 2018-10-09 19:54:34 -07:00
Todd Lipcon 89d4481020 HADOOP-15550. Avoid static initialization of ObjectMappers 
(cherry picked from commit 7a3c6e9c3c)
 2018-10-09 13:50:00 -05:00
Yiqun Lin 665036c5f7 HDFS-13962. Add null check for add-replica pool to avoid lock acquiring. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 1043795f7f)
 2018-10-09 10:36:45 +08:00
Inigo Goiri 9bbf494c7f HDFS-13964. RBF: TestRouterWebHDFSContractAppend fails with No Active Namenode under nameservice. Contributed by Ayush Saxena. 
(cherry picked from commit 3bfff0f79e)
 2018-10-06 10:41:49 -07:00
Yiqun Lin dd70adf318 HDFS-13957. Fix incorrect option used in description of InMemoryAliasMap. 
(cherry picked from commit 619e490333)
 2018-10-05 10:08:28 +08:00
Yiqun Lin 65af98b58a HDFS-13768. Adding replicas to volume map makes DataNode start slowly. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 5689355783)
 2018-10-02 09:46:23 +08:00
Brahma Reddy Battula 02f8b5da47 HDFS-13840. RBW Blocks which are having less GS should be added to Corrupt. Contributed by Brahma Reddy Battula 
(cherry picked from commit e0ff8e2c10)
 2018-09-26 23:51:59 +05:30
Brahma Reddy Battula 97b75f47fd HDFS-13790. RBF: Move ClientProtocol APIs to its own module Contributed by Chao Sun. 2018-09-26 22:40:48 +05:30
Chen Liang b4069343bb HADOOP-15726. Create utility to limit frequency of log statements. Contributed by Erik Krogen. 2018-09-19 14:28:35 -07:00
Arpit Agarwal 58f339e491 HADOOP-15772. Remove the 'Path ... should be specified as a URI' warnings on startup. Contributed by Ayush Saxena. 
(cherry picked from commit 2eb597b151)
 2018-09-19 13:49:06 -07:00
Inigo Goiri 2f7222a27b HADOOP-15684. triggerActiveLogRoll stuck on dead name node, when ConnectTimeoutException happens. Contributed by Rong Tang. 
(cherry picked from commit 7f9a89e1b54a9712af50ffef70bed7cfb91ed34e)
 2018-09-19 12:59:16 -07:00
Inigo Goiri e1af3c9bb7 HDFS-13908. TestDataNodeMultipleRegistrations is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 56e0d635e0)
 2018-09-19 09:31:18 -07:00
Wei-Chiu Chuang 7442b7941d HDFS-13868. WebHDFS: GETSNAPSHOTDIFF API NPE when param "snapshotname" is given but "oldsnapshotname" is not. Contributed by Pranay Singh. 
(cherry picked from commit 28ceb34a72)
 2018-09-19 03:12:55 -07:00
Wei-Chiu Chuang 595ce94570 HDFS-13886. HttpFSFileSystem.getFileStatus() doesn't return "snapshot enabled" bit. Contributed by Siyao Meng. 
(cherry picked from commit 44857476fa)
 2018-09-18 15:40:10 -07:00
Mingliang Liu bff2230591 HDFS-11719. Arrays.fill() wrong index in BlockSender.readChecksum() exception handling. Contributed by Tao Zhang 2018-09-18 10:40:16 -07:00
Inigo Goiri 508fd55c79 HDFS-13844. Fix the fmt_bytes function in the dfs-dust.js. Contributed by yanghuafeng. 
(cherry picked from commit a9c1c55cafc0338bb35d2991f217d4157463f1d3)
 2018-09-17 14:38:08 -07:00
Wei-Chiu Chuang 829399a9f3 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit a1de8cbac5)
 2018-09-14 05:24:24 +08:00
Inigo Goiri 2f7d226589 HDFS-13914. Fix DN UI logs link broken when https is enabled after HDFS-13902. Contributed by Jianfei Jiang. 
(cherry picked from commit e1b242a984)
 2018-09-13 09:17:37 -07:00
Brahma Reddy Battula 74c00641f9 HDFS-13902. Add JMX, conf and stacks menus to the datanode page. Contributed by fengchuang. 
(cherry picked from commit b2432d254c)
 2018-09-12 06:34:35 +05:30
Xiao Chen 2dd27c999b HDFS-13051. Fix dead lock during async editlog rolling if edit queue is full. Contributed by Daryn Sharp. 
(cherry picked from commit 8e54da1511)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLogRace.java
 2018-09-10 22:44:48 -07:00
Brahma Reddy Battula 77dd456461 HDFS-13237. [Documentation] RBF: Mount points across multiple subclusters. Contributed Íñigo Goiri 
(cherry picked from commit 96892c469b)
 2018-09-11 10:13:35 +05:30
Vinayakumar B b8eaeca7e6 HDFS-13895. EC: Fix Intermittent Failure in TestDFSStripedOutputStreamWithFailureWithRandomECPolicy. Contributed by Ayush Saxena. 
(cherry picked from commit eef3bafae8)
 2018-09-10 09:24:08 +05:30
Brahma Reddy Battula cbcdaefa37 HDFS-13862. RBF: Router logs are not capturing few of the dfsrouteradmin commands. Contributed by Ayush Saxena 
(cherry picked from commit 8944407d93)
 2018-09-09 23:21:45 +05:30
Inigo Goiri a400ed3aab HDFS-13836. RBF: Handle mount table znode with null value. Contributed by yanghuafeng. 
(cherry picked from commit 527288ef89)
 2018-09-06 16:48:44 -07:00
Wei-Chiu Chuang d1ea7df43d HADOOP-15696. KMS performance regression due to too many open file descriptors after Jetty migration. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e780556ae9)

Conflicts:
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/http/TestHttpServer.java
 2018-09-05 10:17:58 -07:00
Yiqun Lin c898757f55 HDFS-13815. RBF: Add check to order command. Contributed by Ranith Sardar. 
(cherry picked from commit 9315db5f5d)
 2018-09-05 23:35:30 +08:00
Hrishikesh Gadre fde3b5ac22 HDFS-13812. Fix the inconsistent default refresh interval on Caching documentation. Contributed by Hrishikesh Gadre. 
Signed-off-by: Xiao Chen <xiao@apache.org>
(cherry picked from commit 6ccb809c2d)
 2018-09-04 21:54:43 -07:00
Inigo Goiri a26565960a HDFS-13857. RBF: Choose to enable the default nameservice to read/write files. Contributed by yanghuafeng. 
(cherry picked from commit 54f2044595)
 2018-09-04 12:17:49 -07:00
Vinayakumar B 9cf35d99b9 HDFS-13774. EC: 'hdfs ec -getPolicy' is not retrieving policy details when the special REPLICATION policy set on the directory. Contributed by Ayush Saxena. 
(cherry picked from commit 3801436e49)
 2018-09-03 14:39:11 +05:30
Vinayakumar B 6776db5aec HDFS-13867. RBF: Add validation for max arguments for Router admin ls, clrQuota, setQuota, rm and nameservice commands. Contributed by Ayush Saxena. 
(cherry picked from commit 780df9034f)
 2018-09-03 14:29:28 +05:30
Wei-Chiu Chuang 9f0cd2ade2 Revert "HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng." 
This reverts commit 41732e7b28.
 2018-08-30 11:58:42 -07:00
Vinayakumar B 65977e5d81 HDFS-13027. Handle possible NPEs due to deleted blocks in race condition. Contributed by Vinayakumar B. 2018-08-30 22:14:47 +05:30
Yiqun Lin 8a3be0d5a4 HDFS-13863. FsDatasetImpl should log DiskOutOfSpaceException. Contributed by Fei Hui. 
(cherry picked from commit 582cb10ec7)
 2018-08-30 11:22:57 +08:00
Yiqun Lin b9b2b97e9d HDFS-13634. RBF: Configurable value in xml for async connection request queue size. Contributed by CR Hota. 
(cherry picked from commit a0ebb6b39f)
 2018-08-29 16:18:15 +08:00
Brahma Reddy Battula c01c240f6f HDFS-13854. RBF: The ProcessingAvgTime and ProxyAvgTime should display by JMX with ms unit. Contributed by yanghuafeng. 
(cherry picked from commit 64ad0298d4)
 2018-08-29 08:31:19 +05:30
Zsolt Venczel 5cbb9b1ca9 HDFS-13731. ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints. Contributed by Zsolt Venczel. 
(cherry picked from commit 3e18b957eb)
 2018-08-28 15:14:13 -07:00
Brahma Reddy Battula 450ba6790d HDFS-13861. RBF: Illegal Router Admin command leads to printing usage for all commands. Contributed by Ayush Saxena. 
(cherry picked from commit cb9d371ae2)
 2018-08-29 00:31:16 +05:30
Yiqun Lin 2cbc3c7d43 HDFS-13831. Make block increment deletion number configurable. Contributed by Ryan Wu. 
(cherry picked from commit b9b964d253)
 2018-08-28 09:36:02 -07:00
Vinayakumar B 9265d1a61a HDFS-13858. RBF: Add check to have single valid argument to safemode command. Contributed by Ayush Saxena. 
(cherry picked from commit 75691ad600)
 2018-08-28 09:21:42 +05:30
Wei-Chiu Chuang 41732e7b28 HDFS-13838. WebHdfsFileSystem.getFileStatus() won't return correct "snapshot enabled" status. Contributed by Siyao Meng. 
(cherry picked from commit 26c2a97c56)
 2018-08-27 16:03:52 -07:00
Konstantin V Shvachko 60fd202a59 HDFS-13848. Refactor NameNode failover proxy providers. Contributed by Konstantin Shvachko. 
(cherry picked from commit a4121c71c2)
 2018-08-24 18:44:04 -07:00
Yiqun Lin 0d73497aa7 HDFS-13821. RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache. Contributed by Fei Hui. 
(cherry picked from commit 81847392ba)
 2018-08-22 13:05:23 +08:00
Vinayakumar B 8df2eb8119 HDFS-13772. Erasure coding: Unnecessary NameNode Logs displaying for Enabling/Disabling Erasure coding policies which are already enabled/disabled. Contributed by Ayush Saxena 2018-08-21 09:34:57 +05:30
Wei-Chiu Chuang 865650052b HDFS-10240. Race between close/recoverLease leads to missing block. Contributed by Jinglun, zhouyingchao and Wei-Chiu Chuang. 2018-08-20 14:57:40 -07:00
Xiao Chen 0424715207 HDFS-13732. ECAdmin should print the policy name when an EC policy is set. Contributed by Zsolt Venczel. 2018-08-20 14:57:07 -07:00
Xiao Chen 81fbfe500e HDFS-13217. Audit log all EC policy names during addErasureCodingPolicies. Contributed by liaoyuxiangqin. 2018-08-20 14:56:47 -07:00
Daniel Templeton c15853f87a HDFS-13819. TestDirectoryScanner#testDirectoryScannerInFederatedCluster is flaky 
Change-Id: I1cea6e67fcec72702ad202775dee3373261ac5cd
 2018-08-20 14:54:58 -07:00
drankye 975d60685e HDFS-13668. FSPermissionChecker may throws AIOOE when check inode permission. Contributed by He Xiaoqiao. 2018-08-20 14:53:32 -07:00
Xiao Chen c0ac0a5337 HDFS-13658. Expose HighestPriorityLowRedundancy blocks statistics. Contributed by Kitti Nanasi. 2018-08-20 14:52:28 -07:00
Márton Elek 366517ba99 HDFS-13447. Fix Typos - Node Not Chosen. Contributed by Beluga Behr. 2018-08-20 14:52:04 -07:00
Vinayakumar B a517ee4f5b HDFS-13785. EC: 'removePolicy' is not working for built-in/system Erasure Code policies. Contributed by Ayush Saxena 2018-08-20 14:51:53 -07:00
Xiao Chen 754069906b HDFS-13799. TestEditLogTailer#testTriggersLogRollsForAllStandbyNN fails due to missing synchronization 
between rollEditsRpcExecutor and tailerThread shutdown. Contributed by Hrishikesh Gadre.
 2018-08-20 14:51:07 -07:00
Wei-Chiu Chuang 78fb14ba49 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:24 -07:00
Yiqun Lin d7442c244f HDFS-13750. RBF: Router ID in RouterRpcClient is always null. Contributed by Takanobu Asanuma. 
(cherry picked from commit 01ff817814)
 2018-08-20 23:05:55 +08:00
Xiao Chen 06f0d5e257 HDFS-13747. Statistic for list_located_status is incremented incorrectly by listStatusIterator. Contributed by Antal Mihalyi. 
(cherry picked from commit c67b0650ea)
 2018-08-16 23:15:48 -07:00
Daniel Templeton fbedf89377 HDFS-13746. Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh 
(Contributed by Siyao Meng via Daniel Templeton)

Change-Id: I9fad1537ace38367a463d9fe67aaa28d3178fc69
(cherry picked from commit 8512e1a91b)
 2018-08-16 15:01:34 -07:00
Xiao Chen 83d0d82a69 HDFS-13788. Update EC documentation about rack fault tolerance. Contributed by Kitti Nanasi. 
(cherry picked from commit cede33997f)
 2018-08-14 11:57:50 -07:00
Wei-Chiu Chuang 7a115e8de4 HDFS-13758. DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction. Contributed by chencan. 
(cherry picked from commit 61a9b4f58b)
 2018-08-14 11:52:24 -07:00
Wei-Chiu Chuang 74d4fd975a HDFS-13738. fsck -list-corruptfileblocks has infinite loop if user is not privileged. Contributed by Yuen-Kuei Hsueh. 
(cherry picked from commit 4023eeba05)
 2018-08-13 17:37:12 -07:00
Wei-Chiu Chuang b38649c59a HDFS-13813. Exit NameNode if dangling child inode is detected when saving FsImage. Contributed by Siyao Meng. 
(cherry picked from commit 23854443ef)
 2018-08-13 16:13:36 -07:00
Vinayakumar B b3f29e3504 HDFS-13786. EC: Display erasure coding policy for sub-directories is not working. Contributed by Ayush Saxena. 
(cherry picked from commit 2b0f977241)
 2018-08-13 13:58:27 -07:00
Chen Liang e05b38f132 HDFS-13735. Make QJM HTTP URL connection timeout configurable. Contributed by Chao Sun. 2018-08-13 12:48:12 -07:00
Arpit Agarwal 4e197a4b46 HDFS-13823. NameNode UI : "Utilities -> Browse the file system -> open a file -> Head the file" is not working. Contributed by Nanda kumar. 2018-08-13 11:36:15 -07:00
Chao Sun baca64cc68 Remove super user privilege requirement for NameNode.getServiceStatus. Contributed by Chao Sun. 
(cherry picked from commit a2a8c48699)
 2018-08-10 16:07:02 -07:00
Xiao Chen 7d1e2e1fbb HDFS-13721. NPE in DataNode due to uninitialized DiskBalancer. 2018-08-10 13:53:34 -07:00
Wangda Tan 781620a44d Make 3.1.1 awared by other branches 
(cherry picked from commit 49c687608b)
 2018-08-08 13:04:05 -07:00
Xiao Chen bf03b25f4b HDFS-13728. Disk Balancer should not fail if volume usage is greater than capacity. Contributed by Stephen O'Donnell. 
(cherry picked from commit 6677717c68)
 2018-08-07 22:05:51 -07:00
Wangda Tan 5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00
Brahma Reddy Battula 6f6d9ee9bf HDFS-12716. 'dfs.datanode.failed.volumes.tolerated' to support minimum number of volumes to be available. Contributed by Ranith Sardar and usharani 
(cherry picked from commit 3108d27edd)
 2018-07-30 15:57:58 +05:30
Arpit Agarwal 0eb187f72b HDFS-13765. Fix javadoc for FSDirMkdirOp#createParentDirectories. Contributed by Lokesh Jain. 2018-07-27 10:14:20 -07:00
Anu Engineer 2ebd553072 HDFS-13727. Log full stack trace if DiskBalancer exits with an unhandled exception. 
Contributed by Gabor Bota.

(cherry picked from commit 64e739e344)
 2018-07-27 09:43:44 -07:00
Wei-Chiu Chuang 964f3454d1 HDFS-11060. make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable. Contributed by Lantao Jin. 
(cherry picked from commit e95c5e9f62)
 2018-07-25 11:05:18 -07:00
Daniel Templeton 00c476abd8 HDFS-13448. HDFS Block Placement - Ignore Locality for First Block Replica 
(Contributed by BELUGA BEHR via Daniel Templeton)

Change-Id: I965d1cfa642ad24296038b83e3d5c9983545267d
(cherry picked from commit 849c45db18)
 2018-07-24 16:12:43 -07:00
Yiqun Lin 5aca0588ea HDFS-13583. RBF: Router admin clrQuota is not synchronized with nameservice. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 17a87977f2)
 2018-07-24 11:19:21 +08:00
Yiqun Lin 823d576a66 HDFS-13743. RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver. Contributed by Takanobu Asanuma. 
(cherry picked from commit 7b25fb949b)
 2018-07-20 17:31:13 +08:00
Yiqun Lin 228508a0ee HDFS-13733. RBF: Add Web UI configurations and descriptions to RBF document. Contributed by Takanobu Asanuma. 
(cherry picked from commit 1af87df242)
 2018-07-17 10:47:07 +08:00
Konstantin V Shvachko 53c7d82d53 Fix potential FSImage corruption. Contributed by Ekanth Sethuramalingam & Arpit Agarwal. 
(cherry picked from commit 0a1e922f3d)
 2018-07-16 18:29:43 -07:00
Wei-Chiu Chuang ac9155d6cb HDFS-13524. Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize. Contributed by Siyao Meng. 
(cherry picked from commit 88b2794244)
 2018-07-16 10:52:56 -07:00
Inigo Goiri 4898edf4f7 HDFS-13475. RBF: Admin cannot enforce Router enter SafeMode. Contributed by Chao Sun. 
(cherry picked from commit 359ea4e181)
 2018-07-16 09:47:00 -07:00
Xiao Chen d5d987550a HDFS-12837. Intermittent failure in TestReencryptionWithKMS. 
(cherry picked from commit b37074be5a)
 2018-07-11 21:03:45 -07:00
Akira Ajisaka fe256a98ff
HDFS-13729. Fix broken links to RBF documentation. Contributed by Gabor Bota. 
(cherry picked from commit 418cc7f3ae)
 2018-07-11 14:47:21 -04:00
Wei-Chiu Chuang f9fa3cb157 HDFS-13723. Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh. Contributed by Siyao Meng. 
(cherry picked from commit 162228e8db)
 2018-07-11 10:04:07 -07:00
Yiqun Lin 99e98bf19a HDFS-13726. RBF: Fix RBF configuration links. Contributed by Takanobu Asanuma. 
(cherry picked from commit 2ae13d41dc)
 2018-07-11 22:15:07 +08:00
Andrew Wang 242b5acdb3 HDFS-13719. Docs around dfs.image.transfer.timeout are misleading. Contributed by Kitti Nansi. 
(cherry picked from commit eecb5baaaa)
 2018-07-09 15:17:34 +02:00
Yiqun Lin e994c4f8aa HDFS-13710. RBF: setQuota and getQuotaUsage should check the dfs.federation.router.quota.enable. Contributed by yanghuafeng. 
(cherry picked from commit 43f7fe8aae)
 2018-07-09 15:07:47 +08:00
Wei-Chiu Chuang f5f4d0b7e7 HDFS-13121. NPE when request file descriptors when SC read. Contributed by Zsolt Venczel. 
(cherry picked from commit 0247cb6318)
 2018-07-06 15:01:10 -07:00
Yiqun Lin 48addcdbbe HDFS-13528. RBF: If a directory exceeds quota limit then quota usage is not refreshed for other mount entries. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 3b637155a4)
 2018-07-04 15:30:17 +08:00
Andrew Wang 3b266abdad HDFS-13712. BlockReaderRemote.read() logging improvement. Contributed by Gergo Repas. 
(cherry picked from commit 344f324710)
 2018-07-03 11:07:57 +02:00
Bharat Viswanadham 0d7b811ee6 HDFS-13715:diskbalancer does not work if one of the blockpools are empty on a Federated cluster. Contributed by Bharat Viswanadham 
(cherry picked from commit 59a3038bc3)
 2018-07-02 21:48:09 -07:00
Inigo Goiri 177ccbc43f HDFS-13714. Fix TestNameNodePrunesMissingStorages test failures on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 7296b644f7)
 2018-07-02 16:37:12 -07:00
Wei-Chiu Chuang 2ce892fcf5 HDFS-13635. Incorrect message when block is not found. Contributed by Gabor Bota. 
(cherry picked from commit fef20a446f)
 2018-07-02 09:43:44 -07:00
Andrew Wang 2d2639beaa HDFS-13702. Remove HTrace hooks from DFSClient to reduce CPU usage. Contributed by Todd Lipcon. 
(cherry picked from commit 5d748bd056)
 2018-07-02 12:11:11 +02:00
Andrew Wang 0268b954e0 HDFS-13703. Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit. Contributed by Todd Lipcon. 
(cherry picked from commit 6ba9974108)
 2018-07-02 12:02:48 +02:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Yiqun Lin 6b5b58a495 HDFS-13692. StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet. Contributed by Bharat Viswanadham. 
(cherry picked from commit 30728aced4)
 2018-06-22 10:53:33 +08:00
Xiao Chen e8655b8c72 HDFS-13682. Cannot create encryption zone after KMS auth token expires. 
(cherry picked from commit 32f867a6a9)
 2018-06-20 15:59:23 -07:00
Inigo Goiri 88e8a3e5bc HDFS-13681. Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 8762e9cf10)
 2018-06-15 16:49:51 -07:00
Inigo Goiri 7c06be3912 HDFS-13686. Add overall metrics for FSNamesystemLock. Contributed by Lukas Majercak. 
(cherry picked from commit d31a3ce767)
 2018-06-15 15:59:49 -07:00
Wei-Chiu Chuang 975d4b3d60 HDFS-13174. hdfs mover -p /path times out after 20 min. Contributed by Istvan Fajth. 
(cherry picked from commit c966a3837a)
 2018-06-15 13:37:19 -07:00
Inigo Goiri 271ea7a3d9 HDFS-13676. TestEditLogRace fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit eebeb6033f)
 2018-06-15 13:08:24 -07:00
Inigo Goiri 4add4c1be0 HDFS-13673. TestNameNodeMetrics fails on Windows. Contributed by Zuoming Zhang. 
(cherry picked from commit 43d994e4a6)
 2018-06-15 10:34:08 -07:00
Inigo Goiri fd7cd596b7 HDFS-13675. Speed up TestDFSAdminWithHA. Contributed by Lukas Majercak. 
(cherry picked from commit 5d7449d2b8)
 2018-06-14 13:43:51 -07:00
Inigo Goiri e45541be51 HDFS-13563. TestDFSAdminWithHA times out on Windows. Contributed by Lukas Majercak. 
(cherry picked from commit 8d4926f38b)
 2018-06-14 09:59:28 -07:00
Yiqun Lin 5c1248b2da HDFS-13641. Add metrics for edit log tailing. Contributed by Chao Sun. 
(cherry picked from commit 8e7548d33b)
 2018-06-13 20:10:46 +08:00
Xiao Chen 3b646a9836 HDFS-13511. Provide specialized exception when block length cannot be obtained. Contributed by Gabor Bota. 
(cherry picked from commit 774c1f199e)
 2018-06-12 11:17:54 -07:00
Rohith Sharma K S af0309d793 YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace0)
 2018-06-12 17:23:22 +05:30
Yongjun Zhang 425fe4e217 Update CHANGES, RELEASENOTES, and jdiff for 3.0.3 release. 
(cherry picked from commit 5a82f10e32)
 2018-06-11 14:31:46 -07:00
Inigo Goiri f0150f024f HDFS-13653. Make dfs.client.failover.random.order a per nameservice configuration. Contributed by Ekanth Sethuramalingam. 
(cherry picked from commit 784dbbd2d1fc0c361fee851d3f9b3aa50be5481d)
 2018-06-11 11:17:37 -07:00
Bharat Viswanadham c0d46a84a6 HDFS-13667:Typo: Marking all datandoes as stale. Contributed by Nanda Kumar 
(cherry picked from commit ccfb816d39)
 2018-06-09 16:43:58 -07:00
Arpit Agarwal 33b6a68c81 HDFS-12670. can't renew HDFS tokens with only the hdfs client jar. Contributed by Arpit Agarwal. 2018-06-08 17:57:30 -07:00
Chao Sun d827ffe245 HDFS-13664. Refactor ConfiguredFailoverProxyProvider to make inheritance easier. Contributed by Chao Sun. 
(cherry picked from commit fba1c42adc)
 2018-06-08 16:40:13 -07:00
Wei-Chiu Chuang b2e919dc1a HDFS-13659. Add more test coverage for contentSummary for snapshottable path. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e39b113db0)
 2018-06-07 08:31:30 -07:00
Brahma Reddy Battula 630f43faf7 HDFS-12950. [oiv] ls will fail in secure cluster. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 9654dd1f47)
 2018-06-07 10:21:00 +05:30
Xiao Chen f1c91e9fda HADOOP-15217. FsUrlConnection does not handle paths with spaces. Contributed by Joseph Fourny and Zsolt Venczel. 
(cherry picked from commit ba4011d64f)
 2018-06-05 21:19:11 -07:00
Akira Ajisaka f358a77610
HDFS-13545. "guarded" is misspelled as "gaurded" in FSPermissionChecker.java. Contributed by Jianchao Jia. 
(cherry picked from commit 0e3c31579d)
 2018-06-05 17:55:01 +09:00
Inigo Goiri ec5ef0bb20 HDFS-13650. Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy. Contributed by Anbang Hu. 
(cherry picked from commit 8d31ddcfeb)
 2018-06-04 18:28:40 -07:00
Inigo Goiri 19b37e7080 HDFS-13649. Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy. Contributed by Anbang Hu. 
(cherry picked from commit 16316b6011)
 2018-06-04 18:22:09 -07:00
Inigo Goiri 76546d23e2 HDFS-13652. Randomize baseDir for MiniDFSCluster in TestBlockScanner. Contributed by Anbang Hu. 
(cherry picked from commit a28848104f)
 2018-06-04 18:13:14 -07:00
Rushabh Shah 6646b16450 HDFS-13281 Namenode#createFile should be /.reserved/raw/ aware.. Contributed by Rushabh S Shah 
(cherry picked from commit e2289c8d14)
 2018-06-04 09:21:14 -05:00
Wei-Chiu Chuang 302c36ffdc HDFS-13155. BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL. Contributed by Zsolt Venczel. 
(cherry picked from commit bccdfeee0a)
 2018-06-04 07:03:35 -07:00
Xiao Chen fde45994a3 HDFS-13339. Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume. Contributed by liaoyuxiangqin and Zsolt Venczel. 
(cherry picked from commit 9efb4b7db0)
 2018-06-03 22:37:27 -07:00
Inigo Goiri 90060d52f6 HDFS-13648. Fix TestGetConf#testGetJournalNodes on Windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 8f83b9abf9)
 2018-06-02 20:20:26 -07:00
Inigo Goiri 8a9c2d7b97 HDFS-13651. TestReencryptionHandler fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit e308ac88d3)
 2018-06-02 20:13:21 -07:00
Inigo Goiri fdacc8a088 HDFS-13637. RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE. Contributed by CR Hota. 
(cherry picked from commit e11d674049)
 2018-06-01 16:59:35 -07:00
Sean Mackrory 58b5b3aa75 HDFS-13582. Improve backward compatibility for HDFS-13176 (WebHdfs file path gets truncated when having semicolon (;) inside). Contributed by Zsolt Venczel. 
(cherry picked from commit 1361030e59)
 2018-05-31 22:23:19 -06:00
Chao Sun 2bbdf362cf HDFS-13602. Add checkOperation(WRITE) checks in FSNamesystem. Contributed by Chao Sun. 
(cherry picked from commit ff013d2c95)
 2018-05-31 17:39:46 -07:00
Konstantin V Shvachko 442dd87dcd HDFS-12978. Fine-grained locking while consuming journal stream. Contributed by Konstantin Shvachko. 2018-05-31 14:58:06 -07:00
Inigo Goiri ad11e40557 HDFS-13631. TestDFSAdmin#testCheckNumOfBlocksInReportCommand should use a separate MiniDFSCluster path. Contributed by Anbang Hu. 
(cherry picked from commit 1f324e9661)
 2018-05-31 10:13:14 -07:00
Inigo Goiri 5c09dadd73 HDFS-13629. Some tests in TestDiskBalancerCommand fail on Windows due to MiniDFSCluster path conflict and improper path usage. Contributed by Anbang Hu. 
(cherry picked from commit 47c31ff16b)
 2018-05-30 10:23:00 -07:00
Inigo Goiri 6dc40bab4e HDFS-13632. Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA. Contributed by Anbang Hu. 
(cherry picked from commit e1102865e6e16babaff9346212c824cb1f47c31c)
 2018-05-30 10:02:50 -07:00
Inigo Goiri bc80290f28 HDFS-13591. TestDFSShell#testSetrepLow fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9dbf4f0166)
 2018-05-28 16:34:31 -07:00
Inigo Goiri cc696f80d3 HDFS-13627. TestErasureCodingExerciseAPIs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 91d7c74e6a)
 2018-05-28 10:27:23 -07:00
Takanobu Asanuma 499a90f152
HDFS-13628. Update Archival Storage doc for Provided Storage 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 04757e5864)
 2018-05-28 19:06:58 +09:00
Inigo Goiri c7d77a3bfe HDFS-13620. Randomize the test directory path for TestHDFSFileSystemContract. Contributed by Anbang Hu. 
(cherry picked from commit 8605a38514)
 2018-05-25 19:45:48 -07:00
Inigo Goiri 71ab85adce HDFS-13619. TestAuditLoggerWithCommands fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 13d2528907)
 2018-05-25 13:33:05 -07:00
Inigo Goiri 2d3d7a67f2 HDFS-13618. Fix TestDataNodeFaultInjector test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit a8e9ee3fdf3aa1b889dfdd3bade91ba754ed9ee1)
 2018-05-25 09:11:07 -07:00
Andrew Wang 31d597f0c7 HDFS-13611. Unsafe use of Text as a ConcurrentHashMap key in PBHelperClient. 
(cherry picked from commit c9b63deb53)
 2018-05-24 09:56:37 -07:00
Inigo Goiri f57e91a348 HDFS-13493. Reduce the HttpServer2 thread count on DataNodes. Contributed by Erik Krogen. 
(cherry picked from commit cddbbe5f69)
 2018-05-23 12:12:40 -07:00
Inigo Goiri a138496379 HDFS-13587. TestQuorumJournalManager fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c13dea87d9)
 2018-05-23 11:36:34 -07:00
Bharat Viswanadham d0d3d5b2ad HDFS-13589: Add dfsAdmin command to query if upgrade is finalized. Contributed by Hanisha Koneru 
(cherry picked from commit bc6d9d4c79)
 2018-05-23 10:23:45 -07:00
Inigo Goiri 0c268a0755 HDFS-13588. Fix TestFsDatasetImpl test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit c0c9b7a8ef)
 2018-05-23 09:47:10 -07:00
Sammi Chen 7d71b3a1cc HDFS-13540. DFSStripedInputStream should only allocate new buffers when reading. Contributed by Xiao Chen. 
(cherry picked from commit 34e8b9f9a8)
 2018-05-23 19:12:04 +08:00
Andrew Wang d47c0fc39a HDFS-13601. Optimize ByteString conversions in PBHelper. 
(cherry picked from commit 1d2640b613)
 2018-05-22 23:55:35 -07:00
Inigo Goiri 8a86269c51 HDFS-13593. TestBlockReaderLocalLegacy#testBlockReaderLocalLegacyWithAppend fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 9775ecb235)
 2018-05-18 09:46:29 -07:00
Inigo Goiri da5eff8325 HDFS-13592. TestNameNodePrunesMissingStorages#testNameNodePrunesUnreportedStorages does not shut down cluster properly. Contributed by Anbang Hu. 
(cherry picked from commit 57b893de3d)
 2018-05-18 09:38:23 -07:00
Inigo Goiri df54cdbaea HDFS-13560. Insufficient system resources exist to complete the requested service for some tests on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 53b807a6a8)
 2018-05-17 17:03:56 -07:00
Inigo Goiri f67865ed7e HDFS-13556. TestNestedEncryptionZones does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit a97a2042f2)
 2018-05-17 16:53:53 -07:00
Inigo Goiri 8c463a2a9c HDFS-13554. TestDatanodeRegistration#testForcedRegistration does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 65476458fa)
 2018-05-17 14:48:40 -07:00
Inigo Goiri e5ff9f3922 HDFS-13558. TestDatanodeHttpXFrame does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 26f1e22fc9)
 2018-05-17 13:35:40 -07:00
Inigo Goiri b0c89690ed HDFS-13570. TestQuotaByStorageType,TestQuota,TestDFSOutputStream fail on Windows. Contributed by Anbang Hu. 
(cherry picked from commit c0ec061e28)
 2018-05-17 09:22:29 -07:00
Kihwal Lee 4f1ec6a515 HDFS-13581. DN UI logs link is broken when https is enabled. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0ce6290de6)
 2018-05-17 09:26:13 -05:00
Inigo Goiri 1d1806f862 HDFS-13559. TestBlockScanner does not close TestContext properly. Contributed by Anbang Hu. 
(cherry picked from commit 454de3b543)
 2018-05-16 19:23:33 -07:00
Arpit Agarwal 45dd200a2c HDFS-13512. WebHdfs getFileStatus doesn't return ecPolicy. Contributed by Ajay Kumar. 2018-05-16 11:28:51 -07:00
Inigo Goiri e12ddd9394 HDFS-13550.TestDebugAdmin#testComputeMetaCommand fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 55d5549087)
 2018-05-16 10:09:18 -07:00
Inigo Goiri 5310542fb4 HDFS-13557. TestDFSAdmin#testListOpenFiles fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 6b67161ca3)
 2018-05-16 09:32:15 -07:00
Inigo Goiri 899a7ec2fb HDFS-13567. TestNameNodeMetrics#testGenerateEDEKTime,TestNameNodeMetrics#testResourceCheck should use a different cluster basedir. Contributed by Anbang Hu. 
(cherry picked from commit 63480976a0)
 2018-05-15 15:36:03 -07:00
Inigo Goiri 34fbc1c2eb HDFS-13548. TestResolveHdfsSymlink#testFcResolveAfs fails on Windows. Contributed by Anbang Hu. 
(cherry picked from commit 2d6195c0af)
 2018-05-15 11:21:44 -07:00
Inigo Goiri c168329cc4 HDFS-11700. TestHDFSServerPorts#testBackupNodePorts doesn't pass on Windows. Contributed by Anbang Hu. 
(cherry picked from commit eaca7980ad)
 2018-05-15 10:28:01 -07:00
Inigo Goiri 583b9b418a HDFS-13551. TestMiniDFSCluster#testClusterSetStorageCapacity does not shut down cluster. Contributed by Anbang Hu. 
(cherry picked from commit 92ebd466c7)
 2018-05-15 10:22:19 -07:00
Hanisha Koneru 4f55941390 HDFS-13544. Improve logging for JournalNode in federated cluster. 
(cherry picked from commit 6beb25ab7e)
 2018-05-14 10:14:38 -07:00
Xiao Chen 6e731eb20d HDFS-13539. DFSStripedInputStream NPE when reportCheckSumFailure. 
(cherry picked from commit 960940e0e0)
 2018-05-14 09:29:14 -07:00
Sammi Chen e5281d9f8b Add 2.9.1 release notes and changes documents 2018-05-14 15:26:00 +08:00
Inigo Goiri e332e011ae HDFS-13542. TestBlockManager#testNeededReplicationWhileAppending fails due to improper cluster shutdown in TestBlockManager#testBlockManagerMachinesArray on Windows. Contributed by Anbang Hu. 
(cherry picked from commit d50c4d71dc)
 2018-05-11 09:48:45 -07:00
Yiqun Lin 3665217d50 HDFS-13346. RBF: Fix synchronization of router quota and nameservice quota. 
(cherry picked from commit a922b9c82c)
 2018-05-11 15:09:54 +08:00
Inigo Goiri e3ff9e36df HDFS-13537. TestHdfsHelper does not generate jceks path properly for relative path in Windows. Contributed by Xiao Liang. 
(cherry picked from commit 914b98a713)
 2018-05-09 16:52:45 -07:00
Inigo Goiri c649c76566 HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun. 
(cherry picked from commit 63803e7051)
 2018-05-09 13:56:06 -07:00
Xiao Chen 6f3bbf364c HDFS-12981. renameSnapshot a Non-Existent snapshot to itself should throw error. Contributed by Kitti Nanasi. 
(cherry picked from commit 696a4be0da)
 2018-05-07 15:39:24 -07:00
Xiao Chen d762932d79 Revert "HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445." 
This reverts commit 128b09ee43.
 2018-05-07 13:47:52 -07:00
Inigo Goiri 524dde4006 HDFS-13525. RBF: Add unit test TestStateStoreDisabledNameservice. Contributed by Yiqun Lin. 
(cherry picked from commit a3b416f69d)
 2018-05-03 11:25:30 -07:00
Yiqun Lin 5fef28d0d4 HDFS-13488. RBF: Reject requests when a Router is overloaded. Contributed by Inigo Goiri. 
(cherry picked from commit 37269261d1)
 2018-05-02 14:56:15 +08:00
Inigo Goiri 4aadfb3c92 HDFS-13503. Fix TestFsck test failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit 9e2cfb2d3f)
 2018-05-01 08:13:30 -07:00
Inigo Goiri 4c13e7e3a0 HDFS-13283. Percentage based Reserved Space Calculation for DataNode. Contributed by Lukas Majercak. 
(cherry picked from commit fc074a359c)
 2018-04-30 13:31:03 -07:00
Inigo Goiri ce62991648 HDFS-13509. Bug fix for breakHardlinks() of ReplicaInfo/LocalReplica, and fix TestFileAppend failures on Windows. Contributed by Xiao Liang. 
(cherry picked from commit eb7fe1d588)
 2018-04-28 09:05:59 -07:00
Inigo Goiri 9923fa2490 HDFS-13508. RBF: Normalize paths (automatically) when adding, updating, removing or listing mount table entries. Contributed by Ekanth S. 
(cherry picked from commit 484440602c)
 2018-04-27 16:29:41 -07:00
Inigo Goiri f26f764857 HDFS-13434. RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari. 
(cherry picked from commit f469628bba)
 2018-04-27 15:14:18 -07:00
Yiqun Lin e097998c22 HDFS-13499. RBF: Show disabled name services in the UI. Contributed by Inigo Goiri. 
(cherry picked from commit 0a293de00d)
 2018-04-27 10:39:21 +08:00
Inigo Goiri 921a3e1678 HDFS-13326. RBF: Improve the interfaces to modify and view mount tables. Contributed by Gang Li. 
(cherry picked from commit c394051a3d4d9d531f418503cb519606ae2b2e69)
 2018-04-26 13:01:16 -07:00
Yiqun Lin 999ea44b9f HDFS-13484. RBF: Disable Nameservices from the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 30fef0bf1e5c8c0ca073df99ad9b33cb0e4431a5)
 2018-04-25 15:23:54 +08:00
Yiqun Lin ea2863f2b5 HDFS-13490. RBF: Fix setSafeMode in the Router. Contributed by Inigo Goiri. 
(cherry picked from commit b06601acce)
 2018-04-24 11:27:14 +08:00
Inigo Goiri 5b2427cd73 HDFS-13336. Test cases of TestWriteToReplica failed in windows. Contributed by Xiao Liang. 
(cherry picked from commit df92a17e02)
 2018-04-23 19:12:53 -07:00
Bharat Viswanadham caf4d5000e HDFS-13356. Balancer:Set default value of minBlockSize to 10mb. Contributed by Bharat Viswanadham 
(cherry picked from commit 9b5375e0c1)
 2018-04-23 16:10:28 -07:00
Chris Douglas cf272c5179 HDFS-13408. MiniDFSCluster to support being built on randomized base directory. Contributed by Xiao Liang 
(cherry picked from commit f411de6a79)
 2018-04-23 11:13:52 -07:00
Arpit Agarwal 1f486a0649 HDFS-13433. webhdfs requests can be routed incorrectly in federated cluster. Contributed by Arpit Agarwal. 2018-04-23 10:09:02 -07:00
Yiqun Lin 3f72d077ff HDFS-13478. RBF: Disabled Nameservice store API. Contributed by Inigo Goiri. 
(cherry picked from commit b6dae26f44)
 2018-04-21 13:35:04 +08:00
Sangjin Lee 9f648f6e59 HDFS-10183. Prevent race condition during class initialization. Contributed by Pavel Avgustinov. 
(cherry picked from commit f40969a141)
 2018-04-20 20:36:31 -07:00
Inigo Goiri 18b6a2a2ee HDFS-13453. RBF: getMountPointDates should fetch latest subdir time/date when parent dir is not present but /parent/child dirs are present in mount table. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 1134af9ad1)
 2018-04-19 14:57:09 -07:00
Xiao Chen 128b09ee43 HDFS-13430. Fix TestEncryptionZonesWithKMS failure due to HADOOP-14445. 
(cherry picked from commit 6503593711)
 2018-04-18 22:29:59 -05:00
Shashikant Banerjee a13b3edfd8
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
 2018-04-19 08:36:32 +09:00
Bharat Viswanadham 63c1b9d50c HDFS-13464. Fix javadoc in FsVolumeList#handleVolumeFailures. Contributed by Shashikant Banerjee 
(cherry picked from commit bf7694d020)
 2018-04-18 08:24:52 -07:00
Mukul Kumar Singh a07ba66ee9 HDFS-13426. Fix javadoc in FsDatasetAsyncDiskService#removeVolume. Contributed by Shashikant Banerjee. 2018-04-18 16:30:40 +05:30
Mukul Kumar Singh d99c8aa898 HDFS-13463. Fix javadoc in FsDatasetImpl#checkAndUpdate. Contributed by Shashikant Banerjee. 2018-04-18 16:30:24 +05:30
Erik Krogen fb350e8112
HDFS-12828. OIV ReverseXML Processor fails with escaped characters 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 2d9e791a90)
 2018-04-18 14:40:01 +09:00
Wei Yan 5d50f716e7 HDFS-13466. RBF: Add more router-related information to the UI. 
(cherry picked from commit e4313e7e47)
 2018-04-17 15:03:12 -07:00
Inigo Goiri 4be0fa28da HDFS-13462. Add BIND_HOST configuration for JournalNode's HTTP and RPC Servers. Contributed by Lukas Majercak. 
(cherry picked from commit c9b33514b8)
 2018-04-17 14:20:23 -07:00
Bharat Viswanadham fc11555caf HDFS-13129. Add a test for DfsAdmin refreshSuperUserGroupsConfiguration. Contributed by Mukul Kumar Singh 
(cherry picked from commit a68b04370f)
 2018-04-17 12:41:54 -07:00
Yiqun Lin 1d776f22f9 HDFS-13435. RBF: Improve the error loggings for printing the stack trace. 
(cherry picked from commit c4d3636c21)
 2018-04-17 11:25:14 +08:00
Arpit Agarwal 43a2af5569 HDFS-13439. Add test case for read block operation when it is moved. Contributed by Ajay Kumar. 2018-04-16 14:18:41 -07:00
Xiao Chen 805e33b62c HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-16 11:58:10 -07:00
Wei-Chiu Chuang ca8bb322be HDFS-13330. ShortCircuitCache#fetchOrCreate never retries. Contributed by Gabor Bota. 
(cherry picked from commit e66e287efe)
 2018-04-13 12:18:49 -07:00
Wei-Chiu Chuang 696d786063 HDFS-11900. Hedged reads thread pool creation not synchronized. Contributed by John Zhuge. 2018-04-13 10:16:23 -07:00
Xiaoyu Yao 867135af28 HDFS-11043. TestWebHdfsTimeouts fails. Contributed by Xiaoyu Yao and Chao Sun. 2018-04-13 10:14:47 -07:00
Yongjun Zhang 56acbe476b HDFS-13315. Add a test for the issue reported in HDFS-11481 which is fixed by HDFS-10997. Contributed by Yongjun Zhang. 2018-04-13 10:14:24 -07:00
Chris Douglas ffc39ec3ab HADOOP-14667. Flexible Visual Studio support. Contributed by Allen Wittenauer 2018-04-13 10:11:35 -07:00
Bharat Viswanadham 8df8cad019 HDFS-13438. Fix javadoc in FsVolumeList#removeVolume. Contributed by Shashikant Banerjee. 
(cherry picked from commit fa8b88ab2b)
 2018-04-13 09:05:24 -07:00
Yiqun Lin 16763d8229 HDFS-13418. NetworkTopology should be configurable when enable DFSNetworkTopology. Contributed by Tao Jie. 
(cherry picked from commit 0725953efe)
 2018-04-13 17:57:15 +08:00
Akira Ajisaka 5531c31f16
HDFS-13436. Fix javadoc of package-info.java 
(cherry picked from commit 9b0b9f2af2)
 2018-04-13 13:24:42 +09:00
Inigo Goiri 671e407ced Revert "HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun." 
This reverts commit 8c9dab978e.
 2018-04-12 08:43:12 -07:00
Akira Ajisaka 5f4762d950
HDFS-7101. Potential null dereference in DFSck#doWork(). Contributed by skrho and Ted Yu. 
(cherry picked from commit 113af12cfb)
 2018-04-12 17:48:14 +09:00
Akira Ajisaka 2d82a21b84
HDFS-13427. Fix the section titles of transparent encryption document. 
(cherry picked from commit c7cd362afd)
 2018-04-12 10:46:14 +09:00
Inigo Goiri 3c1cd08f04 HDFS-13386. RBF: Wrong date information in list file(-ls) result. Contributed by Dibyendu Karmakar. 
(cherry picked from commit 18de6f2042)
 2018-04-11 12:03:40 -07:00
Inigo Goiri d032112e6b HDFS-13428. RBF: Remove LinkedList From StateStoreFileImpl.java. Contributed by BELUGA BEHR. 
(cherry picked from commit 6a881cd842fb4eab596a386b39670ea03536891a)
 2018-04-11 10:41:13 -07:00
Wei Yan 32efc77799 HDFS-13045. RBF: Improve error message returned from subcluster. Contributed by Inigo Goiri. 
(cherry picked from commit 0c93d43f3d)
 2018-04-11 08:38:52 -07:00
Rakesh Radhakrishnan 3414bf6dbb HDFS-13328. Abstract ReencryptionHandler recursive logic in separate class. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit f89594f0b8)
 2018-04-10 23:40:26 +05:30
Weiwei Yang b104de3f64 HDFS-13420. License header is displayed in ArchivalStorage/MemoryStorage html pages. Contributed by Akira Ajisaka. 2018-04-10 22:13:08 +08:00
Yiqun Lin 5eb32396d7 HDFS-13384. RBF: Improve timeout RPC call mechanism. Contributed by Inigo Goiri. 
(cherry picked from commit e87be8a2a4)
 2018-04-10 15:36:48 +08:00
Yiqun Lin 9a86cb0494 HDFS-13410. RBF: Support federation with no subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit a92200f4a6)
 2018-04-10 14:31:59 +08:00
Wangda Tan 25bf1d3ac6 Updated branch-3.1 to track 3.1.1 
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
 2018-04-09 10:58:50 -07:00
Inigo Goiri 1c9038bae5 HDFS-13380. RBF: mv/rm fail after the directory exceeded the quota limit. Contributed by Yiqun Lin. 
(cherry picked from commit e9b9f48dad)
 2018-04-09 10:09:55 -07:00
Inigo Goiri 8c9dab978e HDFS-13388. RequestHedgingProxyProvider calls multiple configured NNs all the time. Contributed by Jinglun. 
(cherry picked from commit ac32b3576d)
 2018-04-09 09:17:35 -07:00
Yiqun Lin ac5703207d HDFS-13402. RBF: Fix java doc for StateStoreFileSystemImpl. Contributed by Yiran Wu. 
(cherry picked from commit 5700556cd6)
 2018-04-08 12:03:35 +08:00
Wangda Tan a336561e2b Added CHANGES/RELEASES/Jdiff for 3.1.0 release 
Change-Id: Ied5067a996151c04d15cad46c46ac98b60c37b39
(cherry picked from commit 2d96570452a72569befdf9cfe9b90c9fa2e0e261)
 2018-04-05 15:52:02 -07:00
Wei Yan 89efe30f65 HDFS-13353. RBF: TestRouterWebHDFSContractCreate failed. Contributed by Takanobu Asanuma. 
(cherry picked from commit 3121e8c293)
 2018-04-05 13:13:16 -07:00
Lei Xu 2543a9dd77 HDFS-13350. Negative legacy block ID will confuse Erasure Coding to be considered as striped block. (Contributed by Lei (Eddy) Xu). 
(cherry picked from commit 1cbf23df145af01692b8aaa438642b64e330cd05)
 2018-04-05 10:01:05 -07:00
Yiqun Lin b89dc00302 HDFS-13364. RBF: Support NamenodeProtocol in the Router. Contributed by Inigo Goiri. 
(cherry picked from commit 2be64eb201)
 2018-04-03 15:23:39 +08:00
Xiao Chen dcb2176c5d HADOOP-15317. Improve NetworkTopology chooseRandom's loop. 
(cherry picked from commit 57374c4737)
 2018-04-02 22:48:54 -07:00
Inigo Goiri 3c9983547c HDFS-13289. RBF: TestConnectionManager#testCleanup() test case need correction. Contributed by Dibyendu Karmakar. 
(cherry picked from commit dc8e343201)
 2018-03-31 09:46:44 -07:00
Konstantin V Shvachko 4ce45a8ec7 HADOOP-15253. Should update maxQueueSize when refresh call queue. Contributed by Tao Jie. 
(cherry picked from commit acfd764fcc)
 2018-03-30 18:57:18 -07:00
Xiao Chen 2960592c6f HDFS-13087. Snapshotted encryption zone information should be immutable. Contributed by LiXin Ge. 
(cherry picked from commit 2c6cfad5a3)
 2018-03-29 16:07:01 -07:00
Arpit Agarwal d54e19a913 HDFS-13314. NameNode should optionally exit if it detects FsImage corruption. Contributed by Arpit Agarwal. 2018-03-28 12:49:14 -07:00
Yiqun Lin 99766992e6 HDFS-13347. RBF: Cache datanode reports. Contributed by Inigo Goiri. 
(cherry picked from commit a71656c1c1)
 2018-03-28 11:04:36 +08:00
Inigo Goiri c1fc87dcce HDFS-13352. RBF: Add xsl stylesheet for hdfs-rbf-default.xml. Contributed by Takanobu Asanuma. 
(cherry picked from commit 4cef01147f)
 2018-03-27 10:32:07 -07:00
Inigo Goiri 0dc16559ee HDFS-13204. RBF: Optimize name service safe mode icon. Contributed by liuhongtong. 
(cherry picked from commit 947b282984)
 2018-03-26 09:43:28 -07:00
Yiqun Lin 1e01ddfe4f HDFS-13291. RBF: Implement available space based OrderResolver. Contributed by Yiqun Lin. 
(cherry picked from commit cfc3a1c8f0)
 2018-03-26 18:36:22 +08:00
Lei Xu 4859cd7cc9 Update 3.0.1 jdiff file and jdiff stable api version 2018-03-23 11:53:30 -07:00
weiy d17aa8d052 HDFS-12512. RBF: Add WebHDFS. 
(cherry picked from commit 6e31a09084)
 2018-03-23 08:40:04 -07:00
Inigo Goiri 59ab86311c HDFS-12792. RBF: Test Router-based federation using HDFSContract. Contributed by Inigo Goiri. 
(cherry picked from commit e196d158a2)
 2018-03-22 13:33:24 -07:00
Kihwal Lee c245ecedda HDFS-13195. DataNode conf page cannot display the current value after reconfig. Contributed by maobaolong. 
(cherry picked from commit 66461ed98b)
 2018-03-22 12:28:34 -05:00
Inigo Goiri 485348a337 HDFS-13318. RBF: Fix FindBugs in hadoop-hdfs-rbf. Contributed by Ekanth S. 
(cherry picked from commit 3d0ee8b804)
 2018-03-22 09:22:16 -07:00
Konstantin V Shvachko 21db4e9efe HDFS-12884. BlockUnderConstructionFeature.truncateBlock should be of type BlockInfo. Contributed by chencan. 
(cherry picked from commit 8d898ab25f)
 2018-03-21 16:51:33 -07:00
Yiqun Lin 0db4b587c6 HDFS-13250. RBF: Router to manage requests across multiple subclusters. Contributed by Inigo Goiri. 
(cherry picked from commit 2caba999bb)
 2018-03-21 11:40:02 +08:00
Yiqun Lin 131546002d HDFS-13307. RBF: Improve the use of setQuota command. Contributed by liuhongtong. 
(cherry picked from commit 69fe4407eb)
 2018-03-21 10:53:24 +08:00