Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,851 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7641 Commits

Author SHA1 Message Date
Ayush Saxena 9956d18129 HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:52:12 +05:30
Ayush Saxena 98e3c6af2c HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:37 +05:30
Duo Zhang 07f59c023c HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:30:34 -07:00
Jonathan Hung 4d274f60bc Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aa)
 2019-10-30 21:00:55 -07:00
Eric Badger 50c1f932c3 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2019-10-28 16:54:05 +00:00
Rakesh Radhakrishnan 2db0b8f6ca HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d)
 2019-10-27 22:24:44 +05:30
Rakesh Radhakrishnan cd1277398a HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6f)
 2019-10-27 22:24:30 +05:30
Rakesh Radhakrishnan f2563dcca1 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1)
 2019-10-27 22:24:14 +05:30
Rakesh Radhakrishnan df0dcc7493 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7)
 2019-10-27 22:24:00 +05:30
Sammi Chen 9281b72550 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad44490)
 2019-10-27 22:23:44 +05:30
Rakesh Radhakrishnan f982f96623 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639)
 2019-10-27 22:23:33 +05:30
Rakesh Radhakrishnan 3b2a7aabf7 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-27 22:23:16 +05:30
Uma Maheswara Rao G b00f9d89e4 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-27 22:20:00 +05:30
Rakesh Radhakrishnan 712749c1a0 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d5)
 2019-10-27 22:14:56 +05:30
Uma Maheswara Rao G aab9fefddc HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3e)
 2019-10-27 22:14:39 +05:30
zhaoyim 44ca2fe4db HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59)
(cherry picked from commit 9316ca149f)
 2019-10-25 13:11:07 -07:00
Wei-Chiu Chuang c6e6d1c6cc HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc9)
(cherry picked from commit 8cf4bc4bda)
 2019-10-24 13:14:18 -07:00
Wei-Chiu Chuang a3493fe328 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7e)
(cherry picked from commit 570ffa1cd6)
 2019-10-24 13:14:00 -07:00
Vinayakumar B e640e809a9 HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:26:29 +05:30
Mukul Kumar Singh 51114e12ee HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad8)
(cherry picked from commit c9d07a42dc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
 2019-10-22 16:14:26 -07:00
Wei-Chiu Chuang 8ef6c0aab2 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 
(cherry picked from commit 31243f0d29)
 2019-10-22 09:34:41 -07:00
Ayush Saxena 4d4f5325bd HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:47:08 +05:30
Kihwal Lee 362657c1a3 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef773)
 2019-10-21 16:35:50 -05:00
Ayush Saxena 9067a11b0d HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:43:03 +05:30
Ayush Saxena c5b4a2d115 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:36:27 +05:30
Surendra Singh Lilhore 14beaae65e HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d72)
 2019-10-17 22:49:03 +05:30
Surendra Singh Lilhore a47680a3f9 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd873)
 2019-10-15 00:11:37 +05:30
Surendra Singh Lilhore 626a48d472 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-11 00:09:20 +05:30
Surendra Singh Lilhore b396ebd0d5 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ac)
 2019-10-09 23:47:21 +05:30
Chen Liang 01d9952f63 HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a)
 2019-10-08 12:08:17 -07:00
Ayush Saxena a3599c7e70 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:24:56 +05:30
Vinayakumar B 7ab02a67bc HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 
(cherry picked from commit 30eceec342)
 2019-10-04 13:53:50 -07:00
Eric Yang 166d38ceaa HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0)
(cherry picked from commit 8bb2b00d38)
 2019-10-04 12:08:21 -07:00
He Xiaoqiao c61c114a3d HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37)
(cherry picked from commit e29ae7db12)
 2019-10-04 12:07:44 -07:00
He Xiaoqiao d91c68729c HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

(cherry picked from commit 80392e94b6)
 2019-10-04 12:07:42 -07:00
Erik Krogen 9fdb849e03 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e00143)
(cherry picked from 6630c9b75d)
 2019-10-04 11:34:37 -07:00
Arpit Agarwal 8bb98076be HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 
(cherry picked from commit a64a43b77fb1032dcb66730a6b6257a24726c256)
(cherry picked from commit 8b12381717)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
 2019-10-04 11:23:03 -07:00
Erik Krogen 56caacac1f HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa)
(cherry picked from 69b0c513a9)
 2019-10-04 10:48:17 -07:00
Siyao Meng 61dc877b00 HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b)
 2019-10-04 09:19:30 -07:00
Ayush Saxena 8ce5015e94 HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8)
(cherry picked from commit e3f54a7babf942efbe879aabef65a5b6df57fb65)
 2019-10-04 08:52:03 -07:00
Takanobu Asanuma c48f2730b4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085)
 2019-10-04 16:49:58 +09:00
Stephen O'Donnell 71dcb8f6d4 HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java

(cherry picked from commit 966193153f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
 2019-10-03 22:28:22 -07:00
Inigo Goiri bb1015719c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b2274)
 2019-10-03 18:21:32 -07:00
Inigo Goiri af688579a6 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff5)
 2019-10-03 17:27:52 -07:00
Shashikant Banerjee 2eb5865ad1 HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a)
 2019-10-03 17:26:13 -07:00
Vinayakumar B c41791671c HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a530)
 2019-10-03 17:15:38 -07:00
Vinayakumar B 1a7cb7aba1 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 
(cherry picked from commit 39dc7345b8)
(cherry picked from commit abe14d32d4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
 2019-10-03 16:59:00 -07:00
Shweta Yakkali e68ccca2d7 HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:22:00 -07:00
Ayush Saxena 7ca418c2c0 HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:22:00 -07:00
Shweta Yakkali 8ee3a8768d HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 16:22:00 -07:00
Brahma Reddy Battula ddea9d68a0 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 15d8f592b51d08a658e88c6a7a7596e0edf5793b)
(cherry picked from commit 7e10dd03b5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2019-10-03 16:16:23 -07:00
Kitti Nanasi aeb3ae1c4c HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc9200)
 2019-10-03 15:59:41 -07:00
Ayush Saxena 12f2d121ca HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 
(cherry picked from commit c4c8d5fd0e)
(cherry picked from commit 9a9edfa87e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
 2019-10-03 15:54:39 -07:00
Giovanni Matteo Fumarola 76c30e4f3c HADOOP-16625: Backport HADOOP-14624 to branch-3.1. 
(cherry picked from commit 79c97f6a0b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
 2019-10-03 15:45:23 -07:00
Inigo Goiri 0160e2c97f HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 
(cherry picked from commit 944adc61b1)
(cherry picked from commit 6148acebc3)
 2019-10-03 15:44:56 -07:00
Surendra Singh Lilhore ef9439679d HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced. Contributed by hemanthboyina. 
(cherry picked from commit 4852a90e4b)
 2019-10-03 23:57:32 +05:30
Chen Liang 40c772a750 HDFS-12979. [SBN read] StandbyNode should upload FsImage to ObserverNode after checkpointing. Contributed by Chen Liang. 2019-10-02 21:17:29 -07:00
Surendra Singh Lilhore 7f882570d7 HDFS-14216. NullPointerException happens in NamenodeWebHdfs. Contributed by lujie. 
(cherry picked from commit 92b53c40f0)
(cherry picked from commit 2e939515df)
 2019-10-02 19:50:45 -07:00
LeonGao91 a989779f2d HDFS-14678. Allow triggerBlockReport to a specific namenode. (#1252). Contributed by Leon Gao. 
(cherry picked from commit 9a1d8cfaf5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 55e40227d1d20bf88448e213937f6e3166452dc9)
 2019-10-02 17:35:25 -07:00
Chen Liang 460ef89210 HDFS-14858. [SBN read] Allow configurably enable/disable AlignmentContext on NameNode. Contributed by Chen Liang. 2019-10-02 16:40:01 -07:00
paulward24 ab7ecd6a0f HDFS-14618. Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe). Contributed by Paul Ward. 
(cherry picked from commit d203045c30)
 2019-10-02 15:07:34 -07:00
paulward24 d95dd33ea1 HDFS-14610. HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized. (#1015) 
(cherry picked from commit d8bac50e12)
(cherry picked from commit 76a9135947)
 2019-10-02 14:31:23 -07:00
Ayush Saxena 122b02e288 HDFS-14808. EC: Improper size values for corrupt ec block in LOG. Contributed by Ayush Saxena. 
(cherry picked from commit 66400c1cbb)
(cherry picked from commit 2bb4ac18ec)
 2019-10-02 13:49:20 -07:00
Surendra Singh Lilhore f0bdab02e7 HDFS-14699. Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold. Contributed by Zhao Yi Ming. 
(cherry picked from commit d1c303a497)
(cherry picked from commit eb1ddcd04c)
 2019-10-02 13:41:23 -07:00
Inigo Goiri f1fe3abac9 HADOOP-16161. NetworkTopology#getWeightUsingNetworkLocation return unexpected result. Contributed by He Xiaoqiao. 
(cherry picked from commit 389e640f0c)
 2019-10-01 17:33:08 -07:00
Yiqun Lin 3aea2bcf12 HDFS-14313. Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du. Contributed by Lisheng Sun. 
(cherry picked from commit a5bb1e8ee8)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml

(cherry picked from commit c74027d9d3)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/BlockPoolSlice.java
 2019-10-01 17:16:55 -07:00
Inigo Goiri 2a3eb04996 HDFS-14192. Track missing DFS operations in Statistics and StorageStatistics. Contributed by Ayush Saxena. 
(cherry picked from commit f048512bb8)
(cherry picked from commit e59ced9c60a7007551ee8a9f83ce8e266e4dbae1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
 2019-10-01 16:56:29 -07:00
Inigo Goiri 528dc8199b HDFS-14460. DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured. Contributed by CR Hota. 
(cherry picked from commit 865c328930)
 2019-10-01 09:53:31 -07:00
Inigo Goiri 23eed7b201 HDFS-14202. dfs.disk.balancer.max.disk.throughputInMBperSec property is not working as per set value. Contributed by Ranith Sardar. 
(cherry picked from commit 0e79a86582)
(cherry picked from commit fc8a7a9e5b)
 2019-10-01 09:35:14 -07:00
Inigo Goiri 7a915a0ba3 HDFS-14418. Remove redundant super user priveledge checks from namenode. Contributed by Ayush Saxena. 
(cherry picked from commit be6c8014e6)
(cherry picked from commit 06b2eceb76)
 2019-10-01 09:24:21 -07:00
Konstantin V Shvachko 8fc4e406ad HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by Konstantin V Shvachko. 2019-09-30 17:24:15 -07:00
Konstantin V Shvachko bece1a1f70 Revert "HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao." 
This reverts commit 0feba4396f.
 2019-09-30 17:24:15 -07:00
Surendra Singh Lilhore 1030ac85bb HDFS-14235. Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon. Contributed by Ranith Sardar. 
(cherry picked from commit 41e18feda3)
(cherry picked from commit b93b127956)
 2019-09-30 16:47:08 -07:00
Ayush Saxena 8cc702ad38 HDFS-14853. NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present. Contributed by Ranith Sardar. 2019-09-28 08:30:26 +05:30
Zhankun Tang 670d0657b4 Make upstream aware of 3.1.3 release. 2019-09-27 16:58:36 +08:00
Chen Liang 3d7730266a HDFS-14785. [SBN read] Change client logging to be less aggressive. Contributed by Chen Liang. 2019-09-26 15:59:08 -07:00
Ayush Saxena 5c058549b7 HDFS-14655. [SBN Read] Namenode crashes if one of The JN is down. Contributed by Ayush Saxena. 2019-09-25 01:22:11 +05:30
Chen Liang d25b3339ed HDFS-14822. [SBN read] Revisit GlobalStateIdContext locking when getting server state id. Contributed by Chen Liang. 2019-09-18 17:20:05 -07:00
Kihwal Lee 2661c32c06 HDFS-13959. TestUpgradeDomainBlockPlacementPolicy is flaky. Contributed by Ayush Saxena. 
(cherry picked from commit 1851d06eb3)
 2019-09-18 16:45:29 -05:00
Wei-Chiu Chuang 0fcbeed49a HDFS-14836. FileIoProvider should not increase FileIoErrors metric in datanode volume metric. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 5dd859a8a0)
(cherry picked from commit b906f53b0d)
 2019-09-18 09:36:55 -07:00
Anu Engineer eb5c4ce432 HDFS-14759. HDFS cat logs an info message. 
Contributed by Eric Badger.

(cherry picked from commit 8aaf5e1a14)
 2019-09-14 01:35:12 +09:00
Ayush Saxena cab81f7f5f HDFS-14303. Addendum: check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 2019-09-14 10:20:49 +05:30
Takanobu Asanuma 8d5460a045 HDFS-14838. RBF: Display RPC (instead of HTTP) Port Number in RBF web UI. Contributed by Xieming Li 
(cherry picked from commit c255333e20)
 2019-09-11 17:11:13 +09:00
Xiaoyu Yao ed253c4412
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-05 16:52:09 -07:00
Xiaoyu Yao fbdfb36fdf
Revert "HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun." 
This reverts commit 29e6a97faa.
 2019-09-05 16:52:09 -07:00
Zhankun Tang ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Xiaoyu Yao 29e6a97faa
HDFS-14633. The StorageType quota and consume in QuotaFeature is not handled for rename. Contributed by Jinglun. 
(cherry picked from commit 62d71fbac3789c7d484bc76ced9ec7fa6ff94de1)
 2019-09-03 08:58:59 -07:00
Stephen O'Donnell 63b18d764a HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 915cbc91c0)
(cherry picked from commit 2c7f49743fba587e32398c81412db9eac426cf98)
 2019-09-02 09:58:48 -07:00
Wei-Chiu Chuang 8ffaef78da Revert "HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell." 
This reverts commit d42b48d4d0.
 2019-09-02 09:58:32 -07:00
Istvan Fajth e3c01e174c HDFS-8178. QJM doesn't move aside stale inprogress edits files. Contributed by Istvan Fajth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-30 13:13:48 -07:00
Ayush Saxena e44693953d HDFS-12212. Options.Rename.To_TRASH is considered even when Options.Rename.NONE is specified. Contributed by Vinayakumar B. 2019-08-30 08:59:52 +05:30
leosunli 0011136e66 Make DataNodePeerMetrics#minOutlierDetectionSamples configurable (#1314). Contributed by Lisheng Sun. 
Signed-off-by: sunlisheng <sunlisheng@xiaomi.com>
(cherry picked from commit 915271245b)
(cherry picked from commit b516d2ea7b)
 2019-08-29 18:46:25 -07:00
Stephen O'Donnell d42b48d4d0 HDFS-14706. Checksums are not checked if block meta file is less than 7 bytes. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7bebad61d9)
(cherry picked from commit 9c0d6e1657)
 2019-08-29 18:26:14 -07:00
Jonathan Hung 79364a9c53 HDFS-14779. Fix logging error in TestEditLog#testMultiStreamsLoadEditWithConfMaxTxns 
(cherry picked from commit 8ab7020e64)
(cherry picked from commit d255efa5f8)
 2019-08-27 10:31:21 -07:00
Chen Liang d2bb764ffa HDFS-14674. [SBN read] Got an unexpected txid when tail editlog. Contributed by wangzhaohui. 2019-08-23 12:29:23 -07:00
Erik Krogen 9dc921f5e5 HDFS-13977. Override shouldForceSync in QuorumOutputStream to allow for proper auto-sync behavior. Contributed by Erik Krogen. 
(cherry picked from d699022fce)
(cherry picked from 2408c2491f)
 2019-08-23 11:48:51 -07:00
Chen Liang 0839cf1d4e HDFS-14611. Move handshake secret field from Token to BlockAccessToken. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang ef9b0a1d9a HDFS-13699. Add DFSClient sending handshake token to DataNode, and allow DataNode overwrite downstream QOP. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Chen Liang 3135cc35e3 HDFS-13617. Allow wrapping NN QOP into token in encrypted message. Contributed by Chen Liang 2019-08-23 10:11:22 -07:00
Chen Liang 3835a6c4e8 HDFS-13566. Add configurable additional RPC listener to NameNode. Contributed by Chen Liang. 2019-08-23 10:11:22 -07:00
Akira Ajisaka 83f67cca60
HDFS-14396. Failed to load image from FSImageFile when downgrade from 3.x to 2.x. Contributed by Fei Hui. 
(cherry picked from commit bd7baea5a5)
 2019-08-23 15:29:25 +09:00
Akira Ajisaka b40aa71cf0
HDFS-13596. NN restart fails after RollingUpgrade from 2.x to 3.x. Contributed by Fei Hui. 
(cherry picked from commit abc8fde4ca)
 2019-08-23 15:08:34 +09:00
Xiao Chen a5690b29a7
HDFS-13642. Creating a file with block size smaller than EC policy's cell size should fail. 
(cherry picked from commit cf4108313d)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSStripedOutputStream.java
 2019-08-23 15:03:57 +09:00
Ayush Saxena 900384cf57 HDFS-14276. [SBN read] Reduce tailing overhead. Contributed by Wei-Chiu Chuang. 2019-08-22 22:54:50 +05:30
Wei-Chiu Chuang 2ce0914176 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit e978c6c9ed.
 2019-08-21 09:00:59 -07:00
Surendra Singh Lilhore c2aeeb01fa HDFS-14582. Failed to start DN with ArithmeticException when NULL checksum used. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit f95988113da3f06f6d975f99f1ee51d88a793537)
(cherry picked from commit 03c62c7989)
 2019-08-21 09:00:52 -07:00
Siyao Meng e0a5fee7f5 HDFS-14665. HttpFS: LISTSTATUS response is missing HDFS-specific fields (#1291) Contributed by Siyao Meng. 2019-08-20 10:49:51 -07:00
Wei-Chiu Chuang 7a1f508f58 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 
(cherry picked from commit fbe87eddbc30fe5191c008b496fb83e51ef4ee4a)
(cherry picked from commit f78dcbaae5535e3e05c9f29986b8963ba262baa9)
 2019-08-20 10:36:55 -07:00
Wei-Chiu Chuang d8c60f829f HDFS-13201. Fix prompt message in testPolicyAndStateCantBeNull. Contributed by chencan. 
(cherry picked from commit aa6995fde2)
(cherry picked from commit b004d305e9)
 2019-08-20 10:34:25 -07:00
Wei-Chiu Chuang fc062b6fef HDFS-13709. Report bad block to NN when transfer block encounter EIO exception. Contributed by Chen Zhang. 
(cherry picked from commit 360a96f342)
(cherry picked from commit 6d93886231)
 2019-08-19 13:11:21 -07:00
HUAN-PING SU 18f98491e7 HDFS-14746. Trivial test code update after HDFS-14687. Contributed by kevin su. 
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit abae6ff2a2)
(cherry picked from commit abda503040)
 2019-08-19 11:15:48 -07:00
Wei-Chiu Chuang e978c6c9ed HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 
(cherry picked from commit 8b802d6b44)
 2019-08-19 08:39:00 -07:00
Surendra Singh Lilhore 92c3a581ec HDFS-14687. Standby Namenode never come out of safemode when EC files are being written. Contributed by Surendra Singh Lilhore. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
Reviewed-by: Wei-Chiu CHuang <weichiu@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b8db5b9a98)
(cherry picked from commit 6b01effd01)
 2019-08-18 18:17:50 -07:00
Shashikant Banerjee d18b720c70 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
(cherry picked from commit 0a85af959c)
(cherry picked from commit e89413da88)
 2019-08-17 03:07:03 -07:00
Siyao Meng 224643a58c HDFS-14595. HDFS-11848 breaks API compatibility. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 3c0382f1b9)
(cherry picked from commit 136a97a74dbc12f05b88d0abda101690e7c727d9)
 2019-08-14 07:40:48 -07:00
Wei-Chiu Chuang 6966b76230 HDFS-14491. More Clarity on Namenode UI Around Blocks and Replicas. Contributed by Siyao Meng. 
(cherry picked from commit 6a43d0fbd49b3ff1ce75a2334b51a98ae476e473)
(cherry picked from commit 4784165bb24228d13f4e738e0093ab0dade0bff1)
 2019-08-13 17:15:10 -07:00
Masatake Iwasaki e66ad19395 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)
 2019-08-14 09:07:00 +09:00
Siyao Meng b040eb91c7 HDFS-14148. HDFS OIV ReverseXML SnapshotSection parser throws exception when there are more than one snapshottable directory (#1274) Contributed by Siyao Meng. 
(cherry picked from commit c92b49876a)
(cherry picked from commit c5f433b11f)
 2019-08-12 17:30:32 -07:00
Yiqun Lin f2007818fc HDFS-13359. DataXceiver hung due to the lock in FsDatasetImpl#getBlockInputStream. Contributed by Yiqun Lin. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8a77a224c7)
(cherry picked from commit a7bc09c596)
 2019-08-09 18:41:18 -07:00
Wei-Chiu Chuang 0aa847e158 HDFS-14693. NameNode should log a warning when EditLog IPC logger's pending size exceeds limit. Contributed by Xudong Cao. 
(cherry picked from commit 6ad9a11494)
(cherry picked from commit 2f2fa3dad6)
 2019-08-08 13:51:23 -07:00
Stephen O'Donnell 163fb887a6 HDFS-14459. ClosedChannelException silently ignored in FsVolumeList.addBlockPool(). Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit b0799148cf)
(cherry picked from commit a00a327594)
 2019-08-08 13:37:22 -07:00
Chao Sun 90447baeb0 HDFS-14034. Support getQuotaUsage API in WebHDFS. Contributed by Chao Sun. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>

(cherry picked from 3ae775d740)
(cherry picked from d38b617baa)
 2019-08-08 10:42:27 -07:00
Erik Krogen f6ce2f4a50 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
 2019-08-07 09:33:36 -07:00
Lokesh Jain d06fca0104 HDFS-14692. Upload button should not encode complete url. Contributed by Lokesh Jain. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 954ff36360)
(cherry picked from commit f6fe18a282)
 2019-08-06 15:31:15 -07:00
Stephen O'Donnell 99af4ce0ac HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-06 13:53:01 -07:00
Erik Krogen 69e3ac5cbb HDFS-14697. Backport of HDFS-14513. FSImage which is saving should be clean while NameNode shutdown. Contributed by He Xiaoqiao. 
(cherry picked from ee1547a807)
 2019-08-06 09:49:23 -07:00
Wei-Chiu Chuang a7a348446d Revert "HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell." 
This reverts commit 45ba41570b.
 2019-08-06 08:57:00 -07:00
Stephen O'Donnell 45ba41570b HDFS-14557. JournalNode error: Can't scan a pre-transactional edit log. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 35e0a01d7b)
(cherry picked from commit 8414e6354b)
 2019-08-06 07:58:23 -07:00
Erik Krogen da548cf263 HDFS-14462 Ensure WebHDFS client throws the correct exception during writes. Contributed by Simbarashe Dzinamarira. 
(cherry picked from e7a0b8aa83)
(cherry picked from ba4ca7d996cafa1b1786249cae2b207f0eaa1070)
 2019-08-02 11:51:22 -07:00
Wei-Chiu Chuang 68d618acd2 HDFS-14685. DefaultAuditLogger doesn't print CallerContext. Contributed by xuzq. 
(cherry picked from commit 6757fc1a1d40ffc771bb685886b0d4a16315eca5)
(cherry picked from commit e1659fe009)
 2019-08-02 10:52:05 -07:00
Wei-Chiu Chuang d66ee4bf61 HDFS-14631.The DirectoryScanner doesn't fix the wrongly placed replica. Contributed by Jinglun. 
(cherry picked from commit 32607dbd98)
(cherry picked from commit 5c1f91fd97)
 2019-08-01 15:49:14 -07:00
Christopher Gregorian 7b8f08f59e HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
 2019-07-30 14:45:00 -07:00
Wei-Chiu Chuang 608e7044a5 HDFS-14429. Block remain in COMMITTED but not COMPLETE caused by Decommission. Contributed by Yicong Cai. 
(cherry picked from commit 8053085388)
(cherry picked from commit d28e624958)
 2019-07-29 14:33:08 -07:00
Yiqun Lin 2e1afa9a86 HDFS-14632. Reduce useless #getNumLiveDataNodes call in SafeModeMonitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 993dc8726b)
(cherry picked from commit 69f1c8d77b)
 2019-07-29 06:50:41 -07:00
Inigo Goiri fe40fbbd4d HDFS-12703. Exceptions are fatal to decommissioning monitor. Contributed by He Xiaoqiao. 
(cherry picked from commit 3d396786cf6eaab49c1c9b8b2a4652c2e440b9e3)
(cherry picked from commit 950aa74d5f)
 2019-07-26 15:56:44 -07:00
Erik Krogen 42f10712a5 Revert "HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena." 
This reverts commit bf4a2258ce.
 2019-07-26 11:36:59 -07:00
Masatake Iwasaki bf4a2258ce HDFS-14135. TestWebHdfsTimeouts Fails intermittently in trunk. Contributed by Ayush Saxena. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 6b8107ad97)
 2019-07-26 09:49:42 +09:00
Chen Liang 2a1451aa0e HDFS-13791. Amend fix LOG string format. 2019-07-12 12:58:08 -07:00
Wei-Chiu Chuang 3ba7826d41 HDFS-14537. Journaled Edits Cache is not cleared when formatting the JN. Contributed by Ranith Sardar. 2019-07-12 12:26:23 -07:00
Erik Krogen cc22373606 HDFS-14435. [SBN Read] Enable ObserverReadProxyProvider to gracefully handle StandbyException when fetching HAServiceState. Contributed by Erik Krogen. 2019-07-12 12:25:19 -07:00
Erik Krogen 769b5a802b HDFS-14211. [SBN Read]. Add a configurable flag to enable always-msync mode to ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen 35200b3671 HDFS-14272. [SBN read] Make ObserverReadProxyProvider initialize its state ID against the active NN on startup. Contributed by Erik Krogen. 2019-07-12 12:24:44 -07:00
Erik Krogen 98684fb821 HDFS-14279. [SBN read] Fix race condition in ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 12:24:43 -07:00
Erik Krogen ee0416b456 HDFS-14250. [SBN read]. msync should always direct to active NameNode to get latest stateID. Contributed by Chao Sun. 2019-07-12 12:24:43 -07:00
Konstantin V Shvachko 321e26ced1 HDFS-14170. [SBN read] Fix checkstyle warnings related to SBN reads. Contributed by Konstantin V Shvachko. 2019-07-12 12:24:43 -07:00
Chao Sun b028d62d1f HDFS-14154. [SBN read] Document dfs.ha.tail-edits.period in user guide. Contributed by Chao Sun. 2019-07-12 12:24:42 -07:00
Konstantin V Shvachko b8f9739024 HDFS-14160. [SBN read] ObserverReadInvocationHandler should implement RpcInvocationHandler. Contributed by Konstantin V Shvachko. 2019-07-12 12:24:42 -07:00
Chao Sun 8f36bdc3b9 HDFS-14149. [SBN read] Fix annotations on new interfaces/classes for SBN reads. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Chao Sun 7a650b018c HDFS-14116. [SBN read] Fix class cast error in NNThroughputBenchmark with ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 12:24:41 -07:00
Erik Krogen 6c1bf61628 HDFS-14146. [SBN read] Handle exceptions from and prevent handler threads from blocking within internalQueueCall. Contributed by Chao Sun. 2019-07-12 12:24:40 -07:00
Erik Krogen 67c4768825 HDFS-13873. [SBN read] ObserverNode should reject read requests when it is too far behind. Contributed by Konstantin Shvachko. 2019-07-12 12:24:36 -07:00
Chen Liang 8e77349bee HDFS-14142. Move ipfailover config key out of HdfsClientConfigKeys. Contributed by Chen Liang. 2019-07-12 12:23:56 -07:00
Erik Krogen 265bd68464 HDFS-14131. [SBN read] Create user guide for Consistent Reads from Observer feature. Contributed by Chao Sun. 2019-07-12 12:23:17 -07:00
Chen Liang 6e54d97458 HDFS-14120. [SBN read] ORFPP should also clone DT for the virtual IP. Contributed by Chen Liang. 2019-07-12 12:23:17 -07:00
Konstantin V Shvachko 6cb1b5aa2f HDFS-14094. [SBN read] Fix the order of logging arguments in ObserverReadProxyProvider. Contributed by Ayush Saxena. 2019-07-12 12:23:16 -07:00
Chao Sun c4d944aeb3 HDFS-14067. [SBN read] Allow manual failover between standby and observer. Contributed by Chao Sun. 2019-07-12 12:23:16 -07:00
Chen Liang f5c923be3f HDFS-14017. [SBN read] ObserverReadProxyProviderWithIPFailover should work with HA configuration. Contributed by Chen Liang. 2019-07-12 12:23:11 -07:00
Chen Liang 53a88bfdb4 HDFS-14035. NN status discovery does not leverage delegation token. Contributed by Chen Liang. 2019-07-12 12:22:25 -07:00
Chao Sun e0436b3438 HDFS-14016. [SBN read] ObserverReadProxyProvider should enable observer read by default. Contributed by Chen Liang. 2019-07-12 12:11:19 -07:00
Chao Sun 8100dcbb34 HDFS-13924. [SBN read] Handle BlockMissingException when reading from observer. Contributed by Chao Sun. 2019-07-12 12:11:18 -07:00
Konstantin V Shvachko 66db260cdc HDFS-13925. Unit Test for transitioning between different states. Contributed by Sherwood Zheng. 2019-07-12 12:11:18 -07:00
Konstantin V Shvachko e860882d60 HDFS-13523. Support observer nodes in MiniDFSCluster. Contributed by Konstantin Shvachko. 2019-07-12 12:11:17 -07:00
Konstantin V Shvachko a054c65185 HDFS-13961. [SBN read] TestObserverNode refactoring. Contributed by Konstantin Shvachko. 2019-07-12 12:11:14 -07:00
Chen Liang 91803c8216 HDFS-13791. Limit logging frequency of edit tail related statements. Contributed by Erik Krogen. 2019-07-12 11:50:51 -07:00
Erik Krogen d4adf921a3 HDFS-13898. [SBN read] Throw retriable exception for getBlockLocations when ObserverNameNode is in safemode. Contributed by Chao Sun. 2019-07-12 11:50:05 -07:00
Erik Krogen 07558629db HDFS-13749. [SBN read] Use getServiceStatus to discover observer namenodes. Contributed by Chao Sun. 2019-07-12 11:50:04 -07:00
Konstantin V Shvachko e639297ba0 HDFS-13778. [SBN read] TestStateAlignmentContextWithHA should use real ObserverReadProxyProvider instead of AlignmentContextProxyProvider. Contributed by Konstantin Shvachko and Plamen Jeliazkov. 2019-07-12 11:50:04 -07:00
Chen Liang 8ec17f6aea HDFS-13880. Add mechanism to allow certain RPC calls to bypass sync. Contributed by Chen Liang. 2019-07-12 11:50:03 -07:00
Erik Krogen 504cb70619 HDFS-13779. [SBN read] Implement proper failover and observer failure handling logic for for ObserverReadProxyProvider. Contributed by Erik Krogen. 2019-07-12 11:50:03 -07:00
Konstantin V Shvachko 1be8e187b1 HDFS-13782. ObserverReadProxyProvider should work with IPFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-12 11:50:02 -07:00
Konstantin V Shvachko 1ad1c800a4 HDFS-13851. Remove AlignmentContext from AbstractNNFailoverProxyProvider. Contributed by Konstantin Shvachko. 2019-07-12 11:50:02 -07:00
Chen Liang ee9977cc69 HDFS-13767. Add msync server implementation. Contributed by Chen Liang. 2019-07-12 11:50:01 -07:00
Chao Sun d96ddb3b87 HDFS-13789. Reduce logging frequency of QuorumJournalManager#selectInputStreams. Contributed by Erik Krogen. 2019-07-12 11:50:01 -07:00
Erik Krogen 849a6c6f0d HDFS-13688. [SBN read] Introduce msync API call. Contributed by Chen Liang. 2019-07-12 11:49:58 -07:00
Erik Krogen ac2e79330c HDFS-13610. [SBN read] Edit Tail Fast Path Part 4: Cleanup. Integration test, documentation, remove unnecessary dummy sync, minors fixups. Contributed by Erik Krogen. 2019-07-12 11:05:50 -07:00
Chao Sun 9840dae72f HDFS-12976. [SBN read] Introduce ObserverReadProxyProvider. Contributed by Chao Sun. 2019-07-12 11:05:07 -07:00
Konstantin V Shvachko 405f6b6fbf HDFS-13706. [SBN read] Rename client context to ClientGSIContext. Contributed by Konstantin Shvachko. 2019-07-12 11:05:07 -07:00
Erik Krogen 8e551708d8 HDFS-13609. [SBN read] Edit Tail Fast Path Part 3: NameNode-side changes to support tailing edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:05:04 -07:00
Erik Krogen abb8614dcf HDFS-13608. [SBN read] Edit Tail Fast Path Part 2: Add ability for JournalNode to serve edits via RPC. Contributed by Erik Krogen. 2019-07-12 11:04:24 -07:00
Erik Krogen c40ddf9b5b HDFS-13607. [SBN read] Edit Tail Fast Path Part 1: Enhance JournalNode with an in-memory cache of recent edit transactions. Contributed by Erik Krogen. 2019-07-12 11:03:32 -07:00
Plamen Jeliazkov 7d0c3ba700 HDFS-13399. [SBN read] Make Client field AlignmentContext non-static. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:31 -07:00
Erik Krogen 18e545f0be HDFS-13578. [SBN read] Add ReadOnly annotation to methods in ClientProtocol. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen 07a6510e6a HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-07-12 11:03:31 -07:00
Erik Krogen b999e96c9a HDFS-13331. [SBN read] Add lastSeenStateId to RpcRequestHeader. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:30 -07:00
Plamen Jeliazkov 21a886a4ef HDFS-12977. [SBN read] Add stateId to RPC headers. Contributed by Plamen Jeliazkov. 2019-07-12 11:03:27 -07:00
Chao Sun 4cf63905d0 HDFS-12975. [SBN read] Changes to the NameNode to support reads from standby. Contributed by Chao Sun. 2019-07-12 10:58:43 -07:00
Masatake Iwasaki c6ac7f511c
HDFS-14466. Add a regression test for HDFS-14323. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 00dd843a1a)
 2019-07-12 10:14:49 +09:00
Weiwei Yang 9c3806cf1b HDFS-12748. NameNode memory leak when accessing webhdfs GETHOMEDIRECTORY. Contributed by Weiwei Yang. 2019-07-11 09:46:27 +08:00
Erik Krogen a59553b5e6 HDFS-14547. Improve memory efficiency of quotas when storage type quotas are not set. Contributed by Jinglun. 
(cherry-picked from 4632708148)
(cherry-picked from 491dc7cc9d)
 2019-07-08 15:20:29 -07:00
leosunli 7c7907f756 HDFS-13694. Making md5 computing being in parallel with image loading. 
(cherry picked from commit 9c90729486)
 2019-07-07 20:02:51 -07:00
Masatake Iwasaki 02ae1d6071 HDFS-14629. Property value Hard Coded in DNConf.java. Contributed by hemanthboyina. 
(cherry picked from commit 1c254a8ec7)
 2019-07-04 23:54:18 +09:00
Anu Engineer 1bae5c7024 HDFS-14598. Findbugs warning caused by HDFS-12487. 
Contributed by He Xiaoqiao.

(cherry picked from commit 041e7a7dee)
(cherry picked from commit e7fce2104f)
 2019-06-24 21:20:00 -07:00
Inigo Goiri 90b88db35d HDFS-14247. Repeat adding node description into network topology. Contributed by HuangTao. 
(cherry picked from commit 80b77deb42)
(cherry picked from commit 9637124535)
 2019-06-24 21:17:04 -07:00
Inigo Goiri aaf74ea5d7 HDFS-14541. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException. 
(cherry picked from commit 0966407ad6)
 2019-06-24 19:03:26 -07:00
Santosh Marella 026cc2647d HDFS-12914. Block report leases cause missing blocks until next report. Contributed by Santosh Marella, He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2019-06-23 20:27:01 -07:00
Wei-Chiu Chuang 529d0955f5 HDFS-14074. DataNode runs async disk checks maybe throws NullPointerException, and DataNode failed to register to NameSpace. Contributed by guangyi lu. 
(cherry picked from commit 645d67bc4f)
(cherry picked from commit d5ecca1d4c)
 2019-06-21 18:27:50 -07:00
Wei-Chiu Chuang 4507c2f3a5 HDFS-12487. FsDatasetSpi.isValidBlock() lacks null pointer check inside and neither do the callers. Contributed by liumi. 
(cherry picked from commit 1524e2e6c5)
(cherry picked from commit 845d7611ba)
 2019-06-21 18:18:36 -07:00
Wei-Chiu Chuang f33bcbb687 HDFS-14303. check block directory logic not correct when there is only meta file, print no meaning warn log. Contributed by qiang Liu. 
(cherry picked from commit a5d76a4965)
 2019-06-20 20:54:43 -07:00
Takanobu Asanuma 2f6f42bc7b HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643)
 2019-06-20 20:18:00 -07:00
Wei-Chiu Chuang 58c1920472 Revert "HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti." 
This reverts commit 49587e4df3.
 2019-06-19 06:55:25 -07:00
Virajith Jalaparti 49587e4df3 HDFS-13287. TestINodeFile#testGetBlockType results in NPE when run alone. Contributed by Virajith Jalaparti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 333633d9935acd5787ee8688bc9d2d7b7c058def)
(cherry picked from commit 36dc0810bf)
 2019-06-18 22:17:08 -07:00
Zsolt Venczel 4ea3b04bf3 HDFS-14101. Random failure of testListCorruptFilesCorruptedBlock. Contributed by Zsolt Venczel, Nikhil Navadia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Nikhil Navadia <nikhil.navadiya@cloudera.com>
(cherry picked from commit 7c00756aff)
 2019-06-18 14:38:05 -07:00
Elek, Marton 293f8bbd84 HDFS-14078. Admin helper fails to prettify NullPointerExceptions. Contributed by Elek, Marton. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cc229be9ebb1b3737c9f5fcb29788b51fa584e47)
(cherry picked from commit 1a99f97945)
 2019-06-18 12:00:48 -07:00
Wei-Chiu Chuang 29c4aded34 HDFS-14010. Pass correct DF usage to ReservedSpaceCalculator builder. Contributed by Virajith Jalaparti. 
(cherry picked from commit 1cc1530b4e874391a17ec81e24cb55dfbbabd36a)
(cherry picked from commit 223a582eb8)
 2019-06-18 00:34:55 -07:00
Wei-Chiu Chuang 4362fcb629 HDFS-14340. Lower the log level when can't get postOpAttr. Contributed by Anuhan Torgonshar. 
(cherry picked from commit 098c325a78)
(cherry picked from commit f5428d4062c0a5e98461ea3b8420dc68555681f9)
 2019-06-17 22:48:07 -07:00
Weiwei Yang 3bdb661f35 HDFS-12770. Add doc about how to disable client socket cache. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6ec57fe6a45df0e9216bbbba85b5bf61da1bf2ca)
(cherry picked from commit f36cdc4d7f)
 2019-06-17 21:47:07 -07:00
Wei-Chiu Chuang fc8983a5b8 HDFS-13730. BlockReaderRemote.sendReadResult throws NPE. Contributed by Yuanbo Liu. 
(cherry picked from commit 62ad9885ea)
(cherry picked from commit 45543ffd6a)
 2019-06-17 20:20:08 -07:00
Akira Ajisaka 3d1d8623a1 HDFS-11950. Disable libhdfs zerocopy test on Mac. Contributed by Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a95e87d827)
(cherry picked from commit f68099aa13)
 2019-06-17 17:04:07 -07:00
Wei-Chiu Chuang b3e0430c3b HDFS-14465. When the Block expected replications is larger than the number of DataNodes, entering maintenance will never exit. Contributed by Yicong Cai. 
(cherry picked from commit 7db922c6474cb29e0e9bad320fcda4769366340a)
(cherry picked from commit 0eb42e1e05)
 2019-06-17 15:18:39 -07:00
huzheng b082628e5a HDFS-14535. The default 8KB buffer in requestFileDescriptors#BufferedOutputStream is causing lots of heap allocation in HBase when using short-circut read 
(cherry picked from commit ea3b0a1844)
(cherry picked from commit 9b61ecfcbe)
 2019-06-17 07:21:07 -07:00
Inigo Goiri 76a66998fe HDFS-14560. Allow block replication parameters to be refreshable. Contributed by Stephen O'Donnell. 2019-06-13 18:28:38 -07:00
Sean Mackrory fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Stephen O'Donnell a9b8310584 HDFS-13231. Extend visualization for Decommissioning, Maintenance Mode under Datanode tab in the NameNode UI. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 50de0874d0)
(cherry picked from commit 71159f9540)
 2019-06-12 10:31:21 -07:00
He Xiaoqiao 03ec4c4c7c HDFS-14553. Make queue size of BlockReportProcessingThread configurable. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bd46bdf9f9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit d7560c866e)
 2019-06-10 17:27:48 -07:00
Hanisha Koneru dd795edcc7 HDFS-10659. Namenode crashes after Journalnode re-installation in an HA cluster due to missing paxos directory. Contributed by star, Hanisha Koneru. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7217494f40)
(cherry picked from commit 210480a23c)
 2019-06-10 13:47:17 -07:00
Kihwal Lee a935258e8c HDFS-14521. Suppress setReplication logging. Contributed by Kihwal Lee 
(cherry picked from commit 9122b9b649)
 2019-05-30 17:31:30 -05:00
Ayush Saxena 90cc5ddd49 HDFS-14512. ONE_SSD policy will be violated while write data with DistributedFileSystem.create(....favoredNodes). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c1caab40f2)
(cherry picked from commit 2cd7577a5a)
 2019-05-29 20:58:34 -07:00
Takanobu Asanuma 8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:39:49 +09:00
Akira Ajisaka f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Erik Krogen 9295e95174 HDFS-14500. NameNode StartupProgress should not allow new steps in an already-completed phase. Contributed by Erik Krogen. 
(cherry-picked from 55e0c134f0)
(cherry-picked from c26cf22d6b)
 2019-05-24 13:19:39 -07:00
Konstantin V Shvachko 799ca572f7 HDFS-14502. keepResults option in NNThroughputBenchmark should call saveNamespace(). Contributed by Konstantin V Shvachko. 2019-05-24 12:35:32 -07:00
Siyao Meng 9196c84c9f HDFS-14507. Document -blockingDecommission option for hdfs dfsadmin -listOpenFiles 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 9dff6eff81)
 2019-05-22 13:25:42 +09:00
Srinivasu Majeti e599e3b546 HDFS-14323. Distcp fails in Hadoop 3.x when 2.x source webhdfs url has special characters in hdfs file path. Contributed by Srinivasu Majeti. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3e5e5b028a)
(cherry picked from commit 85f3664bb8)
 2019-05-17 19:22:03 +02:00
Siyao Meng 86fb5e1c8e HDFS-14463. Add Log Level link under NameNode and DataNode Web UI Utilities dropdown. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7cb46f035a)
(cherry picked from commit 0550aa32f1a4b1809c234f8ed785db08b2e33d13)
 2019-05-01 16:49:36 -07:00
Arpit Agarwal 7b51edbab9
HDFS-13677. Dynamic refresh Disk configuration results in overwriting VolumeMap. Contributed by xuzq. 
(cherry picked from commit 4b4200f1f8)
 2019-04-29 14:54:11 -07:00
Puleya7 546ed4937c HDFS-14432. dfs.datanode.shared.file.descriptor.paths duplicated in hdfs-default.xml 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 13907d8479)
 2019-04-17 21:57:06 +09:00
Akira Ajisaka a70076b3a5
HDFS-14407. Fix misuse of SLF4j logging API in DatasetVolumeChecker#checkAllVolumes. Contributed by Wanqiang Ji. 
(cherry picked from commit 67020f0950)
 2019-04-05 12:31:14 +09:00
Stephen O'Donnell d9899015eb HDFS-14389. getAclStatus returns incorrect permissions and owner when an iNodeAttributeProvider is configured. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c528e427aa6745434672b1c1850738795ad1d6d2)
(cherry picked from commit 388f445dde)
 2019-04-04 04:06:33 -07:00
Wei-Chiu Chuang 084fb9de22 HDFS-10477. Stop decommission a rack of DataNodes caused NameNode fail over to standby. Contributed by yunjiong zhao and Wei-Chiu Chuang. 
(cherry picked from commit be488b6070)
(cherry picked from commit c8703dda07)
 2019-04-03 12:41:07 -07:00
Gabor Bota f01d592f3b
HADOOP-16220. Add findbugs ignores for unjustified issues during update to guava to 27.0-jre in hadoop-project 
This closes #665

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 53a86e2b8e)
 2019-04-01 13:52:44 +09:00
Takanobu Asanuma 1fc74178eb HDFS-14037. Fix SSLFactory truststore reloader thread leak in URLConnectionFactory. 
(cherry picked from commit 55fb3c32fb)
 2019-03-27 03:29:42 +09:00
Inigo Goiri 80732146c6 HDFS-14366. Improve HDFS append performance. Contributed by Chao Sun. 
(cherry picked from commit ff06ef0631)
 2019-03-15 13:58:39 -07:00
Erik Krogen cb76fc5618 HDFS-14346. Add better time precision to Configuration#getTimeDuration, allowing return unit and default unit to be specified independently. Contributed by Chao Sun. 
(cherry picked from commit 66357574ae)
(cherry picked from commit fec7c5f3eb)
 2019-03-13 13:20:51 -07:00
Shweta Yakkali a324db2cce HDFS-14081. hdfs dfsadmin -metasave metasave_test results NPE. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 1bea785020)
(cherry picked from commit 1ceefa726e)
 2019-03-12 16:06:16 -07:00
Stephen O'Donnell 327e57ae6e HDFS-14333. Datanode fails to start if any disk has errors during Namenode registration. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 34b14061b3)
(cherry picked from commit a21e2e4dbc)
 2019-03-12 10:19:37 -07:00
Erik Krogen b0b8000240 HDFS-14317. Ensure checkpoints are created when in-progress edit log tailing is enabled with a period shorter than the log roll period. Contributed by Ekanth Sethuramalingam. 
(cherry-picked from commit 1bc282e0b3)
(cherry-picked from commit 6d076dd5e8)
 2019-03-07 08:43:39 -08:00
Wei-Chiu Chuang d951497f57 HDFS-14314. fullBlockReportLeaseId should be reset after registering to NN. Contributed by star. 
(cherry picked from commit 387dbe587a)
(cherry picked from commit e58ccca3ce)
 2019-03-04 10:46:00 -08:00
Erik Krogen b7eb9116eb HDFS-14305. Fix serial number calculation in BlockTokenSecretManager to avoid token key ID overlap between NameNodes. Contributed by He Xiaoqiao. 2019-03-01 08:12:54 -08:00
Surendra Singh Lilhore bd9e2826fe HDFS-14263. Remove unnecessary block file exists check from FsDatasetImpl#getBlockInputStream(). Contributed by Surendra Singh Lilhore 
(cherry picked from commit 29b411d5f0)
 2019-02-13 23:56:34 +05:30
Surendra Singh Lilhore 3bcdff8fac HDFS-14274. EC: NPE While Listing EC Policy For A Directory Following Replication Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 35d4f32b32)
 2019-02-13 23:26:14 +05:30
Vinayakumar B 1f3e737407 HDFS-14266. EC : Fsck -blockId shows null for EC Blocks if One Block Is Not Available. Contributed by Ayush Saxena. 2019-02-12 22:35:55 +05:30
Surendra Singh Lilhore 1f4be45ef6 HDFS-14140. JournalNodeSyncer authentication is failing in secure cluster. Contributed by Surendra Singh Lilhore. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 4be87353e3)
(cherry picked from commit 2501fcd26b)
 2019-02-07 16:53:12 -08:00
Kitti Nanasi c25b641cec HDFS-14231. DataXceiver#run() should not log exceptions caused by InvalidToken exception as an error. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 49ddd8a6ed)
(cherry picked from commit 38541b2375)
 2019-02-05 15:15:31 -08:00
Wangda Tan a1e09b4c0c Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
(cherry picked from commit 649da5af04)
 2019-02-05 14:09:33 -08:00
Masatake Iwasaki dd3e7fc0c2 HDFS-12729. Document special paths in HDFS. Contributed by Masatake Iwasaki. 
(cherry picked from commit dc5af4c6b7)
 2019-01-29 02:29:39 +09:00
Dinesh Chitlangia b63941516f HDFS-14228. Incorrect getSnapshottableDirListing() javadoc. Contributed by Dinesh Chitlangia. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 45caeee6cf)
(cherry picked from commit eb77e60163)
 2019-01-25 07:38:11 -08:00
Takanobu Asanuma 38e4bed224 HDFS-14218. EC: Ls -e throw NPE when directory ec policy is disabled. Contributed by Ayush Saxena. 
(cherry picked from commit 7d6792e5d2)
 2019-01-23 10:05:39 +09:00
Arpit Agarwal f5bfeac62e HDFS-14221. Replace Guava Optional with Java Optional. Contributed by Arpit Agarwal. 
(cherry picked from commit 1ff658b2ef)
 2019-01-21 20:48:05 -08:00
Arpit Agarwal 78b45977b1 HDFS-14222. Make ThrottledAsyncChecker constructor public. Contributed by Arpit Agarwal. 
(cherry picked from commit 6f0756fc0e)
 2019-01-21 20:47:02 -08:00
Weiwei Yang cf520b7d64 HDFS-14207. ZKFC should catch exception when ha configuration missing. Contributed by Fei Hui. 
(cherry picked from commit de34fc148c)
 2019-01-22 12:17:33 +08:00