Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,399 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7436 Commits

Author SHA1 Message Date
Rakesh Radhakrishnan 2db0b8f6ca HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d)
 2019-10-27 22:24:44 +05:30
Rakesh Radhakrishnan cd1277398a HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6f)
 2019-10-27 22:24:30 +05:30
Rakesh Radhakrishnan f2563dcca1 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1)
 2019-10-27 22:24:14 +05:30
Rakesh Radhakrishnan df0dcc7493 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7)
 2019-10-27 22:24:00 +05:30
Sammi Chen 9281b72550 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad44490)
 2019-10-27 22:23:44 +05:30
Rakesh Radhakrishnan f982f96623 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639)
 2019-10-27 22:23:33 +05:30
Rakesh Radhakrishnan 3b2a7aabf7 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-27 22:23:16 +05:30
Uma Maheswara Rao G b00f9d89e4 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-27 22:20:00 +05:30
Rakesh Radhakrishnan 712749c1a0 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d5)
 2019-10-27 22:14:56 +05:30
Uma Maheswara Rao G aab9fefddc HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3e)
 2019-10-27 22:14:39 +05:30
zhaoyim 44ca2fe4db HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59)
(cherry picked from commit 9316ca149f)
 2019-10-25 13:11:07 -07:00
Wei-Chiu Chuang c6e6d1c6cc HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc9)
(cherry picked from commit 8cf4bc4bda)
 2019-10-24 13:14:18 -07:00
Wei-Chiu Chuang a3493fe328 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7e)
(cherry picked from commit 570ffa1cd6)
 2019-10-24 13:14:00 -07:00
Vinayakumar B e640e809a9 HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:26:29 +05:30
Mukul Kumar Singh 51114e12ee HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad8)
(cherry picked from commit c9d07a42dc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
 2019-10-22 16:14:26 -07:00
Wei-Chiu Chuang 8ef6c0aab2 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 
(cherry picked from commit 31243f0d29)
 2019-10-22 09:34:41 -07:00
Ayush Saxena 4d4f5325bd HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:47:08 +05:30
Kihwal Lee 362657c1a3 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef773)
 2019-10-21 16:35:50 -05:00
Ayush Saxena 9067a11b0d HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:43:03 +05:30
Ayush Saxena c5b4a2d115 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:36:27 +05:30
Surendra Singh Lilhore 14beaae65e HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d72)
 2019-10-17 22:49:03 +05:30
Surendra Singh Lilhore a47680a3f9 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd873)
 2019-10-15 00:11:37 +05:30
Surendra Singh Lilhore 626a48d472 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-11 00:09:20 +05:30
Surendra Singh Lilhore b396ebd0d5 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ac)
 2019-10-09 23:47:21 +05:30
Chen Liang 01d9952f63 HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a)
 2019-10-08 12:08:17 -07:00
Ayush Saxena a3599c7e70 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:24:56 +05:30
Vinayakumar B 7ab02a67bc HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 
(cherry picked from commit 30eceec342)
 2019-10-04 13:53:50 -07:00
Eric Yang 166d38ceaa HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0)
(cherry picked from commit 8bb2b00d38)
 2019-10-04 12:08:21 -07:00
He Xiaoqiao c61c114a3d HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37)
(cherry picked from commit e29ae7db12)
 2019-10-04 12:07:44 -07:00
He Xiaoqiao d91c68729c HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

(cherry picked from commit 80392e94b6)
 2019-10-04 12:07:42 -07:00
Erik Krogen 9fdb849e03 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e00143)
(cherry picked from 6630c9b75d)
 2019-10-04 11:34:37 -07:00
Arpit Agarwal 8bb98076be HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 
(cherry picked from commit a64a43b77fb1032dcb66730a6b6257a24726c256)
(cherry picked from commit 8b12381717)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
 2019-10-04 11:23:03 -07:00
Erik Krogen 56caacac1f HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa)
(cherry picked from 69b0c513a9)
 2019-10-04 10:48:17 -07:00
Siyao Meng 61dc877b00 HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b)
 2019-10-04 09:19:30 -07:00
Ayush Saxena 8ce5015e94 HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8)
(cherry picked from commit e3f54a7babf942efbe879aabef65a5b6df57fb65)
 2019-10-04 08:52:03 -07:00
Takanobu Asanuma c48f2730b4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085)
 2019-10-04 16:49:58 +09:00
Stephen O'Donnell 71dcb8f6d4 HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java

(cherry picked from commit 966193153f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
 2019-10-03 22:28:22 -07:00
Inigo Goiri bb1015719c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b2274)
 2019-10-03 18:21:32 -07:00
Inigo Goiri af688579a6 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff5)
 2019-10-03 17:27:52 -07:00
Shashikant Banerjee 2eb5865ad1 HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a)
 2019-10-03 17:26:13 -07:00
Vinayakumar B c41791671c HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a530)
 2019-10-03 17:15:38 -07:00
Vinayakumar B 1a7cb7aba1 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 
(cherry picked from commit 39dc7345b8)
(cherry picked from commit abe14d32d4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
 2019-10-03 16:59:00 -07:00
Shweta Yakkali e68ccca2d7 HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:22:00 -07:00
Ayush Saxena 7ca418c2c0 HDFS-14660. [SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider. Contributed by Chao Sun. 2019-10-03 16:22:00 -07:00
Shweta Yakkali 8ee3a8768d HDFS-14494. Move Server logging of StatedId inside receiveRequestState(). Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-10-03 16:22:00 -07:00
Brahma Reddy Battula ddea9d68a0 HDFS-14064. WEBHDFS: Support Enable/Disable EC Policy. Contributed by Ayush Saxena. 
(cherry picked from commit 15d8f592b51d08a658e88c6a7a7596e0edf5793b)
(cherry picked from commit 7e10dd03b5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2019-10-03 16:16:23 -07:00
Kitti Nanasi aeb3ae1c4c HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc9200)
 2019-10-03 15:59:41 -07:00
Ayush Saxena 12f2d121ca HDFS-14849. Erasure Coding: the internal block is replicated many times when datanode is decommissioning. Contributed by HuangTao. 
(cherry picked from commit c4c8d5fd0e)
(cherry picked from commit 9a9edfa87e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
 2019-10-03 15:54:39 -07:00
Giovanni Matteo Fumarola 76c30e4f3c HADOOP-16625: Backport HADOOP-14624 to branch-3.1. 
(cherry picked from commit 79c97f6a0b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestSaveNamespace.java
 2019-10-03 15:45:23 -07:00
Inigo Goiri 0160e2c97f HDFS-14527. Stop all DataNodes may result in NN terminate. Contributed by He Xiaoqiao. 
(cherry picked from commit 944adc61b1)
(cherry picked from commit 6148acebc3)
 2019-10-03 15:44:56 -07:00