Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,023 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8208 Commits

Author SHA1 Message Date
Ayush Saxena 051dae1f89 HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 
(cherry picked from commit 9e287054a8)
 2019-11-18 10:24:53 -08:00
Erik Krogen 60655bfe54 HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a)
 2019-11-15 10:11:56 -08:00
Erik Krogen dec765b329 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe711)
 2019-11-13 11:57:42 -08:00
Chao Sun 0c12d633b9 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 
(cherry picked from commit 97ec34e117)
 2019-11-12 11:40:56 -08:00
Siyao Meng e0cf1735e1 HADOOP-16676. Backport HADOOP-16152 to branch-3.2. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-11-12 11:38:42 -08:00
Surendra Singh Lilhore c4e22e0c80 HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 
(cherry picked from commit 320008bb7c)
 2019-11-09 22:28:15 +05:30
Ayush Saxena 2e843cb773 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:50:30 +05:30
Chen Liang c9ed0c3a37 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:12:51 -08:00
Chen Liang 431edfcbe4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:00:15 -08:00
Ayush Saxena 5941a91f64 HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-06 22:10:27 +05:30
Surendra Singh Lilhore aa1c795dc9 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c)
 2019-11-06 19:37:41 +05:30
Wei-Chiu Chuang 240b5ee380 HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 
(cherry picked from commit eb73ba6ed5)
 2019-11-04 09:40:57 -08:00
Zhao Junwang c9fc118991 HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
(cherry picked from commit de6b8b0c0b)
 2019-11-01 16:39:18 -07:00
Ayush Saxena b1821be1ec HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:51:50 +05:30
Ayush Saxena 239505a0af HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:06 +05:30
Jonathan Hung 5d2ffcc7aa Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
 2019-10-30 20:59:20 -07:00
Duo Zhang a37a4bc62f HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 20:26:54 -07:00
Eric Badger 3d9127dc97 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2019-10-28 16:38:28 +00:00
Rakesh Radhakrishnan a6cdcf676e HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d)
 2019-10-28 10:41:40 +05:30
Rakesh Radhakrishnan dc2fad4bd3 HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6f)
 2019-10-28 10:41:25 +05:30
Rakesh Radhakrishnan c6d59e6a47 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1)
 2019-10-28 10:41:12 +05:30
Rakesh Radhakrishnan 75c66bf044 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7)
 2019-10-28 10:40:59 +05:30
Sammi Chen 6eb5fb596f HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad44490)
 2019-10-28 10:40:46 +05:30
Rakesh Radhakrishnan 0fe720dc48 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639)
 2019-10-28 10:40:32 +05:30
Rakesh Radhakrishnan f3571c1f09 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-28 10:40:15 +05:30
Uma Maheswara Rao G 6bc73a983d HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-28 10:37:34 +05:30
Rakesh Radhakrishnan 0c6824120a HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d5)
 2019-10-28 10:33:45 +05:30
Uma Maheswara Rao G 3f6f095c06 HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3e)
 2019-10-28 10:33:29 +05:30
zhaoyim 9316ca149f HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59)
 2019-10-25 13:10:56 -07:00
Wei-Chiu Chuang 8cf4bc4bda HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc9)
 2019-10-24 13:13:27 -07:00
Wei-Chiu Chuang 570ffa1cd6 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7e)
 2019-10-24 13:13:19 -07:00
Vinayakumar B e1e3d9c81d HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:15:18 +05:30
Mukul Kumar Singh c9d07a42dc HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad8)
 2019-10-22 13:31:34 -07:00
Wei-Chiu Chuang 31243f0d29 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-22 09:34:11 -07:00
Ayush Saxena 8c74717720 HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:45:19 +05:30
Kihwal Lee 2d1f3611cd HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef773)
 2019-10-21 16:34:17 -05:00
Ayush Saxena 0f64ed89d2 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:42:21 +05:30
Ayush Saxena c2cd847a15 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:29:16 +05:30
Surendra Singh Lilhore 5c50eb2243 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d72)
 2019-10-17 22:39:57 +05:30
Surendra Singh Lilhore d1b30e8450 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd873)
 2019-10-14 23:47:42 +05:30
Surendra Singh Lilhore 2ca05ca02a HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ac)
 2019-10-09 23:32:21 +05:30
Chen Liang a12ab911ee HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a)
 2019-10-08 12:03:04 -07:00
Ayush Saxena 57ab854492 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:22:16 +05:30
Surendra Singh Lilhore a0cdebc409 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 382967be51)
 2019-10-08 00:26:27 +05:30
He Xiaoqiao e29ae7db12 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37)
 2019-10-04 12:07:30 -07:00
He Xiaoqiao 80392e94b6 HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2019-10-04 12:06:55 -07:00
Erik Krogen 6630c9b75d HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e00143)
 2019-10-04 11:15:49 -07:00
Erik Krogen 69b0c513a9 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa)
 2019-10-04 10:48:33 -07:00
Eric Yang 8bb2b00d38 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0)
 2019-10-04 13:16:08 -04:00
Siyao Meng 673c9d53ca HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b)
 2019-10-04 09:40:22 -07:00