Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,205 Commits 326 Branches 371 Tags
Commit Graph

21205 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Chen Liang
c9ed0c3a37 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:12:51 -08:00
Chen Liang
431edfcbe4 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:00:15 -08:00
Ayush Saxena
5941a91f64 HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-06 22:10:27 +05:30
Surendra Singh Lilhore
aa1c795dc9 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c212b26d75268ee3659aa2cadcff349)
 2019-11-06 19:37:41 +05:30
Ahmed Hussein
56988e88f7 MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)
 2019-11-05 15:21:51 -06:00
Jonathan Eagles
254e18dcaf Revert "YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph." 
This reverts commit 11c763c22055fea367b19b338a3d8067f9386ba4.
 2019-11-05 15:10:01 -06:00
Chen Liang
a58f466d07 HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 
(cherry picked from commit 01cb958af44b2376bcf579cc65d90566530f733d)
 2019-11-04 17:18:27 -08:00
Sunil G
597b315811 YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 
(cherry picked from commit 51e7d1b37eb06165657befdae82c015055c58334)
 2019-11-04 23:28:39 +05:30
Wei-Chiu Chuang
240b5ee380 HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 
(cherry picked from commit eb73ba6ed5f7c5500cc0ef36ca22aae4e71046fa)
 2019-11-04 09:40:57 -08:00
Sunil G
11c763c220 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 
(cherry picked from commit d462308e0476053d1aae65eca2752eacce797e22)
 2019-11-03 08:48:04 +05:30
Zhao Junwang
c9fc118991 HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
(cherry picked from commit de6b8b0c0b1933aab2af3e8adc50a2091d428238)
 2019-11-01 16:39:18 -07:00
Ayush Saxena
b1821be1ec HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:51:50 +05:30
Jonathan Hung
9ccfc2d6d3 Add 2.10.0 release notes for HDFS-12943 
(cherry picked from commit ef9d12df24c0db76fd37a95551db7920d27d740c)
 2019-10-31 20:34:36 -07:00
Ayush Saxena
239505a0af HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:06 +05:30
Jonathan Hung
5d2ffcc7aa Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
 2019-10-30 20:59:20 -07:00
Duo Zhang
a37a4bc62f HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 20:26:54 -07:00
Wei-Chiu Chuang
d25211af30 HADOOP-16675. Upgrade jackson-databind to 2.9.10.1. Contributed by Lisheng Sun. 
(cherry picked from commit 012756a0d9eddd02aaba1509cdfb397af2d28d1b)
 2019-10-29 13:37:42 -07:00
Steve Loughran
1932499eaf HADOOP-16669. TestRawLocalFileSystemContract.testPermission fails if no native library. 
Contributed by Steve Loughran.

Change-Id: I622e36f17a63c775c5e33e93b8b26e461d0bc451
 2019-10-29 17:10:46 +00:00
Eric Badger
3d9127dc97 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c83b9752e772ece7a716a33045752bf)
 2019-10-28 16:38:28 +00:00
Rakesh Radhakrishnan
a6cdcf676e HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)
 2019-10-28 10:41:40 +05:30
Rakesh Radhakrishnan
dc2fad4bd3 HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6fcb0ef0a25a00615f16f383e5032373)
 2019-10-28 10:41:25 +05:30
Rakesh Radhakrishnan
c6d59e6a47 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1572129fe7d02f8a784c47ab57ce89a)
 2019-10-28 10:41:12 +05:30
Rakesh Radhakrishnan
75c66bf044 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7da8fa913b86ecf2049444b1d8617d4)
 2019-10-28 10:40:59 +05:30
Sammi Chen
6eb5fb596f HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad444907e1fc5314e8e64529e57c51ed7561c)
 2019-10-28 10:40:46 +05:30
Rakesh Radhakrishnan
0fe720dc48 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db)
 2019-10-28 10:40:32 +05:30
Rakesh Radhakrishnan
f3571c1f09 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-28 10:40:15 +05:30
Uma Maheswara Rao G
6bc73a983d HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-28 10:37:34 +05:30
Rakesh Radhakrishnan
0c6824120a HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d57ef2e0c7e968b6eea56eab578f7e93)
 2019-10-28 10:33:45 +05:30
Uma Maheswara Rao G
3f6f095c06 HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3ead628c3d44d384f7ed4d2b3a55dd07)
 2019-10-28 10:33:29 +05:30
Fei Hui
54d3e212bf HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7be5508d9b35892f483ba6022b6aced7648b8fa3)
 2019-10-25 15:06:46 -07:00
zhaoyim
9316ca149f HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59d250788d029cb2013bb4712ef9b546)
 2019-10-25 13:10:56 -07:00
Eric Badger
fa6b27ea8d YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87a75e16b2cca870d99a5e1e28c31d9b)
 2019-10-25 17:15:48 +00:00
Wei-Chiu Chuang
8cf4bc4bda HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc92976a9fb78ad5d3ab70c52cc0a5fa7)
 2019-10-24 13:13:27 -07:00
Wei-Chiu Chuang
570ffa1cd6 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7ee09b561e42c16ce4e91c0479908ad8)
 2019-10-24 13:13:19 -07:00
Vinayakumar B
e1e3d9c81d HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:15:18 +05:30
Mukul Kumar Singh
c9d07a42dc HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad80b4efee020e1ddd06104121f26e31f)
 2019-10-22 13:31:34 -07:00
Wei-Chiu Chuang
31243f0d29 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 2019-10-22 09:34:11 -07:00
Ayush Saxena
8c74717720 HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:45:19 +05:30
Kihwal Lee
2d1f3611cd HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef7735acd6f91b73d2ecb0227f8dd75a2e4)
 2019-10-21 16:34:17 -05:00
Eric E Payne
ea574087d1 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-10-21 20:56:40 +00:00
Ayush Saxena
0f64ed89d2 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:42:21 +05:30
Ayush Saxena
ae4b1325d1 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:55:43 +05:30
Ayush Saxena
c2cd847a15 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:29:16 +05:30
Surendra Singh Lilhore
5c50eb2243 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d720851eb6017906d664aa0fda2698225)
 2019-10-17 22:39:57 +05:30
Da Zhou
fe96407451
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 
(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)
 2019-10-16 14:27:11 -07:00
Eric E Payne
23b72d8ae1 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-10-16 21:13:02 +00:00
Sunil G
9672b81fa3 YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 
(cherry picked from commit 090f73a9aaeb4c03032a78c0a837478b527c27e0)
 2019-10-16 18:15:34 +05:30
Szilard Nemeth
0bbd48c7a8 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)
 2019-10-16 14:11:30 +02:00
Haibo Chen
3a5474c61e YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af46db2859d7d2caf192861cae9b6a1a8)
 2019-10-15 22:14:33 +00:00
Haibo Chen
1344823d4d YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88541f94cecc7fdbcaad010c4742cfdb)
 2019-10-15 15:32:01 +00:00