Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,662 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

19662 Commits

Author SHA1 Message Date
Ayush Saxena 5841d0b2f9 HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-06 22:17:11 +05:30
Surendra Singh Lilhore 4690142e86 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c)
 2019-11-06 19:48:16 +05:30
Ahmed Hussein cbd501636c MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit ed302f1fed)
 2019-11-05 15:29:19 -06:00
Chen Liang 7b4f44573f HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 
(cherry picked from commit 01cb958af4)
(cherry picked from commit a58f466d07)
 2019-11-04 17:18:37 -08:00
Wei-Chiu Chuang c9ff4b7a1b HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 
(cherry picked from commit eb73ba6ed5)
(cherry picked from commit 240b5ee380)
 2019-11-04 09:41:05 -08:00
Zhao Junwang cda8b2ac69 HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
(cherry picked from commit de6b8b0c0b)
(cherry picked from commit c9fc118991)
 2019-11-01 16:39:29 -07:00
Ayush Saxena 9956d18129 HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:52:12 +05:30
Jonathan Hung f29481f046 Add 2.10.0 release notes for HDFS-12943 
(cherry picked from commit ef9d12df24)
(cherry picked from commit 9ccfc2d6d3)
 2019-10-31 20:35:20 -07:00
Ayush Saxena 98e3c6af2c HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:37 +05:30
Duo Zhang 07f59c023c HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:30:34 -07:00
Jonathan Hung 4d274f60bc Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aa)
 2019-10-30 21:00:55 -07:00
Steve Loughran ad881214bc HADOOP-16669. TestRawLocalFileSystemContract.testPermission fails if no native library. 
Contributed by Steve Loughran.

Change-Id: I622e36f17a63c775c5e33e93b8b26e461d0bc451
 2019-10-29 17:11:25 +00:00
Eric Badger 50c1f932c3 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2019-10-28 16:54:05 +00:00
Rakesh Radhakrishnan 2db0b8f6ca HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d)
 2019-10-27 22:24:44 +05:30
Rakesh Radhakrishnan cd1277398a HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6f)
 2019-10-27 22:24:30 +05:30
Rakesh Radhakrishnan f2563dcca1 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1)
 2019-10-27 22:24:14 +05:30
Rakesh Radhakrishnan df0dcc7493 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7)
 2019-10-27 22:24:00 +05:30
Sammi Chen 9281b72550 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad44490)
 2019-10-27 22:23:44 +05:30
Rakesh Radhakrishnan f982f96623 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639)
 2019-10-27 22:23:33 +05:30
Rakesh Radhakrishnan 3b2a7aabf7 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-27 22:23:16 +05:30
Uma Maheswara Rao G b00f9d89e4 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-27 22:20:00 +05:30
Rakesh Radhakrishnan 712749c1a0 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d5)
 2019-10-27 22:14:56 +05:30
Uma Maheswara Rao G aab9fefddc HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3e)
 2019-10-27 22:14:39 +05:30
Fei Hui 287fab6346 HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7be5508d9b)
(cherry picked from commit 54d3e212bf)
 2019-10-25 15:06:56 -07:00
zhaoyim 44ca2fe4db HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59)
(cherry picked from commit 9316ca149f)
 2019-10-25 13:11:07 -07:00
Eric Badger f5a3a9e138 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:17:14 +00:00
Wei-Chiu Chuang c6e6d1c6cc HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc9)
(cherry picked from commit 8cf4bc4bda)
 2019-10-24 13:14:18 -07:00
Wei-Chiu Chuang a3493fe328 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7e)
(cherry picked from commit 570ffa1cd6)
 2019-10-24 13:14:00 -07:00
Vinayakumar B e640e809a9 HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:26:29 +05:30
Zhankun Tang 7175fc91d0 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:07:40 +08:00
Mukul Kumar Singh 51114e12ee HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad8)
(cherry picked from commit c9d07a42dc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
 2019-10-22 16:14:26 -07:00
Wei-Chiu Chuang 8ef6c0aab2 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 
(cherry picked from commit 31243f0d29)
 2019-10-22 09:34:41 -07:00
Ayush Saxena 4d4f5325bd HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:47:08 +05:30
Kihwal Lee 362657c1a3 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef773)
 2019-10-21 16:35:50 -05:00
Eric E Payne d8b4348036 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f)
 2019-10-21 21:14:23 +00:00
Ayush Saxena 9067a11b0d HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:43:03 +05:30
Ayush Saxena 9434f36b43 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:55:58 +05:30
Ayush Saxena c5b4a2d115 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:36:27 +05:30
Szilard Nemeth 19755b9b36 MAPREDUCE-6441. Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes (wattsinabox, rchiang, haibochen via snemeth) 2019-10-18 15:25:50 +02:00
Duo Zhang 6097e909ec HADOOP-16600. StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-10-17 16:52:09 -07:00
Surendra Singh Lilhore 14beaae65e HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d72)
 2019-10-17 22:49:03 +05:30
Eric E Payne 889255576b YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988)
 2019-10-16 21:23:38 +00:00
Szilard Nemeth f8c06f5a87 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d99)
(cherry picked from commit 0bbd48c7a8)
 2019-10-16 14:24:17 +02:00
Haibo Chen a70c6e9665 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af4)
 2019-10-15 22:20:56 +00:00
Haibo Chen 4ec409fe09 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88)
 2019-10-15 15:41:20 +00:00
Szilard Nemeth c44902212a YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-15 13:25:57 +02:00
Surendra Singh Lilhore a47680a3f9 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd873)
 2019-10-15 00:11:37 +05:30
Surendra Singh Lilhore 626a48d472 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-11 00:09:20 +05:30
Surendra Singh Lilhore b396ebd0d5 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ac)
 2019-10-09 23:47:21 +05:30
Szilard Nemeth aabc18a777 Revert "YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl" 
This reverts commit 73bc8ef9b8.
 2019-10-09 19:59:02 +02:00