Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,734 Commits 326 Branches 371 Tags
Commit Graph

19734 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Szilard Nemeth
0a4e6b7b62 YARN-9836. General usability improvements in showSimulationTrace.html. Contributed by Adam Antal 2019-11-19 21:22:01 +01:00
Siyao Meng
df13f6d485 HADOOP-16703. Backport HADOOP-16152 to branch-3.1. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-11-19 12:14:27 -08:00
Eric Yang
1669a5c1cb YARN-9983. Fixed typo in YARN Service overview. 
Contributed by Denes Gerencser
 2019-11-19 14:18:38 -05:00
Sunil G
9a51380208 YARN-9873. Version Number for each Scheduler Config Change. Contributed by Prabhu Joseph. 
(cherry picked from commit be901f49628a553cfe6a3942478cb528cce2b266)
 2019-11-19 17:17:45 +05:30
Abhishek Modi
96e07b7e97 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 
(cherry picked from commit 751b5a1ac871734c46cb934d100a5367f1dbf791)
 2019-11-19 16:41:04 +05:30
Sunil G
dab4cc6320 YARN-9909. Offline Format of YarnConfigurationStore. Contributed by Prabhu Joseph. 2019-11-19 16:19:19 +05:30
Sunil G
9b17fa7091 YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 
(cherry picked from commit 51e7d1b37eb06165657befdae82c015055c58334)
 2019-11-19 15:43:19 +05:30
Sunil G
f24d1fac35 YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 215f2052fc3b7e366e8bd1bd332663966fa9206c)
 2019-11-19 14:05:51 +05:30
Ayush Saxena
b4f0c63a6a HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun. 
(cherry picked from commit 9e287054a8aa0725643bc5c90601645302fffade)
(cherry picked from commit d16441460215155b25fa649a7ca733e579d97929)
 2019-11-18 10:23:48 -08:00
Ayush Saxena
7d3c67b4a7 HADOOP-16712. Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml. Contributed by Xieming Li. 2019-11-18 19:59:18 +05:30
Prabhu Joseph
d271d174ca YARN-9900. Revert Invalid Config and Refresh Support in SchedulerConfig Format. 
Signed-off-by: prabhujoseph <pjoseph@cloudera.com>
 2019-11-18 15:02:59 +05:30
Lukas Majercak
50d98a6568 HADOOP-16705. MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log. 
MBeanInfoBuilder's get() method DEBUG logs all the MBeanAttributeInfo attributes that it gathered. This can have a high memory churn that can be easily avoided.

(cherry picked from commit c73334a924d2009d136a8ee62278aaac53418a24)
 2019-11-15 13:29:29 -08:00
Erik Krogen
17779adb32 HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen. 
(cherry picked from b2cc8b6b4a78f31cdd937dc4d1a2255f80c5881e)
(cherry picked from 60655bfe54e138957ef5bbf480a4541bd83152fd)
 2019-11-15 11:14:49 -08:00
Akira Ajisaka
b9e4fd5f4c
HADOOP-15097. AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path. Contributed by Xieming Li. 
(cherry picked from commit 92c28c100ee1aa414948cd510321ad13cb8639bc)
 2019-11-15 13:51:35 +09:00
Erik Krogen
5cf36aa2b1 HDFS-14979 Allow Balancer to submit getBlocks calls to Observer Nodes when possible. Contributed by Erik Krogen. 
(cherry picked from 586defe7113ed246ed0275bb3833882a3d873d70)
(cherry picked from dec765b329d3947f30273c0e7f0c4eb607ec42c9)
 2019-11-13 11:57:59 -08:00
Xudong Cao
b4f39c2a8a HADOOP-16677. Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout. (#1687) 
(cherry picked from commit df6b3162c11987ba5299c69cb251332228dacf36)
(cherry picked from commit 975f669141add885d631685d693e81c44c190751)
 2019-11-12 14:21:59 -08:00
Chao Sun
7e0f1eabd8 HDFS-14959: [SBNN read] access time should be turned off (#1706) Contributed by Chao Sun. 
(cherry picked from commit 97ec34e117af71e1a9950b8002131c45754009c7)
(cherry picked from commit 0c12d633b9797f2cb14246fe020993b2e9123e82)
 2019-11-12 11:41:09 -08:00
Surendra Singh Lilhore
9179046368 HDFS-14720. DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE. Contributed by hemanthboyina. 
(cherry picked from commit 320008bb7cc558b1300398178bd2f48cbf0b6c80)
 2019-11-09 22:36:48 +05:30
Ayush Saxena
9aa59c0629 HDFS-14958. TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup. Contributed by Jim Brennan. 2019-11-07 22:42:07 +05:30
Akira Ajisaka
3ac564ddb1 HDFS-14590. [SBN Read] Add the document link to the top page. Contributed by Takanobu Asanuma. 
(cherry picked from commit 48d7f00da8aab25707dc08219122541867edfa65)
 2019-11-07 22:30:58 +09:00
Chen Liang
76ca864406 HDFS-14806. Bootstrap standby may fail if with in-progress tailing. Contributed by Chen Liang and Erik Krogen. 2019-11-06 16:21:41 -08:00
Chen Liang
c6cbf67b82 HDFS-14941. Potential editlog race condition can cause corrupted file. Contributed by Chen Liang and Konstantin Shvachko. 2019-11-06 15:11:09 -08:00
Ayush Saxena
5841d0b2f9 HDFS-14946. Erasure Coding: Block recovery failed during decommissioning. Contributed by Fei Hui. 2019-11-06 22:17:11 +05:30
Surendra Singh Lilhore
4690142e86 HDFS-14384. When lastLocatedBlock token expire, it will take 1~3s second to refetch it. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c36014165c212b26d75268ee3659aa2cadcff349)
 2019-11-06 19:48:16 +05:30
Ahmed Hussein
cbd501636c MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)
 2019-11-05 15:29:19 -06:00
Chen Liang
7b4f44573f HADOOP-16013. DecayRpcScheduler decay thread should run as a daemon. Contributed by Erik Krogen. 
(cherry picked from commit 01cb958af44b2376bcf579cc65d90566530f733d)
(cherry picked from commit a58f466d074becff11b0757b1b3cf9c8f6bb57a8)
 2019-11-04 17:18:37 -08:00
Wei-Chiu Chuang
c9ff4b7a1b HDFS-14945. Revise PacketResponder's log. Contributed by Xudong Cao. 
(cherry picked from commit eb73ba6ed5f7c5500cc0ef36ca22aae4e71046fa)
(cherry picked from commit 240b5ee3802550950707c91f22154386d915fffb)
 2019-11-04 09:41:05 -08:00
Zhao Junwang
cda8b2ac69 HDFS-14925. Rename operation should check nest snapshot (#1670) 
If the src directory or any of its descendant is snapshottable
and the dst directory or any of its ancestors is snapshottable,
we consider this as nested snapshot, which should be denied.

Reviewed-by: Shashikant Banerjee <shashikant@apache.org>
(cherry picked from commit de6b8b0c0b1933aab2af3e8adc50a2091d428238)
(cherry picked from commit c9fc118991104acf9109aa769fe1a0e84f425a61)
 2019-11-01 16:39:29 -07:00
Ayush Saxena
9956d18129 HDFS-13736. BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false. Contributed by hu xiaodong. 2019-11-01 22:52:12 +05:30
Jonathan Hung
f29481f046 Add 2.10.0 release notes for HDFS-12943 
(cherry picked from commit ef9d12df24c0db76fd37a95551db7920d27d740c)
(cherry picked from commit 9ccfc2d6d3f3819da58ea9f0ff056ed03970a6e1)
 2019-10-31 20:35:20 -07:00
Ayush Saxena
98e3c6af2c HDFS-14920. Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission. Contributed by Fei Hui. 2019-10-31 23:53:37 +05:30
Duo Zhang
07f59c023c HADOOP-16598. Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches 
Signed-off-by: stack <stack@apache.org>
 2019-10-30 21:30:34 -07:00
Jonathan Hung
4d274f60bc Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aac608746946dc2ed5711549a01a2055)
 2019-10-30 21:00:55 -07:00
Steve Loughran
ad881214bc HADOOP-16669. TestRawLocalFileSystemContract.testPermission fails if no native library. 
Contributed by Steve Loughran.

Change-Id: I622e36f17a63c775c5e33e93b8b26e461d0bc451
 2019-10-29 17:11:25 +00:00
Eric Badger
50c1f932c3 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c83b9752e772ece7a716a33045752bf)
 2019-10-28 16:54:05 +00:00
Rakesh Radhakrishnan
2db0b8f6ca HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)
 2019-10-27 22:24:44 +05:30
Rakesh Radhakrishnan
cd1277398a HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6fcb0ef0a25a00615f16f383e5032373)
 2019-10-27 22:24:30 +05:30
Rakesh Radhakrishnan
f2563dcca1 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1572129fe7d02f8a784c47ab57ce89a)
 2019-10-27 22:24:14 +05:30
Rakesh Radhakrishnan
df0dcc7493 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7da8fa913b86ecf2049444b1d8617d4)
 2019-10-27 22:24:00 +05:30
Sammi Chen
9281b72550 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad444907e1fc5314e8e64529e57c51ed7561c)
 2019-10-27 22:23:44 +05:30
Rakesh Radhakrishnan
f982f96623 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db)
 2019-10-27 22:23:33 +05:30
Rakesh Radhakrishnan
3b2a7aabf7 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-27 22:23:16 +05:30
Uma Maheswara Rao G
b00f9d89e4 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-27 22:20:00 +05:30
Rakesh Radhakrishnan
712749c1a0 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d57ef2e0c7e968b6eea56eab578f7e93)
 2019-10-27 22:14:56 +05:30
Uma Maheswara Rao G
aab9fefddc HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3ead628c3d44d384f7ed4d2b3a55dd07)
 2019-10-27 22:14:39 +05:30
Fei Hui
287fab6346 HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7be5508d9b35892f483ba6022b6aced7648b8fa3)
(cherry picked from commit 54d3e212bfdcc6b473969a70faca339184849e61)
 2019-10-25 15:06:56 -07:00
zhaoyim
44ca2fe4db HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59d250788d029cb2013bb4712ef9b546)
(cherry picked from commit 9316ca149f5e29d62af78808a7f897c81264cce8)
 2019-10-25 13:11:07 -07:00
Eric Badger
f5a3a9e138 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87a75e16b2cca870d99a5e1e28c31d9b)
 2019-10-25 17:17:14 +00:00
Wei-Chiu Chuang
c6e6d1c6cc HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc92976a9fb78ad5d3ab70c52cc0a5fa7)
(cherry picked from commit 8cf4bc4bda5b0d33e42168ed2688396524598ead)
 2019-10-24 13:14:18 -07:00
Wei-Chiu Chuang
a3493fe328 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7ee09b561e42c16ce4e91c0479908ad8)
(cherry picked from commit 570ffa1cd67990ef0e49c149abc04a92ed3670ac)
 2019-10-24 13:14:00 -07:00