Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,600 Commits 326 Branches 371 Tags
Commit Graph

19600 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric Badger
50c1f932c3 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c83b9752e772ece7a716a33045752bf)
 2019-10-28 16:54:05 +00:00
Rakesh Radhakrishnan
2db0b8f6ca HDFS-14818. Check native pmdk lib by 'hadoop checknative' command. Contributed by Feilong He. 
(cherry picked from commit 659c88801d008bb352d10a1cb3bd0e401486cc9b)
 2019-10-27 22:24:44 +05:30
Rakesh Radhakrishnan
cd1277398a HDFS-14700. Clean up pmem cache before setting pmem cache capacity. Contributed by Feilong He. 
(cherry picked from commit f6fa865d6fcb0ef0a25a00615f16f383e5032373)
 2019-10-27 22:24:30 +05:30
Rakesh Radhakrishnan
f2563dcca1 HDFS-14357. Update documentation for HDFS cache on SCM support. Contributed by Feilong He. 
(cherry picked from commit 30a8f840f1572129fe7d02f8a784c47ab57ce89a)
 2019-10-27 22:24:14 +05:30
Rakesh Radhakrishnan
df0dcc7493 HDFS-14458. Report pmem stats to namenode. Contributed by Feilong He. 
(cherry picked from commit e98adb00b7da8fa913b86ecf2049444b1d8617d4)
 2019-10-27 22:24:00 +05:30
Sammi Chen
9281b72550 HDFS-14356. Implement HDFS cache on SCM with native PMDK libs. Contributed by Feilong He. 
(cherry picked from commit d1aad444907e1fc5314e8e64529e57c51ed7561c)
 2019-10-27 22:23:44 +05:30
Rakesh Radhakrishnan
f982f96623 HDFS-14402. Use FileChannel.transferTo() method for transferring block to SCM cache. Contributed by Feilong He. 
(cherry picked from commit 37900c5639f8ba8d41b9fedc3d41ee0fbda7d5db)
 2019-10-27 22:23:33 +05:30
Rakesh Radhakrishnan
3b2a7aabf7 HDFS-14401. Refine the implementation for HDFS cache on SCM. Contributed by Feilong He. 2019-10-27 22:23:16 +05:30
Uma Maheswara Rao G
b00f9d89e4 HDFS-14355 : Implement HDFS cache on SCM by using pure java mapped byte buffer. Contributed by Feilong He. 2019-10-27 22:20:00 +05:30
Rakesh Radhakrishnan
712749c1a0 HDFS-14393. Refactor FsDatasetCache for SCM cache implementation. Contributed by Rakesh R 
(cherry picked from commit f3f51284d57ef2e0c7e968b6eea56eab578f7e93)
 2019-10-27 22:14:56 +05:30
Uma Maheswara Rao G
aab9fefddc HDFS-14354: Refactor MappableBlock to align with the implementation of SCM cache. Contributed by Feilong He. 
(cherry picked from commit ba50a36a3ead628c3d44d384f7ed4d2b3a55dd07)
 2019-10-27 22:14:39 +05:30
Fei Hui
287fab6346 HDFS-14923. Remove dead code from HealthMonitor. Contributed by Fei Hui. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 7be5508d9b35892f483ba6022b6aced7648b8fa3)
(cherry picked from commit 54d3e212bfdcc6b473969a70faca339184849e61)
 2019-10-25 15:06:56 -07:00
zhaoyim
44ca2fe4db HDFS-14308. DFSStripedInputStream curStripeBuf is not freed by unbuffer() (#1667) 
Reviewed-by: Aravindan Vijayan <avijayan@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 30db895b59d250788d029cb2013bb4712ef9b546)
(cherry picked from commit 9316ca149f5e29d62af78808a7f897c81264cce8)
 2019-10-25 13:11:07 -07:00
Eric Badger
f5a3a9e138 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87a75e16b2cca870d99a5e1e28c31d9b)
 2019-10-25 17:17:14 +00:00
Wei-Chiu Chuang
c6e6d1c6cc HDFS-14910. Rename Snapshot with Pre Descendants Fail With IllegalArgumentException. Contributed by Wei-Chiu Chuang. (#1672) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit a1b4eebcc92976a9fb78ad5d3ab70c52cc0a5fa7)
(cherry picked from commit 8cf4bc4bda5b0d33e42168ed2688396524598ead)
 2019-10-24 13:14:18 -07:00
Wei-Chiu Chuang
a3493fe328 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7ee09b561e42c16ce4e91c0479908ad8)
(cherry picked from commit 570ffa1cd67990ef0e49c149abc04a92ed3670ac)
 2019-10-24 13:14:00 -07:00
Vinayakumar B
e640e809a9 HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:26:29 +05:30
Zhankun Tang
7175fc91d0 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:07:40 +08:00
Mukul Kumar Singh
51114e12ee HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad80b4efee020e1ddd06104121f26e31f)
(cherry picked from commit c9d07a42dc18016b4b2f1e750708d23e8b7f4e28)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
 2019-10-22 16:14:26 -07:00
Wei-Chiu Chuang
8ef6c0aab2 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 
(cherry picked from commit 31243f0d2991b6ceb484f2425fabbd35f53e9e29)
 2019-10-22 09:34:41 -07:00
Ayush Saxena
4d4f5325bd HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:47:08 +05:30
Kihwal Lee
362657c1a3 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef7735acd6f91b73d2ecb0227f8dd75a2e4)
 2019-10-21 16:35:50 -05:00
Eric E Payne
d8b4348036 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-10-21 21:14:23 +00:00
Ayush Saxena
9067a11b0d HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:43:03 +05:30
Ayush Saxena
9434f36b43 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:55:58 +05:30
Ayush Saxena
c5b4a2d115 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:36:27 +05:30
Szilard Nemeth
19755b9b36 MAPREDUCE-6441. Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes (wattsinabox, rchiang, haibochen via snemeth) 2019-10-18 15:25:50 +02:00
Duo Zhang
6097e909ec HADOOP-16600. StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-10-17 16:52:09 -07:00
Surendra Singh Lilhore
14beaae65e HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d720851eb6017906d664aa0fda2698225)
 2019-10-17 22:49:03 +05:30
Eric E Payne
889255576b YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-10-16 21:23:38 +00:00
Szilard Nemeth
f8c06f5a87 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)
(cherry picked from commit 0bbd48c7a8c0d1dd57ffe5d372d749e5e7e41d34)
 2019-10-16 14:24:17 +02:00
Haibo Chen
a70c6e9665 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af46db2859d7d2caf192861cae9b6a1a8)
 2019-10-15 22:20:56 +00:00
Haibo Chen
4ec409fe09 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88541f94cecc7fdbcaad010c4742cfdb)
 2019-10-15 15:41:20 +00:00
Szilard Nemeth
c44902212a YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-15 13:25:57 +02:00
Surendra Singh Lilhore
a47680a3f9 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd8737f3dff38f7bdad9721511c711c522b)
 2019-10-15 00:11:37 +05:30
Surendra Singh Lilhore
626a48d472 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-11 00:09:20 +05:30
Surendra Singh Lilhore
b396ebd0d5 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ace56490a92da70bde9e651ce515f80c)
 2019-10-09 23:47:21 +05:30
Szilard Nemeth
aabc18a777 Revert "YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl" 
This reverts commit 73bc8ef9b86cb9b3ee9bbc1786b49600e0ee600d.
 2019-10-09 19:59:02 +02:00
Szilard Nemeth
57e88a63cf YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-10-09 14:19:56 +02:00
Szilard Nemeth
73bc8ef9b8 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 
(cherry picked from commit 6f1ab9516869dbfdee034f4a17bccc379a39fc31)
(cherry picked from commit 42177e8b786efd7fa1365b68617382de022a5146)
 2019-10-09 13:28:44 +02:00
Szilard Nemeth
d1415b66b7 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 
(cherry picked from commit 35f093f5b3f310af49a8bb476e62173862f5af51)
(cherry picked from commit 0ddb48a3038d9c5132efd9f48110f9d23c53ac67)
 2019-10-09 13:13:51 +02:00
Chen Liang
01d9952f63 HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a6695f45f0d9cea5ae9aae83941d360)
 2019-10-08 12:08:17 -07:00
Hrishikesh Gadre
b74754a448 HADOOP-16491. Upgrade jetty version to 9.3.27. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3cc0ace2037ee039fc2e97d5737e92d3cb64cc72)
(cherry picked from commit aec58731a800a9eab539410f933d98cd3960c08f)
 2019-10-08 11:29:59 -07:00
Jonathan Hung
0dc85bc182 YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:18:52 -07:00
Szilard Nemeth
2c8cfb4f64 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:34:44 +02:00
Ayush Saxena
a3599c7e70 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:24:56 +05:30
Vinayakumar B
7ab02a67bc HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 
(cherry picked from commit 30eceec3420fc6be00d3878ba787bd9518d3ca0e)
 2019-10-04 13:53:50 -07:00
Eric Yang
166d38ceaa HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)
(cherry picked from commit 8bb2b00d38978859b22b892034eb3f559b820942)
 2019-10-04 12:08:21 -07:00
He Xiaoqiao
c61c114a3d HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37bffb77da17c025f0b9b673d7088bc6)
(cherry picked from commit e29ae7db1258f08339cf0f53968fce6f98ada3ac)
 2019-10-04 12:07:44 -07:00
He Xiaoqiao
d91c68729c HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e94442825fe286c2b18518925d980e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

(cherry picked from commit 80392e94b6dca16229fc35426d107184be68c908)
 2019-10-04 12:07:42 -07:00