Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,785 Commits 326 Branches 371 Tags
Commit Graph

19785 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Wei-Chiu Chuang
a3493fe328 HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang. (#1370) 
* HDFS-14492. Snapshot memory leak. Contributed by Wei-Chiu Chuang.

Change-Id: I9e5e450c07ad70aa1905973896c4f627042dbd37

* Fix checkstyle

Change-Id: I16d4bd4f03a971e1ed36cf57d89dc42357ef8fbf
(cherry picked from commit 6ef6594c7ee09b561e42c16ce4e91c0479908ad8)
(cherry picked from commit 570ffa1cd67990ef0e49c149abc04a92ed3670ac)
 2019-10-24 13:14:00 -07:00
Vinayakumar B
e640e809a9 HDFS-14921. Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing. Contributed by Ayush Saxena. 2019-10-24 12:26:29 +05:30
Zhankun Tang
7175fc91d0 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:07:40 +08:00
Mukul Kumar Singh
51114e12ee HDFS-14884. Add sanity check that zone key equals feinfo key while setting Xattrs. Contributed by Mukul Kumar Singh. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit a901405ad80b4efee020e1ddd06104121f26e31f)
(cherry picked from commit c9d07a42dc18016b4b2f1e750708d23e8b7f4e28)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirXAttrOp.java
 2019-10-22 16:14:26 -07:00
Wei-Chiu Chuang
8ef6c0aab2 HDFS-13901. INode access time is ignored because of race between open and rename. Contributed by Jinglun. 
(cherry picked from commit 31243f0d2991b6ceb484f2425fabbd35f53e9e29)
 2019-10-22 09:34:41 -07:00
Ayush Saxena
4d4f5325bd HDFS-14915. Move Superuser Check Before Taking Lock For Encryption API. Contributed by Ayush Saxena. 2019-10-22 18:47:08 +05:30
Kihwal Lee
362657c1a3 HDFS-12749. DN may not send block report to NN after NN restart. Contributed by Xiaoqiao He. 
(cherry picked from commit c4e27ef7735acd6f91b73d2ecb0227f8dd75a2e4)
 2019-10-21 16:35:50 -05:00
Eric E Payne
d8b4348036 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-10-21 21:14:23 +00:00
Ayush Saxena
9067a11b0d HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:43:03 +05:30
Ayush Saxena
9434f36b43 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:55:58 +05:30
Ayush Saxena
c5b4a2d115 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:36:27 +05:30
Szilard Nemeth
19755b9b36 MAPREDUCE-6441. Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes (wattsinabox, rchiang, haibochen via snemeth) 2019-10-18 15:25:50 +02:00
Duo Zhang
6097e909ec HADOOP-16600. StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1 
Signed-off-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: stack <stack@apache.org>
 2019-10-17 16:52:09 -07:00
Surendra Singh Lilhore
14beaae65e HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d720851eb6017906d664aa0fda2698225)
 2019-10-17 22:49:03 +05:30
Eric E Payne
889255576b YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-10-16 21:23:38 +00:00
Szilard Nemeth
f8c06f5a87 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)
(cherry picked from commit 0bbd48c7a8c0d1dd57ffe5d372d749e5e7e41d34)
 2019-10-16 14:24:17 +02:00
Haibo Chen
a70c6e9665 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af46db2859d7d2caf192861cae9b6a1a8)
 2019-10-15 22:20:56 +00:00
Haibo Chen
4ec409fe09 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88541f94cecc7fdbcaad010c4742cfdb)
 2019-10-15 15:41:20 +00:00
Szilard Nemeth
c44902212a YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-15 13:25:57 +02:00
Surendra Singh Lilhore
a47680a3f9 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd8737f3dff38f7bdad9721511c711c522b)
 2019-10-15 00:11:37 +05:30
Surendra Singh Lilhore
626a48d472 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 2019-10-11 00:09:20 +05:30
Surendra Singh Lilhore
b396ebd0d5 HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ace56490a92da70bde9e651ce515f80c)
 2019-10-09 23:47:21 +05:30
Szilard Nemeth
aabc18a777 Revert "YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl" 
This reverts commit 73bc8ef9b86cb9b3ee9bbc1786b49600e0ee600d.
 2019-10-09 19:59:02 +02:00
Szilard Nemeth
57e88a63cf YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-10-09 14:19:56 +02:00
Szilard Nemeth
73bc8ef9b8 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 
(cherry picked from commit 6f1ab9516869dbfdee034f4a17bccc379a39fc31)
(cherry picked from commit 42177e8b786efd7fa1365b68617382de022a5146)
 2019-10-09 13:28:44 +02:00
Szilard Nemeth
d1415b66b7 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 
(cherry picked from commit 35f093f5b3f310af49a8bb476e62173862f5af51)
(cherry picked from commit 0ddb48a3038d9c5132efd9f48110f9d23c53ac67)
 2019-10-09 13:13:51 +02:00
Chen Liang
01d9952f63 HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a6695f45f0d9cea5ae9aae83941d360)
 2019-10-08 12:08:17 -07:00
Hrishikesh Gadre
b74754a448 HADOOP-16491. Upgrade jetty version to 9.3.27. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3cc0ace2037ee039fc2e97d5737e92d3cb64cc72)
(cherry picked from commit aec58731a800a9eab539410f933d98cd3960c08f)
 2019-10-08 11:29:59 -07:00
Jonathan Hung
0dc85bc182 YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:18:52 -07:00
Szilard Nemeth
2c8cfb4f64 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:34:44 +02:00
Ayush Saxena
a3599c7e70 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:24:56 +05:30
Vinayakumar B
7ab02a67bc HDFS-13806. EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory. Contributed by Ayush Saxena. 
(cherry picked from commit 30eceec3420fc6be00d3878ba787bd9518d3ca0e)
 2019-10-04 13:53:50 -07:00
Eric Yang
166d38ceaa HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)
(cherry picked from commit 8bb2b00d38978859b22b892034eb3f559b820942)
 2019-10-04 12:08:21 -07:00
He Xiaoqiao
c61c114a3d HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37bffb77da17c025f0b9b673d7088bc6)
(cherry picked from commit e29ae7db1258f08339cf0f53968fce6f98ada3ac)
 2019-10-04 12:07:44 -07:00
He Xiaoqiao
d91c68729c HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e94442825fe286c2b18518925d980e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

(cherry picked from commit 80392e94b6dca16229fc35426d107184be68c908)
 2019-10-04 12:07:42 -07:00
Erik Krogen
9fdb849e03 HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e0014343f60f853cb796781ca1fa03a72efd)
(cherry picked from 6630c9b75d65deefb5550e355eef7783909a57bc)
 2019-10-04 11:34:37 -07:00
Arpit Agarwal
8bb98076be HDFS-2470. NN should automatically set permissions on dfs.namenode.*.dir. Contributed by Siddharth Wagle. 
(cherry picked from commit a64a43b77fb1032dcb66730a6b6257a24726c256)
(cherry picked from commit 8b1238171752d03712ae69d8464108ef0803ae10)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestStartup.java
 2019-10-04 11:23:03 -07:00
Erik Krogen
56caacac1f HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
(cherry picked from 69b0c513a9b11cc7f795747732173b36aacbe794)
 2019-10-04 10:48:17 -07:00
Siyao Meng
61dc877b00 HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b384af93726e4f2e6f5e28c6e3a8f65)
 2019-10-04 09:19:30 -07:00
Ayush Saxena
8ce5015e94 HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46)
(cherry picked from commit e3f54a7babf942efbe879aabef65a5b6df57fb65)
 2019-10-04 08:52:03 -07:00
Takanobu Asanuma
c48f2730b4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085dbc561c785cef1613bacaf6735d909)
 2019-10-04 16:49:58 +09:00
Stephen O'Donnell
71dcb8f6d4 HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167ff9566012ef32104a3964887d62c899)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java

(cherry picked from commit 966193153f9cd75e009c8db3502e1b3ba2cdfa25)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
 2019-10-03 22:28:22 -07:00
Tsz Wo Nicholas Sze
ce804be72e HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 
(cherry picked from commit 9192f71e21847ad86bc9ff23847d8957dfe8ae58)
 2019-10-03 18:28:13 -07:00
Inigo Goiri
bb1015719c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b227495d748abe8884eee34db507476cee1)
 2019-10-03 18:21:32 -07:00
Wei-Chiu Chuang
aa5dfb2f8a HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 
(cherry picked from commit b94eba9f11af66b10638dd255c224e946d842b8c)
 2019-10-03 17:30:00 -07:00
Inigo Goiri
af688579a6 HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff54f79de0e6439d6c77c2ed437989f10)
 2019-10-03 17:27:52 -07:00
Shashikant Banerjee
2eb5865ad1 HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a53757f8081e8224e0d4b557fe6a0e2)
 2019-10-03 17:26:13 -07:00
Vinayakumar B
c41791671c HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a5305bea1a40c87a4abc0793802bea5c75)
 2019-10-03 17:15:38 -07:00
Vinayakumar B
1a7cb7aba1 HDFS-14124. EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs. Contributed by Ayush Saxena. 
(cherry picked from commit 39dc7345b80e27ba8bd1ff4c19ca241aef5ac0fc)
(cherry picked from commit abe14d32d41b4a2bf630e8a89d794d729cee119b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
 2019-10-03 16:59:00 -07:00
Shweta Yakkali
e68ccca2d7 HDFS-14262. [SBN read] Make Log.WARN message in GlobalStateIdContext more informative. Contributed by Shweta Yakkali. 2019-10-03 16:22:00 -07:00