Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,066 Commits 326 Branches 371 Tags
Commit Graph

21066 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric E Payne
ea574087d1 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-10-21 20:56:40 +00:00
Ayush Saxena
0f64ed89d2 HDFS-14913. Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType(). Contributed by Ayush Saxena. 2019-10-21 18:42:21 +05:30
Ayush Saxena
ae4b1325d1 HADOOP-16662. Remove unnecessary InnerNode check in NetworkTopology#add(). Contributed by Lisheng Sun. 2019-10-20 11:55:43 +05:30
Ayush Saxena
c2cd847a15 HDFS-14847. Erasure Coding: Blocks are over-replicated while EC decommissioning. Contributed by Fei Hui. 2019-10-20 06:29:16 +05:30
Surendra Singh Lilhore
5c50eb2243 HDFS-14909. DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 54dc6b7d720851eb6017906d664aa0fda2698225)
 2019-10-17 22:39:57 +05:30
Da Zhou
fe96407451
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 
(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)
 2019-10-16 14:27:11 -07:00
Eric E Payne
23b72d8ae1 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-10-16 21:13:02 +00:00
Sunil G
9672b81fa3 YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 
(cherry picked from commit 090f73a9aaeb4c03032a78c0a837478b527c27e0)
 2019-10-16 18:15:34 +05:30
Szilard Nemeth
0bbd48c7a8 HADOOP-16580. Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException. Contributed by Adam Antal 
(cherry picked from commit c79a5f2d9930f58ad95864c59cd0a6164cd53280)
 2019-10-16 14:11:30 +02:00
Haibo Chen
3a5474c61e YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af46db2859d7d2caf192861cae9b6a1a8)
 2019-10-15 22:14:33 +00:00
Haibo Chen
1344823d4d YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88541f94cecc7fdbcaad010c4742cfdb)
 2019-10-15 15:32:01 +00:00
Szilard Nemeth
b10fdd136a YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-15 13:24:59 +02:00
Surendra Singh Lilhore
d1b30e8450 HDFS-14886. In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec. Contributed by hemanthboyina. 
(cherry picked from commit 336abbd8737f3dff38f7bdad9721511c711c522b)
 2019-10-14 23:47:42 +05:30
Siyao Meng
793df595f2
HADOOP-16619. Upgrade jackson and jackson-databind to 2.9.10 (#1554). Contributed by Siyao Meng. 
(cherry picked from commit d947ded05388c36f8ac688fc697cfafcbaaa58e7)
 2019-10-10 15:41:19 +09:00
Akira Ajisaka
510b898c24
HADOOP-16533. Upgrade jackson-databind to 2.9.9.3. (#1354). Contributed by Akira Ajisaka. 
Reviewed-by: Siyao Meng <smeng@cloudera.com>
(cherry picked from commit d85d68f6ffb046dc801b263ce664fddb85d8d166)
 2019-10-10 15:41:09 +09:00
Siyao Meng
320337d737
HADOOP-16487. Update jackson-databind to 2.9.9.2. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9680a8b2371c2ff8b5314e8fa39bdb9f8db46f96)
 2019-10-10 15:40:57 +09:00
Siyao Meng
146908cf3d
HADOOP-16451. Update jackson-databind to 2.9.9.1. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 9b8b3acb0a2b87356056c23f3d0f30a97a38cd3d)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/pom.xml
 2019-10-10 15:40:28 +09:00
Shweta Yakkali
43576f4738
HADOOP-16365. Upgrade jackson-databind to 2.9.9. Contributed by Shweta Yakkali. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit cf84881dea11639bed48b4c8e8a785a535510e6d)
 2019-10-10 15:38:40 +09:00
Akira Ajisaka
eb4bd54938
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. (#1634) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e40e2d6ad5cbe782c3a067229270738b501ed27e)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeManagerImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/lifecycle/VolumeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterServiceInterceptor.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
 2019-10-10 15:27:54 +09:00
Akira Ajisaka
ae8ca70fc0
HADOOP-16609. Add Jenkinsfile for all active branches (#1557) 
* Backport HADOOP-16035. Jenkinsfile for Hadoop
* Use Yetus 0.10.0
 2019-10-10 10:47:29 +09:00
Surendra Singh Lilhore
2ca05ca02a HDFS-14754. Erasure Coding : The number of Under-Replicated Blocks never reduced(addendum). Contributed by Surendra Singh Lilhore. 
(cherry picked from commit d76e2655ace56490a92da70bde9e651ce515f80c)
 2019-10-09 23:32:21 +05:30
Szilard Nemeth
da35a22083 Revert "YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl" 
This reverts commit 42177e8b786efd7fa1365b68617382de022a5146.
 2019-10-09 19:58:46 +02:00
Szilard Nemeth
05966ce204 YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-10-09 14:18:06 +02:00
Szilard Nemeth
42177e8b78 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 
(cherry picked from commit 6f1ab9516869dbfdee034f4a17bccc379a39fc31)
 2019-10-09 13:28:01 +02:00
Szilard Nemeth
0ddb48a303 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 
(cherry picked from commit 35f093f5b3f310af49a8bb476e62173862f5af51)
 2019-10-09 13:12:03 +02:00
Sunil G
9cb3ab058f YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit be901f49628a553cfe6a3942478cb528cce2b266)
 2019-10-09 15:54:09 +05:30
Chen Liang
a12ab911ee HDFS-14509. DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x. Contributed by Yuxuan Wang and Konstantin Shvachko. 
(cherry picked from commit 72ae371e7a6695f45f0d9cea5ae9aae83941d360)
 2019-10-08 12:03:04 -07:00
Hrishikesh Gadre
aec58731a8 HADOOP-16491. Upgrade jetty version to 9.3.27. Contributed by Hrishikesh Gadre. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3cc0ace2037ee039fc2e97d5737e92d3cb64cc72)
 2019-10-08 11:29:47 -07:00
Jonathan Hung
ad3c98456d YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 
(cherry picked from commit eebd313d76ed742fe82292bd8c0184970cdc5692)
 2019-10-08 11:17:05 -07:00
Szilard Nemeth
9b4aba49d1 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:41:46 +02:00
Ayush Saxena
57ab854492 HDFS-14859. Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero. Contributed by Srinivasu Majeti. 2019-10-08 14:22:16 +05:30
Surendra Singh Lilhore
a0cdebc409 HDFS-14373. EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 382967be51052d59e31d8d05713645b8d3c2325b)
 2019-10-08 00:26:27 +05:30
Sunil G
5704f15589 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 3a0afcfb7f196bee8e534a3e80211bfa077b2aee.
 2019-10-05 09:16:04 +05:30
He Xiaoqiao
e29ae7db12 HDFS-14497. Addendum: Write lock held by metasave impact following RPC processing. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit dde9399b37bffb77da17c025f0b9b673d7088bc6)
 2019-10-04 12:07:30 -07:00
He Xiaoqiao
80392e94b6 HDFS-14497. Write lock held by metasave impact following RPC processing. Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 33c62f8f4e94442825fe286c2b18518925d980e6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2019-10-04 12:06:55 -07:00
Erik Krogen
6630c9b75d HDFS-14245. [SBN read] Enable ObserverReadProxyProvider to work with non-ClientProtocol proxy types. Contributed by Erik Krogen. 
(cherry picked from 5847e0014343f60f853cb796781ca1fa03a72efd)
 2019-10-04 11:15:49 -07:00
Erik Krogen
69b0c513a9 HDFS-14162. [SBN read] Allow Balancer to work with Observer node. Add a new ProxyCombiner allowing for multiple related protocols to be combined. Allow AlignmentContext to be passed in NameNodeProxyFactory. Contributed by Erik Krogen. 
(cherry picked from 64f28f9efa2ef3cd9dd54a6c5009029721e030ed)
 2019-10-04 10:48:33 -07:00
Eric Yang
8bb2b00d38 HDFS-14890. Fixed namenode and journalnode startup on Windows. 
Contributed by Siddharth Wagle

(cherry picked from commit aa24add8f0e9812d1f787efb3c40155b0fdeed9c)
 2019-10-04 13:16:08 -04:00
Siyao Meng
673c9d53ca HDFS-14686. HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null (#1192) Contributed by Siyao Meng. 
(cherry picked from commit 17e8cf501b384af93726e4f2e6f5e28c6e3a8f65)
 2019-10-04 09:40:22 -07:00
Sunil G
3a0afcfb7f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit 4510970e2f7728d036c750b596985e5ffa357b60)
 2019-10-04 21:49:49 +05:30
Ayush Saxena
702572434c HDFS-13693. Remove unnecessary search in INodeDirectory.addChild during image loading. Contributed by Lisheng Sun. 
(cherry picked from commit 377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46)
 2019-10-04 08:52:59 -07:00
Takanobu Asanuma
732a68cfb4 HDFS-14879. Header was wrong in Snapshot web UI. Contributed by hemanthboyina 
(cherry picked from commit b23bdaf085dbc561c785cef1613bacaf6735d909)
 2019-10-04 16:48:50 +09:00
Stephen O'Donnell
966193153f HDFS-14637. Namenode may not replicate blocks to meet the policy after enabling upgradeDomain. Contributed by Stephen O'Donnell. 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c99a12167ff9566012ef32104a3964887d62c899)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java
	hadoop-tools/hadoop-dynamometer/hadoop-dynamometer-infra/src/main/java/org/apache/hadoop/tools/dynamometer/BlockPlacementPolicyAlwaysSatisfied.java
 2019-10-03 22:18:29 -07:00
Tsz Wo Nicholas Sze
8acd9ab695 HADOOP-16127. In ipc.Client, put a new connection could happen after stop. 
(cherry picked from commit 9192f71e21847ad86bc9ff23847d8957dfe8ae58)
 2019-10-03 18:28:21 -07:00
Inigo Goiri
94af96325c HDFS-14850. Optimize FileSystemAccessService#getFileSystemConfiguration. Contributed by Lisheng Sun. 
(cherry picked from commit d8313b227495d748abe8884eee34db507476cee1)
 2019-10-03 18:21:46 -07:00
Wei-Chiu Chuang
6facb3f7c6 HADOOP-12282. Connection thread's name should be updated after address changing is detected. Contributed by Lisheng Sun. 
(cherry picked from commit b94eba9f11af66b10638dd255c224e946d842b8c)
 2019-10-03 17:30:08 -07:00
Inigo Goiri
21a89d544f HDFS-14624. When decommissioning a node, log remaining blocks to replicate periodically. Contributed by Stephen O'Donnell. 
(cherry picked from commit 5747f6cff54f79de0e6439d6c77c2ed437989f10)
 2019-10-03 17:27:44 -07:00
Shashikant Banerjee
f14fb9081f HDFS-14499. Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory. Contributed by Shashikant Banerjee. 
(cherry picked from commit f9fab9f22a53757f8081e8224e0d4b557fe6a0e2)
 2019-10-03 17:27:06 -07:00
Vinayakumar B
4e223d9968 HDFS-14113. EC : Add Configuration to restrict UserDefined Policies. Contributed by Ayush Saxena. 
(cherry picked from commit c03024a5305bea1a40c87a4abc0793802bea5c75)
 2019-10-03 17:15:45 -07:00
Kitti Nanasi
cb701f4da0 HDFS-14187. Make warning message more clear when there are not enough data nodes for EC write. Contributed by Kitti Nanasi. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 0ab7fc92009fec2f0ab341f3d878e1b8864b8ea9)
 2019-10-03 16:18:23 -07:00