Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,982 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

14982 Commits

Author SHA1 Message Date
Wangda Tan 1f05fa041a YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
(cherry picked from commit 39240b61a1)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
 2019-08-29 17:02:46 -07:00
Sunil G 7c37fbef12 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin. 
(cherry picked from commit 758b771943)
 2019-08-29 17:02:46 -07:00
Wangda Tan c69dd9eea6 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
(cherry picked from commit df3855541a)
 2019-08-29 17:02:46 -07:00
Wangda Tan 43030aa423 YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda) 
Change-Id: I8e40f704b6fcdd5b14faa9548a27986501044fa1
(cherry picked from commit b1fe3a222e)
 2019-08-29 17:02:46 -07:00
Sunil G 7b1c58b037 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan. 
(cherry picked from commit a6989af1d9)
 2019-08-29 17:02:46 -07:00
Sunil G 2fafdc7f8b YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton. 
(cherry picked from commit a0030c8c6f)
 2019-08-29 17:02:46 -07:00
Sunil G 06c38ee7a4 YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G. 
(cherry picked from commit 2b51b262ab)
 2019-08-29 17:02:46 -07:00
Yufei Gu 59cfecd60e YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit d5e9939ebb)
 2019-08-29 17:02:46 -07:00
Daniel Templeton d8f388652e YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)

(cherry picked from commit 3aeaafecb8)
 2019-08-29 17:02:46 -07:00
Sunil G 1dc66dbb3f YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang. 
(cherry picked from commit 4e5632d28e)
 2019-08-29 17:02:46 -07:00
Sunil G ad6f095774 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev. 
(cherry picked from commit ca2808d8bb)
 2019-08-29 17:02:46 -07:00
Jonathan Hung d5963b2be9 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2 2019-08-29 17:02:46 -07:00
Jonathan Hung 08923e324a YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2 2019-08-29 17:02:46 -07:00
Jonathan Hung 7f614f7cee YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2 2019-08-29 17:02:46 -07:00
Sunil G c5e77e8b84 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 
(cherry picked from commit 7805deed48)
 2019-08-29 17:02:46 -07:00
Arun Suresh 6db8c36ba9 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh) 
(cherry picked from commit 6708ac3301)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S f99d603aaf YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit 239c1824a0)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S 015a55a5c6 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 
(cherry picked from commit 9e4ba6aff5)
 2019-08-29 17:02:46 -07:00
Varun Vasudev 7d5050a768 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 
(cherry picked from commit 759114b006)
 2019-08-29 17:02:46 -07:00
Arun Suresh 817837e65c YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 
(cherry picked from commit 7ba698997b)
 2019-08-29 17:02:46 -07:00
Varun Vasudev 133eb31d54 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 
(cherry picked from commit ceb12c59a3)
 2019-08-29 17:02:46 -07:00
Wangda Tan be2afb4a5d YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 
(cherry picked from commit 32c91223f1)
 2019-08-29 17:02:46 -07:00
Wangda Tan afcf9c55b5 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 
(cherry picked from commit 1bbab7c157)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
 2019-08-29 17:02:46 -07:00
Sneha Vijayarajan f56972bfe1 HADOOP-16460: ABFS: fix for Sever Name Indication (SNI) 
Contributed by Sneha Vijayarajan <snvijaya@microsoft.com>
 2019-08-28 22:42:28 +00:00
Da Zhou 6cb74629ec HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-28 22:42:28 +00:00
bilaharith 1376f2ed3d HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-28 22:42:28 +00:00
Arun Singh 8044415ea1 HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-08-28 22:42:28 +00:00
Da Zhou 42816bf698 HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-08-28 22:42:28 +00:00
Aaron Fabbri 67fe5398fd Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest" 
This reverts commit 7c2d7c05a9.

Git Commit name and email were incorrect. Will re-commit.
 2019-08-28 22:42:28 +00:00
DadanielZ 605c812749 HADOOP-16251. ABFS: add FSMainOperationsBaseTest 
Author: Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou 50953c586b HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou e5bb8498df HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.
 2019-08-28 22:42:28 +00:00
Eric E Payne 8e8c16dbc6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec)
 2019-08-28 21:26:03 +00:00
Jonathan Hung b6be5f869a YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
(cherry picked from commit f73842780e)
 2019-08-27 15:49:58 -07:00
Jonathan Hung f083f7fef1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca1)
(cherry picked from commit e4249c3202)
(cherry picked from commit 6baa0d1e4d)
 2019-08-26 16:05:25 -07:00
Chen Liang e786147096 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen. 2019-08-26 14:32:27 -07:00
Weiwei Yang 97f207d1a9 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f)

(cherry picked from commit a0b7abf278)
 2019-08-23 09:59:47 -07:00
Akira Ajisaka 786df37429
HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy. 
This closes #1243

(cherry picked from commit 34dd9ee366)
 2019-08-22 11:04:05 +09:00
Wei-Chiu Chuang 69b80f1ee8 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow." 
This reverts commit 8755de108d.
 2019-08-21 09:04:07 -07:00
Wei-Chiu Chuang bdab03cf71 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:38:52 -07:00
Shashikant Banerjee ab7509efe7 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-18 19:01:12 -07:00
Wei-Chiu Chuang 8755de108d HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:51:44 -07:00
He Xiaoqiao b020c5ccfd HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-08-16 17:53:45 -07:00
Masatake Iwasaki 61bd868e80 HDFS-14707. Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2019-08-15 11:21:17 +09:00
Masatake Iwasaki 670de354b3 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
 2019-08-14 12:35:14 +09:00
Christopher Gregorian 99cd181a78 HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian. 
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
(cherry-picked from 7b8f08f59e)
(cherry picked from ec00431eaa)
 2019-08-13 14:28:18 -07:00
Eric Badger afa9a4084d YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

(cherry picked from commit db88224e8f)
 2019-08-13 17:27:43 +00:00
Erik Krogen 719214ff30 HDFS-14370. Add exponential backoff to the edit log tailer to avoid spinning on empty edit tail requests. Contributed by Erik Krogen. 
(cherry picked from 827dbb11e2)
(cherry picked from 016aa13940)
(cherry picked from f6ce2f4a50)
(cherry picked from 5657e45fb2)
 2019-08-13 08:30:33 -07:00
Chen Liang 885b7f4c98 Fix JDK7 compatibility in branch-2. 2019-08-12 17:42:01 -07:00
Chen Liang 0d153dcb8c HDFS-13898 Addendum: add a missing test helper method 2019-08-12 10:31:37 -07:00