Commit Graph

14986 Commits

Author SHA1 Message Date
Sunil G 45b50c45d6 YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R.
(cherry picked from commit 805095496d)
2019-08-29 17:02:47 -07:00
Wangda Tan 8cea242e94 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda)
Change-Id: I88928a747ee3eec17bc76fb71e9aaa632d091f0f
(cherry picked from commit 53df3eac50)
2019-08-29 17:02:46 -07:00
Wangda Tan f340726c9a YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda)
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
(cherry picked from commit e490602e9b)
2019-08-29 17:02:46 -07:00
Sunil G 74bb2d6ae7 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan.
(cherry picked from commit a333ba54e3)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
2019-08-29 17:02:46 -07:00
Wangda Tan 1f05fa041a YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda)
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
(cherry picked from commit 39240b61a1)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
2019-08-29 17:02:46 -07:00
Sunil G 7c37fbef12 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin.
(cherry picked from commit 758b771943)
2019-08-29 17:02:46 -07:00
Wangda Tan c69dd9eea6 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda)
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
(cherry picked from commit df3855541a)
2019-08-29 17:02:46 -07:00
Wangda Tan 43030aa423 YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda)
Change-Id: I8e40f704b6fcdd5b14faa9548a27986501044fa1
(cherry picked from commit b1fe3a222e)
2019-08-29 17:02:46 -07:00
Sunil G 7b1c58b037 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan.
(cherry picked from commit a6989af1d9)
2019-08-29 17:02:46 -07:00
Sunil G 2fafdc7f8b YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton.
(cherry picked from commit a0030c8c6f)
2019-08-29 17:02:46 -07:00
Sunil G 06c38ee7a4 YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G.
(cherry picked from commit 2b51b262ab)
2019-08-29 17:02:46 -07:00
Yufei Gu 59cfecd60e YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu)
(cherry picked from commit d5e9939ebb)
2019-08-29 17:02:46 -07:00
Daniel Templeton d8f388652e YARN-6788. [YARN-3926] Improve performance of resource profile branch
(Contributed by Sunil Govindan via Daniel Templeton)

(cherry picked from commit 3aeaafecb8)
2019-08-29 17:02:46 -07:00
Sunil G 1dc66dbb3f YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang.
(cherry picked from commit 4e5632d28e)
2019-08-29 17:02:46 -07:00
Sunil G ad6f095774 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev.
(cherry picked from commit ca2808d8bb)
2019-08-29 17:02:46 -07:00
Jonathan Hung d5963b2be9 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2 2019-08-29 17:02:46 -07:00
Jonathan Hung 08923e324a YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2 2019-08-29 17:02:46 -07:00
Jonathan Hung 7f614f7cee YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2 2019-08-29 17:02:46 -07:00
Sunil G c5e77e8b84 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev.
(cherry picked from commit 7805deed48)
2019-08-29 17:02:46 -07:00
Arun Suresh 6db8c36ba9 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh)
(cherry picked from commit 6708ac3301)
2019-08-29 17:02:46 -07:00
Rohith Sharma K S f99d603aaf YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev.
(cherry picked from commit 239c1824a0)
2019-08-29 17:02:46 -07:00
Rohith Sharma K S 015a55a5c6 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev.
(cherry picked from commit 9e4ba6aff5)
2019-08-29 17:02:46 -07:00
Varun Vasudev 7d5050a768 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev.
(cherry picked from commit 759114b006)
2019-08-29 17:02:46 -07:00
Arun Suresh 817837e65c YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh)
(cherry picked from commit 7ba698997b)
2019-08-29 17:02:46 -07:00
Varun Vasudev 133eb31d54 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev.
(cherry picked from commit ceb12c59a3)
2019-08-29 17:02:46 -07:00
Wangda Tan be2afb4a5d YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda)
(cherry picked from commit 32c91223f1)
2019-08-29 17:02:46 -07:00
Wangda Tan afcf9c55b5 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda)
(cherry picked from commit 1bbab7c157)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
2019-08-29 17:02:46 -07:00
Sneha Vijayarajan f56972bfe1 HADOOP-16460: ABFS: fix for Sever Name Indication (SNI)
Contributed by Sneha Vijayarajan <snvijaya@microsoft.com>
2019-08-28 22:42:28 +00:00
Da Zhou 6cb74629ec HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
2019-08-28 22:42:28 +00:00
bilaharith 1376f2ed3d HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
2019-08-28 22:42:28 +00:00
Arun Singh 8044415ea1 HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
Contributed by: Arun Singh
2019-08-28 22:42:28 +00:00
Da Zhou 42816bf698 HADOOP-16376. ABFS: Override access() to no-op.
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
2019-08-28 22:42:28 +00:00
Aaron Fabbri 67fe5398fd Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
This reverts commit 7c2d7c05a9.

Git Commit name and email were incorrect. Will re-commit.
2019-08-28 22:42:28 +00:00
DadanielZ 605c812749 HADOOP-16251. ABFS: add FSMainOperationsBaseTest
Author: Da Zhou
2019-08-28 22:42:28 +00:00
Da Zhou 50953c586b HADOOP-16269. ABFS: add listFileStatus with StartFrom.
Author:    Da Zhou
2019-08-28 22:42:28 +00:00
Da Zhou e5bb8498df HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
Contributed by Da Zhou.
2019-08-28 22:42:28 +00:00
Eric E Payne 8e8c16dbc6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06).
(cherry picked from commit d562050cec)
2019-08-28 21:26:03 +00:00
Jonathan Hung b6be5f869a YARN-9438. launchTime not written to state store for running applications
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
(cherry picked from commit f73842780e)
2019-08-27 15:49:58 -07:00
Jonathan Hung f083f7fef1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph
(cherry picked from commit 8660e48ca1)
(cherry picked from commit e4249c3202)
(cherry picked from commit 6baa0d1e4d)
2019-08-26 16:05:25 -07:00
Chen Liang e786147096 HDFS-13977. NameNode can kill itself if it tries to send too many txns to a QJM simultaneously. Contributed by Erik Krogen. 2019-08-26 14:32:27 -07:00
Weiwei Yang 97f207d1a9 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang.
(Cherry picked from commit 67f239c42f)

(cherry picked from commit a0b7abf278)
2019-08-23 09:59:47 -07:00
Akira Ajisaka 786df37429
HADOOP-16494. Add SHA-512 checksum to release artifact to comply with the release distribution policy.
This closes #1243

(cherry picked from commit 34dd9ee366)
2019-08-22 11:04:05 +09:00
Wei-Chiu Chuang 69b80f1ee8 Revert "HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow."
This reverts commit 8755de108d.
2019-08-21 09:04:07 -07:00
Wei-Chiu Chuang bdab03cf71 HDFS-14311. Multi-threading conflict at layoutVersion when loading block pool storage. Contributed by Yicong Cai. 2019-08-20 10:38:52 -07:00
Shashikant Banerjee ab7509efe7 HDFS-13101. Yet another fsimage corruption related to snapshot. Contributed by Shashikant Banerjee.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-18 19:01:12 -07:00
Wei-Chiu Chuang 8755de108d HDFS-14476. lock too long when fix inconsistent blocks between disk and in-memory. Contributed by Sean Chow. 2019-08-18 17:51:44 -07:00
He Xiaoqiao b020c5ccfd HDFS-14725. Backport HDFS-12914 to branch-2 (Block report leases cause missing blocks until next report). Contributed by He Xiaoqiao.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-16 17:53:45 -07:00
Masatake Iwasaki 61bd868e80 HDFS-14707. Add JAVA_LIBRARY_PATH to HTTPFS startup options in branch-2.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-08-15 11:21:17 +09:00
Masatake Iwasaki 670de354b3 HDFS-14423. Percent (%) and plus (+) characters no longer work in WebHDFS.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit da0006fe04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java

(cherry picked from commit d7ca016d63)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsUrl.java
2019-08-14 12:35:14 +09:00
Christopher Gregorian 99cd181a78 HADOOP-16459. Backport of HADOOP-16266. Add more fine-grained processing time metrics to the RPC layer. Contributed by Christopher Gregorian.
This commit also includes the follow-on commit 827a84778a.

(cherry-picked from f96a2df38d)
(cherry-picked from d4492bdd9e)
(cherry-picked from 7b8f08f59e)
(cherry picked from ec00431eaa)
2019-08-13 14:28:18 -07:00