Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,979 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4118 Commits

Author SHA1 Message Date
Wangda Tan a465026207 YARN-8062. yarn rmadmin -getGroups returns group from which the user has been removed. (Sunil G via wangda) 
Change-Id: I80ed63846502bf7751b890b6c6c6a7c0679e2b4a
(cherry picked from commit 5d381570f8)
 2018-03-26 11:33:24 -07:00
Rohith Sharma K S dc02823d2c YARN-7986. ATSv2 REST API queries do not return results for uppercase application tags. Contributed by Charan Hebri. 
(cherry picked from commit 75fc05f369)
 2018-03-23 12:24:38 +05:30
Sunil G 2f6d6cc0d6 YARN-8063. DistributedShellTimelinePlugin wrongly check for entityId instead of entityType. Contributed by Rohith Sharma K S. 
(cherry picked from commit 22c5ddb7c4)
 2018-03-23 10:55:46 +05:30
Jason Lowe 8f60b50f28 YARN-8054. Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread. Contributed by Jonathan Eagles 
(cherry picked from commit 5aa7052e31)
 2018-03-21 17:23:01 -05:00
Weiwei Yang 7c286fe804 YARN-7636. Re-reservation count may overflow when cluster resource exhausted for a long time. contributed by Tao Yang. 2018-03-16 19:11:54 +08:00
Rohith Sharma K S 117984a2cb YARN-8022. ResourceManager UI cluster/app/<app-id> page fails to render. Contributed by Tarun Parimi. 
(cherry picked from commit e6de10d0a6)
 2018-03-14 22:14:43 +05:30
Wangda Tan 8034faa888 Updated timeline reader to use AuthenticationFilter 
Change-Id: I961771589180c1eb377d36c37a79aa23754effbf
(cherry picked from commit 837338788eb903d0e8bbb1230694782a707891be)
 2018-03-10 13:36:09 -08:00
Owen O'Malley 0f106df4d6 Revert "HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu." 
This reverts commit 172b23af33.

(cherry picked from commit d0d2d4c51e9534e08893ae14cf3fff7b2ee70b1d)
 2018-03-10 13:33:52 -08:00
Sen Zhao 4df200bc6d
YARN-7736. Fix itemization in YARN federation document 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 245751ffdc)
 2018-03-05 11:55:01 -08:00
Jason Lowe 7c224656ce YARN-7511. NPE in ContainerLocalizer when localization failed for running container. Contributed by Tao Yang 
(cherry picked from commit 83798f15f8)
 2018-03-02 10:03:51 -06:00
Sunil G 09bb378c10 YARN-7947. Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps. Contributed by Eric Payne. 
(cherry picked from commit bdd2a184d7)
 2018-02-21 15:06:28 +05:30
Jason Lowe 85c611ad7d YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:38:49 -06:00
Rohith Sharma K S 313c38b502 YARN-7937. Fix http method name in Cluster Application Timeout Update API example request. Contributed by Charan Hebri. 
(cherry picked from commit 87bdde6943)
 2018-02-18 14:06:29 +05:30
Eric Payne 55ded70716 Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit 36c9dda07f.
 2018-02-14 14:38:31 -06:00
Eric Payne 36c9dda07f YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-14 08:58:28 -06:00
Anu Engineer 2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb3)
 2018-02-07 11:16:53 -08:00
Billie Rinaldi 0ec45f73a1 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit ef212b8550.
 2018-02-07 07:51:56 -08:00
Billie Rinaldi 615bd15072 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 
(cherry picked from commit a196ee9362)
 2018-02-06 11:34:49 -08:00
Lei Xu 7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Jason Lowe 07f779483c YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error. Contributed by Botong Huang 
(cherry picked from commit 60656bceca)
 2018-02-05 17:17:15 -06:00
Jason Lowe 84297f69d8 Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 874bdbc401.
 2018-02-01 12:17:23 -06:00
Jason Lowe 874bdbc401 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 
(cherry picked from commit 12eaae383a)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
 2018-01-31 11:45:57 -06:00
Sunil G 0463ede09d YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 
(cherry picked from commit e9c72d04be)
 2018-01-29 20:46:32 +05:30
Jinjiang Ling 1d7e9b9ad8
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3400d0c535)
 2018-01-30 00:12:40 +09:00
Jian He 3316779364 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-27 22:20:49 -08:00
Jason Lowe 217f408aaa YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit ff8378eb1b)
 2018-01-25 17:51:51 -06:00
Sunil G 88bacad50f YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 
(cherry picked from commit 82cc6f6968)
 2018-01-25 14:33:42 +05:30
Zhe Zhang 83a4f92a3f YARN-7737. prelaunch.err file not found exception on container failure. Contributed by Keqiu Hu. 
(cherry picked from commit fa8cf4d1b4)
 2018-01-24 12:57:27 -08:00
Jian He 7598f73320 YARN-7801. AmFilterInitializer should addFilter after fill all parameters. Contributed by Wangda Tan 2018-01-24 11:20:23 -08:00
Sunil G ecfdd01be7 YARN-7806. Distributed Shell should use timeline async api's. Contributed by Rohith Sharma K S 
(cherry picked from commit 97607617ce)
 2018-01-24 16:56:51 +05:30
Miklos Szegedi bf795560fb YARN-7796. Container-executor fails with segfault on certain OS configurations. Contributed by Gergo Repas. 
(cherry picked from commit e7642a3e6f)
 2018-01-23 22:21:12 -08:00
Jian He 92f789a131 YARN-7782. Enable user re-mapping for Docker containers in yarn-default.xml. Contributed by Eric Yang 2018-01-23 14:06:48 -08:00
Haibo Chen 8793e45f53 YARN-5094. some YARN container events have timestamp of -1. 
(cherry picked from commit 4aca4ff759)
 2018-01-19 10:29:33 -08:00
Miklos Szegedi 7629353153 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 
(cherry picked from commit 41049ba5d1)
 2018-01-17 13:47:44 -08:00
Robert Kanter e54c65a326 YARN-7479. TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk (ajisakaa via rkanter) 
(cherry picked from commit 5ac109909a)
 2018-01-16 10:21:12 -08:00
Rohith Sharma K S 2a01652809 YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 
(cherry picked from commit d09058b2fd)
 2018-01-16 08:42:07 +05:30
Miklos Szegedi 75c0b19bea YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 
(cherry picked from commit 2dcfc18)
 2018-01-12 17:55:58 -08:00
Sunil G f270818ed3 YARN-7727. Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector. Contributed by Prabhu Joseph. 
(cherry picked from commit 128d773a23)
 2018-01-12 15:42:06 +05:30
Miklos Szegedi ed9c516bc8 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 
(cherry picked from commit bc285da)
 2018-01-11 10:28:08 -08:00
Akira Ajisaka a13a773edb
YARN-7735. Fix typo in YARN documentation. Contributed by Takanobu Asanuma. 
(cherry picked from commit fbbbf59c82)
 2018-01-11 15:59:55 +09:00
Haibo Chen 7b32d6811c YARN-7665. Allow FS scheduler state dump to be turned on/off separately from FS debug log. (Wilfred Spiegelenburg via Haibo Chen) 
(cherry picked from commit 8ee7080e5d)
 2018-01-09 16:53:10 -08:00
Haibo Chen cf816bc751 YARN-7716. metricsTimeStart and metricsTimeEnd should be all lower case in the doc. 
(cherry picked from commit f725b9e267)
 2018-01-09 08:59:04 -08:00
Jason Lowe effce9b117 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 
(cherry picked from commit 90c86d6e09)
 2018-01-09 10:52:21 -06:00
Jason Lowe d2c2d8c913 YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 
(cherry picked from commit ebff4decfe)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
 2018-01-09 09:57:45 -06:00
bibinchundatt c57e71b542 YARN-7508. NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode. Contributed by Tao Yang. 
(cherry picked from commit b26e30ab16)
 2018-01-09 20:01:42 +05:30
Rohith Sharma K S 281350674c YARN-7699. queueUsagePercentage is coming as INF for getApp REST api call. Contributed by Sunil G. 2018-01-08 15:52:03 +05:30
Sunil G a79abbc03e YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:50:17 +05:30
Jason Lowe c155d8e756 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 
(cherry picked from commit d795661868)
 2018-01-04 10:18:32 -06:00
Haibo Chen 889bebf8bb YARN-7602. NM should reference the singleton JvmMetrics instance. 
(cherry picked from commit 2f6c038be6)
 2018-01-03 09:44:33 -08:00
Rohith Sharma K S 2c67985bd2 YARN-7692. Skip validating priority acls while recovering applications. Contributed by Sunil G. 
(cherry picked from commit c9bf813c9a)
 2018-01-03 18:21:47 +05:30
Arun Suresh 829917e9ba YARN-7691. Add Unit Tests for ContainersLauncher. (Sampada Dehankar via asuresh) 
(cherry picked from commit c0c7cce81d)
 2018-01-02 22:03:56 -08:00
Arun Suresh bd4dcc7772 YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 
(cherry picked from commit a55884c68e)
 2017-12-28 22:21:52 -08:00
Rohith Sharma K S 7d42433f37 YARN-7674. Update Timeline Reader web app address in UI2. Contributed by Sunil G. 
(cherry picked from commit 13ad7479b0)
 2017-12-20 22:29:23 +05:30
Sunil G 2026817a75 YARN-7032. [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Contributed by Rohith Sharma K S. 
(cherry picked from commit d62932c3b2)
 2017-12-20 11:33:12 +05:30
Eric Yang 62c3801af0 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 19:56:40 -05:00
Varun Saxena f92ea68367 YARN-7662. [ATSv2] Define new set of configurations for reader and collectors to bind (Rohith Sharma K S via Varun Saxena) 
(cherry picked from commit c0aeb666a4)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2017-12-19 22:41:52 +05:30
Junping Du 53b9d81f46 Add 2.8.3 release jdiff files. 
(cherry picked from commit c89f99aade)
 2017-12-18 22:40:04 -08:00
Jason Lowe f3c10beb16 YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 
(cherry picked from commit 811fabdebe)
 2017-12-18 15:23:58 -06:00
Akira Ajisaka 01dd11d977
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:52:39 +09:00
Weiwei Yang de2156dc90 YARN-7617. Add a flag in distributed shell to automatically PROMOTE opportunistic containers to guaranteed once they are started. Contributed by Weiwei Yang. 
(cherry picked from commit 9289641020)
 2017-12-18 10:27:17 +08:00
Wangda Tan 90636cf127 YARN-7629. TestContainerLaunch# fails after YARN-7381. (Jason Lowe via wangda) 
Change-Id: Ia6a3f05c9a7e797d8190123d304ecc4e2b018e33
(cherry picked from commit 44825f0960)
 2017-12-15 15:45:35 -08:00
Wangda Tan c3e6ab8c89 YARN-5418. When partial log aggregation is enabled, display the list of aggregated files on the container log page. (Xuan Gong via wangda) 
Change-Id: I1befb0bbaeb89fb315bafe3e2f3379663f8cf1ec
(cherry picked from commit 631b5c2db7)
 2017-12-15 15:45:28 -08:00
Rohith Sharma K S 53033c69f3 YARN-7190. Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath. Contributed by Varun Saxena. 
(cherry picked from commit 09d996fdd4)
 2017-12-15 22:17:43 +05:30
Weiwei Yang 1011bfd220 YARN-7642. Add test case to verify context update after container promotion or demotion with or without auto update. Contributed by Weiwei Yang. 
(cherry picked from commit 89b6c482c1)
 2017-12-15 12:55:03 +08:00
Sunil G 9ee1649cec YARN-7119. Support multiple resource types in rmadmin updateNodeResource command. Contributed by Manikandan R. 
(cherry picked from commit 2564b4d07f)
 2017-12-14 16:12:43 +05:30
Sunil G 6d814ae068 YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák. 
(cherry picked from commit 10fc8d2a7d)
 2017-12-13 22:01:28 +05:30
Weiwei Yang 415c69b36f YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:18:54 +08:00
Jason Lowe 472c7edc62 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 
(cherry picked from commit 2abab1d7c5)
 2017-12-12 16:07:47 -06:00
Weiwei Yang 3dc1ea6050 YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 10:33:46 +08:00
Wangda Tan 41796ea60a YARN-7591. NPE in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I67e086d6d35c3166ec0f6305490ea6527bd6d799
 2017-12-08 21:43:28 -08:00
Weiwei Yang 27c7a1f221 YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-07 17:36:31 +08:00
Robert Kanter cdf305c337 YARN-6483. addendum to skip failing test (asuresh via rkanter) 2017-12-06 12:10:59 -08:00
Weiwei Yang 8bfd30b921 YARN-7610. Extend Distributed Shell to support launching job with opportunistic containers. Contributed by Weiwei Yang. 
(cherry picked from commit 40b0045ebe)
 2017-12-06 19:10:34 +08:00
Weiwei Yang 48af34571f YARN-7611. Node manager web UI should display container type in containers page. Contributed by Weiwei Yang. 
(cherry picked from commit 05c347fe51)
 2017-12-06 12:24:00 +08:00
Wangda Tan 5cb1b97905 YARN-7381. Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled by default in yarn-default.xml. (Xuan Gong via wangda) 
Change-Id: I1ed58dafad5cc276eea5c0b0813cf04f57d73a87
(cherry picked from commit 0311cf0535)
 2017-12-05 13:10:15 -08:00
Sunil G f034668b91 YARN-6907. Node information page in the old web UI should report resource types. Contributed by Gergely Novák. 
(cherry picked from commit 81f6e46b2f)
 2017-12-04 12:19:27 +05:30
Arun Suresh 6f123aae41 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 
(cherry picked from commit 37ca416950)
 2017-12-03 22:23:46 -08:00
Jason Lowe 5db3ca39dc YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 
(cherry picked from commit 60f95fb719)
 2017-12-01 15:48:33 -06:00
Jason Lowe f222b9d362 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df505421)
 2017-12-01 15:38:57 -06:00
Junping Du 58d849b858 YARN-7558. yarn logs command fails to get logs for running containers if UI authentication is enabled. Contributed by Xuan Gong. 
(cherry picked from commit a409425986)
 2017-11-30 13:49:00 -08:00
Jason Lowe f072caa0bc YARN-7495. Improve robustness of the AggregatedLogDeletionService. Contributed by Jonathan Eagles 
(cherry picked from commit 5cfaee2e6d)
 2017-11-30 12:40:45 -06:00
Daniel Templeton 40372be7bb YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd)
 2017-11-30 09:26:25 -08:00
Jason Lowe a4f1e3036d YARN-6647. RM can crash during transitionToStandby due to InterruptedException. Contributed by Bibin A Chundatt 
(cherry picked from commit a2c7a73e33)
 2017-11-28 11:19:43 -06:00
Yufei Gu c0ab3c8407 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 
(cherry picked from commit d8923cdbf1)
 2017-11-27 14:33:13 -08:00
Yufei Gu f335d509d3 YARN-7290. Method canContainerBePreempted can return true when it shouldn't. (Contributed by Steven Rand) 
(cherry picked from commit 2bde3aedf1)
 2017-11-24 23:39:03 -08:00
Wangda Tan 6e038165a6 YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
(cherry picked from commit 834e91ee91)
 2017-11-23 20:02:34 -08:00
Arun Suresh 5884b8a80a YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 
(cherry picked from commit b46ca7e73b)
 2017-11-22 19:19:52 -08:00
Yufei Gu b85eb9bb53 YARN-7524. Remove unused FairSchedulerEventLog. (Contributed by Wilfred Spiegelenburg) 
(cherry picked from commit 4cc9479dae)
 2017-11-22 14:19:36 -08:00
yufei 717634e45c YARN-7513. Remove the scheduler lock in FSAppAttempt.getWeight() (Contributed by Wilfred Spiegelenburg) 
(cherry picked from commit 03c311eae3)
 2017-11-21 10:35:06 -08:00
Haibo Chen b419e8aa25 YARN-7531. ResourceRequest.equal does not check ExecutionTypeRequest.enforceExecutionType(). 2017-11-21 09:12:27 -08:00
Wangda Tan adc5157a3c YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I51ae6c2ab7a3d1febdd7d8d0519b63a13295ac7d
(cherry picked from commit 0d781dd03b)
 2017-11-20 13:14:17 -08:00
bibinchundatt fbcadfe3c4 YARN-7489. ConcurrentModificationException in RMAppImpl#getRMAppMetrics. Contributed by Tao Yang. 
(cherry picked from commit b5b81a4f08)
 2017-11-18 20:01:01 +05:30
bibinchundatt ae8d1b3c34 YARN-7525. Incorrect query parameters in cluster nodes REST API document. Contributed by Tao Yang. 
(cherry picked from commit 96e6a99360)
 2017-11-18 19:11:40 +05:30
Varun Vasudev 6681e7238b YARN-7430. Enable user re-mapping for Docker containers by default. Contributed by Eric Yang. 
(cherry picked from commit 5f0b238a11)
 2017-11-17 12:05:19 +05:30
Haibo Chen 9bbe7eb226 YARN-7390. All reservation related test cases failed when TestYarnClient runs against Fair Scheduler. (Yufei Gu via Haibo Chen) 
(cherry picked from commit 28d0fcbef4)
 2017-11-16 10:49:40 -08:00
Sunil G c3fb49667a YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 
(cherry picked from commit 61ace174cd)
 2017-11-16 22:35:22 +05:30
Jason Lowe 9780d58a63 YARN-7361. Improve the docker container runtime documentation. Contributed by Shane Kumpf 
(cherry picked from commit fac72eef23)
 2017-11-15 12:32:56 -06:00
Daniel Templeton 570bc80901 YARN-7414. FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight() 
(Contributed by Soumabrata Chakraborty via Daniel Templeton)

(cherry picked from commit b246c54749)
 2017-11-15 10:06:17 -08:00
Daniel Templeton 5536fa1e37 YARN-6953. Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources() 
(Contributed by Manikandan R via Daniel Templeton)

(cherry picked from commit e094eb74b9)
 2017-11-15 10:06:11 -08:00
Andrew Wang 1c8d334287 Preparing for 3.0.1 development 2017-11-14 11:17:18 -08:00
Junping Du ef212b8550 YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 
(cherry picked from commit e14f03dfbf)
 2017-11-13 15:28:21 -08:00
Wangda Tan 7b3cd10131 YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2017-11-13 14:45:40 -08:00
Daniel Templeton c2f479c248 YARN-7369. Improve the resource types docs 2017-11-13 11:08:58 -08:00
Daniel Templeton 5b55a74bad YARN-7442. [YARN-7069] Limit format of resource type name (Contributed by Wangda Tan via Daniel Templeton) 
(cherry picked from commit 2c6213a44280f5b3950167131293ff83f48ff56f)
 2017-11-13 11:03:56 -08:00
Subru Krishnan 37ae818660 YARN-7475. Fix Container log link in new YARN UI. (Sunil G via Subru). 
(cherry picked from commit 3c5b46c2ed)
(cherry picked from commit 3e26077848)
 2017-11-12 09:54:55 -08:00
bibinchundatt bfbf94197f YARN-7406. Moving logging APIs over to slf4j in hadoop-yarn-api. Contributed by Yeliang Cang. 
(cherry picked from commit 2c2b7a3672)
 2017-11-11 10:38:47 +05:30
Sean Mackrory abaf90792e YARN-7465. start-yarn.sh fails to start ResourceManagers unless running as root. 
(cherry picked from commit 1883a00249)
 2017-11-09 14:49:54 -07:00
Konstantinos Karanasos e55dc1700d YARN-7437. Rename PlacementSet and SchedulingPlacementSet. (Wangda Tan via kkaranasos) 
(cherry picked from commit ac4d2b1081)
 2017-11-09 13:07:10 -08:00
Robert Kanter 82abc7224f YARN-7386. Duplicate Strings in various places in Yarn memory (misha@cloudera.com via rkanter) 
(cherry picked from commit a2c150a736)
 2017-11-09 12:13:09 -08:00
Daniel Templeton 933a09e886 YARN-7143. FileNotFound handling in ResourceUtils is inconsistent 
Change-Id: Ib1bb487e14a15edd2b5a42cf5078c5a2b295f069
(cherry picked from commit db82a41d94872cea4d0c1bb1336916cebc2faeec)
 2017-11-09 11:59:16 -08:00
Wangda Tan 51d3693a9c YARN-7205. Log improvements for the ResourceUtils. (Sunil G via wangda) 
Change-Id: I0f5b7a7f68ec5d3e1d52211f83fdd089bc0bfd37
(cherry picked from commit 8bcc49e677)
 2017-11-09 11:59:16 -08:00
Haibo Chen 91a7f8d246 YARN-7388. TestAMRestart should be scheduler agnostic. 
(cherry picked from commit a1382a18df)
 2017-11-09 10:50:49 -08:00
bibinchundatt 4370b70e63 YARN-7454. RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup. Contributed by Jason Lowe. 
(cherry picked from commit 0a72c2f56c)
 2017-11-09 21:08:47 +05:30
Daniel Templeton 9bcc9e3f4d YARN-7458. TestContainerManagerSecurity is still flakey 
(Contributed by Robert Kanter via Daniel Templeton)

Change-Id: Ibb1975ad086c3a33f8af0b4f8b9a13c3cdca3f7d
(cherry picked from commit 49b4c0b334)
 2017-11-08 17:32:34 -08:00
Daniel Templeton d2b16e3e83 YARN-7166. Container REST endpoints should report resource types 
Change-Id: If9c2fe58d4cf758bb6b6cf363dc01f35f8720987
(cherry picked from commit 0de10680b7)
 2017-11-08 16:51:20 -08:00
Arun Suresh 91e726c07f YARN-7453. Fix issue where RM fails to switch to active after first successful start. (Rohith Sharma K S via asuresh) 
(cherry picked from commit a9c70b0e84)
 2017-11-08 08:01:54 -08:00
Daniel Templeton 4e7b5824b4 YARN-7401. Reduce lock contention in ClusterNodeTracker#getClusterCapacity() 
(cherry picked from commit 8db9d61ac2)
 2017-11-07 14:58:56 -08:00
Haibo Chen 91e95dc484 YARN-7360. TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic. 
(cherry picked from commit 8f214dc4f8)
 2017-11-06 15:54:14 -08:00
Wangda Tan df27824b80 YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. (wangda) 2017-11-06 15:26:49 -08:00
Wangda Tan 170b6a48c4 YARN-7307. Allow client/AM update supported resource types via YARN APIs. (Sunil G via wangda) 2017-11-06 10:01:10 -08:00
bibinchundatt e49509f024 Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 
(cherry picked from commit dcd99c4b9a)
 2017-11-06 23:07:16 +05:30
Inigo Goiri 3e70fd9af0 YARN-7434. Router getApps REST invocation fails with multiple RMs. Contributed by Inigo Goiri. 
(cherry picked from commit 6e2259264ad9525eeec2a14055d53236711659d7)
 2017-11-02 21:30:20 -07:00
Subru Krishnan 1b6de3fac5 YARN-7432. Fix DominantResourceFairnessPolicy serializable findbugs issues. (Daniel Templeton via Subru). 
(cherry picked from commit ad0fff2b41)
 2017-11-02 17:40:03 -07:00
Eric Payne 3d923d66e8 YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 
(cherry picked from commit e6ec02001f)
 2017-11-02 12:48:07 -05:00
Wangda Tan fb442e9a7c YARN-7364. Queue dash board in new YARN UI has incorrect values. (Sunil G via wangda) 
Change-Id: Ia5336f407147d2985a61b5f9f1e69b5b35f398a4
(cherry picked from commit 752f000e0f)
 2017-11-02 09:50:57 -07:00
Jason Lowe b7dee1f060 YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 
(cherry picked from commit d00b6f7c1f)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java
 2017-11-02 09:47:26 -05:00
Rohith Sharma K S 9efe9bd3d8 addendum patch for YARN-7289. 
(cherry picked from commit 940ffe3f9c)
 2017-11-02 14:02:48 +05:30
Eric Yang be7cfe57fc YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 2017-11-01 18:45:24 -04:00
Junping Du 552f74e6b7 YARN-7400. Incorrect log preview displayed in jobhistory server ui. Contributed by Xuan Gong. 
(cherry picked from commit d57dba9942)
 2017-11-01 15:02:50 -07:00
Inigo Goiri 08c3efa4a0 YARN-7276. Federation Router Web Service fixes. Contributed by Inigo Goiri. 
(cherry picked from commit 8be5707067)
 2017-11-01 13:42:12 -07:00
Inigo Goiri 8d41875219 YARN-7276 addendum to add timeline service depencies. Contributed by Inigo Goiri. 
(cherry picked from commit 70f1a9470c)
 2017-11-01 13:28:50 -07:00
Daniel Templeton c025c9ac62 YARN-7336. Unsafe cast from long to int Resource.hashCode() method 
(Contributed by Miklos Szegedi via Daniel Templeton)

(cherry picked from commit d64736d589)
 2017-10-31 15:06:18 -07:00
Daniel Templeton 7689d4781a YARN-6927. Add support for individual resource types requests in MapReduce 
(Contributed by Gergo Repas via Daniel Templeton)

(cherry picked from commit 9a7e810838)
 2017-10-31 15:06:18 -07:00
Daniel Templeton 77401022a7 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 
(cherry picked from commit 9711b78998)
 2017-10-31 15:06:18 -07:00
Daniel Templeton ab93bf5b00 YARN-7397. Reduce lock contention in FairScheduler#getAppWeight() 
(cherry picked from commit e62bbbca7a)
 2017-10-31 15:06:18 -07:00
Daniel Templeton c818c85238 YARN-6984. DominantResourceCalculator.isAnyMajorResourceZero() should test all resources 
(Contributed by Sunil G via Daniel Templeton)

(cherry picked from commit 679f99b145)
 2017-10-31 15:06:18 -07:00
Daniel Templeton 68585156dc YARN-7270 addendum: Reapplied changes after YARN-3926 backports 2017-10-31 15:06:18 -07:00
Wangda Tan 6f41843906 YARN-7172. ResourceCalculator.fitsIn() should not take a cluster resource parameter. (Sen Zhao via wangda) 
Change-Id: Icc3670c9381ce7591ca69ec12da5aa52d3612d34
(cherry picked from commit e81596d06d)
 2017-10-31 15:06:18 -07:00
Yufei Gu f30f78a736 YARN-6612. Update fair scheduler policies to be aware of resource types. (Contributed by Daniel Templeton via Yufei Gu) 
(cherry picked from commit 09b476e6da)
 2017-10-31 15:06:18 -07:00
Sunil G 74030d808c YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan. 
(cherry picked from commit da0b6a354b)
 2017-10-31 15:06:17 -07:00
Daniel Templeton 0db9ddb728 YARN-7136. Additional Performance Improvement for Resource Profile Feature 
(Contributed by Wangda Tan via Daniel Templeton)

(cherry picked from commit bf2b687412)
 2017-10-31 15:06:17 -07:00
Wangda Tan 335e3a6334 YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda) 
Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e
(cherry picked from commit 6b88cd1901)
 2017-10-31 15:06:17 -07:00
Sunil G 4bf29910a8 YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R. 
(cherry picked from commit 805095496d)
 2017-10-31 15:06:17 -07:00
Wangda Tan 00fcc8593b YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda) 
Change-Id: I88928a747ee3eec17bc76fb71e9aaa632d091f0f
(cherry picked from commit 53df3eac50)
 2017-10-31 15:06:17 -07:00
Wangda Tan 57bbdbeb80 YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
(cherry picked from commit e490602e9b)
 2017-10-31 15:06:17 -07:00
Sunil G 5dd008ca2e YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan. 
(cherry picked from commit a333ba54e3)
 2017-10-31 15:06:17 -07:00
Wangda Tan 605fcde012 YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
(cherry picked from commit 39240b61a1)
 2017-10-31 15:06:17 -07:00
Sunil G f2b881a505 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin. 
(cherry picked from commit 758b771943)
 2017-10-31 15:06:17 -07:00
Wangda Tan 4af3deaeb0 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
(cherry picked from commit df3855541a)
 2017-10-31 15:06:17 -07:00
Wangda Tan 255668a2a6 YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda) 
Change-Id: I8e40f704b6fcdd5b14faa9548a27986501044fa1
(cherry picked from commit b1fe3a222e)
 2017-10-31 15:06:17 -07:00
Sunil G 8827d93ca1 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan. 
(cherry picked from commit a6989af1d9)
 2017-10-31 15:06:17 -07:00
Sunil G 1ba85a9305 YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton. 
(cherry picked from commit a0030c8c6f)
 2017-10-31 15:06:17 -07:00
Sunil G 5231c43bdb YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G. 
(cherry picked from commit 2b51b262ab)
 2017-10-31 15:06:16 -07:00
Yufei Gu 9c4505f3fa YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit d5e9939ebb)
 2017-10-31 15:06:16 -07:00
Daniel Templeton 99c292403d YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)

(cherry picked from commit 3aeaafecb8)
 2017-10-31 15:06:16 -07:00
Sunil G 58f2c07028 YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang. 
(cherry picked from commit 4e5632d28e)
 2017-10-31 15:06:16 -07:00
Sunil G 5fa0dc35c3 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev. 
(cherry picked from commit ca2808d8bb)
 2017-10-31 15:06:16 -07:00
Sunil G 48ddb8f5b4 YARN-6445. [YARN-3926] Performance improvements in resource profile branch with respect to SLS. Contributed by Varun Vasudev. 
(cherry picked from commit 58da54640e)
 2017-10-31 15:06:16 -07:00
Sunil G bda5486dad YARN-6232. Update resource usage and preempted resource calculations to take into account all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit dae65f3bef)
 2017-10-31 15:06:16 -07:00
Sunil G 4045ef047d YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 
(cherry picked from commit 7805deed48)
 2017-10-31 15:06:16 -07:00
Arun Suresh a9fa0349a0 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh) 
(cherry picked from commit 6708ac3301)
 2017-10-31 15:06:16 -07:00
Rohith Sharma K S 3d1b59e04a YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit 239c1824a0)
 2017-10-31 15:06:16 -07:00
Rohith Sharma K S feb6d00e2a YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 
(cherry picked from commit 9e4ba6aff5)
 2017-10-31 15:06:16 -07:00
Varun Vasudev 16beac12ad YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 
(cherry picked from commit 759114b006)
 2017-10-31 15:06:16 -07:00
Arun Suresh 92ca475fdc YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 
(cherry picked from commit 7ba698997b)
 2017-10-31 15:06:16 -07:00
Varun Vasudev f1f9b83e4c YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 
(cherry picked from commit ceb12c59a3)
 2017-10-31 15:06:15 -07:00
Wangda Tan 33fabe1e10 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 
(cherry picked from commit 32c91223f1)
 2017-10-31 15:06:15 -07:00
Wangda Tan 68da5218b7 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 
(cherry picked from commit 1bbab7c157)
 2017-10-31 15:06:15 -07:00
Carlo Curino 1fbc72704e YARN-3661. Basic Federation UI. (Contributed by Inigo Goiri via curino) 
(cherry picked from commit ceca9694f9)
 2017-10-31 13:42:36 -07:00
Haibo Chen e785e83270 YARN-7389. Make TestResourceManager Scheduler agnostic. (Robert Kanter via Haibo Chen) 
(cherry picked from commit d7f3737f3b)
 2017-10-31 13:41:44 -07:00
Subru Krishnan 0d9503d6b5 YARN-6413. FileSystem based Yarn Registry implementation. (Ellen Hui via Subru). 
(cherry picked from commit ed24da3dd7)
 2017-10-31 12:08:32 -07:00
Akira Ajisaka da6ab7727c
YARN-7379. Moving logging APIs over to slf4j in hadoop-yarn-client. Contributed by Yeliang Cang. 
(cherry picked from commit c02d2ba50d)
 2017-10-31 17:11:18 +09:00
Akira Ajisaka 6bbfa158d7
YARN-7407. Moving logging APIs over to slf4j in hadoop-yarn-applications. Contributed by Yeliang Cang. 
(cherry picked from commit 785f1b0d11)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/main/java/org/apache/hadoop/yarn/applications/distributedshell/Client.java
 2017-10-31 16:45:12 +09:00
Junping Du b9a4bf0c46 HADOOP-14990. Clean up jdiff xml files added for 2.8.2 release. 
(cherry picked from commit a8083aaa6f)
 2017-10-30 15:28:58 -07:00
Yufei Gu d9f027adbc YARN-6747. TestFSAppStarvation.testPreemptionEnable fails intermittently. (Contributed by Miklos Szegedi) 
(cherry picked from commit d4811c8cfa)
 2017-10-29 16:48:07 -07:00
Arun Suresh 6b4319cc50 YARN-7299. Fix TestDistributedScheduler. (asuresh) 
(cherry picked from commit 9c5c68745e)
 2017-10-27 23:15:37 -07:00
Arun Suresh b30eb17c75 YARN-7178. Add documentation for Container Update API. (asuresh) 
(cherry picked from commit 24f8c5cce3)
 2017-10-27 22:58:34 -07:00
Jason Lowe 00345eed50 YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 
(cherry picked from commit 665bb147aa)
 2017-10-27 17:00:31 -05:00
Rohith Sharma K S d9b636f2f4 YARN-7289. Application lifetime does not work with FairScheduler. Contributed by Miklos Szegedi. 
(cherry picked from commit 5c799ecf09)
 2017-10-27 23:14:55 +05:30
Sunil G e084d43550 YARN-7375. Possible NPE in RMWebapp when HA is enabled and the active RM fails. Contributed by Chandni Singh. 
(cherry picked from commit 792388e1c0)
 2017-10-27 20:54:58 +05:30
Junping Du ce10c1ef05 Add several jdiff xml files for 2.8.2 release. 
(cherry picked from commit a25b5aa0cf)
 2017-10-26 23:18:50 -07:00
Robert Kanter aa65f6c1ad YARN-7262. Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow (rkanter) 
(cherry picked from commit b1de78619f)
 2017-10-26 17:47:51 -07:00
Robert Kanter cd9078df68 YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ Addendum (misha@cloudera.com via rkanter) 
(cherry picked from commit 088ffee716)
 2017-10-26 15:55:23 -07:00
Subru Krishnan 9665971a65 YARN-5516. Add REST API for supporting recurring reservations. (Sean Po via Subru). 
(cherry picked from commit 25932da6d1)
 2017-10-26 12:11:08 -07:00
Robert Kanter d92dddaf7c YARN-7358. TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler (haibochen via rkanter) 
(cherry picked from commit 2da654e34a)
 2017-10-26 10:45:23 -07:00
Subru Krishnan 926b188871 YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru). 
(cherry picked from commit 3fae675383)
 2017-10-25 15:08:19 -07:00
Eric Yang 107afed848 YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-24 16:59:17 -04:00
Yufei Gu 6e953eb6a3 YARN-7045. Remove FSLeafQueue#addAppSchedulable. (Contributed by Sen Zhao via Yufei Gu) 
(cherry picked from commit a92ef030a2)
 2017-10-24 13:52:02 -07:00
Robert Kanter c5eeb2e4ce YARN-7385. TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE (yufeigu via rkanter) 
(cherry picked from commit 03af442e76)
 2017-10-24 13:40:06 -07:00
Carlo Curino 8b200ab395 YARN-7339. LocalityMulticastAMRMProxyPolicy should handle cancel request properly. (Botong Huang via curino) 
(cherry picked from commit 1c5c2b5dde)
 2017-10-24 10:53:08 -07:00
Robert Kanter 4bae02a97d YARN-7382. NoSuchElementException in FairScheduler after failover causes RM crash (rkanter) 
(cherry picked from commit 025c656572)
 2017-10-24 10:25:43 -07:00
Robert Kanter 61aba4f0c1 YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ (misha@cloudera.com via rkanter) 
(cherry picked from commit 5da295a34e)
 2017-10-23 18:02:11 -07:00
Daniel Templeton 9da9023371 YARN-7357. Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations 
(Contributed by Sen Zhao via Daniel Templeton)

(cherry picked from commit 0553f5a012e14c8ceb40a16161440384f9f3a9a8)
 2017-10-23 13:51:39 -07:00
Eric Payne 3f315a526b YARN-4163: Audit getQueueInfo and getApplications calls 
(cherry picked from commit 921338cd86)
 2017-10-23 11:48:41 -05:00
Sunil G 4b872e5e9f YARN-6871. Add additional deSelects params in RMWebServices#getAppReport. Contributed by Tanuj Nayak. 
(cherry picked from commit 8facf1f976)
 2017-10-20 17:20:39 -07:00
Yufei Gu d375645d50 YARN-7355. TestDistributedShell should be scheduler agnostic. (Contributed by Haibo) 
(cherry picked from commit 6b7c87c945)
 2017-10-20 11:17:05 -07:00
Yufei Gu 2a0ef092d3 YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu) 
(cherry picked from commit 0799fde35e)
 2017-10-20 10:01:52 -07:00
Yufei Gu 96106b8f5f YARN-4090. Make Collections.sort() more efficient by caching resource usage. (Contributed by Yufei Gu, Shilong Zhang and Xianyin Xin) 
(cherry picked from commit 1f4cdf1068)
 2017-10-20 01:34:05 -07:00
Wangda Tan dc2ae3f20b YARN-7170. Improve bower dependencies for YARN UI v2. (Sunil G via wangda) 
(cherry picked from commit 4afd308b62)
 2017-10-19 21:59:28 -07:00
Yufei Gu b2f516118c YARN-7359. TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic. (Contributed by Haibo Chen) 
(cherry picked from commit 7b4b018780)
 2017-10-19 16:53:10 -07:00