Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,186 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3825 Commits

Author SHA1 Message Date
Sunil G 23dc6ef8b0 YARN-7163. RMContext need not to be injected to webapp and other Always Running services. Contributed by Rohith Sharma K S. 2017-09-11 10:34:59 +05:30
Subru Krishnan 2c051dbe9e YARN-5330. SharingPolicy enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 
(cherry picked from commit fa6137501c)
 2017-09-08 14:44:46 -07:00
Chris Douglas 2f8825588e YARN-5164. Use plan RLE to improve CapacityOverTimePolicy efficiency 
(cherry picked from commit d383bfdcd4)
 2017-09-08 14:44:34 -07:00
Nathan Roberts b3a478832a YARN-6219. Fix conf usage in NMWebServer unit tests. Contributed by Jason Lowe. 2017-09-08 16:01:06 -05:00
Arun Suresh 2b09cec218 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 1f53ae7972)
 2017-09-08 09:26:50 -07:00
Sunil G a39829c456 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S. 2017-09-08 19:20:52 +05:30
Jason Lowe c004d57d47 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:24:55 -05:00
Sunil G ef342455cb YARN-6248. user is not removed from UsersManager’s when app is killed with pending container requests. Contributed by Eric Payne. 
(cherry picked from commit c1386a90d7)
 2017-09-07 14:47:07 -05:00
Arun Suresh 48279f7876 YARN-6978. Add updateContainer API to NMClient. (Kartheek Muthyala via asuresh) 
(cherry picked from commit c41118a7f8)
 2017-09-07 10:30:41 -07:00
Sunil G 8cce19241e YARN-6992. Kill application button is visible even if the application is FINISHED in RM UI. Contributed by Suma Shivaprasad. 
(cherry picked from commit 6f101e7df1)
 2017-09-07 13:18:30 +05:30
Subru Krishnan 5ccdd83923 YARN-5328. Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 
(cherry picked from commit b6e7d13696)
 2017-09-06 16:46:49 -07:00
Junping Du 1ef48f1d4b YARN-7144. Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader. Contributed by Xuan Gong. 
(cherry picked from commit 22de9449f8)
 2017-09-06 14:54:29 -07:00
Jason Lowe fc444da891 YARN-7164. TestAMRMClientOnRMRestart fails sporadically with bind address in use. Contributed by Jason Lowe 
(cherry picked from commit 704267cb49)
 2017-09-06 15:25:37 -05:00
Junping Du eaf134c46c YARN-7148. TestLogsCLI fails in trunk and branch-2 and javadoc error. Contributed by Xuan Gong. 
(cherry picked from commit 63231a2a30)
 2017-09-06 13:20:58 -07:00
Wei-Chiu Chuang 92d9ad7761 Revert "Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem." 
This reverts commit 80516b3de7.
 2017-09-05 14:52:43 -07:00
Subru Krishnan 80516b3de7 Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 
(cherry picked from commit 7996eca7dc)
 2017-09-01 15:17:27 -07:00
Oleg Danilov b616462eef
YARN-7023. Incorrect ReservationId.compareTo() implementation. 
This closes #262

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c5281a85e1)
 2017-09-01 21:36:12 +02:00
Jason Lowe 57b5b186ce YARN-7120. CapacitySchedulerPage NPE in "Aggregate scheduler counts" section. Contributed by Eric Payne 
(cherry picked from commit 99a7f5d451)
 2017-09-01 11:10:40 -05:00
Junping Du 119220b88f YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-09-01 03:04:55 -07:00
Daniel Templeton 004231dc08 YARN-6780. ResourceWeights.toString() cleanup (Contributed by weiyuan via Daniel Templeton) 
(cherry picked from commit d4417dae4f)
 2017-08-31 09:38:31 -07:00
Sunil G e8b4427259 YARN-7116. CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage. Contributed by Wangda Tan. 
(cherry picked from commit f9e0cc8cdc)
 2017-08-31 18:43:17 +05:30
Robert Kanter 8523319e15 YARN-7094. Document the current known issue with server-side NM graceful decom (rkanter) 
(cherry picked from commit ce79f7be29)
 2017-08-30 18:06:03 -07:00
Rohith Sharma K S 877d96b83d YARN-6982. Potential issue on setting AMContainerSpec#tokenConf to null before app is completed. Contributed by Manikandan R. 
(cherry picked from commit 4cae120c61)
 2017-08-30 10:50:57 +05:30
Jian He 036cbd7453 YARN-6756. ContainerRequest#executionTypeRequest causes NPE. Contributed by Jian He 2017-08-29 20:36:23 -07:00
Wangda Tan 9bcd64d233 YARN-5219. When an export var command fails in launch_container.sh, the full container launch should fail. (Sunil G via wangda) 
Change-Id: Iaa6b978bb89482e9d1d77ba57f4adfdc48e39a3c
(cherry picked from commit f59332b97b)
 2017-08-29 15:29:53 -07:00
Junping Du b906bddd95 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang. 
(cherry picked from commit ad45d19998)
 2017-08-29 15:17:21 -07:00
Jason Lowe c930aa3ff9 YARN-5816. TestDelegationTokenRenewer#testCancelWithMultipleAppSubmissions is still flakey. Contributed by Robert Kanter 2017-08-29 10:02:44 -05:00
Junping Du f252e1fe28 YARN-7076. yarn application -list -appTypes is not working. Contributed by Jian He. 
(cherry picked from commit 312b1fd9da)
 2017-08-28 18:27:38 -07:00
Wangda Tan 82e1cd08eb YARN-7112. TestAMRMProxy is failing with invalid request. (Jason Lowe via wangda) 
Change-Id: I432f4523921666ac218fcf9936f826a0d8d635e0
(cherry picked from commit 51881a863c)
 2017-08-28 11:38:05 -07:00
Yufei Gu 221c1bdf7e YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu) 
(cherry picked from commit d5bc958279)
 2017-08-28 10:11:01 -07:00
Sunil G 80570737a8 YARN-7051. Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin. Contributed by Eric Payne. 
(cherry picked from commit 02599bda04)
 2017-08-28 12:27:29 +05:30
Li Lu b7f319c53c YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9) 
(cherry picked from commit 3bee7046c5)
 2017-08-25 22:45:20 -07:00
Eric Payne 1617744801 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe. 
(cherry picked from commit e864f81471)
 2017-08-25 15:52:29 -05:00
Jason Lowe ea5bb5898f YARN-7052. RM SchedulingMonitor gives no indication why the spawned thread crashed. Contributed by Eric Payne 
(cherry picked from commit 39a9dc8e4a)
 2017-08-25 15:32:29 -05:00
Jason Lowe 82957ce0d0 YARN-6640. AM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit 3a4e861169)
 2017-08-25 09:20:50 -05:00
Yufei Gu 389b659c94 YARN-7049. FSAppAttempt preemption related fields have confusing names. (Contributed by Karthik Kambatla via Yufei Gu) 
(cherry picked from commit 9e2699ac2c)
 2017-08-24 17:54:00 -07:00
Junping Du e12358c5ce YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 
(cherry picked from commit c2cb7ea1ef)
 2017-08-24 15:27:40 -07:00
Akira Ajisaka bf903396a0
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-25 03:28:02 +09:00
bibinchundatt cc50ca0722 YARN-7074. Fix NM state store update comment. Contributed by Botong Huang 
(cherry picked from commit de0cba700b)
 2017-08-24 16:40:18 +05:30
Junping Du c98917f798 YARN-7090. testRMRestartAfterNodeLabelDisabled get failed when CapacityScheduler is configured. Contributed by Wangda Tan. 
(cherry picked from commit 652dd434d9)
 2017-08-23 18:07:07 -07:00
Wangda Tan 245b49332d YARN-6251. Do async container release to prevent deadlock during container updates. (Arun Suresh via wangda) 
Change-Id: I6c67d20c5dd4d22752830ebf0ed2340824976ecb
(cherry picked from commit f49843a988)
 2017-08-23 10:35:37 -07:00
Subru Krishnan a45ffdcddc YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru). 
(cherry picked from commit 4249172e14)
 2017-08-22 19:21:35 -07:00
Ray Chiang 2299c8dffa YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 
(cherry picked from commit f5f14a2ad6)
 2017-08-22 15:24:02 -07:00
Jason Lowe b15a393440 YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-22 16:28:08 -05:00
Jason Lowe a7893033fe YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. Contributed by Daryn Sharp 
(cherry picked from commit 657dd59cc8)
 2017-08-22 13:17:31 -05:00
Jason Lowe fc7df25a11 YARN-2416. InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time. Contributed by Jonathan Eagles 
(cherry picked from commit 3efcd51c3b)
 2017-08-22 12:59:47 -05:00
Arun Suresh e0ba8a9ed8 YARN-6979. [Addendum patch] Fixed classname and added javadocs. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 7a82d7bcea)
 2017-08-20 10:28:42 -07:00
Arun Suresh f8f193c8e8 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 8410d862d3)
 2017-08-20 08:31:17 -07:00
Yufei Gu 4bd417a056 YARN-6969. Clean up unused code in class FairSchedulerQueueInfo. (Larry Lo via Yufei Gu) 
(cherry picked from commit 8991f0baec)
 2017-08-18 14:45:13 -07:00
Daniel Templeton 78f9dca2cb YARN-6952. Enable scheduling monitor in FS. (Yufei Gu via Daniel Templeton) 2017-08-18 14:40:30 -07:00
bibinchundatt ecf1a029bb YARN-7007. NPE in RM while using YarnClient.getApplications(). Contributed by Lingfeng Su. 
(cherry picked from commit e05fa3451d)
 2017-08-18 20:31:26 +05:30
Sunil G 5511cea86b YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad. 
(cherry picked from commit f9a0e23381)
 2017-08-17 15:10:07 +05:30
Naganarasimha fa3094e436 YARN-6327. Removing queues from CapacitySchedulerQueueManager and ParentQueue should be done with iterator. Contributed by Jonathan Hung. 
(cherry picked from commit 0a3aa40fe7)
 2017-08-17 10:42:00 +05:30
Jason Lowe cf30380d46 YARN-7020. TestAMRMProxy#testAMRMProxyTokenRenewal is flakey. Contributed by Robert Kanter 
(cherry picked from commit 14553061be)
 2017-08-16 13:06:13 -05:00
bibinchundatt 67f9968407 YARN-6741. Deleting all children of a Parent Queue on refresh throws exception. Contributed by Naganarasimha G R. 2017-08-16 14:52:09 +05:30
Akira Ajisaka a28a3dc8f2
YARN-6965. Duplicate instantiation in FairSchedulerQueueInfo. Contributed by Masahiro Tanaka. 
(cherry picked from commit 588c190afd)
 2017-08-16 14:07:08 +09:00
Subru Krishnan ad46a90872 HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru). 
(cherry picked from commit 75dd866bfb)
 2017-08-15 16:54:48 -07:00
Arun Suresh 7b22df3da6 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 4d7be1d857)
 2017-08-14 19:57:34 -07:00
Jason Lowe 56d0e757c7 YARN-6917. Queue path is recomputed from scratch on every allocation. Contributed by Eric Payne 
(cherry picked from commit 5558792894)
 2017-08-14 15:34:50 -05:00
Jason Lowe e5ba40cc98 YARN-6987. Log app attempt during InvalidStateTransition. Contributed by Jonathan Eagles 
(cherry picked from commit 3325ef653d)
 2017-08-14 14:41:38 -05:00
Subru Krishnan a70efb6138 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-14 11:03:50 -07:00
Carlo Curino 394573780b YARN-6687. Validate that the duration of the periodic reservation is less than the periodicity. (subru via curino) 
(cherry picked from commit 28d97b79b6)
 2017-08-14 11:01:56 -07:00
Jian He 630ada1b4e YARN-6959. RM may allocate wrong AM Container for new attempt. Contributed by Yuqi Wang 2017-08-14 10:56:39 -07:00
Daniel Templeton 7d3cea232d Revert "YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator" 
This reverts commit ce0cdc50d9.
 2017-08-11 20:52:07 -07:00
Daniel Templeton ce0cdc50d9 YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator 
(Contributed by Larry Lo via Daniel Templeton)

(cherry picked from commit 0996acde6c)
 2017-08-11 14:51:42 -07:00
Daniel Templeton 11e3673748 YARN-6967. Limit application attempt's diagnostic message size thoroughly 
(Contributed by Chengbing Liu via Daniel Templeton)

(cherry picked from commit 65364defb4)
 2017-08-11 14:31:11 -07:00
Daniel Templeton 2566e5fcf7 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces 
(Contributed by weiyuan via Daniel Templeton)

(cherry picked from commit c7680d4cc4)
 2017-08-11 14:23:17 -07:00
Karthik Kambatla f6c74ac5ea YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha) 
(cherry picked from commit 8c4b6d16a5)
 2017-08-11 12:18:29 -07:00
Akira Ajisaka 1f80e05eb4
YARN-6958. Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice. Contributed by Yeliang Cang. 2017-08-10 22:14:44 +09:00
Jian He 8b7590057e YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-08-09 11:12:38 -07:00
Subru Krishnan e9ec1e8f87 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-08-08 14:01:22 -07:00
Daniel Templeton c5b3d80a4b YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code 
(Contributed by Angela Wang via Daniel Templeton)

(cherry picked from commit e0c24145d2)
 2017-08-08 13:36:49 -07:00
Junping Du e9c21d14b4 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-08 12:45:55 -07:00
Daniel Templeton 675d368950 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path 
(Contributed by Miklos Szegedi via Daniel Templeton)

(cherry picked from commit 47b145b9b4)
 2017-08-08 11:12:34 -07:00
Jian He ad1567c487 YARN-6890. Not display killApp button on UI if UI is unsecured but cluster is secured. Contributed by Junping Du 2017-08-08 11:10:29 -07:00
Akira Ajisaka 9fd5a03611
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-08 19:28:33 +09:00
Arun Suresh 3f50800d41 YARN-6920. Fix resource leak that happens during container re-initialization. (asuresh) 
(cherry picked from commit 8d3fd81980)
 2017-08-07 19:01:13 -07:00
Wangda Tan e18415a350 YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
(cherry picked from commit adb84f34db)
 2017-08-07 11:34:45 -07:00
Akira Ajisaka cfdf29785b
YARN-6957. Moving logging APIs over to slf4j in hadoop-yarn-server-sharedcachemanager. Contributed by Yeliang Cang. 
(cherry picked from commit b0fbf17965)
 2017-08-07 19:26:18 +09:00
Sunil G d15bac141e YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 
(cherry picked from commit 46b7054fa7)
 2017-08-07 13:17:04 +05:30
Yufei Gu b9549e108e YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-08-04 17:23:44 -07:00
Carlo Curino 7654b52167 YARN-6634. [API] Refactor ResourceManager WebServices to make API explicit (Contributed by Giovanni Matteo Fumarola via curino). 
(Edited cherry-pick of 547cb53e8a).
 2017-08-04 17:04:20 -07:00
Arun Suresh 4cfd248a72 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 
(cherry picked from commit 077fcf6a96)
 2017-08-04 16:50:21 -07:00
Arun Suresh 043b7d133e YARN-6776. Refactor ApplicaitonMasterService to move actual processing logic to a separate class. (asuresh) 
(cherry picked from commit 5496a34c0c)
 2017-08-04 16:43:35 -07:00
Arun Suresh 2d62af6545 YARN-5966. AMRMClient changes to support ExecutionType update. (asuresh) 
(cherry picked from commit aaf106fde3)
 2017-08-04 16:19:44 -07:00
Arun Suresh 88d6adf419 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 35dc782923)
 2017-08-03 21:18:43 -07:00
Yufei Gu b6729a7be5 YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-03 16:56:15 -07:00
Yufei Gu ce609d06c0 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 13:52:58 -07:00
Sunil G 82ec700241 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 
(cherry picked from commit f64cfeaf61)
 2017-08-03 19:28:16 +05:30
Eric Payne 223229b99c YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 
(cherry picked from commit 8ce8672b6b)
 2017-08-02 12:58:07 -05:00
Eric Payne 686bdc48b8 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 
(cherry picked from commit 48899134d2)
 2017-08-02 11:29:56 -05:00
Jian He 6ee0fe70c7 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:59 -07:00
Jian He 4006cc266b YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 10:00:04 -07:00
Akira Ajisaka 72bde1dad5
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 
(cherry picked from commit b38a1eea8e)
 2017-08-01 15:16:21 +09:00
Jason Lowe b9640c02bf YARN-6628. Unexpected jackson-core-2.2.3 dependency introduced. Contributed by Jonathan Eagles 2017-07-31 16:20:54 -05:00
Akira Ajisaka 98c3544e94
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 
(cherry picked from commit f8bed5e9a7)
 2017-07-31 11:10:10 +09:00
Arun Suresh 7e643130f2 YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 
(cherry picked from commit 890e14c02a)
 2017-07-28 16:33:42 -07:00
Yufei Gu e29d1c75ea YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit 9902be72cb)
 2017-07-27 23:21:30 -07:00
Varun Saxena 05bbbcbdbe Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit 11ece0bda1)
 2017-07-28 00:33:41 +05:30
Varun Saxena 0b342a644a YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit f66fd11e51)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
 2017-07-28 00:33:37 +05:30
Sunil G 08d5f3acf6 Addendum patch for YARN-6102. 2017-07-26 18:40:52 +05:30
Daniel Templeton 00f4a80e8f YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit f81a4efb8c)
 2017-07-25 13:02:00 -07:00
Akira Ajisaka 22f237c5d9
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 
(cherry picked from commit cca51e916b)
 2017-07-25 15:25:36 +09:00
Akira Ajisaka 09241feb13
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 
(cherry picked from commit 218b1b33ff)
 2017-07-25 15:12:47 +09:00
Daniel Templeton 9b6e787230 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 10583625c1)
 2017-07-24 13:45:09 -07:00
Daniel Templeton eb1751670a YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)

(cherry picked from commit bb30bd3771)
 2017-07-24 12:15:35 -07:00
Daniel Templeton 6b8e353c65 YARN-6844. AMRMClientImpl.checkNodeLabelExpression() has wrong error message 
(Contributed by Manikandan R via Daniel Templeton)

(cherry picked from commit 4c40cd451c)
 2017-07-24 12:06:50 -07:00
Sunil G a657472b42 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 20:57:56 +05:30
Jason Lowe db7c5636b6 YARN-6768. Improve performance of yarn api record toString and fromString. Contributed by Jonathan Eagles 
(cherry picked from commit 24853bf32a)
 2017-07-24 10:22:31 -05:00
Jason Lowe e76e7b9f0b YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 
(cherry picked from commit c8df3668ec)
 2017-07-20 11:06:39 -05:00
Wangda Tan 5070c9ba39 YARN-6775. CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls. (Nathan Roberts via wangda) 
Change-Id: Iaf9bb7e5ed3aa1300abdccf87ae6fcbddbd25e3e
 2017-07-19 15:25:21 -07:00
Rohith Sharma K S f806e8627b YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 
(cherry picked from commit 845c4e52bd)
 2017-07-19 11:16:17 +05:30
Yufei Gu d04bb465d2 YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit daaf530fce)
 2017-07-18 16:40:13 -07:00
Sunil G 487b0a3d7f Addendum patch for YARN-5731 
(cherry picked from commit 0b7afc060c)
 2017-07-18 13:46:43 +05:30
Sunil G 9b1abb448d YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. Contributed by Tao Yang. 2017-07-17 13:22:04 +05:30
Sunil G d462e48336 YARN-6280. Introduce deselect query param to skip ResourceRequest from getApp/getApps REST API. Contributed by Lantao Jin. 2017-07-17 12:25:41 +05:30
Arun Suresh 0101973dbf YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh) 2017-07-14 13:57:22 -07:00
Jason Lowe 9eb93c6f3f YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 
(cherry picked from commit a5ae5ac50e)
 2017-07-14 15:03:48 -05:00
bibinchundatt af80d10e54 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 
(cherry picked from commit 75c0220b44)
 2017-07-14 14:05:38 +05:30
Yufei Gu b2a34fe8d6 YARN-6769. Make schedulables without demand less needy in FairSharePolicy#compare. (Yunfan Zhou via Yufei Gu) 
(cherry picked from commit 4a574e9a84)
 2017-07-13 23:11:27 -07:00
Sunil G fd6935709f YARN-5731. Preemption calculation is not accurate when reserved containers are present in queue. Contributed by Wangda Tan. 2017-07-14 08:45:17 +05:30
Sunil G f5cdee6bfd YARN-6792. Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 228ddaa31d)
 2017-07-14 08:08:32 +05:30
Jason Lowe a77fb561ef YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 
(cherry picked from commit ebc048cc05)
 2017-07-13 17:49:55 -05:00
Jason Lowe 6ce2b0e4d4 YARN-6654. RollingLevelDBTimelineStore backwards incompatible after fst upgrade. Contributed by Jonathan Eagles 
(cherry picked from commit 5f1ee72b0e)
 2017-07-13 17:28:49 -05:00
Sunil G d83e8712f6 YARN-5892. Support user-specific minimum user limit percentage in Capacity Scheduler. Contributed by Eric Payne. 
(cherry picked from commit ca13b224b2)
 2017-07-13 14:50:28 -05:00
Wangda Tan cbb5f601f1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Sunil G via wangda) 
Change-Id: I9b19f69788068be05b3295247cdd7b972f8a573c
(cherry picked from commit c583ab02c7)
 2017-07-13 10:44:07 -05:00
Xuan edc0aa95f1 YARN-6689. PlacementRule should be configurable. (Jonathan Hung via xgong) 2017-07-12 15:45:51 -07:00
Jason Lowe 401b191a3c YARN-6797. TimelineWriter does not fully consume the POST response. Contributed by Jason Lowe 
(cherry picked from commit 655110393b)
 2017-07-12 15:54:17 -05:00
Wangda Tan f2d440b3b3 YARN-5889. Improve and refactor user-limit calculation in Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit 5fb723bb77)
 2017-07-12 10:06:02 -05:00
Eric Payne e6cdf770ca Revert "YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G)" 
This reverts commit eda4ac07c1.
Commit eda4ac07c1 was a separate patch from trunk rather than a cherry-pick. I will cherryp-ick dependencies and then cherry-pick the trunk commit for YARN-2113.
 2017-07-12 08:56:40 -05:00
Akira Ajisaka 3419381e98
YARN-6809. Fix typo in ResourceManagerHA.md. Contributed by Yeliang Cang. 
(cherry picked from commit b628d0da51)
 2017-07-12 19:39:03 +09:00
Yufei Gu 20a2770d72 YARN-6764. Simplify the logic in FairScheduler#attemptScheduling. Contributed by Yufei Gu. 
(cherry picked from commit 3de47ab5ea)
 2017-07-09 16:11:26 -07:00
Naganarasimha 5f603306d8 YARN-6770. A small mistake in the example of TimelineClient. Contributed by Jinjiang Ling. 
(cherry picked from commit ba5b056e8a)
 2017-07-09 21:31:53 +05:30
Naganarasimha b08ef44d9e YARN-6428. Queue AM limit is not honored in CS always. Contributed by Bibin A Chundatt. 
(cherry picked from commit 626d730bfc)
 2017-07-09 17:27:50 +05:30
Daniel Templeton fcef28e53b YARN-6410. FSContext.scheduler should be final (Contributed by Yeliang Cang via Daniel Templeton) 
(cherry picked from commit 9d27880442)
 2017-07-09 19:38:37 +09:00
Daniel Templeton a6d194d720 YARN-6746. SchedulerUtils.checkResourceRequestMatchingNodePartition() is dead code 
(Contributed by Deepti Sawhney via Daniel Templeton)

(cherry picked from commit c5a0c38d8e)
 2017-07-09 19:31:20 +09:00
Daniel Templeton edcfae8258 YARN-6752. Display reserved resources in web UI per application 
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit 0615985886)
 2017-07-09 19:19:07 +09:00
Eric Payne eda4ac07c1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G) 2017-07-06 10:43:00 -05:00
Jason Lowe 53c2441c10 YARN-6708. Nodemanager container crash after ext3 folder limit. Contributed by Bibin A Chundatt 
(cherry picked from commit 7576a688ea)
 2017-07-06 09:44:02 -05:00
Naganarasimha 87c81cfad3 YARN-6681. Eliminate double-copy of child queues in canAssignToThisQueue. Contributed by Daryn Sharp. 
(cherry picked from commit fa1aaee87b)
 2017-07-01 12:32:01 +05:30
Daniel Templeton 93c5421a43 YARN-6751. Display reserved resources in web UI per queue 
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit ec97519779)
 2017-06-29 16:57:23 -07:00
Daniel Templeton 15242786b6 YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description needs spaces in yarn-default.xml 
(Contributed by Lori Loberg via Daniel Templeton)

(cherry picked from commit 25d891a784)
 2017-06-28 17:17:56 -07:00
Junping Du c5f43feb7c YARN-5311. Document graceful decommission CLI and usage. Contributed by Elek, Marton. 
(cherry picked from commit 4e3eebc943)
 2017-06-28 15:32:49 -07:00
Naganarasimha a5ae7c0cae YARN-6467. CSQueueMetrics needs to update the current metrics for default partition only. Contributed by Manikandan R. 2017-06-29 00:09:30 +05:30
Jason Lowe 44b3c6ffaa YARN-6738. LevelDBCacheTimelineStore should reuse ObjectMapper instances. Contributed by Zoltan Haindrich 
(cherry picked from commit 63ce1593c5)
 2017-06-27 17:13:49 -05:00
Naganarasimha 92e266c2b6 YARN-5006. ResourceManager quit due to ApplicationStateData exceed the limit size of znode in zk. Contributed by Bibin A Chundatt. 2017-06-24 23:14:51 +05:30
Yufei Gu 4c59b446e2 YARN-5876. TestResourceTrackerService#testGracefulDecommissionWithApp fails intermittently on trunk. (Robert Kanter via Yufei Gu) 
(cherry picked from commit 0b77262890)
 2017-06-23 13:30:20 -07:00
Arun Suresh 66f780e6ab YARN-6127. Add support for work preserving NM restart when AMRMProxy is enabled. (Botong Huang via asuresh). 2017-06-22 15:07:38 -07:00
Akira Ajisaka 35f6c7c9c7
YARN-6713. Fix dead link in the Javadoc of FairSchedulerEventLog.java. Contributed by Weiwei Yang. 
(cherry picked from commit 099dfe92a9)
 2017-06-20 17:23:06 +09:00
Naganarasimha 578318473c YARN-6680. Avoid locking overhead for NO_LABEL lookups. Contributed by Daryn Sharp 
(cherry picked from commit ee89ac84e6)
 2017-06-19 22:45:21 +05:30
Naganarasimha 3106f256e2 YARN-6517. Fix warnings from Spotbugs in hadoop-yarn-common(addendum). Contributed by Weiwei Yang. 
(cherry picked from commit 7582dedad1)
 2017-06-18 19:56:11 +05:30
bibinchundatt 3f9acf65fb YARN-6703. RM startup failure with old state store due to version mismatch. Contributed by Varun Saxena 
(cherry picked from commit d64c842743)
 2017-06-12 18:32:23 +05:30
Eric Payne 33605e0896 YARN-6585. RM fails to start when upgrading from 2.7 for clusters with node labels. Contributed by Sunil G. 
(cherry picked from commit 5578af8603)
 2017-06-09 16:18:16 -05:00
Naganarasimha 85d561a5b3 YARN-6681. Improve performance of AssignmentInformation datastructures. Contributed by Daryn sharp. 2017-06-09 11:46:51 +05:30
Daniel Templeton c5b28c03a1 YARN-6679. Reduce Resource instance overhead via non-PBImpl 
(Contributed by Daryn Sharp via Daniel Templeton)
 2017-06-08 10:41:32 -07:00
Daniel Templeton 0bcbf2a09b YARN-6208. Improve the log when FinishAppEvent sent to the NodeManager which didn't run the application 
(Contributed by Akira Ajisaka via Daniel Templeton)

(cherry picked from commit 73ecb19312)
 2017-06-02 08:52:27 -07:00
Daniel Templeton 10b7a5fa98 YARN-6246. Identifying starved apps does not need the scheduler writelock 
(Contributed by Karthik Kambatla via Daniel Templeton)

(cherry picked from commit d5b71e4175)
 2017-05-31 15:50:11 -07:00
Nathan Roberts c112bf6838 YARN-6649. RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception. Contributed by Jon Eagles. 
(cherry picked from commit 4369690ce6)
 2017-05-31 12:35:50 -05:00
Jonathan Eagles f5f12b576e YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely Novák via jeagles) 
(cherry picked from commit cbfed0e82f)
 2017-05-31 10:19:51 -05:00
Varun Vasudev 1eecde3355 YARN-6366. Refactor the NodeManager DeletionService to support additional DeletionTask types. Contributed by Shane Kumpf. 
(cherry picked from commit 547f18cb96)
 2017-05-31 16:16:33 +05:30
Jason Lowe 3c0eb9c415 YARN-6641. Non-public resource localization on a bad disk causes subsequent containers failure. Contributed by Kuhu Shukla 
(cherry picked from commit aea42930bb)
 2017-05-26 09:39:22 -05:00
Yufei Gu fc6cb4b2dd YARN-6582. FSAppAttempt demand can be updated atomically in updateDemand(). (Karthik Kambatla via Yufei Gu) 
(cherry picked from commit 87590090c8)
 2017-05-25 14:25:49 -07:00
Jason Lowe 4c44ff69df YARN-6643. TestRMFailover fails rarely due to port conflict. Contributed by Robert Kanter 
(cherry picked from commit 3fd6a2da4e)
 2017-05-25 16:09:08 -05:00
Varun Vasudev 1a6c532301 Addendum patch to fix Docker sanitization. 
(cherry picked from commit 2ff2a1f50e8c7c0f33676b010b256d6c8daf912d)
 2017-05-25 14:54:08 +05:30
Akira Ajisaka ca1c0cbc62
YARN-6141. ppc64le on Linux doesn't trigger __linux get_executable codepath. Contributed by Sonia Garudi and Ayappan. 
(cherry picked from commit bc28da65fb)
 2017-05-25 17:07:50 +09:00
Subru Krishnan d1724ef919 YARN-6587. Refactor of ResourceManager#startWebApp in a Util class. (Giovanni Matteo Fumarola via Subru). 2017-05-24 13:40:49 -07:00
Jason Lowe 96a97e7eb2 YARN-6615. AmIpFilter drops query parameters on redirect. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 8bf1949c0e)
 2017-05-24 11:24:00 -05:00
Sunil G ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Wangda Tan 9a8fdfa385 YARN-6493. Print requested node partition in assignContainer logs. (Jonathan Hung via wangda) 
Change-Id: Ib20326daac10acad363befb0ae0998441fa5b32d
(cherry picked from commit 8e0f83e49a)
 2017-05-22 14:43:44 -07:00
Robert Kanter 2719cc0388 YARN-6602. Impersonation does not work if standby RM is contacted first (rkanter) 
(cherry picked from commit 9855225a79)
 2017-05-19 15:43:43 -07:00
Yufei Gu 50e263a316 YARN-6249. TestFairSchedulerPreemption fails inconsistently. (Tao Jie via Yufei Gu) 
(cherry picked from commit 66bba8c024)
 2017-05-19 15:22:14 -07:00
Eric Payne 46c54bf200 YARN-6618. TestNMLeveldbStateStoreService#testCompactionCycle can fail if compaction occurs more than once. Contributed by Jason Lowe. 
(cherry picked from commit 19482e0d09)
 2017-05-19 11:29:02 -05:00
Varun Vasudev cd1a04e3c6 Sanitize arguments before launching Docker containers. 2017-05-18 11:46:03 +05:30
Varun Vasudev 603a0a316c Validate docker image name before launching container. 2017-05-18 11:31:12 +05:30
Chris Douglas 3d2afb209c YARN-6577. Remove unused ContainerLocalization classes. Contributed by ZhangBing Lin 
(cherry picked from commit b23fcc86c6)
 2017-05-17 20:40:20 -07:00
Rohith Sharma K S 379e5bd26f HADOOP-14412. HostsFileReader#getHostDetails is very expensive on large clusters. Contributed by Jason Lowe. 2017-05-17 08:27:50 +05:30
Jian He 82751ff0cb YARN-6306. NMClient API change for container upgrade. Contributed by Arun Suresh 2017-05-16 10:49:12 -07:00
Jason Lowe feb7e92121 YARN-6603. NPE in RMAppsBlock. Contributed by Jason Lowe 
(cherry picked from commit 489f85933c)
 2017-05-16 09:28:42 -05:00
Jonathan Eagles d370edc6aa YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles) 
(cherry picked from commit c48f2976a3)
 2017-05-15 10:32:47 -05:00
Haibo Chen 6284cc4663 YARN-6580. Incorrect logger for FairSharePolicy. (Vrushali C via Haibo Chen) 
(cherry picked from commit 6c35001b9f)
 2017-05-12 13:07:17 -07:00
Daniel Templeton 717f03ecf7 YARN-6380. FSAppAttempt keeps redundant copy of the queue 
(cherry picked from commit 90cb5b4635)
 2017-05-11 13:48:46 -07:00
Min Shen 9310d24b08 YARN-5543. ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread. Contributed by Min Shen. 
(cherry picked from commit 2ada100da7)
 2017-05-11 13:29:02 -07:00
Brahma Reddy Battula 3766e54893 HADOOP-14410. Correct spelling of 'beginning' and variants. Contributed By Dongtao Zhang 
(cherry picked from commit 4aae2d40a3)
 2017-05-11 23:38:18 +05:30
Daniel Templeton df3d7415c0 YARN-6571. Fix JavaDoc issues in SchedulingPolicy 
(Contributed by Weiwei Yang via Daniel Templeton)

(cherry picked from commit 18d54b5ae2f76513548c453e05cdd6d87777cd06)
 2017-05-10 14:37:36 -07:00
Jason Lowe 3230dd1ab6 YARN-6552. Increase YARN test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 
(cherry picked from commit 6099deebcb)
 2017-05-10 08:48:47 -05:00
Karthik Kambatla 3403ed3148 YARN-3742. YARN RM will shut down if ZKClient creation times out. (Daniel Templeton via kasha) 
(cherry picked from commit 166be0ee95)
 2017-05-09 14:44:39 -07:00
Daniel Templeton 486a160735 YARN-5301. NM mount cpu cgroups failed on some systems 
(Contributed by Miklos Szegedi via Daniel Templeton)

(cherry picked from commit a2f680493f)
 2017-05-09 12:56:28 -07:00
Daniel Templeton 920a3078f6 YARN-6302. Fail the node if Linux Container Executor is not configured properly 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-05-09 10:08:07 -07:00
Subru Krishnan d8f0d409b9 YARN-6281. Cleanup when AMRMProxy fails to initialize a new interceptor chain. (Botong Huang via Subru) 
(cherry picked from commit 57a9afbd45b7ef8e6021cc58f96bd0074bf1389d)
(cherry picked from commit 749e5c09b9)
 2017-05-08 16:58:42 -07:00
Subru Krishnan 7511cfb842 YARN-6234. Support multiple attempts on the node when AMRMProxy is enabled. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit cd9ff27ffc)
 2017-05-08 16:42:15 -07:00
Haibo Chen 86fcb8a66c YARN-6457. Allow custom SSL configuration to be supplied in WebApps. (Sanjay M Pujare via Haibo Chen) 
(cherry picked from commit 1769b12a77)
 2017-05-08 15:47:01 -07:00
Jason Lowe 6ca0c134e4 YARN-3839. Quit throwing NMNotYetReadyException. Contributed by Manikandan R 
(cherry picked from commit 424887ecb7)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
 2017-05-08 17:25:11 -05:00
Subru Krishnan 36d97e7aa8 YARN-6565. Fix memory leak and finish app trigger in AMRMProxy. (Botong Huang via Subru). 
(cherry picked from commit d6eed5acca)
 2017-05-05 16:28:51 -07:00
Sidharta S 6186d2e742 YARN-6374. Improve test coverage and add utility classes for common Docker operations. Contributed by Shane Kumpf 2017-05-04 14:41:04 -07:00
Daniel Templeton b337a1da31 YARN-6481. Yarn top shows negative container number in FS 
(Contributed by Tao Jie via Daniel Templeton)

(cherry picked from commit 9f0aea0ee2)
 2017-05-02 13:15:18 -07:00
Naganarasimha 5e28311341 YARN-6519. Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager. Contributed by Weiwei Yang. 2017-05-03 00:36:33 +05:30
Subru Krishnan 6d53224afc YARN-5331. Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem. (Sangeetha Abdu Jyothi via Subru). 
(cherry picked from commit 6bf42e48ef)
 2017-05-01 18:49:14 -07:00
Subru Krishnan 2030f408e0 YARN-6542. Fix the logger in TestAlignedPlanner and TestGreedyReservationAgent. 
(cherry picked from commit e514fc432a)
 2017-05-01 18:31:08 -07:00
Subru Krishnan 0eae1c6368 YARN-4359. Update LowCost agents logic to take advantage of YARN-4358. (Jonathan Yaniv and Ishai Menache via Subru). 
(cherry picked from commit a3a615eeab)
 2017-05-01 18:30:58 -07:00
Steve Loughran 5ca4f0f82e YARN-524 TestYarnVersionInfo failing if generated properties doesn't include an SVN URL 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1463300 13f79535-47bb-0310-9956-ffa450edef68
(cherry picked from commit 7c65dbb938)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2017-05-01 08:58:12 -05:00
Naganarasimha d9afb237fb YARN-6520. Fix warnings from Spotbugs in hadoop-yarn-client. Contributed by Weiwei Yang. 
(cherry picked from commit 64f68cb0b8)
 2017-05-01 18:52:52 +05:30
Naganarasimha 29370e7784 YARN-6517. Fix warnings from Spotbugs in hadoop-yarn-common. Contributed by Weiwei Yang 
(cherry picked from commit 4b5bd73ac5)
 2017-05-01 16:35:45 +05:30
Daniel Templeton a8512d5aeb YARN-2962. ZKRMStateStore: Limit the number of znodes under a znode (Contributed by Varun Sexena via Daniel Templeton) 
(cherry picked from commit 2e52789edf)
 2017-04-28 13:39:11 -07:00
Eric Payne 92e5650f98 YARN-6536. TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently. Contributed by Jason Lowe. 
(cherry picked from commit fdf5192bbb)
 2017-04-28 15:21:58 -05:00
Robert Kanter 816d98e7cd YARN-5894. fixed license warning caused by de.ruedigermoeller:fst:jar:2.24 (haibochen via rkanter) 
(cherry picked from commit 371b6467dc)
 2017-04-28 09:15:40 -07:00
Haibo Chen 894521673b YARN-6510. Fix profs stat file warning caused by process names that includes parenthesis. (Wilfred Spiegelenburg via Haibo Chen) 
(cherry picked from commit 4f3ca0396a)
 2017-04-26 11:47:23 -07:00
Wangda Tan 503dd650bf YARN-6392. Add submit time to Application Summary log. (Zhihai Xu via wangda) 
(cherry picked from commit 2ba21d6376)
 2017-04-24 13:18:47 -07:00
Haibo Chen 49a2f286f7 YARN-6500. Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler. (Miklos Szegedi via Haibo Chen) 
(cherry picked from commit 8ac50e1322)
 2017-04-24 11:39:55 -07:00
Eric Payne 421e51cb9c YARN-5617. AMs only intended to run one attempt can be run more than once? Contributed by Jason Lowe. 
(cherry picked from commit 52adf71914)
 2017-04-24 12:08:35 -05:00
Sunil G a9495b206b YARN-6164. Expose Queue Configurations per Node Label through YARN client api. Contributed by Benson Qiu. 2017-04-24 12:16:18 +05:30
Daniel Templeton d77137a03f YARN-6453. fairscheduler-statedump.log gets generated regardless of service 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit e74ca7216aecc998d277e5ca1f4d0abae43c7e31)
 2017-04-20 14:14:38 -07:00
Jason Lowe f5a9287bb1 YARN-6501. FSSchedulerNode.java fails to compile with JDK7. Contributed by John Zhuge 2017-04-20 09:52:44 -05:00
Junping Du 9289f4527d YARN-6304. Skip rm.transitionToActive call to RM if RM is already active. Contributed by Rohith Sharma K S. 
(cherry picked from commit 6ed9d36224)
 2017-04-17 15:52:02 -07:00
Karthik Kambatla 6f5912c106 YARN-6432. FairScheduler: Reserve preempted resources for corresponding applications. (Miklos Szegedi via kasha) 2017-04-17 14:34:28 -07:00
Wangda Tan 6ade9e6ce6 YARN-6406. Remove SchedulerRequestKeys when no more pending ResourceRequest. (Arun Suresh via wangda) 
(cherry picked from commit 87e2ef8c98)
 2017-04-15 21:34:29 -07:00
Jason Lowe 1699b20526 YARN-6480. Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart. Contributed by Eric Badger 
(cherry picked from commit 4168805502)
 2017-04-14 17:17:46 -05:00
Karthik Kambatla 53da65ef8b YARN-6433. Only accessible cgroup mount directories should be selected for a controller. (Miklos Szegedi via kasha) 
(cherry picked from commit 8a1d7480f7)
 2017-04-14 15:07:54 -07:00
Eric Payne 157a9f3104 YARN-5994. TestCapacityScheduler.testAMLimitUsage fails intermittently. Contributed by Eric Badger 
(cherry-picked from commit a41f8dd58e)
 2017-04-14 11:04:18 -05:00
Wangda Tan 5756256280 YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 
(cherry picked from commit eac6b4c35c)
 2017-04-13 13:16:31 -07:00
Wangda Tan 408d23477f YARN-6040. Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes. (wangda) 2017-04-13 11:11:57 -07:00
Junping Du d77cada182 YARN-3760. FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close(). Contributed by Haibo Chen. 
(cherry picked from commit c26ccf1adb3a72df3f68e1150b86b813c691203a)
(cherry picked from commit 9d9087a67a)
 2017-04-12 13:54:58 -07:00
Eric Payne 607295d21f YARN-6450. TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTest? Contributed by Jason Lowe. 
(cherry picked from commit a16ab2be91)
 2017-04-12 10:26:31 -05:00
Wangda Tan 2f08c86b7a YARN-6439. Fix ReservationSystem creation of default ReservationQueue. (Carlo Curino via wangda) 
(cherry picked from commit 4d4ad0ebb7)
 2017-04-11 14:59:34 -07:00
Naganarasimha b04c09163b YARN-6463. Correct spelling mistake in FileSystemRMStateStore. Contributed by Yeliang Cang. 
(cherry picked from commit 7d873c465b)
 2017-04-11 22:46:09 +05:30
Jason Lowe b29fb55798 YARN-6461. TestRMAdminCLI has very low test timeouts. Contributed by Eric Badger 
(cherry picked from commit 90d97372ed)
 2017-04-11 09:17:42 -05:00
Jason Lowe 0cb22a0f15 YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. Contributed by Benson Qiu 
(cherry picked from commit 0e065f2ede)
 2017-04-11 08:45:16 -05:00
Konstantinos Karanasos 9b3ba25980 YARN-6344. Add parameter for rack locality delay in CapacityScheduler. (kkaranasos) 
(cherry picked from commit 3587b46164514e6751d7cebb09c8ab5e988853b4)
 2017-04-10 17:22:29 -07:00
Daniel Templeton c1ae753de1 YARN-6372. Add default value for NM disk validator (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit e486569197e10bf48731d9e7ef5c031987c26acc)
 2017-04-10 14:57:19 -07:00
Daniel Templeton 2dae9299ef YARN-6343. Docker docs MR example is broken (Contributed by Prashant Jha via Daniel Templeton) 
(cherry picked from commit 14a39907c6)
 2017-04-10 10:03:06 -07:00
Karthik Kambatla e0b143e51a Revert "YARN-6298. Metric preemptCall is not used in new preemption. (Yufei Gu via kasha)" 
This reverts commit dcfce5fcdd.
 2017-04-07 17:37:31 -07:00
Karthik Kambatla dcfce5fcdd YARN-6298. Metric preemptCall is not used in new preemption. (Yufei Gu via kasha) 
(cherry picked from commit 2aa8967809)
 2017-04-07 17:36:39 -07:00
Robert Kanter cbb1c76195 YARN-6368. Decommissioning an NM results in a -1 exit code (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 63f7322522)
 2017-04-07 14:28:39 -07:00
Akira Ajisaka a467e5412e
YARN-6288. Addendum patch to fix compile error. 2017-04-07 12:19:46 +09:00
Ming Ma db5b4c292b YARN-5797. Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches. (Chris Trezzo via mingma) 2017-04-06 17:08:59 -07:00
Jason Lowe 84b7f2e956 YARN-6288. Exceptions during aggregated log writes are mishandled. Contributed by Akira Ajisaka 
(cherry picked from commit 1b081ca27e)
 2017-04-06 16:25:41 -05:00
Daniel Templeton 59ad5cd68d YARN-6381. FSAppAttempt has several variables that should be final 
(Contributed by Ameet Zaveri via Daniel Templeton)

(cherry picked from commit 984df31a31c25ffc8541467dc41913f62f69e1df)
 2017-04-05 16:44:35 -07:00
Karthik Kambatla 7c0f9bd5e7 YARN-6448. Continuous scheduling thread crashes while sorting nodes. (Yufei Gu via kasha) 
(cherry picked from commit b4c4f36594)
 2017-04-05 16:00:30 -07:00
Varun Saxena 0a7ec1fcf7 YARN-6437. TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently (Jason Lowe via Varun Saxena) 
(cherry picked from commit 7d963c477a)
 2017-04-06 01:58:32 +05:30
Jason Lowe 903278559a YARN-6403. Invalid local resource request can raise NPE and make NM exit. Contributed by Tao Yang 
(cherry picked from commit e8071aa249)
 2017-04-05 12:31:48 -05:00
Varun Saxena 05c008256a YARN-6420. RM startup failure due to wrong order in nodelabel editlog (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit 19b89c4c7b)
 2017-04-05 09:40:09 +05:30
Ming Ma 7507ccd38a YARN-6004. Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines. (Chris Trezzo via mingma) 2017-04-04 18:05:09 -07:00
Karthik Kambatla 1938f97c0b YARN-6436. TestSchedulingPolicy#testParseSchedulingPolicy timeout is too low. (Eric Badger via kasha) 
(cherry picked from commit 9cc04b4708)
 2017-04-04 17:08:59 -07:00
Wangda Tan 70a810adc0 YARN-6109. Add an ability to convert ChildQueue to ParentQueue. (Xuan Gong via wangda) 
(cherry picked from commit 3fdae0a2b6)
 2017-04-04 14:39:38 -07:00
Eric Payne 4396c8e938 YARN-6354. LeveldbRMStateStore can parse invalid keys when recovering reservations. Contributed by Jason Lowe 
(cherry picked from commit 318bfb01bc)
 2017-03-31 12:36:01 -05:00
Jason Lowe 3ae529ba54 YARN-6411. Clean up the overwrite of createDispatcher() in subclass of MockRM. Contributed by Yufei Gu 
(cherry picked from commit 4d1fac5df2)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java
 2017-03-31 10:21:44 -05:00
Varun Saxena 3fe7d36e72 YARN-6352. Header injections are possible in application proxy servlet (Naganarasimha G R via Varun Saxena) 2017-03-30 03:49:32 +05:30
Daniel Templeton c2636468d7 YARN-5685. RM configuration allows all failover methods to disabled when automatic failover is enabled 
(cherry picked from commit 640ba1d23f)
 2017-03-29 12:38:50 -07:00
Akira Ajisaka 554bc8342b
YARN-6329. Remove unnecessary TODO comment from AppLogAggregatorImpl.java. Contributed by victor bertschinger. 
(cherry picked from commit 82fb9ce8df)
 2017-03-29 10:37:52 +09:00
Robert Kanter a65011a128 YARN-6050. AMs can't be scheduled on racks or nodes (rkanter) 2017-03-28 14:33:08 -07:00
Varun Saxena 97c83f2498 YARN-5368. Memory leak in timeline server (Jonathan Eagles via Varun Saxena) 
(cherry picked from commit 01aca54a22)
 2017-03-29 01:56:59 +05:30
Jason Lowe bea8f8190f YARN-6359. TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition. Contributed by Robert Kanter 2017-03-28 09:45:18 -05:00
Wangda Tan c97da481d7 YARN-6339. Improve performance for createAndGetApplicationReport. (Yunjiong Zhao via wangda) 
(cherry picked from commit cd014d57aa)
 2017-03-27 13:59:36 -07:00
Akira Ajisaka e4320305d3 YARN-6379. Remove unused argument in ClientRMService. Contributed by Kai Sasaki. 
(cherry picked from commit b63ae854f9)
 2017-03-27 11:26:23 +09:00
Sunil G f51ee482a7 YARN-5956. Refactor ClientRMService for unify error handling across apis. Contributed by Kai Sasaki. 2017-03-26 17:00:06 +05:30
Junping Du 3ad4d37363 Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 release. 
(cherry picked from commit 6df029db36)
 2017-03-25 14:33:20 -07:00
Daniel Templeton 6209e4c913 YARN-6334. TestRMFailover#testAutomaticFailover always passes even when it should fail 
(Contributed by Yufei Gu via Daniel Templeton)
 2017-03-24 14:02:39 -07:00
Daniel Templeton d4bba011cc YARN-6284. hasAlreadyRun should be final in ResourceManager.StandByTransitionRunnable 
(Contributed by Laura Adams via Daniel Templeton)

(cherry picked from commit 0a05c5c598)
 2017-03-21 14:27:27 -07:00
Junping Du dabe96a72e YARN-6367. YARN logs CLI needs alway check containerLogsInfo/containerLogInfo before parse the JSON object from NMWebService. Contributed by Xuan Gong. 
(cherry picked from commit c01d15ab27)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
 2017-03-21 10:55:19 -07:00
Daniel Templeton f7177d91d6 YARN-6309. Fair scheduler docs should have the queue and queuePlacementPolicy 
elements listed in bold so that they're easier to see
(Contributed by Esmaeil Mirzaee via Daniel Templeton)

(cherry picked from commit 948afe2c92977d6924a62c1a5a1ffca315cd3e68)
 2017-03-20 14:32:01 -07:00
Varun Saxena e2cabb4d21 YARN-6353. Clean up OrderingPolicy javadoc (Daniel Templeton via Varun Saxena) 
(cherry picked from commit 35034653d0)
 2017-03-20 23:51:15 +05:30
Jason Lowe 75e6b7428e YARN-6217. TestLocalCacheDirectoryManager test timeout is too aggressive. Contributed by Miklos Szegedi 
(cherry picked from commit 4a8e304502)
 2017-03-17 14:07:48 -05:00
Jason Lowe a16ba4296e YARN-4051. ContainerKillEvent lost when container is still recovering and application finishes. Contributed by sandflee 2017-03-16 09:31:20 -05:00
Jian He 9f9ccb2784 YARN-6332. Make RegistrySecurity use short user names for ZK ACLs. Contributed by Billie Rinaldi 2017-03-16 13:00:44 +08:00
Junping Du 8b31ff6dba YARN-6294. ATS client should better handle Socket closed case. Contributed by Li Lu. 2017-03-15 11:49:51 -07:00
Sunil G caaa6fa96b YARN-6328. Fix a spelling mistake in CapacityScheduler. Contributed by Jin Yibo. This closes #202 
(cherry picked from commit 2d0e24eb01)
 2017-03-15 14:58:20 +05:30
Ray Chiang fa59f4e490 YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG log. (Yufei Gu via rchiang) 2017-03-14 14:45:13 -07:00
Junping Du 0e7879052a YARN-6313. YARN logs cli should provide logs for a completed container even when application is still running. Contributed by Xuan Gong. 
(cherry picked from commit b88f5e0f7858d1d89b79dfd325b767c34416052d)
 2017-03-14 12:58:41 -07:00
Junping Du f254002f1d YARN-6314. Potential infinite redirection on YARN log redirection web service. Contributed by Xuan Gong. 
(cherry picked from commit 5a9dda796f0e73060ada794ad5752cc6a237ab2e)
 2017-03-14 02:58:44 -07:00
Jason Lowe ef95e74eb1 YARN-6321. TestResources test timeouts are too aggressive. Contributed by Eric Badger 
(cherry picked from commit 9649c27864)
 2017-03-10 13:07:20 -06:00
Jian He 8c44a9db21 HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand 2017-03-10 09:26:19 -08:00
Jason Lowe 1a0358b59a YARN-6310. OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptions. Contributed by Haibo Chen 
(cherry picked from commit deb9f56946)
 2017-03-10 11:09:04 -06:00
Karthik Kambatla 4218671f3c YARN-6264. AM not launched when a single vcore is available on the cluster. (Yufei Gu via kasha) 
(cherry picked from commit a96afae125)
 2017-03-09 23:16:19 -08:00
Daniel Templeton c60cd88ad1 YARN-1047. Expose # of pre-emptions as a queue counter (Contributed by Karthik Kambatla via Daniel Templeton) 2017-03-09 18:18:03 -08:00
Daniel Templeton 6774f9c3c2 YARN-6300. NULL_UPDATE_REQUESTS is redundant in TestFairScheduler (Contributed by Yuanbo Liu via Daniel Templeton) 
(cherry picked from commit 8bc8804be1b96f8a4fc33b5d24dd7f7af93a437e)
 2017-03-09 17:53:24 -08:00
Sidharta S 3b6e5ef917 YARN-5669. Add support for docker pull command (Contribtued by luhuichun) 2017-03-09 16:23:21 -08:00
Jason Lowe 4a1187238a YARN-6165. Intra-queue preemption occurs even when preemption is turned off for a specific queue. Contributed by Eric Payne 
(cherry picked from commit d7762a5511)
 2017-03-08 16:47:31 -06:00
Jian He ea8d53de6a Revert "HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand" 
This reverts commit 36eda5af75.
 2017-03-08 13:20:45 -08:00
Jian He 36eda5af75 HADOOP-14062. ApplicationMasterProtocolPBClientImpl.allocate fails with EOFException when RPC privacy is enabled. Contributed by Steven Rand 2017-03-08 10:49:07 -08:00
Robert Kanter 11d4cdc3a6 YARN-6297. TestAppLogAggregatorImp.verifyFilesUploaded() should check # of filed uploaded with that of files expected (haibochen via rkanter) 
(cherry picked from commit 287ba4ffa6)
 2017-03-08 10:45:59 -08:00
Sunil G ec4c2d42fe YARN-6207. Move application across queues should handle delayed event processing. Contributed by Bibin A Chundatt. 
(cherry picked from commit 1eb8186703)
 2017-03-08 12:13:14 +05:30
Daniel Templeton 83fbb051bb YARN-6287. RMCriticalThreadUncaughtExceptionHandler.rmContext should be final (Contributed by Corey Barker via Daniel Templeton) 
(cherry picked from commit e0c239cdbd)
 2017-03-07 12:01:38 -08:00
Jian He 712434e744 YARN-6270. WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting. Contributed by Xuan Gong 2017-03-03 15:24:23 -08:00
Junping Du a58dfcb421 YARN-6271. yarn rmadin -getGroups returns information from standby RM. Contributed by Jian He. 
(cherry picked from commit 8db7a8c3ae)
 2017-03-03 14:40:32 -08:00
Ray Chiang 303ee13e3c YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via rchiang) 
(cherry picked from commit 30b91ff9540e35195af834d1bd5983114a556c6d)
 2017-03-03 12:57:17 -08:00
Jason Lowe 5eca427da2 YARN-6263. NMTokenSecretManagerInRM.createAndGetNMToken is not thread safe. Contributed by Haibo Chen 
(cherry picked from commit e58fc76030)
 2017-03-03 08:24:05 -06:00
Jian He c7ddf95d4a YARN-6153. KeepContainer does not work when AM retry window is set. Contributed by kyungwan nam 
(cherry picked from commit 235203dffd)
 2017-02-28 13:26:55 -08:00
Xuan 95bd3c3d55 YARN-6189: Improve application status log message when RM restarted when 
app is in NEW state. Contributed by Junping Du

(cherry picked from commit e0bb867c3f)
 2017-02-28 11:06:07 -08:00
Gera Shegalov df35ba81fe YARN-1728. Workaround guice3x-undecoded pathInfo in YARN WebApp. (Yuanbo Liu via gera) 2017-02-28 09:46:57 -08:00
Karthik Kambatla 58eefc71c7 YARN-6231. FairSchedulerTestBase helper methods should call scheduler.update to avoid flakiness. (kasha) 
(cherry picked from commit f187d63816)
 2017-02-27 22:16:32 -08:00
Naganarasimha 1eec911cd9 YARN-5703. ReservationAgents are not correctly configured. Contributed by Manikandan R. 
(cherry picked from commit 5f5b031d1f)
 2017-02-27 20:42:10 +05:30
Sunil G 68b08e96a0 YARN-6123. [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed. Contributed by Wangda Tan. 2017-02-27 17:54:47 +05:30
Karthik Kambatla c24325bbcb YARN-6172. FSLeafQueue demand update needs to be atomic. (Miklos Szegedi via kasha) 
(cherry picked from commit fbfe86deea)
 2017-02-26 20:37:18 -08:00
Karthik Kambatla a95d3e1fe7 YARN-6215. FairScheduler preemption and update should not run concurrently. (Tao Jie via kasha) 
(cherry picked from commit 815d53506f)
 2017-02-26 20:31:03 -08:00
Xuan 19c5b3d622 YARN-6228: EntityGroupFSTimelineStore should allow configurable cache 
stores. Contributed by Li Lu

(cherry picked from commit 53d372a255)
 2017-02-24 11:00:16 -08:00
Wangda Tan 650ff95e00 YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 
(cherry picked from commit 0a55bd841e)
 2017-02-23 17:22:25 -08:00
Karthik Kambatla 40bc9e7ddb YARN-6222. TestFairScheduler.testReservationMetrics is flaky. (Yufei Gu via kasha) 
(cherry picked from commit 694e680d20)
 2017-02-23 15:30:29 -08:00
Wangda Tan a82480c7d4 YARN-6143. Fix incompatible issue caused by YARN-3583. (Sunil G via wangda) 2017-02-23 13:59:20 -08:00
Sunil G 801357726d YARN-6211. Synchronization improvement for moveApplicationAcrossQueues and updateApplicationPriority. Contributed by Bibin A Chundatt. 
(cherry picked from commit a207aa9930)
 2017-02-23 14:25:14 +05:30
Karthik Kambatla 37edbd35f1 YARN-6194. Cluster capacity in SchedulingPolicy is updated only on allocation file reload. (Yufei Gu via kasha) 
(cherry picked from commit b10e962224)
 2017-02-22 15:59:11 -08:00
Karthik Kambatla ca7a6a7365 YARN-6210. FairScheduler: Node reservations can interfere with preemption. (kasha) 
(cherry picked from commit 718ad9f6ee)
 2017-02-22 15:52:44 -08:00
Daniel Templeton 93219b6454 YARN-6125. The application attempt's diagnostic message should have a maximum size 
(Contributed by Andras Piros via Daniel Templeton)

(cherry picked from commit c7a36e6130)
 2017-02-17 14:38:06 -08:00
Karthik Kambatla 5575328bb6 YARN-6193. FairScheduler might not trigger preemption when using DRF. (kasha) 
(cherry picked from commit dbbfcf74ab)
 2017-02-17 14:07:47 -08:00
Daniel Templeton 10fd5627bb YARN-6188. Fix OOM issue with decommissioningNodesWatcher in the case of clusters with 
large number of nodes (Contributed by Ajay Jadhav via Daniel Templeton)

(cherry picked from commit 585168cc1bdb87398d4f388f3eeccd1c71114982)
 2017-02-17 13:14:37 -08:00
Karthik Kambatla afc8124ff6 YARN-6171. ConcurrentModificationException on FSAppAttempt.containersToPreempt. (Miklos Szegedi via kasha) 
(cherry picked from commit a77f432449)
 2017-02-16 15:03:35 -08:00
Li Lu 8fc67e5973 YARN-6177. Yarn client should exit with an informative error message if an incompatible Jersey library is used at client. Contributed by Weiwei Yang. 
(cherry picked from commit 5d339c46f5)
 2017-02-16 13:46:30 -08:00
Rohith Sharma K S f07b52dc5a YARN-6200. Reverting since the same functionality achieved by YARN-1623. 
Revert "YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)"

This reverts commit a37b3694ea.
 2017-02-16 17:58:38 +05:30
Karthik Kambatla 9fecd5063f YARN-5798. Set UncaughtExceptionHandler for all FairScheduler threads. (Yufei Gu via kasha) 
(cherry picked from commit 74dd142250)
 2017-02-16 00:03:26 -08:00
Karthik Kambatla fefac1276a YARN-4212. FairScheduler: Can't create a DRF queue under a FAIR policy queue. (Yufei Gu via kasha) 
(cherry picked from commit 11be3f70e0)
 2017-02-15 23:51:51 -08:00
Karthik Kambatla 46b6c95e0a YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha) 
(cherry picked from commit 6c25dbcdc0)
 2017-02-15 23:23:18 -08:00
Junping Du 323782b389 Addendum patch for YARN-6174 - remove deleted file. 
(cherry picked from commit 0fc6f38379)
 2017-02-15 09:09:27 -08:00
Junping Du 2aa6f317f6 YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong. 
(cherry picked from commit ce2d5bfa5f)
 2017-02-15 09:08:54 -08:00
Varun Saxena 9cab25a63d YARN-6156. AM blacklisting to consider node label partition (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit b7613e0f40)
 2017-02-15 14:58:39 +05:30
Masatake Iwasaki cba7e2c346 YARN-4753. Use doxia macro to generate in-page TOC of YARN site documentation. (iwasakims) 2017-02-15 13:37:25 +09:00
Karthik Kambatla 58acb676a2 YARN-6061. Addendum. Remove extraneous change. 
(cherry picked from commit 353a9b2d91)
 2017-02-14 15:20:11 -08:00
Karthik Kambatla 37921b3fef YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha) 
(cherry picked from commit 652679aa8a)
 2017-02-14 13:40:06 -08:00
Karthik Kambatla 30ff5bff1a YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha) 
(cherry picked from commit 646c6d6509)
 2017-02-13 11:27:02 -08:00
Junping Du 205fc1f10c YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong. 
(cherry picked from commit 464ff479ce)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
 2017-02-13 06:16:04 -08:00
Daniel Templeton 7c3578ca17 YARN-6166. Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run (Contributed by Grant Whiteheart via Daniel Templeton) 
(cherry picked from commit 1238fa71f8)
 2017-02-10 09:48:50 -08:00
Sunil G 7b2b6caf12 YARN-6118. Add javadoc for Resources.isNone. Contributed by Andres Perez. 
(cherry picked from commit 3a0a0a485a)
 2017-02-10 15:54:59 +05:30
Karthik Kambatla e259a818ad YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha) 
(cherry picked from commit 08f93978f3)
 2017-02-09 14:45:21 -08:00
Karthik Kambatla 41361ec39e YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:12:21 -08:00
Jason Lowe e330c3806c YARN-6137. Yarn client implicitly invoke ATS client which accesses HDFS. Contributed by Li Lu 
(cherry picked from commit 37b4acf7ce)
 2017-02-08 15:00:50 -06:00
Junping Du 33e7c3044f YARN-6145. Improve log message on fail over. Contributed by Jian He. 
(cherry picked from commit eec52e158b)
 2017-02-08 02:15:11 -08:00
Sidharta S 127ed38d13 YARN-5258. Document Use of Docker with LinuxContainerExecutor (Contributed by Daniel Templeton) 2017-02-07 16:35:56 -08:00
Junping Du ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong. 
(cherry picked from commit 9dbfab1284)
 2017-02-07 02:14:09 -08:00
Junping Du b7d59e790b HADOOP-12097. Allow port range to be specified while starting webapp. Contributed by Varun Saxena. 
(cherry picked from commit cce35c3815)

Conflicts:

	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
 2017-02-05 20:13:49 -08:00
Junping Du c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Daniel Templeton be14bda5c0 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton) 
(cherry picked from commit 3e06475307)
 2017-01-31 16:26:41 -08:00
Daniel Templeton ccf33bc36c YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 
(cherry picked from commit 87852b6ef4)
 2017-01-30 21:46:38 -08:00
Ray Chiang 9f0b813350 YARN-6106. Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei Gu via rchiang) 
(cherry picked from commit 3f970ac363dc2faf4f54180ee5b64d0c60807097)
 2017-01-30 13:42:11 -08:00
Karthik Kambatla c9824c5ff6 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha) 
(cherry picked from commit 11e44bdda7)
 2017-01-30 13:39:35 -08:00
Jason Lowe 4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Junping Du 660f4d8631 YARN-6126. Obtaining app logs for Running application fails with json parse error. Contributed by Xuan Gong. 
(cherry picked from commit 1a16431bd0)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
 2017-01-27 06:33:44 -08:00
Karthik Kambatla 1b1a25cca6 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha) 
(cherry picked from commit 4d3f73acc0)
 2017-01-26 14:11:50 -08:00
Daniel Templeton c4eeac02de YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit f85b74ccf9)
 2017-01-26 14:08:46 -08:00
Daniel Templeton 443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton) 
(cherry picked from commit b3befc021b)
 2017-01-26 14:07:46 -08:00
Jason Lowe b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger" 
This reverts commit 45407acaed.
 2017-01-26 14:42:22 +00:00
Sangjin Lee b799ea7641 YARN-3637. Handle localization sym-linking correctly at the YARN level. Contributed by Chris Trezzo. 
(cherry picked from commit 425a7e5028)
 2017-01-25 15:52:05 -08:00
Sangjin Lee c4d3b28c2d YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo. 
(cherry picked from commit dc6ec97048)
 2017-01-25 15:36:07 -08:00
Jason Lowe 45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 
(cherry picked from commit 9e19f758c1)
 2017-01-25 21:54:17 +00:00
Karthik Kambatla 73497f08fc YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 
(cherry picked from commit abedb8a9d8)
 2017-01-25 12:19:27 -08:00
Karthik Kambatla 71026cc3f6 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 
(cherry picked from commit e224c96234)
 2017-01-25 12:19:20 -08:00
Karthik Kambatla 0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Junping Du 8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Wangda Tan 1309accd68 YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-24 14:44:42 -08:00
Jason Lowe 1672a06135 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:19:02 +00:00
Rohith Sharma K S 26c4cfb653 YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 
(cherry picked from commit 0101267d9d)
 2017-01-24 11:01:52 +05:30
Jason Lowe 94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 
(cherry picked from commit 69fa81679f)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
 2017-01-23 17:23:32 +00:00
Sunil G f8e49bb343 YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 
(cherry picked from commit 3fa0d540df)
 2017-01-23 12:24:39 +05:30
Jian He d374087670 YARN-6104. RegistrySecurity overrides zookeeper sasl system properties. Contributed by Billie Rinaldi 2017-01-19 10:19:24 -08:00
Chris Nauroth 5251de00fa YARN-5456. container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. Contributed by Allen Wittenauer. 
(cherry picked from commit b913677365)
 2017-01-19 20:48:27 +05:30
Chris Nauroth 198bd84b33 YARN-5121. fix some container-executor portability issues. Contributed by Allen Wittenauer. 
(cherry picked from commit ef501b1a0b)
 2017-01-19 20:48:27 +05:30
Arun Suresh 3c6ec20955 YARN-6110. Fix opportunistic containers documentation. (Akira Ajisaka via asuresh) 
(cherry picked from commit 7e8d32147c)
 2017-01-19 06:16:30 -08:00
Wangda Tan 8d6fa4569b YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 
(cherry picked from commit 72054a817d)
 2017-01-18 13:35:30 -08:00
Subru Krishnan fb639aed90 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru). 
(cherry picked from commit 4d1f3d9020)
 2017-01-17 14:51:27 -08:00
bibinchundatt 580275628b YARN-6057. yarn.scheduler.minimum-allocation-* descriptions are incorrect about behavior when a request is out of bounds. Contributed by Julia Sommer. 
(cherry picked from commit a9d946d4df)
 2017-01-17 14:18:41 +05:30
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit e407449ddb)
 2017-01-17 10:22:29 +09:00
Junping Du 861e275646 YARN-6011. Add a new web service to list the files on a container in AHSWebService. Contributed by Xuan Gong. 
(cherry picked from commit cf695577aa)
 2017-01-16 16:41:16 -08:00
Sunil G f166bb8f09 YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 
(cherry picked from commit d3170f9eba)
 2017-01-13 18:30:31 +05:30
Naganarasimha 4bd05840fc YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 
(cherry picked from commit a6b06f7179)
 2017-01-12 06:42:18 +05:30
Daniel Templeton 718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton) 
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
 2017-01-11 15:36:18 -08:00
Jason Lowe cc09b1a49a YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du 
(cherry picked from commit 357eab9566)
 2017-01-11 17:15:14 +00:00
Junping Du ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 
(cherry picked from commit 4db119b7b5)
 2017-01-10 18:22:45 -08:00
Wangda Tan c00a468b0b YARN-6022. Revert changes of AbstractResourceRequest. (Contributed by Wangda Tan / Daniel Templeton) 2017-01-10 10:17:33 -08:00
Naganarasimha a7cdd39234 YARN-6054. TimelineServer fails to start when some LevelDb state files are missing. Contributed by Ravi Prakash. 
(cherry picked from commit 4c431a6940)
 2017-01-10 15:55:46 +05:30
Naganarasimha ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Sunil G 3190a4ba44 YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang. 
(cherry picked from commit 9594c35dcb)
 2017-01-10 14:10:25 +05:30
Daniel Templeton 74bf9c6e4b YARN-6073. Misuse of format specifier in Preconditions.checkArgument (Contributed by Yuanbo Liu via Daniel Templeton) 
(cherry picked from commit 6332a318bc)
 2017-01-09 22:14:46 -08:00
Junping Du b600577095 YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe. 
(cherry picked from commit 945db55f2e)
 2017-01-09 18:15:21 -08:00
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804)
 2017-01-09 08:42:38 -08:00
Varun Saxena 4fab250fc0 YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena) 
(cherry picked from commit f59e36b4ce)
 2017-01-09 11:31:39 +05:30
Arun Suresh 4985217de4 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh) 2017-01-08 08:27:10 -08:00
Junping Du f615c1fbbc Addendum patch to YARN-3866: fix incompatible API change. Contributed by Wangda Tan. 2017-01-07 19:58:42 -08:00
Jian He ba5d482366 YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S 
(cherry picked from commit 020316458d)
 2017-01-06 18:27:36 -08:00
Arun Suresh 81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh) 
(cherry picked from commit ac1e5d4f77)
 2017-01-06 11:25:49 -08:00
Arun Suresh 2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2273a74c1f)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
 2017-01-06 11:15:10 -08:00
Arun Suresh 3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit b0aace21b1)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
 2017-01-06 11:15:09 -08:00
Arun Suresh 44774eb21c YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. (Bibin A Chundatt via asuresh) 
(cherry picked from commit 005850b28f)
(cherry picked from commit cbff10b4147f98a89b393519b17e16385294af07)
 2017-01-06 11:15:09 -08:00
Arun Suresh 368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 
(cherry picked from commit 3219b7b4ac)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
 2017-01-06 11:15:09 -08:00
Arun Suresh 047772f15f YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 283fa33feb)
(cherry picked from commit 1c4cc88a754ac9f557cdc8c859b8aadec19a5067)
 2017-01-06 11:15:09 -08:00
Arun Suresh 0e60c7cd33 YARN-5377. Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit f38a6d03a1)
(cherry picked from commit 83ccab8a653a4ef943276651c9604bd434675980)
 2017-01-06 11:15:09 -08:00
Arun Suresh 7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d4)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Arun Suresh 28f3bb38bf YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh) 
(cherry picked from commit aa3cab1eb2)
(cherry picked from commit ce10e98826884245e733431297d0275177965f17)
 2017-01-06 11:15:09 -08:00
Arun Suresh f9ea753414 YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 10be45986c)
(cherry picked from commit e3baa0988b4dfb592afe426ffeba3c7091b271c5)
 2017-01-06 11:15:09 -08:00
Sunil G 65e7ae5dcf YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2017-01-06 21:30:52 +05:30
Rohith Sharma K S 3207762192 YARN-5988. RM unable to start in secure setup. Contributed by Ajith S. 
(cherry picked from commit e49e0a6e37)
 2017-01-04 12:10:05 +05:30
Junping Du dc2e44e8f8 YARN-5923. Unable to access logs for a running application if YARN_ACL_ENABLE is enabled. Contributed by Xuan Gong. 
(cherry picked from commit 8fadd69047)
 2017-01-03 15:19:36 -08:00
Wangda Tan 23f6a33156 YARN-6025. Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
(cherry picked from commit 554e6656bc)
 2017-01-03 14:52:31 -08:00
Wangda Tan cb64e8eb19 Revert "Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)" because forgot to add JIRA Number 
This reverts commit 554e6656bc.
 2017-01-03 14:51:59 -08:00
Wangda Tan 554e6656bc Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
 2017-01-03 14:50:19 -08:00
Daniel Templeton 8cb7aa2b52 YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton) 
(cherry picked from commit 165d01a73e)
 2016-12-30 08:56:03 -08:00
Robert Kanter f9b48206dd YARN-4882. Change the log level to DEBUG for recovering completed applications (templedf via rkanter) 
(cherry picked from commit f216276d21)
 2016-12-28 15:29:06 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Sunil G b3231b480d YARN-6024. Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max. Contributed by Wangda Tan. 
(cherry picked from commit f6715b26b6)
 2016-12-28 16:00:52 +05:30
Rohith Sharma K S cb40e3cffa YARN-6001. Improve moveApplicationQueues command line. Contributed by Sunil G. 
(cherry picked from commit e297be74ed)
 2016-12-28 12:17:38 +05:30
Wangda Tan 42f571728c YARN-5756. Add state-machine implementation for scheduler queues. (Xuan Gong via wangda) 
(cherry picked from commit 0840b4329b)
 2016-12-27 21:19:10 -08:00
Robert Kanter 88597991b0 YARN-5257. Fix unreleased resources and null dereferences (yufeigu via rkanter) 
(cherry picked from commit 9262797e86)
 2016-12-27 14:14:41 -08:00
Robert Kanter 8fbd36c60f YARN-5962. Spelling errors in logging and exceptions for resource manager code (gsohn via rkanter) 
(cherry picked from commit 1bbd023275)
 2016-12-27 13:35:55 -08:00
Karthik Kambatla 10a154cedd YARN-5969. FairShareComparator: Cache value of getResourceUsage for better performance. (zhangshilong via kasha) 
(cherry picked from commit c3973e7080)
 2016-12-27 12:30:50 -08:00
Daniel Templeton 28b605dd1a HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton) 
(cherry picked from commit d432173737a532b7106682116013b751ba162ff0)
 2016-12-27 07:19:37 -08:00
Naganarasimha 4a9f965314 YARN-6026. A couple of spelling errors in the docs. Contributed by Grant Sohn. 2016-12-26 06:50:06 +05:30
Akira Ajisaka 522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539ea)
 2016-12-23 04:06:53 +09:00
Varun Saxena 3ba589cd1e YARN-5903. Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method (Haibo Chen via Varun Saxena) 
(cherry picked from commit 38e66d4d64)
 2016-12-22 23:11:18 +05:30
Junping Du cc42dbb703 YARN-4990. Re-direction of a particular log file within in a container in NM UI does not redirect properly. Contributed by Xuan Gong. 
(cherry picked from commit 736f54b727)
 2016-12-21 14:16:19 -08:00
Sunil G 238863c11b YARN-5650. Render Application Timeout value in web UI. Contributed by Akhil PB. 2016-12-20 11:03:57 +05:30
Naganarasimha e9ce2df1f2 YARN-5524. Yarn live log aggregation does not throw if command line arg is wrong. Contributed by Xuan Gong. 
(cherry picked from commit 358341398a)
 2016-12-20 06:45:52 +05:30
Xuan 236dbe3485 YARN-5999. AMRMClientAsync will stop if any exceptions thrown on allocate call. Contributed by Jian He 2016-12-14 15:05:42 -08:00
Sunil G 23bd68a4a4 YARN-5970. Validate application update timeout request parameters. Contributed by Rohith Sharma K S. 2016-12-12 11:59:14 +05:30
Jian He 71fdca4600 YARN-5709. Cleanup leader election configs and pluggability. Contribtued by Karthik Kambatla 
(cherry picked from commit b817c565c8be1d4a682d119bfac6f43ee09e87f0)
 2016-12-09 16:59:52 -08:00
Robert Kanter 9cba7c6d71 YARN-5963. Spelling errors in logging and exceptions for node manager, client, web-proxy, common, and app history code (gsohn via rkanter) 
(cherry picked from commit 72fe546841)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
 2016-12-07 15:09:37 -08:00
Sunil a636a87270 YARN-5965. Retrospect ApplicationReport#getApplicationTimeouts. Contributed by Rohith Sharma K S 
(cherry picked from commit ab923a53fc)
 2016-12-08 00:34:45 +05:30
Daniel Templeton d8b4961515 YARN-5136. Error in handling event type APP_ATTEMPT_REMOVED to the scheduler 
(Contributed by Wilfred Spiegelenburg via Daniel Templeton)
 2016-12-07 11:01:09 -08:00
Rohith Sharma K S 602c998443 YARN-5932. Retrospect moveApplicationToQueue in align with YARN-5611. Contributed by Sunil G. 2016-12-07 22:45:25 +05:30
Junping Du 3350d0c080 YARN-5184. Fix up incompatible changes introduced on ContainerStatus and NodeReport. Contributed by Sangjin Lee. 
(cherry picked from commit a7288da595)
 2016-12-06 14:37:46 -08:00
Karthik Kambatla 8773cd5f76 HADOOP-13859. TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties. (Haibo Chen via kasha) 
(cherry picked from commit c8d0a049b0)
 2016-12-06 14:16:38 -08:00
Naganarasimha 54c5880cf7 YARN-5921. Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState. Contributed by Varun Saxena 
(cherry picked from commit f3b8ff54ab)
 2016-12-06 11:01:14 +05:30
Jian He 0f6fbfc0db YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. Contributed by Akira Ajisaka & Wangda Tan 
(cherry picked from commit 43ebff2e35)
 2016-12-05 11:40:26 -08:00
Jian He d58fca0102 YARN-5746. The state of the parentQueue and its childQueues should be synchronized. Contributed by Xuan Gong 2016-12-02 16:18:39 -08:00
Daniel Templeton 1216bad0f7 YARN-5929. Missing scheduling policy in the FS queue metric. (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 9b076327eac9da244f94e226816465e7d17fda54)
 2016-12-02 14:02:35 -08:00
Daniel Templeton eb301bb3ec YARN-5901. Fix race condition in TestGetGroups beforeclass setup() (Contributed by Haibo Chen via Daniel Templeton) 2016-12-02 12:05:29 -08:00
Jason Lowe 6b17d449bf YARN-5915. ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write. Contributed by Atul Sikaria 2016-12-02 16:58:43 +00:00
Robert Kanter 99b046f8a9 MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job overview page in JHS (haibochen via rkanter) 
(cherry picked from commit c87b3a448a)
 2016-12-01 17:30:02 -08:00
Li Lu 8ffe86f780 YARN-5761. Separate QueueManager from Scheduler. (Xuan Gong via gtcarrera9) 
(cherry picked from commit 69fb70c31a)
 2016-11-30 14:02:31 -08:00
Daniel Templeton 3013b028ba YARN-5942. "Overridden" is misspelled as "overriden" in FairScheduler.md 
(Contributed by Heather Sutherland via Daniel Templeton)

(cherry picked from commit 57194889fc39a36dc0697b29eafc4ac07ee42d00)
 2016-11-30 11:24:44 -08:00
Daniel Templeton c22ceb0a9b YARN-5851. TestContainerManagerSecurity testContainerManager[1] failed (Contributed by Haibo Chen via Daniel Templeton) 
(cherry picked from commit dd9a96cb17)
 2016-11-29 14:23:42 -08:00
Daniel Templeton 803aba03f2 YARN-5890. FairScheduler should log information about AM-resource-usage and max-AM-share for queues 
(Contributed by Yufei Gu via Daniel Templeton)
 2016-11-29 14:14:43 -08:00
Daniel Templeton b8bebb8607 YARN-5774. MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
if set yarn.scheduler.minimum-allocation-mb to 0. (Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 25f9872be6)
 2016-11-29 10:12:10 -08:00
Sunil c28dc5e5a4 YARN-4206. Add Application timeouts in Application report and CLI. Contributed by Rohith Sharma K S. 
(cherry picked from commit eb0a483ed0)
 2016-11-24 18:27:50 +05:30
Sunil 7b676c5632 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-24 17:37:36 +05:30
Rohith Sharma K S 6b47a7fb8e YARN-5920. Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang. Contributed by Varun Saxena. 
(cherry picked from commit e15c20edba)
 2016-11-24 12:19:19 +05:30
Rohith Sharma K S 87bb3c51d6 Reverting for fixing compilation errors in branch-2. 
Revert "YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S"

This reverts commit 4379658042.
 2016-11-24 09:18:07 +05:30
Jian He 4379658042 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-23 16:27:12 -08:00
Rohith Sharma K S 2d068f1793 YARN-5865. Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611. Contributed by Sunil G. 2016-11-23 23:29:39 +05:30
Naganarasimha 2d94f938c5 YARN-4330. MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages. Contributed by Varun Saxena 
(cherry picked from commit 3541ed8068)
 2016-11-23 14:14:56 +05:30
Naganarasimha b96ed43897 YARN-5911. DrainDispatcher does not drain all events on stop even if setDrainEventsOnStop is true. Contributed by Varun Saxena. 
(cherry picked from commit 4667564162)
 2016-11-23 09:36:43 +05:30
Daniel Templeton 06f5d27bd6 YARN-5722. FairScheduler hides group resolution exceptions when assigning queue (Contributed by Wilfred Spiegelenburg via Daniel Templeton) 
(cherry picked from commit 83cc7263af)
 2016-11-22 11:17:39 -08:00
Jason Lowe 38024eb8f7 YARN-5859. TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails. Contributed by Eric Badger 
(cherry picked from commit 009452bb6d)
 2016-11-21 16:37:59 +00:00
Wei-Chiu Chuang c5a3ec493b YARN-5271. ATS client doesn't work with Jersey 2 on the classpath. Contributed by Weiwei Yang. 
(cherry picked from commit 09520cb439)
 2016-11-17 22:23:58 -06:00
Subru Krishnan e5a5a5a618 YARN-5904. Reduce the number of default server threads for AMRMProxyService. 
(cherry picked from commit 140b9939da)
 2016-11-17 18:27:17 -08:00
Junping Du a8ad6b6171 YARN-3538. TimelineWebService doesn't catch runtime exception. Contributed by Steve Loughran and Tsuyoshi Ozawa. 
(cherry picked from commit f05a9ceb4a)
 2016-11-17 16:56:21 -08:00
Rohith Sharma K S bbe993238d YARN-5870. Expose getApplications API in YarnClient with GetApplicationsRequest parameter. Contributed by Jian He. 
(cherry picked from commit ce2847e794)
 2016-11-17 08:54:33 +05:30
Jason Lowe 46b7d6233c YARN-5836. Malicious AM can kill containers of other apps running in any node its containers are running. Contributed by Botong Huang 
(cherry picked from commit 59bfcbf357)
 2016-11-16 22:25:15 +00:00
Varun Saxena 6cb1fda99d YARN-5891. yarn rmadmin -help contains a misspelled ResourceManager (Grant Sohn via Varun Saxena) 
(cherry picked from commit 6a11877ab5)
 2016-11-17 03:28:08 +05:30
Varun Saxena 2893316dd0 YARN-5843. Incorrect documentation for timeline service entityType/events REST end points (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit c8bc7a8475)
 2016-11-17 00:52:08 +05:30
Rohith Sharma K S 74ac78b3c0 YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures. Contributed by sandflee. 
(cherry picked from commit d65603517e)
 2016-11-16 15:17:46 +05:30
Xiao Chen 3497221fcc YARN-5875. TestTokenClientRMService#testTokenRenewalWrongUser fails. Contributed by Gergely Novák. 
(cherry picked from commit d5c396ccb361e1a0cf4307ce1d1b37fae0eec250)
 2016-11-15 13:58:02 -08:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b644)
 2016-11-15 11:36:08 -08:00
Naganarasimha 260f3a9dc9 YARN-4355. NPE while processing localizer heartbeat. Contributed by Varun Saxena & Jonathan Hung. 
(cherry picked from commit 7ffb9943b8)
 2016-11-15 15:44:25 +05:30
Naganarasimha 7e9a6b653e Reverting the patch due to the issue raised in YARN-5765 
Revert "YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang."

This reverts commit 93b768d0d1.
 2016-11-15 11:05:03 +05:30
Rohith Sharma K S 528ef2407c YARN-5874. RM -format-state-store and -remove-application-from-state-store commands fail with NPE. Contributed by Varun Saxena. 
(cherry picked from commit b7070f3308)
 2016-11-15 11:03:38 +05:30