Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,610 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3449 Commits

Author SHA1 Message Date
Jason Lowe c930aa3ff9 YARN-5816. TestDelegationTokenRenewer#testCancelWithMultipleAppSubmissions is still flakey. Contributed by Robert Kanter 2017-08-29 10:02:44 -05:00
Junping Du f252e1fe28 YARN-7076. yarn application -list -appTypes is not working. Contributed by Jian He. 
(cherry picked from commit 312b1fd9da)
 2017-08-28 18:27:38 -07:00
Wangda Tan 82e1cd08eb YARN-7112. TestAMRMProxy is failing with invalid request. (Jason Lowe via wangda) 
Change-Id: I432f4523921666ac218fcf9936f826a0d8d635e0
(cherry picked from commit 51881a863c)
 2017-08-28 11:38:05 -07:00
Yufei Gu 221c1bdf7e YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu) 
(cherry picked from commit d5bc958279)
 2017-08-28 10:11:01 -07:00
Sunil G 80570737a8 YARN-7051. Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin. Contributed by Eric Payne. 
(cherry picked from commit 02599bda04)
 2017-08-28 12:27:29 +05:30
Li Lu b7f319c53c YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9) 
(cherry picked from commit 3bee7046c5)
 2017-08-25 22:45:20 -07:00
Eric Payne 1617744801 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe. 
(cherry picked from commit e864f81471)
 2017-08-25 15:52:29 -05:00
Jason Lowe ea5bb5898f YARN-7052. RM SchedulingMonitor gives no indication why the spawned thread crashed. Contributed by Eric Payne 
(cherry picked from commit 39a9dc8e4a)
 2017-08-25 15:32:29 -05:00
Jason Lowe 82957ce0d0 YARN-6640. AM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit 3a4e861169)
 2017-08-25 09:20:50 -05:00
Yufei Gu 389b659c94 YARN-7049. FSAppAttempt preemption related fields have confusing names. (Contributed by Karthik Kambatla via Yufei Gu) 
(cherry picked from commit 9e2699ac2c)
 2017-08-24 17:54:00 -07:00
Junping Du e12358c5ce YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 
(cherry picked from commit c2cb7ea1ef)
 2017-08-24 15:27:40 -07:00
Akira Ajisaka bf903396a0
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-25 03:28:02 +09:00
bibinchundatt cc50ca0722 YARN-7074. Fix NM state store update comment. Contributed by Botong Huang 
(cherry picked from commit de0cba700b)
 2017-08-24 16:40:18 +05:30
Junping Du c98917f798 YARN-7090. testRMRestartAfterNodeLabelDisabled get failed when CapacityScheduler is configured. Contributed by Wangda Tan. 
(cherry picked from commit 652dd434d9)
 2017-08-23 18:07:07 -07:00
Wangda Tan 245b49332d YARN-6251. Do async container release to prevent deadlock during container updates. (Arun Suresh via wangda) 
Change-Id: I6c67d20c5dd4d22752830ebf0ed2340824976ecb
(cherry picked from commit f49843a988)
 2017-08-23 10:35:37 -07:00
Subru Krishnan a45ffdcddc YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru). 
(cherry picked from commit 4249172e14)
 2017-08-22 19:21:35 -07:00
Ray Chiang 2299c8dffa YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 
(cherry picked from commit f5f14a2ad6)
 2017-08-22 15:24:02 -07:00
Jason Lowe b15a393440 YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-22 16:28:08 -05:00
Jason Lowe a7893033fe YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. Contributed by Daryn Sharp 
(cherry picked from commit 657dd59cc8)
 2017-08-22 13:17:31 -05:00
Jason Lowe fc7df25a11 YARN-2416. InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time. Contributed by Jonathan Eagles 
(cherry picked from commit 3efcd51c3b)
 2017-08-22 12:59:47 -05:00
Arun Suresh e0ba8a9ed8 YARN-6979. [Addendum patch] Fixed classname and added javadocs. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 7a82d7bcea)
 2017-08-20 10:28:42 -07:00
Arun Suresh f8f193c8e8 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 8410d862d3)
 2017-08-20 08:31:17 -07:00
Yufei Gu 4bd417a056 YARN-6969. Clean up unused code in class FairSchedulerQueueInfo. (Larry Lo via Yufei Gu) 
(cherry picked from commit 8991f0baec)
 2017-08-18 14:45:13 -07:00
Daniel Templeton 78f9dca2cb YARN-6952. Enable scheduling monitor in FS. (Yufei Gu via Daniel Templeton) 2017-08-18 14:40:30 -07:00
bibinchundatt ecf1a029bb YARN-7007. NPE in RM while using YarnClient.getApplications(). Contributed by Lingfeng Su. 
(cherry picked from commit e05fa3451d)
 2017-08-18 20:31:26 +05:30
Sunil G 5511cea86b YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad. 
(cherry picked from commit f9a0e23381)
 2017-08-17 15:10:07 +05:30
Naganarasimha fa3094e436 YARN-6327. Removing queues from CapacitySchedulerQueueManager and ParentQueue should be done with iterator. Contributed by Jonathan Hung. 
(cherry picked from commit 0a3aa40fe7)
 2017-08-17 10:42:00 +05:30
Jason Lowe cf30380d46 YARN-7020. TestAMRMProxy#testAMRMProxyTokenRenewal is flakey. Contributed by Robert Kanter 
(cherry picked from commit 14553061be)
 2017-08-16 13:06:13 -05:00
bibinchundatt 67f9968407 YARN-6741. Deleting all children of a Parent Queue on refresh throws exception. Contributed by Naganarasimha G R. 2017-08-16 14:52:09 +05:30
Akira Ajisaka a28a3dc8f2
YARN-6965. Duplicate instantiation in FairSchedulerQueueInfo. Contributed by Masahiro Tanaka. 
(cherry picked from commit 588c190afd)
 2017-08-16 14:07:08 +09:00
Subru Krishnan ad46a90872 HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru). 
(cherry picked from commit 75dd866bfb)
 2017-08-15 16:54:48 -07:00
Arun Suresh 7b22df3da6 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 4d7be1d857)
 2017-08-14 19:57:34 -07:00
Jason Lowe 56d0e757c7 YARN-6917. Queue path is recomputed from scratch on every allocation. Contributed by Eric Payne 
(cherry picked from commit 5558792894)
 2017-08-14 15:34:50 -05:00
Jason Lowe e5ba40cc98 YARN-6987. Log app attempt during InvalidStateTransition. Contributed by Jonathan Eagles 
(cherry picked from commit 3325ef653d)
 2017-08-14 14:41:38 -05:00
Subru Krishnan a70efb6138 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-14 11:03:50 -07:00
Carlo Curino 394573780b YARN-6687. Validate that the duration of the periodic reservation is less than the periodicity. (subru via curino) 
(cherry picked from commit 28d97b79b6)
 2017-08-14 11:01:56 -07:00
Jian He 630ada1b4e YARN-6959. RM may allocate wrong AM Container for new attempt. Contributed by Yuqi Wang 2017-08-14 10:56:39 -07:00
Daniel Templeton 7d3cea232d Revert "YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator" 
This reverts commit ce0cdc50d9.
 2017-08-11 20:52:07 -07:00
Daniel Templeton ce0cdc50d9 YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator 
(Contributed by Larry Lo via Daniel Templeton)

(cherry picked from commit 0996acde6c)
 2017-08-11 14:51:42 -07:00
Daniel Templeton 11e3673748 YARN-6967. Limit application attempt's diagnostic message size thoroughly 
(Contributed by Chengbing Liu via Daniel Templeton)

(cherry picked from commit 65364defb4)
 2017-08-11 14:31:11 -07:00
Daniel Templeton 2566e5fcf7 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces 
(Contributed by weiyuan via Daniel Templeton)

(cherry picked from commit c7680d4cc4)
 2017-08-11 14:23:17 -07:00
Karthik Kambatla f6c74ac5ea YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha) 
(cherry picked from commit 8c4b6d16a5)
 2017-08-11 12:18:29 -07:00
Akira Ajisaka 1f80e05eb4
YARN-6958. Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice. Contributed by Yeliang Cang. 2017-08-10 22:14:44 +09:00
Jian He 8b7590057e YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-08-09 11:12:38 -07:00
Subru Krishnan e9ec1e8f87 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-08-08 14:01:22 -07:00
Daniel Templeton c5b3d80a4b YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code 
(Contributed by Angela Wang via Daniel Templeton)

(cherry picked from commit e0c24145d2)
 2017-08-08 13:36:49 -07:00
Junping Du e9c21d14b4 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-08 12:45:55 -07:00
Daniel Templeton 675d368950 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path 
(Contributed by Miklos Szegedi via Daniel Templeton)

(cherry picked from commit 47b145b9b4)
 2017-08-08 11:12:34 -07:00
Jian He ad1567c487 YARN-6890. Not display killApp button on UI if UI is unsecured but cluster is secured. Contributed by Junping Du 2017-08-08 11:10:29 -07:00
Akira Ajisaka 9fd5a03611
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-08 19:28:33 +09:00
Arun Suresh 3f50800d41 YARN-6920. Fix resource leak that happens during container re-initialization. (asuresh) 
(cherry picked from commit 8d3fd81980)
 2017-08-07 19:01:13 -07:00
Wangda Tan e18415a350 YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
(cherry picked from commit adb84f34db)
 2017-08-07 11:34:45 -07:00
Akira Ajisaka cfdf29785b
YARN-6957. Moving logging APIs over to slf4j in hadoop-yarn-server-sharedcachemanager. Contributed by Yeliang Cang. 
(cherry picked from commit b0fbf17965)
 2017-08-07 19:26:18 +09:00
Sunil G d15bac141e YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 
(cherry picked from commit 46b7054fa7)
 2017-08-07 13:17:04 +05:30
Yufei Gu b9549e108e YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-08-04 17:23:44 -07:00
Carlo Curino 7654b52167 YARN-6634. [API] Refactor ResourceManager WebServices to make API explicit (Contributed by Giovanni Matteo Fumarola via curino). 
(Edited cherry-pick of 547cb53e8a).
 2017-08-04 17:04:20 -07:00
Arun Suresh 4cfd248a72 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 
(cherry picked from commit 077fcf6a96)
 2017-08-04 16:50:21 -07:00
Arun Suresh 043b7d133e YARN-6776. Refactor ApplicaitonMasterService to move actual processing logic to a separate class. (asuresh) 
(cherry picked from commit 5496a34c0c)
 2017-08-04 16:43:35 -07:00
Arun Suresh 2d62af6545 YARN-5966. AMRMClient changes to support ExecutionType update. (asuresh) 
(cherry picked from commit aaf106fde3)
 2017-08-04 16:19:44 -07:00
Arun Suresh 88d6adf419 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 35dc782923)
 2017-08-03 21:18:43 -07:00
Yufei Gu b6729a7be5 YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-03 16:56:15 -07:00
Yufei Gu ce609d06c0 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 13:52:58 -07:00
Sunil G 82ec700241 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 
(cherry picked from commit f64cfeaf61)
 2017-08-03 19:28:16 +05:30
Eric Payne 223229b99c YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 
(cherry picked from commit 8ce8672b6b)
 2017-08-02 12:58:07 -05:00
Eric Payne 686bdc48b8 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 
(cherry picked from commit 48899134d2)
 2017-08-02 11:29:56 -05:00
Jian He 6ee0fe70c7 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:59 -07:00
Jian He 4006cc266b YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 10:00:04 -07:00
Akira Ajisaka 72bde1dad5
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 
(cherry picked from commit b38a1eea8e)
 2017-08-01 15:16:21 +09:00
Jason Lowe b9640c02bf YARN-6628. Unexpected jackson-core-2.2.3 dependency introduced. Contributed by Jonathan Eagles 2017-07-31 16:20:54 -05:00
Akira Ajisaka 98c3544e94
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 
(cherry picked from commit f8bed5e9a7)
 2017-07-31 11:10:10 +09:00
Arun Suresh 7e643130f2 YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 
(cherry picked from commit 890e14c02a)
 2017-07-28 16:33:42 -07:00
Yufei Gu e29d1c75ea YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit 9902be72cb)
 2017-07-27 23:21:30 -07:00
Varun Saxena 05bbbcbdbe Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit 11ece0bda1)
 2017-07-28 00:33:41 +05:30
Varun Saxena 0b342a644a YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit f66fd11e51)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
 2017-07-28 00:33:37 +05:30
Sunil G 08d5f3acf6 Addendum patch for YARN-6102. 2017-07-26 18:40:52 +05:30
Daniel Templeton 00f4a80e8f YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit f81a4efb8c)
 2017-07-25 13:02:00 -07:00
Akira Ajisaka 22f237c5d9
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 
(cherry picked from commit cca51e916b)
 2017-07-25 15:25:36 +09:00
Akira Ajisaka 09241feb13
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 
(cherry picked from commit 218b1b33ff)
 2017-07-25 15:12:47 +09:00
Daniel Templeton 9b6e787230 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 10583625c1)
 2017-07-24 13:45:09 -07:00
Daniel Templeton eb1751670a YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)

(cherry picked from commit bb30bd3771)
 2017-07-24 12:15:35 -07:00
Daniel Templeton 6b8e353c65 YARN-6844. AMRMClientImpl.checkNodeLabelExpression() has wrong error message 
(Contributed by Manikandan R via Daniel Templeton)

(cherry picked from commit 4c40cd451c)
 2017-07-24 12:06:50 -07:00
Sunil G a657472b42 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 20:57:56 +05:30
Jason Lowe db7c5636b6 YARN-6768. Improve performance of yarn api record toString and fromString. Contributed by Jonathan Eagles 
(cherry picked from commit 24853bf32a)
 2017-07-24 10:22:31 -05:00
Jason Lowe e76e7b9f0b YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 
(cherry picked from commit c8df3668ec)
 2017-07-20 11:06:39 -05:00
Wangda Tan 5070c9ba39 YARN-6775. CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls. (Nathan Roberts via wangda) 
Change-Id: Iaf9bb7e5ed3aa1300abdccf87ae6fcbddbd25e3e
 2017-07-19 15:25:21 -07:00
Rohith Sharma K S f806e8627b YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 
(cherry picked from commit 845c4e52bd)
 2017-07-19 11:16:17 +05:30
Yufei Gu d04bb465d2 YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit daaf530fce)
 2017-07-18 16:40:13 -07:00
Sunil G 487b0a3d7f Addendum patch for YARN-5731 
(cherry picked from commit 0b7afc060c)
 2017-07-18 13:46:43 +05:30
Sunil G 9b1abb448d YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. Contributed by Tao Yang. 2017-07-17 13:22:04 +05:30
Sunil G d462e48336 YARN-6280. Introduce deselect query param to skip ResourceRequest from getApp/getApps REST API. Contributed by Lantao Jin. 2017-07-17 12:25:41 +05:30
Arun Suresh 0101973dbf YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh) 2017-07-14 13:57:22 -07:00
Jason Lowe 9eb93c6f3f YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 
(cherry picked from commit a5ae5ac50e)
 2017-07-14 15:03:48 -05:00
bibinchundatt af80d10e54 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 
(cherry picked from commit 75c0220b44)
 2017-07-14 14:05:38 +05:30
Yufei Gu b2a34fe8d6 YARN-6769. Make schedulables without demand less needy in FairSharePolicy#compare. (Yunfan Zhou via Yufei Gu) 
(cherry picked from commit 4a574e9a84)
 2017-07-13 23:11:27 -07:00
Sunil G fd6935709f YARN-5731. Preemption calculation is not accurate when reserved containers are present in queue. Contributed by Wangda Tan. 2017-07-14 08:45:17 +05:30
Sunil G f5cdee6bfd YARN-6792. Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 228ddaa31d)
 2017-07-14 08:08:32 +05:30
Jason Lowe a77fb561ef YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 
(cherry picked from commit ebc048cc05)
 2017-07-13 17:49:55 -05:00
Jason Lowe 6ce2b0e4d4 YARN-6654. RollingLevelDBTimelineStore backwards incompatible after fst upgrade. Contributed by Jonathan Eagles 
(cherry picked from commit 5f1ee72b0e)
 2017-07-13 17:28:49 -05:00
Sunil G d83e8712f6 YARN-5892. Support user-specific minimum user limit percentage in Capacity Scheduler. Contributed by Eric Payne. 
(cherry picked from commit ca13b224b2)
 2017-07-13 14:50:28 -05:00
Wangda Tan cbb5f601f1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Sunil G via wangda) 
Change-Id: I9b19f69788068be05b3295247cdd7b972f8a573c
(cherry picked from commit c583ab02c7)
 2017-07-13 10:44:07 -05:00