Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,475 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4860 Commits

Author SHA1 Message Date
Sunil G 2564b4d07f YARN-7119. Support multiple resource types in rmadmin updateNodeResource command. Contributed by Manikandan R. 2017-12-14 16:11:57 +05:30
Eric Yang 438c1d333e YARN-7540. Route YARN service CLI function through YARN Service API. (Contributed by Eric Yang) 2017-12-13 19:29:04 -05:00
Chen Liang 46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Sunil G cb87e4dc92 YARN-7643. Handle recovery of applications in case of auto-created leaf queue mapping. Contributed by Suma Shivaprasad. 2017-12-13 22:49:58 +05:30
Sunil G 10fc8d2a7d YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák. 2017-12-13 22:00:07 +05:30
Sunil G 880cd754a5 YARN-7536. em-table improvement for better filtering in new YARN UI. Contributed by Vasudevan Skm. This closes #313. 2017-12-13 16:46:12 +05:30
Sunil G fa4ecd8b98 YARN-7641. Allow searchable filter for Application page log viewer in new YARN UI. Contributed by Vasudevan Skm. This closes #312. 2017-12-13 15:51:49 +05:30
Weiwei Yang 7efc4f7688 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:11:41 +08:00
Jason Lowe 2abab1d7c5 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-12 16:04:15 -06:00
Jian He 3ebe6a7819 YARN-7565. Yarn service pre-maturely releases the container after AM restart. Contributed by Chandni Singh 2017-12-12 13:35:56 -08:00
Jason Lowe 06f0eb2dce YARN-7625. Expose NM node/containers resource utilization in JVM metrics. Contributed by Weiwei Yang 2017-12-12 12:56:26 -06:00
Sunil G 8bb83a8f62 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues. Contributed by Suma Shivaprasad. 2017-12-12 15:20:59 +05:30
Sunil G 5c87fb2f62 YARN-7635. TestRMWebServicesSchedulerActivities fails in trunk. Contributed by Sunil G. 2017-12-12 15:08:18 +05:30
Sunil G 312ceebde8 YARN-7632. Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management. Contributed by Suma Shivaprasad. 2017-12-11 19:20:02 +05:30
Weiwei Yang a2edc4cbf5 YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 10:31:46 +08:00
Subru Krishnan 670e8d4ec7 YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 2017-12-08 15:39:18 -08:00
Wangda Tan 04b84da245 YARN-7443. Add native FPGA module support to do isolation with cgroups. (Zhankun Tang via wangda) 
Change-Id: Ic4b7f9f3e032986b8f955139c9fe4d3a6c818a53
 2017-12-08 15:18:22 -08:00
Wangda Tan adca1a72e4 YARN-7591. NPE in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I46689e530550ee0a6ac7a29786aab2cc1bdf314f
 2017-12-08 15:17:02 -08:00
Wangda Tan a8316df8c0 YARN-7520. Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues. (Suma Shivaprasad via wangda) 
Change-Id: I482f086945bd448d512cb5b3879d7371e37ee134
 2017-12-08 15:11:28 -08:00
Wangda Tan f548bfffbd YARN-7420. YARN UI changes to depict auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: I8039d3772a191ddede132cd1f8b08a8ca2e275b7
 2017-12-08 15:10:47 -08:00
Wangda Tan b38643c9a8 YARN-7473. Implement Framework and policy for capacity management of auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: Icca7805fe12f6f7fb335effff4b121b6f7f6337b
 2017-12-08 15:10:16 -08:00
Wangda Tan 74665e3a7d YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
 2017-12-08 15:07:56 -08:00
Sunil G 532bbf4602 YARN-7533. Documentation for absolute resource support in Capacity Scheduler. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G 4db4a4a165 YARN-7575. NPE in scheduler UI when max-capacity is not configured. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G daa1cdd062 YARN-7564. Cleanup to fix checkstyle issues of YARN-5881 branch. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan 1012b901c8 YARN-7544. Use queue-path.capacity/maximum-capacity to specify absolute min/max resources. (Sunil G via wangda) 
Change-Id: I685341be213eee500f51e02f01c91def89391c17
 2017-12-07 18:56:54 -08:00
Wangda Tan b7b8cd5324 YARN-7538. Fix performance regression introduced by Capacity Scheduler absolute min/max resource refactoring. (Sunil G via wangda) 
Change-Id: Ic9bd7e599c56970fe01cb0e1bba6df7d1f77eb29
 2017-12-07 18:56:54 -08:00
Wangda Tan 7462c38277 YARN-7483. CapacityScheduler test cases cleanup post YARN-5881. (Sunil G via wangda) 
Change-Id: I9741a6baf5cb7352d05636efb6c0b24790e7589a
 2017-12-07 18:56:54 -08:00
Rohith Sharma K S e65ca92fb6 YARN-7482. Max applications calculation per queue has to be retrospected with absolute resource support. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan 034b312d9f YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
 2017-12-07 18:56:54 -08:00
Wangda Tan aa3f62740f YARN-7332. Compute effectiveCapacity per each resource vector. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan d52627a7cb YARN-7254. UI and metrics changes related to absolute resource configuration. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan 5e798b1a0d YARN-6471. Support to add min/max resource configuration for a queue. (Sunil G via wangda) 
Change-Id: I9213f5297a6841fab5c573e85ee4c4e5f4a0b7ff
 2017-12-07 18:56:54 -08:00
Weiwei Yang e411dd6666 YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-07 17:29:40 +08:00
Weiwei Yang 40b0045ebe YARN-7610. Extend Distributed Shell to support launching job with opportunistic containers. Contributed by Weiwei Yang. 2017-12-06 17:52:41 +08:00
Weiwei Yang 05c347fe51 YARN-7611. Node manager web UI should display container type in containers page. Contributed by Weiwei Yang. 2017-12-06 12:21:52 +08:00
Wangda Tan 0311cf0535 YARN-7381. Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled by default in yarn-default.xml. (Xuan Gong via wangda) 
Change-Id: I1ed58dafad5cc276eea5c0b0813cf04f57d73a87
 2017-12-05 13:09:49 -08:00
Sunil G a957f1c60e YARN-7438. Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Contributed by Wangda Tan 2017-12-05 22:50:07 +05:30
Sunil G 99ccca341f YARN-7092. Render application specific log under application tab in new YARN UI. Contributed by Akhil PB. 2017-12-05 19:41:07 +05:30
Sunil G f9f317b702 YARN-7586. Application Placement should be done before ACL checks in ResourceManager. Contributed by Suma Shivaprasad. 2017-12-05 18:28:31 +05:30
Arpit Agarwal e00c7f78c1 HADOOP-14976. Set HADOOP_SHELL_EXECNAME explicitly in scripts. 2017-12-04 21:02:04 -08:00
Robert Kanter d8863fc16f YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 2017-12-04 13:14:55 -08:00
Eric Yang d30d57828f YARN-6669. Implemented Kerberos security for YARN service framework. (Contributed by Jian He) 2017-12-04 15:11:00 -05:00
Arun Suresh 37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Sunil G 81f6e46b2f YARN-6907. Node information page in the old web UI should report resource types. Contributed by Gergely Novák. 2017-12-04 11:27:23 +05:30
Sunil G 30f2646b15 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2017-12-04 10:45:07 +05:30
Jason Lowe 60f95fb719 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 2017-12-01 15:47:01 -06:00
Jason Lowe 25df505421 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 2017-12-01 15:37:36 -06:00
Robert Kanter c83fe44917 YARN-4813. TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently (grepas via rkanter) 2017-12-01 12:18:13 -08:00
Wangda Tan 7225ec0ceb YARN-6507. Add support in NodeManager to isolate FPGA devices with CGroups. (Zhankun Tang via wangda) 
Change-Id: Ic9afd841805f1035423915a0b0add5f3ba96cf9d
 2017-12-01 10:50:49 -08:00
Sunil G 556aea3f36 YARN-7487. Ensure volume to include GPU base libraries after created by plugin. Contributed by Wangda Tan. 2017-12-01 13:36:28 +05:30
Sunil G 4653aa3eb3 YARN-7546. Layout changes in Queue UI to show queue details on right pane. Contributed by Vasudevan Skm. 2017-12-01 13:26:01 +05:30
Wangda Tan a63d19d365 YARN-6124. Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues. (Zian Chen via wangda) 
Change-Id: Id93656f3af7dcd78cafa94e33663c78d410d43c2
 2017-11-30 15:57:22 -08:00
Junping Du a409425986 YARN-7558. yarn logs command fails to get logs for running containers if UI authentication is enabled. Contributed by Xuan Gong. 2017-11-30 13:47:47 -08:00
Jason Lowe 5cfaee2e6d YARN-7495. Improve robustness of the AggregatedLogDeletionService. Contributed by Jonathan Eagles 2017-11-30 12:39:18 -06:00
Wangda Tan c9a54aab6b YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2017-11-29 17:46:16 -08:00
Wei Yan 333ef303ff YARN-6851. Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat. 2017-11-29 14:38:07 -08:00
Daniel Templeton 8498d287cd YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 2017-11-29 11:11:36 -08:00
Jason Lowe a2c7a73e33 YARN-6647. RM can crash during transitionToStandby due to InterruptedException. Contributed by Bibin A Chundatt 2017-11-28 11:15:44 -06:00
Sunil G 6b76695f88 YARN-7480. Render tooltips on columns where text is clipped in new YARN UI. Contributed by Vasudevan Skm. This closes #293 2017-11-28 22:41:52 +05:30
Sunil G 641ba5c7a1 YARN-7499. Layout changes to Application details page in new YARN UI. Contributed by Vasudevan Skm. 2017-11-28 18:37:11 +05:30
Yufei Gu d8923cdbf1 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 2017-11-27 14:31:52 -08:00
Jian He fedabcad42 YARN-6168. Restarted RM may not inform AM about all existing containers. Contributed by Chandni Singh 2017-11-27 10:19:58 -08:00
Yufei Gu 2bde3aedf1 YARN-7290. Method canContainerBePreempted can return true when it shouldn't. (Contributed by Steven Rand) 2017-11-24 23:32:46 -08:00
Wangda Tan 834e91ee91 YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
 2017-11-23 19:59:03 -08:00
Arun Suresh b46ca7e73b YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 2017-11-22 19:18:30 -08:00
Sunil G aab439593b YARN-7330. Add support to show GPU in UI including metrics. Contributed by Wangda Tan. 2017-11-23 07:54:20 +05:30
Yufei Gu 4cc9479dae YARN-7524. Remove unused FairSchedulerEventLog. (Contributed by Wilfred Spiegelenburg) 2017-11-22 14:18:36 -08:00
Eric Yang d42a336cfa YARN-5534. Allow user provided Docker volume mount list. (Contributed by Shane Kumpf) 2017-11-22 13:05:34 -05:00
Steve Loughran de8b6ca5ef HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints. 
Contributed by Steve Loughran and Ryan Blue.
 2017-11-22 15:28:12 +00:00
yufei 03c311eae3 YARN-7513. Remove the scheduler lock in FSAppAttempt.getWeight() (Contributed by Wilfred Spiegelenburg) 2017-11-21 10:33:34 -08:00
Haibo Chen 67bbbe1c0c YARN-7531. ResourceRequest.equal does not check ExecutionTypeRequest.enforceExecutionType(). 2017-11-21 09:09:16 -08:00
Wangda Tan 0d781dd03b YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I51ae6c2ab7a3d1febdd7d8d0519b63a13295ac7d
 2017-11-20 11:48:15 -08:00
Billie Rinaldi 6f9d7a146d YARN-7529. TestYarnNativeServices#testRecoverComponentsAfterRMRestart() fails intermittently. Contributed by Chandni Singh 2017-11-20 07:37:04 -08:00
bibinchundatt b5b81a4f08 YARN-7489. ConcurrentModificationException in RMAppImpl#getRMAppMetrics. Contributed by Tao Yang. 2017-11-18 19:25:29 +05:30
bibinchundatt 96e6a99360 YARN-7525. Incorrect query parameters in cluster nodes REST API document. Contributed by Tao Yang. 2017-11-18 19:02:11 +05:30
Subru Krishnan d5f66888b8 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-17 17:39:06 -08:00
Eric Yang 0940e4f692 YARN-7218. Decouple YARN Services REST API namespace from RM. (Contributed by Eric Yang) 2017-11-17 12:28:12 -05:00
Varun Vasudev 5f0b238a11 YARN-7430. Enable user re-mapping for Docker containers by default. Contributed by Eric Yang. 2017-11-17 12:04:47 +05:30
Wangda Tan 0987a7b8cb YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue mapping. (Suma Shivaprasad via wangda) 
Change-Id: Ia1704bb8cb5070e5b180b5a85787d7b9ca57ebc6
 2017-11-16 11:25:52 -08:00
Jian He 6bf2c30192 YARN-7503. Configurable heap size / JVM opts in service AM. Contributed by Jonathan Hung 2017-11-16 10:53:55 -08:00
Haibo Chen 28d0fcbef4 YARN-7390. All reservation related test cases failed when TestYarnClient runs against Fair Scheduler. (Yufei Gu via Haibo Chen) 2017-11-16 10:48:24 -08:00
Sunil G 61ace174cd YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 2017-11-16 22:34:23 +05:30
Billie Rinaldi f4d5d20286 YARN-7486. Race condition in service AM that can cause NPE. Contributed by Jian He 2017-11-16 07:58:06 -08:00
Sunil G 09a1342608 YARN-7492. Set up SASS for new YARN UI styling. Contributed by Vasudevan Skm. 2017-11-16 13:44:21 +05:30
Jason Lowe fac72eef23 YARN-7361. Improve the docker container runtime documentation. Contributed by Shane Kumpf 2017-11-15 12:32:02 -06:00
Daniel Templeton b246c54749 YARN-7414. FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight() 
(Contributed by Soumabrata Chakraborty via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Daniel Templeton e094eb74b9 YARN-6953. Clean up ResourceUtils.setMinimumAllocationForMandatoryResources() and setMaximumAllocationForMandatoryResources() 
(Contributed by Manikandan R via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Sunil G 2f379d412b YARN-7464. Introduce filters in Nodes page of new YARN UI. Contributed by Vasudevan Skm. 2017-11-15 14:02:53 +05:30
Sunil G c4c57b80e1 YARN-7462. Render outstanding resource requests on application page of new YARN UI. Contributed by Vasudevan Skm. 2017-11-15 12:01:29 +05:30
Jian He 4f40cd314a YARN-7488. Make ServiceClient.getAppId method public to return ApplicationId for a service name. Contributed by Gour Saha 2017-11-13 18:57:56 -08:00
Jian He 5323b0048b YARN-7466. ResourceRequest has a different default for allocationRequestId than Container. Contributed by Chandni Singh 2017-11-13 15:37:39 -08:00
Junping Du e14f03dfbf YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 15:27:37 -08:00
Daniel Templeton 040a38dc49 YARN-7369. Improve the resource types docs 2017-11-13 11:05:07 -08:00
Daniel Templeton 2e512f016e YARN-7442. [YARN-7069] Limit format of resource type name (Contributed by Wangda Tan via Daniel Templeton) 2017-11-13 11:03:30 -08:00
Eric Yang fa4b5c669c YARN-7447. Fixed bug in create YARN services via RM. (Contributed by Billie Rinaldi) 2017-11-13 13:59:58 -05:00
Sunil G fb62bd625f YARN-7445. Render Applications and Services page with filters in new YARN UI. Contributed by Vasudevan Skm. 2017-11-13 19:41:49 +05:30
Subru Krishnan 3e26077848 YARN-7475. Fix Container log link in new YARN UI. (Sunil G via Subru). 
(cherry picked from commit 3c5b46c2ed)
 2017-11-12 09:53:39 -08:00
Sunil G ff9f7fcf7f YARN-7452. Decommissioning node default value to be zero in new YARN UI. Contributed by Vasudevan Skm. 2017-11-11 16:52:31 +05:30
bibinchundatt 2c2b7a3672 YARN-7406. Moving logging APIs over to slf4j in hadoop-yarn-api. Contributed by Yeliang Cang. 2017-11-11 10:36:27 +05:30
Wangda Tan dd07038ffa YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2017-11-09 14:51:15 -08:00
Sean Mackrory 1883a00249 YARN-7465. start-yarn.sh fails to start ResourceManagers unless running as root. 2017-11-09 14:47:43 -07:00
Konstantinos Karanasos ac4d2b1081 YARN-7437. Rename PlacementSet and SchedulingPlacementSet. (Wangda Tan via kkaranasos) 2017-11-09 13:01:24 -08:00
Robert Kanter a2c150a736 YARN-7386. Duplicate Strings in various places in Yarn memory (misha@cloudera.com via rkanter) 2017-11-09 12:12:52 -08:00
Daniel Templeton 462f6c490e YARN-7143. FileNotFound handling in ResourceUtils is inconsistent 
Change-Id: Ib1bb487e14a15edd2b5a42cf5078c5a2b295f069
 2017-11-09 11:58:49 -08:00
Haibo Chen a1382a18df YARN-7388. TestAMRestart should be scheduler agnostic. 2017-11-09 10:49:50 -08:00
bibinchundatt 0a72c2f56c YARN-7454. RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup. Contributed by Jason Lowe. 2017-11-09 21:01:19 +05:30
Daniel Templeton 49b4c0b334 YARN-7458. TestContainerManagerSecurity is still flakey 
(Contributed by Robert Kanter via Daniel Templeton)

Change-Id: Ibb1975ad086c3a33f8af0b4f8b9a13c3cdca3f7d
 2017-11-08 17:31:14 -08:00
Daniel Templeton 0de10680b7 YARN-7166. Container REST endpoints should report resource types 
Change-Id: If9c2fe58d4cf758bb6b6cf363dc01f35f8720987
 2017-11-08 16:43:49 -08:00
Arun Suresh cb35a59589 YARN-7343. Add a junit test for ContainerScheduler recovery. (Sampada Dehankar via asuresh) 2017-11-08 08:14:02 -08:00
Arun Suresh a9c70b0e84 YARN-7453. Fix issue where RM fails to switch to active after first successful start. (Rohith Sharma K S via asuresh) 2017-11-08 08:00:53 -08:00
Daniel Templeton 8db9d61ac2 YARN-7401. Reduce lock contention in ClusterNodeTracker#getClusterCapacity() 2017-11-07 14:53:48 -08:00
Wangda Tan 13fa2d4e3e YARN-7394. Merge code paths for Reservation/Plan queues and Auto Created queues. (Suma Shivaprasad via wangda) 2017-11-06 21:38:24 -08:00
Haibo Chen 8f214dc4f8 YARN-7360. TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic. 2017-11-06 15:45:37 -08:00
Jian He a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Billie Rinaldi a127f7b0fb YARN-7423. Improve service client loggings. Contributed by Jian He 2017-11-06 13:30:20 -08:00
Billie Rinaldi a6c4bd74b6 YARN-6394. Support specifying YARN related params in the service REST API. Contributed by Jian He 2017-11-06 13:30:20 -08:00
Jian He a288b0accf YARN-7331. Change in few metrics in new YARN UI related to native-services. Contributed by Sunil G 2017-11-06 13:30:20 -08:00
Jian He ba7ed7b66a YARN-7384. Remove apiserver cmd and merge service cmd into application cmd. Contributed by Billie Rinaldi 2017-11-06 13:30:19 -08:00
Billie Rinaldi 8ec9e72cc3 YARN-7393. RegistryDNS doesn't work in tcp channel. Contributed by Eric Yang 2017-11-06 13:30:19 -08:00
Billie Rinaldi d2775fbcca YARN-7351. Fix high CPU usage issue in RegistryDNS. Contributed by Jian He 2017-11-06 13:30:19 -08:00
Billie Rinaldi 501be9b4be YARN-7326. Add recursion support and configure RegistryDNS to lookup upstream. Contributed by Eric Yang 2017-11-06 13:30:19 -08:00
Billie Rinaldi 68acd88dcb YARN-7323. Data structure update in service REST API. Contributed by Jian He 2017-11-06 13:30:18 -08:00
Jian He 3c30b1a97d Rebase to trunk 2017-11-06 13:30:18 -08:00
Jian He 9273480084 YARN-7198. Add jsvc support for RegistryDNS. Contributed by Billie Rinaldi 2017-11-06 13:30:18 -08:00
Jian He b57144a172 YARN-7202. Add UT for api-server. Contributed by Eric Yang 2017-11-06 13:30:18 -08:00
Jian He b8a7ef1b64 YARN-6744. Recover component information on YARN native services AM restart. Contributed by Billie Rinaldi 2017-11-06 13:30:18 -08:00
Jian He c723021579 YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Jian He 673c0db43c Revert "YARN-6626. Embed REST API service into RM. Contributed by Eric Yang" 
This reverts commit 63d1084e9781e0fee876916190b69f6242dd00e4.
 2017-11-06 13:30:17 -08:00
Jian He 9e677fa05c YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Billie Rinaldi 8851209788 YARN-7191. Improve yarn-service documentation. Contributed by Jian He 2017-11-06 13:30:17 -08:00
Jian He c70e5df100 YARN-7201. Added an apache httpd example YARN service. Contributed by Billie Rinaldi 2017-11-06 13:30:16 -08:00
Billie Rinaldi ce74e64363 YARN-7210. Some NPE fixes in Registry DNS. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Billie Rinaldi 37c9b7327d YARN-7186. Add examples in yarn-service. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Jian He 307d55b3e1 Rebased onto trunk - fix conflicts 2017-11-06 13:30:15 -08:00
Gour Saha 541b64c798 YARN-7165. Miscellaneous fixes in yarn-native-services. Contributed by Jian He 2017-11-06 13:30:15 -08:00
Jian He b9ca13e34b YARN-7161. Fix special chars in DNS documentation. Contributed by Gour Saha 2017-11-06 13:30:15 -08:00
Jian He eed9e3e8f4 Rebase onto trunk - fix some issues. Contributed by Rohith Sharma K S 2017-11-06 13:30:14 -08:00
Jian He e660e52a04 YARN-7126. Create introductory site documentation for YARN native services. Contributed by Gour Saha 2017-11-06 13:30:14 -08:00
Jian He 7dd4490a44 YARN-7073. Yarn native services rest API documentation. Contributed by Gour Saha 2017-11-06 13:30:14 -08:00
Jian He bd96c4c235 Rebase onto latest trunk. minor conflicts 2017-11-06 13:30:13 -08:00
Jian He 59fe98cd50 YARN-7100. Added JsonIgnore annotation to Resource#getMemoryMB. Contributed by Eric Yang 2017-11-06 13:30:13 -08:00
Jian He fecf22b2fd YARN-7113. Clean up packaging and dependencies for yarn-native-services. Contributed by Billie Rinaldi 2017-11-06 13:30:13 -08:00
Billie Rinaldi 40ab068eab YARN-7091. Rename application to service in yarn-native-services. Contributed by Jian He 2017-11-06 13:30:13 -08:00
Gour Saha 2ba38ce8ec YARN-5244. Documentation required for DNS Server implementation. Contributed by Jon Maron 2017-11-06 13:30:12 -08:00
Billie Rinaldi 2191f5b533 YARN-7050. Post cleanup after YARN-6903, removal of org.apache.slider package. Contributed by Jian He 2017-11-06 13:30:12 -08:00
Billie Rinaldi 79bf5c084d YARN-7029. Add more UTs in yarn-native-services. Contributed by Jian He 2017-11-06 13:30:11 -08:00
Billie Rinaldi 1888318c89 YARN-6903. Yarn-native-service framework core rewrite. Contributed by Jian He 2017-11-06 13:30:11 -08:00
Jian He 27dcc67959 Update pom version to 3.0.0-beta1-SNAPSHOT 2017-11-06 13:30:10 -08:00
Jian He 8d335e59cf YARN-6804. [yarn-native-services changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-11-06 13:30:10 -08:00
Billie Rinaldi 14f3f3b42e YARN-6655. Fix compilation failure in TestServiceApiUtil due to changes in YARN-6716. 2017-11-06 13:30:10 -08:00
Jian He 59a9b4f438 YARN-6655. Validate yarn native services application submission side to ensure that the hostname should be less than 63 characters. Contributed by Billie Rinaldi 2017-11-06 13:30:09 -08:00
Jian He e86d828548 YARN-6716. Native services support for specifying component start order. Contributed by Billie Rinaldi 2017-11-06 13:30:09 -08:00
Akira Ajisaka b7e5d739ba YARN-6722. Bumping up pom file hadoop version. Contributed by Panagiotis Garefalakis. 2017-11-06 13:30:09 -08:00
Billie Rinaldi a0b9e8a630 YARN-6617. Services API delete call first attempt usually fails. Contributed by Jian He 2017-11-06 13:30:09 -08:00
Jian He a0574e7f4f YARN-6613. Update json validation for new native services providers. Contributed by Billie Rinaldi 2017-11-06 13:30:08 -08:00
Jian He 9085cd5c6e YARN-6160. Create an agent-less docker-less provider in the native services framework. Contributed by Billie Rinaldi 2017-11-06 13:30:08 -08:00
Gour Saha a22256bd40 YARN-6544. Add Null check RegistryDNS service while parsing registry records. Contributed by Karam Singh 2017-11-06 13:30:08 -08:00
Billie Rinaldi 02d68c4bcf YARN-6588. Add native-service AM log4j properties in classpath. Contributed by Jian He 2017-11-06 13:30:08 -08:00
Jian He a041373dd4 YARN-6533. Race condition in writing service record to registry in yarn native services. Contributed by Billie Rinaldi 2017-11-06 13:30:08 -08:00
Billie Rinaldi ce05c6e981 YARN-6545. Followup fix for YARN-6405. Contributed by Jian He 2017-11-06 13:30:07 -08:00
Sunil G 399525c2e0 YARN-6398. Support to add native-service specific details in new YARN UI. Contributed by Akhil PB. 2017-11-06 13:30:07 -08:00
Sunil G ba812bf67f YARN-6446. addendum patch. Contributed by Rohith Sharma K S. 2017-11-06 13:30:07 -08:00
Jian He a8a273b064 YARN-6446. Revisit ATSv2 integration to ensure all required information is published. Contributed by Rohith Sharma K S 2017-11-06 13:30:07 -08:00
Sunil G 08cdf231a0 YARN-6419. Support to launch new native-service from new YARN UI. Contributed by Akhil PB. 2017-11-06 13:30:06 -08:00
Billie Rinaldi 845764da14 YARN-6405. Improve configuring services through REST API. Contributed by Jian He 2017-11-06 13:28:42 -08:00
Jian He 384ee13eec YARN-6335. Port slider's groovy unit tests to yarn native services. Contributed by Billie Rinaldi 2017-11-06 13:28:42 -08:00
Jian He c3ac67e3e1 Rebase to latest trunk, fix conflicts and pom.xml 2017-11-06 13:28:41 -08:00
Jian He fd9d0d4ead YARN-6400. Remove some unneeded code after YARN-6255. Contributed by Jian He 2017-11-06 13:28:41 -08:00
Jian He f4216b7bba YARN-6395. Integrate service app master to write data into ATSv2. Contributed by Rohith Sharma K S 2017-11-06 13:28:41 -08:00
Jian He c31cd981eb YARN-6255. Refactor yarn-native-services framework. Contributed by Jian He 2017-11-06 13:28:41 -08:00
Billie Rinaldi 024e90a508 YARN-6192. Move yarn-native-service API records into hadoop-yarn-slider-core module. Contributed by Jian He 2017-11-06 13:28:41 -08:00
Gour Saha 0ed8d4ebe6 YARN-6205. Default lifetime for native services app is invalid. Contributed by Billie Rinaldi 2017-11-06 13:28:40 -08:00
Gour Saha 500695d726 YARN-6185. Apply SLIDER-1199 to yarn native services for blacklisting nodes. Contributed by Billie Rinaldi 2017-11-06 13:28:40 -08:00
Gour Saha 8967a1b812 YARN-6186 Handle InvalidResourceRequestException in native services AM onError 2017-11-06 13:28:40 -08:00
Billie Rinaldi e34ca6b02d YARN-6173. Add artifact info and privileged container details to the container info in API GET response. Contributed by Gour Saha 2017-11-06 13:28:40 -08:00
Jian He 68940ab8f6 Updated pom to point to 3.0.0-alpha3-SNAPSHOT 2017-11-06 13:28:40 -08:00
Billie Rinaldi d943eb4c15 YARN-6132. SliderClient bondToCluster should call findInstance with live states filter. Contributed by Gour Saha 2017-11-06 13:28:40 -08:00
Billie Rinaldi 7edc154e0e YARN-6115. Few additional paths in Slider client still uses get all Applications without tags/states filter. Contributed by Gour Saha 2017-11-06 13:28:40 -08:00
Gour Saha a5e20f0fc1 YARN-5993. Allow native services quicklinks to be exported for each component. Contributed by Billie Rinaldi 2017-11-06 13:28:39 -08:00
Varun Vasudev 7d685f2fb3 YARN-4757. Add the ability to split reverse zone subnets. Contributed by Shane Kumpf. 
(cherry picked from commit 9bff70f131)
 2017-11-06 13:28:39 -08:00
Jian He f934f62fc7 YARN-5218. Initial core change for DNS for YARN. Contributed by Jonathan Maron 2017-11-06 13:28:39 -08:00
Billie Rinaldi 74fff4086e YARN-6014. Followup fix for slider core module findbugs. Contributed by Jian He 2017-11-06 13:28:39 -08:00
Billie Rinaldi 1ea43c443e YARN-6010. Fix findbugs, site warnings in yarn-services-api module. Contributed by Jian He 2017-11-06 13:28:38 -08:00
Jian He 0b00c8e56b YARN-5968. Fix slider core module javadocs. Contributed by Billie Rinaldi 2017-11-06 13:28:38 -08:00
Billie Rinaldi db96e8aa21 YARN-5967. Fix slider core module findbugs warnings. Contributed by Jian He 2017-11-06 13:28:38 -08:00
Gour Saha c11f4b3c21 YARN-5996. Native services AM kills app on AMRMClientAsync onError call. Contributed by Billie Rinaldi 2017-11-06 13:28:38 -08:00
Gour Saha 2e1fa61253 YARN-5740. Add a new field in Slider status output - lifetime (remaining). Contributed by Jian He 2017-11-06 13:28:37 -08:00
Billie Rinaldi 2cea59dc54 YARN-5975. Remove the agent - slider AM ssl related code. Contributed by Jian He 2017-11-06 13:28:37 -08:00
Jian He efe7d44ead YARN-5961. Generate native services protobuf classes during build. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha e4f0263e51 YARN-5958. Fix ASF license warnings for slider core module. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha 1f44e6a5f4 YARN-5944. Native services AM should remain up if RM is down. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha 6143aa135b YARN-5769. Integrate update app lifetime using feature implemented in YARN-5611. Contributed by Jian He 2017-11-06 13:28:37 -08:00
Gour Saha bd0e9f91b6 YARN-5941. Slider handles "per.component" for multiple components incorrectly. Contributed by Billie Rinaldi 2017-11-06 13:28:37 -08:00
Gour Saha 4f43ebdb9f YARN-5943. Write native services container stderr file to log directory. Contributed by Billie Rinaldi 2017-11-06 13:28:36 -08:00
Billie Rinaldi 7b8fd3abd6 YARN-5883 Avoid or eliminate expensive YARN get all applications call. Contributed by Gour Saha 2017-11-06 13:28:36 -08:00
Jian He 234dba84a1 YARN-5909. Remove agent related code in slider AM. Contributed by Jian He 2017-11-06 13:28:36 -08:00
Gour Saha 2c4e0f33e9 YARN-5808. Add gc log options to the yarn daemon script when starting services-api. Contributed by Billie Rinaldi 2017-11-06 13:28:35 -08:00
Gour Saha d1ee95981e YARN-5828. Native services client errors out when config formats are uppercase. Contributed by Billie Rinaldi 2017-11-06 13:28:35 -08:00
Jian He 7757f6fe4a YARN-5812. Exception during GET call - "Failed to retrieve application: null". Contributed by Gour Saha 2017-11-06 13:28:35 -08:00
Gour Saha 25411a1742 YARN-5813. Slider should not try to set a negative lifetime timeout value. Contributed by Jian He 2017-11-06 13:28:35 -08:00
Jian He 3741e5518f YARN-5796. Convert enums values in service code to upper case and special handling of an error. Contributed by Gour Saha 2017-11-06 13:28:34 -08:00
Gour Saha 09e4b9e897 YARN-5690. Integrate native services modules into maven build. Contributed by Billie Rinaldi 2017-11-06 13:28:34 -08:00
Billie Rinaldi ef5a3628c2 YARN-5770. Performance improvement of native-services REST API service. Contributed by Gour Saha 2017-11-06 13:28:34 -08:00
Gour Saha 1d7a42e795 YARN-5680. Add 2 new fields in Slider status output - image-name and is-privileged-container. Contributed by Billie Rinaldi 2017-11-06 13:28:34 -08:00
Billie Rinaldi 773c565ac5 YARN-5775. Convert enums in swagger definition to uppercase. Contributed by Gour Saha 2017-11-06 13:28:34 -08:00
Gour Saha 6bdfce3ac7 YARN-5778. Add .keep file for yarn native services AM web app. Contributed by Billie Rinaldi 2017-11-06 13:28:34 -08:00
Jian He f206a43954 YARN-5701. Fix issues in yarn native services apps-of-apps. Contributed by Billie Rinaldi 2017-11-06 13:28:33 -08:00
Gour Saha 4ec1cbe86d YARN-5735. Make the service REST API use the app timeout feature YARN-4205. Contributed by Jian He 2017-11-06 13:28:33 -08:00
Jian He 7da243ebe0 YARN-5729. Bug fixes for the service Rest API. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He c0120e51b9 YARN-5689. Update native services REST API to use agentless docker provider. Contributed by Billie Rinaldi & Gour Saha 2017-11-06 13:28:33 -08:00
Jian He f66ecfecf6 Addendum patch for YARN-5610. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He 295b87d0fb YARN-5675. Swagger definition for YARN service API. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He 18bf0dce28 YARN-5610. Initial code for native services REST API. Contributed by Gour Saha 2017-11-06 13:28:33 -08:00
Jian He 9dc46aa379 YARN-5623. Apply SLIDER-1166 to yarn-native-services branch. Contributed by Gour Saha 2017-11-06 13:28:32 -08:00
Jian He bce06ed1af YARN-5505. Create an agent-less docker provider in the native-services framework. Contributed by Billie Rinaldi 2017-11-06 13:28:32 -08:00
Jian He 3279baecb9 YARN-5538. Apply SLIDER-875 to yarn-native-services. Contributed by Billie Rinaldi 2017-11-06 13:28:32 -08:00
Jian He 02e2740bdf YARN-5513. Move Java only tests from slider develop to yarn-native-services. Contributed by Gour Saha 2017-11-06 13:28:32 -08:00
Jian He 54ec9e2440 Modify pom file for slider 2017-11-06 13:28:32 -08:00
Jian He 804bb82d1b Rename org.apache.slider.core.build to org.apache.slider.core.buildutils 2017-11-06 13:28:32 -08:00
Jian He f47df51791 YARN-5461. Initial code ported from slider-core module. (jianhe) 2017-11-06 13:28:31 -08:00
bibinchundatt dcd99c4b9a Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 2017-11-06 22:39:10 +05:30
Sunil G c41728486b YARN-7392. Render cluster information on new YARN web ui. Contributed by Vasudevan Skm. 2017-11-03 11:45:50 +05:30
Inigo Goiri 6fc09beac4 YARN-7434. Router getApps REST invocation fails with multiple RMs. Contributed by Inigo Goiri. 2017-11-02 21:29:53 -07:00
Sunil G 1700adc6f7 YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. Contributed by Wangda Tan. 2017-11-03 07:25:29 +05:30
Subru Krishnan ad0fff2b41 YARN-7432. Fix DominantResourceFairnessPolicy serializable findbugs issues. (Daniel Templeton via Subru). 2017-11-02 17:39:23 -07:00
Eric Payne e6ec02001f YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 2017-11-02 12:37:33 -05:00
Wangda Tan 04c604cf1d YARN-7364. Queue dash board in new YARN UI has incorrect values. (Sunil G via wangda) 
Change-Id: Ia5336f407147d2985a61b5f9f1e69b5b35f398a4
(cherry picked from commit 752f000e0f)
 2017-11-02 09:51:28 -07:00
Jason Lowe d00b6f7c1f YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 2017-11-02 09:37:17 -05:00
Rohith Sharma K S 940ffe3f9c addendum patch for YARN-7289. 2017-11-02 13:55:19 +05:30
Jian He 0cc98ae0ec YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2017-11-01 17:00:32 -07:00
Eric Yang 7a49ddfdde YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 2017-11-01 18:39:56 -04:00
Junping Du d57dba9942 YARN-7400. Incorrect log preview displayed in jobhistory server ui. Contributed by Xuan Gong. 2017-11-01 14:48:16 -07:00
Inigo Goiri 70f1a9470c YARN-7276 addendum to add timeline service depencies. Contributed by Inigo Goiri. 2017-11-01 13:26:37 -07:00
Subru Krishnan ed24da3dd7 YARN-6413. FileSystem based Yarn Registry implementation. (Ellen Hui via Subru). 2017-10-31 12:05:43 -07:00
Akira Ajisaka c02d2ba50d
YARN-7379. Moving logging APIs over to slf4j in hadoop-yarn-client. Contributed by Yeliang Cang. 2017-10-31 17:09:45 +09:00
Akira Ajisaka 785f1b0d11
YARN-7407. Moving logging APIs over to slf4j in hadoop-yarn-applications. Contributed by Yeliang Cang. 2017-10-31 16:36:02 +09:00
Junping Du a8083aaa6f HADOOP-14990. Clean up jdiff xml files added for 2.8.2 release. 2017-10-30 15:16:51 -07:00
Daniel Templeton d64736d589 YARN-7336. Unsafe cast from long to int Resource.hashCode() method 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-10-30 12:41:28 -07:00
Daniel Templeton 9a7e810838 YARN-6927. Add support for individual resource types requests in MapReduce 
(Contributed by Gergo Repas via Daniel Templeton)
 2017-10-30 11:04:22 -07:00
Daniel Templeton 9711b78998 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 2017-10-29 18:54:33 -07:00
Yufei Gu d4811c8cfa YARN-6747. TestFSAppStarvation.testPreemptionEnable fails intermittently. (Contributed by Miklos Szegedi) 2017-10-29 16:44:16 -07:00
Sunil G 9114d7a5a0 YARN-7224. Support GPU isolation for docker container. Contributed by Wangda Tan. 2017-10-29 11:08:44 +05:30
Daniel Templeton e62bbbca7a YARN-7397. Reduce lock contention in FairScheduler#getAppWeight() 2017-10-28 09:13:13 -07:00
Arun Suresh 9c5c68745e YARN-7299. Fix TestDistributedScheduler. (asuresh) 2017-10-27 23:08:18 -07:00
Arun Suresh 24f8c5cce3 YARN-7178. Add documentation for Container Update API. (asuresh) 2017-10-27 22:56:41 -07:00
Inigo Goiri 8be5707067 YARN-7276. Federation Router Web Service fixes. Contributed by Inigo Goiri. 2017-10-27 16:46:05 -07:00
Jason Lowe 665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Rohith Sharma K S 5c799ecf09 YARN-7289. Application lifetime does not work with FairScheduler. Contributed by Miklos Szegedi. 2017-10-27 22:46:38 +05:30
Sunil G 792388e1c0 YARN-7375. Possible NPE in RMWebapp when HA is enabled and the active RM fails. Contributed by Chandni Singh. 2017-10-27 20:53:57 +05:30
Wangda Tan 36e158ae98 YARN-7307. Allow client/AM update supported resource types via YARN APIs. (Sunil G via wangda) 
Change-Id: I14c5ea7252b7c17e86ab38f692b5f9d43196dbe0
 2017-10-26 20:15:19 -07:00
Junping Du a25b5aa0cf Add several jdiff xml files for 2.8.2 release. 2017-10-26 19:30:48 -07:00
Robert Kanter b1de78619f YARN-7262. Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow (rkanter) 2017-10-26 17:47:32 -07:00
Robert Kanter 088ffee716 YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ Addendum (misha@cloudera.com via rkanter) 2017-10-26 15:50:14 -07:00
Subru Krishnan 25932da6d1 YARN-5516. Add REST API for supporting recurring reservations. (Sean Po via Subru). 2017-10-26 12:10:14 -07:00
Robert Kanter 2da654e34a YARN-7358. TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler (haibochen via rkanter) 2017-10-26 10:25:10 -07:00
Subru Krishnan 3fae675383 YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru). 2017-10-25 15:07:50 -07:00
Haibo Chen d7f3737f3b YARN-7389. Make TestResourceManager Scheduler agnostic. (Robert Kanter via Haibo Chen) 2017-10-24 22:17:56 -07:00
Robert Kanter 03af442e76 YARN-7385. TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE (yufeigu via rkanter) 2017-10-24 13:36:50 -07:00
Carlo Curino 1c5c2b5dde YARN-7339. LocalityMulticastAMRMProxyPolicy should handle cancel request properly. (Botong Huang via curino) 2017-10-24 10:39:04 -07:00
Robert Kanter 025c656572 YARN-7382. NoSuchElementException in FairScheduler after failover causes RM crash (rkanter) 2017-10-24 10:21:44 -07:00
Daniel Templeton 679f99b145 YARN-6984. DominantResourceCalculator.isAnyMajorResourceZero() should test all resources 
(Contributed by Sunil G via Daniel Templeton)
 2017-10-24 10:08:26 -07:00
Robert Kanter 5da295a34e YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ (misha@cloudera.com via rkanter) 2017-10-23 17:56:56 -07:00
Daniel Templeton 9e77dc2bd1 YARN-7357. Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-10-23 13:51:19 -07:00
Eric Payne 921338cd86 YARN-4163: Audit getQueueInfo and getApplications calls 2017-10-23 11:43:41 -05:00
Haibo Chen 480187aebb YARN-7372. TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic is flaky. 2017-10-20 14:24:17 -07:00
Yufei Gu 6b7c87c945 YARN-7355. TestDistributedShell should be scheduler agnostic. (Contributed by Haibo) 2017-10-20 11:15:20 -07:00
Yufei Gu 0799fde35e YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu) 2017-10-20 10:00:13 -07:00
Eric Yang b61144a93d YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-20 12:02:06 -04:00
Yufei Gu 1f4cdf1068 YARN-4090. Make Collections.sort() more efficient by caching resource usage. (Contributed by Yufei Gu, Shilong Zhang and Xianyin Xin) 2017-10-20 01:32:20 -07:00
Wangda Tan 4afd308b62 YARN-7170. Improve bower dependencies for YARN UI v2. (Sunil G via wangda) 2017-10-19 21:58:40 -07:00
Yufei Gu 7b4b018780 YARN-7359. TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic. (Contributed by Haibo Chen) 2017-10-19 16:51:47 -07:00
Yufei Gu cbd2b73ef8 YARN-7294. TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair Scheduler. (Contributed by Miklos Szegedi) 2017-10-19 16:39:25 -07:00
Wangda Tan c1b08ba720 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2017-10-19 14:45:44 -07:00
Wangda Tan 298b174f66 YARN-7338. Support same origin policy for cross site scripting prevention. (Sunil G via wangda) 2017-10-19 14:44:42 -07:00
Subru Krishnan 75323394fb YARN-7311. Fix TestRMWebServicesReservation parametrization for fair scheduler. (Yufei Gu via Subru). 2017-10-17 12:38:06 -07:00
Haibo Chen acabc657ff YARN-7341. TestRouterWebServiceUtil#testMergeMetrics is flakey. (Robert Kanter via Haibo Chen) 2017-10-17 10:15:53 -07:00
Robert Kanter 8a61525928 YARN-7308. TestApplicationACLs fails with FairScheduler (rkanter) 2017-10-16 15:34:32 -07:00
Nathan Roberts 4540ffd15f YARN-7333. container-executor fails to remove entries from a directory that is not writable or executable. Contributed by Jason Lowe. 2017-10-16 17:00:38 -05:00
Junping Du 1fcbe7cf5f YARN-7124. LogAggregationTFileController deletes/renames while file is open. Contributed by Jason Lowe. 2017-10-16 13:57:03 -07:00
Arun Suresh a50be1b8f4 YARN-7275. NM Statestore cleanup for Container updates. (Kartheek Muthyala via asuresh) 2017-10-16 13:12:15 -07:00
Wangda Tan 87ea1dff9c Addendum fix for: YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. (Wangda Tan) 
Change-Id: If1fe4a62e07b25e6f1b8ba803495da87e2cb2df6
 2017-10-14 10:41:58 -07:00
Haibo Chen 3fb4718886 YARN-7310. TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler. (Robert Kanter via Haibo Chen) 2017-10-13 15:49:21 -07:00
Yufei Gu 7a27c2c367 YARN-7270. Fix unsafe casting from long to int for class Resource and 
its sub-classes. (Yufei)
 2017-10-13 12:41:59 -07:00
Rohith Sharma K S f4fb6695a3 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-13 15:17:05 +05:30
Carlo Curino 13fcfb3d46 YARN-7317. Fix overallocation resulted from ceiling in LocalityMulticastAMRMProxyPolicy. (contributed by Botong Huang via curino) 2017-10-12 10:38:58 -07:00
Wangda Tan 8bcc49e677 YARN-7205. Log improvements for the ResourceUtils. (Sunil G via wangda) 
Change-Id: I0f5b7a7f68ec5d3e1d52211f83fdd089bc0bfd37
 2017-10-11 15:25:28 -07:00
Sunil G fa5cfc68f3 YARN-6620. Add support in NodeManager to isolate GPU devices by using CGroups. Contributed by Wangda Tan. 2017-10-11 23:44:33 +05:30
Jason Lowe ebb34c7053 YARN-7082. TestContainerManagerSecurity failing in trunk. Contributed by Akira Ajisaka 2017-10-11 10:31:02 -05:00
Robert Kanter ec8bf9e48a YARN-7309. TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by FairScheduler (rkanter) 2017-10-10 10:18:42 -07:00
Sunil G 41351b05f3 YARN-7296. convertToProtoFormat(Resource r) is not setting for all resource types. Contributed by lovekesh bansal. 2017-10-10 12:02:18 +05:30
Wangda Tan 5cfbad68c0 YARN-7237. Cleanup usages of ResourceProfiles. (Wangda Tan) 
Change-Id: I959c8531bd48231956068cb7d2312f3235549f9c
(cherry picked from commit 8536db5a7549fa44d9f9e444b41bd5978b5c6ae1)
 2017-10-09 20:36:57 -07:00
Jonathan Hung 1d36b53ab6 YARN-7252. Removing queue then failing over results in exception 2017-10-09 11:11:23 -07:00
Jonathan Hung 09c5dfe937 YARN-7251. Misc changes to YARN-5734 2017-10-09 11:11:23 -07:00
Jonathan Hung a3075de9ab YARN-7238. Documentation for API based scheduler configuration management 2017-10-09 11:11:23 -07:00
Jonathan Hung d6622daaa3 YARN-7046. Add closing logic to configuration store 2017-10-09 11:11:23 -07:00
Wangda Tan ff39c0de20 YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:11:23 -07:00
Xuan 4d8abd84f4 YARN-7024: Fix issues on recovery in LevelDB store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan 4c8b208adb YARN-6322: Disable queue refresh when configuration mutation is enabled. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan 74ba6ffa0b YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan 04f9e80bb2 YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to package org.apache.hadoop.yarn.webapp.dao 2017-10-09 11:11:23 -07:00
Xuan 916bdbd6be YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung via xgong) 2017-10-09 11:11:23 -07:00
Xuan e566fd8b58 YARN-6575. Support global configuration mutation in MutableConfProvider. (Jonathan Hung via Xuan Gong) 2017-10-09 11:11:23 -07:00
Wangda Tan a4e6253046 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:11:23 -07:00
Wangda Tan d8a5136b1f YARN-5952. Create REST API for changing YARN scheduler configurations. (Jonathan Hung via wangda) 2017-10-09 11:11:23 -07:00
Jonathan Hung ef686635c1 YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:11:23 -07:00
Xuan e3579a8c3b YARN-5946: Create YarnConfigurationStore interface and 
InMemoryConfigurationStore class. Contributed by Jonathan Hung
 2017-10-09 11:11:23 -07:00
Jonathan Hung ef59cbe08a YARN-5951. Changes to allow CapacityScheduler to use configuration store 2017-10-09 11:11:22 -07:00
Daniel Templeton 6d6ca4c923 YARN-2612 addendum: fixed javadoc error. 2017-10-09 14:12:13 +02:00
Sunil G 8137254a1f YARN-7245. Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user. Contributed by Eric Payne. 2017-10-09 14:39:41 +05:30
Rohith Sharma K S 6fe6a72352 Revert "YARN-7180. Remove class ResourceType. Contributed by Sunil G." 
This reverts commit 6bd81ab054.
 2017-10-09 13:21:13 +05:30
Rohith Sharma K S 6bd81ab054 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-09 10:20:40 +05:30
Yufei Gu 72d22b753a YARN-7207. Cache the RM proxy server address. (Yufei Gu) 2017-10-06 09:49:11 -07:00
Yufei Gu 49ae538164 YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage(addendum). (Yufei Gu) 2017-10-06 00:08:25 -07:00
Yufei Gu 99292adcef YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage. (Yufei Gu) 2017-10-05 23:51:58 -07:00
Haibo Chen c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Naganarasimha d0046bb5c6 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 2017-10-06 00:07:53 +05:30
Chris Trezzo 7e76f85bc6 YARN-2960. Add documentation for the YARN shared cache. 2017-10-05 10:38:41 -07:00
Arun Suresh b733348dde YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 2017-10-05 09:58:04 -07:00
Sunil G f702c95754 YARN-6182. Fix alignment issues and missing information in new YARN UI's Queue page. Contributed by Akhil PB. 2017-10-05 11:34:44 +05:30
Sunil G 592bf2d550 YARN-7279. Fix typo in helper message of ContainerLauncher. Contributed by Elek, Marton. 2017-10-05 10:52:00 +05:30
Subru Krishnan e6e614e380 YARN-5329. Placement Agent enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-10-04 19:28:27 -07:00
Akira Ajisaka 2df1b2ac05
YARN-7044. TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails. 2017-10-05 06:07:32 +09:00
Akira Ajisaka acf5b880d8
YARN-6975. Moving logging APIs over to slf4j in hadoop-yarn-server-tests, hadoop-yarn-server-web-proxy and hadoop-yarn-server-router. Contributed by Yeliang Cang. 2017-10-04 14:53:41 +09:00
Ray Chiang 67e12169b0 YARN-7219. Make AllocateRequestProto compatible with branch-2/branch-2.8. (rchiang) 2017-10-03 15:06:07 -07:00
Akira Ajisaka c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Akira Ajisaka 4a87773718
YARN-6916. Moving logging APIs over to slf4j in hadoop-yarn-server-common. Contributed by Bibin A Chundatt and Akira Ajisaka. 2017-10-04 06:06:39 +09:00
Allen Wittenauer 4d5dd75b60
HADOOP-14908. CrossOriginFilter should trigger regex on more input (Johannes Alberti via aw) 2017-10-03 10:58:28 -07:00
Jian He 4111e6c781 YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. Contributed by Wangda Tan 2017-10-03 09:54:38 -07:00
Sidharta S 7eb8499996 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-02 19:04:49 -07:00
Subru Krishnan d4d2fd1acd YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru). 2017-10-02 18:14:44 -07:00
Wangda Tan 280080fad0 YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda) 
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
 2017-10-02 15:30:22 -07:00
Carlo Curino ceca9694f9 YARN-3661. Basic Federation UI. (Contributed by Inigo Goiri via curino) 2017-10-02 13:03:32 -07:00
Yufei Gu 66c417167a YARN-6333. Improve doc for minSharePreemptionTimeout, fairSharePreemptionTimeout and fairSharePreemptionThreshold. (Contributed by Chetna Chaudhari via Yufei Gu) 2017-09-29 18:07:33 -07:00
Wangda Tan ec2ae3060a YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda) 
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
 2017-09-29 16:42:56 -07:00
Wangda Tan febeead5f9 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda) 
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
 2017-09-29 16:39:46 -07:00
Eric Payne 7f6118f918 YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe. 2017-09-29 10:33:57 -05:00
Wangda Tan d3b1c63195 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda) 
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
 2017-09-28 16:41:09 -07:00
Chris Trezzo c114da5e64 YARN-7250. Update Shared cache client api to use URLs. 2017-09-28 15:28:06 -07:00
Subru Krishnan ca669f9f8b YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru). 2017-09-28 13:04:03 -07:00
Jason Lowe 85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Xuan 28c4957fcc YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad 
'start' value. Contributed by Jason Lowe
 2017-09-27 15:46:18 -07:00
Chris Trezzo c87db8d154 YARN-7253. Shared Cache Manager daemon command listed as admin subcmd in yarn script. 2017-09-27 11:32:09 -07:00
Sunil G 8facf1f976 YARN-6871. Add additional deSelects params in RMWebServices#getAppReport. Contributed by Tanuj Nayak. 2017-09-27 14:37:32 +05:30
Jason Lowe 14fec04ed3 Revert "YARN-6570. No logs were found for running application, running" 
This reverts commit 7c732924a8.
 2017-09-26 10:38:03 -05:00
Rohith Sharma K S 06e5a7b5cf YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R. 2017-09-26 14:34:09 +05:30
Arun Suresh df800f6cf3 YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh) 2017-09-25 16:02:55 -07:00
Akira Ajisaka e928ee583c
YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. Contributed by Sen Zhao. 2017-09-26 02:03:30 +09:00
Daniel Templeton 164a063e74 YARN-6878. TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-09-22 23:53:32 +02:00
Junping Du 4002bf0a9e YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi. 2017-09-22 13:58:09 -07:00
Jason Lowe bfd1a72ba8 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 2017-09-21 17:41:34 -05:00
Junping Du e5e1851d80 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:01:16 -07:00
Yufei Gu a92ef030a2 YARN-7045. Remove FSLeafQueue#addAppSchedulable. (Contributed by Sen Zhao via Yufei Gu) 2017-09-21 12:31:07 -07:00
Wangda Tan 263e2c692a YARN-6991. "Kill application" button does not show error if other user tries to kill the application for secure cluster. (Suma Shivaprasad via wangda) 
Change-Id: I7e7894b24609709f89064ee5882f055dbb09080b
 2017-09-21 12:00:53 -07:00
Jason Lowe 10d7493587 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger 2017-09-21 08:38:08 -05:00
Sangjin Lee 8b336632ac YARN-6771. Use classloader inside configuration class to make new 
classes. Contributed by Jongyoul Lee.
 2017-09-20 22:22:43 -07:00
Yufei Gu a9019e1fb7 YARN-6499. Remove the doc about Schedulable#redistributeShare(). (Contributed by Chetna Chaudhari via Yufei Gu) 2017-09-19 18:28:31 -07:00
Junping Du 647b7527a9 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh. 2017-09-19 18:31:15 -07:00
Arun Suresh 56ef5279c1 YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh) 2017-09-18 15:49:31 -07:00
Andrew Wang 3cf3540f19 HADOOP-14835. mvn site build throws SAX errors. Contributed by Andrew Wang and Sean Mackrory. 2017-09-18 15:13:42 -07:00
Xuan 7c732924a8 YARN-6570. No logs were found for running application, running 
container. Contributed by Junping Du
 2017-09-18 14:04:05 -07:00
Arun Suresh 29dd55153e YARN-7199. Fix TestAMRMClientContainerRequest.testOpportunisticAndGuaranteedRequests. (Botong Huang via asuresh) 2017-09-18 11:26:44 -07:00
Jason Lowe a4f9c7c924 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 2017-09-18 10:16:09 -05:00
Wangda Tan e81596d06d YARN-7172. ResourceCalculator.fitsIn() should not take a cluster resource parameter. (Sen Zhao via wangda) 
Change-Id: Icc3670c9381ce7591ca69ec12da5aa52d3612d34
 2017-09-17 21:20:43 -07:00
Wangda Tan 38c14ef8d8 YARN-7149. Cross-queue preemption sometimes starves an underserved queue. (Eric Payne via wangda) 
Change-Id: Ib269991dbebce160378e8372ee6d24849c4a5ed6
(cherry picked from commit 3dfa937a1fadfc62947755872515f549b3b15e6a)
 2017-09-15 21:29:39 -07:00
Wangda Tan 958e8c0e25 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda) 
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
 2017-09-15 21:29:31 -07:00
Junping Du 1a84c24b0c YARN-7174. Add retry logic in LogsCLI when fetch running application logs. Contributed by Xuan Gong. 2017-09-15 15:33:24 -07:00
Yufei Gu 09b476e6da YARN-6612. Update fair scheduler policies to be aware of resource types. (Contributed by Daniel Templeton via Yufei Gu) 2017-09-14 11:23:37 -07:00
Arun Suresh 66ca0a6540 YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh 4f8194430f YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh 864fbacd45 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh) 2017-09-14 08:48:39 -07:00
Junping Du c92c1d521e YARN-7194. Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat. Contributed by Xuan Gong. 2017-09-13 22:16:06 -07:00
Robert Kanter bb34ae9554 YARN-7146. Many RM unit tests failing with FairScheduler (rkanter) 2017-09-13 15:44:54 -07:00
Eric Payne 3860be7961 YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe. 2017-09-13 14:41:55 -05:00
Eric Payne a3c44195be Revert 'YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe.' 
This reverts commit 729d05f529.
 2017-09-13 14:38:58 -05:00
Eric Payne 729d05f529 YARN-4727. Unable to override the /home/ericp/run/conf/ env variable for container. Contributed by Jason Lowe. 2017-09-13 14:29:08 -05:00
Rohith Sharma K S 5324388cf2 YARN-7157. Add admin configuration to filter per-user's apps in secure cluster. Contributed by Sunil G. 2017-09-13 23:36:47 +05:30
Arun Suresh a1c398e0dc YARN-7185. [Addendum patch] Minor javadoc and checkstyle fix. 2017-09-12 17:03:32 -07:00
Yufei Gu 82c5dd1d50 YARN-7057. FSAppAttempt#getResourceUsage doesn't need to consider resources queued for preemption. (Contributed by Karthik via Yufei Gu) 2017-09-12 16:13:39 -07:00
Arun Suresh 2ae72692fc YARN-7185. ContainerScheduler should only look at availableResource for GUARANTEED containers when OPPORTUNISTIC container queuing is enabled. (Wangda Tan via asuresh) 2017-09-12 16:10:08 -07:00
Sunil G da0b6a354b YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Daniel Templeton bf2b687412 YARN-7136. Additional Performance Improvement for Resource Profile Feature 
(Contributed by Wangda Tan via Daniel Templeton)
 2017-09-12 09:19:12 -07:00
Wangda Tan 6b88cd1901 YARN-7056. Document Resource Profiles feature. (Sunil G via wangda) 
Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e
 2017-09-12 09:19:12 -07:00
Sunil G 805095496d YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R. 2017-09-12 09:19:12 -07:00
Wangda Tan 53df3eac50 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda) 
Change-Id: I88928a747ee3eec17bc76fb71e9aaa632d091f0f
 2017-09-12 09:19:12 -07:00
Wangda Tan e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Sunil G a333ba54e3 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Wangda Tan 39240b61a1 YARN-7043. Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
 2017-09-12 09:19:12 -07:00
Sunil G 758b771943 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin. 2017-09-12 09:19:12 -07:00
Wangda Tan df3855541a YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
 2017-09-12 09:19:12 -07:00
Wangda Tan b1fe3a222e YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda) 
Change-Id: I8e40f704b6fcdd5b14faa9548a27986501044fa1
 2017-09-12 09:19:12 -07:00
Sunil G a6989af1d9 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Sunil G a0030c8c6f YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton. 2017-09-12 09:19:12 -07:00
Daniel Templeton 6746f8cadb YARN-6908. ResourceProfilesManagerImpl is missing @Overrides on methods 
(Contributed by Sunil G. via Daniel Templeton)
 2017-09-12 09:19:12 -07:00
Sunil G 2b51b262ab YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G. 2017-09-12 09:19:12 -07:00
Yufei Gu d5e9939ebb YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu) 2017-09-12 09:19:11 -07:00
Daniel Templeton 5c4ab4c291 YARN-6935. [YARN-3926] ResourceProfilesManagerImpl.parseResource() has no need of the key parameter 
(Contributed by Manikandan R via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Daniel Templeton 3aeaafecb8 YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Sunil G 4e5632d28e YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang. 2017-09-12 09:19:11 -07:00
Sunil G ca2808d8bb YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G 58da54640e YARN-6445. [YARN-3926] Performance improvements in resource profile branch with respect to SLS. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G dae65f3bef YARN-6232. Update resource usage and preempted resource calculations to take into account all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G 7805deed48 YARN-5588. [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Arun Suresh 6708ac3301 YARN-5587. Add support for resource profiles. (vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev c2032e251e YARN-5708. Implement APIs to get resource profiles from the RM. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev a9b4426302 YARN-5707. Add manager class for resource profiles. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Rohith Sharma K S 239c1824a0 YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Rohith Sharma K S 9e4ba6aff5 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev 759114b006 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Arun Suresh 7ba698997b YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev ceb12c59a3 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Wangda Tan 32c91223f1 YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 2017-09-12 09:19:09 -07:00
Wangda Tan 1bbab7c157 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 2017-09-12 09:19:09 -07:00
Wangda Tan e74d1be04b YARN-7173. Container update RM-NM communication fix for backward compatibility. (Arun Suresh via wangda) 
Change-Id: I1c39ed5c59dee739ba5044b61b3ef5ed203b79c1
 2017-09-11 20:52:08 -07:00
Haibo Chen fa531788fd YARN-7132. FairScheduler.initScheduler() contains a surprising unary plus. (Yeliang Cang via Haibo Chen) 2017-09-11 20:42:49 -07:00
Haibo Chen 661f5eb0c6 YARN-7128. The error message in TimelineSchemaCreator is not enough to find out the error. (Jinjiang Ling via Haibo Chen) 2017-09-11 16:20:20 -07:00
Daniel Templeton 6651cbcc72 YARN-6622. Document Docker work as experimental (Contributed by Varun Vasudev) 2017-09-11 16:14:31 -07:00
Daniel Templeton 07951b586e Revert "YARN-6022. Document Docker work as experimental (Contributed by Varun Vasudev)" 
This reverts commit 66be6c848b.
 2017-09-11 16:14:18 -07:00
Daniel Templeton 66be6c848b YARN-6022. Document Docker work as experimental (Contributed by Varun Vasudev) 2017-09-11 15:46:23 -07:00
Yufei Gu 5ab21dfe9c YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu) 2017-09-10 23:17:59 -07:00
Sunil G 722ee84194 YARN-7163. RMContext need not to be injected to webapp and other Always Running services. Contributed by Rohith Sharma K S. 2017-09-11 10:28:10 +05:30
Junping Du 3fddabc2fe YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong. 2017-09-08 15:16:19 -07:00
Haibo Chen 8edc60531f HADOOP-14847. Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc. (Bharat Viswanadham via Haibo Chen) 2017-09-08 14:33:11 -07:00
Arun Suresh 1f53ae7972 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh) 2017-09-08 09:24:05 -07:00
Rohith Sharma K S 4a83170be4 YARN-7140. CollectorInfo should have Public visibility. Contributed by Varun Saxena. 2017-09-08 20:23:16 +05:30
Rohith Sharma K S 0bfb3a2566 YARN-7130. ATSv2 documentation changes post merge. Contributed by Varun Saxena. 2017-09-08 20:21:50 +05:30
Sunil G ab8368d2e0 YARN-7104. Improve Nodes Heatmap in new YARN UI with better color coding. Contributed by Da Ding. 2017-09-08 16:16:01 +05:30
Subru Krishnan fa6137501c YARN-5330. SharingPolicy enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-09-07 19:07:17 -07:00
Sunil G 56d93d2e39 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S. 2017-09-08 07:15:17 +05:30
Jason Lowe b0b535d9d5 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:17:03 -05:00
Wangda Tan f155ab7cfa YARN-7033. Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan) 
Change-Id: Iffd18bb95debe1c8cc55e30abc1d8f663e9d0e30
 2017-09-07 14:13:37 -07:00
Arun Suresh c41118a7f8 YARN-6978. Add updateContainer API to NMClient. (Kartheek Muthyala via asuresh) 2017-09-07 10:23:12 -07:00
Sunil G 6f101e7df1 YARN-6992. Kill application button is visible even if the application is FINISHED in RM UI. Contributed by Suma Shivaprasad. 2017-09-07 13:16:20 +05:30
Subru Krishnan b6e7d13696 YARN-5328. Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-06 16:46:01 -07:00
Subru Krishnan e3345e985b Revert "Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem." 
This reverts commit 7996eca7dc.
 2017-09-06 16:39:23 -07:00
Junping Du 22de9449f8 YARN-7144. Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader. Contributed by Xuan Gong. 2017-09-06 14:53:31 -07:00
Jason Lowe 704267cb49 YARN-7164. TestAMRMClientOnRMRestart fails sporadically with bind address in use. Contributed by Jason Lowe 2017-09-06 15:23:49 -05:00
Junping Du 63231a2a30 YARN-7148. TestLogsCLI fails in trunk and branch-2 and javadoc error. Contributed by Xuan Gong. 2017-09-06 13:19:34 -07:00
Sunil G ed162b7aa0 YARN-7022. Improve click interaction in queue topology in new YARN UI. Contributed by Abdullah Yousufi. 2017-09-05 08:06:43 +05:30
Varun Saxena ef87d34a0e YARN-7152. [ATSv2] Registering timeline client before AMRMClient service init throw exception (Rohith Sharma K S via Varun Saxena) 2017-09-04 20:11:02 +05:30
Sunil G 275980bb1e YARN-7075. Better styling for donut charts in new YARN UI. Contributed by Da Ding. 2017-09-02 08:08:59 +05:30
Subru Krishnan 7996eca7dc Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-01 15:16:40 -07:00
Oleg Danilov c5281a85e1
YARN-7023. Incorrect ReservationId.compareTo() implementation. 
This closes #262

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2017-09-01 21:33:31 +02:00
Andrew Wang 0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Jason Lowe 99a7f5d451 YARN-7120. CapacitySchedulerPage NPE in "Aggregate scheduler counts" section. Contributed by Eric Payne 2017-09-01 11:09:14 -05:00
Sunil G a756704f5a YARN-7071. Add vcores and number of containers in new YARN UI node heat map. Contributed by Abdullah Yousufi. 2017-09-01 20:46:39 +05:30
bibinchundatt dcd0bedcc8 YARN-7141. Move logging APIs to slf4j in timelineservice after ATSv2 merge. Contributed by Varun Saxena 2017-09-01 11:29:16 +05:30
Allen Wittenauer 0adc3a0533
YARN-6721. container-executor should have stack checking 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-31 19:39:31 -07:00
Allen Wittenauer 190410085b
HADOOP-14364. refresh changelog/release notes with newer Apache Yetus build 
Signed-off-by: Andrew Wang <wang@apache.org>
 2017-08-31 19:06:49 -07:00
Junping Du 91cc070d67 YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-08-31 16:41:43 -07:00
Subru Krishnan bac4e8cca8 YARN-7095. Federation: routing getNode/getNodes/getMetrics REST invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 2017-08-31 15:05:41 -07:00
Daniel Templeton d4417dae4f YARN-6780. ResourceWeights.toString() cleanup (Contributed by weiyuan via Daniel Templeton) 2017-08-31 09:35:01 -07:00
Sunil G f9e0cc8cdc YARN-7116. CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage. Contributed by Wangda Tan. 2017-08-31 18:42:01 +05:30
Allen Wittenauer 71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Robert Kanter ce79f7be29 YARN-7094. Document the current known issue with server-side NM graceful decom (rkanter) 2017-08-30 18:04:55 -07:00
Daniel Templeton cc23514aba YARN-7115. Move BoundedAppender to org.hadoop.yarn.util pacakge 
(Contributed by Jian He via Daniel Templeton)
 2017-08-30 17:26:13 -07:00
Haibo Chen a20e7105ea YARN-6868. Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml. (Ray Chiang via Haibo Chen) 2017-08-30 09:14:59 -07:00
Varun Saxena 3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Varun Saxena 512068a01f YARN-6905. Addendum to fix TestTimelineReaderWebServicesHBaseStorage due to missing FastNumberFormat 2017-08-30 11:29:54 +05:30
Varun Saxena 16ba4f544f MAPREDUCE-6838. Addendum to fix code comment 2017-08-30 11:29:54 +05:30
Jian He 08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Vrushali C bea3e4df76 YARN-6047 Documentation updates for TimelineService v2 (Contributed by Rohith Sharma) 2017-08-30 11:29:54 +05:30
Haibo Chen 9b08f365d7 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-08-30 11:29:54 +05:30
Varun Saxena b2efebdd07 YARN-6861. Reader API for sub application entities (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:54 +05:30
Jian He 7fd6ae2479 YARN-6134. [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Rohith Sharma K S e276c75ec1 YARN-7041. Nodemanager NPE running jobs with security off. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Rohith Sharma K S 32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Jian He b664569586 YARN-7006. [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jason Lowe d5ff965fee YARN-6820. Restrict read access to timelineservice v2 data. Contributed by Vrushali C 2017-08-30 11:29:53 +05:30
Varun Saxena 60765aff9b YARN-6874. Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite (Vrushali C via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 354be99dbf YARN-6133. [ATSv2 Security] Renew delegation token for app automatically if an app collector is active. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 7594d1de7b YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Varun Saxena 9f6540535d YARN-6734. Ensure sub-application user is extracted & sent to timeline service (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 3fb71b1393 YARN-6888. Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S a990ff70c2 YARN-6733. Add table for storing sub-application entities. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Vrushali C 61136d03f2 YARN-6850 Ensure that supplemented timestamp is stored only for flow run metrics (Contributed by Varun Saxena via Vrushali C) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 70078e91e3 YARN-4455. Support fetching metrics by time range. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 660413165a YARN-6801. NPE in RM while setting collectors map in NodeHeartbeatResponse. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Jian He ac7f52df83 YARN-5648. [ATSv2 Security] Client side changes for authentication. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jian He d3f11e3f13 YARN-6638. [ATSv2 Security] Timeline reader side changes for loading auth filters and principals. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jian He 879de51206 YARN-5647. [ATSv2 Security] Collector side changes for loading auth filters and principals. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
vrushali 24447b3626 YARN-6316 Provide help information and documentation for TimelineSchemaCreator (Contributed by Haibo Chen via Vrushali C) 2017-08-30 11:29:52 +05:30
Varun Saxena f6a51dce18 YARN-6658. Remove columnFor() methods of Columns in HBaseTimeline backend (Haibo Chen via Varun Saxena) 2017-08-30 11:29:52 +05:30
Haibo Chen a8f082a180 YARN-6555. Store application flow context in NM state store for work-preserving restart. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 47474fffac)
 2017-08-30 11:29:52 +05:30
Varun Saxena b87b72b40a YARN-6146. Add Builder methods for TimelineEntityFilters (Haibo Chen via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena 44999aabc2 YARN-6318. timeline service schema creator fails if executed from a remote machine (Sangjin Lee via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena 18b3a80df7 YARN-6237. Move UID constant to TimelineReaderUtils (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena c3bd8d6ad3 YARN-6256. Add FROM_ID info key for timeline entities in reader response (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena 8bb2646595 YARN-6027. Support fromid(offset) filter for /flows API (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Sangjin Lee cf30b3b914 YARN-6253. FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp. Contributed by Haibo Chen. 2017-08-30 11:29:52 +05:30
Varun Saxena ff43b8db7a YARN-6069. CORS support in timeline v2 (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Sangjin Lee ccb38c1991 YARN-5928. Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers. Contributed by Haibo Chen. 2017-08-30 11:29:52 +05:30
Sangjin Lee 580d884913 YARN-6094. Update the coprocessor to be a dynamically loaded one. Contributed by Vrushali C. 2017-08-30 11:29:52 +05:30
Varun Saxena 5e0acee75e Addendum for YARN-6064. Support fromId for flowRuns and flow/flowRun apps REST API's 2017-08-30 11:29:52 +05:30
Varun Saxena 6f65cf27bb YARN-6064. Support fromId for flowRuns and flow/flowRun apps REST API's (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena 71847ed44d YARN-5378. Accommodate app-id->cluster mapping (Sangjin Lee via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee 7f54ac48c6 YARN-5980. Update documentation for single node hbase deploy. Contributed by Vrushali C. 2017-08-30 11:29:51 +05:30
Varun Saxena 02a9710a09 YARN-5585. [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee 2556c012e3 YARN-5925. Extract hbase-backend-exclusive utility methods from TimelineStorageUtil. Contributed by Haibo Chen. 2017-08-30 11:29:51 +05:30
Sangjin Lee ecbc8d4816 YARN-5922. Remove direct references of HBaseTimelineWriter/Reader in core ATS classes. Contributed by Haibo Chen. 
(cherry picked from commit a5a55a54ab)
 2017-08-30 11:29:51 +05:30
Sangjin Lee 4481561e4a YARN-5739. Provide timeline reader API to list available timeline entity types for one application. Contributed by Li Lu. 2017-08-30 11:29:51 +05:30
Sangjin Lee 092fead5d9 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 2017-08-30 11:29:51 +05:30
Varun Saxena c92a7ab31c YARN-3359. Addendum for Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena 91d3443806 YARN-3359. Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee 643a20a358 YARN-5265. Make HBase configuration for the timeline service configurable. Contributed by Joep Rottinghuis. 2017-08-30 11:29:51 +05:30
Sangjin Lee 05ff04439e YARN-5715. Introduce entity prefix for return and sort order. Contributed by Rohith Sharma K S. 2017-08-30 11:29:51 +05:30
Varun Saxena 9bb5cc3d49 YARN-3649. Allow configurable prefix for hbase table names like prod, exp, test etc (Vrushali C via Varun Saxena) 2017-08-30 11:29:50 +05:30
Sangjin Lee 78b7e070d8 YARN-5638. Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery. Contributed by Li Lu. 2017-08-30 11:29:50 +05:30
Li Lu e72e6260da YARN-5229. Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl. (Vrushali C via gtcarrera9) 2017-08-30 11:29:50 +05:30
Sangjin Lee 278bd6fcff YARN-5156. YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state (Vrushali C via sjlee) 2017-08-30 11:29:50 +05:30
Rohith Sharma K S 4cae120c61 YARN-6982. Potential issue on setting AMContainerSpec#tokenConf to null before app is completed. Contributed by Manikandan R. 2017-08-30 10:45:11 +05:30
Jian He 8201ed8009 YARN-6756. ContainerRequest#executionTypeRequest causes NPE. Contributed by Jian He 2017-08-29 20:35:53 -07:00
Wangda Tan f59332b97b YARN-5219. When an export var command fails in launch_container.sh, the full container launch should fail. (Sunil G via wangda) 
Change-Id: Iaa6b978bb89482e9d1d77ba57f4adfdc48e39a3c
 2017-08-29 15:29:22 -07:00
Junping Du ad45d19998 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang. 2017-08-29 15:16:23 -07:00
Carlo Curino cc8893edc0 YARN-7010. Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-29 14:53:09 -07:00
Jason Lowe 1f6dc4ee9a YARN-5816. TestDelegationTokenRenewer#testCancelWithMultipleAppSubmissions is still flakey. Contributed by Robert Kanter 2017-08-29 09:52:24 -05:00
Sunil G 7ddc7aa26e YARN-6386. Show decommissioning nodes in new YARN UI. Contributed by Elek Marton. 2017-08-29 16:51:14 +05:30
Junping Du 312b1fd9da YARN-7076. yarn application -list -appTypes is not working. Contributed by Jian He. 2017-08-28 18:26:51 -07:00
Wangda Tan 51881a863c YARN-7112. TestAMRMProxy is failing with invalid request. (Jason Lowe via wangda) 
Change-Id: I432f4523921666ac218fcf9936f826a0d8d635e0
 2017-08-28 11:37:35 -07:00
Yufei Gu d5bc958279 YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu) 2017-08-28 10:09:46 -07:00
Sunil G 02599bda04 YARN-7051. Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin. Contributed by Eric Payne. 2017-08-28 12:22:56 +05:30
Li Lu 3bee7046c5 YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9) 2017-08-25 22:43:25 -07:00
Eric Payne e864f81471 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe. 2017-08-25 15:37:54 -05:00
Jason Lowe 39a9dc8e4a YARN-7052. RM SchedulingMonitor gives no indication why the spawned thread crashed. Contributed by Eric Payne 2017-08-25 15:31:29 -05:00
Arpit Agarwal 8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Jason Lowe 3a4e861169 YARN-6640. AM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2017-08-25 09:16:17 -05:00
Yufei Gu 9e2699ac2c YARN-7049. FSAppAttempt preemption related fields have confusing names. (Contributed by Karthik Kambatla via Yufei Gu) 2017-08-24 17:52:41 -07:00
Junping Du c2cb7ea1ef YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 2017-08-24 13:36:49 -07:00
bibinchundatt de0cba700b YARN-7074. Fix NM state store update comment. Contributed by Botong Huang 2017-08-24 16:34:38 +05:30
Junping Du 652dd434d9 YARN-7090. testRMRestartAfterNodeLabelDisabled get failed when CapacityScheduler is configured. Contributed by Wangda Tan. 2017-08-23 18:06:29 -07:00
Wangda Tan f49843a988 YARN-6251. Do async container release to prevent deadlock during container updates. (Arun Suresh via wangda) 
Change-Id: I6c67d20c5dd4d22752830ebf0ed2340824976ecb
 2017-08-23 09:56:20 -07:00
Subru Krishnan 4249172e14 YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru). 2017-08-22 19:20:57 -07:00
Jason Lowe 657dd59cc8 YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. Contributed by Daryn Sharp 2017-08-22 13:16:24 -05:00
Jason Lowe 3efcd51c3b YARN-2416. InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time. Contributed by Jonathan Eagles 2017-08-22 12:56:09 -05:00
Akira Ajisaka d5ff57a08f
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-22 17:14:12 +09:00
Arun Suresh ae8fb13b31 YARN-6923. Metrics for Federation Router. (Giovanni Matteo Fumarola via asuresh) 2017-08-21 22:50:24 -07:00
Arun Suresh 75abc9a8e2 YARN-5603. Metrics for Federation StateStore. (Ellen Hui via asuresh) 2017-08-21 22:43:08 -07:00
Arun Suresh 7a82d7bcea YARN-6979. [Addendum patch] Fixed classname and added javadocs. (Kartheek Muthyala via asuresh) 2017-08-20 10:24:05 -07:00
Arun Suresh 8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Wangda Tan 436c2638f9 YARN-6852. Native code changes to support isolate GPU devices by using CGroups. (wangda) 
Change-Id: I4869cc4d8ad539539ccba4bea5a178cacdb741ab
 2017-08-18 18:26:36 -07:00
Yufei Gu 8991f0baec YARN-6969. Clean up unused code in class FairSchedulerQueueInfo. (Larry Lo via Yufei Gu) 2017-08-18 14:38:44 -07:00
bibinchundatt e05fa3451d YARN-7007. NPE in RM while using YarnClient.getApplications(). Contributed by Lingfeng Su. 2017-08-18 20:28:50 +05:30
Jason Lowe ab1a8ae85f YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-17 15:50:14 -05:00
Sunil G f9a0e23381 YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad. 2017-08-17 15:07:15 +05:30
Subru Krishnan de462da04e YARN-6900. ZooKeeper based implementation of the FederationStateStore. (Íñigo Goiri via Subru). 2017-08-16 11:43:24 -07:00
Jason Lowe 14553061be YARN-7020. TestAMRMProxy#testAMRMProxyTokenRenewal is flakey. Contributed by Robert Kanter 2017-08-16 13:04:36 -05:00
Akira Ajisaka 588c190afd
YARN-6965. Duplicate instantiation in FairSchedulerQueueInfo. Contributed by Masahiro Tanaka. 2017-08-16 14:06:22 +09:00
Subru Krishnan 75dd866bfb HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru). 2017-08-15 16:53:59 -07:00
Nathan Roberts d265459024 YARN-7014. Fix off-by-one error causing heap corruption (Jason Lowe via nroberts) 2017-08-15 15:52:48 -05:00
Sunil G dadb0c2225 YARN-5146. Support for Fair Scheduler in new YARN UI. Contributed by Abdullah Yousufi. 2017-08-15 21:58:44 +05:30
Arun Suresh 4d7be1d857 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh) 2017-08-14 19:46:17 -07:00
Jason Lowe 5558792894 YARN-6917. Queue path is recomputed from scratch on every allocation. Contributed by Eric Payne 2017-08-14 15:31:34 -05:00
Jason Lowe 3325ef653d YARN-6987. Log app attempt during InvalidStateTransition. Contributed by Jonathan Eagles 2017-08-14 14:40:08 -05:00
Daniel Templeton 6b09c32705 YARN-6881. LOG is unused in AllocationConfiguration (Contributed by weiyuan via Daniel Templeton) 2017-08-14 11:55:33 -07:00
Vrushali C 608a06cca5 YARN-6905 Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat (Contributed by Haibo Chen) 2017-08-14 11:41:11 -07:00
Subru Krishnan 18f3603bce YARN-6996. Change javax.cache library implementation from JSR107 to Apache Geronimo. (Ray Chiang via Subru). 2017-08-14 11:10:00 -07:00
Jian He e2f6299f6f YARN-6959. RM may allocate wrong AM Container for new attempt. Contributed by Yuqi Wang 2017-08-14 10:51:30 -07:00
Lei Xu ce797a1706 HDFS-12221. Replace xcerces in XmlEditsVisitor. (Ajay Kumar via lei) 2017-08-14 10:27:47 -07:00
bibinchundatt d8f74c3964 YARN-6741. Deleting all children of a Parent Queue on refresh throws exception. Contributed by Naganarasimha G R. 2017-08-14 09:39:00 +05:30
Carlo Curino 28d97b79b6 YARN-6687. Validate that the duration of the periodic reservation is less than the periodicity. (subru via curino) 2017-08-11 16:58:04 -07:00
Carlo Curino cc59b5fb26 YARN-6896. Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-11 15:58:01 -07:00
Daniel Templeton 0996acde6c YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator 
(Contributed by Larry Lo via Daniel Templeton)
 2017-08-11 14:50:46 -07:00
Daniel Templeton 65364defb4 YARN-6967. Limit application attempt's diagnostic message size thoroughly 
(Contributed by Chengbing Liu via Daniel Templeton)
 2017-08-11 14:28:55 -07:00
Daniel Templeton c7680d4cc4 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces 
(Contributed by weiyuan via Daniel Templeton)
 2017-08-11 14:22:02 -07:00
Daniel Templeton 218588be77 YARN-6952. Enable scheduling monitor in FS (Contributed by Yufei Gu via Daniel Templeton) 2017-08-11 14:04:19 -07:00
Subru Krishnan bbbf0e2a41 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-11 13:58:45 -07:00
Karthik Kambatla 8c4b6d16a5 YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha) 2017-08-11 12:15:43 -07:00
Sunil G 8d953c2359 YARN-6631. Refactor loader.js in new Yarn UI. Contributed by Akhil P B. 2017-08-10 11:53:26 +05:30
Wangda Tan ec694145cf YARN-6033. Add support for sections in container-executor configuration file. (Varun Vasudev via wandga) 
Change-Id: Ibc6d2a959debe5d8ff2b51504149742449d1f1da
 2017-08-09 10:51:29 -07:00
Akira Ajisaka 63cfcb90ac
YARN-6958. Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice. Contributed by Yeliang Cang. 2017-08-09 23:58:22 +09:00
Akira Ajisaka 1a18d5e514
YARN-6515. Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager. Contributed by Naganarasimha G R. 2017-08-09 21:56:43 +09:00
Subru Krishnan ad2a350662 YARN-6970. Add PoolInitializationException as retriable exception in FederationFacade. (Giovanni Matteo Fumarola via Subru). 2017-08-08 16:48:29 -07:00
Daniel Templeton e0c24145d2 YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code 
(Contributed by Angela Wang via Daniel Templeton)
 2017-08-08 13:35:22 -07:00
Wangda Tan 1794de3ea4 YARN-6726. Fix issues with docker commands executed by container-executor. (Shane Kumpf via wangda) 
Change-Id: If1b1827345f98f0a49cc7e39d1ba41fbeed5e911
 2017-08-08 12:56:29 -07:00
Jian He acf9bd8b1d YARN-6890. Not display killApp button on UI if UI is unsecured but cluster is secured. Contributed by Junping Du 2017-08-08 11:09:38 -07:00
Daniel Templeton 47b145b9b4 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-08-08 10:33:26 -07:00
Akira Ajisaka 98912950b6
YARN-6961. Remove commons-logging dependency from hadoop-yarn-server-applicationhistoryservice module. Contributed by Yeliang Cang. 2017-08-08 19:38:58 +09:00
Arun Suresh 8d3fd81980 YARN-6920. Fix resource leak that happens during container re-initialization. (asuresh) 2017-08-07 18:59:25 -07:00
Subru Krishnan c61f2c4198 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru). 2017-08-07 16:58:29 -07:00
Wangda Tan adb84f34db YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
 2017-08-07 11:32:21 -07:00
Akira Ajisaka b0fbf17965
YARN-6957. Moving logging APIs over to slf4j in hadoop-yarn-server-sharedcachemanager. Contributed by Yeliang Cang. 2017-08-07 19:25:40 +09:00
Akira Ajisaka 839e077faf
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-07 18:56:00 +09:00
Sunil G 46b7054fa7 YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 2017-08-07 13:15:46 +05:30
Junping Du f44b349b81 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-04 16:03:56 -07:00
Arun Suresh 35dc782923 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 2017-08-03 21:15:40 -07:00
Subru Krishnan 0542e6f86e YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 2017-08-03 14:18:03 -07:00
Yufei Gu c617fe02b3 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 11:33:37 -07:00
Haibo Chen 293c74a81b YARN-6673 Add cpu cgroup configurations for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:56:51 -07:00
Haibo Chen c5d256c760 YARN-6674 Add memory cgroup settings for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:52:35 -07:00
Sunil G f64cfeaf61 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 2017-08-03 19:27:10 +05:30
Sean Mackrory 1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Eric Payne 8ce8672b6b YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 2017-08-02 12:53:22 -05:00
Yufei Gu 45535f8afa YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-02 09:25:19 -07:00
Eric Payne 48899134d2 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 2017-08-02 10:59:33 -05:00
Jian He f9139ac8f6 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:25 -07:00
Carlo Curino 9625a030de YARN-6853. Add MySql Scripts for FederationStateStore. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit 874ddbf0b5)
 2017-08-01 17:28:28 -07:00
Carlo Curino acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92ab)
 2017-08-01 17:28:28 -07:00
Subru Krishnan 86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec)
 2017-08-01 17:28:28 -07:00
Subru Krishnan 8737c2a8e9 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed2680)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 712e97d4cf Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5)
 2017-08-01 17:28:27 -07:00