Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,805 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3628 Commits

Author SHA1 Message Date
Jason Lowe 8013475d44 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-02-15 17:09:00 -06:00
Konstantinos Karanasos 0b489e564c YARN-7920. Simplify configuration for PlacementConstraints. Contributed by Wangda Tan. 2018-02-15 14:23:38 -08:00
Eric Payne bddfe42e2c Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit c5e6e3de1c.
 2018-02-14 14:29:20 -06:00
Wangda Tan 042ef2fa7b YARN-7789. Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator. (Zian Chen via wangda) 
Change-Id: I1f6a451fe16758def6f47c046a9b8a67ad7255af
 2018-02-14 23:11:10 +08:00
Eric Payne c5e6e3de1c YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-13 10:11:02 -06:00
Jason Lowe 5a1db60ab1 YARN-7914. Fix exit code handling for short lived Docker containers. Contributed by Shane Kumpf 2018-02-12 15:50:10 -06:00
Jason Lowe 3414fd1243 YARN-7917. Fix failing test TestDockerContainerRuntime#testLaunchContainerWithDockerTokens. Contributed by Shane Kumpf 2018-02-12 09:27:43 -06:00
Wangda Tan d02e42cee4 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
 2018-02-12 10:29:37 +08:00
Wangda Tan 60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Yufei Gu 1bc03ddf97 YARN-7655. Avoid AM preemption caused by RRs for specific nodes or racks. Contributed by Steven Rand. 2018-02-08 12:32:43 -08:00
Jian He eb2449d539 YARN-5428. Allow for specifying the docker client configuration directory. Contributed by Shane Kumpf 2018-02-08 11:35:30 -08:00
Jason Lowe 456705a07c YARN-7815. Make the YARN mounts added to Docker containers more restrictive. Contributed by Shane Kumpf 2018-02-07 13:09:08 -06:00
Billie Rinaldi 266da25c04 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit e14f03dfbf.
 2018-02-07 07:17:00 -08:00
Billie Rinaldi aa461f9091 YARN-7516. Add security check for trusted docker images. Contributed by Eric Yang 2018-02-07 07:09:45 -08:00
Robert Kanter 814d701d46 YARN-7841. Cleanup AllocationFileLoaderService's reloadAllocations method (snemeth via rkanter) 2018-02-06 14:36:49 -08:00
Anu Engineer 4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Weiwei Yang bff858e910 YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 2018-02-06 19:33:04 +08:00
Billie Rinaldi a196ee9362 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 2018-02-05 18:04:09 -08:00
Jason Lowe 60656bceca YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error. Contributed by Botong Huang 2018-02-05 17:16:04 -06:00
Konstantinos Karanasos 50723889cc YARN-7778. Merging of placement constraints defined at different levels. Contributed by Weiwei Yang. 2018-02-02 14:46:20 -08:00
Arun Suresh 6e5ba9366f YARN-7839. Modify PlacementAlgorithm to Check node capacity before placing request on node. (Panagiotis Garefalakis via asuresh) 2018-02-02 10:28:22 -08:00
Jason Lowe 682ea21f2b Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 12eaae383a.
 2018-02-01 12:14:09 -06:00
Jason Lowe 12eaae383a YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-01-31 11:37:54 -06:00
Arun Suresh d481344783 YARN-7822. Constraint satisfaction checker support for composite OR and AND constraints. (Weiwei Yang via asuresh) 2018-01-31 01:51:08 -08:00
Arun Suresh add993e26a YARN-6597. Add RMContainer recovery test to verify tag population in the AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh f8c5f5b237 YARN-7784. Fix Cluster metrics when placement processor is enabled. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh c23980c4f2 YARN-7795. Fix jenkins issues of YARN-6592 branch. (Sunil G via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 644afe5fd8 YARN-7807. Assume intra-app anti-affinity as default for scheduling request inside AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh a4c539fcdb YARN-7783. Add validation step to ensure constraints are not violated due to order in which a request is processed. (asuresh) 2018-01-31 01:30:17 -08:00
Sunil G 9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos adbe87abf8 YARN-7788. Factor out management of temp tags from AllocationTagsManager. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Arun Suresh 8bf7c44436 YARN-7763. Allow Constraints specified in the SchedulingRequest to override application level constraints. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 28fe7f3318 YARN-7774. Miscellaneous fixes to the PlacementProcessor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 8779a35742 YARN-7709. Remove SELF from TargetExpression type. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Wangda Tan 29d9e4d581 YARN-6619. AMRMClient Changes to use the PlacementConstraint and SchcedulingRequest objects. (Arun Suresh via wangda) 2018-01-31 01:30:17 -08:00
Arun Suresh a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 4eda58c136 YARN-7681. Double-check placement constraints in scheduling phase before actual allocation is made. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh bdba01f73b YARN-7682. Expose canSatisfyConstraints utility function to validate a placement against a constraint. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh a52d11fb8c YARN-7613. Implement Basic algorithm for constraint based placement. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh f9af15d659 YARN-7612. Add Processor Framework for Rich Placement Constraints. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 1efb2b6f25 YARN-6596. Introduce Placement Constraint Manager module. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 37f1a7b64f YARN-7653. Node group support for AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 06eb63e64b YARN-7669. API and interface modifications for placement constraint processor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 88d8d3f40b YARN-7670. Modifications to the ResourceScheduler API to support SchedulingRequests. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 801c0988b5 YARN-7522. Introduce AllocationTagsManager to associate allocation tags to nodes. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Sunil G 6463e10c72 YARN-7723. Avoid using docker volume --format option to run against to older docker releases. Contributed by Wangda Tan 2018-01-30 15:58:11 +05:30
Sunil G e9c72d04be YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 2018-01-29 20:44:38 +05:30
Jinjiang Ling 3400d0c535
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-30 00:00:57 +09:00
Jian He 443523f9c0 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-27 22:18:51 -08:00
Haibo Chen 649ef7ac33 YARN-7064. Use cgroup to get container resource utilization. (Miklos Szegedi via Haibo Chen) 2018-01-26 16:27:31 -08:00
Billie Rinaldi f2fa736f0a YARN-7797. Docker host network can not obtain IP address for RegistryDNS. Contributed by Eric Yang 2018-01-26 09:32:23 -08:00
Wangda Tan 75618c0dec YARN-7760. [UI2] Clicking 'Master Node' or link next to 'AM Node Web UI' under application's appAttempt page goes to OLD RM UI. (Vasudevan Skm / Sunil G via wangda) 
Change-Id: Ia7e2b955d3551ca43c6852fe253fd01983b56565
 2018-01-26 15:44:55 +08:00
Wangda Tan e0cfb0a31a YARN-7817. Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. (Sunil G via wangda) 
Change-Id: Ia7ceeabd82046645ddeaf487c763288f36cfbdee
 2018-01-26 15:43:27 +08:00
Miklos Szegedi 2e5865606b YARN-7814. Remove automatic mounting of the cgroups root directory into Docker containers. Contributed by Shane Kumpf. 2018-01-25 17:37:23 -08:00
Jason Lowe ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Sunil G 82cc6f6968 YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 2018-01-25 14:32:54 +05:30
Eric Yang 59828be197 YARN-7810. Update TestDockerContainerRuntime to test with current user credential. 
(contributed by Shane Kumpf)
 2018-01-24 21:24:19 -05:00
Zhe Zhang fa8cf4d1b4 YARN-7737. prelaunch.err file not found exception on container failure. Contributed by Keqiu Hu. 2018-01-24 12:54:54 -08:00
Jian He 55c32776b1 YARN-7801. AmFilterInitializer should addFilter after fill all parameters. Contributed by Wangda Tan 2018-01-24 11:17:12 -08:00
Rohith Sharma K S f3e33aeb80 Revert "YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma" 
This reverts commit ec8f47e7fa.
 2018-01-24 15:04:15 +05:30
Miklos Szegedi e7642a3e6f YARN-7796. Container-executor fails with segfault on certain OS configurations. Contributed by Gergo Repas. 2018-01-23 21:07:05 -08:00
Billie Rinaldi e307edcb47 YARN-7540 and YARN-7605. Convert yarn app cli to call yarn api services and implement doAs for Api Service REST API. Contributed by Eric Yang and Jian He 2018-01-23 17:54:39 -08:00
Eric Yang 97fe3cc187 YARN-7729. Add support for setting Docker PID namespace mode. (Contributed by Billie Rinaldi) 2018-01-22 16:33:38 -05:00
Yufei Gu 22ee6f77e5 YARN-7755. Clean up deprecation messages for allocation increments in FS config. Contributed by Wilfred Spiegelenburg. 2018-01-22 10:30:01 -08:00
Vrushali C ec8f47e7fa YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma 2018-01-19 15:34:40 -08:00
Haibo Chen 4aca4ff759 YARN-5094. some YARN container events have timestamp of -1. 2018-01-19 10:27:46 -08:00
Miklos Szegedi bc93ac229e YARN-7139. FairScheduler: finished applications are always restored to default queue. Contributed by Wilfred Spiegelenburg. 2018-01-18 17:43:47 -08:00
Billie Rinaldi 37f4696a9c YARN-7740. Fix logging for destroy yarn service cli when app does not exist and some minor bugs. Contributed by Jian He 2018-01-18 12:11:19 -08:00
Sunil G 06cceba1cb YARN-7738. CapacityScheduler: Support refresh maximum allocation for multiple resource types. Contributed by Wangda Tan. 2018-01-18 19:05:26 +05:30
Miklos Szegedi a68e445dc6 YARN-7717. Add configuration consistency for module.enabled and docker.privileged-containers.enabled. Contributed by Eric Badger. 2018-01-17 14:11:14 -08:00
Miklos Szegedi 41049ba5d1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-16 15:40:43 -08:00
Yufei Gu 370f1c6283 YARN-6486. FairScheduler: Deprecate continuous scheduling. (Contributed by Wilfred Spiegelenburg) 2018-01-16 14:46:55 -08:00
Robert Kanter 5ac109909a YARN-7479. TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk (ajisakaa via rkanter) 2018-01-16 10:16:42 -08:00
Rohith Sharma K S d09058b2fd YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 2018-01-16 07:58:29 +05:30
Miklos Szegedi 2dcfc1876e YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 2018-01-12 17:00:01 -08:00
Wangda Tan edcc3a95d5 YARN-7468. Provide means for container network policy control. (Xuan Gong via wangda) 
Change-Id: I73678c343f663412917758feef35d8308c216e76
 2018-01-12 16:14:10 -08:00
Eric Yang 3d65dbe032 YARN-5366. Improve signal handling and delete delay for Docker on Yarn. 
(Contributed by Shane Kumpf)
 2018-01-12 13:41:27 -05:00
Sunil G 128d773a23 YARN-7727. Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector. Contributed by Prabhu Joseph. 2018-01-12 15:41:19 +05:30
Miklos Szegedi bc285da107 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 2018-01-11 09:40:09 -08:00
Sunil G afd8caba27 YARN-7722. Rename variables in MockNM, MockRM for better clarity. Contributed by Lovekesh bansal 2018-01-10 20:51:34 +05:30
Miklos Szegedi 55066cc53d YARN-7689. TestRMContainerAllocator fails after YARN-6124. Contributed by Wilfred Spiegelenburg. 2018-01-09 16:59:39 -08:00
Haibo Chen 8ee7080e5d YARN-7665. Allow FS scheduler state dump to be turned on/off separately from FS debug log. (Wilfred Spiegelenburg via Haibo Chen) 2018-01-09 16:45:35 -08:00
Jason Lowe 90c86d6e09 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 2018-01-09 10:51:15 -06:00
Jason Lowe ebff4decfe YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 2018-01-09 09:52:59 -06:00
bibinchundatt b26e30ab16 YARN-7508. NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode. Contributed by Tao Yang. 2018-01-09 17:57:01 +05:30
Ray Chiang 59ab5da0a0 YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang) 2018-01-08 16:19:02 -08:00
Rohith Sharma K S c2d6fa3656 YARN-7699. queueUsagePercentage is coming as INF for getApp REST api call. Contributed by Sunil G. 2018-01-08 10:39:31 +05:30
Wangda Tan a81144daa0 YARN-7666. Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations. (Sunil G via wangda) 
Change-Id: I0fd826490f5160d47d42af2a9ac0bd8ec4e959dc
 2018-01-05 15:12:04 -08:00
Robert Kanter 2aa4f0a559 YARN-7645. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers is flakey with FairScheduler (rkanter) 2018-01-05 13:55:09 -08:00
Robert Kanter f8e7dd9b10 YARN-7557. It should be possible to specify resource types in the fair scheduler increment value (grepas via rkanter) 2018-01-05 11:15:06 -08:00
Sunil G 0c75d0634b YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:42:17 +05:30
Jason Lowe d795661868 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:15:52 -06:00
Robert Kanter 7a55044803 YARN-7622. Allow fair-scheduler configuration on HDFS (gphillips via rkanter) 2018-01-03 15:31:50 -08:00
Haibo Chen 2f6c038be6 YARN-7602. NM should reference the singleton JvmMetrics instance. 2018-01-03 09:41:26 -08:00
Rohith Sharma K S c9bf813c9a YARN-7692. Skip validating priority acls while recovering applications. Contributed by Sunil G. 2018-01-03 18:20:04 +05:30
Arun Suresh c0c7cce81d YARN-7691. Add Unit Tests for ContainersLauncher. (Sampada Dehankar via asuresh) 2018-01-02 22:03:00 -08:00
Miklos Szegedi 7f515f57ed YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 2018-01-02 18:03:04 -08:00
Miklos Szegedi b82049b4f0 YARN-7580. ContainersMonitorImpl logged message lacks detail when exceeding memory limits. Contributed by Wilfred Spiegelenburg. 2017-12-29 12:49:37 -08:00
Arun Suresh a55884c68e YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 2017-12-28 22:20:42 -08:00
Robert Kanter 382215c72b YARN-7577. Unit Fail: TestAMRestart#testPreemptedAMRestartOnRMRestart (miklos.szegedi@cloudera.com via rkanter) 2017-12-20 13:39:00 -08:00
Sunil G d62932c3b2 YARN-7032. [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Contributed by Rohith Sharma K S. 2017-12-20 11:31:15 +05:30
Eric Yang 94a2ac6b71 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 18:42:27 -05:00
Varun Saxena c0aeb666a4 YARN-7662. [ATSv2] Define new set of configurations for reader and collectors to bind (Rohith Sharma K S via Varun Saxena) 2017-12-19 22:29:24 +05:30
Jason Lowe 811fabdebe YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 2017-12-18 15:20:06 -06:00
Akira Ajisaka 001008958d
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:24:51 +09:00
Wangda Tan 44825f0960 YARN-7629. TestContainerLaunch# fails after YARN-7381. (Jason Lowe via wangda) 
Change-Id: Ia6a3f05c9a7e797d8190123d304ecc4e2b018e33
 2017-12-15 15:40:56 -08:00
Wangda Tan 631b5c2db7 YARN-5418. When partial log aggregation is enabled, display the list of aggregated files on the container log page. (Xuan Gong via wangda) 
Change-Id: I1befb0bbaeb89fb315bafe3e2f3379663f8cf1ec
 2017-12-15 15:38:36 -08:00
Rohith Sharma K S 09d996fdd4 YARN-7190. Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath. Contributed by Varun Saxena. 2017-12-15 21:50:28 +05:30
Sunil G 890d3d0645 YARN-7638. Unit tests related to preemption for auto created leaf queues feature.Contributed by Suma Shivaprasad. 2017-12-15 13:00:57 +05:30
Subru Krishnan 17ba74be29 YARN-7630. Fix AMRMToken rollover handling in AMRMProxy. Contributed by Botong Huang. 2017-12-14 14:03:55 -08:00
Chen Liang 46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Sunil G cb87e4dc92 YARN-7643. Handle recovery of applications in case of auto-created leaf queue mapping. Contributed by Suma Shivaprasad. 2017-12-13 22:49:58 +05:30
Weiwei Yang 7efc4f7688 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:11:41 +08:00
Jason Lowe 2abab1d7c5 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-12 16:04:15 -06:00
Jason Lowe 06f0eb2dce YARN-7625. Expose NM node/containers resource utilization in JVM metrics. Contributed by Weiwei Yang 2017-12-12 12:56:26 -06:00
Sunil G 8bb83a8f62 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues. Contributed by Suma Shivaprasad. 2017-12-12 15:20:59 +05:30
Sunil G 5c87fb2f62 YARN-7635. TestRMWebServicesSchedulerActivities fails in trunk. Contributed by Sunil G. 2017-12-12 15:08:18 +05:30
Sunil G 312ceebde8 YARN-7632. Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management. Contributed by Suma Shivaprasad. 2017-12-11 19:20:02 +05:30
Weiwei Yang a2edc4cbf5 YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 10:31:46 +08:00
Subru Krishnan 670e8d4ec7 YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 2017-12-08 15:39:18 -08:00
Wangda Tan 04b84da245 YARN-7443. Add native FPGA module support to do isolation with cgroups. (Zhankun Tang via wangda) 
Change-Id: Ic4b7f9f3e032986b8f955139c9fe4d3a6c818a53
 2017-12-08 15:18:22 -08:00
Wangda Tan adca1a72e4 YARN-7591. NPE in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I46689e530550ee0a6ac7a29786aab2cc1bdf314f
 2017-12-08 15:17:02 -08:00
Wangda Tan a8316df8c0 YARN-7520. Queue Ordering policy changes for ordering auto created leaf queues within Managed parent Queues. (Suma Shivaprasad via wangda) 
Change-Id: I482f086945bd448d512cb5b3879d7371e37ee134
 2017-12-08 15:11:28 -08:00
Wangda Tan f548bfffbd YARN-7420. YARN UI changes to depict auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: I8039d3772a191ddede132cd1f8b08a8ca2e275b7
 2017-12-08 15:10:47 -08:00
Wangda Tan b38643c9a8 YARN-7473. Implement Framework and policy for capacity management of auto created queues. (Suma Shivaprasad via wangda) 
Change-Id: Icca7805fe12f6f7fb335effff4b121b6f7f6337b
 2017-12-08 15:10:16 -08:00
Wangda Tan 74665e3a7d YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
 2017-12-08 15:07:56 -08:00
Sunil G 4db4a4a165 YARN-7575. NPE in scheduler UI when max-capacity is not configured. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Sunil G daa1cdd062 YARN-7564. Cleanup to fix checkstyle issues of YARN-5881 branch. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan 1012b901c8 YARN-7544. Use queue-path.capacity/maximum-capacity to specify absolute min/max resources. (Sunil G via wangda) 
Change-Id: I685341be213eee500f51e02f01c91def89391c17
 2017-12-07 18:56:54 -08:00
Wangda Tan b7b8cd5324 YARN-7538. Fix performance regression introduced by Capacity Scheduler absolute min/max resource refactoring. (Sunil G via wangda) 
Change-Id: Ic9bd7e599c56970fe01cb0e1bba6df7d1f77eb29
 2017-12-07 18:56:54 -08:00
Wangda Tan 7462c38277 YARN-7483. CapacityScheduler test cases cleanup post YARN-5881. (Sunil G via wangda) 
Change-Id: I9741a6baf5cb7352d05636efb6c0b24790e7589a
 2017-12-07 18:56:54 -08:00
Rohith Sharma K S e65ca92fb6 YARN-7482. Max applications calculation per queue has to be retrospected with absolute resource support. Contributed by Sunil G. 2017-12-07 18:56:54 -08:00
Wangda Tan 034b312d9f YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
 2017-12-07 18:56:54 -08:00
Wangda Tan aa3f62740f YARN-7332. Compute effectiveCapacity per each resource vector. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan d52627a7cb YARN-7254. UI and metrics changes related to absolute resource configuration. (Sunil G via wangda) 2017-12-07 18:56:54 -08:00
Wangda Tan 5e798b1a0d YARN-6471. Support to add min/max resource configuration for a queue. (Sunil G via wangda) 
Change-Id: I9213f5297a6841fab5c573e85ee4c4e5f4a0b7ff
 2017-12-07 18:56:54 -08:00
Weiwei Yang e411dd6666 YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-07 17:29:40 +08:00
Weiwei Yang 05c347fe51 YARN-7611. Node manager web UI should display container type in containers page. Contributed by Weiwei Yang. 2017-12-06 12:21:52 +08:00
Sunil G a957f1c60e YARN-7438. Additional changes to make SchedulingPlacementSet agnostic to ResourceRequest / placement algorithm. Contributed by Wangda Tan 2017-12-05 22:50:07 +05:30
Sunil G f9f317b702 YARN-7586. Application Placement should be done before ACL checks in ResourceManager. Contributed by Suma Shivaprasad. 2017-12-05 18:28:31 +05:30
Robert Kanter d8863fc16f YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 2017-12-04 13:14:55 -08:00
Arun Suresh 37ca416950 YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 2017-12-03 22:22:01 -08:00
Sunil G 81f6e46b2f YARN-6907. Node information page in the old web UI should report resource types. Contributed by Gergely Novák. 2017-12-04 11:27:23 +05:30
Sunil G 30f2646b15 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2017-12-04 10:45:07 +05:30
Jason Lowe 60f95fb719 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 2017-12-01 15:47:01 -06:00
Robert Kanter c83fe44917 YARN-4813. TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently (grepas via rkanter) 2017-12-01 12:18:13 -08:00
Wangda Tan 7225ec0ceb YARN-6507. Add support in NodeManager to isolate FPGA devices with CGroups. (Zhankun Tang via wangda) 
Change-Id: Ic9afd841805f1035423915a0b0add5f3ba96cf9d
 2017-12-01 10:50:49 -08:00
Sunil G 556aea3f36 YARN-7487. Ensure volume to include GPU base libraries after created by plugin. Contributed by Wangda Tan. 2017-12-01 13:36:28 +05:30
Wangda Tan a63d19d365 YARN-6124. Make SchedulingEditPolicy can be enabled / disabled / updated with RMAdmin -refreshQueues. (Zian Chen via wangda) 
Change-Id: Id93656f3af7dcd78cafa94e33663c78d410d43c2
 2017-11-30 15:57:22 -08:00
Wangda Tan c9a54aab6b YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2017-11-29 17:46:16 -08:00
Daniel Templeton 8498d287cd YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 2017-11-29 11:11:36 -08:00
Jason Lowe a2c7a73e33 YARN-6647. RM can crash during transitionToStandby due to InterruptedException. Contributed by Bibin A Chundatt 2017-11-28 11:15:44 -06:00
Yufei Gu d8923cdbf1 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 2017-11-27 14:31:52 -08:00
Jian He fedabcad42 YARN-6168. Restarted RM may not inform AM about all existing containers. Contributed by Chandni Singh 2017-11-27 10:19:58 -08:00
Yufei Gu 2bde3aedf1 YARN-7290. Method canContainerBePreempted can return true when it shouldn't. (Contributed by Steven Rand) 2017-11-24 23:32:46 -08:00
Wangda Tan 834e91ee91 YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
 2017-11-23 19:59:03 -08:00
Arun Suresh b46ca7e73b YARN-6483. Add nodes transitioning to DECOMMISSIONING state to the list of updated nodes returned to the AM. (Juan Rodriguez Hortala via asuresh) 2017-11-22 19:18:30 -08:00
Sunil G aab439593b YARN-7330. Add support to show GPU in UI including metrics. Contributed by Wangda Tan. 2017-11-23 07:54:20 +05:30
Yufei Gu 4cc9479dae YARN-7524. Remove unused FairSchedulerEventLog. (Contributed by Wilfred Spiegelenburg) 2017-11-22 14:18:36 -08:00
Eric Yang d42a336cfa YARN-5534. Allow user provided Docker volume mount list. (Contributed by Shane Kumpf) 2017-11-22 13:05:34 -05:00
yufei 03c311eae3 YARN-7513. Remove the scheduler lock in FSAppAttempt.getWeight() (Contributed by Wilfred Spiegelenburg) 2017-11-21 10:33:34 -08:00
Wangda Tan 0d781dd03b YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I51ae6c2ab7a3d1febdd7d8d0519b63a13295ac7d
 2017-11-20 11:48:15 -08:00
bibinchundatt b5b81a4f08 YARN-7489. ConcurrentModificationException in RMAppImpl#getRMAppMetrics. Contributed by Tao Yang. 2017-11-18 19:25:29 +05:30
Subru Krishnan d5f66888b8 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-17 17:39:06 -08:00
Eric Yang 0940e4f692 YARN-7218. Decouple YARN Services REST API namespace from RM. (Contributed by Eric Yang) 2017-11-17 12:28:12 -05:00
Wangda Tan 0987a7b8cb YARN-7419. CapacityScheduler: Allow auto leaf queue creation after queue mapping. (Suma Shivaprasad via wangda) 
Change-Id: Ia1704bb8cb5070e5b180b5a85787d7b9ca57ebc6
 2017-11-16 11:25:52 -08:00
Sunil G 61ace174cd YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 2017-11-16 22:34:23 +05:30
Daniel Templeton b246c54749 YARN-7414. FairScheduler#getAppWeight() should be moved into FSAppAttempt#getWeight() 
(Contributed by Soumabrata Chakraborty via Daniel Templeton)
 2017-11-15 10:03:29 -08:00
Junping Du e14f03dfbf YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 15:27:37 -08:00
Wangda Tan dd07038ffa YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2017-11-09 14:51:15 -08:00
Konstantinos Karanasos ac4d2b1081 YARN-7437. Rename PlacementSet and SchedulingPlacementSet. (Wangda Tan via kkaranasos) 2017-11-09 13:01:24 -08:00
Robert Kanter a2c150a736 YARN-7386. Duplicate Strings in various places in Yarn memory (misha@cloudera.com via rkanter) 2017-11-09 12:12:52 -08:00
Haibo Chen a1382a18df YARN-7388. TestAMRestart should be scheduler agnostic. 2017-11-09 10:49:50 -08:00
bibinchundatt 0a72c2f56c YARN-7454. RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup. Contributed by Jason Lowe. 2017-11-09 21:01:19 +05:30
Daniel Templeton 49b4c0b334 YARN-7458. TestContainerManagerSecurity is still flakey 
(Contributed by Robert Kanter via Daniel Templeton)

Change-Id: Ibb1975ad086c3a33f8af0b4f8b9a13c3cdca3f7d
 2017-11-08 17:31:14 -08:00
Daniel Templeton 0de10680b7 YARN-7166. Container REST endpoints should report resource types 
Change-Id: If9c2fe58d4cf758bb6b6cf363dc01f35f8720987
 2017-11-08 16:43:49 -08:00
Arun Suresh cb35a59589 YARN-7343. Add a junit test for ContainerScheduler recovery. (Sampada Dehankar via asuresh) 2017-11-08 08:14:02 -08:00
Arun Suresh a9c70b0e84 YARN-7453. Fix issue where RM fails to switch to active after first successful start. (Rohith Sharma K S via asuresh) 2017-11-08 08:00:53 -08:00
Daniel Templeton 8db9d61ac2 YARN-7401. Reduce lock contention in ClusterNodeTracker#getClusterCapacity() 2017-11-07 14:53:48 -08:00
Wangda Tan 13fa2d4e3e YARN-7394. Merge code paths for Reservation/Plan queues and Auto Created queues. (Suma Shivaprasad via wangda) 2017-11-06 21:38:24 -08:00
Haibo Chen 8f214dc4f8 YARN-7360. TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic. 2017-11-06 15:45:37 -08:00
Jian He a55d0738f1 YARN-7371. Added allocateRequestId in NMContainerStatus for recovery. Contributed by Chandni Singh 2017-11-06 13:30:20 -08:00
Jian He c723021579 YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Jian He 673c0db43c Revert "YARN-6626. Embed REST API service into RM. Contributed by Eric Yang" 
This reverts commit 63d1084e9781e0fee876916190b69f6242dd00e4.
 2017-11-06 13:30:17 -08:00
Jian He 9e677fa05c YARN-6626. Embed REST API service into RM. Contributed by Eric Yang 2017-11-06 13:30:17 -08:00
Billie Rinaldi ce74e64363 YARN-7210. Some NPE fixes in Registry DNS. Contributed by Jian He 2017-11-06 13:30:16 -08:00
Jian He bd96c4c235 Rebase onto latest trunk. minor conflicts 2017-11-06 13:30:13 -08:00
Billie Rinaldi 1888318c89 YARN-6903. Yarn-native-service framework core rewrite. Contributed by Jian He 2017-11-06 13:30:11 -08:00
Jian He 8d335e59cf YARN-6804. [yarn-native-services changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-11-06 13:30:10 -08:00
Billie Rinaldi ce05c6e981 YARN-6545. Followup fix for YARN-6405. Contributed by Jian He 2017-11-06 13:30:07 -08:00
bibinchundatt dcd99c4b9a Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 2017-11-06 22:39:10 +05:30
Inigo Goiri 6fc09beac4 YARN-7434. Router getApps REST invocation fails with multiple RMs. Contributed by Inigo Goiri. 2017-11-02 21:29:53 -07:00
Eric Payne e6ec02001f YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 2017-11-02 12:37:33 -05:00
Jason Lowe d00b6f7c1f YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 2017-11-02 09:37:17 -05:00
Rohith Sharma K S 940ffe3f9c addendum patch for YARN-7289. 2017-11-02 13:55:19 +05:30
Jian He 0cc98ae0ec YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2017-11-01 17:00:32 -07:00
Eric Yang 7a49ddfdde YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 2017-11-01 18:39:56 -04:00
Inigo Goiri 70f1a9470c YARN-7276 addendum to add timeline service depencies. Contributed by Inigo Goiri. 2017-11-01 13:26:37 -07:00
Daniel Templeton 9711b78998 YARN-7374. Improve performance of DRF comparisons for resource types in fair scheduler 2017-10-29 18:54:33 -07:00
Yufei Gu d4811c8cfa YARN-6747. TestFSAppStarvation.testPreemptionEnable fails intermittently. (Contributed by Miklos Szegedi) 2017-10-29 16:44:16 -07:00
Sunil G 9114d7a5a0 YARN-7224. Support GPU isolation for docker container. Contributed by Wangda Tan. 2017-10-29 11:08:44 +05:30
Daniel Templeton e62bbbca7a YARN-7397. Reduce lock contention in FairScheduler#getAppWeight() 2017-10-28 09:13:13 -07:00
Arun Suresh 9c5c68745e YARN-7299. Fix TestDistributedScheduler. (asuresh) 2017-10-27 23:08:18 -07:00
Inigo Goiri 8be5707067 YARN-7276. Federation Router Web Service fixes. Contributed by Inigo Goiri. 2017-10-27 16:46:05 -07:00
Jason Lowe 665bb147aa YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 2017-10-27 16:56:05 -05:00
Rohith Sharma K S 5c799ecf09 YARN-7289. Application lifetime does not work with FairScheduler. Contributed by Miklos Szegedi. 2017-10-27 22:46:38 +05:30
Sunil G 792388e1c0 YARN-7375. Possible NPE in RMWebapp when HA is enabled and the active RM fails. Contributed by Chandni Singh. 2017-10-27 20:53:57 +05:30
Wangda Tan 36e158ae98 YARN-7307. Allow client/AM update supported resource types via YARN APIs. (Sunil G via wangda) 
Change-Id: I14c5ea7252b7c17e86ab38f692b5f9d43196dbe0
 2017-10-26 20:15:19 -07:00
Robert Kanter b1de78619f YARN-7262. Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow (rkanter) 2017-10-26 17:47:32 -07:00
Robert Kanter 088ffee716 YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ Addendum (misha@cloudera.com via rkanter) 2017-10-26 15:50:14 -07:00
Subru Krishnan 25932da6d1 YARN-5516. Add REST API for supporting recurring reservations. (Sean Po via Subru). 2017-10-26 12:10:14 -07:00
Robert Kanter 2da654e34a YARN-7358. TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler (haibochen via rkanter) 2017-10-26 10:25:10 -07:00
Subru Krishnan 3fae675383 YARN-4827. Document configuration of ReservationSystem for FairScheduler. (Yufei Gu via Subru). 2017-10-25 15:07:50 -07:00
Haibo Chen d7f3737f3b YARN-7389. Make TestResourceManager Scheduler agnostic. (Robert Kanter via Haibo Chen) 2017-10-24 22:17:56 -07:00
Robert Kanter 03af442e76 YARN-7385. TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE (yufeigu via rkanter) 2017-10-24 13:36:50 -07:00
Carlo Curino 1c5c2b5dde YARN-7339. LocalityMulticastAMRMProxyPolicy should handle cancel request properly. (Botong Huang via curino) 2017-10-24 10:39:04 -07:00
Robert Kanter 025c656572 YARN-7382. NoSuchElementException in FairScheduler after failover causes RM crash (rkanter) 2017-10-24 10:21:44 -07:00
Robert Kanter 5da295a34e YARN-7320. Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_ (misha@cloudera.com via rkanter) 2017-10-23 17:56:56 -07:00
Daniel Templeton 9e77dc2bd1 YARN-7357. Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-10-23 13:51:19 -07:00
Eric Payne 921338cd86 YARN-4163: Audit getQueueInfo and getApplications calls 2017-10-23 11:43:41 -05:00
Haibo Chen 480187aebb YARN-7372. TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic is flaky. 2017-10-20 14:24:17 -07:00
Yufei Gu 0799fde35e YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu) 2017-10-20 10:00:13 -07:00
Eric Yang b61144a93d YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-20 12:02:06 -04:00
Yufei Gu 1f4cdf1068 YARN-4090. Make Collections.sort() more efficient by caching resource usage. (Contributed by Yufei Gu, Shilong Zhang and Xianyin Xin) 2017-10-20 01:32:20 -07:00
Yufei Gu 7b4b018780 YARN-7359. TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic. (Contributed by Haibo Chen) 2017-10-19 16:51:47 -07:00
Yufei Gu cbd2b73ef8 YARN-7294. TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair Scheduler. (Contributed by Miklos Szegedi) 2017-10-19 16:39:25 -07:00
Wangda Tan c1b08ba720 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2017-10-19 14:45:44 -07:00
Subru Krishnan 75323394fb YARN-7311. Fix TestRMWebServicesReservation parametrization for fair scheduler. (Yufei Gu via Subru). 2017-10-17 12:38:06 -07:00
Haibo Chen acabc657ff YARN-7341. TestRouterWebServiceUtil#testMergeMetrics is flakey. (Robert Kanter via Haibo Chen) 2017-10-17 10:15:53 -07:00
Robert Kanter 8a61525928 YARN-7308. TestApplicationACLs fails with FairScheduler (rkanter) 2017-10-16 15:34:32 -07:00
Nathan Roberts 4540ffd15f YARN-7333. container-executor fails to remove entries from a directory that is not writable or executable. Contributed by Jason Lowe. 2017-10-16 17:00:38 -05:00
Arun Suresh a50be1b8f4 YARN-7275. NM Statestore cleanup for Container updates. (Kartheek Muthyala via asuresh) 2017-10-16 13:12:15 -07:00
Wangda Tan 87ea1dff9c Addendum fix for: YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. (Wangda Tan) 
Change-Id: If1fe4a62e07b25e6f1b8ba803495da87e2cb2df6
 2017-10-14 10:41:58 -07:00
Rohith Sharma K S f4fb6695a3 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-13 15:17:05 +05:30
Carlo Curino 13fcfb3d46 YARN-7317. Fix overallocation resulted from ceiling in LocalityMulticastAMRMProxyPolicy. (contributed by Botong Huang via curino) 2017-10-12 10:38:58 -07:00
Sunil G fa5cfc68f3 YARN-6620. Add support in NodeManager to isolate GPU devices by using CGroups. Contributed by Wangda Tan. 2017-10-11 23:44:33 +05:30
Jason Lowe ebb34c7053 YARN-7082. TestContainerManagerSecurity failing in trunk. Contributed by Akira Ajisaka 2017-10-11 10:31:02 -05:00
Robert Kanter ec8bf9e48a YARN-7309. TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by FairScheduler (rkanter) 2017-10-10 10:18:42 -07:00
Wangda Tan 5cfbad68c0 YARN-7237. Cleanup usages of ResourceProfiles. (Wangda Tan) 
Change-Id: I959c8531bd48231956068cb7d2312f3235549f9c
(cherry picked from commit 8536db5a7549fa44d9f9e444b41bd5978b5c6ae1)
 2017-10-09 20:36:57 -07:00
Jonathan Hung 1d36b53ab6 YARN-7252. Removing queue then failing over results in exception 2017-10-09 11:11:23 -07:00
Jonathan Hung 09c5dfe937 YARN-7251. Misc changes to YARN-5734 2017-10-09 11:11:23 -07:00
Jonathan Hung d6622daaa3 YARN-7046. Add closing logic to configuration store 2017-10-09 11:11:23 -07:00
Wangda Tan ff39c0de20 YARN-6840. Implement zookeeper based store for scheduler configuration updates. (Jonathan Hung via wangda) 
Change-Id: I9debea674fe8c7e4109d4ca136965a1ea4c48bcc
 2017-10-09 11:11:23 -07:00
Xuan 4d8abd84f4 YARN-7024: Fix issues on recovery in LevelDB store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan 4c8b208adb YARN-6322: Disable queue refresh when configuration mutation is enabled. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan 74ba6ffa0b YARN-5947: Create LeveldbConfigurationStore class using Leveldb as backing store. Contributed by Jonathan Hung 2017-10-09 11:11:23 -07:00
Xuan 04f9e80bb2 YARN-5953 addendum: Move QueueConfigInfo and SchedConfUpdateInfo to package org.apache.hadoop.yarn.webapp.dao 2017-10-09 11:11:23 -07:00
Xuan 916bdbd6be YARN-5953:Create CLI for changing YARN configurations. (Jonathan Hung via xgong) 2017-10-09 11:11:23 -07:00
Xuan e566fd8b58 YARN-6575. Support global configuration mutation in MutableConfProvider. (Jonathan Hung via Xuan Gong) 2017-10-09 11:11:23 -07:00
Wangda Tan a4e6253046 YARN-5949. Add pluggable configuration ACL policy interface and implementation. (Jonathan Hung via wangda) 
Change-Id: Ib98e82ff753bede21fcab2e6ca9ec1e7a5a2008f
 2017-10-09 11:11:23 -07:00
Wangda Tan d8a5136b1f YARN-5952. Create REST API for changing YARN scheduler configurations. (Jonathan Hung via wangda) 2017-10-09 11:11:23 -07:00
Jonathan Hung ef686635c1 YARN-5948. Implement MutableConfigurationManager for handling storage into configuration store 2017-10-09 11:11:23 -07:00
Xuan e3579a8c3b YARN-5946: Create YarnConfigurationStore interface and 
InMemoryConfigurationStore class. Contributed by Jonathan Hung
 2017-10-09 11:11:23 -07:00
Jonathan Hung ef59cbe08a YARN-5951. Changes to allow CapacityScheduler to use configuration store 2017-10-09 11:11:22 -07:00
Daniel Templeton 6d6ca4c923 YARN-2612 addendum: fixed javadoc error. 2017-10-09 14:12:13 +02:00
Sunil G 8137254a1f YARN-7245. Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user. Contributed by Eric Payne. 2017-10-09 14:39:41 +05:30
Rohith Sharma K S 6fe6a72352 Revert "YARN-7180. Remove class ResourceType. Contributed by Sunil G." 
This reverts commit 6bd81ab054.
 2017-10-09 13:21:13 +05:30
Rohith Sharma K S 6bd81ab054 YARN-7180. Remove class ResourceType. Contributed by Sunil G. 2017-10-09 10:20:40 +05:30
Yufei Gu 72d22b753a YARN-7207. Cache the RM proxy server address. (Yufei Gu) 2017-10-06 09:49:11 -07:00
Yufei Gu 49ae538164 YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage(addendum). (Yufei Gu) 2017-10-06 00:08:25 -07:00
Yufei Gu 99292adcef YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage. (Yufei Gu) 2017-10-05 23:51:58 -07:00
Haibo Chen c071aad5da YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen) 2017-10-05 15:50:13 -07:00
Naganarasimha d0046bb5c6 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe. 2017-10-06 00:07:53 +05:30
Arun Suresh b733348dde YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 2017-10-05 09:58:04 -07:00
Sunil G 592bf2d550 YARN-7279. Fix typo in helper message of ContainerLauncher. Contributed by Elek, Marton. 2017-10-05 10:52:00 +05:30
Subru Krishnan e6e614e380 YARN-5329. Placement Agent enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-10-04 19:28:27 -07:00
Akira Ajisaka 2df1b2ac05
YARN-7044. TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails. 2017-10-05 06:07:32 +09:00
Akira Ajisaka acf5b880d8
YARN-6975. Moving logging APIs over to slf4j in hadoop-yarn-server-tests, hadoop-yarn-server-web-proxy and hadoop-yarn-server-router. Contributed by Yeliang Cang. 2017-10-04 14:53:41 +09:00
Akira Ajisaka c9b525bd94
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari. 2017-10-04 07:01:54 +09:00
Akira Ajisaka 4a87773718
YARN-6916. Moving logging APIs over to slf4j in hadoop-yarn-server-common. Contributed by Bibin A Chundatt and Akira Ajisaka. 2017-10-04 06:06:39 +09:00
Jian He 4111e6c781 YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. Contributed by Wangda Tan 2017-10-03 09:54:38 -07:00
Sidharta S 7eb8499996 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-02 19:04:49 -07:00
Subru Krishnan d4d2fd1acd YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru). 2017-10-02 18:14:44 -07:00
Carlo Curino ceca9694f9 YARN-3661. Basic Federation UI. (Contributed by Inigo Goiri via curino) 2017-10-02 13:03:32 -07:00
Wangda Tan febeead5f9 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda) 
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
 2017-09-29 16:39:46 -07:00
Eric Payne 7f6118f918 YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe. 2017-09-29 10:33:57 -05:00
Wangda Tan d3b1c63195 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda) 
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
 2017-09-28 16:41:09 -07:00
Subru Krishnan ca669f9f8b YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru). 2017-09-28 13:04:03 -07:00
Jason Lowe 85d81ae58e YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh 2017-09-28 14:10:15 -05:00
Sunil G 8facf1f976 YARN-6871. Add additional deSelects params in RMWebServices#getAppReport. Contributed by Tanuj Nayak. 2017-09-27 14:37:32 +05:30
Jason Lowe 14fec04ed3 Revert "YARN-6570. No logs were found for running application, running" 
This reverts commit 7c732924a8.
 2017-09-26 10:38:03 -05:00
Rohith Sharma K S 06e5a7b5cf YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R. 2017-09-26 14:34:09 +05:30
Arun Suresh df800f6cf3 YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh) 2017-09-25 16:02:55 -07:00
Daniel Templeton 164a063e74 YARN-6878. TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order 
(Contributed by Sen Zhao via Daniel Templeton)
 2017-09-22 23:53:32 +02:00
Junping Du 4002bf0a9e YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi. 2017-09-22 13:58:09 -07:00
Jason Lowe bfd1a72ba8 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 2017-09-21 17:41:34 -05:00
Junping Du e5e1851d80 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:01:16 -07:00
Yufei Gu a92ef030a2 YARN-7045. Remove FSLeafQueue#addAppSchedulable. (Contributed by Sen Zhao via Yufei Gu) 2017-09-21 12:31:07 -07:00
Wangda Tan 263e2c692a YARN-6991. "Kill application" button does not show error if other user tries to kill the application for secure cluster. (Suma Shivaprasad via wangda) 
Change-Id: I7e7894b24609709f89064ee5882f055dbb09080b
 2017-09-21 12:00:53 -07:00
Jason Lowe 10d7493587 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger 2017-09-21 08:38:08 -05:00
Yufei Gu a9019e1fb7 YARN-6499. Remove the doc about Schedulable#redistributeShare(). (Contributed by Chetna Chaudhari via Yufei Gu) 2017-09-19 18:28:31 -07:00
Junping Du 647b7527a9 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh. 2017-09-19 18:31:15 -07:00
Arun Suresh 56ef5279c1 YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh) 2017-09-18 15:49:31 -07:00
Xuan 7c732924a8 YARN-6570. No logs were found for running application, running 
container. Contributed by Junping Du
 2017-09-18 14:04:05 -07:00
Jason Lowe a4f9c7c924 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh 2017-09-18 10:16:09 -05:00
Wangda Tan e81596d06d YARN-7172. ResourceCalculator.fitsIn() should not take a cluster resource parameter. (Sen Zhao via wangda) 
Change-Id: Icc3670c9381ce7591ca69ec12da5aa52d3612d34
 2017-09-17 21:20:43 -07:00
Wangda Tan 38c14ef8d8 YARN-7149. Cross-queue preemption sometimes starves an underserved queue. (Eric Payne via wangda) 
Change-Id: Ib269991dbebce160378e8372ee6d24849c4a5ed6
(cherry picked from commit 3dfa937a1fadfc62947755872515f549b3b15e6a)
 2017-09-15 21:29:39 -07:00
Wangda Tan 958e8c0e25 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda) 
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
 2017-09-15 21:29:31 -07:00
Yufei Gu 09b476e6da YARN-6612. Update fair scheduler policies to be aware of resource types. (Contributed by Daniel Templeton via Yufei Gu) 2017-09-14 11:23:37 -07:00
Arun Suresh 66ca0a6540 YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh 4f8194430f YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh) 2017-09-14 08:51:26 -07:00
Arun Suresh 864fbacd45 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh) 2017-09-14 08:48:39 -07:00
Robert Kanter bb34ae9554 YARN-7146. Many RM unit tests failing with FairScheduler (rkanter) 2017-09-13 15:44:54 -07:00
Eric Payne 3860be7961 YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe. 2017-09-13 14:41:55 -05:00
Eric Payne a3c44195be Revert 'YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe.' 
This reverts commit 729d05f529.
 2017-09-13 14:38:58 -05:00
Eric Payne 729d05f529 YARN-4727. Unable to override the /home/ericp/run/conf/ env variable for container. Contributed by Jason Lowe. 2017-09-13 14:29:08 -05:00
Rohith Sharma K S 5324388cf2 YARN-7157. Add admin configuration to filter per-user's apps in secure cluster. Contributed by Sunil G. 2017-09-13 23:36:47 +05:30
Arun Suresh a1c398e0dc YARN-7185. [Addendum patch] Minor javadoc and checkstyle fix. 2017-09-12 17:03:32 -07:00
Yufei Gu 82c5dd1d50 YARN-7057. FSAppAttempt#getResourceUsage doesn't need to consider resources queued for preemption. (Contributed by Karthik via Yufei Gu) 2017-09-12 16:13:39 -07:00
Arun Suresh 2ae72692fc YARN-7185. ContainerScheduler should only look at availableResource for GUARANTEED containers when OPPORTUNISTIC container queuing is enabled. (Wangda Tan via asuresh) 2017-09-12 16:10:08 -07:00
Sunil G da0b6a354b YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Daniel Templeton bf2b687412 YARN-7136. Additional Performance Improvement for Resource Profile Feature 
(Contributed by Wangda Tan via Daniel Templeton)
 2017-09-12 09:19:12 -07:00
Wangda Tan e490602e9b YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
 2017-09-12 09:19:12 -07:00
Sunil G a333ba54e3 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan. 2017-09-12 09:19:12 -07:00
Wangda Tan 39240b61a1 YARN-7043. Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
 2017-09-12 09:19:12 -07:00
Wangda Tan df3855541a YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
 2017-09-12 09:19:12 -07:00
Daniel Templeton 6746f8cadb YARN-6908. ResourceProfilesManagerImpl is missing @Overrides on methods 
(Contributed by Sunil G. via Daniel Templeton)
 2017-09-12 09:19:12 -07:00
Daniel Templeton 5c4ab4c291 YARN-6935. [YARN-3926] ResourceProfilesManagerImpl.parseResource() has no need of the key parameter 
(Contributed by Manikandan R via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Daniel Templeton 3aeaafecb8 YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)
 2017-09-12 09:19:11 -07:00
Sunil G 58da54640e YARN-6445. [YARN-3926] Performance improvements in resource profile branch with respect to SLS. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G dae65f3bef YARN-6232. Update resource usage and preempted resource calculations to take into account all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Sunil G 7805deed48 YARN-5588. [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 2017-09-12 09:19:11 -07:00
Arun Suresh 6708ac3301 YARN-5587. Add support for resource profiles. (vvasudev via asuresh) 2017-09-12 09:19:10 -07:00
Varun Vasudev c2032e251e YARN-5708. Implement APIs to get resource profiles from the RM. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev a9b4426302 YARN-5707. Add manager class for resource profiles. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Rohith Sharma K S 239c1824a0 YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Varun Vasudev 759114b006 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 2017-09-12 09:19:10 -07:00
Wangda Tan e74d1be04b YARN-7173. Container update RM-NM communication fix for backward compatibility. (Arun Suresh via wangda) 
Change-Id: I1c39ed5c59dee739ba5044b61b3ef5ed203b79c1
 2017-09-11 20:52:08 -07:00
Haibo Chen fa531788fd YARN-7132. FairScheduler.initScheduler() contains a surprising unary plus. (Yeliang Cang via Haibo Chen) 2017-09-11 20:42:49 -07:00
Haibo Chen 661f5eb0c6 YARN-7128. The error message in TimelineSchemaCreator is not enough to find out the error. (Jinjiang Ling via Haibo Chen) 2017-09-11 16:20:20 -07:00
Yufei Gu 5ab21dfe9c YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu) 2017-09-10 23:17:59 -07:00
Sunil G 722ee84194 YARN-7163. RMContext need not to be injected to webapp and other Always Running services. Contributed by Rohith Sharma K S. 2017-09-11 10:28:10 +05:30
Junping Du 3fddabc2fe YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong. 2017-09-08 15:16:19 -07:00
Arun Suresh 1f53ae7972 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh) 2017-09-08 09:24:05 -07:00
Subru Krishnan fa6137501c YARN-5330. SharingPolicy enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru). 2017-09-07 19:07:17 -07:00
Sunil G 56d93d2e39 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S. 2017-09-08 07:15:17 +05:30
Jason Lowe b0b535d9d5 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:17:03 -05:00
Wangda Tan f155ab7cfa YARN-7033. Add support for NM Recovery of assigned resources (e.g. GPU's, NUMA, FPGA's) to container. (Devaraj K and Wangda Tan) 
Change-Id: Iffd18bb95debe1c8cc55e30abc1d8f663e9d0e30
 2017-09-07 14:13:37 -07:00
Sunil G 6f101e7df1 YARN-6992. Kill application button is visible even if the application is FINISHED in RM UI. Contributed by Suma Shivaprasad. 2017-09-07 13:16:20 +05:30
Subru Krishnan b6e7d13696 YARN-5328. Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-06 16:46:01 -07:00
Subru Krishnan e3345e985b Revert "Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem." 
This reverts commit 7996eca7dc.
 2017-09-06 16:39:23 -07:00
Subru Krishnan 7996eca7dc Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem. 2017-09-01 15:16:40 -07:00
Andrew Wang 0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Jason Lowe 99a7f5d451 YARN-7120. CapacitySchedulerPage NPE in "Aggregate scheduler counts" section. Contributed by Eric Payne 2017-09-01 11:09:14 -05:00
bibinchundatt dcd0bedcc8 YARN-7141. Move logging APIs to slf4j in timelineservice after ATSv2 merge. Contributed by Varun Saxena 2017-09-01 11:29:16 +05:30
Allen Wittenauer 0adc3a0533
YARN-6721. container-executor should have stack checking 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-31 19:39:31 -07:00
Junping Du 91cc070d67 YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-08-31 16:41:43 -07:00
Subru Krishnan bac4e8cca8 YARN-7095. Federation: routing getNode/getNodes/getMetrics REST invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 2017-08-31 15:05:41 -07:00
Daniel Templeton d4417dae4f YARN-6780. ResourceWeights.toString() cleanup (Contributed by weiyuan via Daniel Templeton) 2017-08-31 09:35:01 -07:00
Sunil G f9e0cc8cdc YARN-7116. CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage. Contributed by Wangda Tan. 2017-08-31 18:42:01 +05:30
Allen Wittenauer 71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms 
Signed-off-by: Chris Douglas <cdouglas@apache.org>
 2017-08-30 21:47:24 -07:00
Daniel Templeton cc23514aba YARN-7115. Move BoundedAppender to org.hadoop.yarn.util pacakge 
(Contributed by Jian He via Daniel Templeton)
 2017-08-30 17:26:13 -07:00
Haibo Chen a20e7105ea YARN-6868. Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml. (Ray Chiang via Haibo Chen) 2017-08-30 09:14:59 -07:00
Varun Saxena 3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Varun Saxena 512068a01f YARN-6905. Addendum to fix TestTimelineReaderWebServicesHBaseStorage due to missing FastNumberFormat 2017-08-30 11:29:54 +05:30
Jian He 08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Haibo Chen 9b08f365d7 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-08-30 11:29:54 +05:30
Varun Saxena b2efebdd07 YARN-6861. Reader API for sub application entities (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:54 +05:30
Jian He 7fd6ae2479 YARN-6134. [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Rohith Sharma K S e276c75ec1 YARN-7041. Nodemanager NPE running jobs with security off. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Rohith Sharma K S 32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Jian He b664569586 YARN-7006. [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jason Lowe d5ff965fee YARN-6820. Restrict read access to timelineservice v2 data. Contributed by Vrushali C 2017-08-30 11:29:53 +05:30
Varun Saxena 60765aff9b YARN-6874. Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite (Vrushali C via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 354be99dbf YARN-6133. [ATSv2 Security] Renew delegation token for app automatically if an app collector is active. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 7594d1de7b YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Varun Saxena 9f6540535d YARN-6734. Ensure sub-application user is extracted & sent to timeline service (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 3fb71b1393 YARN-6888. Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S a990ff70c2 YARN-6733. Add table for storing sub-application entities. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Vrushali C 61136d03f2 YARN-6850 Ensure that supplemented timestamp is stored only for flow run metrics (Contributed by Varun Saxena via Vrushali C) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 70078e91e3 YARN-4455. Support fetching metrics by time range. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 660413165a YARN-6801. NPE in RM while setting collectors map in NodeHeartbeatResponse. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Jian He ac7f52df83 YARN-5648. [ATSv2 Security] Client side changes for authentication. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jian He d3f11e3f13 YARN-6638. [ATSv2 Security] Timeline reader side changes for loading auth filters and principals. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jian He 879de51206 YARN-5647. [ATSv2 Security] Collector side changes for loading auth filters and principals. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Varun Saxena f6a51dce18 YARN-6658. Remove columnFor() methods of Columns in HBaseTimeline backend (Haibo Chen via Varun Saxena) 2017-08-30 11:29:52 +05:30
Haibo Chen a8f082a180 YARN-6555. Store application flow context in NM state store for work-preserving restart. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 47474fffac)
 2017-08-30 11:29:52 +05:30
Varun Saxena b87b72b40a YARN-6146. Add Builder methods for TimelineEntityFilters (Haibo Chen via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena 44999aabc2 YARN-6318. timeline service schema creator fails if executed from a remote machine (Sangjin Lee via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena 18b3a80df7 YARN-6237. Move UID constant to TimelineReaderUtils (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena c3bd8d6ad3 YARN-6256. Add FROM_ID info key for timeline entities in reader response (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Varun Saxena 8bb2646595 YARN-6027. Support fromid(offset) filter for /flows API (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:52 +05:30
Sangjin Lee cf30b3b914 YARN-6253. FlowAcitivityColumnPrefix.store(byte[] rowKey, ...) drops timestamp. Contributed by Haibo Chen. 2017-08-30 11:29:52 +05:30
Sangjin Lee ccb38c1991 YARN-5928. Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers. Contributed by Haibo Chen. 2017-08-30 11:29:52 +05:30
Sangjin Lee 580d884913 YARN-6094. Update the coprocessor to be a dynamically loaded one. Contributed by Vrushali C. 2017-08-30 11:29:52 +05:30
Varun Saxena 5e0acee75e Addendum for YARN-6064. Support fromId for flowRuns and flow/flowRun apps REST API's 2017-08-30 11:29:52 +05:30
Varun Saxena 6f65cf27bb YARN-6064. Support fromId for flowRuns and flow/flowRun apps REST API's (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena 71847ed44d YARN-5378. Accommodate app-id->cluster mapping (Sangjin Lee via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena 02a9710a09 YARN-5585. [Atsv2] Reader side changes for entity prefix and support for pagination via additional filters (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee 2556c012e3 YARN-5925. Extract hbase-backend-exclusive utility methods from TimelineStorageUtil. Contributed by Haibo Chen. 2017-08-30 11:29:51 +05:30
Sangjin Lee 4481561e4a YARN-5739. Provide timeline reader API to list available timeline entity types for one application. Contributed by Li Lu. 2017-08-30 11:29:51 +05:30
Sangjin Lee 092fead5d9 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 2017-08-30 11:29:51 +05:30
Varun Saxena c92a7ab31c YARN-3359. Addendum for Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30
Varun Saxena 91d3443806 YARN-3359. Recover collector list when RM fails over (Li Lu via Varun Saxena) 2017-08-30 11:29:51 +05:30
Sangjin Lee 643a20a358 YARN-5265. Make HBase configuration for the timeline service configurable. Contributed by Joep Rottinghuis. 2017-08-30 11:29:51 +05:30
Sangjin Lee 05ff04439e YARN-5715. Introduce entity prefix for return and sort order. Contributed by Rohith Sharma K S. 2017-08-30 11:29:51 +05:30
Varun Saxena 9bb5cc3d49 YARN-3649. Allow configurable prefix for hbase table names like prod, exp, test etc (Vrushali C via Varun Saxena) 2017-08-30 11:29:50 +05:30
Sangjin Lee 78b7e070d8 YARN-5638. Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery. Contributed by Li Lu. 2017-08-30 11:29:50 +05:30
Li Lu e72e6260da YARN-5229. Refactor #isApplicationEntity and #getApplicationEvent from HBaseTimelineWriterImpl. (Vrushali C via gtcarrera9) 2017-08-30 11:29:50 +05:30
Sangjin Lee 278bd6fcff YARN-5156. YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state (Vrushali C via sjlee) 2017-08-30 11:29:50 +05:30
Rohith Sharma K S 4cae120c61 YARN-6982. Potential issue on setting AMContainerSpec#tokenConf to null before app is completed. Contributed by Manikandan R. 2017-08-30 10:45:11 +05:30
Wangda Tan f59332b97b YARN-5219. When an export var command fails in launch_container.sh, the full container launch should fail. (Sunil G via wangda) 
Change-Id: Iaa6b978bb89482e9d1d77ba57f4adfdc48e39a3c
 2017-08-29 15:29:22 -07:00
Junping Du ad45d19998 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang. 2017-08-29 15:16:23 -07:00
Carlo Curino cc8893edc0 YARN-7010. Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-29 14:53:09 -07:00
Jason Lowe 1f6dc4ee9a YARN-5816. TestDelegationTokenRenewer#testCancelWithMultipleAppSubmissions is still flakey. Contributed by Robert Kanter 2017-08-29 09:52:24 -05:00
Junping Du 312b1fd9da YARN-7076. yarn application -list -appTypes is not working. Contributed by Jian He. 2017-08-28 18:26:51 -07:00
Yufei Gu d5bc958279 YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu) 2017-08-28 10:09:46 -07:00
Sunil G 02599bda04 YARN-7051. Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin. Contributed by Eric Payne. 2017-08-28 12:22:56 +05:30
Li Lu 3bee7046c5 YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9) 2017-08-25 22:43:25 -07:00
Eric Payne e864f81471 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe. 2017-08-25 15:37:54 -05:00
Jason Lowe 39a9dc8e4a YARN-7052. RM SchedulingMonitor gives no indication why the spawned thread crashed. Contributed by Eric Payne 2017-08-25 15:31:29 -05:00
Arpit Agarwal 8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Jason Lowe 3a4e861169 YARN-6640. AM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2017-08-25 09:16:17 -05:00
Yufei Gu 9e2699ac2c YARN-7049. FSAppAttempt preemption related fields have confusing names. (Contributed by Karthik Kambatla via Yufei Gu) 2017-08-24 17:52:41 -07:00
Junping Du c2cb7ea1ef YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong. 2017-08-24 13:36:49 -07:00
bibinchundatt de0cba700b YARN-7074. Fix NM state store update comment. Contributed by Botong Huang 2017-08-24 16:34:38 +05:30
Junping Du 652dd434d9 YARN-7090. testRMRestartAfterNodeLabelDisabled get failed when CapacityScheduler is configured. Contributed by Wangda Tan. 2017-08-23 18:06:29 -07:00
Wangda Tan f49843a988 YARN-6251. Do async container release to prevent deadlock during container updates. (Arun Suresh via wangda) 
Change-Id: I6c67d20c5dd4d22752830ebf0ed2340824976ecb
 2017-08-23 09:56:20 -07:00
Subru Krishnan 4249172e14 YARN-7053. Move curator transaction support to ZKCuratorManager. (Jonathan Hung via Subru). 2017-08-22 19:20:57 -07:00
Jason Lowe 657dd59cc8 YARN-7048. Fix tests faking kerberos to explicitly set ugi auth type. Contributed by Daryn Sharp 2017-08-22 13:16:24 -05:00
Jason Lowe 3efcd51c3b YARN-2416. InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time. Contributed by Jonathan Eagles 2017-08-22 12:56:09 -05:00
Akira Ajisaka d5ff57a08f
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-22 17:14:12 +09:00
Arun Suresh ae8fb13b31 YARN-6923. Metrics for Federation Router. (Giovanni Matteo Fumarola via asuresh) 2017-08-21 22:50:24 -07:00
Arun Suresh 75abc9a8e2 YARN-5603. Metrics for Federation StateStore. (Ellen Hui via asuresh) 2017-08-21 22:43:08 -07:00
Arun Suresh 7a82d7bcea YARN-6979. [Addendum patch] Fixed classname and added javadocs. (Kartheek Muthyala via asuresh) 2017-08-20 10:24:05 -07:00
Arun Suresh 8410d862d3 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 2017-08-20 07:54:09 -07:00
Wangda Tan 436c2638f9 YARN-6852. Native code changes to support isolate GPU devices by using CGroups. (wangda) 
Change-Id: I4869cc4d8ad539539ccba4bea5a178cacdb741ab
 2017-08-18 18:26:36 -07:00
Yufei Gu 8991f0baec YARN-6969. Clean up unused code in class FairSchedulerQueueInfo. (Larry Lo via Yufei Gu) 2017-08-18 14:38:44 -07:00
bibinchundatt e05fa3451d YARN-7007. NPE in RM while using YarnClient.getApplications(). Contributed by Lingfeng Su. 2017-08-18 20:28:50 +05:30
Jason Lowe ab1a8ae85f YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-17 15:50:14 -05:00
Sunil G f9a0e23381 YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad. 2017-08-17 15:07:15 +05:30
Subru Krishnan de462da04e YARN-6900. ZooKeeper based implementation of the FederationStateStore. (Íñigo Goiri via Subru). 2017-08-16 11:43:24 -07:00
Akira Ajisaka 588c190afd
YARN-6965. Duplicate instantiation in FairSchedulerQueueInfo. Contributed by Masahiro Tanaka. 2017-08-16 14:06:22 +09:00
Subru Krishnan 75dd866bfb HADOOP-14773. Extend ZKCuratorManager API for more reusability. (Íñigo Goiri via Subru). 2017-08-15 16:53:59 -07:00
Nathan Roberts d265459024 YARN-7014. Fix off-by-one error causing heap corruption (Jason Lowe via nroberts) 2017-08-15 15:52:48 -05:00
Arun Suresh 4d7be1d857 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh) 2017-08-14 19:46:17 -07:00
Jason Lowe 5558792894 YARN-6917. Queue path is recomputed from scratch on every allocation. Contributed by Eric Payne 2017-08-14 15:31:34 -05:00
Jason Lowe 3325ef653d YARN-6987. Log app attempt during InvalidStateTransition. Contributed by Jonathan Eagles 2017-08-14 14:40:08 -05:00
Daniel Templeton 6b09c32705 YARN-6881. LOG is unused in AllocationConfiguration (Contributed by weiyuan via Daniel Templeton) 2017-08-14 11:55:33 -07:00
Vrushali C 608a06cca5 YARN-6905 Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat (Contributed by Haibo Chen) 2017-08-14 11:41:11 -07:00
Subru Krishnan 18f3603bce YARN-6996. Change javax.cache library implementation from JSR107 to Apache Geronimo. (Ray Chiang via Subru). 2017-08-14 11:10:00 -07:00
Jian He e2f6299f6f YARN-6959. RM may allocate wrong AM Container for new attempt. Contributed by Yuqi Wang 2017-08-14 10:51:30 -07:00
bibinchundatt d8f74c3964 YARN-6741. Deleting all children of a Parent Queue on refresh throws exception. Contributed by Naganarasimha G R. 2017-08-14 09:39:00 +05:30
Carlo Curino 28d97b79b6 YARN-6687. Validate that the duration of the periodic reservation is less than the periodicity. (subru via curino) 2017-08-11 16:58:04 -07:00
Carlo Curino cc59b5fb26 YARN-6896. Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution). (Contributed by Giovanni Matteo Fumarola via curino) 2017-08-11 15:58:01 -07:00
Daniel Templeton 0996acde6c YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator 
(Contributed by Larry Lo via Daniel Templeton)
 2017-08-11 14:50:46 -07:00
Daniel Templeton 65364defb4 YARN-6967. Limit application attempt's diagnostic message size thoroughly 
(Contributed by Chengbing Liu via Daniel Templeton)
 2017-08-11 14:28:55 -07:00
Daniel Templeton c7680d4cc4 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces 
(Contributed by weiyuan via Daniel Templeton)
 2017-08-11 14:22:02 -07:00
Daniel Templeton 218588be77 YARN-6952. Enable scheduling monitor in FS (Contributed by Yufei Gu via Daniel Templeton) 2017-08-11 14:04:19 -07:00
Subru Krishnan bbbf0e2a41 HADOOP-14741. Refactor curator based ZooKeeper communication into common library. (Íñigo Goiri via Subru). 2017-08-11 13:58:45 -07:00
Karthik Kambatla 8c4b6d16a5 YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha) 2017-08-11 12:15:43 -07:00
Wangda Tan ec694145cf YARN-6033. Add support for sections in container-executor configuration file. (Varun Vasudev via wandga) 
Change-Id: Ibc6d2a959debe5d8ff2b51504149742449d1f1da
 2017-08-09 10:51:29 -07:00
Akira Ajisaka 63cfcb90ac
YARN-6958. Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice. Contributed by Yeliang Cang. 2017-08-09 23:58:22 +09:00
Akira Ajisaka 1a18d5e514
YARN-6515. Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager. Contributed by Naganarasimha G R. 2017-08-09 21:56:43 +09:00
Subru Krishnan ad2a350662 YARN-6970. Add PoolInitializationException as retriable exception in FederationFacade. (Giovanni Matteo Fumarola via Subru). 2017-08-08 16:48:29 -07:00
Daniel Templeton e0c24145d2 YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code 
(Contributed by Angela Wang via Daniel Templeton)
 2017-08-08 13:35:22 -07:00
Wangda Tan 1794de3ea4 YARN-6726. Fix issues with docker commands executed by container-executor. (Shane Kumpf via wangda) 
Change-Id: If1b1827345f98f0a49cc7e39d1ba41fbeed5e911
 2017-08-08 12:56:29 -07:00
Jian He acf9bd8b1d YARN-6890. Not display killApp button on UI if UI is unsecured but cluster is secured. Contributed by Junping Du 2017-08-08 11:09:38 -07:00
Daniel Templeton 47b145b9b4 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path 
(Contributed by Miklos Szegedi via Daniel Templeton)
 2017-08-08 10:33:26 -07:00
Akira Ajisaka 98912950b6
YARN-6961. Remove commons-logging dependency from hadoop-yarn-server-applicationhistoryservice module. Contributed by Yeliang Cang. 2017-08-08 19:38:58 +09:00
Arun Suresh 8d3fd81980 YARN-6920. Fix resource leak that happens during container re-initialization. (asuresh) 2017-08-07 18:59:25 -07:00
Subru Krishnan c61f2c4198 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru). 2017-08-07 16:58:29 -07:00
Wangda Tan adb84f34db YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
 2017-08-07 11:32:21 -07:00
Akira Ajisaka b0fbf17965
YARN-6957. Moving logging APIs over to slf4j in hadoop-yarn-server-sharedcachemanager. Contributed by Yeliang Cang. 2017-08-07 19:25:40 +09:00
Akira Ajisaka 839e077faf
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-07 18:56:00 +09:00
Sunil G 46b7054fa7 YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 2017-08-07 13:15:46 +05:30
Junping Du f44b349b81 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-04 16:03:56 -07:00
Arun Suresh 35dc782923 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 2017-08-03 21:15:40 -07:00
Subru Krishnan 0542e6f86e YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 2017-08-03 14:18:03 -07:00
Yufei Gu c617fe02b3 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 11:33:37 -07:00
Haibo Chen 293c74a81b YARN-6673 Add cpu cgroup configurations for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:56:51 -07:00
Haibo Chen c5d256c760 YARN-6674 Add memory cgroup settings for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:52:35 -07:00
Sunil G f64cfeaf61 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 2017-08-03 19:27:10 +05:30
Eric Payne 8ce8672b6b YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 2017-08-02 12:53:22 -05:00
Yufei Gu 45535f8afa YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-02 09:25:19 -07:00
Eric Payne 48899134d2 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 2017-08-02 10:59:33 -05:00
Jian He f9139ac8f6 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:25 -07:00
Carlo Curino acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92ab)
 2017-08-01 17:28:28 -07:00
Subru Krishnan 86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec)
 2017-08-01 17:28:28 -07:00
Subru Krishnan 8737c2a8e9 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed2680)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 712e97d4cf Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 3e1dc7ece0 YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 825108d08a YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd7)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 43a97174fe YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971)
 2017-08-01 17:28:26 -07:00
Subru Krishnan 4cfec943b1 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de5969)
 2017-08-01 17:28:26 -07:00
Subru Krishnan 70b1a757f1 YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235e)
 2017-08-01 17:28:26 -07:00
Subru Krishnan bed1832c93 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru). 
(cherry picked from commit 2399eb8200)
 2017-08-01 17:28:25 -07:00
Subru Krishnan 42949a6e01 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a)
 2017-08-01 17:28:25 -07:00
Subru Krishnan f317e0cb48 Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase. 
(cherry picked from commit bd9c7b1bec)
 2017-08-01 17:28:25 -07:00
Subru Krishnan 859aa1f9d6 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce)
 2017-08-01 17:28:24 -07:00
Subru Krishnan 7444406d6d YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519)
 2017-08-01 17:28:24 -07:00
Subru Krishnan 80e1904000 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061)
 2017-08-01 17:28:24 -07:00
Carlo Curino 5272af8c7e YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2e)
 2017-08-01 17:28:24 -07:00
Carlo Curino c4bbdfeb66 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29)
 2017-08-01 17:28:23 -07:00
Subru Krishnan e1da8f0667 YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde)
 2017-08-01 17:28:23 -07:00
Carlo Curino 1c64e1709b YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c3)
 2017-08-01 17:28:23 -07:00
Subru Krishnan 86b2bec56e YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c)
 2017-08-01 17:28:23 -07:00
Carlo Curino 8623644f45 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 193ec456d6 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 98b45b0ed3 YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 91803305e5 YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade05)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 78032e51d9 YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 4128c9522d YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 083a05bc14 YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bd)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 93d9fdeca6 YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682ec)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 1dadd0b45a YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 0662996b6a YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 1298127bda)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 81472778d7 YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru). 
(cherry picked from commit f8208fe0b5)
 2017-08-01 17:28:20 -07:00
Jian He 2797507d51 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan 
(cherry picked from commit 9ca2aba9cc)
 2017-08-01 17:28:20 -07:00
Subru Krishnan 1882bc10fb YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ac1ba2a304)
 2017-08-01 17:28:20 -07:00
Jian He 94a24567d6 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan 
(cherry picked from commit 8573c286e2)
 2017-08-01 17:28:20 -07:00
Jian He bdfad4523f YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f)
 2017-08-01 17:28:20 -07:00
Subru Krishnan cfafd173bd YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru) 
(cherry picked from commit bd44182e70)
 2017-08-01 17:28:19 -07:00
Jian He 5c84382397 YARN-3672. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b9)
 2017-08-01 17:28:19 -07:00
Subru Krishnan 52558df620 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru) 
(cherry picked from commit e4f928cf9c)
 2017-08-01 17:28:19 -07:00
Subru Krishnan 0733088c29 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru) 
(cherry picked from commit b747d59f41)
 2017-08-01 17:28:19 -07:00
Subru Krishnan e0c3a44396 YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru). 
(cherry picked from commit 268389d021)
 2017-08-01 17:28:19 -07:00
Subru Krishnan f1a508bdef YARN-3664. Federation PolicyStore internal APIs 
(cherry picked from commit 2f377263a1)
 2017-08-01 17:28:18 -07:00
Subru Krishnan 047a16bf15 YARN-5307. Federation Application State Store internal APIs 
(cherry picked from commit 1af5292042)
 2017-08-01 17:28:18 -07:00
Subru Krishnan d19b677301 YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a93)
 2017-08-01 17:28:18 -07:00
Subru Krishnan dcf8c52d80 YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui. 
(cherry picked from commit 20d1d2be91)
 2017-08-01 17:28:18 -07:00
Subru Krishnan 3e22896f86 YARN-3662. Federation Membership State Store internal APIs. 
(cherry picked from commit 5424d0899c)
 2017-08-01 17:28:17 -07:00
Jian He 91f120f743 YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 09:56:33 -07:00
Akira Ajisaka b38a1eea8e
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 2017-08-01 15:15:43 +09:00
Akira Ajisaka a4aa1cb405
Revert "YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang." 
This reverts commit 1a78c0ff01.
 2017-08-01 12:15:03 +09:00
Akira Ajisaka 1a78c0ff01
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-01 10:53:32 +09:00
Akira Ajisaka f8bed5e9a7
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 2017-07-31 11:09:13 +09:00
Arun Suresh 890e14c02a YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 2017-07-28 16:32:43 -07:00
Subru Krishnan bcde66bed1 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-07-28 15:44:36 -07:00
Yufei Gu 9ea01fd956 YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-07-28 09:52:53 -07:00
Yufei Gu 9902be72cb YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 2017-07-27 23:19:39 -07:00
Akira Ajisaka 38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier. 2017-07-28 09:10:52 +09:00
Varun Saxena 11ece0bda1 Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-28 00:10:45 +05:30
Varun Saxena f66fd11e51 YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-27 23:14:50 +05:30
Daniel Templeton f81a4efb8c YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 2017-07-25 13:00:31 -07:00
Jian He ac9489f7fc YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-25 09:57:14 -07:00
Akira Ajisaka cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Akira Ajisaka 218b1b33ff
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 2017-07-25 15:12:08 +09:00
Daniel Templeton 10583625c1 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)
 2017-07-24 13:44:00 -07:00
Daniel Templeton bb30bd3771 YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)
 2017-07-24 12:13:50 -07:00
Sunil G e315328428 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 11:39:03 +05:30
Brahma Reddy Battula 2054324d47 Revert "YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi" 
This reverts commit 4a771d9010.
 2017-07-23 12:56:18 +08:00
Jian He 4a771d9010 YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-21 11:31:33 -07:00
Jason Lowe c8df3668ec YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 2017-07-20 11:03:04 -05:00
Arun Suresh 077fcf6a96 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 2017-07-19 12:26:40 -07:00
Rohith Sharma K S 845c4e52bd YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 2017-07-19 11:10:52 +05:30
Yufei Gu daaf530fce YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 2017-07-18 16:38:07 -07:00
Ray Chiang f5f14a2ad6 YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 2017-07-18 12:35:08 -07:00
Sunil G 0b7afc060c Addendum patch for YARN-5731 2017-07-18 11:49:09 +05:30