Commit Graph

3552 Commits

Author SHA1 Message Date
Eric Yang b4fa1830a8 YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator.
Contributed by Tarun Parimi

(cherry picked from commit 71e0b0d800)
2018-12-17 12:08:35 -05:00
Eric Yang 690d760174 YARN-9125. Fixed Carriage Return detection in Docker container launch command.
Contributed by Billie Rinaldi

(cherry picked from commit b2d7204ed0)
2018-12-14 17:55:38 -05:00
Weiwei Yang 14ecdb62b6 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters.
(cherry picked from commit 1c09a10e96)
2018-12-10 12:17:00 +08:00
Jonathan Hung 7b523e6a77 YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics
(cherry picked from commit 978ab3e958227220cb6f1a08ae6e7cdb8a46628b)
(cherry picked from commit dca69d178dba21c41fd1293187f29143f7e81e19)
2018-12-07 10:45:57 -08:00
Eric Yang 7ef4ff1905 YARN-9071. Improved status update for reinitialized containers.
Contributed by Chandni Singh

(cherry picked from commit 1b790f4dd1)
2018-12-05 19:05:26 -05:00
Jonathan Hung 2cb9479bfc YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:16:39 -08:00
bibinchundatt 8be2d16b94 YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
(cherry picked from commit 07142f54a8)
2018-11-29 22:08:35 +05:30
Jason Lowe d9457df989 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko
(cherry picked from commit 3ce99e32f7)
2018-11-28 08:54:04 -06:00
Eric Yang bec5036397 YARN-8665. Added Yarn service cancel upgrade option.
Contributed by Chandni Singh
2018-11-27 16:29:08 -05:00
Eric Yang 463de48f04 YARN-8986. Added port publish for Docker container running with bridge.
Contributed by Charo Zhang
2018-11-27 14:28:37 -05:00
Weiwei Yang 17a41f5d86 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun.
(cherry picked from commit d027a24f03)
2018-11-18 23:24:37 +08:00
Eric Yang fce0350289 YARN-8160. Support upgrade of service that use docker containers.
Contributed by Chandni Singh
2018-11-16 16:01:25 -05:00
Rohith Sharma K S 095635d984 YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report.
(cherry picked from commit ee3355be3c)
2018-11-16 18:38:11 +05:30
Akira Ajisaka daad077121
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-10 14:37:35 +09:00
Weiwei Yang a3b61baf94 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji.
(cherry picked from commit c96cbe8659)
2018-11-07 22:50:05 +08:00
Akira Ajisaka 52af95fdce
Revert "YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang."
This reverts commit dd8479e80d.
2018-11-07 11:33:31 +09:00
Akira Ajisaka dd8479e80d
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang.
(cherry picked from commit 951c98f890)
2018-11-07 11:19:11 +09:00
Jason Lowe 7335d940de YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
(cherry picked from commit ab6aa4c726)
2018-11-06 08:52:29 -06:00
Weiwei Yang 631b31110c YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang.
(cherry picked from commit 5d6554c722)
2018-11-06 14:53:28 +08:00
Weiwei Yang 71999f4464 YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji.
(cherry picked from commit c7fcca0d7e)
2018-11-06 13:23:42 +08:00
Jonathan Hung 221494a75c YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne
(cherry picked from commit 2ab611d48b)
2018-11-01 14:31:22 -07:00
Rohith Sharma K S d9a494b1e0 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
(cherry picked from commit 4ec4ec6971)
2018-10-30 11:50:56 +05:30
Weiwei Yang 70efe253f3 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
(cherry picked from commit 1d90a0dd23)
2018-10-29 11:56:31 +08:00
Jason Lowe 3be72b7aa2 YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg
(cherry picked from commit 93fb3b4b9c)
2018-10-23 12:55:48 -05:00
Rohith Sharma K S 3e3b088856 YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
(cherry picked from commit 0b62983c5a)
2018-10-23 14:08:06 +05:30
Sunil G 0cb184d6e9 YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh.
(cherry picked from commit 2202e00ba8)
2018-10-23 09:58:05 +05:30
Eric Yang e86efa8712 YARN-8910. Fixed misleading log statement when container max retries is infinite.
Contributed by Chandni Singh

(cherry picked from commit 47ad98b2e1)
2018-10-19 13:50:32 -04:00
Weiwei Yang beca90ece8 YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang.
(cherry picked from commit 13cc0f50ea)
2018-10-19 10:02:46 +08:00
Wangda Tan 46baafedf1 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07)
2018-10-18 12:29:19 -07:00
Weiwei Yang a0060cf8ee Revert "YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach."
This reverts commit ce4a0898df.
2018-10-10 21:41:00 +08:00
Weiwei Yang 3968ce1073 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan.
(cherry picked from commit edce866489)
2018-10-10 09:48:56 +08:00
Weiwei Yang ce4a0898df YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach. 2018-10-09 22:30:42 +08:00
Wangda Tan 86a1ad4428 YARN-8844. TestNMProxy unit test is failing. (Eric Yang via wangda)
Change-Id: I241fa8701b6f1dbcad87fd2e9a429e32e7aa40f5
(cherry picked from commit b3ac886933)
2018-10-04 10:49:29 -07:00
Shane Kumpf adbc010d0f YARN-8785. Improve the error message when a bind mount is not whitelisted. Contributed by Simon Prewo
(cherry picked from commit 5edb9d3b97)
2018-10-02 07:26:45 -06:00
Eric E Payne c306da08ec YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang.
(cherry picked from commit 8598b498bc)
2018-09-28 15:34:23 +00:00
Vrushali C 5b72aa04e1 YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks.
(cherry picked from commit 90e2e493b3)
2018-09-28 10:31:38 +05:30
Jason Lowe a56a345e07 YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang
(cherry picked from commit 6b988d821e)
2018-09-26 16:15:48 -07:00
Rohith Sharma K S d473152e6a YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt. 2018-09-26 20:36:34 +05:30
Rohith Sharma K S 47306cc2db YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt.
(cherry picked from commit 50bc7746d7)
2018-09-25 11:40:09 +05:30
Eric Yang e9315f6688 YARN-8801. Fixed header comments for docker utility functions.
Contributed by Zian Chen
2018-09-20 13:12:29 -04:00
Jason Lowe 3fb6787295 YARN-8784. DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk. Contributed by Eric Badger
(cherry picked from commit 6b5838ed32)
2018-09-19 16:49:21 -05:00
Weiwei Yang aaf0b119e5 YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang.
(cherry picked from commit 0712537e79)
2018-09-19 19:38:09 +08:00
Jason Lowe 3d77094cf2 YARN-8648. Container cgroups are leaked when using docker. Contributed by Jim Brennan
(cherry picked from commit 2df0a8dcb3)
2018-09-18 15:43:10 -05:00
Weiwei Yang 00a469138d YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi.
(cherry picked from commit f1a893fdbc)
2018-09-14 16:40:35 +08:00
Jason Lowe 88687213cc YARN-8680. YARN NM: Implement Iterable Abstraction for LocalResourceTracker state. Contributed by Pradeep Ambati
(cherry picked from commit 250b50018e)
2018-09-13 14:12:20 -05:00
Weiwei Yang a7b1d1e006 YARN-8729. Node status updater thread could be lost after it is restarted. Contributed by Tao Yang.
(cherry picked from commit 39c1ea1ed4)
2018-09-13 23:16:52 +08:00
Sunil G c879ca38de YARN-8630. ATSv2 REST APIs should honor filter-entity-list-by-user in non-secure cluster when ACls are enabled. Contributed by Rohith Sharma K S.
(cherry picked from commit f4bda5e8e9)
2018-09-13 17:48:01 +05:30
Eric E Payne b6bc0f409a YARN-8709: CS preemption monitor always fails since one under-served queue was deleted. Contributed by Tao Yang.
(cherry picked from commit 987d8191ad)
2018-09-10 20:02:39 +00:00
Eric Yang 0b97dc5869 YARN-8751. Reduce conditions that mark node manager as unhealthy.
Contributed by Craig Condit

(cherry picked from commit 7d62334387)
2018-09-07 20:32:11 -04:00
Shane Kumpf 2d68708a1d YARN-8638. Allow linux container runtimes to be pluggable. Contributed by Craig Condit
(cherry picked from commit dffb7bfe6c)
2018-09-05 06:55:25 -06:00
bibinchundatt e2e0fc26a2 YARN-8535. Fix DistributedShell unit tests. Contributed by Abhishek Modi.
(cherry picked from commit eed8415dc1)
2018-09-02 13:37:38 +05:30
Shane Kumpf b8618556ee YARN-8642. Add support for tmpfs mounts with the Docker runtime. Contributed by Craig Condit
(cherry picked from commit 73625168c0)
2018-08-29 07:11:38 -06:00
Weiwei Yang f164568b47 YARN-8723. Fix a typo in CS init error message when resource calculator is not correctly set. Contributed by Abhishek Modi.
(cherry picked from commit 3fa4639421)
2018-08-29 11:15:02 +08:00
Billie Rinaldi eefd780918 YARN-8675. Remove default hostname for docker containers when net=host. Contributed by Suma Shivaprasad
(cherry picked from commit 05b2bbeb35)
2018-08-27 11:42:09 -07:00
Haibo Chen e4282c077b YARN-8051. TestRMEmbeddedElector#testCallbackSynchronization is flaky. (Robert Kanter via Haibo Chen)
(cherry picked from commit 93d47a0ed5)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
2018-08-24 13:24:08 -05:00
Jason Lowe 84654451fa YARN-8649. NPE in localizer hearbeat processing if a container is killed while localizing. Contributed by lujie
(cherry picked from commit 585ebd873a)
2018-08-23 09:37:43 -05:00
Rohith Sharma K S 63d5214332 YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi.
(cherry picked from commit d3fef7a5c5)
2018-08-21 12:11:12 +05:30
Wei-Chiu Chuang 0d155de159 HADOOP-14212. Addendum patch: Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:28 -07:00
Wei-Chiu Chuang 78fb14ba49 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:24 -07:00
Jason Lowe 44c4928b64 YARN-8242. YARN NM: OOM error while reading back the state store on recovery. Contributed by Pradeep Ambati and Kanwaljeet Sachdev
(cherry picked from commit 65e7469712)
2018-08-20 10:21:57 -05:00
Rohith Sharma K S a3d4a25bbf YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan.
(cherry picked from commit 4aacbfff60)
2018-08-18 11:04:09 +05:30
Eric Yang 5237bdfb5a YARN-8667. Cleanup symlinks when container restarted by NM.
Contributed by Chandni Singh

(cherry picked from commit d42806160e)
2018-08-16 18:44:47 -04:00
Jason Lowe 819a2a6f10 YARN-8656. container-executor should not write cgroup tasks files for docker containers. Contributed by Jim Brennan
(cherry picked from commit cb21eaa026)
2018-08-16 10:09:56 -05:00
Jason Lowe 95cd6de5c6 YARN-8640. Restore previous state in container-executor after failure. Contributed by Jim Brennan
(cherry picked from commit d1d129aa9d)
2018-08-14 10:26:21 -05:00
Weiwei Yang 734bc42289 YARN-8575. Avoid committing allocation proposal to unavailable nodes in async scheduling. Contributed by Tao Yang.
(cherry picked from commit 0a71bf1452)
2018-08-10 15:10:27 +08:00
Weiwei Yang 991514f7c3 YARN-8521. NPE in AllocationTagsManager when a container is removed more than once. Contributed by Weiwei Yang.
(cherry picked from commit 08d5060605)
2018-08-10 08:44:53 +08:00
Wangda Tan 68279fcd65 YARN-8588. Logging improvements for better debuggability. (Suma Shivaprasad via wangda)
Change-Id: I66aa4b0ec031ae5ce0fae558e2f8cbcbbfebc442
(cherry picked from commit 344c335a92)
2018-08-09 12:04:25 -07:00
Weiwei Yang 0ee7e80047 YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang.
(cherry picked from commit d352f167eb)
2018-08-10 00:43:53 +08:00
Jason Lowe 3dd299a770 YARN-8331. Race condition in NM container launched after done. Contributed by Pradeep Ambati
(cherry picked from commit cd04e954d2)
2018-08-09 10:23:02 -05:00
Wangda Tan 450c791ecf YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma Shivaprasad via wangda)
Change-Id: I392ef4f8baa84d5d7b1f2e438c560b5426b6d4f2
(cherry picked from commit d4258fcad7)
2018-08-07 12:41:55 -07:00
Jason Lowe 619019ccca YARN-8263. DockerClient still touches hadoop.tmp.dir. Contributed by Craig Condit
(cherry picked from commit 7526815e32)
2018-08-02 10:45:52 -05:00
Sunil G 1f77b20f08 YARN-8593. Add RM web service endpoint to get user information. Contributed by Akhil PB.
(cherry picked from commit 735b492556)
2018-08-02 08:35:54 +05:30
Billie Rinaldi 2a94823f32 YARN-8403. Change the log level for fail to download resource from INFO to ERROR. Contributed by Eric Yang
(cherry picked from commit 67c65da261)
2018-08-01 08:58:15 -07:00
Sunil G ff35f0c308 YARN-8606. Opportunistic scheduling does not work post RM failover. Contributed by Bibin A Chundatt.
(cherry picked from commit a48a0cc7fd)
2018-08-01 12:17:53 +05:30
Sunil G cbfd7358d2 YARN-8397. Potential thread leak in ActivitiesManager. Contributed by Rohith Sharma K S.
(cherry picked from commit 6310c0d17d)
2018-08-01 08:34:09 +05:30
Eric Yang 7640d62716 YARN-8579. Recover NMToken of previous attempted component data.
Contributed by Gour Saha
2018-07-31 18:35:31 -04:00
Wangda Tan 5583711419 Preparing for 3.1.2 release
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
2018-07-31 13:08:55 -07:00
Wangda Tan 7b552c9d72 YARN-8418. App local logs could leaked if log aggregation fails to initialize for the app. (Bibin A Chundatt via wangda)
Change-Id: I29a23ca4b219b48c92e7975cd44cddb8b0e04104
(cherry picked from commit 4b540bbfcf)
2018-07-31 12:13:36 -07:00
Jonathan Hung b91cf90e1c YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung 2018-07-30 17:57:25 -07:00
bibinchundatt 8cd2a73777 YARN-8584. Several typos in Log Aggregation related classes. Contributed by Szilard Nemeth.
(cherry picked from commit 2b39ad2698)
2018-07-31 00:07:08 +05:30
Sunil G f1eb5777a0 YARN-8591. [ATSv2] NPE while checking for entity acl in non-secure cluster. Contributed by Rohith Sharma K S.
(cherry picked from commit 63e08ec071)
2018-07-30 14:49:03 +05:30
bibinchundatt 2e7876a725 YARN-8558. NM recovery level db not cleaned up properly on container finish. Contributed by Bibin A Chundatt.
(cherry picked from commit 3d586841ab)
2018-07-28 20:56:35 +05:30
Eric Yang c2c3eee69c YARN-8508. Release GPU resource for killed container.
Contributed by Chandni Singh

(cherry picked from commit ed9d60e888)
2018-07-27 19:36:21 -04:00
Eric Yang 8e3807afe0 YARN-8330. Improved publishing ALLOCATED events to ATS.
Contributed by Suma Shivaprasad

(cherry picked from commit f93ecf5c1e)
2018-07-25 18:51:42 -04:00
Eric E Payne 830ef12af8 YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R
(cherry picked from commit 9485c9aee6)
2018-07-25 16:30:30 +00:00
bibinchundatt 8e65057eb1 YARN-8541. RM startup failure on recovery after user deletion. Contributed by Bibin A Chundatt. 2018-07-25 15:54:32 +05:30
Weiwei Yang b89624a943 YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang.
(Cherry-picked from commit 5be9f4a5d0)
2018-07-25 17:53:40 +08:00
Haibo Chen 7e7792dd7b YARN-6966. NodeManager metrics may return wrong negative values when NM restart. (Szilard Nemeth via Haibo Chen)
(cherry picked from commit 9d3c39e9dd)
2018-07-24 12:50:43 -07:00
Sunil G 4488fd8295 YARN-7748. TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events. Contributed by Weiwei Yang.
(cherry picked from commit 35ce6eb1f5)
2018-07-24 22:21:15 +05:30
bibinchundatt a684a2efb8 YARN-8548. AllocationRespose proto setNMToken initBuilder not done. Contributed by Bilwa S T.
(cherry picked from commit ff7c2eda34)
2018-07-24 16:30:31 +05:30
bibinchundatt 0710107f8d YARN-8544. [DS] AM registration fails when hadoop authorization is enabled. Contributed by Bibin A Chundatt.
(cherry picked from commit 8461278833)
2018-07-24 13:11:31 +05:30
Eric Yang 23b8546a80 YARN-8380. Support bind propagation options for mounts in docker runtime.
Contributed by Billie Rinaldi

(cherry picked from commit 8688a0c7f8)
2018-07-23 20:13:41 -04:00
Weiwei Yang 004e1f248e YARN-8528. Final states in ContainerAllocation might be modified externally causing unexpected allocation results. Contributed by Xintong Song. 2018-07-20 22:43:47 +08:00
Eric Yang 76b8beb289 YARN-8501. Reduce complexity of RMWebServices getApps method.
Contributed by Szilard Nemeth

(cherry picked from commit 5836e0a46b)
2018-07-19 12:32:55 -04:00
Robert Kanter dfa71428ea YARN-8518. test-container-executor test_is_empty() is broken (Jim_Brennan via rkanter)
(cherry picked from commit 1bc106a738)
2018-07-18 16:07:48 -07:00
Robert Kanter 1c7d916347 Only mount non-empty directories for cgroups (miklos.szegedi@cloudera.com via rkanter)
(cherry picked from commit 0838fe8337)
2018-07-18 16:07:48 -07:00
Robert Kanter 27e2b4b364 Disable mounting cgroups by default (miklos.szegedi@cloudera.com via rkanter)
(cherry picked from commit 351cf87c92)
2018-07-18 16:07:48 -07:00
Eric Yang d82edec3c0 YARN-8538. Fixed memory leaks in container-executor and test cases.
Contributed by Billie Rinaldi
2018-07-18 13:44:49 -04:00
Wangda Tan 44beab0b63 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda)
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
(cherry picked from commit 752dcce5f4)
2018-07-16 11:04:08 -07:00
Eric E Payne 9a79e893f7 YARN-8421: when moving app, activeUsers is increased, even though app does not have outstanding request. Contributed by Kyungwan Nam
(cherry picked from commit 937ef39b3f)
2018-07-16 16:32:05 +00:00
Jason Lowe 7cbb9597c4 YARN-8515. container-executor can crash with SIGPIPE after nodemanager restart. Contributed by Jim Brennan
(cherry picked from commit 17118f446c)
2018-07-13 10:06:38 -05:00
Sunil G 9b4ead92c8 YARN-8512. ATSv2 entities are not published to HBase from second attempt onwards. Contributed by Rohith Sharma K S.
(cherry picked from commit 7f1d3d0e9d)
2018-07-11 12:27:12 +05:30
Sunil G 6f10491e64 YARN-8473. Containers being launched as app tears down can leave containers in NEW state. Contributed by Jason Lowe.
(cherry picked from commit 705e2c1f7c)
2018-07-10 20:12:47 +05:30
Rohith Sharma K S 785ed3f8ae YARN-8302. ATS v2 should handle HBase connection issue properly. Contributed by Billie Rinaldi.
(cherry picked from commit ba68320449)
2018-07-06 15:20:39 -07:00
Sunil G 844538d669 YARN-8459. Improve Capacity Scheduler logs to debug invalid states. Contributed by Wangda Tan.
(cherry picked from commit 51654a3962)
2018-07-03 10:00:05 -07:00
Junping Du 3d6ba2dd4e yarn.resourcemanager.fail-fast is used inconsistently. Contributed by Yuanbo Liu.
(cherry picked from commit d9ba6f3656)
2018-07-03 22:36:24 +08:00
Sunil G 8ffc191f61 YARN-8415. TimelineWebServices.getEntity should throw ForbiddenException instead of 404 when ACL checks fail. Contributed by Suma Shivaprasad.
(cherry picked from commit fa9ef15ecd)
2018-07-02 15:35:45 -07:00
Shane Kumpf 2f264764a4 YARN-8485. Priviledged container app launch is failing intermittently. Contributed by Eric Yang
(cherry picked from commit 53e267fa72)
2018-07-02 16:23:23 -06:00
Eric Yang 0aa7272199 YARN-8465. Fixed docker container status for node manager restart.
Contributed by Shane Kumpf
2018-07-02 13:40:02 -04:00
Jason Lowe dbb0a5f51f YARN-8451. Multiple NM heartbeat thread created when a slow NM resync with RM. Contributed by Botong Huang
(cherry picked from commit 100470140d)
2018-06-29 13:11:41 -05:00
Sunil G e7f2c9886a YARN-8455. Add basic ACL check for all ATS v2 REST APIs. Contributed by Rohith Sharma K S.
(cherry picked from commit 469b29c081)
2018-06-29 10:03:53 -07:00
Sunil G 122cb81b65 YARN-8379. Improve balancing resources in already satisfied queues by using Capacity Scheduler preemption. Contributed by Zian Chen.
(cherry picked from commit 291194302c)
2018-06-28 10:24:03 -07:00
Eric Yang 3ce3571a34 YARN-8409. Fixed NPE in ActiveStandbyElectorBasedElectorService.
Contributed by Chandni Singh
2018-06-28 13:11:56 -04:00
Wangda Tan ec37e02bc9 YARN-8464. Async scheduling thread could be interrupted when there are no NodeManagers in cluster. (Sunil G via wangda)
Change-Id: I4f5f856373378685713e77752ba6cf0988a66065
(cherry picked from commit bedc4fe079)
2018-06-26 19:34:15 -07:00
Wangda Tan 8f226f4f1d YARN-8423. GPU does not get released even though the application gets killed. (Sunil G via wangda)
Change-Id: I570db7d60f8c6c21762dd618a9207d1107c486a0
(cherry picked from commit ada8f63d0b)
2018-06-26 19:34:10 -07:00
Eric Yang f6aa6a6981 YARN-8108. Added option to disable loading existing filters to prevent
security filter from initialize twice.
            Contributed by Sunil Govindan

(cherry picked from commit b69ba0f330)
2018-06-26 17:45:13 -04:00
Weiwei Yang bfdbc9dea2 YARN-8443. Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers. Contributed by Tao Yang.
(Cherry picked from commit 440140cea6)
2018-06-25 09:28:57 +08:00
Eric Yang 3b3a85002d YARN-8326. Removed exit code file check for launched container.
Contributed by Shane Kumpf

(cherry picked from commit 8a32bc39eb)
2018-06-22 19:23:57 -04:00
Rohith Sharma K S 1aa6c9407c YARN-8155. Improve ATSv2 client logging in RM and NM publisher. Contributed by Abhishek Modi.
(cherry picked from commit 9119b3cf8f)
2018-06-14 13:51:13 +05:30
Eric Yang 934a005f72 YARN-8259. Improve privileged docker container liveliness checks.
Contributed by Shane Kumpf

(cherry picked from commit 22994889dc)
2018-06-13 19:26:23 -04:00
Sunil G ed20c59d36 YARN-8404. Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down. Contributed by Rohith Sharma K S
(cherry picked from commit 6307962b93)
2018-06-13 16:09:52 +05:30
Weiwei Yang 4488ad5297 YARN-8394. Improve data locality documentation for Capacity Scheduler. Contributed by Weiwei Yang. 2018-06-13 13:54:09 +08:00
Inigo Goiri d5cd12c082 HADOOP-15529. ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit 6e756e8a62)
2018-06-12 10:25:05 -07:00
Rohith Sharma K S af0309d793 YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri.
(cherry picked from commit 2df73dace0)
2018-06-12 17:23:22 +05:30
Inigo Goiri 65d2554ad8 YARN-8370. Some Node Manager tests fail on Windows due to improper path/file separator. Contributed by Anbang Hu.
(cherry picked from commit 2b2f672022)
2018-06-11 19:26:27 -07:00
Inigo Goiri 63c20f9021 YARN-8359. Exclude containermanager.linux test classes on Windows. Contributed by Jason Lowe.
(cherry picked from commit 3492a1db2c0654ce5375360caa74a34f928f23be)
2018-06-07 17:10:09 -07:00
Rohith Sharma K S a95f216945 YARN-8399. NodeManager is giving 403 GSS exception post upgrade to 3.1 in secure mode. Contributed by Sunil Govindan.
(cherry picked from commit 58bc34f1e3)
2018-06-07 10:27:08 +05:30
Robert Kanter 93d6ed859e YARN-4677. RMNodeResourceUpdateEvent update from scheduler can lead to race condition (wilfreds and gphillips via rkanter)
(cherry picked from commit 0cd145a443)
2018-06-04 15:32:47 -07:00
Miklos Szegedi 21948b048e YARN-8382. cgroup file leak in NM. Contributed by Hu Ziqian.
(cherry picked from commit 925fdf761a513130e23c10575c7328c8681cff1d)
2018-06-04 10:56:13 -07:00
Billie Rinaldi e59e9feefa YARN-8342. Enable untrusted docker image to run with launch command. Contributed by Eric Yang
(cherry picked from commit 31998643a5)
2018-06-02 14:49:21 -07:00
Rohith Sharma K S f7d0ca71ac More YARN pages need to honor yarn.resourcemanager.display.per-user-apps(addendum). Contributed by Sunil G.
(cherry picked from commit 8261f9e571)
2018-06-02 09:43:08 +05:30
Wangda Tan 0be93a7973 YARN-8349. Remove YARN registry entries when a service is killed by the RM. (Billie Rinaldi via wangda)
Change-Id: Ia58db3637789a8921482f564aa9bdf99c45cc36c
(cherry picked from commit ff583d3fa3)
2018-06-01 15:53:26 -07:00
Wangda Tan 1cee85d8c4 YARN-7962. Race Condition When Stopping DelegationTokenRenewer causes RM crash during failover. (BELUGA BEHR via wangda)
Change-Id: I617e2645f60a57080058ad5f06af860fb3f682c8
(cherry picked from commit 931f78718f)
2018-06-01 15:47:51 -07:00
Wangda Tan cc460fea2d YARN-8384. stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop. (Eric Yang via wangda)
Change-Id: Idbb2cd250f4f62dc32993e0d3ca3ec5684616baa
(cherry picked from commit 3a6bd77550)
2018-06-01 15:47:45 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) a468400bad YARN-8197. Fixed AM IP Filter and Webapp proxy to redirect app tracking-URLs correctly when UI is secure. Contributed by Sunil Govindan.
(cherry picked from commit 6b74f5d7fc)
2018-05-31 17:04:45 -07:00
Weiwei Yang ec4240a7fa YARN-8367. Fix NPE in SingleConstraintAppPlacementAllocator when placement constraint in SchedulingRequest is null. Contributed by Weiwei Yang.
(Cherry picked from commit 6468071f13)
2018-05-31 21:04:41 +08:00
Eric E Payne 3d06d75423 YARN-8377: Javadoc build failed in hadoop-yarn-server-nodemanager. Contributed by Takanobu Asanuma
(cherry picked from commit e44c0849d7)
2018-05-30 17:28:14 +00:00
Eric Yang 03209e8966 YARN-8362. Bugfix logic in container retries in node manager.
Contributed by Chandni Singh

(cherry picked from commit 135941e00d)
2018-05-29 17:04:01 -04:00
Jason Lowe a1fd04c4f4 YARN-8329. Docker client configuration can still be set incorrectly. Contributed by Shane Kumpf
(cherry picked from commit 4827e9a908)
2018-05-29 14:48:01 -05:00
Wangda Tan 3eb1cb18c7 YARN-8369. Javadoc build failed due to 'bad use of >'. (Takanobu Asanuma via wangda)
Change-Id: I79a42154e8f86ab1c3cc939b3745024b8eebe5f4
(cherry picked from commit 17aa40f669)
2018-05-29 09:28:34 -07:00
Jason Lowe 500b0ee2ce YARN-8338. TimelineService V1.5 doesn't come up after HADOOP-15406. Contributed by Vinod Kumar Vavilapalli
(cherry picked from commit 31ab960f4f)
2018-05-29 11:03:19 -05:00
Sunil G 7f9caf8986 YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne.
(cherry picked from commit 7c343669ba)
2018-05-28 16:33:48 +05:30
Wangda Tan 4d41cb1696 YARN-8213. Add Capacity Scheduler performance metrics. (Weiwei Yang via wangda)
Change-Id: Ieea6f3eeb83c90cd74233fea896f0fcd0f325d5f
(cherry picked from commit f24c842d52)
2018-05-25 22:04:54 -07:00
Eric E Payne 053937f436 YARN-8292: Fix the dominant resource preemption cannot happen when some of the resource vector becomes negative. Contributed by Wangda Tan.
(cherry picked from commit 8d5509c681)
2018-05-25 16:27:09 +00:00
Shane Kumpf de43b41d0f YARN-6919. Add default volume mount list. Contributed by Eric Badger 2018-05-24 09:35:35 -06:00
Rohith Sharma K S 923c833da4 YARN-8319. More YARN pages need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G.
(cherry picked from commit c05b5d424b)
2018-05-24 14:22:43 +05:30
Inigo Goiri 61b5b2f4f7 YARN-8344. Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit e99e5bf104)
2018-05-23 14:16:03 -07:00
Wangda Tan 6aaf33e50c YARN-8290. SystemMetricsPublisher.appACLsUpdated should be invoked after application information is published to ATS to avoid "User is not set in the application report" Exception. (Eric Yang via wangda)
Change-Id: I0ac6ddd19740d1aa7dd07111cd11af71ddc2fcaf
(cherry picked from commit bd15d2396e)
2018-05-22 13:39:15 -07:00
Eric Yang 7c3f236783 YARN-7960. Added security flag no-new-privileges for YARN Docker integration.
Contributed by Eric Badger

(cherry picked from commit 6176d2b35c)
2018-05-22 13:47:08 -04:00
Jason Lowe 9788c8c01e YARN-8206. Sending a kill does not immediately kill docker containers. Contributed by Eric Badger
(cherry picked from commit 5f11288e41)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java
2018-05-22 09:49:31 -05:00
Eric E Payne 5b2bbff141 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam.
(cherry picked from commit 0b4c44bdee)
2018-05-21 20:19:44 +00:00
Eric Yang aafaa5f99b YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag.
Contributed by Chandni Singh

(cherry-picked from commit d45a0b7d73)
2018-05-17 17:32:29 -04:00
Jason Lowe 79b2a508e0 YARN-8284. get_docker_command refactoring. Contributed by Eric Badger
(cherry picked from commit d47c09dcb1)
2018-05-16 09:34:24 -05:00
Weiwei Yang 2bb39330a4 YARN-8278. DistributedScheduling is not working in HA. Contributed by Bibin A Chundatt.
(Cherry picked from commit 2bb647bb91)
2018-05-15 18:47:15 +08:00
Haibo Chen 8d3598525d YARN-8130 Race condition when container events are published for KILLED applications. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit 2d00a0c71b)
2018-05-15 11:40:47 +05:30
Eric Yang 8edcbe9460 YARN-8265. Improve DNS handling on docker IP changes.
Contributed by Billie Rinaldi

(cherry picked from commit 0ff94563b9)
2018-05-11 22:41:30 -07:00
Jason Lowe dfe73334c2 YARN-7654. Support ENTRY_POINT for docker container. Contributed by Eric Yang
(cherry picked from commit 6c8e51ca7e)
2018-05-11 18:57:20 -05:00
Jason Lowe f6d6a2ab04 YARN-8244. TestContainerSchedulerQueuing.testStartMultipleContainers failed. Contributed by Jim Brennan
(cherry picked from commit dc912994a1)
2018-05-11 14:13:55 -05:00
Eric Yang 645429ca28 YARN-8274. Fixed a bug on docker start command.
Contributed by Jason Lowe
2018-05-11 14:26:20 -04:00
Weiwei Yang b4213bfd90 YARN-7003. DRAINING state of queues is not recovered after RM restart. Contributed by Tao Yang.
(Cherry picked from commit 9db9cd95bd)
2018-05-11 10:55:05 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 4c0a6b67f3 YARN-8249. Fixed few REST APIs in RMWebServices to have static-user check. Contributed by Sunil Govindan.
(cherry picked from commit d76fbbc9b8)
2018-05-10 19:07:23 -07:00
Robert Kanter 0506c762b2 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter)
(cherry picked from commit c8b53c4364)
2018-05-10 09:52:11 -07:00
Vrushali C ad4d4153da YARN-8247 Incorrect HTTP status code returned by ATSv2 for non-whitelisted users. Contributed by Rohith Sharma K S 2018-05-09 22:30:59 -07:00
bibinchundatt b29479c93a YARN-8201. Skip stacktrace of few exception from ClientRMService. Contributed by Bilwa S T.
(cherry picked from commit cc0310a526)
2018-05-10 09:29:43 +05:30
Eric Yang 6ce89ea2a8 YARN-8261. Fixed a bug in creation of localized container directory.
Contributed by Jason Lowe

(cherry picked from commit af4fc2e628)
2018-05-09 13:34:54 -04:00
Rohith Sharma K S e3a7dbc4ec YARN-8239. [UI2] Clicking on Node Manager UI under AM container info / App Attempt page goes to old RM UI. Contributed by Sunil G.
(cherry picked from commit eb7b256cc2)
2018-05-09 12:33:49 +05:30
Jason Lowe 39236da2ff YARN-8207. Docker container launch use popen have risk of shell expansion. Contributed by Eric Yang.
(cherry picked from commit a2ea756420)
2018-05-08 15:37:24 -05:00
Wangda Tan 0873cfcb56 YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian via wangda)
Change-Id: Ia21e1da6871570c993bbedde76ce32929e95970f
(cherry picked from commit 6b96a73bb0)
2018-05-08 11:35:11 -07:00
Rohith Sharma K S b69c65fc6b YARN-8253. HTTPS Ats v2 api call fails with 'bad HTTP parsed'. Contributed by Charan Hebri.
(cherry picked from commit 7450583721)
2018-05-08 12:31:35 +05:30
Weiwei Yang 68dd01efc1 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang.
(Cherry picked from commit 67f239c42f)
2018-05-07 11:41:01 +08:00
Eric Yang a984abc472 YARN-8223. Improved yarn auxiliary service to load jar file from HDFS.
Contributed by Zian Chen

(cherry picked from commit 8cdb032aff)
2018-05-04 12:38:28 -04:00
Billie Rinaldi 04c3f11de4 YARN-7818. Remove privileged operation warnings during container launch for the ContainerRuntimes. Contributed by Shane Kumpf
(cherry picked from commit 502914ca32)
2018-05-04 08:55:30 -07:00
Eric Yang cf46533bca YARN-7973. Added ContainerRelaunch feature for Docker containers.
Addendum patch 001
            Contributed by Shane Kumpf
2018-05-03 12:46:04 -04:00
Eric Yang f729fb0baa YARN-8194. Fixed reinitialization error for LinuxContainerExecutor.
Contributed by Chandni Singh

(cherry picked from commit f4d280f02b)
2018-05-03 11:59:35 -04:00
Eric Yang e4b9981612 YARN-7973. Added ContainerRelaunch feature for Docker containers.
Contributed by Shane Kumpf
2018-05-03 11:57:01 -04:00
Rohith Sharma K S 4c93cd228c YARN-8217. RmAuthenticationFilterInitializer and TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator. Contributed by Suma Shivaprasad.
(cherry picked from commit ee2ce923a9)
2018-05-03 18:20:21 +05:30
Eric Yang 61fac20c9e YARN-8209. Fixed NPE in Yarn Service deletion.
Contributed by Eric Badger
2018-05-02 15:05:30 -04:00
Weiwei Yang 6fce88765a YARN-8222. Fix potential NPE when gets RMApp from RM context. Contributed by Tao Yang. 2018-05-02 18:02:24 +08:00
Yiqun Lin 2741af505f
YARN-6385. Fix checkstyle warnings in TestFileSystemApplicationHistoryStore
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3265b55119)
2018-05-02 18:15:53 +09:00
Weiwei Yang f34f4e9c11 YARN-8212. Pending backlog for async allocation threads should be configurable. Contributed by Tao Yang.
(cherry picked from commit 2d319e3793)
2018-05-01 09:57:15 +08:00
Eric Yang 0fc55a8f57 YARN-8228. Added hostname length check for docker container.
Contributed by Shane Kumpf

(cherry picked from commit a966ec6e23)
2018-04-30 19:14:37 -04:00
Konstantinos Karanasos 9d2967098d YARN-8195. Fix constraint cardinality check in the presence of multiple target allocation tags. Contributed by Weiwei Yang.
(cherry picked from commit 9b09555451)
2018-04-30 11:55:26 -07:00
Wangda Tan 090a8a274e YARN-8005. Add unit tests for queue priority with dominant resource calculator. (Zian Chen via wangda)
Change-Id: I17a645f20869a1e5d86fa7a325c93fec908b91dc
(cherry picked from commit 92c5331423)
2018-04-27 13:13:52 -07:00
Rohith Sharma K S b3fd464e55 YARN-8221. RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G.
(cherry picked from commit ef3ecc308d)
2018-04-27 22:59:16 +05:30
Sunil G 28e63f3d88 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen.
(cherry picked from commit 71220d218d)
2018-04-27 10:42:59 +05:30
Sunil G a1a2f1ddc1 YARN-8215. ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM. Contributed by Rohith Sharma K S.
(cherry picked from commit 7fc09c4f72)
2018-04-27 10:34:29 +05:30
Sunil G e2901d97ba YARN-8205. Application State is not updated to ATS if AM launching is delayed. Contributed by Rohith Sharma K S.
(cherry picked from commit 1634de0fc1)
2018-04-27 10:28:18 +05:30
Eric Yang 050bd4cd2a YARN-8043. Added the exception message for failed launches running under LCE.
Contributed by Shane Kumpf
2018-04-26 08:26:10 -07:00
Wangda Tan af286319fd YARN-8193. YARN RM hangs abruptly (stops allocating resources) when running successive applications. (Zian Chen via wangda)
Change-Id: Ia83dd2499ee9000b9e09ae5a932f21a13c0ddee6
(cherry picked from commit af986b442b7be0f2fad1241ca9e267a15c20bf43)
2018-04-25 22:16:07 -07:00
Sunil G 44d28692a8 YARN-8188. RM Nodes UI data table index for sorting column need to be corrected post Application tags display. Contributed by Weiwei Yang.
(cherry picked from commit 626690612cd0957316628376744a8be62f891665)
2018-04-25 14:58:07 +05:30
Wangda Tan 3043a93d46 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda)
Change-Id: I347871d672001653a3afe2e99adefd74e0d798cd
(cherry picked from commit bb3c504764f807fccba7f28298a12e2296f284cb)
2018-04-24 17:44:58 -07:00
Robert Kanter ea7ad50499 HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter)
(cherry picked from commit 7ab08a9c37)
2018-04-23 16:25:41 -07:00
Eric Yang 5556cf397c YARN-8064. Bugfix for clean up container-executor cmd helper file.
Contributed by Eric Badger
2018-04-20 19:12:06 -04:00
Konstantinos Karanasos 18c86a3fb6 YARN-8111. Simplify PlacementConstraints API by removing allocationTagToIntraApp. Contributed by Weiwei Yang.
(cherry picked from commit 28e2244390)
2018-04-20 12:25:08 -07:00
Jason Lowe 71b0d52987 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie
(cherry picked from commit 766544c0b0)
2018-04-20 13:23:46 -05:00
Sunil G 162db6e826 YARN-6827. [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. Contributed by Rohith Sharma K S.
(cherry picked from commit 7d06806dfd)
2018-04-20 00:06:44 +05:30
Weiwei Yang c3ca10c22a YARN-7734. Fix UT failure TestContainerLogsPage#testContainerLogPageAccess. Contributed by Tao Yang.
(cherry picked from commit 411993f6e5)
2018-04-18 19:40:34 -07:00
Shashikant Banerjee a13b3edfd8
HADOOP-15396. Some java source files are executable
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
2018-04-19 08:36:32 +09:00
Rohith Sharma K S fa007d986b YARN-8145. yarn rmadmin -getGroups doesn't return updated groups for user. Contributed by Sunil G.
(cherry picked from commit 241d985467)
2018-04-18 13:40:32 +05:30
Inigo Goiri 8f300c9ce1 YARN-8165. Incorrect queue name logging in AbstractContainerAllocator. Contributed by Weiwei Yan.
(cherry picked from commit dd5e18c4ae)
2018-04-17 15:28:13 +05:30