Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,912 Commits 326 Branches 371 Tags
Commit Graph

3408 Commits

Author SHA1 Message Date
Weiwei Yang
4edd883d48 YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang. 
(cherry picked from commit abde1e1f58d5b699e4b8e460cff68e154738169b)
 2019-01-21 21:27:40 +08:00
Wangda Tan
a685ffe9a9 YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda) 
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
(cherry picked from commit 6d7eedfd28cc1712690db2f6ca8a281b0901ee28)
(cherry picked from commit fe7cb2d84ac160c5fed00640d85e2c5c4c6d2412)
 2019-01-17 15:17:34 -08:00
Weiwei Yang
91e9c9f96e YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed Wilfred Spiegelenburg. 2019-01-08 13:56:21 +08:00
Wangda Tan
3570713a69 YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda) 
Change-Id: I416268888a7b6f097d218d84e8497dd70b4b6d8f
 2019-01-07 12:30:30 -08:00
Wangda Tan
31ea2f7806 Preparing for 3.1.3 development 
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
 2019-01-07 10:04:58 -08:00
Weiwei Yang
d6464629ca YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie. 
(cherry picked from commit cfe89e6f963ba25b5fff1ce48cad36d74b3c789c)
 2019-01-04 01:37:47 +08:00
Eric Yang
b4fa1830a8 YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator. 
Contributed by Tarun Parimi

(cherry picked from commit 71e0b0d8005ea1952dc7e582db15c2ac09df7c91)
 2018-12-17 12:08:35 -05:00
Eric Yang
690d760174 YARN-9125. Fixed Carriage Return detection in Docker container launch command. 
Contributed by Billie Rinaldi

(cherry picked from commit b2d7204ed0b64405a0adbe07e3eaf385e046efa1)
 2018-12-14 17:55:38 -05:00
Weiwei Yang
14ecdb62b6 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters. 
(cherry picked from commit 1c09a10e9601bd628f1e887f3bf92c5a4ac286ab)
 2018-12-10 12:17:00 +08:00
Jonathan Hung
7b523e6a77 YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics 
(cherry picked from commit 978ab3e958227220cb6f1a08ae6e7cdb8a46628b)
(cherry picked from commit dca69d178dba21c41fd1293187f29143f7e81e19)
 2018-12-07 10:45:57 -08:00
Eric Yang
7ef4ff1905 YARN-9071. Improved status update for reinitialized containers. 
Contributed by Chandni Singh

(cherry picked from commit 1b790f4dd1f682423d5dbb8e70c6225cbddce989)
 2018-12-05 19:05:26 -05:00
Jonathan Hung
2cb9479bfc YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:16:39 -08:00
bibinchundatt
8be2d16b94 YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T. 
(cherry picked from commit 07142f54a8c7f70857e99c041f3a2a5189c809b5)
 2018-11-29 22:08:35 +05:30
Jason Lowe
d9457df989 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 3ce99e32f7d7887412cae8337cd4ebeb3b2ee308)
 2018-11-28 08:54:04 -06:00
Eric Yang
bec5036397 YARN-8665. Added Yarn service cancel upgrade option. 
Contributed by Chandni Singh
 2018-11-27 16:29:08 -05:00
Eric Yang
463de48f04 YARN-8986. Added port publish for Docker container running with bridge. 
Contributed by Charo Zhang
 2018-11-27 14:28:37 -05:00
Weiwei Yang
17a41f5d86 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 
(cherry picked from commit d027a24f0349b60efa5125c330058f123771748f)
 2018-11-18 23:24:37 +08:00
Eric Yang
fce0350289 YARN-8160. Support upgrade of service that use docker containers. 
Contributed by Chandni Singh
 2018-11-16 16:01:25 -05:00
Rohith Sharma K S
095635d984 YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report. 
(cherry picked from commit ee3355be3cd1cd956296a1b79b264177ff139ce6)
 2018-11-16 18:38:11 +05:30
Akira Ajisaka
daad077121
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-10 14:37:35 +09:00
Weiwei Yang
a3b61baf94 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji. 
(cherry picked from commit c96cbe8659587cfc114a96aab1be5cc85029fe44)
 2018-11-07 22:50:05 +08:00
Akira Ajisaka
52af95fdce
Revert "YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang." 
This reverts commit dd8479e80d3f0fe87a6edb099e7f617bff42106a.
 2018-11-07 11:33:31 +09:00
Akira Ajisaka
dd8479e80d
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 
(cherry picked from commit 951c98f89059d64fda8456366f680eff4a7a6785)
 2018-11-07 11:19:11 +09:00
Jason Lowe
7335d940de YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c7265db5bcbb446c2f779289023d454b81)
 2018-11-06 08:52:29 -06:00
Weiwei Yang
631b31110c YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang. 
(cherry picked from commit 5d6554c722f08f79bce904e021243605ee75bae3)
 2018-11-06 14:53:28 +08:00
Weiwei Yang
71999f4464 YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji. 
(cherry picked from commit c7fcca0d7ec9e31d43ef3040ecd576ec808f1f8b)
 2018-11-06 13:23:42 +08:00
Jonathan Hung
221494a75c YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 
(cherry picked from commit 2ab611d48b7669b31bd2c9b918f47251da77d0f6)
 2018-11-01 14:31:22 -07:00
Rohith Sharma K S
d9a494b1e0 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. 
(cherry picked from commit 4ec4ec69711180d642c5b56cd3d3dbdf44d3c61f)
 2018-10-30 11:50:56 +05:30
Weiwei Yang
70efe253f3 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23c5d1bd52d04c303506806d0dc61cd2)
 2018-10-29 11:56:31 +08:00
Jason Lowe
3be72b7aa2 YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 93fb3b4b9cbc63e65d590676a5a318d555a25904)
 2018-10-23 12:55:48 -05:00
Rohith Sharma K S
3e3b088856 YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli. 
(cherry picked from commit 0b62983c5a9361eb832784f134f140f9926c9ec6)
 2018-10-23 14:08:06 +05:30
Sunil G
0cb184d6e9 YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh. 
(cherry picked from commit 2202e00ba8a44ad70f0a90e6c519257e3ae56a36)
 2018-10-23 09:58:05 +05:30
Eric Yang
e86efa8712 YARN-8910. Fixed misleading log statement when container max retries is infinite. 
Contributed by Chandni Singh

(cherry picked from commit 47ad98b2e1b4e9ad9304aca4d452da428778e7ee)
 2018-10-19 13:50:32 -04:00
Weiwei Yang
beca90ece8 YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 13cc0f50ea1c5f8978455e34f49716ddb0e3a143)
 2018-10-19 10:02:46 +08:00
Wangda Tan
46baafedf1 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07df2a3ed6273b96ae857c278fd72718)
 2018-10-18 12:29:19 -07:00
Weiwei Yang
a0060cf8ee Revert "YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach." 
This reverts commit ce4a0898df5e9891f0388aa65a754f0ff85b56ac.
 2018-10-10 21:41:00 +08:00
Weiwei Yang
3968ce1073 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan. 
(cherry picked from commit edce866489d83744f3f47a3b884b0c6136885e4a)
 2018-10-10 09:48:56 +08:00
Weiwei Yang
ce4a0898df YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach. 2018-10-09 22:30:42 +08:00
Wangda Tan
86a1ad4428 YARN-8844. TestNMProxy unit test is failing. (Eric Yang via wangda) 
Change-Id: I241fa8701b6f1dbcad87fd2e9a429e32e7aa40f5
(cherry picked from commit b3ac8869338faeabedbc27e95e3166c830a9a761)
 2018-10-04 10:49:29 -07:00
Shane Kumpf
adbc010d0f YARN-8785. Improve the error message when a bind mount is not whitelisted. Contributed by Simon Prewo 
(cherry picked from commit 5edb9d3b972ad51e23b9cac7928b38876068a463)
 2018-10-02 07:26:45 -06:00
Eric E Payne
c306da08ec YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang. 
(cherry picked from commit 8598b498bcaf4deffa822f871a26635bdf3d9d5c)
 2018-09-28 15:34:23 +00:00
Vrushali C
5b72aa04e1 YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks. 
(cherry picked from commit 90e2e493b3dc8be54f655b957b98a4bc0e003684)
 2018-09-28 10:31:38 +05:30
Jason Lowe
a56a345e07 YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang 
(cherry picked from commit 6b988d821e62d29c118e10a7213583b92c302baf)
 2018-09-26 16:15:48 -07:00
Rohith Sharma K S
d473152e6a YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt. 2018-09-26 20:36:34 +05:30
Rohith Sharma K S
47306cc2db YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt. 
(cherry picked from commit 50bc7746d7593bf6adc92fc9807a99cf53b4aef3)
 2018-09-25 11:40:09 +05:30
Eric Yang
e9315f6688 YARN-8801. Fixed header comments for docker utility functions. 
Contributed by Zian Chen
 2018-09-20 13:12:29 -04:00
Jason Lowe
3fb6787295 YARN-8784. DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk. Contributed by Eric Badger 
(cherry picked from commit 6b5838ed3220f992092c7348f92f1d9d0d4a3061)
 2018-09-19 16:49:21 -05:00
Weiwei Yang
aaf0b119e5 YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang. 
(cherry picked from commit 0712537e799bc03855d548d1f4bd690dd478b871)
 2018-09-19 19:38:09 +08:00
Jason Lowe
3d77094cf2 YARN-8648. Container cgroups are leaked when using docker. Contributed by Jim Brennan 
(cherry picked from commit 2df0a8dcb3dfde15d216481cc1296d97d2cb5d43)
 2018-09-18 15:43:10 -05:00
Weiwei Yang
00a469138d YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi. 
(cherry picked from commit f1a893fdbc2dbe949cae786f08bdb2651b88d673)
 2018-09-14 16:40:35 +08:00