Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,579 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3558 Commits

Author SHA1 Message Date
Robert Kanter dfa71428ea YARN-8518. test-container-executor test_is_empty() is broken (Jim_Brennan via rkanter) 
(cherry picked from commit 1bc106a738)
 2018-07-18 16:07:48 -07:00
Robert Kanter 1c7d916347 Only mount non-empty directories for cgroups (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 0838fe8337)
 2018-07-18 16:07:48 -07:00
Robert Kanter 27e2b4b364 Disable mounting cgroups by default (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 351cf87c92)
 2018-07-18 16:07:48 -07:00
Eric Yang d82edec3c0 YARN-8538. Fixed memory leaks in container-executor and test cases. 
Contributed by Billie Rinaldi
 2018-07-18 13:44:49 -04:00
Wangda Tan 44beab0b63 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
(cherry picked from commit 752dcce5f4)
 2018-07-16 11:04:08 -07:00
Eric E Payne 9a79e893f7 YARN-8421: when moving app, activeUsers is increased, even though app does not have outstanding request. Contributed by Kyungwan Nam 
(cherry picked from commit 937ef39b3f)
 2018-07-16 16:32:05 +00:00
Jason Lowe 7cbb9597c4 YARN-8515. container-executor can crash with SIGPIPE after nodemanager restart. Contributed by Jim Brennan 
(cherry picked from commit 17118f446c)
 2018-07-13 10:06:38 -05:00
Sunil G 9b4ead92c8 YARN-8512. ATSv2 entities are not published to HBase from second attempt onwards. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7f1d3d0e9d)
 2018-07-11 12:27:12 +05:30
Sunil G 6f10491e64 YARN-8473. Containers being launched as app tears down can leave containers in NEW state. Contributed by Jason Lowe. 
(cherry picked from commit 705e2c1f7c)
 2018-07-10 20:12:47 +05:30
Rohith Sharma K S 785ed3f8ae YARN-8302. ATS v2 should handle HBase connection issue properly. Contributed by Billie Rinaldi. 
(cherry picked from commit ba68320449)
 2018-07-06 15:20:39 -07:00
Sunil G 844538d669 YARN-8459. Improve Capacity Scheduler logs to debug invalid states. Contributed by Wangda Tan. 
(cherry picked from commit 51654a3962)
 2018-07-03 10:00:05 -07:00
Junping Du 3d6ba2dd4e yarn.resourcemanager.fail-fast is used inconsistently. Contributed by Yuanbo Liu. 
(cherry picked from commit d9ba6f3656)
 2018-07-03 22:36:24 +08:00
Sunil G 8ffc191f61 YARN-8415. TimelineWebServices.getEntity should throw ForbiddenException instead of 404 when ACL checks fail. Contributed by Suma Shivaprasad. 
(cherry picked from commit fa9ef15ecd)
 2018-07-02 15:35:45 -07:00
Shane Kumpf 2f264764a4 YARN-8485. Priviledged container app launch is failing intermittently. Contributed by Eric Yang 
(cherry picked from commit 53e267fa72)
 2018-07-02 16:23:23 -06:00
Eric Yang 0aa7272199 YARN-8465. Fixed docker container status for node manager restart. 
Contributed by Shane Kumpf
 2018-07-02 13:40:02 -04:00
Jason Lowe dbb0a5f51f YARN-8451. Multiple NM heartbeat thread created when a slow NM resync with RM. Contributed by Botong Huang 
(cherry picked from commit 100470140d)
 2018-06-29 13:11:41 -05:00
Sunil G e7f2c9886a YARN-8455. Add basic ACL check for all ATS v2 REST APIs. Contributed by Rohith Sharma K S. 
(cherry picked from commit 469b29c081)
 2018-06-29 10:03:53 -07:00
Sunil G 122cb81b65 YARN-8379. Improve balancing resources in already satisfied queues by using Capacity Scheduler preemption. Contributed by Zian Chen. 
(cherry picked from commit 291194302c)
 2018-06-28 10:24:03 -07:00
Eric Yang 3ce3571a34 YARN-8409. Fixed NPE in ActiveStandbyElectorBasedElectorService. 
Contributed by Chandni Singh
 2018-06-28 13:11:56 -04:00
Wangda Tan ec37e02bc9 YARN-8464. Async scheduling thread could be interrupted when there are no NodeManagers in cluster. (Sunil G via wangda) 
Change-Id: I4f5f856373378685713e77752ba6cf0988a66065
(cherry picked from commit bedc4fe079)
 2018-06-26 19:34:15 -07:00
Wangda Tan 8f226f4f1d YARN-8423. GPU does not get released even though the application gets killed. (Sunil G via wangda) 
Change-Id: I570db7d60f8c6c21762dd618a9207d1107c486a0
(cherry picked from commit ada8f63d0b)
 2018-06-26 19:34:10 -07:00
Eric Yang f6aa6a6981 YARN-8108. Added option to disable loading existing filters to prevent 
security filter from initialize twice.
            Contributed by Sunil Govindan

(cherry picked from commit b69ba0f330)
 2018-06-26 17:45:13 -04:00
Weiwei Yang bfdbc9dea2 YARN-8443. Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers. Contributed by Tao Yang. 
(Cherry picked from commit 440140cea6)
 2018-06-25 09:28:57 +08:00
Eric Yang 3b3a85002d YARN-8326. Removed exit code file check for launched container. 
Contributed by Shane Kumpf

(cherry picked from commit 8a32bc39eb)
 2018-06-22 19:23:57 -04:00
Rohith Sharma K S 1aa6c9407c YARN-8155. Improve ATSv2 client logging in RM and NM publisher. Contributed by Abhishek Modi. 
(cherry picked from commit 9119b3cf8f)
 2018-06-14 13:51:13 +05:30
Eric Yang 934a005f72 YARN-8259. Improve privileged docker container liveliness checks. 
Contributed by Shane Kumpf

(cherry picked from commit 22994889dc)
 2018-06-13 19:26:23 -04:00
Sunil G ed20c59d36 YARN-8404. Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down. Contributed by Rohith Sharma K S 
(cherry picked from commit 6307962b93)
 2018-06-13 16:09:52 +05:30
Weiwei Yang 4488ad5297 YARN-8394. Improve data locality documentation for Capacity Scheduler. Contributed by Weiwei Yang. 2018-06-13 13:54:09 +08:00
Inigo Goiri d5cd12c082 HADOOP-15529. ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 6e756e8a62)
 2018-06-12 10:25:05 -07:00
Rohith Sharma K S af0309d793 YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace0)
 2018-06-12 17:23:22 +05:30
Inigo Goiri 65d2554ad8 YARN-8370. Some Node Manager tests fail on Windows due to improper path/file separator. Contributed by Anbang Hu. 
(cherry picked from commit 2b2f672022)
 2018-06-11 19:26:27 -07:00
Inigo Goiri 63c20f9021 YARN-8359. Exclude containermanager.linux test classes on Windows. Contributed by Jason Lowe. 
(cherry picked from commit 3492a1db2c0654ce5375360caa74a34f928f23be)
 2018-06-07 17:10:09 -07:00
Rohith Sharma K S a95f216945 YARN-8399. NodeManager is giving 403 GSS exception post upgrade to 3.1 in secure mode. Contributed by Sunil Govindan. 
(cherry picked from commit 58bc34f1e3)
 2018-06-07 10:27:08 +05:30
Robert Kanter 93d6ed859e YARN-4677. RMNodeResourceUpdateEvent update from scheduler can lead to race condition (wilfreds and gphillips via rkanter) 
(cherry picked from commit 0cd145a443)
 2018-06-04 15:32:47 -07:00
Miklos Szegedi 21948b048e YARN-8382. cgroup file leak in NM. Contributed by Hu Ziqian. 
(cherry picked from commit 925fdf761a513130e23c10575c7328c8681cff1d)
 2018-06-04 10:56:13 -07:00
Billie Rinaldi e59e9feefa YARN-8342. Enable untrusted docker image to run with launch command. Contributed by Eric Yang 
(cherry picked from commit 31998643a5)
 2018-06-02 14:49:21 -07:00
Rohith Sharma K S f7d0ca71ac More YARN pages need to honor yarn.resourcemanager.display.per-user-apps(addendum). Contributed by Sunil G. 
(cherry picked from commit 8261f9e571)
 2018-06-02 09:43:08 +05:30
Wangda Tan 0be93a7973 YARN-8349. Remove YARN registry entries when a service is killed by the RM. (Billie Rinaldi via wangda) 
Change-Id: Ia58db3637789a8921482f564aa9bdf99c45cc36c
(cherry picked from commit ff583d3fa3)
 2018-06-01 15:53:26 -07:00
Wangda Tan 1cee85d8c4 YARN-7962. Race Condition When Stopping DelegationTokenRenewer causes RM crash during failover. (BELUGA BEHR via wangda) 
Change-Id: I617e2645f60a57080058ad5f06af860fb3f682c8
(cherry picked from commit 931f78718f)
 2018-06-01 15:47:51 -07:00
Wangda Tan cc460fea2d YARN-8384. stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop. (Eric Yang via wangda) 
Change-Id: Idbb2cd250f4f62dc32993e0d3ca3ec5684616baa
(cherry picked from commit 3a6bd77550)
 2018-06-01 15:47:45 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) a468400bad YARN-8197. Fixed AM IP Filter and Webapp proxy to redirect app tracking-URLs correctly when UI is secure. Contributed by Sunil Govindan. 
(cherry picked from commit 6b74f5d7fc)
 2018-05-31 17:04:45 -07:00
Weiwei Yang ec4240a7fa YARN-8367. Fix NPE in SingleConstraintAppPlacementAllocator when placement constraint in SchedulingRequest is null. Contributed by Weiwei Yang. 
(Cherry picked from commit 6468071f13)
 2018-05-31 21:04:41 +08:00
Eric E Payne 3d06d75423 YARN-8377: Javadoc build failed in hadoop-yarn-server-nodemanager. Contributed by Takanobu Asanuma 
(cherry picked from commit e44c0849d7)
 2018-05-30 17:28:14 +00:00
Eric Yang 03209e8966 YARN-8362. Bugfix logic in container retries in node manager. 
Contributed by Chandni Singh

(cherry picked from commit 135941e00d)
 2018-05-29 17:04:01 -04:00
Jason Lowe a1fd04c4f4 YARN-8329. Docker client configuration can still be set incorrectly. Contributed by Shane Kumpf 
(cherry picked from commit 4827e9a908)
 2018-05-29 14:48:01 -05:00
Wangda Tan 3eb1cb18c7 YARN-8369. Javadoc build failed due to 'bad use of >'. (Takanobu Asanuma via wangda) 
Change-Id: I79a42154e8f86ab1c3cc939b3745024b8eebe5f4
(cherry picked from commit 17aa40f669)
 2018-05-29 09:28:34 -07:00
Jason Lowe 500b0ee2ce YARN-8338. TimelineService V1.5 doesn't come up after HADOOP-15406. Contributed by Vinod Kumar Vavilapalli 
(cherry picked from commit 31ab960f4f)
 2018-05-29 11:03:19 -05:00
Sunil G 7f9caf8986 YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne. 
(cherry picked from commit 7c343669ba)
 2018-05-28 16:33:48 +05:30
Wangda Tan 4d41cb1696 YARN-8213. Add Capacity Scheduler performance metrics. (Weiwei Yang via wangda) 
Change-Id: Ieea6f3eeb83c90cd74233fea896f0fcd0f325d5f
(cherry picked from commit f24c842d52)
 2018-05-25 22:04:54 -07:00
Eric E Payne 053937f436 YARN-8292: Fix the dominant resource preemption cannot happen when some of the resource vector becomes negative. Contributed by Wangda Tan. 
(cherry picked from commit 8d5509c681)
 2018-05-25 16:27:09 +00:00
Shane Kumpf de43b41d0f YARN-6919. Add default volume mount list. Contributed by Eric Badger 2018-05-24 09:35:35 -06:00
Rohith Sharma K S 923c833da4 YARN-8319. More YARN pages need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 
(cherry picked from commit c05b5d424b)
 2018-05-24 14:22:43 +05:30
Inigo Goiri 61b5b2f4f7 YARN-8344. Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit e99e5bf104)
 2018-05-23 14:16:03 -07:00
Wangda Tan 6aaf33e50c YARN-8290. SystemMetricsPublisher.appACLsUpdated should be invoked after application information is published to ATS to avoid "User is not set in the application report" Exception. (Eric Yang via wangda) 
Change-Id: I0ac6ddd19740d1aa7dd07111cd11af71ddc2fcaf
(cherry picked from commit bd15d2396e)
 2018-05-22 13:39:15 -07:00
Eric Yang 7c3f236783 YARN-7960. Added security flag no-new-privileges for YARN Docker integration. 
Contributed by Eric Badger

(cherry picked from commit 6176d2b35c)
 2018-05-22 13:47:08 -04:00
Jason Lowe 9788c8c01e YARN-8206. Sending a kill does not immediately kill docker containers. Contributed by Eric Badger 
(cherry picked from commit 5f11288e41)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java
 2018-05-22 09:49:31 -05:00
Eric E Payne 5b2bbff141 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdee)
 2018-05-21 20:19:44 +00:00
Eric Yang aafaa5f99b YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag. 
Contributed by Chandni Singh

(cherry-picked from commit d45a0b7d73)
 2018-05-17 17:32:29 -04:00
Jason Lowe 79b2a508e0 YARN-8284. get_docker_command refactoring. Contributed by Eric Badger 
(cherry picked from commit d47c09dcb1)
 2018-05-16 09:34:24 -05:00
Weiwei Yang 2bb39330a4 YARN-8278. DistributedScheduling is not working in HA. Contributed by Bibin A Chundatt. 
(Cherry picked from commit 2bb647bb91)
 2018-05-15 18:47:15 +08:00
Haibo Chen 8d3598525d YARN-8130 Race condition when container events are published for KILLED applications. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 2d00a0c71b)
 2018-05-15 11:40:47 +05:30
Eric Yang 8edcbe9460 YARN-8265. Improve DNS handling on docker IP changes. 
Contributed by Billie Rinaldi

(cherry picked from commit 0ff94563b9)
 2018-05-11 22:41:30 -07:00
Jason Lowe dfe73334c2 YARN-7654. Support ENTRY_POINT for docker container. Contributed by Eric Yang 
(cherry picked from commit 6c8e51ca7e)
 2018-05-11 18:57:20 -05:00
Jason Lowe f6d6a2ab04 YARN-8244. TestContainerSchedulerQueuing.testStartMultipleContainers failed. Contributed by Jim Brennan 
(cherry picked from commit dc912994a1)
 2018-05-11 14:13:55 -05:00
Eric Yang 645429ca28 YARN-8274. Fixed a bug on docker start command. 
Contributed by Jason Lowe
 2018-05-11 14:26:20 -04:00
Weiwei Yang b4213bfd90 YARN-7003. DRAINING state of queues is not recovered after RM restart. Contributed by Tao Yang. 
(Cherry picked from commit 9db9cd95bd)
 2018-05-11 10:55:05 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 4c0a6b67f3 YARN-8249. Fixed few REST APIs in RMWebServices to have static-user check. Contributed by Sunil Govindan. 
(cherry picked from commit d76fbbc9b8)
 2018-05-10 19:07:23 -07:00
Robert Kanter 0506c762b2 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c4364)
 2018-05-10 09:52:11 -07:00
Vrushali C ad4d4153da YARN-8247 Incorrect HTTP status code returned by ATSv2 for non-whitelisted users. Contributed by Rohith Sharma K S 2018-05-09 22:30:59 -07:00
bibinchundatt b29479c93a YARN-8201. Skip stacktrace of few exception from ClientRMService. Contributed by Bilwa S T. 
(cherry picked from commit cc0310a526)
 2018-05-10 09:29:43 +05:30
Eric Yang 6ce89ea2a8 YARN-8261. Fixed a bug in creation of localized container directory. 
Contributed by Jason Lowe

(cherry picked from commit af4fc2e628)
 2018-05-09 13:34:54 -04:00
Rohith Sharma K S e3a7dbc4ec YARN-8239. [UI2] Clicking on Node Manager UI under AM container info / App Attempt page goes to old RM UI. Contributed by Sunil G. 
(cherry picked from commit eb7b256cc2)
 2018-05-09 12:33:49 +05:30
Jason Lowe 39236da2ff YARN-8207. Docker container launch use popen have risk of shell expansion. Contributed by Eric Yang. 
(cherry picked from commit a2ea756420)
 2018-05-08 15:37:24 -05:00
Wangda Tan 0873cfcb56 YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian via wangda) 
Change-Id: Ia21e1da6871570c993bbedde76ce32929e95970f
(cherry picked from commit 6b96a73bb0)
 2018-05-08 11:35:11 -07:00
Rohith Sharma K S b69c65fc6b YARN-8253. HTTPS Ats v2 api call fails with 'bad HTTP parsed'. Contributed by Charan Hebri. 
(cherry picked from commit 7450583721)
 2018-05-08 12:31:35 +05:30
Weiwei Yang 68dd01efc1 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f)
 2018-05-07 11:41:01 +08:00
Eric Yang a984abc472 YARN-8223. Improved yarn auxiliary service to load jar file from HDFS. 
Contributed by Zian Chen

(cherry picked from commit 8cdb032aff)
 2018-05-04 12:38:28 -04:00
Billie Rinaldi 04c3f11de4 YARN-7818. Remove privileged operation warnings during container launch for the ContainerRuntimes. Contributed by Shane Kumpf 
(cherry picked from commit 502914ca32)
 2018-05-04 08:55:30 -07:00
Eric Yang cf46533bca YARN-7973. Added ContainerRelaunch feature for Docker containers. 
Addendum patch 001
            Contributed by Shane Kumpf
 2018-05-03 12:46:04 -04:00
Eric Yang f729fb0baa YARN-8194. Fixed reinitialization error for LinuxContainerExecutor. 
Contributed by Chandni Singh

(cherry picked from commit f4d280f02b)
 2018-05-03 11:59:35 -04:00
Eric Yang e4b9981612 YARN-7973. Added ContainerRelaunch feature for Docker containers. 
Contributed by Shane Kumpf
 2018-05-03 11:57:01 -04:00
Rohith Sharma K S 4c93cd228c YARN-8217. RmAuthenticationFilterInitializer and TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator. Contributed by Suma Shivaprasad. 
(cherry picked from commit ee2ce923a9)
 2018-05-03 18:20:21 +05:30
Eric Yang 61fac20c9e YARN-8209. Fixed NPE in Yarn Service deletion. 
Contributed by Eric Badger
 2018-05-02 15:05:30 -04:00
Weiwei Yang 6fce88765a YARN-8222. Fix potential NPE when gets RMApp from RM context. Contributed by Tao Yang. 2018-05-02 18:02:24 +08:00
Yiqun Lin 2741af505f
YARN-6385. Fix checkstyle warnings in TestFileSystemApplicationHistoryStore 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3265b55119)
 2018-05-02 18:15:53 +09:00
Weiwei Yang f34f4e9c11 YARN-8212. Pending backlog for async allocation threads should be configurable. Contributed by Tao Yang. 
(cherry picked from commit 2d319e3793)
 2018-05-01 09:57:15 +08:00
Eric Yang 0fc55a8f57 YARN-8228. Added hostname length check for docker container. 
Contributed by Shane Kumpf

(cherry picked from commit a966ec6e23)
 2018-04-30 19:14:37 -04:00
Konstantinos Karanasos 9d2967098d YARN-8195. Fix constraint cardinality check in the presence of multiple target allocation tags. Contributed by Weiwei Yang. 
(cherry picked from commit 9b09555451)
 2018-04-30 11:55:26 -07:00
Wangda Tan 090a8a274e YARN-8005. Add unit tests for queue priority with dominant resource calculator. (Zian Chen via wangda) 
Change-Id: I17a645f20869a1e5d86fa7a325c93fec908b91dc
(cherry picked from commit 92c5331423)
 2018-04-27 13:13:52 -07:00
Rohith Sharma K S b3fd464e55 YARN-8221. RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 
(cherry picked from commit ef3ecc308d)
 2018-04-27 22:59:16 +05:30
Sunil G 28e63f3d88 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218d)
 2018-04-27 10:42:59 +05:30
Sunil G a1a2f1ddc1 YARN-8215. ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7fc09c4f72)
 2018-04-27 10:34:29 +05:30
Sunil G e2901d97ba YARN-8205. Application State is not updated to ATS if AM launching is delayed. Contributed by Rohith Sharma K S. 
(cherry picked from commit 1634de0fc1)
 2018-04-27 10:28:18 +05:30
Eric Yang 050bd4cd2a YARN-8043. Added the exception message for failed launches running under LCE. 
Contributed by Shane Kumpf
 2018-04-26 08:26:10 -07:00
Wangda Tan af286319fd YARN-8193. YARN RM hangs abruptly (stops allocating resources) when running successive applications. (Zian Chen via wangda) 
Change-Id: Ia83dd2499ee9000b9e09ae5a932f21a13c0ddee6
(cherry picked from commit af986b442b7be0f2fad1241ca9e267a15c20bf43)
 2018-04-25 22:16:07 -07:00
Sunil G 44d28692a8 YARN-8188. RM Nodes UI data table index for sorting column need to be corrected post Application tags display. Contributed by Weiwei Yang. 
(cherry picked from commit 626690612cd0957316628376744a8be62f891665)
 2018-04-25 14:58:07 +05:30
Wangda Tan 3043a93d46 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda) 
Change-Id: I347871d672001653a3afe2e99adefd74e0d798cd
(cherry picked from commit bb3c504764f807fccba7f28298a12e2296f284cb)
 2018-04-24 17:44:58 -07:00
Robert Kanter ea7ad50499 HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter) 
(cherry picked from commit 7ab08a9c37)
 2018-04-23 16:25:41 -07:00
Eric Yang 5556cf397c YARN-8064. Bugfix for clean up container-executor cmd helper file. 
Contributed by Eric Badger
 2018-04-20 19:12:06 -04:00
Konstantinos Karanasos 18c86a3fb6 YARN-8111. Simplify PlacementConstraints API by removing allocationTagToIntraApp. Contributed by Weiwei Yang. 
(cherry picked from commit 28e2244390)
 2018-04-20 12:25:08 -07:00
Jason Lowe 71b0d52987 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie 
(cherry picked from commit 766544c0b0)
 2018-04-20 13:23:46 -05:00
Sunil G 162db6e826 YARN-6827. [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7d06806dfd)
 2018-04-20 00:06:44 +05:30
Weiwei Yang c3ca10c22a YARN-7734. Fix UT failure TestContainerLogsPage#testContainerLogPageAccess. Contributed by Tao Yang. 
(cherry picked from commit 411993f6e5)
 2018-04-18 19:40:34 -07:00
Shashikant Banerjee a13b3edfd8
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
 2018-04-19 08:36:32 +09:00
Rohith Sharma K S fa007d986b YARN-8145. yarn rmadmin -getGroups doesn't return updated groups for user. Contributed by Sunil G. 
(cherry picked from commit 241d985467)
 2018-04-18 13:40:32 +05:30
Inigo Goiri 8f300c9ce1 YARN-8165. Incorrect queue name logging in AbstractContainerAllocator. Contributed by Weiwei Yan. 
(cherry picked from commit dd5e18c4ae)
 2018-04-17 15:28:13 +05:30
Wangda Tan 6ee62e6b1c YARN-8138. Add unit test to validate queue priority preemption works under node partition. (Zian Chen via wangda) 
Change-Id: Ibebfab98a714c12c2dc643b6d7b9754a7f813632
 2018-04-14 11:04:49 -07:00
Rohith Sharma K S eb1026dffa YARN-8156. Increase the default value of yarn.timeline-service.app-collector.linger-period.ms. Contributed by Charan Hebri. 
(cherry picked from commit 669eb7bdea)
 2018-04-14 10:57:02 +05:30
Wangda Tan 994c7d66e0 YARN-8091. Revisit checkUserAccessToQueue RM REST API. (wangda) 
Change-Id: I5fab3fe229c34e967487b7327c7b3c8ddf7cb795
 2018-04-13 10:19:10 -07:00
Wangda Tan 7031a853f4 YARN-8048. Support auto-spawning of admin configured services during bootstrap of RM (Rohith Sharma K S via wangda) 
Change-Id: I2d8d61ccad55e1118009294d7e17822df3cd0fd5
 2018-04-13 10:18:48 -07:00
Sunil G 21717db6a0 YARN-8028. Support authorizeUserAccessToQueue in RMWebServices. Contributed by Wangda Tan. 2018-04-13 10:18:12 -07:00
Wangda Tan c962371430 YARN-1151. Ability to configure auxiliary services from HDFS-based JAR files. (Xuan Gong via wangda) 
Change-Id: Ied37ff11e507fc86847753ba79486652c8fadfe9
 2018-04-13 10:17:47 -07:00
Weiwei Yang 84531ad9b6 YARN-8153. Guaranteed containers always stay in SCHEDULED on NM after restart. Contributed by Yang Wang. 2018-04-13 13:29:43 +08:00
Eric E Payne 9693d8b99d YARN-8147. TestClientRMService#testGetApplications sporadically fails. Contributed by Jason Lowe 
(cherry picked from commit 18844599ae)
 2018-04-12 17:58:41 +00:00
Eric E Payne f955d52f39 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 
(cherry picked from commit 6bb128dfb8)
 2018-04-12 16:11:57 +00:00
Billie Rinaldi 11f1d49823 YARN-7221. Add security check for privileged docker container. Contributed by Eric Yang 
(cherry picked from commit 933477e9e0)
 2018-04-11 12:15:00 -07:00
Weiwei Yang c2036af196 YARN-8127. Resource leak when async scheduling is enabled. Contributed by Tao Yang. 
(cherry picked from commit 7eb783e263)
 2018-04-11 17:21:47 +08:00
Vrushali C 67cd7acf99 YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 
(cherry picked from commit 345e7624d5)
 2018-04-11 09:41:04 +05:30
Wangda Tan f5dc7d1edf YARN-8116. Nodemanager fails with NumberFormatException: For input string: . (Chandni Singh via wangda) 
Change-Id: Idd30cfca59982d3fc6e47aa1b88f844a78fae94d
(cherry picked from commit 2bf9cc2c73)
 2018-04-10 17:38:31 -07:00
Wangda Tan 25bf1d3ac6 Updated branch-3.1 to track 3.1.1 
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
 2018-04-09 10:58:50 -07:00
Sunil G 8311fcc75b YARN-7574. Add support for Node Labels on Auto Created Leaf Queue Template. Contributed by Suma Shivaprasad. 
(cherry picked from commit 821b0de4c5)
 2018-04-09 21:17:56 +05:30
bibinchundatt 091db4d0eb YARN-7905. Parent directory permission incorrect during public localization. Contributed by Bilwa S T. 
(cherry picked from commit eb47c3de74)
 2018-04-08 14:49:36 +05:30
Subru Krishnan b9bad94596 YARN-8110. AMRMProxy recover should catch for all throwable to avoid premature exit. (Botong Huang via Subru). 
(cherry picked from commit 00905efab2)
 2018-04-06 16:35:29 -07:00
Haibo Chen b232dcab33 YARN-8107. Give an informative message when incorrect format is used in ATSv2 filter attributes. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 024d7c0870)
 2018-04-06 09:39:45 -07:00
Haibo Chen 6658018410 YARN-6936. [Atsv2] Retrospect storing entities into sub application table from client perspective. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit f8b8bd53c4)
 2018-04-05 11:31:17 -07:00
Konstantinos Karanasos fa464c12ad YARN-8013. Support application tags when defining application namespaces for placement constraints. Contributed by Weiwei Yang. 
(cherry picked from commit 7853ec8d2f)
 2018-04-04 10:53:33 -07:00
Wangda Tan 78832eca8a YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. (Weiwei Yang via wangda) 
Change-Id: I63b4e4192a95bf7ded98c54e46a2871c72869700
 2018-04-04 09:42:49 +08:00
Sunil G 73cd67d7e3 YARN-8085. ResourceProfilesManager should be set in RMActiveServiceContext. Contributed by Tao Yang. 
(cherry picked from commit 7a59d60e0c)
 2018-03-29 21:41:52 +05:30
Subru Krishnan a0091ec4b3 YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru). 
(cherry picked from commit 09999d7e01)
 2018-03-28 11:35:45 -07:00
Subru Krishnan ed49f59e18 Revert "YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru)." 
This reverts commit 85eebf1beb.
 2018-03-28 11:35:30 -07:00
Wangda Tan a2b4daab55 YARN-6629. NPE occurred when container allocation proposal is applied but its resource requests are removed before. (Tao Yang via wangda) 
Change-Id: I805880f90b3f6798ec96ed8e8e75755f390a9ad5
(cherry picked from commit 47f711eebc)
 2018-03-28 11:22:26 -07:00
Subru Krishnan 85eebf1beb YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru). 
(cherry picked from commit 2a2ef15caf)
 2018-03-27 17:44:00 -07:00
Wangda Tan dc99611878 YARN-8072. RM log is getting flooded with MemoryPlacementConstraintManager info logs. (Zian Chen via wangda) 
Change-Id: I2567f4669ad868e13c4301fe44c4bde85bb92c37
(cherry picked from commit 24bc5e0e4d)
 2018-03-26 11:27:40 -07:00
Wangda Tan 62b9537332 YARN-8068. Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ (Sunil G via wangda) 
Change-Id: I7910bd1064a1b4dbbe2084080c060822ea6f3b48
(cherry picked from commit 9eef19b2ad)
 2018-03-26 11:27:32 -07:00
Wangda Tan 54da6a0d78 YARN-8062. yarn rmadmin -getGroups returns group from which the user has been removed. (Sunil G via wangda) 
Change-Id: I80ed63846502bf7751b890b6c6c6a7c0679e2b4a
(cherry picked from commit 5d381570f8)
 2018-03-26 11:27:21 -07:00
Jason Lowe 5d4b2c31a1 YARN-8054. Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread. Contributed by Jonathan Eagles 
(cherry picked from commit 5aa7052e31)
 2018-03-21 17:20:36 -05:00
Rohith Sharma K S bbfe36d686 YARN-7581. HBase filters are not constructed correctly in ATSv2. Contributed by Habio Chen. 
(cherry picked from commit 29acea5000)
 2018-03-21 08:26:56 +05:30
Haibo Chen d8764f1c90 YARN-8053. Add hadoop-distcp in exclusion in hbase-server dependencies for timelineservice-hbase packages. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 3ff6977d3e)
 2018-03-21 08:26:56 +05:30
Jason Lowe a4f0dbcb44 YARN-8027. Setting hostname of docker container breaks for --net=host in docker 1.13. Contributed by Jim Brennan 
(cherry picked from commit f480367af6)
 2018-03-19 17:44:06 -05:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 0bd99621c5 Preparing for 3.1.0 development 2018-03-19 15:16:54 -07:00
Yufei Gu 49c747ab18 YARN-8039. Clean up log dir configuration in TestLinuxContainerExecutorWithMocks.testStartLocalizer. Contributed by Miklos Szegedi. 2018-03-16 16:25:33 -07:00
Weiwei Yang 154cfb2b62 YARN-7636. Re-reservation count may overflow when cluster resource exhausted for a long time. contributed by Tao Yang. 2018-03-16 18:57:31 +08:00
Wangda Tan 4bf622043f YARN-7952. RM should be able to recover log aggregation status after restart/fail-over. (Xuan Gong via wangda) 
Change-Id: I725c9afe64831eda0aa6b0bebdbc79d2dd165707
 2018-03-15 13:26:45 -07:00
fang zhenyi 5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-03-15 16:12:07 +09:00
Eric Yang a82be7754d YARN-7999. Added file cache initializer for Linux container-executor. 
Contributed by Jason Lowe
 2018-03-14 12:21:57 -04:00
Rohith Sharma K S e6de10d0a6 YARN-8022. ResourceManager UI cluster/app/<app-id> page fails to render. Contributed by Tarun Parimi. 2018-03-14 09:42:20 +05:30
Wangda Tan b167d60763 YARN-7657. Queue Mapping could provide options to provide 'user' specific auto-created queues under a specified group parent queue. (Suma Shivaprasad via wangda) 
Change-Id: I32d566e8727840e43c0d66e39a77edef017e3a83
 2018-03-13 17:55:51 -07:00
Wangda Tan a5b27b3c67 YARN-5015. Support sliding window retry capability for container restart. (Chandni Singh via wangda) 
Change-Id: I07addd3e4ba8d98456ee2ff1d5c540a38fe61dea
 2018-03-13 17:55:17 -07:00
Miklos Szegedi a82d4a2e3a YARN-5764. NUMA awareness support for launching containers. Contributed by Devaraj K. 2018-03-13 12:36:57 -07:00
Yufei Gu ff31d8aefa YARN-8024. LOG in class MaxRunningAppsEnforcer is initialized with a faulty class. Contributed by Sen Zhao. 2018-03-12 16:35:26 -07:00
Wangda Tan ea18e70a74 Updated timeline reader to use AuthenticationFilter 
Change-Id: I961771589180c1eb377d36c37a79aa23754effbf
(cherry picked from commit 837338788eb903d0e8bbb1230694782a707891be)
 2018-03-09 22:51:08 -08:00
Owen O'Malley 3a8dade9b1 Revert "HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu." 
This reverts commit 172b23af33.

(cherry picked from commit d0d2d4c51e9534e08893ae14cf3fff7b2ee70b1d)
 2018-03-09 22:46:30 -08:00
Weiwei Yang b451889e8e YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Tao Yang. 2018-03-08 18:13:36 +08:00
Wangda Tan e718ac597f Revert "YARN-7891. LogAggregationIndexedFileController should support read from HAR file. (Xuan Gong via wangda)" 
This reverts commit 4d53ef7eef.
 2018-03-07 15:46:47 -08:00
Wangda Tan 037d783483 YARN-7626. Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount. (Zian Chen via wangda) 
Change-Id: If461277d4557922ab7e4dce9dd8dc5d0d5f22710
(cherry picked from commit 88f9138e12d2d5a1bd13f0915acef93037c1d086)
 2018-03-07 11:30:15 -08:00
Wangda Tan 4d53ef7eef YARN-7891. LogAggregationIndexedFileController should support read from HAR file. (Xuan Gong via wangda) 
Change-Id: Ie16e34039d57df50128c73b37516ad0bc7c9590e
 2018-03-07 11:30:06 -08:00
Jason Lowe d69b31f7f7 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-03-07 09:20:34 -06:00
Rohith Sharma K S 55ba49dd07 YARN-7346. Add a profile to allow optional compilation for ATSv2 with HBase-2.0. Contributed by Haibo Chen and Rohith. 2018-03-06 11:55:21 +05:30
Arun Suresh 1054b48c27 YARN-7972. Support inter-app placement constraints for allocation tags by application ID. (Weiwei Yang via asuresh) 2018-03-05 11:24:17 -08:00
Billie Rinaldi 628be58a4c YARN-7915. Trusted image log message repeated multiple times. Contributed by Shane Kumpf 2018-03-05 09:08:44 -08:00
Jason Lowe 83798f15f8 YARN-7511. NPE in ContainerLocalizer when localization failed for running container. Contributed by Tao Yang 2018-03-02 09:59:46 -06:00
Ray Chiang 923e1773e1 HADOOP-14652. Update metrics-core version to 3.2.4. (rchiang) 2018-03-01 15:36:10 -08:00
Sunil G 22928c0c60 YARN-7637. GPU volume creation command fails when work preserving is disabled at NM. Contributed by Zian Chen. 2018-03-01 22:39:32 +05:30
Yufei Gu 6ca4e3017b YARN-5028. RMStateStore should trim down app state for completed applications(addendum). Contributed by Rohith Sharma K S. 2018-03-01 08:38:53 -08:00
Haibo Chen 10663b78c8 Revert "yarn-7346.07.patch" 
This reverts commit 5e37ca5bb4.
 2018-02-28 21:11:36 -08:00
Haibo Chen d1274c3b71 YARN-7835. Race condition in NM while publishing events if second attempt is launched on the same node. (Rohith Sharma K S via Haibo Chen) 2018-02-28 21:10:42 -08:00
Haibo Chen 5e37ca5bb4 yarn-7346.07.patch 2018-02-28 21:04:54 -08:00
Billie Rinaldi 727c033997 YARN-7446. Remove --user flag when running privileged mode docker container. Contributed by Eric Yang 2018-02-27 14:33:57 -08:00
Arpit Agarwal 2fa7963c3d HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi. 2018-02-24 14:41:55 -08:00
Jason Lowe 8e728f39c9 YARN-5714. ContainerExecutor does not order environment map. Contributed by Remi Catherinot and Jim Brennan 2018-02-23 15:46:35 -06:00
Carlo Curino 514794e1a5 YARN-7934. [GQ] Refactor preemption calculators to allow overriding for Federation Global Algos. (Contributed by curino) 2018-02-22 18:12:12 -08:00
Jason Lowe 3132709b46 YARN-7916. Remove call to docker logs on failure in container-executor. Contributed by Shane Kumpf 2018-02-21 16:54:02 -06:00
Yufei Gu 92cbbfe79e YARN-5028. RMStateStore should trim down app state for completed applications. Contributed by Gergo Repas. 2018-02-21 11:42:51 -08:00
Sunil G bdd2a184d7 YARN-7947. Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps. Contributed by Eric Payne. 2018-02-21 14:35:57 +05:30
Jason Lowe 9497215084 YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:06:28 -06:00
Jason Lowe b9a429bb28 Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 8013475d44.
 2018-02-19 08:16:25 -06:00
fang zhenyi 4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-02-18 22:19:39 +09:00
Rohith Sharma K S 9af30d46c6 YARN-7919. Refactor timelineservice-hbase module into submodules. Contributed by Haibo Chen. 2018-02-17 20:30:28 +05:30
Arun Suresh a1e56a6286 YARN-7918. Fix TestAMRMClientPlacementConstraints. (Gergely Novák via asuresh) 2018-02-17 03:24:55 -08:00
Sunil G aae629913c YARN-7292. Retrospect Resource Profile Behavior for overriding capability. Contributed by Wangda Tan. 2018-02-16 12:37:40 +05:30
Jason Lowe 8013475d44 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-02-15 17:09:00 -06:00
Konstantinos Karanasos 0b489e564c YARN-7920. Simplify configuration for PlacementConstraints. Contributed by Wangda Tan. 2018-02-15 14:23:38 -08:00
Eric Payne bddfe42e2c Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit c5e6e3de1c.
 2018-02-14 14:29:20 -06:00
Wangda Tan 042ef2fa7b YARN-7789. Should fail RM if 3rd resource type is configured but RM uses DefaultResourceCalculator. (Zian Chen via wangda) 
Change-Id: I1f6a451fe16758def6f47c046a9b8a67ad7255af
 2018-02-14 23:11:10 +08:00
Eric Payne c5e6e3de1c YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-13 10:11:02 -06:00
Jason Lowe 5a1db60ab1 YARN-7914. Fix exit code handling for short lived Docker containers. Contributed by Shane Kumpf 2018-02-12 15:50:10 -06:00
Jason Lowe 3414fd1243 YARN-7917. Fix failing test TestDockerContainerRuntime#testLaunchContainerWithDockerTokens. Contributed by Shane Kumpf 2018-02-12 09:27:43 -06:00
Wangda Tan d02e42cee4 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
 2018-02-12 10:29:37 +08:00
Wangda Tan 60f9e60b3b Preparing for 3.2.0 development 
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
 2018-02-11 11:17:38 +08:00
Yufei Gu 1bc03ddf97 YARN-7655. Avoid AM preemption caused by RRs for specific nodes or racks. Contributed by Steven Rand. 2018-02-08 12:32:43 -08:00
Jian He eb2449d539 YARN-5428. Allow for specifying the docker client configuration directory. Contributed by Shane Kumpf 2018-02-08 11:35:30 -08:00
Jason Lowe 456705a07c YARN-7815. Make the YARN mounts added to Docker containers more restrictive. Contributed by Shane Kumpf 2018-02-07 13:09:08 -06:00
Billie Rinaldi 266da25c04 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit e14f03dfbf.
 2018-02-07 07:17:00 -08:00
Billie Rinaldi aa461f9091 YARN-7516. Add security check for trusted docker images. Contributed by Eric Yang 2018-02-07 07:09:45 -08:00
Robert Kanter 814d701d46 YARN-7841. Cleanup AllocationFileLoaderService's reloadAllocations method (snemeth via rkanter) 2018-02-06 14:36:49 -08:00
Anu Engineer 4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Weiwei Yang bff858e910 YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 2018-02-06 19:33:04 +08:00
Billie Rinaldi a196ee9362 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 2018-02-05 18:04:09 -08:00
Jason Lowe 60656bceca YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error. Contributed by Botong Huang 2018-02-05 17:16:04 -06:00
Konstantinos Karanasos 50723889cc YARN-7778. Merging of placement constraints defined at different levels. Contributed by Weiwei Yang. 2018-02-02 14:46:20 -08:00
Arun Suresh 6e5ba9366f YARN-7839. Modify PlacementAlgorithm to Check node capacity before placing request on node. (Panagiotis Garefalakis via asuresh) 2018-02-02 10:28:22 -08:00
Jason Lowe 682ea21f2b Revert "YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan" 
This reverts commit 12eaae383a.
 2018-02-01 12:14:09 -06:00
Jason Lowe 12eaae383a YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan 2018-01-31 11:37:54 -06:00
Arun Suresh d481344783 YARN-7822. Constraint satisfaction checker support for composite OR and AND constraints. (Weiwei Yang via asuresh) 2018-01-31 01:51:08 -08:00
Arun Suresh add993e26a YARN-6597. Add RMContainer recovery test to verify tag population in the AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh f8c5f5b237 YARN-7784. Fix Cluster metrics when placement processor is enabled. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh c23980c4f2 YARN-7795. Fix jenkins issues of YARN-6592 branch. (Sunil G via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 644afe5fd8 YARN-7807. Assume intra-app anti-affinity as default for scheduling request inside AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh a4c539fcdb YARN-7783. Add validation step to ensure constraints are not violated due to order in which a request is processed. (asuresh) 2018-01-31 01:30:17 -08:00
Sunil G 9b81cb0537 YARN-7779. Display allocation tags in RM web UI and expose same through REST API. Contributed by Weiwei Yang. 2018-01-31 01:30:17 -08:00
Konstantinos Karanasos adbe87abf8 YARN-7788. Factor out management of temp tags from AllocationTagsManager. (Arun Suresh via kkaranasos) 2018-01-31 01:30:17 -08:00
Arun Suresh 8bf7c44436 YARN-7763. Allow Constraints specified in the SchedulingRequest to override application level constraints. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 28fe7f3318 YARN-7774. Miscellaneous fixes to the PlacementProcessor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 38af237969 YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 8779a35742 YARN-7709. Remove SELF from TargetExpression type. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Wangda Tan 29d9e4d581 YARN-6619. AMRMClient Changes to use the PlacementConstraint and SchcedulingRequest objects. (Arun Suresh via wangda) 2018-01-31 01:30:17 -08:00
Arun Suresh a5c1fc881e YARN-7696. Add container tags to ContainerTokenIdentifier, api.Container and NMContainerStatus to handle all recovery cases. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 4eda58c136 YARN-7681. Double-check placement constraints in scheduling phase before actual allocation is made. (Weiwei Yang via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh bdba01f73b YARN-7682. Expose canSatisfyConstraints utility function to validate a placement against a constraint. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh a52d11fb8c YARN-7613. Implement Basic algorithm for constraint based placement. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh f9af15d659 YARN-7612. Add Processor Framework for Rich Placement Constraints. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 1efb2b6f25 YARN-6596. Introduce Placement Constraint Manager module. (Konstantinos Karanasos via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 37f1a7b64f YARN-7653. Node group support for AllocationTagsManager. (Panagiotis Garefalakis via asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 06eb63e64b YARN-7669. API and interface modifications for placement constraint processor. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 88d8d3f40b YARN-7670. Modifications to the ResourceScheduler API to support SchedulingRequests. (asuresh) 2018-01-31 01:30:17 -08:00
Arun Suresh 801c0988b5 YARN-7522. Introduce AllocationTagsManager to associate allocation tags to nodes. (Wangda Tan via asuresh) 2018-01-31 01:30:17 -08:00
Sunil G 6463e10c72 YARN-7723. Avoid using docker volume --format option to run against to older docker releases. Contributed by Wangda Tan 2018-01-30 15:58:11 +05:30
Sunil G e9c72d04be YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 2018-01-29 20:44:38 +05:30
Jinjiang Ling 3400d0c535
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-01-30 00:00:57 +09:00
Jian He 443523f9c0 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-27 22:18:51 -08:00
Haibo Chen 649ef7ac33 YARN-7064. Use cgroup to get container resource utilization. (Miklos Szegedi via Haibo Chen) 2018-01-26 16:27:31 -08:00
Billie Rinaldi f2fa736f0a YARN-7797. Docker host network can not obtain IP address for RegistryDNS. Contributed by Eric Yang 2018-01-26 09:32:23 -08:00
Wangda Tan 75618c0dec YARN-7760. [UI2] Clicking 'Master Node' or link next to 'AM Node Web UI' under application's appAttempt page goes to OLD RM UI. (Vasudevan Skm / Sunil G via wangda) 
Change-Id: Ia7e2b955d3551ca43c6852fe253fd01983b56565
 2018-01-26 15:44:55 +08:00
Wangda Tan e0cfb0a31a YARN-7817. Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. (Sunil G via wangda) 
Change-Id: Ia7ceeabd82046645ddeaf487c763288f36cfbdee
 2018-01-26 15:43:27 +08:00
Miklos Szegedi 2e5865606b YARN-7814. Remove automatic mounting of the cgroups root directory into Docker containers. Contributed by Shane Kumpf. 2018-01-25 17:37:23 -08:00
Jason Lowe ff8378eb1b YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:47:19 -06:00
Sunil G 82cc6f6968 YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 2018-01-25 14:32:54 +05:30
Eric Yang 59828be197 YARN-7810. Update TestDockerContainerRuntime to test with current user credential. 
(contributed by Shane Kumpf)
 2018-01-24 21:24:19 -05:00
Zhe Zhang fa8cf4d1b4 YARN-7737. prelaunch.err file not found exception on container failure. Contributed by Keqiu Hu. 2018-01-24 12:54:54 -08:00
Jian He 55c32776b1 YARN-7801. AmFilterInitializer should addFilter after fill all parameters. Contributed by Wangda Tan 2018-01-24 11:17:12 -08:00
Rohith Sharma K S f3e33aeb80 Revert "YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma" 
This reverts commit ec8f47e7fa.
 2018-01-24 15:04:15 +05:30
Miklos Szegedi e7642a3e6f YARN-7796. Container-executor fails with segfault on certain OS configurations. Contributed by Gergo Repas. 2018-01-23 21:07:05 -08:00
Billie Rinaldi e307edcb47 YARN-7540 and YARN-7605. Convert yarn app cli to call yarn api services and implement doAs for Api Service REST API. Contributed by Eric Yang and Jian He 2018-01-23 17:54:39 -08:00
Eric Yang 97fe3cc187 YARN-7729. Add support for setting Docker PID namespace mode. (Contributed by Billie Rinaldi) 2018-01-22 16:33:38 -05:00
Yufei Gu 22ee6f77e5 YARN-7755. Clean up deprecation messages for allocation increments in FS config. Contributed by Wilfred Spiegelenburg. 2018-01-22 10:30:01 -08:00
Vrushali C ec8f47e7fa YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma 2018-01-19 15:34:40 -08:00
Haibo Chen 4aca4ff759 YARN-5094. some YARN container events have timestamp of -1. 2018-01-19 10:27:46 -08:00
Miklos Szegedi bc93ac229e YARN-7139. FairScheduler: finished applications are always restored to default queue. Contributed by Wilfred Spiegelenburg. 2018-01-18 17:43:47 -08:00
Billie Rinaldi 37f4696a9c YARN-7740. Fix logging for destroy yarn service cli when app does not exist and some minor bugs. Contributed by Jian He 2018-01-18 12:11:19 -08:00
Sunil G 06cceba1cb YARN-7738. CapacityScheduler: Support refresh maximum allocation for multiple resource types. Contributed by Wangda Tan. 2018-01-18 19:05:26 +05:30
Miklos Szegedi a68e445dc6 YARN-7717. Add configuration consistency for module.enabled and docker.privileged-containers.enabled. Contributed by Eric Badger. 2018-01-17 14:11:14 -08:00
Miklos Szegedi 41049ba5d1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-16 15:40:43 -08:00
Yufei Gu 370f1c6283 YARN-6486. FairScheduler: Deprecate continuous scheduling. (Contributed by Wilfred Spiegelenburg) 2018-01-16 14:46:55 -08:00
Robert Kanter 5ac109909a YARN-7479. TestContainerManagerSecurity.testContainerManager[Simple] flaky in trunk (ajisakaa via rkanter) 2018-01-16 10:16:42 -08:00
Rohith Sharma K S d09058b2fd YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 2018-01-16 07:58:29 +05:30
Miklos Szegedi 2dcfc1876e YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 2018-01-12 17:00:01 -08:00
Wangda Tan edcc3a95d5 YARN-7468. Provide means for container network policy control. (Xuan Gong via wangda) 
Change-Id: I73678c343f663412917758feef35d8308c216e76
 2018-01-12 16:14:10 -08:00
Eric Yang 3d65dbe032 YARN-5366. Improve signal handling and delete delay for Docker on Yarn. 
(Contributed by Shane Kumpf)
 2018-01-12 13:41:27 -05:00
Sunil G 128d773a23 YARN-7727. Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector. Contributed by Prabhu Joseph. 2018-01-12 15:41:19 +05:30
Miklos Szegedi bc285da107 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 2018-01-11 09:40:09 -08:00
Sunil G afd8caba27 YARN-7722. Rename variables in MockNM, MockRM for better clarity. Contributed by Lovekesh bansal 2018-01-10 20:51:34 +05:30
Miklos Szegedi 55066cc53d YARN-7689. TestRMContainerAllocator fails after YARN-6124. Contributed by Wilfred Spiegelenburg. 2018-01-09 16:59:39 -08:00
Haibo Chen 8ee7080e5d YARN-7665. Allow FS scheduler state dump to be turned on/off separately from FS debug log. (Wilfred Spiegelenburg via Haibo Chen) 2018-01-09 16:45:35 -08:00
Jason Lowe 90c86d6e09 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 2018-01-09 10:51:15 -06:00
Jason Lowe ebff4decfe YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 2018-01-09 09:52:59 -06:00
bibinchundatt b26e30ab16 YARN-7508. NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode. Contributed by Tao Yang. 2018-01-09 17:57:01 +05:30
Ray Chiang 59ab5da0a0 YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang) 2018-01-08 16:19:02 -08:00
Rohith Sharma K S c2d6fa3656 YARN-7699. queueUsagePercentage is coming as INF for getApp REST api call. Contributed by Sunil G. 2018-01-08 10:39:31 +05:30
Wangda Tan a81144daa0 YARN-7666. Introduce scheduler specific environment variable support in ApplicationSubmissionContext for better scheduling placement configurations. (Sunil G via wangda) 
Change-Id: I0fd826490f5160d47d42af2a9ac0bd8ec4e959dc
 2018-01-05 15:12:04 -08:00
Robert Kanter 2aa4f0a559 YARN-7645. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers is flakey with FairScheduler (rkanter) 2018-01-05 13:55:09 -08:00
Robert Kanter f8e7dd9b10 YARN-7557. It should be possible to specify resource types in the fair scheduler increment value (grepas via rkanter) 2018-01-05 11:15:06 -08:00
Sunil G 0c75d0634b YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:42:17 +05:30
Jason Lowe d795661868 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:15:52 -06:00
Robert Kanter 7a55044803 YARN-7622. Allow fair-scheduler configuration on HDFS (gphillips via rkanter) 2018-01-03 15:31:50 -08:00
Haibo Chen 2f6c038be6 YARN-7602. NM should reference the singleton JvmMetrics instance. 2018-01-03 09:41:26 -08:00
Rohith Sharma K S c9bf813c9a YARN-7692. Skip validating priority acls while recovering applications. Contributed by Sunil G. 2018-01-03 18:20:04 +05:30
Arun Suresh c0c7cce81d YARN-7691. Add Unit Tests for ContainersLauncher. (Sampada Dehankar via asuresh) 2018-01-02 22:03:00 -08:00
Miklos Szegedi 7f515f57ed YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 2018-01-02 18:03:04 -08:00
Miklos Szegedi b82049b4f0 YARN-7580. ContainersMonitorImpl logged message lacks detail when exceeding memory limits. Contributed by Wilfred Spiegelenburg. 2017-12-29 12:49:37 -08:00
Arun Suresh a55884c68e YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 2017-12-28 22:20:42 -08:00
Robert Kanter 382215c72b YARN-7577. Unit Fail: TestAMRestart#testPreemptedAMRestartOnRMRestart (miklos.szegedi@cloudera.com via rkanter) 2017-12-20 13:39:00 -08:00
Sunil G d62932c3b2 YARN-7032. [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Contributed by Rohith Sharma K S. 2017-12-20 11:31:15 +05:30
Eric Yang 94a2ac6b71 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 18:42:27 -05:00
Varun Saxena c0aeb666a4 YARN-7662. [ATSv2] Define new set of configurations for reader and collectors to bind (Rohith Sharma K S via Varun Saxena) 2017-12-19 22:29:24 +05:30
Jason Lowe 811fabdebe YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 2017-12-18 15:20:06 -06:00
Akira Ajisaka 001008958d
YARN-7664. Several javadoc errors. Contributed by Sean Mackrory. 2017-12-18 22:24:51 +09:00
Wangda Tan 44825f0960 YARN-7629. TestContainerLaunch# fails after YARN-7381. (Jason Lowe via wangda) 
Change-Id: Ia6a3f05c9a7e797d8190123d304ecc4e2b018e33
 2017-12-15 15:40:56 -08:00
Wangda Tan 631b5c2db7 YARN-5418. When partial log aggregation is enabled, display the list of aggregated files on the container log page. (Xuan Gong via wangda) 
Change-Id: I1befb0bbaeb89fb315bafe3e2f3379663f8cf1ec
 2017-12-15 15:38:36 -08:00
Rohith Sharma K S 09d996fdd4 YARN-7190. Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath. Contributed by Varun Saxena. 2017-12-15 21:50:28 +05:30
Sunil G 890d3d0645 YARN-7638. Unit tests related to preemption for auto created leaf queues feature.Contributed by Suma Shivaprasad. 2017-12-15 13:00:57 +05:30
Subru Krishnan 17ba74be29 YARN-7630. Fix AMRMToken rollover handling in AMRMProxy. Contributed by Botong Huang. 2017-12-14 14:03:55 -08:00
Chen Liang 46e18c8da7 HADOOP-14914. Change to a safely casting long to int. Contributed by Ajay Kumar. 2017-12-13 14:56:14 -08:00
Sunil G cb87e4dc92 YARN-7643. Handle recovery of applications in case of auto-created leaf queue mapping. Contributed by Suma Shivaprasad. 2017-12-13 22:49:58 +05:30
Weiwei Yang 7efc4f7688 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 2017-12-13 13:11:41 +08:00
Jason Lowe 2abab1d7c5 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-12 16:04:15 -06:00
Jason Lowe 06f0eb2dce YARN-7625. Expose NM node/containers resource utilization in JVM metrics. Contributed by Weiwei Yang 2017-12-12 12:56:26 -06:00
Sunil G 8bb83a8f62 Queue ACL validations should validate parent queue ACLs before auto-creating leaf queues. Contributed by Suma Shivaprasad. 2017-12-12 15:20:59 +05:30
Sunil G 5c87fb2f62 YARN-7635. TestRMWebServicesSchedulerActivities fails in trunk. Contributed by Sunil G. 2017-12-12 15:08:18 +05:30
Sunil G 312ceebde8 YARN-7632. Effective min and max resource need to be set for auto created leaf queues upon creation and capacity management. Contributed by Suma Shivaprasad. 2017-12-11 19:20:02 +05:30
Weiwei Yang a2edc4cbf5 YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 10:31:46 +08:00