Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,072 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4826 Commits

Author SHA1 Message Date
Shane Kumpf eea10c7b20 YARN-8623. Update Docker examples to use image which exists. Contributed by Craig Condit 
(cherry picked from commit 26c94a0fd0)
 2018-09-25 06:42:20 -06:00
Rohith Sharma K S 47306cc2db YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt. 
(cherry picked from commit 50bc7746d7)
 2018-09-25 11:40:09 +05:30
Sunil G 0196ffaa22 YARN-8742. [UI2] Container logs on Application / Service pages on UI2 are not available many case, improve error messages in such cases. Contributed by Akhil PB. 
(cherry picked from commit 32a35dc94b)
 2018-09-24 12:21:15 +05:30
Sunil G 6699b9658e YARN-8628. [UI2] Few duplicated or inconsistent information displayed in UI2. Contributed by Akhil PB. 
(cherry picked from commit a2752779ac)
 2018-09-21 16:06:22 +05:30
Eric Yang e9315f6688 YARN-8801. Fixed header comments for docker utility functions. 
Contributed by Zian Chen
 2018-09-20 13:12:29 -04:00
Jason Lowe 3fb6787295 YARN-8784. DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk. Contributed by Eric Badger 
(cherry picked from commit 6b5838ed32)
 2018-09-19 16:49:21 -05:00
Weiwei Yang aaf0b119e5 YARN-8771. CapacityScheduler fails to unreserve when cluster resource contains empty resource type. Contributed by Tao Yang. 
(cherry picked from commit 0712537e79)
 2018-09-19 19:38:09 +08:00
Jason Lowe 3d77094cf2 YARN-8648. Container cgroups are leaked when using docker. Contributed by Jim Brennan 
(cherry picked from commit 2df0a8dcb3)
 2018-09-18 15:43:10 -05:00
Weiwei Yang 46eeba6234 YARN-8787. Fix broken list items in PlacementConstraints documentation. Contributed by Masahiro Tanaka. 
(cherry picked from commit 78a0d173e4)
 2018-09-18 17:21:21 +08:00
Sunil G 75d2d0fdb1 YARN-8652. [UI2] YARN UI2 breaks if getUserInfo REST API is not available in older versions. Contributed by Akhil PB. 
(cherry picked from commit bbeca0107e)
 2018-09-18 12:45:08 +05:30
Sunil G 8dccc4ef12 YARN-8726. [UI2] YARN UI2 is not accessible when config.env file failed to load. Contributed by Akhil PB. 
(cherry picked from commit 0cc6e03945)
 2018-09-18 12:32:30 +05:30
Weiwei Yang f0e5651cf4 YARN-8782. Fix exception message in Resource.throwExceptionWhenArrayOutOfBound. Contributed by Gergely Pollak. 
(cherry picked from commit 9a265fa673)
 2018-09-17 22:26:09 +08:00
Weiwei Yang 00a469138d YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi. 
(cherry picked from commit f1a893fdbc)
 2018-09-14 16:40:35 +08:00
Jason Lowe 88687213cc YARN-8680. YARN NM: Implement Iterable Abstraction for LocalResourceTracker state. Contributed by Pradeep Ambati 
(cherry picked from commit 250b50018e)
 2018-09-13 14:12:20 -05:00
Weiwei Yang a7b1d1e006 YARN-8729. Node status updater thread could be lost after it is restarted. Contributed by Tao Yang. 
(cherry picked from commit 39c1ea1ed4)
 2018-09-13 23:16:52 +08:00
Sunil G c879ca38de YARN-8630. ATSv2 REST APIs should honor filter-entity-list-by-user in non-secure cluster when ACls are enabled. Contributed by Rohith Sharma K S. 
(cherry picked from commit f4bda5e8e9)
 2018-09-13 17:48:01 +05:30
Eric E Payne b6bc0f409a YARN-8709: CS preemption monitor always fails since one under-served queue was deleted. Contributed by Tao Yang. 
(cherry picked from commit 987d8191ad)
 2018-09-10 20:02:39 +00:00
Eric Yang 0b97dc5869 YARN-8751. Reduce conditions that mark node manager as unhealthy. 
Contributed by Craig Condit

(cherry picked from commit 7d62334387)
 2018-09-07 20:32:11 -04:00
Giovanni Matteo Fumarola 7fed1a07ab HADOOP-15731. TestDistributedShell fails on Windows. Contributed by Botong Huang. 2018-09-07 16:37:03 -07:00
Shane Kumpf 2d68708a1d YARN-8638. Allow linux container runtimes to be pluggable. Contributed by Craig Condit 
(cherry picked from commit dffb7bfe6c)
 2018-09-05 06:55:25 -06:00
bibinchundatt e2e0fc26a2 YARN-8535. Fix DistributedShell unit tests. Contributed by Abhishek Modi. 
(cherry picked from commit eed8415dc1)
 2018-09-02 13:37:38 +05:30
Shane Kumpf b8618556ee YARN-8642. Add support for tmpfs mounts with the Docker runtime. Contributed by Craig Condit 
(cherry picked from commit 73625168c0)
 2018-08-29 07:11:38 -06:00
Weiwei Yang f164568b47 YARN-8723. Fix a typo in CS init error message when resource calculator is not correctly set. Contributed by Abhishek Modi. 
(cherry picked from commit 3fa4639421)
 2018-08-29 11:15:02 +08:00
Billie Rinaldi eefd780918 YARN-8675. Remove default hostname for docker containers when net=host. Contributed by Suma Shivaprasad 
(cherry picked from commit 05b2bbeb35)
 2018-08-27 11:42:09 -07:00
Weiwei Yang 271ed86ac0 YARN-8719. Typo correction for yarn configuration in OpportunisticContainers(federation) docs. Contributed by Y. SREENIVASULU REDDY. 
(cherry picked from commit e8b063f630)
 2018-08-28 01:07:05 +08:00
Haibo Chen e4282c077b YARN-8051. TestRMEmbeddedElector#testCallbackSynchronization is flaky. (Robert Kanter via Haibo Chen) 
(cherry picked from commit 93d47a0ed5)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMEmbeddedElector.java
 2018-08-24 13:24:08 -05:00
Inigo Goiri 246086984e YARN-7773. YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'. Contributed by Yiran Wu. 
(cherry picked from commit 60e5c1b516)
 2018-08-24 07:10:02 +05:30
Jason Lowe 84654451fa YARN-8649. NPE in localizer hearbeat processing if a container is killed while localizing. Contributed by lujie 
(cherry picked from commit 585ebd873a)
 2018-08-23 09:37:43 -05:00
Rohith Sharma K S 63d5214332 YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi. 
(cherry picked from commit d3fef7a5c5)
 2018-08-21 12:11:12 +05:30
Wei-Chiu Chuang 0d155de159 HADOOP-14212. Addendum patch: Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:28 -07:00
Wei-Chiu Chuang 78fb14ba49 HADOOP-14212. Expose SecurityEnabled boolean field in JMX for other services besides NameNode. Contributed by Adam Antal. 2018-08-20 14:49:24 -07:00
Jason Lowe 44c4928b64 YARN-8242. YARN NM: OOM error while reading back the state store on recovery. Contributed by Pradeep Ambati and Kanwaljeet Sachdev 
(cherry picked from commit 65e7469712)
 2018-08-20 10:21:57 -05:00
Rohith Sharma K S a3d4a25bbf YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan. 
(cherry picked from commit 4aacbfff60)
 2018-08-18 11:04:09 +05:30
Rohith Sharma K S 3532bd5c8b YARN-8612. Fix NM Collector Service Port issue in YarnConfiguration. Contributed by Prabha Manepalli. 
(cherry picked from commit 1697a02306)
 2018-08-17 11:13:25 +05:30
Eric Yang 5237bdfb5a YARN-8667. Cleanup symlinks when container restarted by NM. 
Contributed by Chandni Singh

(cherry picked from commit d42806160e)
 2018-08-16 18:44:47 -04:00
Eric Yang 97c1934247 YARN-8474. Fixed ApiServiceClient kerberos negotiation. 
Contributed by Billie Rinaldi

(cherry picked from commit 8990eaf592)
 2018-08-16 12:50:52 -04:00
Jason Lowe 819a2a6f10 YARN-8656. container-executor should not write cgroup tasks files for docker containers. Contributed by Jim Brennan 
(cherry picked from commit cb21eaa026)
 2018-08-16 10:09:56 -05:00
Weiwei Yang 05547b1e0b YARN-8614. Fix few annotation typos in YarnConfiguration. Contributed by Sen Zhao. 
(cherry picked from commit 8dc07b4723)
 2018-08-15 12:38:55 +08:00
Jason Lowe 95cd6de5c6 YARN-8640. Restore previous state in container-executor after failure. Contributed by Jim Brennan 
(cherry picked from commit d1d129aa9d)
 2018-08-14 10:26:21 -05:00
Shane Kumpf ba6b48e19f YARN-8520. Document best practice for user management. Contributed by Eric Yang 
(cherry picked from commit e7951c69cb)
 2018-08-10 14:47:14 -06:00
Weiwei Yang 734bc42289 YARN-8575. Avoid committing allocation proposal to unavailable nodes in async scheduling. Contributed by Tao Yang. 
(cherry picked from commit 0a71bf1452)
 2018-08-10 15:10:27 +08:00
Weiwei Yang 991514f7c3 YARN-8521. NPE in AllocationTagsManager when a container is removed more than once. Contributed by Weiwei Yang. 
(cherry picked from commit 08d5060605)
 2018-08-10 08:44:53 +08:00
Wangda Tan beea18736b YARN-8136. Add version attribute to site doc examples and quickstart. (Eric Yang via wangda) 
Change-Id: I4541b239f490ca0a6edf9698e0d3deaf83669151
(cherry picked from commit 8244abb7ae)
 2018-08-09 12:04:33 -07:00
Wangda Tan 68279fcd65 YARN-8588. Logging improvements for better debuggability. (Suma Shivaprasad via wangda) 
Change-Id: I66aa4b0ec031ae5ce0fae558e2f8cbcbbfebc442
(cherry picked from commit 344c335a92)
 2018-08-09 12:04:25 -07:00
Weiwei Yang 0ee7e80047 YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang. 
(cherry picked from commit d352f167eb)
 2018-08-10 00:43:53 +08:00
Jason Lowe 3dd299a770 YARN-8331. Race condition in NM container launched after done. Contributed by Pradeep Ambati 
(cherry picked from commit cd04e954d2)
 2018-08-09 10:23:02 -05:00
Sunil G c6013b060d YARN-8633. Update DataTables version in yarn-common in line with JQuery 3 upgrade. Contributed by Akhil PB. 
(cherry picked from commit 00013d6ef7)
 2018-08-09 12:35:21 +05:30
Bharat Viswanadham 5430aab133 YARN-8426:Upgrade jquery-ui to 1.12.1 in YARN. Contributed by Sunil Govindan 
(cherry picked from commit 361ffb26be)
 2018-08-09 12:27:50 +05:30
Robert Kanter 0ed91309d3 YARN-8568. Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md (bsteinbach via rkanter) 
(cherry picked from commit 8478732bb2)
 2018-08-08 15:10:52 -07:00
Wangda Tan f16c311486 Make 3.1.1 awared by other branches - adding missing files 
(cherry picked from commit 3214cd75ac)
 2018-08-08 13:06:14 -07:00
Wangda Tan 781620a44d Make 3.1.1 awared by other branches 
(cherry picked from commit 49c687608b)
 2018-08-08 13:04:05 -07:00
bibinchundatt b4f957e655 YARN-8601. Print ExecutionType in Container report CLI. Contributed by Bilwa S T. 
(cherry picked from commit ff06bd1be8)
 2018-08-08 22:46:16 +05:30
Wangda Tan f2768eaa38 YARN-8407. Container launch exception in AM log should be printed in ERROR level. (Yesha Vora via wangda) 
Change-Id: I154e873df1df3503a09c41d6b3874ca195af91d9
(cherry picked from commit 861095f761)
 2018-08-07 13:07:28 -07:00
Wangda Tan 450c791ecf YARN-8629. Container cleanup fails while trying to delete Cgroups. (Suma Shivaprasad via wangda) 
Change-Id: I392ef4f8baa84d5d7b1f2e438c560b5426b6d4f2
(cherry picked from commit d4258fcad7)
 2018-08-07 12:41:55 -07:00
Eric Yang 947dc34229 YARN-8624. Updated verbiage around entry point support. 
Contributed by Craig Condit
 2018-08-06 12:39:22 -04:00
Shane Kumpf 7773904974 YARN-8287. Update documentation and yarn-default related to the Docker runtime. Contributed by Craig Condit 
(cherry picked from commit bcfc9851f4)
 2018-08-03 18:23:37 -06:00
Sunil G d8fdcf8b42 YARN-8615. [UI2] Resource Usage tab shows only memory related info. No info available for vcores/gpu. Contributed by Akhil PB. 
(cherry picked from commit 3426f4062a)
 2018-08-03 19:36:19 +05:30
Sunil G 1bfce639e8 YARN-8620. [UI2] YARN Services UI new submission failures are not debuggable. Contributed by Akhil PB. 
(cherry picked from commit 29417dbbf4)
 2018-08-03 19:19:17 +05:30
Rohith Sharma K S 4816d10ddb YARN-8608. [UI2] No information available per application appAttempt about 'Total Outstanding Resource Requests'. Contributed by Akhil PB. 
(cherry picked from commit 022592ae79)
 2018-08-03 15:33:36 +05:30
Rohith Sharma K S 83ccb75b91 YARN-8603. [UI2] Latest run application should be listed first in the RM UI. Contributed by Akhil PB. 
(cherry picked from commit 33482d35e8)
 2018-08-03 15:33:24 +05:30
Jason Lowe 619019ccca YARN-8263. DockerClient still touches hadoop.tmp.dir. Contributed by Craig Condit 
(cherry picked from commit 7526815e32)
 2018-08-02 10:45:52 -05:00
Sunil G d76ab31961 YARN-8318. [UI2] IP address in component page shows N/A. Contributed by Yesha Vora. 
(cherry picked from commit 5033d7da8f)
 2018-08-02 20:10:08 +05:30
Sunil G b1fcbe7c2d YARN-8592. [UI2] rmip:port/ui2 endpoint shows a blank page in windows OS and Chrome browser. Contributed by Akhil PB. 
(cherry picked from commit 97870ec1f6)
 2018-08-02 16:12:12 +05:30
Sunil G 35b0686f24 YARN-8594. [UI2] Display current logged in user. Contributed by Akhil PB. 
(cherry picked from commit 1ea81169ba)
 2018-08-02 12:41:59 +05:30
Sunil G 1f77b20f08 YARN-8593. Add RM web service endpoint to get user information. Contributed by Akhil PB. 
(cherry picked from commit 735b492556)
 2018-08-02 08:35:54 +05:30
Eric Yang 680dac26d8 YARN-8610. Fixed initiate upgrade error message. 
Contributed by Chandni Singh

(cherry picked from commit 23f394240e)
 2018-08-01 20:43:22 -04:00
Shane Kumpf 62cc373dc5 YARN-8600. RegistryDNS hang when remote lookup does not reply. Contributed by Eric Yang 
(cherry picked from commit 603a57476c)
 2018-08-01 12:31:28 -06:00
Billie Rinaldi 2a94823f32 YARN-8403. Change the log level for fail to download resource from INFO to ERROR. Contributed by Eric Yang 
(cherry picked from commit 67c65da261)
 2018-08-01 08:58:15 -07:00
Sunil G a6525e0744 YARN-8595. [UI2] Container diagnostic information is missing from container page. Contributed by Akhil PB. 
(cherry picked from commit d920b9db77)
 2018-08-01 14:28:35 +05:30
Sunil G ff35f0c308 YARN-8606. Opportunistic scheduling does not work post RM failover. Contributed by Bibin A Chundatt. 
(cherry picked from commit a48a0cc7fd)
 2018-08-01 12:17:53 +05:30
Wangda Tan f4a3b26f9a YARN-8522. Application fails with InvalidResourceRequestException. (Zian Chen via wangda) 
Change-Id: I34dd7fa49bd4d10580c4a78051033b1068d28f1e
(cherry picked from commit 5cc8e99147)
 2018-07-31 22:35:16 -07:00
Sunil G cbfd7358d2 YARN-8397. Potential thread leak in ActivitiesManager. Contributed by Rohith Sharma K S. 
(cherry picked from commit 6310c0d17d)
 2018-08-01 08:34:09 +05:30
Eric Yang 7640d62716 YARN-8579. Recover NMToken of previous attempted component data. 
Contributed by Gour Saha
 2018-07-31 18:35:31 -04:00
Wangda Tan 5583711419 Preparing for 3.1.2 release 
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
 2018-07-31 13:08:55 -07:00
Wangda Tan 7b552c9d72 YARN-8418. App local logs could leaked if log aggregation fails to initialize for the app. (Bibin A Chundatt via wangda) 
Change-Id: I29a23ca4b219b48c92e7975cd44cddb8b0e04104
(cherry picked from commit 4b540bbfcf)
 2018-07-31 12:13:36 -07:00
Jonathan Hung b91cf90e1c YARN-7974. Allow updating application tracking url after registration. Contributed by Jonathan Hung 2018-07-30 17:57:25 -07:00
bibinchundatt 8cd2a73777 YARN-8584. Several typos in Log Aggregation related classes. Contributed by Szilard Nemeth. 
(cherry picked from commit 2b39ad2698)
 2018-07-31 00:07:08 +05:30
Sunil G f1eb5777a0 YARN-8591. [ATSv2] NPE while checking for entity acl in non-secure cluster. Contributed by Rohith Sharma K S. 
(cherry picked from commit 63e08ec071)
 2018-07-30 14:49:03 +05:30
bibinchundatt 2e7876a725 YARN-8558. NM recovery level db not cleaned up properly on container finish. Contributed by Bibin A Chundatt. 
(cherry picked from commit 3d586841ab)
 2018-07-28 20:56:35 +05:30
Eric Yang c2c3eee69c YARN-8508. Release GPU resource for killed container. 
Contributed by Chandni Singh

(cherry picked from commit ed9d60e888)
 2018-07-27 19:36:21 -04:00
Billie Rinaldi 06166f2255 YARN-8571. Validate service principal format prior to launching yarn service. Contributed by Eric Yang 
(cherry picked from commit b429f19d32)
 2018-07-27 11:35:41 -07:00
Eric Yang d6627cda9d YARN-8429. Improve diagnostic message when artifact is not set properly. 
Contributed by Gour Saha

(cherry picked from commit 8d3c068e59)
 2018-07-27 11:29:43 -04:00
Eric Yang 177f6045ac YARN-8545. Return allocated resource to RM for failed container. 
Contributed by Chandni Singh

(cherry picked from commit 40fad32824)
 2018-07-26 18:25:41 -04:00
Eric Yang 8e3807afe0 YARN-8330. Improved publishing ALLOCATED events to ATS. 
Contributed by Suma Shivaprasad

(cherry picked from commit f93ecf5c1e)
 2018-07-25 18:51:42 -04:00
Eric E Payne 830ef12af8 YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R 
(cherry picked from commit 9485c9aee6)
 2018-07-25 16:30:30 +00:00
bibinchundatt 8e65057eb1 YARN-8541. RM startup failure on recovery after user deletion. Contributed by Bibin A Chundatt. 2018-07-25 15:54:32 +05:30
Weiwei Yang b89624a943 YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang. 
(Cherry-picked from commit 5be9f4a5d0)
 2018-07-25 17:53:40 +08:00
Haibo Chen 7e7792dd7b YARN-6966. NodeManager metrics may return wrong negative values when NM restart. (Szilard Nemeth via Haibo Chen) 
(cherry picked from commit 9d3c39e9dd)
 2018-07-24 12:50:43 -07:00
Sunil G 4488fd8295 YARN-7748. TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events. Contributed by Weiwei Yang. 
(cherry picked from commit 35ce6eb1f5)
 2018-07-24 22:21:15 +05:30
bibinchundatt a684a2efb8 YARN-8548. AllocationRespose proto setNMToken initBuilder not done. Contributed by Bilwa S T. 
(cherry picked from commit ff7c2eda34)
 2018-07-24 16:30:31 +05:30
bibinchundatt 0710107f8d YARN-8544. [DS] AM registration fails when hadoop authorization is enabled. Contributed by Bibin A Chundatt. 
(cherry picked from commit 8461278833)
 2018-07-24 13:11:31 +05:30
Eric Yang 23b8546a80 YARN-8380. Support bind propagation options for mounts in docker runtime. 
Contributed by Billie Rinaldi

(cherry picked from commit 8688a0c7f8)
 2018-07-23 20:13:41 -04:00
Eric Yang e665c0a9dd YARN-8360. Improve YARN service restart policy and node manager auto restart policy. 
Contributed by Suma Shivaprasad

(cherry picked from commit 84d7bf1eef)
 2018-07-23 12:59:37 -04:00
Eric Yang 4f2a129f2e YARN-8301. Added YARN service upgrade instructions. 
Contributed by Chandni Singh

(cherry picked from commit 10014a4d88f239d3c072e51bc0739cba1fca9406)
 2018-07-20 19:48:19 -04:00
Weiwei Yang 004e1f248e YARN-8528. Final states in ContainerAllocation might be modified externally causing unexpected allocation results. Contributed by Xintong Song. 2018-07-20 22:43:47 +08:00
Eric Yang 76b8beb289 YARN-8501. Reduce complexity of RMWebServices getApps method. 
Contributed by Szilard Nemeth

(cherry picked from commit 5836e0a46b)
 2018-07-19 12:32:55 -04:00
Robert Kanter dfa71428ea YARN-8518. test-container-executor test_is_empty() is broken (Jim_Brennan via rkanter) 
(cherry picked from commit 1bc106a738)
 2018-07-18 16:07:48 -07:00
Robert Kanter 1c7d916347 Only mount non-empty directories for cgroups (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 0838fe8337)
 2018-07-18 16:07:48 -07:00
Robert Kanter 27e2b4b364 Disable mounting cgroups by default (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 351cf87c92)
 2018-07-18 16:07:48 -07:00
Eric Yang d82edec3c0 YARN-8538. Fixed memory leaks in container-executor and test cases. 
Contributed by Billie Rinaldi
 2018-07-18 13:44:49 -04:00
Wangda Tan 44beab0b63 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda) 
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
(cherry picked from commit 752dcce5f4)
 2018-07-16 11:04:08 -07:00
Eric E Payne 9a79e893f7 YARN-8421: when moving app, activeUsers is increased, even though app does not have outstanding request. Contributed by Kyungwan Nam 
(cherry picked from commit 937ef39b3f)
 2018-07-16 16:32:05 +00:00
bibinchundatt 677bbdcdc4 YARN-8434. Update federation documentation of Nodemanager configurations. Contributed by Bibin A Chundatt. 
(cherry picked from commit 4523cc5637)
 2018-07-15 13:57:14 +05:30
Jason Lowe 7cbb9597c4 YARN-8515. container-executor can crash with SIGPIPE after nodemanager restart. Contributed by Jim Brennan 
(cherry picked from commit 17118f446c)
 2018-07-13 10:06:38 -05:00
bibinchundatt 2aaad4000a YARN-8491. TestServiceCLI#testEnableFastLaunch fail when umask is 077. Contributed by K G Bakthavachalam. 
(cherry picked from commit 52e1bc8539)
 2018-07-11 16:24:37 +05:30
Sunil G 9b4ead92c8 YARN-8512. ATSv2 entities are not published to HBase from second attempt onwards. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7f1d3d0e9d)
 2018-07-11 12:27:12 +05:30
Sunil G 6f10491e64 YARN-8473. Containers being launched as app tears down can leave containers in NEW state. Contributed by Jason Lowe. 
(cherry picked from commit 705e2c1f7c)
 2018-07-10 20:12:47 +05:30
Wangda Tan 8ad82ea5b5 YARN-8506. Make GetApplicationsRequestPBImpl thread safe. (wangda) 
Change-Id: I2c006965375823c83036e7f45f7163d13c0bdf90
 2018-07-09 11:35:15 -07:00
Rohith Sharma K S 785ed3f8ae YARN-8302. ATS v2 should handle HBase connection issue properly. Contributed by Billie Rinaldi. 
(cherry picked from commit ba68320449)
 2018-07-06 15:20:39 -07:00
Xiaoyu Yao ffe2019887 HADOOP-15571. Multiple FileContexts created with the same configuration object should be allowed to have different umask. Contributed by Vinod Kumar Vavilapalli. 
(cherry picked from commit 629f7008b531bad469903e6a1f6f8cb4be691f36)
 2018-07-06 11:56:51 -07:00
Sunil G 844538d669 YARN-8459. Improve Capacity Scheduler logs to debug invalid states. Contributed by Wangda Tan. 
(cherry picked from commit 51654a3962)
 2018-07-03 10:00:05 -07:00
Junping Du 3d6ba2dd4e yarn.resourcemanager.fail-fast is used inconsistently. Contributed by Yuanbo Liu. 
(cherry picked from commit d9ba6f3656)
 2018-07-03 22:36:24 +08:00
Sunil G 8ffc191f61 YARN-8415. TimelineWebServices.getEntity should throw ForbiddenException instead of 404 when ACL checks fail. Contributed by Suma Shivaprasad. 
(cherry picked from commit fa9ef15ecd)
 2018-07-02 15:35:45 -07:00
Shane Kumpf 2f264764a4 YARN-8485. Priviledged container app launch is failing intermittently. Contributed by Eric Yang 
(cherry picked from commit 53e267fa72)
 2018-07-02 16:23:23 -06:00
Eric Yang 0aa7272199 YARN-8465. Fixed docker container status for node manager restart. 
Contributed by Shane Kumpf
 2018-07-02 13:40:02 -04:00
Jason Lowe dbb0a5f51f YARN-8451. Multiple NM heartbeat thread created when a slow NM resync with RM. Contributed by Botong Huang 
(cherry picked from commit 100470140d)
 2018-06-29 13:11:41 -05:00
Rohith Sharma K S 00406f583b YARN-8469. [UI2] URL needs to be trimmed to handle index.html redirection while accessing via knox. Contributed by Sunil Govindan. 
(cherry picked from commit a820738e34)
 2018-06-29 10:39:30 -07:00
Sunil G e7f2c9886a YARN-8455. Add basic ACL check for all ATS v2 REST APIs. Contributed by Rohith Sharma K S. 
(cherry picked from commit 469b29c081)
 2018-06-29 10:03:53 -07:00
Sunil G 122cb81b65 YARN-8379. Improve balancing resources in already satisfied queues by using Capacity Scheduler preemption. Contributed by Zian Chen. 
(cherry picked from commit 291194302c)
 2018-06-28 10:24:03 -07:00
Eric Yang 3ce3571a34 YARN-8409. Fixed NPE in ActiveStandbyElectorBasedElectorService. 
Contributed by Chandni Singh
 2018-06-28 13:11:56 -04:00
Sunil G 3c68d4ce45 YARN-8401. [UI2] new ui is not accessible with out internet connection. Contributed by Bibin A Chundatt. 
(cherry picked from commit fbaff369e9)
 2018-06-27 10:35:50 -07:00
Wangda Tan ec37e02bc9 YARN-8464. Async scheduling thread could be interrupted when there are no NodeManagers in cluster. (Sunil G via wangda) 
Change-Id: I4f5f856373378685713e77752ba6cf0988a66065
(cherry picked from commit bedc4fe079)
 2018-06-26 19:34:15 -07:00
Wangda Tan 8f226f4f1d YARN-8423. GPU does not get released even though the application gets killed. (Sunil G via wangda) 
Change-Id: I570db7d60f8c6c21762dd618a9207d1107c486a0
(cherry picked from commit ada8f63d0b)
 2018-06-26 19:34:10 -07:00
Sunil G aa8db00fc3 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-26 15:02:49 -07:00
Eric Yang f6aa6a6981 YARN-8108. Added option to disable loading existing filters to prevent 
security filter from initialize twice.
            Contributed by Sunil Govindan

(cherry picked from commit b69ba0f330)
 2018-06-26 17:45:13 -04:00
Eric Yang 9e9530f505 YARN-8214. Change default RegistryDNS port. 
Contributed by Billie Rinaldi

(cherry picked from commit 3e586330eb)
 2018-06-26 14:26:26 -04:00
Rohith Sharma K S e925de6483 YARN-8457. Compilation is broken with -Pyarn-ui. 
(cherry picked from commit 4ffe68a6f7)
 2018-06-25 10:39:26 -07:00
Weiwei Yang bfdbc9dea2 YARN-8443. Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers. Contributed by Tao Yang. 
(Cherry picked from commit 440140cea6)
 2018-06-25 09:28:57 +08:00
Eric Yang 3b3a85002d YARN-8326. Removed exit code file check for launched container. 
Contributed by Shane Kumpf

(cherry picked from commit 8a32bc39eb)
 2018-06-22 19:23:57 -04:00
Eric Yang 63ce762959 YARN-8445. Improved error message for duplicated service and component names. 
Contributed by Chandni Singh

(cherry picked from commit 9f15483c5d)
 2018-06-21 11:21:34 -04:00
Eric Yang d69c2e7867 YARN-8410. Fixed a bug in A record lookup by CNAME record. 
Contributed by Shane Kumpf

(cherry picked from commit 9591765040)
 2018-06-14 15:55:33 -04:00
Rohith Sharma K S 1aa6c9407c YARN-8155. Improve ATSv2 client logging in RM and NM publisher. Contributed by Abhishek Modi. 
(cherry picked from commit 9119b3cf8f)
 2018-06-14 13:51:13 +05:30
Eric Yang 934a005f72 YARN-8259. Improve privileged docker container liveliness checks. 
Contributed by Shane Kumpf

(cherry picked from commit 22994889dc)
 2018-06-13 19:26:23 -04:00
Eric Yang aba67c360c YARN-8411. Restart stopped system service during RM start. 
Contributed by Billie Rinaldi

(cherry picked from commit 69b0596897)
 2018-06-13 19:10:13 -04:00
Sunil G ed20c59d36 YARN-8404. Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down. Contributed by Rohith Sharma K S 
(cherry picked from commit 6307962b93)
 2018-06-13 16:09:52 +05:30
Weiwei Yang 4488ad5297 YARN-8394. Improve data locality documentation for Capacity Scheduler. Contributed by Weiwei Yang. 2018-06-13 13:54:09 +08:00
Inigo Goiri d5cd12c082 HADOOP-15529. ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 6e756e8a62)
 2018-06-12 10:25:05 -07:00
Sunil G 6f67e4db51 YARN-8419. [UI2] User cannot submit a new service as submit button is always disabled. Contributed by Suma Shivaprasad. 
(cherry picked from commit b3612dd90c)
 2018-06-12 17:41:09 +05:30
Rohith Sharma K S ba0a8f8792 YARN-8413. Flow activity page is failing with 'Timeline server failed with an error'. Contributed by Sunil Govindan. 
(cherry picked from commit 95303f50d3)
 2018-06-12 17:23:28 +05:30
Rohith Sharma K S af0309d793 YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace0)
 2018-06-12 17:23:22 +05:30
Inigo Goiri 65d2554ad8 YARN-8370. Some Node Manager tests fail on Windows due to improper path/file separator. Contributed by Anbang Hu. 
(cherry picked from commit 2b2f672022)
 2018-06-11 19:26:27 -07:00
Inigo Goiri 63c20f9021 YARN-8359. Exclude containermanager.linux test classes on Windows. Contributed by Jason Lowe. 
(cherry picked from commit 3492a1db2c0654ce5375360caa74a34f928f23be)
 2018-06-07 17:10:09 -07:00
Inigo Goiri 1fa51d4542 YARN-8400. Fix typos in YARN Federation documentation page. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 67fc70e09f)
 2018-06-07 16:10:51 -07:00
Rohith Sharma K S e9367ccb41 YARN-8386. App log can not be viewed from Logs tab in secure cluster. Contributed by Sunil Govindan. 
(cherry picked from commit 377ea1bcdf)
 2018-06-07 21:43:17 +05:30
Rohith Sharma K S a95f216945 YARN-8399. NodeManager is giving 403 GSS exception post upgrade to 3.1 in secure mode. Contributed by Sunil Govindan. 
(cherry picked from commit 58bc34f1e3)
 2018-06-07 10:27:08 +05:30
Rohith Sharma K S e4eac4a9f3 YARN-8396. Click on an individual container continuously spins and doesn't load the page. Contributed by Sunil Govindan. 
(cherry picked from commit 745f3a2441)
 2018-06-05 19:29:04 +05:30
Eric Yang 9dd79d1dcd YARN-8365. Set DNS query type according to client request. 
Contributed by Shane Kumpf

(cherry picked from commit 5889a3c0f931ebe1a9e6b8023c0557922b7d9e49)
 2018-06-04 19:03:07 -04:00
Robert Kanter 93d6ed859e YARN-4677. RMNodeResourceUpdateEvent update from scheduler can lead to race condition (wilfreds and gphillips via rkanter) 
(cherry picked from commit 0cd145a443)
 2018-06-04 15:32:47 -07:00
Miklos Szegedi 21948b048e YARN-8382. cgroup file leak in NM. Contributed by Hu Ziqian. 
(cherry picked from commit 925fdf761a513130e23c10575c7328c8681cff1d)
 2018-06-04 10:56:13 -07:00
Inigo Goiri 83d0964d82 YARN-8389. Improve the description of machine-list property in Federation docs. Contributed by Takanobu Asanuma. 
(cherry picked from commit 61fc7f73f2)
 2018-06-04 09:28:40 -07:00
Sunil G ea2fa52d92 YARN-8276. [UI2] After version field became mandatory, form-based submission of new YARN service doesn't work. Contributed by Gergely Novák. 
(cherry picked from commit 9c4cbed8d1)
 2018-06-04 00:09:44 +05:30
Billie Rinaldi e59e9feefa YARN-8342. Enable untrusted docker image to run with launch command. Contributed by Eric Yang 
(cherry picked from commit 31998643a5)
 2018-06-02 14:49:21 -07:00
Rohith Sharma K S f7d0ca71ac More YARN pages need to honor yarn.resourcemanager.display.per-user-apps(addendum). Contributed by Sunil G. 
(cherry picked from commit 8261f9e571)
 2018-06-02 09:43:08 +05:30
Wangda Tan 0be93a7973 YARN-8349. Remove YARN registry entries when a service is killed by the RM. (Billie Rinaldi via wangda) 
Change-Id: Ia58db3637789a8921482f564aa9bdf99c45cc36c
(cherry picked from commit ff583d3fa3)
 2018-06-01 15:53:26 -07:00
Wangda Tan ed20576413 YARN-8372. Distributed shell app master should not release containers when shutdown if keep-container is true. (Suma Shivaprasad via wangda) 
Change-Id: Ief04d1ca865621f348fba4ac85fa78bc47465904
(cherry picked from commit 8956e5b8db)
 2018-06-01 15:47:59 -07:00
Wangda Tan 1cee85d8c4 YARN-7962. Race Condition When Stopping DelegationTokenRenewer causes RM crash during failover. (BELUGA BEHR via wangda) 
Change-Id: I617e2645f60a57080058ad5f06af860fb3f682c8
(cherry picked from commit 931f78718f)
 2018-06-01 15:47:51 -07:00
Wangda Tan cc460fea2d YARN-8384. stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop. (Eric Yang via wangda) 
Change-Id: Idbb2cd250f4f62dc32993e0d3ca3ec5684616baa
(cherry picked from commit 3a6bd77550)
 2018-06-01 15:47:45 -07:00
Eric Yang 4193892de6 YARN-8308. Fixed YARN Service AM failure with HDFS token renewal. 
Contributed by Gour Saha

(cherry picked from commit 7c7ac07f013f46db89c73681806300699ee534d6)
 2018-05-31 21:10:04 -04:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) a468400bad YARN-8197. Fixed AM IP Filter and Webapp proxy to redirect app tracking-URLs correctly when UI is secure. Contributed by Sunil Govindan. 
(cherry picked from commit 6b74f5d7fc)
 2018-05-31 17:04:45 -07:00
Billie Rinaldi 1bb112cd96 YARN-8333. Load balance YARN services using RegistryDNS multiple A records. Contributed by Eric Yang 
(cherry picked from commit 6bc92e304f)
 2018-05-31 06:49:04 -07:00
Weiwei Yang ec4240a7fa YARN-8367. Fix NPE in SingleConstraintAppPlacementAllocator when placement constraint in SchedulingRequest is null. Contributed by Weiwei Yang. 
(Cherry picked from commit 6468071f13)
 2018-05-31 21:04:41 +08:00
Billie Rinaldi 58e7d6c3c0 YARN-8350. NPE in service AM related to placement policy. Contributed by Gour Saha 
(cherry picked from commit 778a4a24be)
 2018-05-30 13:21:17 -07:00
Billie Rinaldi 7bd5d79177 YARN-8368. yarn app start cli should print applicationId. Contributed by Rohith Sharma K S 
(cherry picked from commit 96eefcc84a)
 2018-05-30 12:38:53 -07:00
Eric E Payne 3d06d75423 YARN-8377: Javadoc build failed in hadoop-yarn-server-nodemanager. Contributed by Takanobu Asanuma 
(cherry picked from commit e44c0849d7)
 2018-05-30 17:28:14 +00:00
Eric Yang 03209e8966 YARN-8362. Bugfix logic in container retries in node manager. 
Contributed by Chandni Singh

(cherry picked from commit 135941e00d)
 2018-05-29 17:04:01 -04:00
Jason Lowe a1fd04c4f4 YARN-8329. Docker client configuration can still be set incorrectly. Contributed by Shane Kumpf 
(cherry picked from commit 4827e9a908)
 2018-05-29 14:48:01 -05:00
Wangda Tan 3eb1cb18c7 YARN-8369. Javadoc build failed due to 'bad use of >'. (Takanobu Asanuma via wangda) 
Change-Id: I79a42154e8f86ab1c3cc939b3745024b8eebe5f4
(cherry picked from commit 17aa40f669)
 2018-05-29 09:28:34 -07:00
Wangda Tan b262ea1381 YARN-8339. Service AM should localize static/archive resource types to container working directory instead of 'resources'. (Suma Shivaprasad via wangda) 
Change-Id: I9f8e8f621650347f6c2f9e3420edee9eb2f356a4
(cherry picked from commit 3061bfcde5)
 2018-05-29 09:28:27 -07:00
Jason Lowe 500b0ee2ce YARN-8338. TimelineService V1.5 doesn't come up after HADOOP-15406. Contributed by Vinod Kumar Vavilapalli 
(cherry picked from commit 31ab960f4f)
 2018-05-29 11:03:19 -05:00
Sunil G 7f9caf8986 YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne. 
(cherry picked from commit 7c343669ba)
 2018-05-28 16:33:48 +05:30
Rohith Sharma K S f19e8e2246 MAPREDUCE-7097. MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user. Contributed by Sunil Govindan. 
(cherry picked from commit 88cbe57c06)
 2018-05-28 14:06:50 +05:30
Wangda Tan 4d41cb1696 YARN-8213. Add Capacity Scheduler performance metrics. (Weiwei Yang via wangda) 
Change-Id: Ieea6f3eeb83c90cd74233fea896f0fcd0f325d5f
(cherry picked from commit f24c842d52)
 2018-05-25 22:04:54 -07:00
Eric E Payne 053937f436 YARN-8292: Fix the dominant resource preemption cannot happen when some of the resource vector becomes negative. Contributed by Wangda Tan. 
(cherry picked from commit 8d5509c681)
 2018-05-25 16:27:09 +00:00
Miklos Szegedi 1483c90379 YARN-8310. Handle old NMTokenIdentifier, AMRMTokenIdentifier, and ContainerTokenIdentifier formats. Contributed by Robert Kanter. 
(cherry picked from commit 3e5f7ea986)
 2018-05-24 15:14:15 -07:00
Eric Yang f49e697cc8 YARN-8357. Fixed NPE when YARN service is saved and not deployed. 
Contributed by Chandni Singh

(cherry picked from commit d9852eb589)
 2018-05-24 16:34:32 -04:00
Eric Yang 514c05abd0 YARN-8316. Improved diagnostic message for ATS unavailability for YARN Service. 
Contributed by Billie Rinaldi
 2018-05-24 16:26:53 -04:00
Shane Kumpf de43b41d0f YARN-6919. Add default volume mount list. Contributed by Eric Badger 2018-05-24 09:35:35 -06:00
Rohith Sharma K S 923c833da4 YARN-8319. More YARN pages need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 
(cherry picked from commit c05b5d424b)
 2018-05-24 14:22:43 +05:30
Rohith Sharma K S 67842c56fe YARN-8346. Upgrading to 3.1 kills running containers with error 'Opportunistic container queue is full'. Contributed by Jason Lowe. 
(cherry picked from commit 4cc0c9b0ba)
 2018-05-24 12:25:33 +05:30
Eric Yang 05d905f586 YARN-7530. Refactored YARN service API project location. 
Contributed by Chandni Singh
 2018-05-23 22:43:54 -04:00
Inigo Goiri fa8af4aab5 YARN-8327. Fix TestAggregatedLogFormat#testReadAcontainerLogs1 on Windows. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit f09dc73001)
 2018-05-23 15:59:59 -07:00
Inigo Goiri 61b5b2f4f7 YARN-8344. Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit e99e5bf104)
 2018-05-23 14:16:03 -07:00
Rohith Sharma K S 8dcc7b853b YARN-8297. Incorrect ATS Url used for Wire encrypted cluster.(addendum). Contributed by Sunil G. 
(cherry picked from commit f61e3e752e)
 2018-05-23 18:32:57 +05:30
Wangda Tan 2bbf1b4435 YARN-8332. Incorrect min/max allocation property name in resource types doc. (Weiwei Yang via wangda) 
Change-Id: If74f1ceed9c045a2cb2d6593741278b65ac44a9f
(cherry picked from commit 83f53e5c62)
 2018-05-22 13:39:20 -07:00
Wangda Tan 6aaf33e50c YARN-8290. SystemMetricsPublisher.appACLsUpdated should be invoked after application information is published to ATS to avoid "User is not set in the application report" Exception. (Eric Yang via wangda) 
Change-Id: I0ac6ddd19740d1aa7dd07111cd11af71ddc2fcaf
(cherry picked from commit bd15d2396e)
 2018-05-22 13:39:15 -07:00
Eric Yang 7c3f236783 YARN-7960. Added security flag no-new-privileges for YARN Docker integration. 
Contributed by Eric Badger

(cherry picked from commit 6176d2b35c)
 2018-05-22 13:47:08 -04:00
Jason Lowe 9788c8c01e YARN-8206. Sending a kill does not immediately kill docker containers. Contributed by Eric Badger 
(cherry picked from commit 5f11288e41)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java
 2018-05-22 09:49:31 -05:00
Eric E Payne 5b2bbff141 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdee)
 2018-05-21 20:19:44 +00:00
Eric Yang 61e8974049 YARN-7530. Refactored YARN service API project location. 
Contributed by Chandni Singh
 2018-05-18 17:33:18 -04:00
Eric Yang 0791785574 YARN-8296. Removed unique_component_support from YARN services. 
Contributed by Suma Shivaprasad

(cherry picked from commit 2f2dd22aad)
 2018-05-17 20:59:50 -04:00
Eric Yang 79351d4fda YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag. 
Contributed by Chandni Singh

(cherry picked from 989cfdc1e0)
 2018-05-17 20:57:14 -04:00
Eric Yang aafaa5f99b YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag. 
Contributed by Chandni Singh

(cherry-picked from commit d45a0b7d73)
 2018-05-17 17:32:29 -04:00
Eric Yang 044573b628 YARN-8080. Add restart policy for YARN services. 
Contributed by Suma Shivaprasad

(cherry picked from commit 7f083ed869)
 2018-05-17 17:20:45 -04:00
Eric Yang c6c3b0afa0 YARN-8293. Removed "User Name for service" for deploying secure YARN service. 
Contributed by Sunil G

(cherry picked from commit 7802af6e9a)
 2018-05-17 17:07:55 -04:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 0460201c3a YARN-8128. Documenting the per-node per-app file limit in YARN log aggregation. Contributed by Xuan Gong. 
(cherry picked from commit 328f0847e3)
 2018-05-17 13:09:09 -07:00
Rohith Sharma K S 4222e8f21b YARN-8297. Incorrect ATS Url used for Wire encrypted cluster. Contributed by Sunil G. 
(cherry picked from commit c95795cd8f)
 2018-05-17 16:25:38 +05:30
Eric Yang 2c3f300464 YARN-8300. Fixed NPE in DefaultUpgradeComponentsFinder. 
Contributed by Suma Shivaprasad

(cherry picked from 91357c22ef)
 2018-05-16 12:42:22 -04:00
Jason Lowe 79b2a508e0 YARN-8284. get_docker_command refactoring. Contributed by Eric Badger 
(cherry picked from commit d47c09dcb1)
 2018-05-16 09:34:24 -05:00
Eric Yang 051d9cff9c YARN-8081. Add support to upgrade a component. 
Contributed by Chandni Singh

(cherry picked from commit 8d3b39de89)
 2018-05-15 20:42:24 -04:00
Weiwei Yang 2bb39330a4 YARN-8278. DistributedScheduling is not working in HA. Contributed by Bibin A Chundatt. 
(Cherry picked from commit 2bb647bb91)
 2018-05-15 18:47:15 +08:00