Commit Graph

4676 Commits

Author SHA1 Message Date
Sunil G ea2fa52d92 YARN-8276. [UI2] After version field became mandatory, form-based submission of new YARN service doesn't work. Contributed by Gergely Novák.
(cherry picked from commit 9c4cbed8d1)
2018-06-04 00:09:44 +05:30
Billie Rinaldi e59e9feefa YARN-8342. Enable untrusted docker image to run with launch command. Contributed by Eric Yang
(cherry picked from commit 31998643a5)
2018-06-02 14:49:21 -07:00
Rohith Sharma K S f7d0ca71ac More YARN pages need to honor yarn.resourcemanager.display.per-user-apps(addendum). Contributed by Sunil G.
(cherry picked from commit 8261f9e571)
2018-06-02 09:43:08 +05:30
Wangda Tan 0be93a7973 YARN-8349. Remove YARN registry entries when a service is killed by the RM. (Billie Rinaldi via wangda)
Change-Id: Ia58db3637789a8921482f564aa9bdf99c45cc36c
(cherry picked from commit ff583d3fa3)
2018-06-01 15:53:26 -07:00
Wangda Tan ed20576413 YARN-8372. Distributed shell app master should not release containers when shutdown if keep-container is true. (Suma Shivaprasad via wangda)
Change-Id: Ief04d1ca865621f348fba4ac85fa78bc47465904
(cherry picked from commit 8956e5b8db)
2018-06-01 15:47:59 -07:00
Wangda Tan 1cee85d8c4 YARN-7962. Race Condition When Stopping DelegationTokenRenewer causes RM crash during failover. (BELUGA BEHR via wangda)
Change-Id: I617e2645f60a57080058ad5f06af860fb3f682c8
(cherry picked from commit 931f78718f)
2018-06-01 15:47:51 -07:00
Wangda Tan cc460fea2d YARN-8384. stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop. (Eric Yang via wangda)
Change-Id: Idbb2cd250f4f62dc32993e0d3ca3ec5684616baa
(cherry picked from commit 3a6bd77550)
2018-06-01 15:47:45 -07:00
Eric Yang 4193892de6 YARN-8308. Fixed YARN Service AM failure with HDFS token renewal.
Contributed by Gour Saha

(cherry picked from commit 7c7ac07f013f46db89c73681806300699ee534d6)
2018-05-31 21:10:04 -04:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) a468400bad YARN-8197. Fixed AM IP Filter and Webapp proxy to redirect app tracking-URLs correctly when UI is secure. Contributed by Sunil Govindan.
(cherry picked from commit 6b74f5d7fc)
2018-05-31 17:04:45 -07:00
Billie Rinaldi 1bb112cd96 YARN-8333. Load balance YARN services using RegistryDNS multiple A records. Contributed by Eric Yang
(cherry picked from commit 6bc92e304f)
2018-05-31 06:49:04 -07:00
Weiwei Yang ec4240a7fa YARN-8367. Fix NPE in SingleConstraintAppPlacementAllocator when placement constraint in SchedulingRequest is null. Contributed by Weiwei Yang.
(Cherry picked from commit 6468071f13)
2018-05-31 21:04:41 +08:00
Billie Rinaldi 58e7d6c3c0 YARN-8350. NPE in service AM related to placement policy. Contributed by Gour Saha
(cherry picked from commit 778a4a24be)
2018-05-30 13:21:17 -07:00
Billie Rinaldi 7bd5d79177 YARN-8368. yarn app start cli should print applicationId. Contributed by Rohith Sharma K S
(cherry picked from commit 96eefcc84a)
2018-05-30 12:38:53 -07:00
Eric E Payne 3d06d75423 YARN-8377: Javadoc build failed in hadoop-yarn-server-nodemanager. Contributed by Takanobu Asanuma
(cherry picked from commit e44c0849d7)
2018-05-30 17:28:14 +00:00
Eric Yang 03209e8966 YARN-8362. Bugfix logic in container retries in node manager.
Contributed by Chandni Singh

(cherry picked from commit 135941e00d)
2018-05-29 17:04:01 -04:00
Jason Lowe a1fd04c4f4 YARN-8329. Docker client configuration can still be set incorrectly. Contributed by Shane Kumpf
(cherry picked from commit 4827e9a908)
2018-05-29 14:48:01 -05:00
Wangda Tan 3eb1cb18c7 YARN-8369. Javadoc build failed due to 'bad use of >'. (Takanobu Asanuma via wangda)
Change-Id: I79a42154e8f86ab1c3cc939b3745024b8eebe5f4
(cherry picked from commit 17aa40f669)
2018-05-29 09:28:34 -07:00
Wangda Tan b262ea1381 YARN-8339. Service AM should localize static/archive resource types to container working directory instead of 'resources'. (Suma Shivaprasad via wangda)
Change-Id: I9f8e8f621650347f6c2f9e3420edee9eb2f356a4
(cherry picked from commit 3061bfcde5)
2018-05-29 09:28:27 -07:00
Jason Lowe 500b0ee2ce YARN-8338. TimelineService V1.5 doesn't come up after HADOOP-15406. Contributed by Vinod Kumar Vavilapalli
(cherry picked from commit 31ab960f4f)
2018-05-29 11:03:19 -05:00
Sunil G 7f9caf8986 YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne.
(cherry picked from commit 7c343669ba)
2018-05-28 16:33:48 +05:30
Rohith Sharma K S f19e8e2246 MAPREDUCE-7097. MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user. Contributed by Sunil Govindan.
(cherry picked from commit 88cbe57c06)
2018-05-28 14:06:50 +05:30
Wangda Tan 4d41cb1696 YARN-8213. Add Capacity Scheduler performance metrics. (Weiwei Yang via wangda)
Change-Id: Ieea6f3eeb83c90cd74233fea896f0fcd0f325d5f
(cherry picked from commit f24c842d52)
2018-05-25 22:04:54 -07:00
Eric E Payne 053937f436 YARN-8292: Fix the dominant resource preemption cannot happen when some of the resource vector becomes negative. Contributed by Wangda Tan.
(cherry picked from commit 8d5509c681)
2018-05-25 16:27:09 +00:00
Miklos Szegedi 1483c90379 YARN-8310. Handle old NMTokenIdentifier, AMRMTokenIdentifier, and ContainerTokenIdentifier formats. Contributed by Robert Kanter.
(cherry picked from commit 3e5f7ea986)
2018-05-24 15:14:15 -07:00
Eric Yang f49e697cc8 YARN-8357. Fixed NPE when YARN service is saved and not deployed.
Contributed by Chandni Singh

(cherry picked from commit d9852eb589)
2018-05-24 16:34:32 -04:00
Eric Yang 514c05abd0 YARN-8316. Improved diagnostic message for ATS unavailability for YARN Service.
Contributed by Billie Rinaldi
2018-05-24 16:26:53 -04:00
Shane Kumpf de43b41d0f YARN-6919. Add default volume mount list. Contributed by Eric Badger 2018-05-24 09:35:35 -06:00
Rohith Sharma K S 923c833da4 YARN-8319. More YARN pages need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G.
(cherry picked from commit c05b5d424b)
2018-05-24 14:22:43 +05:30
Rohith Sharma K S 67842c56fe YARN-8346. Upgrading to 3.1 kills running containers with error 'Opportunistic container queue is full'. Contributed by Jason Lowe.
(cherry picked from commit 4cc0c9b0ba)
2018-05-24 12:25:33 +05:30
Eric Yang 05d905f586 YARN-7530. Refactored YARN service API project location.
Contributed by Chandni Singh
2018-05-23 22:43:54 -04:00
Inigo Goiri fa8af4aab5 YARN-8327. Fix TestAggregatedLogFormat#testReadAcontainerLogs1 on Windows. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit f09dc73001)
2018-05-23 15:59:59 -07:00
Inigo Goiri 61b5b2f4f7 YARN-8344. Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit e99e5bf104)
2018-05-23 14:16:03 -07:00
Rohith Sharma K S 8dcc7b853b YARN-8297. Incorrect ATS Url used for Wire encrypted cluster.(addendum). Contributed by Sunil G.
(cherry picked from commit f61e3e752e)
2018-05-23 18:32:57 +05:30
Wangda Tan 2bbf1b4435 YARN-8332. Incorrect min/max allocation property name in resource types doc. (Weiwei Yang via wangda)
Change-Id: If74f1ceed9c045a2cb2d6593741278b65ac44a9f
(cherry picked from commit 83f53e5c62)
2018-05-22 13:39:20 -07:00
Wangda Tan 6aaf33e50c YARN-8290. SystemMetricsPublisher.appACLsUpdated should be invoked after application information is published to ATS to avoid "User is not set in the application report" Exception. (Eric Yang via wangda)
Change-Id: I0ac6ddd19740d1aa7dd07111cd11af71ddc2fcaf
(cherry picked from commit bd15d2396e)
2018-05-22 13:39:15 -07:00
Eric Yang 7c3f236783 YARN-7960. Added security flag no-new-privileges for YARN Docker integration.
Contributed by Eric Badger

(cherry picked from commit 6176d2b35c)
2018-05-22 13:47:08 -04:00
Jason Lowe 9788c8c01e YARN-8206. Sending a kill does not immediately kill docker containers. Contributed by Eric Badger
(cherry picked from commit 5f11288e41)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java
2018-05-22 09:49:31 -05:00
Eric E Payne 5b2bbff141 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam.
(cherry picked from commit 0b4c44bdee)
2018-05-21 20:19:44 +00:00
Eric Yang 61e8974049 YARN-7530. Refactored YARN service API project location.
Contributed by Chandni Singh
2018-05-18 17:33:18 -04:00
Eric Yang 0791785574 YARN-8296. Removed unique_component_support from YARN services.
Contributed by Suma Shivaprasad

(cherry picked from commit 2f2dd22aad)
2018-05-17 20:59:50 -04:00
Eric Yang 79351d4fda YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag.
Contributed by Chandni Singh

(cherry picked from 989cfdc1e0)
2018-05-17 20:57:14 -04:00
Eric Yang aafaa5f99b YARN-8141. Removed YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS flag.
Contributed by Chandni Singh

(cherry-picked from commit d45a0b7d73)
2018-05-17 17:32:29 -04:00
Eric Yang 044573b628 YARN-8080. Add restart policy for YARN services.
Contributed by Suma Shivaprasad

(cherry picked from commit 7f083ed869)
2018-05-17 17:20:45 -04:00
Eric Yang c6c3b0afa0 YARN-8293. Removed "User Name for service" for deploying secure YARN service.
Contributed by Sunil G

(cherry picked from commit 7802af6e9a)
2018-05-17 17:07:55 -04:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 0460201c3a YARN-8128. Documenting the per-node per-app file limit in YARN log aggregation. Contributed by Xuan Gong.
(cherry picked from commit 328f0847e3)
2018-05-17 13:09:09 -07:00
Rohith Sharma K S 4222e8f21b YARN-8297. Incorrect ATS Url used for Wire encrypted cluster. Contributed by Sunil G.
(cherry picked from commit c95795cd8f)
2018-05-17 16:25:38 +05:30
Eric Yang 2c3f300464 YARN-8300. Fixed NPE in DefaultUpgradeComponentsFinder.
Contributed by Suma Shivaprasad

(cherry picked from 91357c22ef)
2018-05-16 12:42:22 -04:00
Jason Lowe 79b2a508e0 YARN-8284. get_docker_command refactoring. Contributed by Eric Badger
(cherry picked from commit d47c09dcb1)
2018-05-16 09:34:24 -05:00
Eric Yang 051d9cff9c YARN-8081. Add support to upgrade a component.
Contributed by Chandni Singh

(cherry picked from commit 8d3b39de89)
2018-05-15 20:42:24 -04:00
Weiwei Yang 2bb39330a4 YARN-8278. DistributedScheduling is not working in HA. Contributed by Bibin A Chundatt.
(Cherry picked from commit 2bb647bb91)
2018-05-15 18:47:15 +08:00
Sunil G 0a30a548f1 YARN-8236. Invalid kerberos principal file name cause NPE in native service. Contributed by Gour Saha.
(cherry picked from commit 58b97c79e3)
2018-05-15 12:18:09 +05:30
Sunil G d997959560 YARN-8166. [UI2] Service page header links are broken. Contributed by Yesha Vora.
(cherry picked from commit ffb9210ded)
2018-05-15 12:13:44 +05:30
Sunil G d0452370cc YARN-8266. [UI2] Clicking on application from cluster view should redirect to application attempt page. Contributed by Yesha Vora.
(cherry picked from commit 796b2b0ee3)
2018-05-15 12:02:40 +05:30
Haibo Chen 8d3598525d YARN-8130 Race condition when container events are published for KILLED applications. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit 2d00a0c71b)
2018-05-15 11:40:47 +05:30
Naganarasimha 26e8fdffd6 YARN-8288. Fix wrong number of table columns in Resource Model doc. Contributed by Weiwei Yang.
(cherry picked from commit 8a2b5914f3)
2018-05-15 00:10:58 +08:00
Sunil G e6c607cc2c YARN-8271. [UI2] Improve labeling of certain tables. Contributed by Yesha Vora.
(cherry picked from commit 89d0b87ad3)
2018-05-14 21:11:59 +05:30
Eric Yang 8edcbe9460 YARN-8265. Improve DNS handling on docker IP changes.
Contributed by Billie Rinaldi

(cherry picked from commit 0ff94563b9)
2018-05-11 22:41:30 -07:00
Jason Lowe dfe73334c2 YARN-7654. Support ENTRY_POINT for docker container. Contributed by Eric Yang
(cherry picked from commit 6c8e51ca7e)
2018-05-11 18:57:20 -05:00
Billie Rinaldi 355ff085e6 YARN-8243. Flex down should remove instance with largest component instance ID first. Contributed by Gour Saha
(cherry picked from commit ca612e353f)
2018-05-11 12:50:40 -07:00
Jason Lowe f6d6a2ab04 YARN-8244. TestContainerSchedulerQueuing.testStartMultipleContainers failed. Contributed by Jim Brennan
(cherry picked from commit dc912994a1)
2018-05-11 14:13:55 -05:00
Eric Yang 645429ca28 YARN-8274. Fixed a bug on docker start command.
Contributed by Jason Lowe
2018-05-11 14:26:20 -04:00
Weiwei Yang b4213bfd90 YARN-7003. DRAINING state of queues is not recovered after RM restart. Contributed by Tao Yang.
(Cherry picked from commit 9db9cd95bd)
2018-05-11 10:55:05 +08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 4c0a6b67f3 YARN-8249. Fixed few REST APIs in RMWebServices to have static-user check. Contributed by Sunil Govindan.
(cherry picked from commit d76fbbc9b8)
2018-05-10 19:07:23 -07:00
Robert Kanter 0506c762b2 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter)
(cherry picked from commit c8b53c4364)
2018-05-10 09:52:11 -07:00
Vrushali C ad4d4153da YARN-8247 Incorrect HTTP status code returned by ATSv2 for non-whitelisted users. Contributed by Rohith Sharma K S 2018-05-09 22:30:59 -07:00
bibinchundatt b29479c93a YARN-8201. Skip stacktrace of few exception from ClientRMService. Contributed by Bilwa S T.
(cherry picked from commit cc0310a526)
2018-05-10 09:29:43 +05:30
Eric Yang 6ce89ea2a8 YARN-8261. Fixed a bug in creation of localized container directory.
Contributed by Jason Lowe

(cherry picked from commit af4fc2e628)
2018-05-09 13:34:54 -04:00
Rohith Sharma K S ee19c47d08 YARN-8260. [UI2] Per-application tracking URL is no longer available in YARN UI2. Contributed by Sunil G.
(cherry picked from commit cd68c7cc69)
2018-05-09 12:34:11 +05:30
Rohith Sharma K S 438c5b12aa YARN-8264. [UI2] GPU Info tab disappears if we click any sub link under List of Applications or List of Containers. Contributed by Sunil G.
(cherry picked from commit 9acfaee1f0)
2018-05-09 12:34:01 +05:30
Rohith Sharma K S e3a7dbc4ec YARN-8239. [UI2] Clicking on Node Manager UI under AM container info / App Attempt page goes to old RM UI. Contributed by Sunil G.
(cherry picked from commit eb7b256cc2)
2018-05-09 12:33:49 +05:30
Billie Rinaldi f7cbfeb726 YARN-7894. Improve ATS response for DShell DS_CONTAINER when container launch fails. Contributed by Chandni Singh
(cherry picked from commit 1ef0a1db1d)
2018-05-08 13:51:02 -07:00
Jason Lowe 39236da2ff YARN-8207. Docker container launch use popen have risk of shell expansion. Contributed by Eric Yang.
(cherry picked from commit a2ea756420)
2018-05-08 15:37:24 -05:00
Wangda Tan 0873cfcb56 YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian via wangda)
Change-Id: Ia21e1da6871570c993bbedde76ce32929e95970f
(cherry picked from commit 6b96a73bb0)
2018-05-08 11:35:11 -07:00
Rohith Sharma K S b69c65fc6b YARN-8253. HTTPS Ats v2 api call fails with 'bad HTTP parsed'. Contributed by Charan Hebri.
(cherry picked from commit 7450583721)
2018-05-08 12:31:35 +05:30
Sunil G 9355f56b36 YARN-8251. [UI2] Clicking on Application link at the header goes to Diagnostics Tab instead of AppAttempt Tab. Contributed by Yesha Vora.
(cherry picked from commit 2916c90500)
2018-05-08 11:37:43 +05:30
Sunil G 9eb51a3004 YARN-5151. [UI2] Support kill application from new YARN UI. Contributed by Gergely Novák.
(cherry picked from commit 9832265e1d)
2018-05-08 10:38:04 +05:30
Weiwei Yang 68dd01efc1 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang.
(Cherry picked from commit 67f239c42f)
2018-05-07 11:41:01 +08:00
Billie Rinaldi e933ed0ee3 YARN-8079. Support static and archive unmodified local resources in service AM. Contributed by Suma Shivaprasad
(cherry picked from commit 6795f8072f)
2018-05-04 09:47:06 -07:00
Eric Yang a984abc472 YARN-8223. Improved yarn auxiliary service to load jar file from HDFS.
Contributed by Zian Chen

(cherry picked from commit 8cdb032aff)
2018-05-04 12:38:28 -04:00
Billie Rinaldi 04c3f11de4 YARN-7818. Remove privileged operation warnings during container launch for the ContainerRuntimes. Contributed by Shane Kumpf
(cherry picked from commit 502914ca32)
2018-05-04 08:55:30 -07:00
Eric Yang 2148fc1c3a YARN-8226. Improved anti-affinity description in YARN Service doc.
Contributed by Gour Saha

(cherry picked from commit 7698737207)
2018-05-03 13:38:21 -04:00
Eric Yang b09af11117 YARN-7961. Improve status message for YARN service.
Contributed by Gour Saha
(cherry picked from commit 7fe3214d4b)
2018-05-03 13:29:07 -04:00
Eric Yang cf46533bca YARN-7973. Added ContainerRelaunch feature for Docker containers.
Addendum patch 001
            Contributed by Shane Kumpf
2018-05-03 12:46:04 -04:00
Eric Yang f729fb0baa YARN-8194. Fixed reinitialization error for LinuxContainerExecutor.
Contributed by Chandni Singh

(cherry picked from commit f4d280f02b)
2018-05-03 11:59:35 -04:00
Eric Yang e4b9981612 YARN-7973. Added ContainerRelaunch feature for Docker containers.
Contributed by Shane Kumpf
2018-05-03 11:57:01 -04:00
Rohith Sharma K S 4c93cd228c YARN-8217. RmAuthenticationFilterInitializer and TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator. Contributed by Suma Shivaprasad.
(cherry picked from commit ee2ce923a9)
2018-05-03 18:20:21 +05:30
Eric Yang 61fac20c9e YARN-8209. Fixed NPE in Yarn Service deletion.
Contributed by Eric Badger
2018-05-02 15:05:30 -04:00
Konstantinos Karanasos 62ad9d512d YARN-8113. Update placement constraints doc with application namespaces and inter-app constraints. Contributed by Weiwei Yang.
(cherry picked from commit 3b34fca4b5)
2018-05-02 11:51:45 -07:00
Weiwei Yang 6fce88765a YARN-8222. Fix potential NPE when gets RMApp from RM context. Contributed by Tao Yang. 2018-05-02 18:02:24 +08:00
Yiqun Lin 2741af505f
YARN-6385. Fix checkstyle warnings in TestFileSystemApplicationHistoryStore
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3265b55119)
2018-05-02 18:15:53 +09:00
Eric Yang a6f3d6ffcb YARN-7799. Improved YARN service jar file handling.
Contributed by Billie Rinaldi
(cherry picked from commit 24eeea8b18)
2018-05-01 16:49:29 -04:00
Billie Rinaldi 4174a0ed25 YARN-2674. Fix distributed shell AM container relaunch during RM work preserving restart. Contributed by Shane Kumpf
(cherry picked from commit 4e1382aca4)
2018-05-01 08:04:39 -07:00
Sunil G 12fe48b5e5 YARN-8187. [UI2] Individual Node page does not contain breadcrumb trail. Contributed by Zian Chen.
(cherry picked from commit d6139c5106)
2018-05-01 14:02:20 +05:30
Weiwei Yang f34f4e9c11 YARN-8212. Pending backlog for async allocation threads should be configurable. Contributed by Tao Yang.
(cherry picked from commit 2d319e3793)
2018-05-01 09:57:15 +08:00
Eric Yang 0fc55a8f57 YARN-8228. Added hostname length check for docker container.
Contributed by Shane Kumpf

(cherry picked from commit a966ec6e23)
2018-04-30 19:14:37 -04:00
Konstantinos Karanasos 9d2967098d YARN-8195. Fix constraint cardinality check in the presence of multiple target allocation tags. Contributed by Weiwei Yang.
(cherry picked from commit 9b09555451)
2018-04-30 11:55:26 -07:00
Weiwei Yang 36fc9a6939 YARN-8225. YARN precommit build failing in TestPlacementConstraintTransformations. Contributed by Shane Kumpf.
(cherry picked from commit 2c95eb8111)
2018-04-28 18:50:51 +08:00
Wangda Tan 090a8a274e YARN-8005. Add unit tests for queue priority with dominant resource calculator. (Zian Chen via wangda)
Change-Id: I17a645f20869a1e5d86fa7a325c93fec908b91dc
(cherry picked from commit 92c5331423)
2018-04-27 13:13:52 -07:00
Wangda Tan f9add5fdd9 YARN-8210. AMRMClient logging on every heartbeat to track updation of AM RM token causes too many log lines to be generated in AM logs. (Suma Shivaprasad via wangda)
Change-Id: I70edd6e301fd5e78d479e1882aedc9332a0827aa
(cherry picked from commit b1833d9ba2)
2018-04-27 13:13:47 -07:00
Rohith Sharma K S b3fd464e55 YARN-8221. RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G.
(cherry picked from commit ef3ecc308d)
2018-04-27 22:59:16 +05:30
Eric Yang e75aeacf29 YARN-7781. Update YARN service documentation.
Contributed by Gour Saha
(cherry picked from commit 24a5ccbf4b)
2018-04-27 12:40:24 -04:00
Eric Yang 0e19c0dd72 YARN-8204. Added a flag to disable YARN service upgrade.
Contributed by Chandni Singh
(cherry picked from commit 14b47990af)
2018-04-27 12:25:57 -04:00
Billie Rinaldi 90a31dc61c YARN-8211. Yarn registry dns log finds BufferUnderflowException on port ping. Contributed by Eric Yang
(cherry picked from commit 9ab3f97089)
2018-04-27 08:00:04 -07:00
Billie Rinaldi b20f68ddaa YARN-8208. Add log statement for Docker client configuration file at INFO level. Contributed by Yesha Vora
(cherry picked from commit 914d6b5cb1)
2018-04-27 06:11:34 -07:00
Sunil G 28e63f3d88 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen.
(cherry picked from commit 71220d218d)
2018-04-27 10:42:59 +05:30
Sunil G a1a2f1ddc1 YARN-8215. ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM. Contributed by Rohith Sharma K S.
(cherry picked from commit 7fc09c4f72)
2018-04-27 10:34:29 +05:30
Sunil G e2901d97ba YARN-8205. Application State is not updated to ATS if AM launching is delayed. Contributed by Rohith Sharma K S.
(cherry picked from commit 1634de0fc1)
2018-04-27 10:28:18 +05:30
Eric Yang 2253616064 YARN-8122. Added component health monitoring for YARN service.
Contributed by Gour Saha
(cherry picked from commit 93979db840)
2018-04-26 19:19:29 -04:00
Billie Rinaldi 91b24278da YARN-8140. Improve log message when launch cmd is ran for stopped yarn service. Contributed by Eric Yang
(cherry picked from commit ed4e0676c357d5d18c15998b1f6b5d305848cb04)
2018-04-26 13:18:56 -07:00
Eric Yang b39d183d92 YARN-7939. Added support to upgrade a component instance.
Contributed by Chandni Singh

(cherry picked from commit 86ae380e55ca3b1b12744b9338d60b34e8c717a6)
2018-04-26 15:54:12 -04:00
Eric Yang 050bd4cd2a YARN-8043. Added the exception message for failed launches running under LCE.
Contributed by Shane Kumpf
2018-04-26 08:26:10 -07:00
Wangda Tan af286319fd YARN-8193. YARN RM hangs abruptly (stops allocating resources) when running successive applications. (Zian Chen via wangda)
Change-Id: Ia83dd2499ee9000b9e09ae5a932f21a13c0ddee6
(cherry picked from commit af986b442b7be0f2fad1241ca9e267a15c20bf43)
2018-04-25 22:16:07 -07:00
Sunil G 44d28692a8 YARN-8188. RM Nodes UI data table index for sorting column need to be corrected post Application tags display. Contributed by Weiwei Yang.
(cherry picked from commit 626690612cd0957316628376744a8be62f891665)
2018-04-25 14:58:07 +05:30
Wangda Tan 3043a93d46 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda)
Change-Id: I347871d672001653a3afe2e99adefd74e0d798cd
(cherry picked from commit bb3c504764f807fccba7f28298a12e2296f284cb)
2018-04-24 17:44:58 -07:00
Eric Yang 678e599879 YARN-8196. Updated documentation for enabling YARN Service REST API.
Contributed by Billie Rinaldi

(cherry picked from commit f64501fcdc9dfa2e9848db0fb4749c6bd4a54d7f)
2018-04-24 19:17:15 -04:00
Junping Du 9209ebae18 YARN-7598. Document how to use classpath isolation for aux-services in YARN. Contributed by Xuan Gong.
(cherry picked from commit 56788d759f)
2018-04-24 18:30:58 +08:00
Robert Kanter ea7ad50499 HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter)
(cherry picked from commit 7ab08a9c37)
2018-04-23 16:25:41 -07:00
Sunil G 74f181e5dd YARN-8177. Documentation changes for auto creation of Leaf Queues with node label. Contributed by Suma Shivaprasad.
(cherry picked from commit 83e5f25d5e)
2018-04-23 13:37:13 +05:30
Sunil G f53e5e000a YARN-7956. [UI2] Avoid duplicating Components link under Services/<ServiceName>/Components. Contributed by Yesha Vora.
(cherry picked from commit b6d94fad08)
2018-04-23 13:32:54 +05:30
Eric Yang 5556cf397c YARN-8064. Bugfix for clean up container-executor cmd helper file.
Contributed by Eric Badger
2018-04-20 19:12:06 -04:00
Konstantinos Karanasos 18c86a3fb6 YARN-8111. Simplify PlacementConstraints API by removing allocationTagToIntraApp. Contributed by Weiwei Yang.
(cherry picked from commit 28e2244390)
2018-04-20 12:25:08 -07:00
Jason Lowe 71b0d52987 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie
(cherry picked from commit 766544c0b0)
2018-04-20 13:23:46 -05:00
Sunil G 25b91fa98d YARN-7830. [UI2] Post selecting grid view in Attempt page, attempt info page should also be opened with grid view. Contributed by Gergely Novák.
(cherry picked from commit 55e4252fc5)
2018-04-20 20:05:47 +05:30
Rohith Sharma K S 71661cec77 YARN-8182. [UI2] Proxy- Clicking on nodes under Nodes HeatMap gives 401 error. Contributed by Sunil G.
(cherry picked from commit 87e09859f1)
2018-04-20 19:44:15 +05:30
Rohith Sharma K S 3f5b4276e5 YARN-8189. [UI2] Nodes page column headers are half truncated. Contributed by Sunil G.
(cherry picked from commit b0753cc2c0)
2018-04-20 19:44:06 +05:30
Sunil G 162db6e826 YARN-6827. [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. Contributed by Rohith Sharma K S.
(cherry picked from commit 7d06806dfd)
2018-04-20 00:06:44 +05:30
Sunil G aa64def228 YARN-5888. [UI2] Improve unit tests for new YARN UI. Contributed by Akhil PB.
(cherry picked from commit c6d7d3eb05)
2018-04-19 22:00:34 +05:30
Weiwei Yang c3ca10c22a YARN-7734. Fix UT failure TestContainerLogsPage#testContainerLogPageAccess. Contributed by Tao Yang.
(cherry picked from commit 411993f6e5)
2018-04-18 19:40:34 -07:00
Shashikant Banerjee a13b3edfd8
HADOOP-15396. Some java source files are executable
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247)
2018-04-19 08:36:32 +09:00
Rohith Sharma K S 2e8053600e YARN-8171. [UI2] AM Node link from attempt page should not redirect to new tab. Contributed by Sunil G.
(cherry picked from commit 034da8fc97)
2018-04-18 13:40:42 +05:30
Rohith Sharma K S fa007d986b YARN-8145. yarn rmadmin -getGroups doesn't return updated groups for user. Contributed by Sunil G.
(cherry picked from commit 241d985467)
2018-04-18 13:40:32 +05:30
Jason Lowe 42aef3b655 YARN-7996. Allow user supplied Docker client configurations with YARN native services. Contributed by Shane Kumpf
(cherry picked from commit 1d6e43da51)
2018-04-17 11:32:58 -05:00
Inigo Goiri 8f300c9ce1 YARN-8165. Incorrect queue name logging in AbstractContainerAllocator. Contributed by Weiwei Yan.
(cherry picked from commit dd5e18c4ae)
2018-04-17 15:28:13 +05:30
Sunil G d5adc59017 YARN-8126. Support auto-spawning of admin configured services during bootstrap of RM. Contributed by Rohith Sharma K S.
(cherry picked from commit 427ad7ecc4)
2018-04-17 15:25:31 +05:30
Eric Yang 2911b5406b YARN-8060. Added default readiness check for service components.
Contributed by Billie Rinaldi

(cherry picked from 2d0662cfd5)
2018-04-16 14:53:42 -04:00
Eric Yang 20453488e8 YARN-8032. Added ability to configure failure validity interval for YARN service.
Contributed by Chandni Singh

(cherry-picked from 647058efc0)
2018-04-16 14:52:09 -04:00
Eric Yang 148322ca72 YARN-8142. Improve SIGTERM handling for YARN Service Application Master.
Contributed by Billie Rinaldi

(cherry picked from commit 9031a76d44)
2018-04-14 12:10:37 -07:00
Wangda Tan 6ee62e6b1c YARN-8138. Add unit test to validate queue priority preemption works under node partition. (Zian Chen via wangda)
Change-Id: Ibebfab98a714c12c2dc643b6d7b9754a7f813632
2018-04-14 11:04:49 -07:00
Wangda Tan 0b88683627 YARN-7142. Support placement policy in yarn native services. (Gour Saha via wangda)
Change-Id: I166c67a7a34430627c17365f60bac75b6da1b434
(cherry picked from commit a0bde7d525)
2018-04-14 11:01:59 -07:00
Rohith Sharma K S eb1026dffa YARN-8156. Increase the default value of yarn.timeline-service.app-collector.linger-period.ms. Contributed by Charan Hebri.
(cherry picked from commit 669eb7bdea)
2018-04-14 10:57:02 +05:30
Wangda Tan 994c7d66e0 YARN-8091. Revisit checkUserAccessToQueue RM REST API. (wangda)
Change-Id: I5fab3fe229c34e967487b7327c7b3c8ddf7cb795
2018-04-13 10:19:10 -07:00
Wangda Tan 7031a853f4 YARN-8048. Support auto-spawning of admin configured services during bootstrap of RM (Rohith Sharma K S via wangda)
Change-Id: I2d8d61ccad55e1118009294d7e17822df3cd0fd5
2018-04-13 10:18:48 -07:00
Sunil G d98c3ca5fd YARN-8040. [UI2] New YARN UI webapp does not respect current pathname for REST api. Contributed by Sunil G. 2018-04-13 10:18:39 -07:00
Sunil G 21717db6a0 YARN-8028. Support authorizeUserAccessToQueue in RMWebServices. Contributed by Wangda Tan. 2018-04-13 10:18:12 -07:00
Wangda Tan c962371430 YARN-1151. Ability to configure auxiliary services from HDFS-based JAR files. (Xuan Gong via wangda)
Change-Id: Ied37ff11e507fc86847753ba79486652c8fadfe9
2018-04-13 10:17:47 -07:00
Weiwei Yang 84531ad9b6 YARN-8153. Guaranteed containers always stay in SCHEDULED on NM after restart. Contributed by Yang Wang. 2018-04-13 13:29:43 +08:00
Weiwei Yang 7be71ec55b YARN-8154. Fix missing titles in PlacementConstraints document. Contributed by Weiwei Yang. 2018-04-13 13:10:05 +08:00
Eric Yang a375fe8263 YARN-7984. Improved YARN service stop/destroy and clean up.
Contributed by Billie Rinaldi

(cherry picked from commit d553799030)
2018-04-12 18:19:58 -07:00
Eric Yang 150085cc64 YARN-8018. Added support for initiating yarn service upgrade.
Contributed by Chandni Singh

(cherry picked from commit 27d60a1634)
2018-04-12 17:08:27 -04:00
Billie Rinaldi 23179c06a3 YARN-7936. Add default service AM Xmx. Contributed by Jian He
(cherry picked from commit 53b3e59473)
2018-04-12 12:43:30 -07:00
Eric E Payne 9693d8b99d YARN-8147. TestClientRMService#testGetApplications sporadically fails. Contributed by Jason Lowe
(cherry picked from commit 18844599ae)
2018-04-12 17:58:41 +00:00
Eric E Payne f955d52f39 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe.
(cherry picked from commit 6bb128dfb8)
2018-04-12 16:11:57 +00:00
Billie Rinaldi 11f1d49823 YARN-7221. Add security check for privileged docker container. Contributed by Eric Yang
(cherry picked from commit 933477e9e0)
2018-04-11 12:15:00 -07:00
Weiwei Yang c2036af196 YARN-8127. Resource leak when async scheduling is enabled. Contributed by Tao Yang.
(cherry picked from commit 7eb783e263)
2018-04-11 17:21:47 +08:00
Vrushali C 67cd7acf99 YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S
(cherry picked from commit 345e7624d5)
2018-04-11 09:41:04 +05:30
Rohith Sharma K S c1b4c6adf4 YARN-7941. Transitive dependencies for component are not resolved. Contributed by Billie Rinaldi.
(cherry picked from commit c048711099)
2018-04-11 09:19:48 +05:30
Wangda Tan 53abef4e30 YARN-8133. Doc link broken for yarn-service from overview page. (Rohith Sharma K S via wangda)
Change-Id: Iacf9a004585dd59e1c0cd8f8c618a38f047cc0fe
(cherry picked from commit d919eb6efa)
2018-04-10 17:38:37 -07:00
Wangda Tan f5dc7d1edf YARN-8116. Nodemanager fails with NumberFormatException: For input string: . (Chandni Singh via wangda)
Change-Id: Idd30cfca59982d3fc6e47aa1b88f844a78fae94d
(cherry picked from commit 2bf9cc2c73)
2018-04-10 17:38:31 -07:00
Sunil G 5387b3f066 YARN-7804. [UI2] Refresh action on Grid view page should not be redirected to graph view. Contributed by Gergely Novák.
(cherry picked from commit 7c1e77dda4)
2018-04-10 16:09:49 +05:30
Wangda Tan 25bf1d3ac6 Updated branch-3.1 to track 3.1.1
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
2018-04-09 10:58:50 -07:00
Sunil G 8311fcc75b YARN-7574. Add support for Node Labels on Auto Created Leaf Queue Template. Contributed by Suma Shivaprasad.
(cherry picked from commit 821b0de4c5)
2018-04-09 21:17:56 +05:30
bibinchundatt 091db4d0eb YARN-7905. Parent directory permission incorrect during public localization. Contributed by Bilwa S T.
(cherry picked from commit eb47c3de74)
2018-04-08 14:49:36 +05:30
Subru Krishnan b9bad94596 YARN-8110. AMRMProxy recover should catch for all throwable to avoid premature exit. (Botong Huang via Subru).
(cherry picked from commit 00905efab2)
2018-04-06 16:35:29 -07:00
Haibo Chen b232dcab33 YARN-8107. Give an informative message when incorrect format is used in ATSv2 filter attributes. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit 024d7c0870)
2018-04-06 09:39:45 -07:00
Sunil G 58a5ee378a YARN-8083. [UI2] All YARN related configurations are paged together in conf page. Contributed by Gergely Novák.
(cherry picked from commit b17dc9f5f5)
2018-04-06 21:53:49 +05:30
Wangda Tan a336561e2b Added CHANGES/RELEASES/Jdiff for 3.1.0 release
Change-Id: Ied5067a996151c04d15cad46c46ac98b60c37b39
(cherry picked from commit 2d96570452a72569befdf9cfe9b90c9fa2e0e261)
2018-04-05 15:52:02 -07:00
Haibo Chen 6658018410 YARN-6936. [Atsv2] Retrospect storing entities into sub application table from client perspective. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit f8b8bd53c4)
2018-04-05 11:31:17 -07:00
Rohith Sharma K S b11c228660 YARN-8119. [UI2] Timeline Server address' url scheme should be removed while accessing via KNOX. Contributed by Sunil G.
(cherry picked from commit f32d6275ba)
2018-04-05 23:34:40 +05:30
Konstantinos Karanasos fa464c12ad YARN-8013. Support application tags when defining application namespaces for placement constraints. Contributed by Weiwei Yang.
(cherry picked from commit 7853ec8d2f)
2018-04-04 10:53:33 -07:00
Sunil G 4db13cb944 YARN-8115. [UI2] URL data like nodeHTTPAddress must be encoded in UI before using to access NM. Contributed by Sreenath Somarajapuram.
(cherry picked from commit 42cd367c93)
2018-04-04 22:24:31 +05:30
Wangda Tan 2f326159f1 YARN-8106. Update LogAggregationIndexedFileController to use readFull instead read to avoid IOException while loading log meta. (Prabhu Joseph via wangda)
Change-Id: I63a65f73f8d1636e2c99ed9c8c2bbd05efcff80f
(cherry picked from commit b779f4f0f6)
2018-04-03 21:15:37 -07:00
Sunil G 1a125dd92f YARN-7764. Findbugs warning: Resource#getResources may expose internal representation. Contributed by Weiwei Yang.
(cherry picked from commit f7a17b029d)
2018-04-04 09:23:39 +05:30
Wangda Tan 78832eca8a YARN-8002. Support NOT_SELF and ALL namespace types for allocation tag. (Weiwei Yang via wangda)
Change-Id: I63b4e4192a95bf7ded98c54e46a2871c72869700
2018-04-04 09:42:49 +08:00
Wangda Tan b64bf56131 YARN-8086. ManagedParentQueue with no leaf queues cause JS error in new UI. (Suma Shivaprasad via wangda)
Change-Id: I6d82af91adec02698afddde6883b1fe1924e6041
(cherry picked from commit e7e20190df)
2018-03-29 15:19:31 -07:00
Sunil G 73cd67d7e3 YARN-8085. ResourceProfilesManager should be set in RMActiveServiceContext. Contributed by Tao Yang.
(cherry picked from commit 7a59d60e0c)
2018-03-29 21:41:52 +05:30
Sunil G 04161bad78 YARN-8076. Support to specify application tags in distributed shell. Contributed by Weiwei Yang.
(cherry picked from commit 431076f637)
2018-03-29 16:02:49 +05:30
Eric Yang 570d7f3072 YARN-8069. Clean up example hostnames for RegistryDNS.
Contributed by Billie Rinaldi

(cherry picked from commit 3d185d62fc)
2018-03-28 20:20:06 -04:00
Zhe Zhang 0db6b8c93c YARN-7623. Fix the CapacityScheduler Queue configuration documentation. Contributed by Jonathan Hung.
(cherry picked from commit 0b1c2b5fe1)
2018-03-28 11:42:16 -07:00
Subru Krishnan a0091ec4b3 YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru).
(cherry picked from commit 09999d7e01)
2018-03-28 11:35:45 -07:00
Subru Krishnan ed49f59e18 Revert "YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru)."
This reverts commit 85eebf1beb.
2018-03-28 11:35:30 -07:00
Wangda Tan a2b4daab55 YARN-6629. NPE occurred when container allocation proposal is applied but its resource requests are removed before. (Tao Yang via wangda)
Change-Id: I805880f90b3f6798ec96ed8e8e75755f390a9ad5
(cherry picked from commit 47f711eebc)
2018-03-28 11:22:26 -07:00
Sunil G 6c904388f4 YARN-8075. DShell does not fail when we ask more GPUs than available even though AM throws 'InvalidResourceRequestException'. Contributed by Wangda Tan.
(cherry picked from commit d1e378d02b)
2018-03-28 08:03:59 +05:30
Subru Krishnan 85eebf1beb YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru).
(cherry picked from commit 2a2ef15caf)
2018-03-27 17:44:00 -07:00
Wangda Tan dc99611878 YARN-8072. RM log is getting flooded with MemoryPlacementConstraintManager info logs. (Zian Chen via wangda)
Change-Id: I2567f4669ad868e13c4301fe44c4bde85bb92c37
(cherry picked from commit 24bc5e0e4d)
2018-03-26 11:27:40 -07:00
Wangda Tan 62b9537332 YARN-8068. Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ (Sunil G via wangda)
Change-Id: I7910bd1064a1b4dbbe2084080c060822ea6f3b48
(cherry picked from commit 9eef19b2ad)
2018-03-26 11:27:32 -07:00
Wangda Tan 54da6a0d78 YARN-8062. yarn rmadmin -getGroups returns group from which the user has been removed. (Sunil G via wangda)
Change-Id: I80ed63846502bf7751b890b6c6c6a7c0679e2b4a
(cherry picked from commit 5d381570f8)
2018-03-26 11:27:21 -07:00
Wangda Tan c98dd3bf18 YARN-8070. Yarn Service API site doc broken due to unwanted character in YarnServiceAPI.md (Gour Saha via wangda)
Change-Id: I22428b2f128d16e79ebbdeaf32869566963d0c6f
(cherry picked from commit 24f75e097a)
2018-03-23 17:53:33 -07:00
Rohith Sharma K S 91791a8abd YARN-7986. ATSv2 REST API queries do not return results for uppercase application tags. Contributed by Charan Hebri.
(cherry picked from commit 75fc05f369)
2018-03-23 12:18:43 +05:30
Sunil G e10e6c9012 YARN-8063. DistributedShellTimelinePlugin wrongly check for entityId instead of entityType. Contributed by Rohith Sharma K S.
(cherry picked from commit 22c5ddb7c4)
2018-03-23 10:55:13 +05:30
Jason Lowe 5d4b2c31a1 YARN-8054. Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread. Contributed by Jonathan Eagles
(cherry picked from commit 5aa7052e31)
2018-03-21 17:20:36 -05:00
Rohith Sharma K S bbfe36d686 YARN-7581. HBase filters are not constructed correctly in ATSv2. Contributed by Habio Chen.
(cherry picked from commit 29acea5000)
2018-03-21 08:26:56 +05:30
Haibo Chen d8764f1c90 YARN-8053. Add hadoop-distcp in exclusion in hbase-server dependencies for timelineservice-hbase packages. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit 3ff6977d3e)
2018-03-21 08:26:56 +05:30
Jason Lowe a4f0dbcb44 YARN-8027. Setting hostname of docker container breaks for --net=host in docker 1.13. Contributed by Jim Brennan
(cherry picked from commit f480367af6)
2018-03-19 17:44:06 -05:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 0bd99621c5 Preparing for 3.1.0 development 2018-03-19 15:16:54 -07:00
Yufei Gu 49c747ab18 YARN-8039. Clean up log dir configuration in TestLinuxContainerExecutorWithMocks.testStartLocalizer. Contributed by Miklos Szegedi. 2018-03-16 16:25:33 -07:00
Weiwei Yang 154cfb2b62 YARN-7636. Re-reservation count may overflow when cluster resource exhausted for a long time. contributed by Tao Yang. 2018-03-16 18:57:31 +08:00
Wangda Tan 4bf622043f YARN-7952. RM should be able to recover log aggregation status after restart/fail-over. (Xuan Gong via wangda)
Change-Id: I725c9afe64831eda0aa6b0bebdbc79d2dd165707
2018-03-15 13:26:45 -07:00
fang zhenyi 5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-15 16:12:07 +09:00
Eric Yang a82be7754d YARN-7999. Added file cache initializer for Linux container-executor.
Contributed by Jason Lowe
2018-03-14 12:21:57 -04:00
Rohith Sharma K S e6de10d0a6 YARN-8022. ResourceManager UI cluster/app/<app-id> page fails to render. Contributed by Tarun Parimi. 2018-03-14 09:42:20 +05:30