Commit Graph

5195 Commits

Author SHA1 Message Date
Weiwei Yang ac2f4b64f9 YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang.
(cherry picked from commit abde1e1f58)
2019-01-21 21:20:01 +08:00
Wangda Tan fe7cb2d84a YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda)
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
(cherry picked from commit 6d7eedfd28)
2019-01-17 15:13:42 -08:00
rahul3 4aee7946dc
YARN-9203. Fix typos in yarn-default.xml.
This closes #437

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0a46baecd3)
2019-01-17 16:19:49 +09:00
Akira Ajisaka a415446705
YARN-8747. [UI2] YARN UI2 page loading failed due to js error under some time zone configuration. Contributed by collinma.
(cherry picked from commit 104ef5df36)
2019-01-16 14:37:43 +09:00
Wangda Tan 1dc2b49bfd YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda)
Change-Id: Ib8044307a4241f6b1b7b9b8266b9256f39b16384
2019-01-07 12:21:33 -08:00
Eric Yang bba76b6f31 HADOOP-16031. Fixed TestSecureLogins unit test. Contributed by Akira Ajisaka 2019-01-07 13:24:01 -05:00
Weiwei Yang 2b549e32e1 YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed by Wilfred Spiegelenburg.
(cherry picked from commit 944cf87223)
2019-01-07 16:05:57 +08:00
Wangda Tan cffe5c1ba0 YARN-9160. [Submarine] Document 'PYTHONPATH' environment variable setting when using -localization options. (Zhankun Tang via wangda)
Change-Id: I115b6710eb11501e21ab3e9e7cd1a87f322e5cf8
(cherry picked from commit 2c02aa6ec2)
2019-01-06 11:10:47 -08:00
Wangda Tan 1ad1ab5d7a YARN-9141. [Submarine] JobStatus outputs with system UTC clock, not local clock. (Zac Zhou via wangda)
Change-Id: Ib2b17cf7860ddc9188f06498086de3326bc09519
(cherry picked from commit 8f004fe4e0)
2019-01-06 11:10:41 -08:00
Weiwei Yang a24cca11f2 YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie.
(cherry picked from commit cfe89e6f96)
2019-01-04 01:04:39 +08:00
Sunil G 9bfea98746 YARN-9090. [Submarine] Update submarine installation script document. Contributed by Xun Liu.
(cherry picked from commit 742fd978b5)
2018-12-26 08:27:50 +05:30
Akira Ajisaka d6baf9d5bf
YARN-9154. Fix itemization in YARN service quickstart document. Contributed by Ayush Saxena.
(cherry picked from commit e98e963e51)
2018-12-22 13:26:41 +09:00
Weiwei Yang 7deef08eb8 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. 2018-12-21 16:31:03 +08:00
Eric Yang 29c9c8a893 YARN-9126. Fix container clean up for reinitialization.
Contributed by Chandni Singh

(cherry picked from commit e815fd9c49)
2018-12-19 14:58:19 -05:00
Eric Yang 4e1cef3625 YARN-9084. Reset container state and defer readiness check for upgrade.
Contributed by Chandni Singh

(cherry picked from commit ccdd982e51)
2018-12-18 18:05:40 -05:00
Eric Yang 28ca14e71b YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator.
Contributed by Tarun Parimi

(cherry picked from commit 71e0b0d800)
2018-12-17 12:08:09 -05:00
Akira Ajisaka 71d8ffb551
YARN-9122. Add table of contents to YARN Service API document. Contributed by Zhankun Tang.
(cherry picked from commit 26e55d4ceb)
2018-12-15 10:30:37 +09:00
Eric Yang 52aafb9789 YARN-9125. Fixed Carriage Return detection in Docker container launch command.
Contributed by Billie Rinaldi

(cherry picked from commit b2d7204ed0)
2018-12-14 17:55:10 -05:00
Wangda Tan 29e4e5f62c YARN-9112. [Submarine] Support polling applicationId when it's not ready in cluster. (Zhankun Tang via wangda)
Change-Id: I73d73f3d631b28fb9866faa56571839b13824a97
(cherry picked from commit 9fba6cc247)
2018-12-12 11:52:16 -08:00
Wangda Tan ef0f835537 YARN-8714. [Submarine] Support files/tarballs to be localized for a training job. (Zhankun Tang via wangda)
Change-Id: I845131273e52a9d81dbc813ea6d4af06b205e334
(cherry picked from commit c771fe6e10)
2018-12-12 11:52:07 -08:00
Weiwei Yang 2b3c3d2a32 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters.
(cherry picked from commit 1c09a10e96)
2018-12-10 12:07:23 +08:00
Jonathan Hung 3ab6ea7aca YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics
(cherry picked from commit 978ab3e958227220cb6f1a08ae6e7cdb8a46628b)
2018-12-07 10:45:47 -08:00
Eric Yang 8c70728f7f YARN-9071. Improved status update for reinitialized containers.
Contributed by Chandni Singh

(cherry picked from commit 1b790f4dd1)
2018-12-05 19:04:55 -05:00
Wangda Tan 559cb99bf6 Revert "YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)"
This reverts commit 920401b296.
2018-12-04 14:13:39 -08:00
Jonathan Hung 6b01e4d2a8 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu
(cherry picked from commit 1c8bd7128c99d8215ef16438bd2ce6b1f025a966)
2018-11-30 10:16:00 -08:00
Eric Yang a8f67ad7c9 YARN-9067. Fixed Resource Manager resource leak via YARN service.
Contributed by Eric Yang
2018-11-29 13:50:06 -05:00
bibinchundatt 183ec39c4b YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
(cherry picked from commit 07142f54a8)
2018-11-29 22:16:32 +05:30
Wangda Tan ee9deb6e9c YARN-9030. Log aggregation changes to handle filesystems which do not support setting permissions. (Suma Shivaprasad via wangda)
Change-Id: I80f1e8196b8624e24d74494719fdedfd7061dced
(cherry picked from commit 9de8e8d049)
2018-11-28 15:06:35 -08:00
Jason Lowe df0e7766e4 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko
(cherry picked from commit 3ce99e32f7)
2018-11-28 08:50:18 -06:00
Eric Yang db8b2a130c YARN-9056. Improved YARN service upgrade state logic for readiness check.
Contributed by Chandni Singh

(cherry picked from commit f657a2a661)
2018-11-27 19:45:23 -05:00
Eric Yang 838190482d YARN-8986. Added port publish for Docker container running with bridge.
Contributed by Charo Zhang
2018-11-27 14:27:13 -05:00
Weiwei Yang 01477108d9 YARN-8984. AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty. Contributed by Yang Wang.
(cherry picked from commit 176bb3f812)
2018-11-22 17:04:16 +08:00
Wangda Tan 34387599c4 YARN-8960. [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment. (Zac Zhou via wangda)
Change-Id: I21b1addc9c32817650ea744a8f2e6b5602f2f4d4
(cherry picked from commit 8b23814415)
2018-11-19 09:01:21 -08:00
Weiwei Yang 650581a19d YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun.
(cherry picked from commit d027a24f03)
2018-11-18 23:23:09 +08:00
Wangda Tan bee5bf867b YARN-9001. [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs. (Zac Zhou via wangda)
Change-Id: I7e8d1c27ebd37e0907ca570c4f3d56fe7a859635
2018-11-16 10:26:50 -08:00
Rohith Sharma K S 13e3670e7f YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report.
(cherry picked from commit ee3355be3c)
2018-11-16 18:37:20 +05:30
Vrushali C da05328824 YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader. Contributed by Abhishek Modi
(cherry picked from commit a3edfddcf7)
2018-11-16 18:37:01 +05:30
Wangda Tan ea753046b0 YARN-8918. [Submarine] Correct method usage of str.subString in CliUtils. (Zhankun Tang via wangda)
Change-Id: Id1f11dbab3aa838dee3a0ec8b8fd5dc32f5dd946
(cherry picked from commit 076b795b2e)
2018-11-13 11:35:06 -08:00
Weiwei Yang 2415f8a5be YARN-8987. Usability improvements node-attributes CLI. Contributed by Bibin A Chundatt.
(cherry picked from commit c741109522)
2018-11-12 18:21:11 +08:00
Eric Yang 5399ff8177 YARN-9002. Improve keytab loading for YARN Service.
Contributed by Gour Saha

(cherry picked from commit 2664248797)
2018-11-10 01:54:59 -05:00
Weiwei Yang b3321d9933 YARN-8988. Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled. Contributed by Tao Yang.
(cherry picked from commit e1bbf7dcdf)
2018-11-08 17:50:16 +08:00
Weiwei Yang b10ec0aa14 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji.
(cherry picked from commit c96cbe8659)
2018-11-07 22:44:01 +08:00
Takanobu Asanuma 02abd0f80a YARN-8866. Fix a parsing error for crossdomain.xml.
(cherry picked from commit 8dc1f6dbf7)
2018-11-07 18:29:28 +09:00
Weiwei Yang 4ff7255d6e YARN-8976. Remove redundant modifiers in interface ApplicationConstants. Contributed by Zhankun Tang.
(cherry picked from commit 482716e5a4)
2018-11-07 10:49:09 +08:00
Akira Ajisaka 41c4e9583d
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang.
(cherry picked from commit 951c98f890)
2018-11-07 11:18:48 +09:00
Jason Lowe 9265934201 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
(cherry picked from commit ab6aa4c726)
2018-11-06 08:47:30 -06:00
Weiwei Yang f00125e2d9 YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang.
(cherry picked from commit 5d6554c722)
2018-11-06 14:52:10 +08:00
Weiwei Yang b474239e0b YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji.
(cherry picked from commit c7fcca0d7e)
2018-11-06 13:17:19 +08:00
Jonathan Hung b9a3c988c0 YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne
(cherry picked from commit 2ab611d48b)
2018-11-01 14:31:09 -07:00
Rohith Sharma K S 7920f79049 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
(cherry picked from commit 4ec4ec6971)
2018-10-30 11:50:33 +05:30
Weiwei Yang 288bc3f1eb YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang.
(cherry picked from commit 78ea897b67)
2018-10-29 12:05:45 +08:00
Weiwei Yang 5db4272d91 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
(cherry picked from commit 1d90a0dd23)
2018-10-29 11:55:22 +08:00
Eric Yang 3665c46176 YARN-8938. Updated YARN service upgrade document.
Contributed by Chandni Singh

(cherry picked from commit bbc6dcd3d0)
2018-10-24 11:52:50 -04:00
Jason Lowe 7097755925 YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg
(cherry picked from commit 93fb3b4b9c)
2018-10-23 12:54:04 -05:00
Rohith Sharma K S 660fff3138 YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
(cherry picked from commit 0b62983c5a)
2018-10-23 14:06:13 +05:30
Sunil G 30998fea28 YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh.
(cherry picked from commit 2202e00ba8)
2018-10-23 09:57:26 +05:30
Sunil G 8724c2900f YARN-7502. Nodemanager restart docs should describe nodemanager supervised property. Contributed by Suma Shivaprasad.
(cherry picked from commit e50334513c)
2018-10-22 14:16:24 +05:30
Eric Yang 04600cf5da YARN-8910. Fixed misleading log statement when container max retries is infinite.
Contributed by Chandni Singh

(cherry picked from commit 47ad98b2e1)
2018-10-19 13:49:51 -04:00
Wangda Tan 9ed9e185d7 YARN-8916. Define a constant docker string in ContainerRuntimeConstants.java for better maintainability. (Zhankun Tang via wangda)
Change-Id: I1349e740037f81afdbe30edbe741f20e88fd0a90
(cherry picked from commit 5e02b4915b)
2018-10-19 09:50:18 -07:00
Wangda Tan bd08554aa4 YARN-8908. Fix errors in yarn-default.xml related to GPU/FPGA. (Zhankun Tang via wangda)
Change-Id: I46f511a9b51b65b069e15d5b827530110c41edab
(cherry picked from commit 071b9e19aa)
2018-10-19 09:50:12 -07:00
Weiwei Yang 042f2df19b YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang.
(cherry picked from commit 13cc0f50ea)
2018-10-19 10:01:45 +08:00
Eric Yang 91c916e209 YARN-8618. Added detection for non-upgradable service.
Contributed by Chandni Singh

(cherry picked from commit 66f059ed1d)
2018-10-18 20:00:50 -04:00
Wangda Tan 7252f8e117 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07)
2018-10-18 12:12:19 -07:00
Wangda Tan 5f7ed043c8 YARN-8456. Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable. (Zhankun Tang via wangda)
Change-Id: Iff150ea98ba0c60d448474fd940eb121afce6965
(cherry picked from commit a457a8951a)
2018-10-18 12:12:14 -07:00
Wangda Tan a7785d2764 YARN-6098. Add documentation for Delete Queue. (Suma Shivaprasad via wangda)
Change-Id: I23bf7ca0acbe1168a7f805ceccdd62fe41bfeb35
(cherry picked from commit bfb88b10f4)
2018-10-18 12:12:09 -07:00
Eric Yang 4681e17a36 YARN-8687. Update YARN service file type in documentation.
Contributed by Suma Shivaprasad
2018-10-18 12:04:04 -04:00
Sunil G bde4fd5ed9 Preparing for 3.2.0 release 2018-10-18 17:07:45 +05:30
Sunil G 6380ee5512 YARN-8759. Copy of resource-types.xml is not deleted if test fails, causes other test failures. Contributed by Antal Bálint Steinbach.
(cherry picked from commit 5085e5fa9e)
2018-10-17 16:06:07 +05:30
Eric Yang fc25def7d7 YARN-8810. Fixed a YARN service bug in comparing ConfigFile object.
Contributed by Chandni Singh
2018-10-16 18:57:50 -04:00
Wangda Tan 80ee60c88c YARN-8892. YARN UI2 doc changes to update security status (verified under security environment). (Sunil G via wangda)
Change-Id: I8bc8622936861b8d6de3e42a0b75af86ad8a3961
(cherry picked from commit 538250db26)
2018-10-16 13:53:08 -07:00
Wangda Tan aa23c8d739 YARN-8798. [Submarine] Job should not be submitted if --input_path option is missing. (Zhankun Tang via wangda)
Change-Id: I7ae0e44eb5179b04a6ac861ec1c65f3b18c38f0f
(cherry picked from commit 143d74775b)
2018-10-16 13:53:03 -07:00
Wangda Tan 920401b296 YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)
Change-Id: I46e8d9fd32c7745c313030da62da41486a77b3ea
(cherry picked from commit 46d6e00166)
2018-10-16 13:52:58 -07:00
Wangda Tan 8610c20724 YARN-8875. [Submarine] Add documentation for submarine installation script details. (Xun Liu via wangda)
Change-Id: I1c8d39c394e5a30f967ea514919835b951f2c124
(cherry picked from commit ed08dd3b0c)
2018-10-16 13:52:53 -07:00
Sunil G a439aa79dc YARN-8879. Kerberos principal is needed when submitting a submarine job. Contributed by Zac Zhou.
(cherry picked from commit 753f149fd3)
2018-10-16 22:18:30 +05:30
Sunil G 1ca5f974f1 YARN-8869. YARN Service Client might not work correctly with RM REST API for Kerberos authentication. Contributed by Eric Yang.
(cherry picked from commit fa94d370b6)
2018-10-15 21:22:45 +05:30
Vrushali C 62d329cac0 YARN-5742 Serve aggregated logs of historical apps from timeline service. Contributed by Rohith Sharma KS
(cherry picked from commit 8d1981806f)
2018-10-12 07:18:13 +05:30
Jason Lowe cdbca8b133 YARN-8861. executorLock is misleading in ContainerLaunch. Contributed by Chandni Singh
(cherry picked from commit e787d65a08)
2018-10-11 10:58:48 -05:00
Sunil G f29a7fdb14 YARN-8753. [UI2] Lost nodes representation missing from Nodemanagers Chart. Contributed by Yesha Vora.
(cherry picked from commit 8a37983990)
2018-10-11 16:16:05 +05:30
Sunil G 0bf0fb5499 YARN-8666. [UI2] Remove application tab from YARN Queue Page. Contributed by Yesha Vora.
(cherry picked from commit d3da741fa2)
2018-10-11 16:07:08 +05:30
Jason Lowe 145c7aa663 YARN-7644. NM gets backed up deleting docker containers. Contributed by Chandni Singh
(cherry picked from commit 5ce70e1211)
2018-10-10 10:01:52 -05:00
Weiwei Yang eb0147a4c7 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan.
(cherry picked from commit edce866489)
2018-10-10 09:42:45 +08:00
Eric Yang 614d6cf99c YARN-8845. Removed unused hadoop.registry.rm reference.
Contributed by Íñigo Goiri

(cherry picked from commit bf04f19456)
2018-10-09 15:56:04 -04:00
Wangda Tan 8e8b748724 YARN-8852. Add documentation for submarine installation details. (Zac Zhou via wangda)
Change-Id: If5681d1ef37ff5dc916735eeef15a6120173d653
(cherry picked from commit a23ea68b97)
2018-10-09 10:19:16 -07:00
Sunil G f6a73d181f YARN-8853. [UI2] Application Attempts tab is not shown correctly when there are no attempts. Contributed by Akhil PB.
(cherry picked from commit 6f11919beb)
2018-10-09 22:05:53 +05:30
Inigo Goiri 21553e22b6 YARN-8843. updateNodeResource does not support units for memory. Contributed by Manikandan R.
(cherry picked from commit 745f64012a)
2018-10-08 10:40:59 -07:00
Sunil G 50acba2e39 YARN-7825. [UI2] Maintain constant horizontal application info bar for Application Attempt page. Contributed by Akhil PB.
(cherry picked from commit 228dc192f1)
2018-10-08 11:55:50 +05:30
Vidura Mudalige 843e7f81fe
YARN-8819. Fix findbugs warnings in YarnServiceUtils
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b85ee0d75d)
2018-10-07 00:55:08 +09:00
Sunil G a7711073ac YARN-8797. [UI2] Improve error pages in new YARN UI. Contributed by Akhil PB.
(cherry picked from commit 94d82f4162)
2018-10-05 14:33:47 +05:30
Sunil G 8957a61ed9 YARN-7957. [UI2] YARN service delete option disappears after stopping application. Contributed by Akhil PB.
(cherry picked from commit 751f626e50)
2018-10-05 12:29:08 +05:30
Wangda Tan b3ac886933 YARN-8844. TestNMProxy unit test is failing. (Eric Yang via wangda)
Change-Id: I241fa8701b6f1dbcad87fd2e9a429e32e7aa40f5
2018-10-04 10:48:47 -07:00
Wangda Tan abe4a8e5d8 YARN-8758. Support getting PreemptionMessage when using AMRMClientAsyn. (Zian Chen via wangda)
Change-Id: Ibf5d165f49957b582eeadeb41dc285c84d2f05e7
(cherry picked from commit 6926fd0ec6)
2018-10-04 10:43:16 -07:00
Akira Ajisaka 4de3cf1968
YARN-8840. Add missing cleanupSSLConfig() call for TestTimelineClient test. Contributed by Aki Tanaka.
(cherry picked from commit 39b35036ba)
2018-10-04 09:52:50 +09:00
Shane Kumpf 5edb9d3b97 YARN-8785. Improve the error message when a bind mount is not whitelisted. Contributed by Simon Prewo 2018-10-02 07:16:29 -06:00
Haibo Chen d0ee6fbe28 YARN-8621. Add test coverage of custom Resource Types for the apps/<appId> REST API endpoint. (Contributed by Szilard Nemeth) 2018-10-01 14:46:42 -07:00
Giovanni Matteo Fumarola 59d5af21b7 YARN-8760. [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer. Contributed by Botong Huang. 2018-10-01 13:12:38 -07:00
Weiwei Yang fd6be5898a YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach. 2018-09-29 17:47:12 +08:00
Sunil G 19ad5be651 YARN-8800. Updated documentation of Submarine with latest examples. Contributed by Wangda Tan. 2018-09-29 00:01:04 +05:30
Billie Rinaldi d1c1dde309 YARN-8734. Readiness check for remote service belonging to the same user. Contributed by Eric Yang 2018-09-28 09:40:48 -07:00
Eric E Payne 8598b498bc YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang. 2018-09-28 15:32:07 +00:00
bibinchundatt 7093afd874 YARN-8829. Cluster metrics can fail with IndexOutOfBoundsException. Contributed by Akshay Agarwal. 2018-09-28 12:35:33 +05:30