Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,930 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5330 Commits

Author SHA1 Message Date
Zhankun Tang 1e7201f9aa YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji. 
(cherry picked from commit 67414a1a80)
 2019-06-18 13:18:27 +08:00
Inigo Goiri 65f7ec2f39 YARN-8856. TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError. Contributed by Sushil Ks. 
(cherry picked from commit eeaf8edaa7)
 2019-06-13 14:22:16 -07:00
Sean Mackrory e0b3cbd221 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:53:40 -06:00
Sunil G 253dcde517 YARN-9543. [UI2] Handle ATSv2 server down or failures cases gracefully in YARN UI v2. Contributed by Zoltan Siegl and Akhil P B. 
(cherry picked from commit 52128e352a)
 2019-06-12 19:25:02 +05:30
Sunil G 72203f7a12 YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 2019-06-12 19:23:40 +05:30
Sunil G f1ead03672 Revert "YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl." 
This reverts commit f1d3a17d3e.
 2019-06-12 19:10:23 +05:30
bibinchundatt 3303723f55 YARN-9547. ContainerStatusPBImpl default execution type is not returned. Contributed by Bilwa S T. 2019-06-11 23:42:29 +05:30
bibinchundatt d9284d4a57 YARN-9565. RMAppImpl#ranNodes not cleared on FinalTransition. Contributed by Bilwa S T. 
(cherry picked from commit 60c95e9b6a)
 2019-06-11 23:13:18 +05:30
bibinchundatt a37011bd5e YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 
(cherry picked from commit 6d80b9bc3f)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java
 2019-06-11 23:01:03 +05:30
Eric Yang 68aec0a98d YARN-9581. Fixed yarn logs cli to access RM2. 
Contributed by Prabhu Joseph

(cherry picked from commit cb9bc6e64c)
 2019-06-06 16:43:25 -04:00
Sunil G f1d3a17d3e YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl. 2019-06-06 06:24:01 +05:30
Weiwei Yang 6e2b091515 YARN-9580. Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers. Contributed by Tao Yang. 2019-06-04 15:24:37 +08:00
Sunil G 2f012044ff YARN-8906. [UI2] NM hostnames not displayed correctly in Node Heatmap Chart. Contributed by Akhil PB. 
(cherry picked from commit 59719dc560)
 2019-06-03 15:54:07 +05:30
Sunil G 58042dadc3 YARN-8947. [UI2] Active User info missing from UI2. Contributed by Akhil PB. 
(cherry picked from commit 7f46dda513)
 2019-06-03 12:25:16 +05:30
Weiwei Yang e027c87da2 YARN-9507. Fix NPE in NodeManager#serviceStop on startup failure. Contributed by Bilwa S T. 
(cherry picked from commit 4530f4500d)
 2019-06-03 14:15:20 +08:00
Eric Yang b2a39e8883 YARN-9542. Fix LogsCLI guessAppOwner ignores custome file format suffix. 
Contributed by Prabhu Joseph
 2019-05-29 18:04:13 -04:00
Eric E Payne 2e561cef47 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 
(cherry picked from commit 3c63551101)
 2019-05-29 18:43:13 +00:00
Ahmed Hussein 777f7345ef YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit abf76ac371)
 2019-05-29 12:14:01 -05:00
Takanobu Asanuma a9a3450560 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:34:16 +09:00
Akira Ajisaka c917ba782e
YARN-9500. Fix typos in ResourceModel.md. Contributed by leiqiang. 
(cherry picked from commit 4a692bc3be)
 2019-05-28 16:54:43 +09:00
Akira Ajisaka 855dc997d6
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:27:33 +09:00
bibinchundatt 71f5bfb822 YARN-9508. YarnConfiguration areNodeLabel enabled is costly in allocation flow. Contributed by Bilwa S T. 
(cherry picked from commit 570fa2da20)
 2019-05-15 13:31:07 +05:30
Sunil G f4ee38df29 YARN-9519. TFile log aggregation file format is not working for yarn.log-aggregation.TFile.remote-app-log-dir config. Contributed by Adam Antal. 
(cherry picked from commit 7d831eca64)
 2019-05-14 10:49:09 -07:00
Sunil G b5b152aded YARN-9504. [UI2] Fair scheduler queue view page does not show actual capacity. Contributed by Zoltan Siegl. 
(cherry picked from commit 64c7f36ab1)
 2019-05-10 14:28:33 +05:30
Haibo Chen c6573562cb YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 
(cherry picked from commit 597fa47ad1)
 2019-05-06 11:58:31 -07:00
Eric E Payne 6fce24fb40 YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein. 
(cherry picked from commit b094b94d43)
 2019-05-02 19:48:06 +00:00
Weiwei Yang cc0c85f04a YARN-9325. TestQueueManagementDynamicEditPolicy fails intermittent. Contributed by Prabhu Joseph. 
(cherry picked from commit 1c8046d67e)
 2019-04-23 14:24:15 +08:00
Eric Yang ac85aa80d9 YARN-8587. Added retries for fetching docker exit code. 
Contributed by Charo Zhang

(cherry picked from commit c16c49b8c3)
 2019-04-19 15:40:23 -04:00
Eric Yang 4a64dab0dd YARN-8622. Fixed container-executor compilation on MacOSX. 
Contributed by Siyao Meng

(cherry picked from commit ef97a20831)
 2019-04-18 19:01:11 -04:00
Eric Yang 2503409977 YARN-6695. Fixed NPE in publishing appFinished events to ATSv2. 
Contributed by Prabhu Joseph

(cherry picked from commit df76cdc895)
 2019-04-18 12:30:55 -04:00
Siyao Meng 742a3ad24b YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 6e4399ea61)
 2019-04-17 22:57:33 -07:00
Weiwei Yang db185de31c YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 
(cherry picked from commit 8c1bba375b)
 2019-04-10 23:02:24 +08:00
Weiwei Yang 7a80b1b481 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 
(cherry picked from commit ec143cbf67)
 2019-04-06 20:19:03 +08:00
Weiwei Yang 06cf8ce30f YARN-9394. Use new API of RackResolver to get better performance. Contributed by Lantao Jin. 
(cherry picked from commit 945e8c6064)
 2019-04-04 18:41:04 +08:00
Eric Yang 10642a6205 YARN-9391. Fixed node manager environment leaks into Docker containers. 
Contributed by Jim Brennan

(cherry picked from commit 3c45762a0b)
 2019-03-25 15:54:52 -04:00
Sunil G aff5973401 YARN-8803. [UI2] Show flow runs in the order of recently created time in graph widgets. Contributed by Akhil PB. 
(cherry picked from commit c79f139519)
 2019-03-06 16:49:49 +05:30
Sunil G d721634fea YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth. 
(cherry picked from commit 46045c5cb3)
 2019-03-06 16:01:56 +05:30
bibinchundatt 63ed16e076 Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph" 
This reverts commit cf1944eb6e.
 2019-03-04 17:01:40 +05:30
Weiwei Yang 4ceb4e4f05 YARN-9332. RackResolver tool should accept multiple hosts. Contributed by Lantao Jin. 
(cherry picked from commit fe6b2b2f23e69f0643e870d9c500117088983209)
 2019-03-02 16:04:24 +00:00
Sunil G d045f02a8d YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-01 19:27:03 +05:30
Eric Yang 3f3548b66a YARN-9334. Allow YARN Service client to send SPNEGO challenge header when authentication type is not simple. 
Contributed by Billie Rinaldi

(cherry picked from commit 04b228e43b)
 2019-02-28 09:33:05 -08:00
Weiwei Yang 7575e3090d YARN-9324. TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11. Contributed by Prabhu Joseph. 2019-02-28 09:32:07 +08:00
Weiwei Yang 7fa5373ec4 YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie. 
(cherry picked from commit 8c30114b00)
 2019-02-27 17:35:09 +08:00
Sunil G 809e3f2453 YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth. 
(cherry picked from commit 5e91ebd91a)
 2019-02-27 12:03:58 +05:30
Sunil G a95a0cbf2f YARN-9087. Improve logging for initialization of Resource plugins. Contributed by Szilard Nemeth. 2019-02-27 11:54:43 +05:30
Weiwei Yang bdde6a612e YARN-9329. updatePriority is blocked when using FairScheduler. Contributed by Jiandan Yang. 
(cherry picked from commit 3e1739d589)
 2019-02-26 00:18:24 +08:00
Sunil G 359e459df1 YARN-9168. DistributedShell client timeout should be -1 by default. Contributed by Zhankun Tang. 
(cherry picked from commit 6cec90653d)
 2019-02-25 15:29:31 +05:30
Sunil G f282f9c362 YARN-9213. RM Web UI v1 does not show custom resource allocations for containers page. Contributed by Szilard Nemeth. 2019-02-25 11:37:42 +05:30
Weiwei Yang cdce1c17a0 YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph. 
(cherry picked from commit 9cd5c5447f)
 2019-02-24 22:48:55 +08:00
Weiwei Yang 604a915bab YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie. 
(cherry picked from commit 9c88695bcd)
 2019-02-24 22:21:53 +08:00
bibinchundatt 3e1bd53a37 YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph 2019-02-23 07:59:51 +05:30
bibinchundatt cf1944eb6e YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph 2019-02-22 20:51:47 +05:30
Sunil G d75aa33612 YARN-9118. Handle exceptions with parsing user defined GPU devices in GpuDiscoverer. Contributed by Szilard Nemeth. 
(cherry picked from commit 95fbbfed75)
 2019-02-22 20:23:24 +05:30
Weiwei Yang fb2735de16 YARN-9258. Support to specify allocation tags without constraint in distributed shell CLI. Contributed by Prabhu Joseph. 
(cherry picked from commit 6c4ab0312b)
 2019-02-22 00:23:21 +08:00
Weiwei Yang c2ef443359 YARN-9315. TestCapacitySchedulerMetrics fails intermittently. Contributed by Prabhu Joseph. 2019-02-21 18:06:26 +08:00
bibinchundatt e6f2b8730f YARN-9286. [Timeline Server] Sorting based on FinalStatus shows pop-up message. Contributed by Bilwa S T. 
(cherry picked from commit b8de78c570)
 2019-02-20 01:20:15 +05:30
Sunil G a060e8cb51 YARN-7824. [UI2] Yarn Component Instance page should include link to container logs. Contributed by Akhil PB. 2019-02-17 20:19:46 +05:30
Adam Antal 830aaac023
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9385ec45d7)
 2019-02-15 18:48:21 +09:00
Sunil G b4dc62a120 YARN-8295. [UI2] Improve Resource Usage tab error message when there are no data available. Contributed by Charan Hebri. 
(cherry picked from commit 5b55f3538c)
 2019-02-15 12:43:01 +05:30
Akira Ajisaka f0b476bb31
YARN-9284. Fix the unit of yarn.service.am-resource.memory in the document. Contributed by Masahiro Tanaka. 
(cherry picked from commit 3a39d9a2d2)
 2019-02-15 15:43:32 +09:00
bibinchundatt a8563d23f5 YARN-9295. [UI2] Fix label typo in Cluster Overview page. Contributed by Charan Hebri. 
(cherry picked from commit b66d5ae9e2)
 2019-02-14 23:10:21 +05:30
Sunil G d321d0e747 YARN-7761. [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link. Contributed by Akhil PB. 2019-02-14 20:55:43 +05:30
Weiwei Yang 4d4968646a YARN-9252. Allocation Tag Namespace support in Distributed Shell. Contributed by Prabhu Joseph. 
(cherry picked from commit 2b7f828d46)
 2019-02-12 16:31:38 +08:00
Weiwei Yang 9cfa5ce0c1 YARN-9253. Add UT to verify Placement Constraint in Distributed Shell. Contributed by Prabhu Joseph. 
(cherry picked from commit 711d22f166)
 2019-02-12 16:22:34 +08:00
Giovanni Matteo Fumarola 314c559971 YARN-9191. Add cli option in DS to support enforceExecutionType in resource requests. Contributed by Abhishek Modi. 
(cherry picked from commit f738b397ae)
 2019-02-12 14:21:06 +08:00
Weiwei Yang fbd03543d8 YARN-8555. Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options. Contributed by Prabhu Joseph. 
(cherry picked from commit 0a1637c750)
 2019-02-11 15:56:34 +08:00
Eric Yang 3d552b24ba YARN-8761. Service AM support for decommissioning component instances. 
Contributed by Billie Rinaldi

(cherry picked from commit 4c465f5535)
 2019-02-10 09:26:27 -08:00
Masatake Iwasaki 6229469574 YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 
(cherry picked from commit e0ab1bdece)
 2019-02-09 00:28:59 +09:00
Sunil G fbc08145cf YARN-9257. Distributed Shell client throws a NPE for a non-existent queue. Contributed by Charan Hebri. 2019-02-08 11:22:44 +05:30
Eric E Payne 55dde827e6 YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein 
(cherry picked from commit d1ca9432dd)
 2019-02-07 16:47:15 +00:00
Vinayakumar B e2b91b2ccb YARN-8498. Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04. Contributed by Ayush Saxena. 2019-02-07 13:03:42 +05:30
Wangda Tan 649da5af04 Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa)
 2019-02-05 14:05:51 -08:00
Weiwei Yang 3a51e50fd8 YARN-9275. Add link to NodeAttributes doc in PlacementConstraints document. Contributed by Masatake Iwasaki. 
(cherry picked from commit aa7ce50e55)
 2019-02-05 16:09:15 +08:00
Weiwei Yang b64e9df949 YARN-9262. TestRMAppAttemptTransitions is failing with an NPE. Contributed by lujie. 
(cherry picked from commit 28ad20a711)
 2019-02-04 14:00:30 +05:30
Sunil G 99876a5ab8 YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 
(cherry picked from commit 604b2489a9)
 2019-02-04 12:49:06 +05:30
Weiwei Yang a0fafbc3ef YARN-9263. TestConfigurationNodeAttributesProvider fails after Mockito updated. Contributed by Weiwei Yang. 
(cherry picked from commit f20b043a02)
 2019-02-04 12:45:40 +05:30
Sunil G 0e7060a1d5 YARN-9099. GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way. Contributed by Szilard Nemeth. 
(cherry picked from commit 71c49fa60f)
 2019-01-31 09:26:07 +05:30
Eric E Payne 4052b7ee60 YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 
(cherry picked from commit 04105bbfdb)
 2019-01-29 17:52:54 +00:00
Weiwei Yang 6b8dd8d113 YARN-9237. NM should ignore sending finished apps to RM during RM fail-over. Contributed by Jiandan Yang. 
(cherry picked from commit 4f63ffe444)
 2019-01-29 10:42:09 +08:00
Eric Yang 8c332affa8 YARN-8901. Fixed restart policy NEVER/ON_FAILURE with component dependency. 
Contributed by Suma Shivaprasad

(cherry picked from commit f5a95f7998)
 2019-01-28 18:12:04 -05:00
Jonathan Hung bf760e7e81 YARN-9222. Print launchTime in ApplicationSummary 
(cherry picked from commit 6cace58e21)
 2019-01-25 13:23:37 -08:00
Sunil G 4aea4b7c9c YARN-8961. [UI2] Flow Run End Time shows 'Invalid date'. Contributed by Akhil PB 
(cherry picked from commit c726445990)
 2019-01-24 15:02:21 +05:30
Weiwei Yang bc6374f282 YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 2019-01-23 18:10:28 +08:00
Weiwei Yang 8ad7711605 YARN-8101. Add UT to verify node-attributes in RM nodes rest API. Contributed by Prabhu Joseph. 
(cherry picked from commit 721d5c2a5f)
 2019-01-23 18:07:45 +08:00
Weiwei Yang 9114489566 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 
(cherry picked from commit d43df31751)
 2019-01-22 10:46:37 +08:00
Weiwei Yang ac2f4b64f9 YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang. 
(cherry picked from commit abde1e1f58)
 2019-01-21 21:20:01 +08:00
Wangda Tan fe7cb2d84a YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda) 
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
(cherry picked from commit 6d7eedfd28)
 2019-01-17 15:13:42 -08:00
rahul3 4aee7946dc
YARN-9203. Fix typos in yarn-default.xml. 
This closes #437

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0a46baecd3)
 2019-01-17 16:19:49 +09:00
Akira Ajisaka a415446705
YARN-8747. [UI2] YARN UI2 page loading failed due to js error under some time zone configuration. Contributed by collinma. 
(cherry picked from commit 104ef5df36)
 2019-01-16 14:37:43 +09:00
Wangda Tan 1dc2b49bfd YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda) 
Change-Id: Ib8044307a4241f6b1b7b9b8266b9256f39b16384
 2019-01-07 12:21:33 -08:00
Eric Yang bba76b6f31 HADOOP-16031. Fixed TestSecureLogins unit test. Contributed by Akira Ajisaka 2019-01-07 13:24:01 -05:00
Weiwei Yang 2b549e32e1 YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 944cf87223)
 2019-01-07 16:05:57 +08:00
Wangda Tan cffe5c1ba0 YARN-9160. [Submarine] Document 'PYTHONPATH' environment variable setting when using -localization options. (Zhankun Tang via wangda) 
Change-Id: I115b6710eb11501e21ab3e9e7cd1a87f322e5cf8
(cherry picked from commit 2c02aa6ec2)
 2019-01-06 11:10:47 -08:00
Wangda Tan 1ad1ab5d7a YARN-9141. [Submarine] JobStatus outputs with system UTC clock, not local clock. (Zac Zhou via wangda) 
Change-Id: Ib2b17cf7860ddc9188f06498086de3326bc09519
(cherry picked from commit 8f004fe4e0)
 2019-01-06 11:10:41 -08:00
Weiwei Yang a24cca11f2 YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie. 
(cherry picked from commit cfe89e6f96)
 2019-01-04 01:04:39 +08:00
Sunil G 9bfea98746 YARN-9090. [Submarine] Update submarine installation script document. Contributed by Xun Liu. 
(cherry picked from commit 742fd978b5)
 2018-12-26 08:27:50 +05:30
Akira Ajisaka d6baf9d5bf
YARN-9154. Fix itemization in YARN service quickstart document. Contributed by Ayush Saxena. 
(cherry picked from commit e98e963e51)
 2018-12-22 13:26:41 +09:00
Weiwei Yang 7deef08eb8 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. 2018-12-21 16:31:03 +08:00
Eric Yang 29c9c8a893 YARN-9126. Fix container clean up for reinitialization. 
Contributed by Chandni Singh

(cherry picked from commit e815fd9c49)
 2018-12-19 14:58:19 -05:00
Eric Yang 4e1cef3625 YARN-9084. Reset container state and defer readiness check for upgrade. 
Contributed by Chandni Singh

(cherry picked from commit ccdd982e51)
 2018-12-18 18:05:40 -05:00
Eric Yang 28ca14e71b YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator. 
Contributed by Tarun Parimi

(cherry picked from commit 71e0b0d800)
 2018-12-17 12:08:09 -05:00
Akira Ajisaka 71d8ffb551
YARN-9122. Add table of contents to YARN Service API document. Contributed by Zhankun Tang. 
(cherry picked from commit 26e55d4ceb)
 2018-12-15 10:30:37 +09:00
Eric Yang 52aafb9789 YARN-9125. Fixed Carriage Return detection in Docker container launch command. 
Contributed by Billie Rinaldi

(cherry picked from commit b2d7204ed0)
 2018-12-14 17:55:10 -05:00
Wangda Tan 29e4e5f62c YARN-9112. [Submarine] Support polling applicationId when it's not ready in cluster. (Zhankun Tang via wangda) 
Change-Id: I73d73f3d631b28fb9866faa56571839b13824a97
(cherry picked from commit 9fba6cc247)
 2018-12-12 11:52:16 -08:00
Wangda Tan ef0f835537 YARN-8714. [Submarine] Support files/tarballs to be localized for a training job. (Zhankun Tang via wangda) 
Change-Id: I845131273e52a9d81dbc813ea6d4af06b205e334
(cherry picked from commit c771fe6e10)
 2018-12-12 11:52:07 -08:00
Weiwei Yang 2b3c3d2a32 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters. 
(cherry picked from commit 1c09a10e96)
 2018-12-10 12:07:23 +08:00
Jonathan Hung 3ab6ea7aca YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics 
(cherry picked from commit 978ab3e958227220cb6f1a08ae6e7cdb8a46628b)
 2018-12-07 10:45:47 -08:00
Eric Yang 8c70728f7f YARN-9071. Improved status update for reinitialized containers. 
Contributed by Chandni Singh

(cherry picked from commit 1b790f4dd1)
 2018-12-05 19:04:55 -05:00
Wangda Tan 559cb99bf6 Revert "YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)" 
This reverts commit 920401b296.
 2018-12-04 14:13:39 -08:00
Jonathan Hung 6b01e4d2a8 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 
(cherry picked from commit 1c8bd7128c99d8215ef16438bd2ce6b1f025a966)
 2018-11-30 10:16:00 -08:00
Eric Yang a8f67ad7c9 YARN-9067. Fixed Resource Manager resource leak via YARN service. 
Contributed by Eric Yang
 2018-11-29 13:50:06 -05:00
bibinchundatt 183ec39c4b YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T. 
(cherry picked from commit 07142f54a8)
 2018-11-29 22:16:32 +05:30
Wangda Tan ee9deb6e9c YARN-9030. Log aggregation changes to handle filesystems which do not support setting permissions. (Suma Shivaprasad via wangda) 
Change-Id: I80f1e8196b8624e24d74494719fdedfd7061dced
(cherry picked from commit 9de8e8d049)
 2018-11-28 15:06:35 -08:00
Jason Lowe df0e7766e4 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 3ce99e32f7)
 2018-11-28 08:50:18 -06:00
Eric Yang db8b2a130c YARN-9056. Improved YARN service upgrade state logic for readiness check. 
Contributed by Chandni Singh

(cherry picked from commit f657a2a661)
 2018-11-27 19:45:23 -05:00
Eric Yang 838190482d YARN-8986. Added port publish for Docker container running with bridge. 
Contributed by Charo Zhang
 2018-11-27 14:27:13 -05:00
Weiwei Yang 01477108d9 YARN-8984. AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty. Contributed by Yang Wang. 
(cherry picked from commit 176bb3f812)
 2018-11-22 17:04:16 +08:00
Wangda Tan 34387599c4 YARN-8960. [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment. (Zac Zhou via wangda) 
Change-Id: I21b1addc9c32817650ea744a8f2e6b5602f2f4d4
(cherry picked from commit 8b23814415)
 2018-11-19 09:01:21 -08:00
Weiwei Yang 650581a19d YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 
(cherry picked from commit d027a24f03)
 2018-11-18 23:23:09 +08:00
Wangda Tan bee5bf867b YARN-9001. [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs. (Zac Zhou via wangda) 
Change-Id: I7e8d1c27ebd37e0907ca570c4f3d56fe7a859635
 2018-11-16 10:26:50 -08:00
Rohith Sharma K S 13e3670e7f YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report. 
(cherry picked from commit ee3355be3c)
 2018-11-16 18:37:20 +05:30
Vrushali C da05328824 YARN-8834 Provide Java client for fetching Yarn specific entities from TimelineReader. Contributed by Abhishek Modi 
(cherry picked from commit a3edfddcf7)
 2018-11-16 18:37:01 +05:30
Wangda Tan ea753046b0 YARN-8918. [Submarine] Correct method usage of str.subString in CliUtils. (Zhankun Tang via wangda) 
Change-Id: Id1f11dbab3aa838dee3a0ec8b8fd5dc32f5dd946
(cherry picked from commit 076b795b2e)
 2018-11-13 11:35:06 -08:00
Weiwei Yang 2415f8a5be YARN-8987. Usability improvements node-attributes CLI. Contributed by Bibin A Chundatt. 
(cherry picked from commit c741109522)
 2018-11-12 18:21:11 +08:00
Eric Yang 5399ff8177 YARN-9002. Improve keytab loading for YARN Service. 
Contributed by Gour Saha

(cherry picked from commit 2664248797)
 2018-11-10 01:54:59 -05:00
Weiwei Yang b3321d9933 YARN-8988. Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled. Contributed by Tao Yang. 
(cherry picked from commit e1bbf7dcdf)
 2018-11-08 17:50:16 +08:00
Weiwei Yang b10ec0aa14 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji. 
(cherry picked from commit c96cbe8659)
 2018-11-07 22:44:01 +08:00
Takanobu Asanuma 02abd0f80a YARN-8866. Fix a parsing error for crossdomain.xml. 
(cherry picked from commit 8dc1f6dbf7)
 2018-11-07 18:29:28 +09:00
Weiwei Yang 4ff7255d6e YARN-8976. Remove redundant modifiers in interface ApplicationConstants. Contributed by Zhankun Tang. 
(cherry picked from commit 482716e5a4)
 2018-11-07 10:49:09 +08:00
Akira Ajisaka 41c4e9583d
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 
(cherry picked from commit 951c98f890)
 2018-11-07 11:18:48 +09:00
Jason Lowe 9265934201 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit ab6aa4c726)
 2018-11-06 08:47:30 -06:00
Weiwei Yang f00125e2d9 YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang. 
(cherry picked from commit 5d6554c722)
 2018-11-06 14:52:10 +08:00
Weiwei Yang b474239e0b YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji. 
(cherry picked from commit c7fcca0d7e)
 2018-11-06 13:17:19 +08:00
Jonathan Hung b9a3c988c0 YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 
(cherry picked from commit 2ab611d48b)
 2018-11-01 14:31:09 -07:00
Rohith Sharma K S 7920f79049 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. 
(cherry picked from commit 4ec4ec6971)
 2018-10-30 11:50:33 +05:30
Weiwei Yang 288bc3f1eb YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 78ea897b67)
 2018-10-29 12:05:45 +08:00
Weiwei Yang 5db4272d91 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23)
 2018-10-29 11:55:22 +08:00
Eric Yang 3665c46176 YARN-8938. Updated YARN service upgrade document. 
Contributed by Chandni Singh

(cherry picked from commit bbc6dcd3d0)
 2018-10-24 11:52:50 -04:00
Jason Lowe 7097755925 YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 93fb3b4b9c)
 2018-10-23 12:54:04 -05:00
Rohith Sharma K S 660fff3138 YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli. 
(cherry picked from commit 0b62983c5a)
 2018-10-23 14:06:13 +05:30
Sunil G 30998fea28 YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh. 
(cherry picked from commit 2202e00ba8)
 2018-10-23 09:57:26 +05:30
Sunil G 8724c2900f YARN-7502. Nodemanager restart docs should describe nodemanager supervised property. Contributed by Suma Shivaprasad. 
(cherry picked from commit e50334513c)
 2018-10-22 14:16:24 +05:30
Eric Yang 04600cf5da YARN-8910. Fixed misleading log statement when container max retries is infinite. 
Contributed by Chandni Singh

(cherry picked from commit 47ad98b2e1)
 2018-10-19 13:49:51 -04:00
Wangda Tan 9ed9e185d7 YARN-8916. Define a constant docker string in ContainerRuntimeConstants.java for better maintainability. (Zhankun Tang via wangda) 
Change-Id: I1349e740037f81afdbe30edbe741f20e88fd0a90
(cherry picked from commit 5e02b4915b)
 2018-10-19 09:50:18 -07:00
Wangda Tan bd08554aa4 YARN-8908. Fix errors in yarn-default.xml related to GPU/FPGA. (Zhankun Tang via wangda) 
Change-Id: I46f511a9b51b65b069e15d5b827530110c41edab
(cherry picked from commit 071b9e19aa)
 2018-10-19 09:50:12 -07:00
Weiwei Yang 042f2df19b YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 13cc0f50ea)
 2018-10-19 10:01:45 +08:00
Eric Yang 91c916e209 YARN-8618. Added detection for non-upgradable service. 
Contributed by Chandni Singh

(cherry picked from commit 66f059ed1d)
 2018-10-18 20:00:50 -04:00
Wangda Tan 7252f8e117 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07)
 2018-10-18 12:12:19 -07:00
Wangda Tan 5f7ed043c8 YARN-8456. Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable. (Zhankun Tang via wangda) 
Change-Id: Iff150ea98ba0c60d448474fd940eb121afce6965
(cherry picked from commit a457a8951a)
 2018-10-18 12:12:14 -07:00
Wangda Tan a7785d2764 YARN-6098. Add documentation for Delete Queue. (Suma Shivaprasad via wangda) 
Change-Id: I23bf7ca0acbe1168a7f805ceccdd62fe41bfeb35
(cherry picked from commit bfb88b10f4)
 2018-10-18 12:12:09 -07:00