Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
23,331 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5745 Commits

Author SHA1 Message Date
Eric Yang a54c1e3ace YARN-8551. Project setup for MaWo application. 
Contributed by Yesha Vora
 2019-04-22 16:56:26 -04:00
Inigo Goiri 96e3027e46 YARN-2889. Limit the number of opportunistic container allocated per AM heartbeat. Contributed by Abhishek Modi. 2019-04-22 09:49:03 -07:00
Weiwei Yang adefd37105 YARN-9495. Fix findbugs warnings in hadoop-yarn-server-resourcemanager module. Contributed by Tao Yang. 2019-04-20 21:51:00 +08:00
Billie Rinaldi 317fcbabe3 YARN-9254. Add support for storing application catalog data to HDFS. Contributed by Eric Yang 2019-04-19 10:29:57 -07:00
Inigo Goiri aeadb9432f YARN-9448. Fix Opportunistic Scheduling for node local allocations. Contributed by Abhishek Modi. 2019-04-19 09:41:06 -07:00
Eric Yang ef97a20831 YARN-8622. Fixed container-executor compilation on MacOSX. 
Contributed by Siyao Meng
 2019-04-18 18:59:21 -04:00
Eric Yang df76cdc895 YARN-6695. Fixed NPE in publishing appFinished events to ATSv2. 
Contributed by Prabhu Joseph
 2019-04-18 12:29:37 -04:00
Prabhu Joseph aa4c744aef
YARN-9470. Fix order of actual and expected expression in assert statements 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-18 15:40:37 +09:00
Siyao Meng 6e4399ea61 YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-17 22:56:57 -07:00
Eric Yang 9cf7401794 YARN-9349. Improved log level practices for InvalidStateTransitionException. 
Contributed by Anuhan Torgonshar

(cherry picked from commit fe2370e039e1ee980d74769ae85d67434e0993cf)
 2019-04-16 19:53:45 -04:00
Szilard Nemeth b8086aed86 YARN-9123. Clean up and split testcases in TestNMWebServices for GPU support. Contributed by Szilard Nemeth. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2019-04-16 11:06:25 -07:00
Billie Rinaldi 2364c7d0bf YARN-9466. Fixed application catalog navigation bar height in Safari. Contributed by Eric Yang 2019-04-16 10:04:27 -07:00
Billie Rinaldi ad865888a6 YARN-8530. Add SPNEGO filter to application catalog. Contributed by Eric Yang 2019-04-16 09:52:14 -07:00
Akira Ajisaka c65218ee8e YARN-8943. Upgrade JUnit from 4 to 5 in hadoop-yarn-api. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-04-16 12:13:49 +09:00
Eric Badger 5583e1b6fc YARN-7848 Force removal of docker containers that do not get removed on first try. Contributed by Eric Yang 2019-04-15 20:47:09 +00:00
Eric Badger 254efc9358 YARN-9379. Can't specify docker runtime through environment. Contributed by caozhiqiang 2019-04-15 18:24:37 +00:00
Weiwei Yang 7fa73fac26 YARN-9439. Support asynchronized scheduling mode and multi-node lookup mechanism for app activities. Contributed by Tao Yang. 2019-04-16 00:12:43 +08:00
Akira Ajisaka c4c16cae7d
YARN-9481. [JDK 11] Build fails due to hard-coded target version in hadoop-yarn-applications-catalog-webapp. Contributed by Kei Kori. 2019-04-15 13:08:28 +09:00
Inigo Goiri 7a68e7abd5 YARN-9474. Remove hard coded sleep from Opportunistic Scheduler tests. Contributed by Abhishek Modi. 2019-04-14 20:11:20 -07:00
Billie Rinaldi b2cdf809bc YARN-9281. Add express upgrade button to Appcatalog UI. Contributed by Eric Yang 2019-04-13 18:55:11 +03:00
Gabor Bota 1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre. 
Author:    Gabor Bota <gabor.bota@cloudera.com>
 2019-04-12 18:28:38 -07:00
Giovanni Matteo Fumarola ed3747c1cc YARN-9435. Add Opportunistic Scheduler metrics in ResourceManager. Contributed by Abhishek Modi. 2019-04-11 11:49:19 -07:00
Weiwei Yang 8c1bba375b YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu. 2019-04-10 22:51:28 +08:00
Igor Rudenko 32722d2661
YARN-9433. Remove unused constants in YARN resource manager 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-04-10 18:37:27 +09:00
Giovanni Matteo Fumarola cfec455c45 YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Contributed by Inigo Goiri. 2019-04-09 10:59:43 -07:00
Weiwei Yang fc05b0e70e YARN-9313. Support asynchronized scheduling mode and multi-node lookup mechanism for scheduler activities. Contributed by Tao Yang. 2019-04-08 13:40:53 +08:00
Weiwei Yang ec143cbf67 YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang. 2019-04-06 19:59:36 +08:00
Vrushali C 22362c876d YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls. Contributed by Abhishesk Modi. 2019-04-05 12:06:51 -07:00
Vrushali C 27039a29ae YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi. 2019-04-05 12:02:43 -07:00
Eric Yang 2bf82aee82 YARN-9441. Updated YARN app catalog name for consistency. 
Contributed by Weiwei Yang
 2019-04-04 13:21:30 -04:00
Eric Yang 8d150067e2 YARN-9396. Fixed duplicated RM Container created event to ATS. 
Contributed by Prabhu Joseph
 2019-04-04 13:01:56 -04:00
Weiwei Yang 945e8c6064 YARN-9394. Use new API of RackResolver to get better performance. Contributed by Lantao Jin. 2019-04-04 18:05:01 +08:00
Vrushali C eb03f7c419 YARN-9303 Username splits won't help timelineservice.app_flow table. Contributed by Prabhu Joseph. 2019-04-03 22:53:05 -07:00
Gabor Bota d7979079ea HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota. 2019-04-03 12:59:39 -06:00
Sunil G 002dcc4ebf YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 2019-04-03 18:57:28 +05:30
Yufei Gu 2f752830ba YARN-9214. Add AbstractYarnScheduler#getValidQueues method to remove duplication. Contributed by Wanqiang Ji. 2019-04-01 20:05:15 -07:00
Giovanni Matteo Fumarola ab2bda57bd YARN-9428. Add metrics for paused containers in NodeManager. Contributed by Abhishek Modi. 2019-04-01 14:21:17 -07:00
Giovanni Matteo Fumarola da7f8c244d YARN-9431. Fix flaky junit test fair.TestAppRunnability after YARN-8967. Contributed by Wilfred Spiegelenburg. 2019-04-01 11:21:31 -07:00
Giovanni Matteo Fumarola 332cab5518 YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph. 2019-04-01 11:06:51 -07:00
Giovanni Matteo Fumarola b0d24ef39c YARN-9227. DistributedShell RelativePath is not removed at end. Contributed by Prabhu Joseph. 2019-04-01 10:59:14 -07:00
Billie Rinaldi 35b0a381e7 YARN-9255. Improve recommend applications order and fix findbugs warnings. Contributed by Eric Yang 2019-04-01 09:21:36 -07:00
Billie Rinaldi d9e9e5657d YARN-9348. Application catalog build system bug fixes. Contributed by Eric Yang 2019-03-29 13:51:21 -07:00
Billie Rinaldi 7ee32a2eb1 YARN-7129. Application Catalog for YARN applications. Contributed by Eric Yang 2019-03-29 13:51:21 -07:00
Devaraj K 56f1e131ec YARN-9270. Minor cleanup in TestFpgaDiscoverer. Contributed by Peter Bacsko. 2019-03-29 10:58:56 -07:00
Giovanni Matteo Fumarola 9cd66198ee YARN-9411. TestYarnNativeServices fails sporadically with bind address in use. Contributed by Prabhu Joseph. 2019-03-27 10:44:54 -07:00
Devaraj K a4cd75e09c YARN-9269. Minor cleanup in FpgaResourceAllocator. Contributed by Peter Bacsko. 2019-03-27 10:08:07 -07:00
yufei 5257f50abb YARN-8967. Change FairScheduler to use PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-25 22:47:24 -07:00
Eric Yang 710cbc9bd6 YARN-9405. Fixed flaky tests in TestYarnNativeServices. 
Contributed by Prabhu Joseph
 2019-03-25 16:34:04 -04:00
Devaraj K eeda6891e4 YARN-9268. General improvements in FpgaDevice. Contributed by Peter Bacsko. 2019-03-25 13:22:53 -07:00
Eric Yang 3c45762a0b YARN-9391. Fixed node manager environment leaks into Docker containers. 
Contributed by Jim Brennan
 2019-03-25 15:53:24 -04:00
Giovanni Matteo Fumarola 509b20b292 YARN-9404. TestApplicationLifetimeMonitor#testApplicationLifetimeMonitor fails intermittent. Contributed by Prabhu Joseph. 2019-03-22 11:45:39 -07:00
Zoltan Siegl ce5eb9cb2e YARN-9358. Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322 
(Contributed by Zoltan Siegl via Daniel Templeton)

Change-Id: I92d52c0ca630e71afb26b2b7587cbdbe79254a05
 2019-03-22 12:28:34 +01:00
Giovanni Matteo Fumarola 548997d6c9 YARN-9402. Opportunistic containers should not be scheduled on Decommissioning nodes. Contributed by Abhishek Modi. 2019-03-21 12:04:05 -07:00
Devaraj K a99eb80659 YARN-9267. General improvements in FpgaResourceHandlerImpl. Contributed by Peter Bacsko. 2019-03-21 11:15:56 -07:00
Eric Yang 506502bb83 YARN-9370. Added logging for recovering assigned GPU devices. 
Contributed by Yesha Vora
 2019-03-20 19:12:19 -04:00
Eric Yang f2b862cac6 YARN-9398. Fixed javadoc errors for FPGA related java files. 
Contributed by Peter Bacsko
 2019-03-20 15:45:37 -04:00
Rohith Sharma K S b3b0e332e6 YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph. 2019-03-20 21:24:31 +05:30
Rohith Sharma K S 0d24684eee YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph. 2019-03-20 21:22:54 +05:30
Rohith Sharma K S f6e60e966e YARN-9387. Update document for ATS HBase Custom tablenames (-entityTableName). Contributed by Prabhu Joseph. 2019-03-20 21:20:50 +05:30
Rohith Sharma K S c1a4eeb7c8 YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph. 2019-03-20 21:18:19 +05:30
Giovanni Matteo Fumarola 5d8bd0e5cb YARN-9392. Handle missing scheduler events in Opportunistic Scheduler. Contributed by Abhishek Modi. 2019-03-19 11:00:21 -07:00
Rohith Sharma K S c9e50c48de YARN-9365. Fix table creation command in TimelineServiceV2.md 2019-03-19 12:05:37 +05:30
Eric Yang 09eabda314 YARN-9364. Remove commons-logging dependency from YARN. 
Contributed by Prabhu Joseph
 2019-03-18 19:58:42 -04:00
Eric Yang 5f6e225166 YARN-9363. Replaced debug logging with SLF4J parameterized log message. 
Contributed by Prabhu Joseph
 2019-03-18 13:57:18 -04:00
Eric Yang 19b22c4385 YARN-9385. Fixed ApiServiceClient to use current UGI. 
Contributed by Eric Yang
 2019-03-18 13:16:34 -04:00
Shweta Yakkali 0e7e9013d4 YARN-9340. [Clean-up] Remove NULL check before instanceof in ResourceRequestSetKey 
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I932e29b36f086f7b7c76a250e33b473617ddbda1
 2019-03-18 15:08:37 +01:00
Eric Yang 2064ca015d YARN-9349. Changed logging to use slf4j api. 
Contributed by Prabhu Joseph
 2019-03-15 19:20:59 -04:00
Eric Yang 03f3c8aed2 YARN-4404. Corrected typo in javadoc. 
Contributed by Yesha Vora
 2019-03-15 18:04:04 -04:00
Eric Badger 688b177fc6 YARN-8376. Separate white list for docker.trusted.registries and docker.privileged-container.registries. Contributed by Eric Yang 2019-03-14 19:39:00 +00:00
Vrushali C f235a942d5 YARN-9016 DocumentStore as a backend for ATSv2. Contributed by Sushil Ks. 2019-03-13 16:45:23 -07:00
Vrushali C 17a3e14d25 YARN-9338 Timeline related testcases are failing. Contributed by Abhishek Modi. 2019-03-12 21:33:17 -07:00
Giovanni Matteo Fumarola 67cc24a7a4 YARN-9381. The yarn-default.xml has two identical property named yarn.timeline-service.http-cross-origin.enabled. Contributed by Abhishek Modi. 2019-03-12 15:21:03 -07:00
Sunil G 8e1539eca8 YARN-9266. General improvements in IntelFpgaOpenclPlugin. Contributed by Peter Bacsko. 2019-03-13 02:45:17 +05:30
Billie Rinaldi ab574ffd72 Revert "YARN-7129. Application Catalog for YARN applications. Contributed by Eric Yang" 
This reverts commit 2576401928.
 2019-03-12 11:50:16 -07:00
Billie Rinaldi 55069bfb09 Revert "YARN-9348. Application catalog build system bug fixes." 
This reverts commit 01ada40ea4.
 2019-03-12 11:49:55 -07:00
Billie Rinaldi 024b3bae8a Revert "YARN-9255. Improve recommend applications order and fix findbugs warnings. Contributed by Eric Yang" 
This reverts commit aab7b77536.
 2019-03-12 11:48:55 -07:00
Billie Rinaldi aab7b77536 YARN-9255. Improve recommend applications order and fix findbugs warnings. Contributed by Eric Yang 2019-03-12 10:06:09 -07:00
Rohith Sharma K S c24af4b0d6 YARN-9336. JobHistoryServer leaks CLOSE_WAIT tcp connections when using LogAggregationIndexedFileController. Contributed by Tarun Parimi. 2019-03-12 20:57:27 +05:30
Eric Badger fa7a0b269a YARN-8805. Automatically convert the launch command to the exec form when using entrypoint support 2019-03-11 12:55:01 -05:00
Sunil G de15a66d78 YARN-9265. FPGA plugin fails to recognize Intel Processing Accelerator Card. Contributed by Peter Bacsko. 2019-03-08 17:39:22 +05:30
Weiwei Yang 373705fcea YARN-9239. Document docker registry deployment with Ozone CSI driver. Contributed by Eric Yang. 2019-03-08 09:47:20 +08:00
Eric Yang 39b4a37e02 YARN-9341. Fixed enentrant lock usage in YARN project. 
Contributed by Prabhu Joseph
 2019-03-07 16:47:45 -05:00
Vrushali C 491313ab84 YARN-8218 Add application launch time to ATSV1. Contributed by Abhishek Modi 2019-03-06 21:47:29 -08:00
Eric Yang 01ada40ea4 YARN-9348. Application catalog build system bug fixes. 
Contributed by Eric Yang
 2019-03-06 17:55:48 -05:00
Sunil G c79f139519 YARN-8803. [UI2] Show flow runs in the order of recently created time in graph widgets. Contributed by Akhil PB. 2019-03-06 16:49:08 +05:30
Sunil G 46045c5cb3 YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-06 16:01:08 +05:30
Eric Yang 7b42e0e32a YARN-7266. Fixed deadlock in Timeline Server thread initialization. 
Contributed by Prabhu Joseph
 2019-03-05 12:17:01 -05:00
Billie Rinaldi 2576401928 YARN-7129. Application Catalog for YARN applications. Contributed by Eric Yang 2019-03-05 06:14:14 -08:00
Yufei Gu 0aefe2846f YARN-9298. Implement FS placement rules using PlacementRule interface. Contributed by Wilfred Spiegelenburg. 2019-03-04 23:49:07 -08:00
Prabhu Joseph e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-03-05 14:10:08 +09:00
bibinchundatt 15098df744 Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph." 
This reverts commit a63c358b78.
 2019-03-04 16:57:31 +05:30
Akira Ajisaka bd8d299ded
YARN-7477. Moving logging APIs over to slf4j in hadoop-yarn-common. Contributed by Prabhu Joseph. 2019-03-04 16:59:20 +09:00
Weiwei Yang e20b5ef52c
YARN-9332. RackResolver tool should accept multiple hosts. Contributed by Lantao Jin. 2019-03-04 16:59:20 +09:00
Suma Shivaprasad cab8529ecb YARN-7904. Privileged, trusted containers should be supported only in ENTRYPOINT mode. Contributed by Eric Yang. 2019-03-01 11:06:09 -08:00
Sunil G dcaca19871 YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth. 2019-03-01 19:24:35 +05:30
Szilard Nemeth 538bb4880d YARN-9323. FSLeafQueue#computeMaxAMResource does not override zero values for custom resources 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: Id844ccf09488f367c0c7de0a3b2d4aca1bba31cc
 2019-02-27 19:59:48 -08:00
Szilard Nemeth 7b928f19a4 YARN-9322. Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: I14c12f1265999d62102f2ec5506d90015efeefe8
 2019-02-27 19:43:50 -08:00
Weiwei Yang 1779fc57a1 YARN-9324. TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11. Contributed by Prabhu Joseph. 2019-02-28 09:56:29 +08:00
Eric Yang 04b228e43b YARN-9334. Allow YARN Service client to send SPNEGO challenge header when authentication type is not simple. 
Contributed by Billie Rinaldi
 2019-02-27 18:47:14 -05:00
Vrushali C ea3cdc60b3 YARN-3841 [atsv2 Storage implementation] Adding retry semantics to HDFS backing storage. Contributed by Abhishek Modi. 2019-02-27 14:55:35 -08:00
Vrushali C 0ec962ac8f YARN-5336 Limit the flow name size & consider cleanup for hex chars. Contributed by Sushil Ks 2019-02-27 14:43:39 -08:00
Eric Yang fbc7bb315f YARN-9245. Added query docker image command ability to node manager. 
Contributed by Chandni Singh
 2019-02-27 14:57:24 -05:00
Szilard Nemeth 29e27faf96 YARN-9318. Resources#multiplyAndRoundUp does not consider Resource Types 
(Contributed by Szilard Nemeth via Daniel Templeton)

Change-Id: Ia45f528574c2b054f6f764d1d140e592bdb7e217
 2019-02-27 10:29:17 -08:00
Weiwei Yang 8c30114b00 YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie. 2019-02-27 17:29:02 +08:00
Rohith Sharma K S 6c96f5e4b6 YARN-8378. ApplicationHistoryManagerImpl#getApplications doesn't honor filters. Contributed by Lantao Jin. 2019-02-27 10:32:58 +05:30
Rohith Sharma K S 8eae260af5 YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph. 2019-02-27 10:28:16 +05:30
Suma Shivaprasad a106d2dc9d YARN-8783. Improve the documentation for the docker.trusted.registries configuration. Contributed by Eric Yang. 2019-02-26 10:19:06 -08:00
Weiwei Yang c6ea28c480 YARN-9331. [YARN-8851] Fix a bug that lacking cgroup initialization when bootstrap DeviceResourceHandlerImpl. Contributed by Zhankun Tang. 2019-02-26 10:05:31 +08:00
Giovanni Matteo Fumarola 95372657fc YARN-9287. Consecutive StringBuilder append should be reuse. Contributed by Ayush Saxena. 2019-02-25 11:45:37 -08:00
Weiwei Yang 3e1739d589 YARN-9329. updatePriority is blocked when using FairScheduler. Contributed by Jiandan Yang. 2019-02-26 00:08:13 +08:00
Sunil G 6cec90653d YARN-9168. DistributedShell client timeout should be -1 by default. Contributed by Zhankun Tang. 2019-02-25 15:28:45 +05:30
Sunil G 5e91ebd91a YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth. 2019-02-25 11:30:46 +05:30
Weiwei Yang 9cd5c5447f YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph. 2019-02-24 22:42:27 +08:00
Weiwei Yang 50094d7fef YARN-9300. Lazy preemption should trigger an update on queue preemption metrics for CapacityScheduler. Contributed by Tao Yang. 2019-02-24 22:17:29 +08:00
Sunil G dddcfa4d9f YARN-8821. [YARN-8851] GPU hierarchy/topology scheduling support based on pluggable device framework. Contributed by Zhankun Tang. 2019-02-24 14:37:06 +05:30
Eric Yang 05bce33d6e YARN-9244. Document docker registry deployment with direct S3 driver. 
Contributed by Suma Shivaprasad
 2019-02-22 19:13:52 -05:00
bibinchundatt ed13cf8406 YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph 2019-02-22 21:01:34 +05:30
Sunil G 95fbbfed75 YARN-9118. Handle exceptions with parsing user defined GPU devices in GpuDiscoverer. Contributed by Szilard Nemeth. 2019-02-22 20:22:17 +05:30
Sunil G 9636fe4114 YARN-8891. Documentation of the pluggable device framework. Contributed by Zhankun Tang. 2019-02-22 20:00:13 +05:30
Weiwei Yang 9c88695bcd YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie. 2019-02-22 20:58:33 +08:00
Weiwei Yang 632d5e8a98 YARN-9156. Improve debug message in device plugin method compatibility check of ResourcePluginManager. Contributed by Zhankun Tang. 2019-02-22 11:27:30 +08:00
Wangda Tan b17a2602d1 YARN-9319. Fix compilation issue of handling typedef an existing name by gcc compiler. (Zhankun Tang via wangda) 
Change-Id: Ie8293cc5b710ff355fb6a1ea7201be53ed0385bf
 2019-02-21 11:17:32 -08:00
Weiwei Yang 6c4ab0312b YARN-9258. Support to specify allocation tags without constraint in distributed shell CLI. Contributed by Prabhu Joseph. 2019-02-22 00:18:07 +08:00
Weiwei Yang eedcc8e26e YARN-9315. TestCapacitySchedulerMetrics fails intermittently. Contributed by Prabhu Joseph. 2019-02-21 17:21:21 +08:00
bibinchundatt a63c358b78 YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph. 2019-02-21 13:29:01 +05:30
bibinchundatt b8de78c570 YARN-9286. [Timeline Server] Sorting based on FinalStatus shows pop-up message. Contributed by Bilwa S T. 2019-02-20 01:19:20 +05:30
Sunil G db4d1a1e2f YARN-9060. [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example. Contributed by Zhankun Tang. 2019-02-18 15:58:04 +05:30
Akira Ajisaka 0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
Sunil G ba56bc24f5 YARN-9213. RM Web UI v1 does not show custom resource allocations for containers page. Contributed by Szilard Nemeth. 2019-02-17 20:08:20 +05:30
Sunil G 9057aa9fc5 SUBMARINE-1. Move code base of submarine from yarn-applications to top directory. Contributed by Wangda Tan. 2019-02-17 20:03:56 +05:30
Eric Yang 7c1b561e33 YARN-8927. Added support for top level Dockerhub images to trusted registry using library keyword. 
Contributed by Zhankun Tang
 2019-02-15 11:18:07 -05:00
Adam Antal 9385ec45d7
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-02-15 18:47:10 +09:00
Sunil G 5b55f3538c YARN-8295. [UI2] Improve Resource Usage tab error message when there are no data available. Contributed by Charan Hebri. 2019-02-15 12:42:21 +05:30
Akira Ajisaka 3a39d9a2d2
YARN-9284. Fix the unit of yarn.service.am-resource.memory in the document. Contributed by Masahiro Tanaka. 2019-02-15 15:41:53 +09:00
bibinchundatt b66d5ae9e2 YARN-9295. [UI2] Fix label typo in Cluster Overview page. Contributed by Charan Hebri. 2019-02-14 23:08:19 +05:30
Sunil G dfe0f42835 YARN-7824. [UI2] Yarn Component Instance page should include link to container logs. Contributed by Akhil PB. 2019-02-14 15:42:19 +05:30
Eric Yang 3dc2523266 YARN-9184. Add a system flag to allow update to latest docker images. 
Contributed by Zhaohui Xin
 2019-02-12 16:16:35 -05:00
Giovanni Matteo Fumarola 7536488bbd YARN-996. REST API support for node resource configuration. Contributed by Inigo Goiri. 2019-02-11 15:27:43 -08:00
Suma Shivaprasad 1ce2e91c4b YARN-9229. Document docker registry deployment with NFS Gateway. Contributed by Eric Yang. 2019-02-11 14:54:30 -08:00
Weiwei Yang 0a1637c750 YARN-8555. Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options. Contributed by Prabhu Joseph. 2019-02-11 15:53:50 +08:00
Weiwei Yang 2b7f828d46 YARN-9252. Allocation Tag Namespace support in Distributed Shell. Contributed by Prabhu Joseph. 2019-02-11 14:42:21 +08:00
Masatake Iwasaki e0ab1bdece YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 2019-02-09 00:26:50 +09:00
Eric E Payne d1ca9432dd YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein 2019-02-07 16:38:11 +00:00
Vinayakumar B 3c96a03dea YARN-8498. Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04. Contributed by Ayush Saxena. 2019-02-07 13:02:35 +05:30
Rohith Sharma K S 7fa62e150c YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek Modi. 2019-02-07 09:58:53 +05:30
Rohith Sharma K S 1e5e08d83b YARN-7627. [ATSv2] When passing a non-number as metricslimit, the error message is wrong. Contributed by Charan Hebri. 2019-02-07 09:58:53 +05:30
Weiwei Yang 711d22f166 YARN-9253. Add UT to verify Placement Constraint in Distributed Shell. Contributed by Prabhu Joseph. 2019-02-06 22:36:51 +08:00
Wangda Tan 308f3168fa Make upstream aware of 3.1.2 release 
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
 2019-02-05 14:03:18 -08:00
Suma Shivaprasad 2044967e75 YARN-9246 NPE when executing a command yarn node -status or -states without additional arguments. Contributed by Masahiro Tanaka 2019-02-05 09:39:38 -08:00
Rohith Sharma K S e3ec18b0c4 YARN-6735. Have a way to turn off container metrics from NMs. Contributed by Abhishek Modi. 2019-02-05 13:48:04 +05:30
Sunil G ba38db4f5b YARN-9257. Distributed Shell client throws a NPE for a non-existent queue. Contributed by Charan Hebri. 2019-02-05 13:41:44 +05:30
Weiwei Yang aa7ce50e55 YARN-9275. Add link to NodeAttributes doc in PlacementConstraints document. Contributed by Masatake Iwasaki. 2019-02-05 16:07:41 +08:00
Rohith Sharma K S 571838920d YARN-9149. yarn container -status misses logUrl when integrated with ATSv2. Contributed by Abhishek Modi. 2019-02-05 12:39:10 +05:30
Sunil G 604b2489a9 YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 2019-02-04 12:48:28 +05:30
Weiwei Yang 69bcff380f YARN-9231. TestDistributedShell fix timeout. Contributed by Prabhu Joseph. 2019-02-02 23:27:31 +08:00
Weiwei Yang f20b043a02 YARN-9263. TestConfigurationNodeAttributesProvider fails after Mockito updated. Contributed by Weiwei Yang. 2019-02-02 23:04:34 +08:00
Weiwei Yang 28ad20a711 YARN-9262. TestRMAppAttemptTransitions is failing with an NPE. Contributed by lujie. 2019-02-02 23:02:45 +08:00
Vrushali C 2c13513040 YARN-8549 Adding a NoOp timeline writer and reader plugin classes for ATSv2. Contributed by Prabha Manepalli. 2019-02-01 14:26:50 -08:00
Giovanni Matteo Fumarola f738b397ae YARN-9191. Add cli option in DS to support enforceExecutionType in resource requests. Contributed by Abhishek Modi. 2019-01-31 11:24:15 -08:00
Sunil G 71c49fa60f YARN-9099. GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way. Contributed by Szilard Nemeth. 2019-01-31 09:25:29 +05:30
Akira Ajisaka a3a9ae3cea
YARN-9251. Build failure for -Dhbase.profile=2.0. Contributed by Rohith Sharma K S. 2019-01-30 05:17:04 -08:00
Akira Ajisaka 1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Eric E Payne 04105bbfdb YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 2019-01-29 17:22:24 +00:00
Weiwei Yang 085f0e8ae7 YARN-9086. [CSI] Run csi-driver-adaptor as aux service. Contributed by Weiwei Yang. 2019-01-29 14:53:08 +08:00
Weiwei Yang 4f63ffe444 YARN-9237. NM should ignore sending finished apps to RM during RM fail-over. Contributed by Jiandan Yang. 2019-01-29 10:34:26 +08:00
Eric Yang f5a95f7998 YARN-8901. Fixed restart policy NEVER/ON_FAILURE with component dependency. 
Contributed by Suma Shivaprasad
 2019-01-28 18:10:33 -05:00
Eric Yang 2e636dd3c4 YARN-9074. Consolidate docker removal logic in ContainerCleanup. 
Contributed by Zhaohui Xin
 2019-01-28 18:05:53 -05:00
Eric Yang 1ab69a9543 YARN-9221. Added flag to disable dynamic auxiliary service feature. 
Contributed by Billie Rinaldi
 2019-01-25 19:05:36 -05:00
Jonathan Hung 6cace58e21 YARN-9222. Print launchTime in ApplicationSummary 2019-01-25 13:17:00 -08:00
Rohith Sharma K S 84bb9808a2 YARN-7761. [UI2] Clicking 'master container log' or 'Link' next to 'log' under application's appAttempt goes to Old UI's Log link. Contributed by Akhil PB. 2019-01-25 14:00:35 +05:30
Eric Yang a33ef4fd31 YARN-8867. Added resource localization status to YARN service status call. 
Contributed by Chandni Singh
 2019-01-24 18:43:21 -05:00
Weiwei Yang a4bd64e724 YARN-9116. Capacity Scheduler: implements queue level maximum-allocation inheritance. Contributed by Aihua Xu. 2019-01-24 17:53:43 +08:00
Sunil G c726445990 YARN-8961. [UI2] Flow Run End Time shows 'Invalid date'. Contributed by Akhil PB 2019-01-24 15:01:43 +05:30
Weiwei Yang 721d5c2a5f YARN-8101. Add UT to verify node-attributes in RM nodes rest API. Contributed by Prabhu Joseph. 2019-01-23 17:59:36 +08:00
Weiwei Yang e3e076dc78 YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 2019-01-23 14:14:49 +08:00
Eric Yang 2fa9389c2e YARN-9146. Added REST API to configure auxiliary service. 
Contributed by Billie Rinaldi
 2019-01-22 18:24:43 -05:00
Weiwei Yang d43df31751 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 2019-01-22 10:23:32 +08:00
Weiwei Yang abde1e1f58 YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang. 2019-01-21 16:54:14 +08:00
Takanobu Asanuma 27aa6e8899 HADOOP-16046. [JDK 11] Correct the compiler exclusion of org/apache/hadoop/yarn/webapp/hamlet/** classes for >= Java 9. Contributed by Devaraj K. 2019-01-21 10:54:58 +09:00
Eric Yang 824dfa3b09 YARN-8489. Support "dominant" component concept in YARN service. 
Contributed by Zac Zhou
 2019-01-18 20:23:04 -05:00
Eric Yang 8c7f6b2d4d YARN-9197. Add safe guard against NPE for component instance failure. 
Contributed by kyungwan nam
 2019-01-18 13:49:56 -05:00
Wangda Tan 6d7eedfd28 YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda) 
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
 2019-01-17 14:20:10 -08:00
rahul3 0a46baecd3
YARN-9203. Fix typos in yarn-default.xml. 
This closes #437

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2019-01-17 16:18:33 +09:00
Akira Ajisaka 104ef5df36
YARN-8747. [UI2] YARN UI2 page loading failed due to js error under some time zone configuration. Contributed by collinma. 2019-01-16 14:36:36 +09:00
Vrushali C 713ded6b15 YARN-9150 Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2. Contributed by Sushil Ks 2019-01-15 21:28:10 -08:00
Vrushali C 6be39230a6 YARN-9150 Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2. Contributed by Sushil Ks 2019-01-15 21:25:37 -08:00
Akira Ajisaka 614af50625
YARN-9179. Fix NPE in AbstractYarnScheduler#updateNewContainerInfo. 2019-01-15 12:59:59 +09:00
Jason Lowe 6a923464af YARN-6523. Optimize system credentials sent in node heartbeat responses. Contributed by Manikandan R 2019-01-08 16:54:05 -06:00
Weiwei Yang 0921b706f7 YARN-9037. [CSI] Ignore volume resource in resource calculators based on tags. Contributed by Sunil Govindan. 2019-01-08 14:30:53 +08:00
Giovanni Matteo Fumarola 489411579c YARN-9169. Add metrics for queued opportunistic and guaranteed containers. Contributed by Abhishek Modi. 2019-01-07 15:16:55 -08:00
Wangda Tan 0a01d49917 YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda) 
Change-Id: Id8af27134d3286a7a10d85eda9be25df9689d0e7
 2019-01-07 12:07:26 -08:00
Akira Ajisaka 5db7c49062
YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. 2019-01-07 17:32:28 +09:00
Weiwei Yang 944cf87223 YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed by Wilfred Spiegelenburg. 2019-01-07 15:57:31 +08:00
Weiwei Yang d3321fbed0 Revert "YARN-9178. TestRMAdminCli#testHelp is failing in trunk. Contributed by Abhishek Modi." 
This reverts commit 650b9cbe16.
 2019-01-07 08:58:09 +08:00
Wangda Tan 2c02aa6ec2 YARN-9160. [Submarine] Document 'PYTHONPATH' environment variable setting when using -localization options. (Zhankun Tang via wangda) 
Change-Id: I115b6710eb11501e21ab3e9e7cd1a87f322e5cf8
 2019-01-06 10:25:35 -08:00
Wangda Tan 8f004fe4e0 YARN-9141. [Submarine] JobStatus outputs with system UTC clock, not local clock. (Zac Zhou via wangda) 
Change-Id: Ib2b17cf7860ddc9188f06498086de3326bc09519
 2019-01-06 10:24:29 -08:00
Weiwei Yang 650b9cbe16 YARN-9178. TestRMAdminCli#testHelp is failing in trunk. Contributed by Abhishek Modi. 2019-01-06 11:29:59 +08:00
Giovanni Matteo Fumarola 6e35f7130f YARN-9166. Fix logging for preemption of Opportunistic containers for Guaranteed containers. Contributed by Abhishek Modi. 2019-01-04 11:16:41 -08:00
Rohith Sharma K S 8c6978c3ba YARN-6149. Allow port range to be specified while starting NM Timeline collector manager. Contributed by Abhishek Modi. 2019-01-04 14:07:02 +05:30
Rohith Sharma K S 573b158791 YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi. 2019-01-04 14:01:35 +05:30
Sunil G f4906ac019 YARN-9038. [CSI] Add ability to publish/unpublish volumes on node managers. Contributed by Weiwei Yang. 2019-01-04 12:10:00 +05:30
Eric Yang dfceffa70d YARN-9147. Rmove auxiliary services when manifest file is removed. 
Contributed by Billie Rinaldi
 2019-01-03 12:57:21 -05:00
Weiwei Yang cfe89e6f96 YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie. 2019-01-03 23:56:28 +08:00
Akira Ajisaka 128f340f4f
YARN-8894. Improve InMemoryPlan#toString. Contributed by BELUGA BEHR. 2018-12-28 17:43:47 +09:00
Akira Ajisaka 15fe1b34dd
YARN-8878. Remove StringBuffer from ManagedParentQueue.java. Contributed by BELUGA BEHR. 2018-12-28 17:29:22 +09:00
Sunil G 742fd978b5 YARN-9090. [Submarine] Update submarine installation script document. Contributed by Xun Liu. 2018-12-26 08:26:48 +05:30
Erik Krogen 091ad974cd HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2018-12-24 09:33:59 -08:00
Botong Huang 657aa433e2 YARN 9108. Fix FederationIntercepter merge home and secondary allocate response typo. Contributed by Abhishek Modi. 2018-12-22 12:41:49 -08:00
Akira Ajisaka e98e963e51
YARN-9154. Fix itemization in YARN service quickstart document. Contributed by Ayush Saxena. 2018-12-22 13:25:39 +09:00
Eric Yang f82922dcfa YARN-5168. Added exposed port information for Docker container. 
Contributed by Xun Liu
 2018-12-21 19:44:07 -05:00
Eric Yang ea724181d6 YARN-9132. Added file permission check for auxiliary services manifest file. 
Contributed by Billie Rinaldi
 2018-12-21 14:56:39 -05:00
Weiwei Yang f659485ee8 YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang. 2018-12-21 10:56:42 +08:00
Eric Yang a80d321074 YARN-9152. Improved AuxServices REST API output. 
Contributed by Billie Rinaldi
 2018-12-20 19:21:55 -05:00
Eric Yang 7affa3053c YARN-9131. Updated document usage for dynamic auxiliary service. 
Contributed by Billie Rinaldi
 2018-12-20 18:45:49 -05:00
Inigo Goiri 5df9fb16b9 YARN-9130. Add Bind_HOST configuration for Yarn Web Proxy. Contributed by Rong Tang. 2018-12-19 18:58:00 -08:00
Billie Rinaldi 2e544dc921 YARN-9129. Ensure flush after printing to log plus additional cleanup. Contributed by Eric Yang 2018-12-19 16:45:23 -08:00
Eric Yang e815fd9c49 YARN-9126. Fix container clean up for reinitialization. 
Contributed by Chandni Singh
 2018-12-19 14:55:56 -05:00
Eric Yang ccdd982e51 YARN-9084. Reset container state and defer readiness check for upgrade. 
Contributed by Chandni Singh
 2018-12-18 18:02:03 -05:00
Eric Yang c7a5a4435e YARN-9075. Add ability to register/remove auxiliary service without restart node manager. 
Contributed by Billie Rinaldi
 2018-12-18 17:05:51 -05:00
Billie Rinaldi a6e9d27c29 YARN-9117. Add a check for insecure setup for container terminal. Contributed by Eric Yang 2018-12-18 12:49:10 -08:00
Billie Rinaldi c5c73182e5 YARN-9072. Send exit command to terminate docker exec on connection close. Contributed by Eric Yang 2018-12-18 10:06:33 -08:00
Eric Yang 71e0b0d800 YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator. 
Contributed by Tarun Parimi
 2018-12-17 12:04:25 -05:00
Vidura Mudalige 04c03476c9
YARN-9095. Removed Unused field from Resource: NUM_MANDATORY_RESOURCES 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2018-12-15 16:07:27 +09:00
Billie Rinaldi 4f622337ee YARN-9091. Improve terminal message when connection is refused. Contributed by Eric Yang 2018-12-14 19:55:50 -08:00
Billie Rinaldi 42297f7d99 YARN-8963. Add flag to disable interactive shell. Contributed by Eric Yang 2018-12-14 18:44:36 -08:00
Billie Rinaldi aab310978f YARN-9089. Add Terminal Link to Service component instance page for UI2. Contributed by Eric Yang 2018-12-14 18:38:03 -08:00
Akira Ajisaka 26e55d4ceb
YARN-9122. Add table of contents to YARN Service API document. Contributed by Zhankun Tang. 2018-12-15 10:29:47 +09:00
Eric Yang b2d7204ed0 YARN-9125. Fixed Carriage Return detection in Docker container launch command. 
Contributed by Billie Rinaldi
 2018-12-14 17:52:26 -05:00
Billie Rinaldi ca90b2480e YARN-9032. Support sh shell for interactive container shell at command line. Contributed by Eric Yang 2018-12-14 14:08:22 -08:00
Billie Rinaldi 72e7c6a489 YARN-8962. Add ability to use interactive shell with normal yarn container. Contributed by Eric Yang 2018-12-12 18:18:57 -08:00
Wangda Tan 37eb919c59 YARN-8885. [DevicePlugin] Support NM APIs to query device resource allocation. (Zhankun Tang via wangda) 
Change-Id: I2a9870709b512af1ac6c09c9701d0b3c0791ff32
 2018-12-12 11:45:47 -08:00
Wangda Tan 9fba6cc247 YARN-9112. [Submarine] Support polling applicationId when it's not ready in cluster. (Zhankun Tang via wangda) 
Change-Id: I73d73f3d631b28fb9866faa56571839b13824a97
 2018-12-12 11:44:58 -08:00
Wangda Tan 61bdcb7b2b YARN-9015. [DevicePlugin] Add an interface for device plugin to provide customized scheduler. (Zhankun Tang via wangda) 
Change-Id: Ib2e4ae47a6f29bb3082c1f8520cf5a52ca720979
 2018-12-12 11:44:22 -08:00
Wangda Tan c771fe6e10 YARN-8714. [Submarine] Support files/tarballs to be localized for a training job. (Zhankun Tang via wangda) 
Change-Id: I845131273e52a9d81dbc813ea6d4af06b205e334
 2018-12-12 11:43:23 -08:00
Wangda Tan 8a5361116e YARN-9078. [Submarine] Clean up the code of CliUtils#parseResourcesString. (Zhankun Tang via wangda) 
Change-Id: I21ac5757b6115d55ec3157ba25db1b65bc85a37b
 2018-12-12 11:42:44 -08:00
Sunil G 5a03c8b9ba YARN-9114. [UI2] log service redirect url need to support user name. Contributed by Akhil PB 2018-12-12 14:19:49 +05:30
Haibo Chen fb55e5201e YARN-9008. Extend YARN distributed shell with file localization feature. (Contributed by Peter Bacsko) 2018-12-11 14:13:34 -08:00
Haibo Chen 881230da21 YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth) 2018-12-11 11:41:32 -08:00
Haibo Chen ac578c0e82 YARN-9087. Improve logging for initialization of Resource plugins. (Contributed by Szilard Nemeth) 2018-12-10 11:12:54 -08:00
Haibo Chen 64411a6ff7 YARN-8738. FairScheduler should not parse negative maxResources or minResources values as positive. (Contributed by Szilard Nemeth) 2018-12-10 11:06:50 -08:00
Weiwei Yang 1c09a10e96 YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters. 2018-12-10 11:55:24 +08:00
Billie Rinaldi 154449fbd8 YARN-8914. Add xtermjs to YARN UI2. Contributed by Eric Yang and Akhil PB 2018-12-07 10:56:17 -08:00
Jonathan Hung cc51607ccd YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics 2018-12-07 10:45:37 -08:00
Haibo Chen 5d4a432660 YARN-9025. TestFairScheduler#testChildMaxResources is flaky. (Contributed by Szilard Nemeth) 2018-12-05 16:36:13 -08:00
Haibo Chen 912b1f9d64 YARN-9019. Ratio calculation of ResourceCalculator implementations could return NaN. (Contributed by Szilard Nemeth) 2018-12-05 15:16:41 -08:00
Haibo Chen b3c75c1f1d YARN-8985. Improve debug log in FSParentQueue when assigning container. (Contributed by Wilfred Spiegelenburg) 2018-12-05 15:02:57 -08:00
Haibo Chen 2bba329c14 YARN-8994. Fix race condition between move app and queue cleanup in Fair Scheduler. (Contributed by Wilfred Spiegelenburg) 2018-12-05 14:38:15 -08:00
Eric Yang 1b790f4dd1 YARN-9071. Improved status update for reinitialized containers. 
Contributed by Chandni Singh
 2018-12-05 17:00:56 -05:00
Eric Yang 1dabb31cdf YARN-9057. Removed third party class bundle from CSI jar file. 
Contributed by Weiwei Yang
 2018-12-05 15:56:10 -05:00
Wangda Tan 228156cfd1 Revert "YARN-8870. [Submarine] Add submarine installation scripts. (Xun Liu via wangda)" 
This reverts commit 46d6e00166.
 2018-12-04 14:13:06 -08:00
Yufei Gu e89941fdbb YARN-9041. Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt. Contributed by Wanqiang Ji. 2018-12-04 10:08:45 -08:00
Jonathan Hung 6d7b44c489 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:13:52 -08:00
Wangda Tan bad12031f6 YARN-9010. Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl. (Zhankun Tang via wangda) 
Change-Id: Iaecc66d57781cc10f19ead4647e47fc9556676da
 2018-11-29 14:56:07 -08:00
bibinchundatt 07142f54a8 YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T. 2018-11-29 22:02:59 +05:30
bibinchundatt a68d766e87 YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt. 2018-11-29 21:43:34 +05:30
Weiwei Yang efc4d91cbe YARN-9067. YARN Resource Manager is running OOM because of leak of Configuration Object. Contributed by Eric Yang. 2018-11-29 09:34:14 +08:00
Wangda Tan fe7dab8ef5 YARN-8989. [YARN-8851] Move DockerCommandPlugin volume related APIs' invocation from DockerLinuxContainerRuntime#prepareContainer to #launchContainer. (Zhankun Tang via wangda) 
Change-Id: Ia6d532c687168448416dfdf46f0ac34bff20e6ca
 2018-11-28 15:03:06 -08:00
Wangda Tan 897643928c YARN-8975. [Submarine] Use predefined Charset object StandardCharsets.UTF_8 instead of String UTF-8. (Zhankun Tang via wangda) 
Change-Id: If6c7904aa17895e543cfca245264249eb7328bdc
 2018-11-28 14:39:06 -08:00
Wangda Tan 8ebeda98a9 YARN-8974. Improve the assertion message in TestGPUResourceHandler. (Zhankun Tang via wangda) 
Change-Id: I4eb58e9d251d5f54e7feffc4fbb813b4f5ae4b1b
 2018-11-28 14:36:30 -08:00
Wangda Tan 9ed87567ad YARN-9061. Improve the GPU/FPGA module log message of container-executor. (Zhankun Tang via wangda) 
Change-Id: Iece9b47438357077a53984a820d4d6423f480518
 2018-11-28 14:31:31 -08:00
Wangda Tan 579ef4be06 YARN-8882. [YARN-8851] Add a shared device mapping manager (scheduler) for device plugins. (Zhankun Tang via wangda) 
Change-Id: I9435136642c3d556971a357bf687f69df90bb45e
 2018-11-28 14:09:52 -08:00
Wangda Tan 9de8e8d049 YARN-9030. Log aggregation changes to handle filesystems which do not support setting permissions. (Suma Shivaprasad via wangda) 
Change-Id: I80f1e8196b8624e24d74494719fdedfd7061dced
 2018-11-28 13:36:21 -08:00
Jason Lowe 3ce99e32f7 YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko 2018-11-28 08:46:11 -06:00
Suma Shivaprasad b3a052d199 YARN-9044. LogsCLI should contact ATSv2 for -am option. Contributed by Rohith Sharma K S 2018-11-28 00:46:53 -08:00
Suma Shivaprasad 7dc272199f YARN-9034. ApplicationCLI should have option to take clusterId. Contributed by Rohith Sharma K S. 2018-11-28 00:42:14 -08:00
Eric Yang f657a2a661 YARN-9056. Improved YARN service upgrade state logic for readiness check. 
Contributed by Chandni Singh
 2018-11-27 18:36:59 -05:00
Eric Yang 33e0df4b35 YARN-8986. Added port publish for Docker container running with bridge network. 
Contributed by Charo Zhang
 2018-11-26 19:45:05 -05:00
Giovanni Matteo Fumarola 0d8406135f YARN-9054. Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest. Contributed by Bibin A Chundatt. 2018-11-26 10:57:38 -08:00
Rohith Sharma K S e223a790a7 YARN-9042. Fix javadoc error in deviceplugin package. Contributed by Zhankun Tang. 2018-11-22 20:04:30 +05:30
Weiwei Yang 176bb3f812 YARN-8984. AMRMClient#OutstandingSchedRequests leaks when AllocationTags is null or empty. Contributed by Yang Wang. 2018-11-22 16:52:29 +08:00
Sunil G 5fe1dbf195 YARN-8964. [UI2] YARN ui2 should use clusters/{cluster name} for all ATSv2 REST APIs. Contributed by Akhil PB. 2018-11-22 10:25:22 +05:30
Haibo Chen a41b648e98 YARN-8992. Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue. (Contributed by Wilfred Spiegelenburg) 2018-11-20 14:12:37 -08:00
Billie Rinaldi 49824ed260 YARN-8838. Check that container user is same as websocket user for interactive shell. Contributed by Eric Yang 2018-11-20 11:12:24 -08:00
Sunil G 5fb14e0635 YARN-8953. [CSI] CSI driver adaptor module support in NodeManager. Contributed by Weiwei Yang. 2018-11-20 08:02:11 +05:30
Billie Rinaldi 27ffec7ba7 YARN-8778. Add command line interface to invoke interactive docker shell. Contributed by Eric Yang 2018-11-19 17:59:12 -08:00
Wangda Tan 8b23814415 YARN-8960. [Submarine] Can't get submarine service status using the command of "yarn app -status" under security environment. (Zac Zhou via wangda) 
Change-Id: I21b1addc9c32817650ea744a8f2e6b5602f2f4d4
 2018-11-19 08:55:30 -08:00
Wangda Tan 6357803645 YARN-8881. [YARN-8851] Add basic pluggable device plugin framework. (Zhankun Tang via wangda) 
Change-Id: If9a2f68cd4713b4ec932cdeda68106f17437c3d3
 2018-11-19 08:54:31 -08:00
Weiwei Yang d027a24f03 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 2018-11-18 23:18:26 +08:00
Giovanni Matteo Fumarola abd6d48c46 YARN-8860. Federation client intercepter class contains unwanted character. Contributed by Abhishek Modi. 2018-11-16 10:23:06 -08:00
Rohith Sharma K S ee3355be3c YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report. 2018-11-16 17:58:19 +05:30
Inigo Goiri eeaf8edaa7 YARN-8856. TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError. Contributed by Sushil Ks. 2018-11-15 09:25:25 -08:00
Eric Yang 21ec4bdaef YARN-8672. Improve token filename management for localization. 
Contributed by Chandni Singh
 2018-11-14 15:22:01 -05:00
Wangda Tan fcbd205cc3 YARN-9001. [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs. (Zac Zhou via wangda) 
Change-Id: Ic3d6c1e439df9cdf74448b345b925343224efe51
 2018-11-13 13:13:27 -08:00
Wangda Tan 076b795b2e YARN-8918. [Submarine] Correct method usage of str.subString in CliUtils. (Zhankun Tang via wangda) 
Change-Id: Id1f11dbab3aa838dee3a0ec8b8fd5dc32f5dd946
 2018-11-13 11:24:15 -08:00
Eric Yang 4c465f5535 YARN-8761. Service AM support for decommissioning component instances. 
Contributed by Billie Rinaldi
 2018-11-12 19:53:10 -05:00
Giovanni Matteo Fumarola e269c3fb5a YARN-8997. [Submarine] Small refactors of modifier, condition check and redundant local variables. Contributed by Zhankun Tang. 2018-11-12 15:06:43 -08:00
Billie Rinaldi 1f9c4f32e8 YARN-8776. Implement Container Exec feature in LinuxContainerExecutor. Contributed by Eric Yang 2018-11-12 10:42:30 -08:00
Sunil G 42f3a7082a YARN-8877. [CSI] Extend service spec to allow setting resource attributes. Contributed by Weiwei Yang. 2018-11-12 21:09:30 +05:30
Weiwei Yang c741109522 YARN-8987. Usability improvements node-attributes CLI. Contributed by Bibin A Chundatt. 2018-11-12 18:18:23 +08:00
Sunil G 4e7284443e YARN-8902. [CSI] Add volume manager that manages CSI volume lifecycle. Contributed by Weiwei Yang. 2018-11-12 11:57:02 +05:30
Botong Huang b5ec85d966 YARN-8933. [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor. Contributed by Botong Huang. 2018-11-11 11:12:53 -08:00
Eric Yang 2664248797 YARN-9002. Improve keytab loading for YARN Service. 
Contributed by Gour Saha
 2018-11-10 01:52:19 -05:00
Haibo Chen 524a7523c4 YARN-8990. Fix fair scheduler race condition in app submit and queue cleanup. (Contributed by Wilfred Spiegelenburg) 2018-11-08 16:02:48 -08:00
Weiwei Yang e1bbf7dcdf YARN-8988. Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled. Contributed by Tao Yang. 2018-11-08 17:47:18 +08:00
Weiwei Yang f8c72d7b3a YARN-8880. Add configurations for pluggable plugin framework. Contributed by Zhankun Tang. 2018-11-08 12:23:00 +08:00
Weiwei Yang c96cbe8659 YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji. 2018-11-07 22:42:22 +08:00
Takanobu Asanuma 8dc1f6dbf7 YARN-8866. Fix a parsing error for crossdomain.xml. 2018-11-07 18:26:07 +09:00
Weiwei Yang 482716e5a4 YARN-8976. Remove redundant modifiers in interface ApplicationConstants. Contributed by Zhankun Tang. 2018-11-07 10:48:07 +08:00
Akira Ajisaka 951c98f890
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-07 11:17:35 +09:00
Eric Yang 08d69d91f2 YARN-8957. Add Serializable interface to ComponentContainers. 
Contributed by Zhankun Tang
 2018-11-06 14:46:05 -05:00
Jason Lowe ab6aa4c726 YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg 2018-11-06 08:40:59 -06:00
Weiwei Yang 5d6554c722 YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang. 2018-11-06 14:50:09 +08:00
Weiwei Yang c7fcca0d7e YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji. 2018-11-06 13:14:57 +08:00
bibinchundatt f84a278baa YARN-8905. [Router] Add JvmMetricsInfo and pause monitor. Contributed by Bilwa S T. 2018-11-03 20:35:31 +05:30
Giovanni Matteo Fumarola 989715ec50 YARN-8893. [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client. Contributed by Botong Huang. 2018-11-02 15:30:08 -07:00
Giovanni Matteo Fumarola aed836efbf YARN-8897. LoadBasedRouterPolicy throws NPE in case of sub cluster unavailability. Contributed by Bilwa S T. 2018-11-02 11:27:11 -07:00
Giovanni Matteo Fumarola babc946d40 YARN-8954. Reservations list field in ReservationListInfo is not accessible. Contributed by Oleksandr Shevchenko. 2018-11-02 11:10:08 -07:00
Jonathan Hung 2ab611d48b YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 2018-11-01 14:22:00 -07:00
Steve Loughran d174b91635 HADOOP-15895. [JDK9+] Add missing javax.annotation-api dependency to hadoop-yarn-csi. 
Contributed by Takanobu Asanuma.
 2018-11-01 12:56:20 +00:00
Akira Ajisaka f747f5b06c
HADOOP-15886. Fix findbugs warnings in RegistryDNS.java. 2018-10-31 10:01:31 +09:00
Shane Kumpf 277a3d8d9f YARN-6729. Clarify documentation on how to enable cgroup support. Contributed by Zhankun Tang 2018-10-30 11:36:55 -06:00
Sunil G d36012b69f YARN-8854. Upgrade jquery datatable version references to v1.10.19. Contributed by Akhil PB. 2018-10-30 22:56:46 +05:30
Rohith Sharma K S 486b9a4a75 YARN-7754. [Atsv2] Update document for running v1 and v2 TS. Contributed by Suma Shivaprasad. 2018-10-30 11:35:01 +05:30
Rohith Sharma K S a283da2167 YARN-8871. Document ATSv2 integrated LogWebService. Contributed by Suma Shivaprasad. 2018-10-30 11:34:20 +05:30
Rohith Sharma K S 4ec4ec6971 YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0. 2018-10-30 11:30:08 +05:30
Weiwei Yang 78ea897b67 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 2018-10-29 12:04:32 +08:00
Weiwei Yang 1d90a0dd23 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 2018-10-29 11:53:10 +08:00
Billie Rinaldi d07e873b7d YARN-8569. Create an interface to provide cluster information to application. Contributed by Eric Yang 2018-10-26 17:57:05 -07:00
Robert Kanter f76e3c3db7 YARN-8930. CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController (haibochen via rkanter) 2018-10-25 11:09:47 -07:00
Weiwei Yang ddc1e0b727 YARN-8941. Fix findbugs warnings in hadoop-yarn-csi. Contributed by Weiwei Yang. 2018-10-25 07:55:51 +08:00
Eric Yang c16c49b8c3 YARN-8587. Added retries for fetching docker exit code. 
Contributed by Charo Zhang
 2018-10-24 17:29:46 -04:00
Robert Kanter 69b328943e YARN-8929. DefaultOOMHandler should only pick running containers to kill upon oom events (haibochen via rkanter) 2018-10-24 13:15:50 -07:00
Eric Yang bbc6dcd3d0 YARN-8938. Updated YARN service upgrade document. 
Contributed by Chandni Singh
 2018-10-24 11:51:44 -04:00
Haibo Chen 766b78ee07 YARN-8911. ContainerScheduler incorrectly uses percentage number as the cpu resource utlization. 2018-10-24 07:58:26 -07:00
Weiwei Yang 553044818d YARN-8939. Javadoc build fails in hadoop-yarn-csi. Contributed by Weiwei Yang. 2018-10-24 18:02:45 +08:00
Eric Yang a0c0b793c3 YARN-8814. Yarn Service Upgrade: Update the swagger definition. 
Contributed by Chandni Singh
 2018-10-23 19:04:58 -04:00
Haibo Chen 823bb5dda8 MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter) 2018-10-23 15:28:13 -07:00
Jason Lowe 93fb3b4b9c YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg 2018-10-23 12:49:15 -05:00
Rohith Sharma K S 0b62983c5a YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli. 2018-10-23 12:58:41 +05:30
Sunil G 0efddd85f0 YARN-8873. [YARN-8811] Add CSI java-based client library. Contributed by Weiwei Yang. 2018-10-23 11:46:06 +05:30
Eric Yang e98a506227 YARN-8923. Cleanup references to ENV file type in YARN service code. 
Contributed by Suma Shivaprasad
 2018-10-22 18:39:57 -04:00
Eric Yang 292c9e017f YARN-8542. Added YARN service REST API to list containers. 
Contributed by Chandni Singh
 2018-10-22 15:59:52 -04:00
Eric Yang 66e7a2c787 YARN-8922. Fixed test-container-executor test setup and clean up. 
Contributed by Robert Kanter
 2018-10-22 15:28:58 -04:00
Sunil G e50334513c YARN-7502. Nodemanager restart docs should describe nodemanager supervised property. Contributed by Suma Shivaprasad. 2018-10-22 14:15:51 +05:30
Eric Yang e2a9fa8448 HADOOP-15821. Move YARN Registry to Hadoop Registry. 
Contributed by Íñigo Goiri
 2018-10-19 19:46:48 -04:00
Eric Yang 47ad98b2e1 YARN-8910. Fixed misleading log statement when container max retries is infinite. 
Contributed by Chandni Singh
 2018-10-19 13:49:04 -04:00
Wangda Tan 5e02b4915b YARN-8916. Define a constant docker string in ContainerRuntimeConstants.java for better maintainability. (Zhankun Tang via wangda) 
Change-Id: I1349e740037f81afdbe30edbe741f20e88fd0a90
 2018-10-19 09:49:26 -07:00
Wangda Tan 071b9e19aa YARN-8908. Fix errors in yarn-default.xml related to GPU/FPGA. (Zhankun Tang via wangda) 
Change-Id: I46f511a9b51b65b069e15d5b827530110c41edab
 2018-10-19 09:49:26 -07:00
Haibo Chen 285d2c0753 YARN-8449. RM HA for AM web server HTTPS Support. (Contributed by Robert Kanter) 2018-10-18 21:24:36 -07:00
Weiwei Yang 13cc0f50ea YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang. 2018-10-19 09:55:59 +08:00
Eric Yang 66f059ed1d YARN-8618. Added detection for non-upgradable service. 
Contributed by Chandni Singh
 2018-10-18 19:59:11 -04:00
Wangda Tan 780be14f07 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
 2018-10-18 10:58:21 -07:00
Wangda Tan a457a8951a YARN-8456. Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable. (Zhankun Tang via wangda) 
Change-Id: Iff150ea98ba0c60d448474fd940eb121afce6965
 2018-10-18 10:57:11 -07:00
Wangda Tan bfb88b10f4 YARN-6098. Add documentation for Delete Queue. (Suma Shivaprasad via wangda) 
Change-Id: I23bf7ca0acbe1168a7f805ceccdd62fe41bfeb35
 2018-10-18 10:55:39 -07:00
Eric Yang ba7e81667c YARN-8687. Update YARN service file type in documentation. 
Contributed by Suma Shivaprasad
 2018-10-18 12:02:10 -04:00
Haibo Chen 32fe351bb6 YARN-8864. NM incorrectly logs container user as the user who sent a start/stop container request in its audit log. (Contributed by Wilfred Spiegelenburg) 2018-10-18 08:28:07 -07:00
Sunil G 2202e00ba8 YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh. 2018-10-18 15:23:20 +05:30
Sunil G 5085e5fa9e YARN-8759. Copy of resource-types.xml is not deleted if test fails, causes other test failures. Contributed by Antal Bálint Steinbach. 2018-10-17 16:05:08 +05:30
Haibo Chen 22f85f2927 YARN-8582. Document YARN support for HTTPS in AM Web server. (Contributed by Robert Kanter) 2018-10-16 17:23:33 -07:00
Eric Yang 3bfd214a59 YARN-8810. Fixed a YARN service bug in comparing ConfigFile object. 
Contributed by Chandni Singh
 2018-10-16 18:54:40 -04:00