e7d44e48f7
YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar
2019-09-06 10:17:07 -07:00
0ccf4b0fe1
YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta
2019-09-05 14:01:42 -07:00
03489124ea
YARN-9797. LeafQueue#activateApplications should use resourceCalculator#fitsIn. Contributed by Bilwa S T.
2019-09-03 11:53:34 +05:30
f4d6e82075
YARN-9400. Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId. Contributed by Prabhu Joseph.
2019-09-02 10:58:23 +05:30
b939ae98e6
YARN-8174. Add containerId to ResourceLocalizationService fetch failure log statement. Contributed by Prabhu Joseph.
2019-09-02 10:45:59 +05:30
bc27f7fb6d
YARN-7982. Do ACLs check while retrieving entity-types per application. Contribued by Prabhu Joseph.
2019-09-02 10:15:59 +05:30
751b5a1ac8
YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph.
2019-09-01 22:40:40 +05:30
c7ef4fb74a
YARN-8678. Queue Management API - rephrase error messages. Contributed by Prabhu Joseph.
2019-09-01 10:00:04 +05:30
d2d963f3d4
YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam.
2019-09-01 09:54:46 +05:30
7d998cb627
YARN-9800. TestRMDelegationTokens can fail in testRemoveExpiredMasterKeyInRMStateStore. Contributed by Adam Antal.
2019-08-31 10:03:09 +05:30
eeccd2f6f6
YARN-9798. ApplicationMasterServiceTestBase#testRepeatedFinishApplicationMaster fails intermittently. Contributed by Tao Yang.
2019-08-31 09:57:45 +05:30
c929b383f8
YARN-9540. TestRMAppTransitions fails intermittently. Contributed by Tao Yang.
2019-08-30 17:17:43 +05:30
5b557de721
YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang.
2019-08-30 10:35:09 +05:30
8c0759d02a
YARN-9664. Improve response of scheduler/app activities for better understanding. Contributed by Tao Yang.
2019-08-29 18:14:39 +08:00
6f2226a013
YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06).
2019-08-28 20:40:34 +00:00
8ef46595da
YARN-9438. launchTime not written to state store for running applications
2019-08-27 15:53:43 -07:00
717c853873
YARN-9755. Fixed RM failing to start when FileSystemBasedConfigurationProvider is configured.
...
Contributed by Prabhu Joseph
2019-08-27 13:14:59 -04:00
d70f5231a7
YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt.
2019-08-27 08:38:12 +05:30
8660e48ca1
YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph
2019-08-26 15:50:33 -07:00
d3ce53e507
YARN-9642. Fix Memory Leak in AbstractYarnScheduler caused by timer. Contributed by Bibin A Chundatt.
2019-08-26 23:21:33 +05:30
689d2e6105
YARN-8917. Absolute (maximum) capacity of level3+ queues is wrongly calculated for absolute resource. Contributed by Tao Yang.
2019-08-26 21:06:15 +05:30
7ab88dbfa6
YARN-7291. Better input parsing for resource in allocation file. Contributed by Zoltan Siegl
2019-08-21 17:01:18 +02:00
e8fa192f07
YARN-9217. Nodemanager will fail to start if GPU is misconfigured on the node or GPU drivers missing. Contributed by Peter Bacsko
2019-08-21 16:44:22 +02:00
e684b17e6f
YARN-5857. TestLogAggregationService.testFixedSizeThreadPool fails intermittently on trunk. Contributed by Bilwa S T.
2019-08-21 17:14:42 +05:30
0e0ddfaf24
YARN-2599. Standby RM should expose jmx endpoint. Contributed by Rohith Sharma K S.
2019-08-17 15:43:19 +05:30
9b8359bb08
YARN-9461. TestRMWebServicesDelegationTokenAuthentication.testCancelledDelegationToken fails with HTTP 400. Contributed by Peter Bacsko
2019-08-16 12:31:58 +02:00
4456ea67b9
YARN-8586. Extract log aggregation related fields and methods from RMAppImpl. Contributed by Peter Bacsko
2019-08-16 11:36:14 +02:00
2216ec54e5
YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko
2019-08-16 09:13:20 +02:00
2a05e0ff3b
YARN-9749. TestAppLogAggregatorImpl#testDFSQuotaExceeded fails on trunk. Contributed by Adam Antal
2019-08-16 08:52:09 +02:00
22c4f38c4b
YARN-9679. Regular code cleanup in TestResourcePluginManager ( #1122 )
2019-08-15 17:32:05 +02:00
1845a83cec
YARN-9488. Skip YARNFeatureNotEnabledException from ClientRMService. Contributed by Prabhu Joseph
2019-08-15 17:15:38 +02:00
167acd87da
YARN-9683. Remove reapDockerContainerNoPid left behind by YARN-9074 ( #1212 ) Contributed by Kevin Su.
...
Reviewed-by: Eric Yang <eyang@apache.org>
Reviewed-by: Adam Antal <adam.antal@cloudera.com>
2019-08-14 10:42:29 -07:00
c89bdfacc8
YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl… ( #1261 )
...
* YARN-9676. Add DEBUG and TRACE level messages to AppLogAggregatorImpl and connected classes
* Using {} placeholder, and increasing loglevel if log aggregation failed.
2019-08-14 17:35:16 +02:00
3e0410449f
YARN-9133. Make tests more easy to comprehend in TestGpuResourceHandler. Contributed by Peter Bacsko
2019-08-14 17:13:54 +02:00
e5e609384f
YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko
2019-08-14 16:58:22 +02:00
89a53c7eb4
YARN-9747. Reduce additional namenode call by EntityGroupFSTimelineStore#cleanLogs. Contributed by Prabhu Joseph.
2019-08-14 13:46:23 +05:30
2ac029b949
YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan.
2019-08-13 16:21:18 +00:00
b4097b96a3
YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph.
2019-08-13 19:04:00 +05:30
e4b538bbda
YARN-9723. ApplicationPlacementContext is not required for terminated jobs during recovery. Contributed by Prabhu Joseph
2019-08-12 15:15:43 +02:00
13a5803ccf
YARN-9464. Support pending resource metrics in RM's RESTful API. Contributed by Prabhu Joseph.
2019-08-12 14:31:24 +05:30
8fbf8b2eb0
YARN-9722. PlacementRule logs object ID in place of queue name. Contributed by Prabhu Joseph.
2019-08-12 10:44:46 +05:30
6ff0453ede
YARN-9527. Prevent rogue Localizer Runner from downloading same file repeatly.
...
Contributed by Jim Brennan
2019-08-09 14:12:17 -04:00
a79564fed0
YARN-9732. yarn.system-metrics-publisher.enabled=false is not honored by RM. Contributed by KWON BYUNGCHANG.
2019-08-09 22:25:30 +05:30
e0c21c6da9
YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak
2019-08-09 10:18:34 +02:00
742e30b473
YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak
2019-08-09 09:59:19 +02:00
72d7e570a7
YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak
2019-08-09 09:49:18 +02:00
3b38f2019e
YARN-9685: NPE when rendering the info table of leaf queue in non-accessible partitions. Contributed by Tao Yang.
2019-08-08 12:37:50 +00:00
22d7d1f8bf
YARN-9601.Potential NPE in ZookeeperFederationStateStore#getPoliciesConfigurations ( #908 ) Contributed by hunshenshi.
2019-08-07 21:26:14 -07:00
f51702d539
YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung)
2019-08-06 13:52:30 -07:00
7c2042a44d
YARN-9678. Addendum: TestGpuResourceHandler / TestFpgaResourceHandler should be renamed. Contributed by kevin su.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-06 10:21:55 -07:00
b8bf09ba3d
YARN-9678. TestGpuResourceHandler / TestFpgaResourceHandler should be renamed. Contributed by kevin su.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-06 09:05:53 -07:00
d6697da5e8
YARN-9667. Use setbuf with line buffer to reduce fflush complexity in container-executor.
...
Contributed by Peter Bacsko
2019-08-05 13:59:12 -04:00
54ac80176e
Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph
2019-08-02 13:38:06 +02:00
1930a7bf60
YARN-9093. Remove commented code block from the beginning of Tes… ( #444 )
2019-08-02 13:16:19 +02:00
95fc38f2e9
YARN-9375. Use Configured in GpuDiscoverer and FpgaDiscoverer ( #1131 )
...
Contributed by Adam Antal
2019-08-02 11:24:09 +02:00
42683aef1a
YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan.
2019-07-30 18:58:36 +00:00
c34ceb5fde
YARN-9568. Fixed NPE in MiniYarnCluster during FileSystemNodeAttributeStore.recover.
...
Contributed by Steve Loughran
2019-07-18 12:30:53 -04:00
5915c902aa
YARN-9646. DistributedShell tests failed to bind to a local host name. (Contributed by Ray Yang)
2019-07-16 17:36:49 -07:00
7a93be0f60
YARN-9645. Fix Invalid event FINISHED_CONTAINERS_PULLED_BY_AM at NEW on NM restart. Contributed by Bilwa S T.
2019-07-16 14:03:22 +05:30
18ee1092b4
YARN-9127. Create more tests to verify GpuDeviceInformationParser. Contributed by Peter Bacsko
2019-07-15 11:59:11 +02:00
91ce09e706
YARN-9360. Do not expose innards of QueueMetrics object into FSLeafQueue#computeMaxAMResource. Contributed by Peter Bacsko
2019-07-15 10:47:20 +02:00
61b0c2bb7c
YARN-9337. GPU auto-discovery script runs even when the resource is given by hand. Contributed by Adam Antal
2019-07-12 17:28:14 +02:00
8b3c6791b1
YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko
2019-07-12 17:20:42 +02:00
c416284bb7
YARN-9235. If linux container executor is not set for a GPU cluster GpuResourceHandlerImpl is not initialized and NPE is thrown. Contributed by Antal Balint Steinbach, Adam Antal
2019-07-12 16:51:58 +02:00
9b54dd7186
YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonathan Hung)
2019-07-11 13:57:08 -07:00
ccaa99c923
HADOOP-16381. The JSON License is included in binary tarball via azure-documentdb:1.16.2. Contributed by Sushil Ks.
2019-07-11 13:49:42 +09:00
a2a8be18cb
YARN-9629. Support configurable MIN_LOG_ROLLING_INTERVAL. Contributed by Adam Antal.
2019-07-03 13:45:00 +02:00
15d82fcb75
YARN-9658. Fix UT failures in TestLeafQueue. Contributed by Tao Yang.
2019-07-03 12:08:45 +08:00
e966edd025
YARN-9644. First RMContext object is always leaked during switch over. Contributed by Bibin A Chundatt.
2019-07-02 12:18:16 +05:30
570eee30e5
YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. Contributed by hunshenshi.
2019-07-02 09:55:25 +08:00
b1dafc3506
YARN-9661:Fix typo in LocalityMulticastAMRMProxyPolicy.java and AbstractConfigurableFederationPolicy.java ( #1042 )
2019-07-01 10:46:33 -07:00
29465bf169
YARN-9560. Restructure DockerLinuxContainerRuntime to extend OCIContainerRuntime.
...
Contributed by Eric Badger, Jim Brennan, Craig Condit
2019-06-28 17:18:53 -04:00
f09c31a97e
Revert "YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023"
...
This reverts commit 5e7caf1287
2019-06-29 00:29:17 +08:00
5e7caf1287
YARN-9655. AllocateResponse in FederationInterceptor lost applicationPriority. (hunshenshi via wwei) closes apache/hadoop#1023
2019-06-29 00:08:40 +08:00
cbae241320
YARN-9623. Auto adjust max queue length of app activities to make sure activities on all nodes can be covered. Contributed by Tao Yang.
2019-06-28 23:24:53 +08:00
be80334cdf
YARN-9639. DecommissioningNodesWatcher cause memory leak. Contributed by Bilwa S T.
2019-06-27 09:59:44 +05:30
1ac967a6b7
YARN-6055. ContainersMonitorImpl need be adjusted when NM resource changed. Contributed by Inigo Goiri.
2019-06-26 14:01:31 -07:00
062eb605ac
YARN-9477. Implement VE discovery using libudev. Contributed by Peter Bacsko.
2019-06-26 23:53:14 +08:00
b220ec6f61
YARN-9374. Improve Timeline service resilience when HBase is unavailable.
...
Contributed by Prabhu Joseph and Szilard Nemeth
2019-06-24 12:19:14 -04:00
83dcb9d87e
YARN-9209. When nodePartition is not set in Placement Constraints, containers are allocated only in default partition. Contributed by Tarun Parimi.
2019-06-21 17:41:05 +08:00
67414a1a80
YARN-9584. Should put initializeProcessTrees method call before get pid. Contributed by Wanqiang Ji.
2019-06-18 12:23:52 +08:00
304a47e22c
YARN-9608. DecommissioningNodesWatcher should get lists of running applications on node from RMNode. Contributed by Abhishek Modi.
2019-06-17 17:09:56 +08:00
cda9f33745
YARN-8499 ATSv2 Generalize TimelineStorageMonitor.
...
Contributed by Prabhu Joseph
2019-06-14 18:59:14 -04:00
3ba090f436
HADOOP-16366. Fixed ProxyUserAuthenticationFilterInitializer for timeline server.
...
Contributed by Prabhu Joseph
2019-06-14 12:54:16 -04:00
bcfd228336
YARN-9599. TestContainerSchedulerQueuing#testQueueShedding fails intermittently. Contributed by Abhishek Modi.
2019-06-13 11:08:35 -07:00
970b0b0c02
YARN-9578. Add limit/actions/summarize options for app activities REST API. Contributed by Tao Yang.
2019-06-13 10:44:47 +08:00
205dd2d8e1
HADOOP-16367. Fixed MiniYarnCluster AuthenticationFilter initialization.
...
Contributed by Prabhu Joseph
2019-06-12 18:03:33 -04:00
2263ead365
YARN-9557. Application fails in diskchecker when ReadWriteDiskValidator is configured. Contributed by Bilwa S T.
2019-06-11 23:20:28 +05:30
60c95e9b6a
YARN-9565. RMAppImpl#ranNodes not cleared on FinalTransition. Contributed by Bilwa S T.
2019-06-11 23:11:49 +05:30
6d80b9bc3f
YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie.
2019-06-11 22:49:21 +05:30
f7df55f4a8
YARN-9602. Use logger format in Container Executor. Contributed by Abhishek Modi.
2019-06-11 22:29:00 +05:30
9191e08f0a
YARN-9569. Auto-created leaf queues do not honor cluster-wide min/max memory/vcores. Contributed by Craig Condit.
2019-06-10 14:33:24 -07:00
0976392502
YARN-9590. Correct incompatible, incomplete and redundant activities. Contributed by Tao Yang.
2019-06-06 21:59:01 +08:00
294695dd57
HADOOP-16314. Make sure all web end points are covered by the same authentication filter.
...
Contributed by Prabhu Joseph
2019-06-05 18:55:13 -04:00
433e97cd34
YARN-9600. Support self-adaption width for columns of containers table on app attempt page. Contributed by Tao Yang.
2019-06-05 13:55:30 +08:00
d45669cd3c
YARN-7537. Add ability to load hbase config from distributed file system.
...
Contributed by Prabhu Joseph
2019-06-04 19:26:06 -04:00
606061aa14
YARN-9595. FPGA plugin: NullPointerException in FpgaNodeResourceUpdateHandler.updateConfiguredResource(). Contributed by Peter Bacsko.
2019-06-04 09:56:59 +08:00
bd2590d71b
YARN-9580. Fulfilled reservation information in assignment is lost when transferring in ParentQueue#assignContainers. Contributed by Tao Yang.
2019-06-03 22:59:02 +08:00
4530f4500d
YARN-9507. Fix NPE in NodeManager#serviceStop on startup failure. Contributed by Bilwa S T.
2019-06-03 14:09:37 +08:00
2210897609
YARN-9592. Use Logger format in ContainersMonitorImpl. Contributed by Inigo Goiri.
2019-05-31 17:35:49 -07:00
4cb559ea7b
YARN-9027. Fixed LevelDBCacheTimelineStore initialization.
...
Contributed by Prabhu Joseph
2019-05-31 14:31:44 -04:00
e49162f4b3
YARN-9545. Create healthcheck REST endpoint for ATSv2. Contributed by Zoltan Siegl.
2019-05-31 10:28:09 +05:30
7861a5eb1a
YARN-9033. ResourceHandlerChain#bootstrap is invoked twice during NM start if LinuxContainerExecutor enabled. Contributed by Zhankun Tang.
2019-05-31 10:22:26 +05:30
f1552f6edb
YARN-9553. Fix NPE in EntityGroupFSTimelineStore#getEntityTimelines. Contributed by Prabhu Joseph.
2019-05-30 11:42:27 -07:00
30c6dd92e1
YARN-9452. Fix TestDistributedShell and TestTimelineAuthFilterForV2 failures. Contributed by Prabhu Joseph.
2019-05-30 22:32:41 +05:30
abf76ac371
YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles)
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2019-05-29 11:24:08 -05:00
3c63551101
YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph.
2019-05-29 16:05:39 +00:00
544876fe12
YARN-8693. Add signalToContainer REST API for RMWebServices. Contributed by Tao Yang.
2019-05-29 16:34:48 +08:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
9078e28a24
YARN-9503. Fix JavaDoc error in TestSchedulerOvercommit. Contributed by Wanqiang Ji.
2019-05-28 15:52:39 +09:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
9f056d905f
YARN-9497. Support grouping by diagnostics for query results of scheduler and app activities. Contributed by Tao Yang.
2019-05-26 09:56:36 -04:00
460ba7fb14
YARN-9558. Fixed LogAggregation test cases.
...
Contributed by Prabhu Joseph
2019-05-23 18:38:47 -04:00
7b03072fd4
YARN-9080. Added clean up of bucket directories.
...
Contributed by Prabhu Joseph, Peter Bacsko, Szilard Nemeth
2019-05-23 12:08:44 -04:00
12c81610e0
YARN-9505. Add container allocation latency for Opportunistic Scheduler. Contributed by Abhishek Modi.
2019-05-17 12:03:21 -07:00
fab5b80a36
YARN-9554. Fixed TimelineEntity DAO serialization handling.
...
Contributed by Prabhu Joseph
2019-05-16 16:39:50 -04:00
55bd35921c
YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko.
2019-05-15 11:50:46 -07:00
570fa2da20
YARN-9508. YarnConfiguration areNodeLabel enabled is costly in allocation flow. Contributed by Bilwa S T.
2019-05-15 13:30:09 +05:30
2de1e30658
YARN-9547. ContainerStatusPBImpl default execution type is not returned. Contributed by Bilwa S T.
2019-05-15 13:21:39 +05:30
29ff7fb140
YARN-9493. Scheduler Page does not display the right page by query string. Contributed by Wanqiang Ji.
2019-05-13 10:57:12 -07:00
1a47c2b7ae
YARN-9539.Improve cleanup process of app activities and make some conditions configurable. Contributed by Tao Yang.
2019-05-12 22:31:39 -07:00
1b48100a5e
YARN-9522. AppBlock ignores full qualified class name of PseudoAuthenticationHandler. Contributed by Prabhu Joseph.
2019-05-09 14:02:58 -07:00
90add05caa
YARN-9489. Support filtering by request-priorities and allocation-request-ids for query results of app activities. Contributed by Tao Yang.
2019-05-09 21:54:09 +08:00
3172f6cbf9
YARN-9513. Addendum patch: Fix ASF License warnings. Contributed by Giovanni Matteo Fumarola.
2019-05-08 14:56:23 +09:00
c336af3847
YARN-9432. Reserved containers leak after its request has been cancelled or satisfied when multi-nodes enabled. Contributed by Tao Yang.
2019-05-08 09:54:16 +08:00
8ecbf61cca
YARN-9513. [JDK11] Fix TestMetricsInvariantChecker#testManyRuns in case of JDK greater than 8. Contributed by Adam Antal.
2019-05-07 10:59:02 -07:00
597fa47ad1
YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung)
2019-05-06 11:56:22 -07:00
12b7059ddc
YARN-9440. Improve diagnostics for scheduler and app activities. Contributed by Tao Yang.
2019-05-06 20:00:15 +08:00
b094b94d43
YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein.
2019-05-02 19:39:26 +00:00
accb811e57
YARN-6929. Improved partition algorithm for yarn remote-app-log-dir.
...
Contributed by Prabhu Joseph
2019-04-30 17:04:59 -04:00
7fbaa7d66f
YARN-9476. [YARN-9473] Create unit tests for VE plugin. Contributed by Peter Bacsko.
2019-04-30 11:06:44 +08:00
79d3d35398
YARN-9486. Docker container exited with failure does not get clean up correctly. Contributed by Eric Yang
2019-04-26 01:21:28 +00:00
a703dae25e
HADOOP-16222. Fix new deprecations after guava 27.0 update in trunk. Contributed by Gabor Bota.
2019-04-24 10:39:00 -06:00
3f2f4186f6
YARN-9424. Change getDeclaredMethods to getMethods in FederationClientInterceptor#invokeConcurrent. Contributed by Shen Yinjie.
2019-04-23 19:58:41 -07:00
fec9bf4b0b
YARN-9501. TestCapacitySchedulerOvercommit#testReducePreemptAndCancel fails intermittent. Contributed by Prabhu Joseph.
2019-04-23 15:42:56 -07:00
4a0ba24959
YARN-9491. TestApplicationMasterServiceFair#ApplicationMasterServiceTestBase.testUpdateTrackingUrl fails intermittent. Contributed by Prabhu Joseph.
2019-04-23 15:27:04 -07:00
c504eee0c2
YARN-9339. Apps pending metric incorrect after moving app to a new queue. Contributed by Abhishek Modi.
2019-04-23 12:40:44 -07:00
8a95ea61e1
YARN-9475. [YARN-9473] Create basic VE plugin. Contributed by Peter Bacsko.
2019-04-23 17:33:58 +08:00
1c8046d67e
YARN-9325. TestQueueManagementDynamicEditPolicy fails intermittent. Contributed by Prabhu Joseph.
2019-04-23 14:21:13 +08:00
96e3027e46
YARN-2889. Limit the number of opportunistic container allocated per AM heartbeat. Contributed by Abhishek Modi.
2019-04-22 09:49:03 -07:00
aeadb9432f
YARN-9448. Fix Opportunistic Scheduling for node local allocations. Contributed by Abhishek Modi.
2019-04-19 09:41:06 -07:00
ef97a20831
YARN-8622. Fixed container-executor compilation on MacOSX.
...
Contributed by Siyao Meng
2019-04-18 18:59:21 -04:00
df76cdc895
YARN-6695. Fixed NPE in publishing appFinished events to ATSv2.
...
Contributed by Prabhu Joseph
2019-04-18 12:29:37 -04:00
aa4c744aef
YARN-9470. Fix order of actual and expected expression in assert statements
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-18 15:40:37 +09:00
6e4399ea61
YARN-9487. NodeManager native build shouldn't link against librt on macOS. Contributed by Siyao Meng.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-17 22:56:57 -07:00
9cf7401794
YARN-9349. Improved log level practices for InvalidStateTransitionException.
...
Contributed by Anuhan Torgonshar
(cherry picked from commit fe2370e039e1ee980d74769ae85d67434e0993cf)
2019-04-16 19:53:45 -04:00
b8086aed86
YARN-9123. Clean up and split testcases in TestNMWebServices for GPU support. Contributed by Szilard Nemeth.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-16 11:06:25 -07:00
5583e1b6fc
YARN-7848 Force removal of docker containers that do not get removed on first try. Contributed by Eric Yang
2019-04-15 20:47:09 +00:00
254efc9358
YARN-9379. Can't specify docker runtime through environment. Contributed by caozhiqiang
2019-04-15 18:24:37 +00:00
7fa73fac26
YARN-9439. Support asynchronized scheduling mode and multi-node lookup mechanism for app activities. Contributed by Tao Yang.
2019-04-16 00:12:43 +08:00
7a68e7abd5
YARN-9474. Remove hard coded sleep from Opportunistic Scheduler tests. Contributed by Abhishek Modi.
2019-04-14 20:11:20 -07:00
1943db5571
HADOOP-16237. Fix new findbugs issues after updating guava to 27.0-jre.
...
Author: Gabor Bota <gabor.bota@cloudera.com>
2019-04-12 18:28:38 -07:00
ed3747c1cc
YARN-9435. Add Opportunistic Scheduler metrics in ResourceManager. Contributed by Abhishek Modi.
2019-04-11 11:49:19 -07:00
8c1bba375b
YARN-9463. Add queueName info when failing with queue capacity sanity check. Contributed by Aihua Xu.
2019-04-10 22:51:28 +08:00
32722d2661
YARN-9433. Remove unused constants in YARN resource manager
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-04-10 18:37:27 +09:00
cfec455c45
YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Contributed by Inigo Goiri.
2019-04-09 10:59:43 -07:00
fc05b0e70e
YARN-9313. Support asynchronized scheduling mode and multi-node lookup mechanism for scheduler activities. Contributed by Tao Yang.
2019-04-08 13:40:53 +08:00
ec143cbf67
YARN-9413. Queue resource leak after app fail for CapacityScheduler. Contributed by Tao Yang.
2019-04-06 19:59:36 +08:00
22362c876d
YARN-9335 [atsv2] Restrict the number of elements held in timeline collector when backend is unreachable for async calls. Contributed by Abhishesk Modi.
2019-04-05 12:06:51 -07:00
27039a29ae
YARN-9382 Publish container killed, paused and resumed events to ATSv2. Contributed by Abhishesk Modi.
2019-04-05 12:02:43 -07:00
8d150067e2
YARN-9396. Fixed duplicated RM Container created event to ATS.
...
Contributed by Prabhu Joseph
2019-04-04 13:01:56 -04:00
eb03f7c419
YARN-9303 Username splits won't help timelineservice.app_flow table. Contributed by Prabhu Joseph.
2019-04-03 22:53:05 -07:00
002dcc4ebf
YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko.
2019-04-03 18:57:28 +05:30
2f752830ba
YARN-9214. Add AbstractYarnScheduler#getValidQueues method to remove duplication. Contributed by Wanqiang Ji.
2019-04-01 20:05:15 -07:00
ab2bda57bd
YARN-9428. Add metrics for paused containers in NodeManager. Contributed by Abhishek Modi.
2019-04-01 14:21:17 -07:00
da7f8c244d
YARN-9431. Fix flaky junit test fair.TestAppRunnability after YARN-8967. Contributed by Wilfred Spiegelenburg.
2019-04-01 11:21:31 -07:00
332cab5518
YARN-9418. ATSV2 /apps//entities/YARN_CONTAINER rest api does not show metrics. Contributed by Prabhu Joseph.
2019-04-01 11:06:51 -07:00
56f1e131ec
YARN-9270. Minor cleanup in TestFpgaDiscoverer. Contributed by Peter Bacsko.
2019-03-29 10:58:56 -07:00
a4cd75e09c
YARN-9269. Minor cleanup in FpgaResourceAllocator. Contributed by Peter Bacsko.
2019-03-27 10:08:07 -07:00
5257f50abb
YARN-8967. Change FairScheduler to use PlacementRule interface. Contributed by Wilfred Spiegelenburg.
2019-03-25 22:47:24 -07:00
eeda6891e4
YARN-9268. General improvements in FpgaDevice. Contributed by Peter Bacsko.
2019-03-25 13:22:53 -07:00
3c45762a0b
YARN-9391. Fixed node manager environment leaks into Docker containers.
...
Contributed by Jim Brennan
2019-03-25 15:53:24 -04:00
509b20b292
YARN-9404. TestApplicationLifetimeMonitor#testApplicationLifetimeMonitor fails intermittent. Contributed by Prabhu Joseph.
2019-03-22 11:45:39 -07:00
ce5eb9cb2e
YARN-9358. Add javadoc to new methods introduced in FSQueueMetrics with YARN-9322
...
(Contributed by Zoltan Siegl via Daniel Templeton)
Change-Id: I92d52c0ca630e71afb26b2b7587cbdbe79254a05
2019-03-22 12:28:34 +01:00
548997d6c9
YARN-9402. Opportunistic containers should not be scheduled on Decommissioning nodes. Contributed by Abhishek Modi.
2019-03-21 12:04:05 -07:00
a99eb80659
YARN-9267. General improvements in FpgaResourceHandlerImpl. Contributed by Peter Bacsko.
2019-03-21 11:15:56 -07:00
506502bb83
YARN-9370. Added logging for recovering assigned GPU devices.
...
Contributed by Yesha Vora
2019-03-20 19:12:19 -04:00
f2b862cac6
YARN-9398. Fixed javadoc errors for FPGA related java files.
...
Contributed by Peter Bacsko
2019-03-20 15:45:37 -04:00
b3b0e332e6
YARN-9299. TestTimelineReaderWhitelistAuthorizationFilter ignores Http Errors. Contributed by Prabhu Joseph.
2019-03-20 21:24:31 +05:30
0d24684eee
YARN-9357. Modify HBase Liveness monitor log to debug. Contributed by Prabhu Joseph.
2019-03-20 21:22:54 +05:30
c1a4eeb7c8
YARN-9389. FlowActivity and FlowRun table prefix is wrong. Contributed by Prabhu Joseph.
2019-03-20 21:18:19 +05:30
5d8bd0e5cb
YARN-9392. Handle missing scheduler events in Opportunistic Scheduler. Contributed by Abhishek Modi.
2019-03-19 11:00:21 -07:00
09eabda314
YARN-9364. Remove commons-logging dependency from YARN.
...
Contributed by Prabhu Joseph
2019-03-18 19:58:42 -04:00
5f6e225166
YARN-9363. Replaced debug logging with SLF4J parameterized log message.
...
Contributed by Prabhu Joseph
2019-03-18 13:57:18 -04:00
0e7e9013d4
YARN-9340. [Clean-up] Remove NULL check before instanceof in ResourceRequestSetKey
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I932e29b36f086f7b7c76a250e33b473617ddbda1
2019-03-18 15:08:37 +01:00
2064ca015d
YARN-9349. Changed logging to use slf4j api.
...
Contributed by Prabhu Joseph
2019-03-15 19:20:59 -04:00
03f3c8aed2
YARN-4404. Corrected typo in javadoc.
...
Contributed by Yesha Vora
2019-03-15 18:04:04 -04:00
688b177fc6
YARN-8376. Separate white list for docker.trusted.registries and docker.privileged-container.registries. Contributed by Eric Yang
2019-03-14 19:39:00 +00:00
f235a942d5
YARN-9016 DocumentStore as a backend for ATSv2. Contributed by Sushil Ks.
2019-03-13 16:45:23 -07:00
17a3e14d25
YARN-9338 Timeline related testcases are failing. Contributed by Abhishek Modi.
2019-03-12 21:33:17 -07:00
8e1539eca8
YARN-9266. General improvements in IntelFpgaOpenclPlugin. Contributed by Peter Bacsko.
2019-03-13 02:45:17 +05:30
de15a66d78
YARN-9265. FPGA plugin fails to recognize Intel Processing Accelerator Card. Contributed by Peter Bacsko.
2019-03-08 17:39:22 +05:30
39b4a37e02
YARN-9341. Fixed enentrant lock usage in YARN project.
...
Contributed by Prabhu Joseph
2019-03-07 16:47:45 -05:00
491313ab84
YARN-8218 Add application launch time to ATSV1. Contributed by Abhishek Modi
2019-03-06 21:47:29 -08:00
46045c5cb3
YARN-9138. Improve test coverage for nvidia-smi binary execution of GpuDiscoverer. Contributed by Szilard Nemeth.
2019-03-06 16:01:08 +05:30
7b42e0e32a
YARN-7266. Fixed deadlock in Timeline Server thread initialization.
...
Contributed by Prabhu Joseph
2019-03-05 12:17:01 -05:00
0aefe2846f
YARN-9298. Implement FS placement rules using PlacementRule interface. Contributed by Wilfred Spiegelenburg.
2019-03-04 23:49:07 -08:00
e40e2d6ad5
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-03-05 14:10:08 +09:00
15098df744
Revert "YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph."
...
This reverts commit a63c358b78
2019-03-04 16:57:31 +05:30
cab8529ecb
YARN-7904. Privileged, trusted containers should be supported only in ENTRYPOINT mode. Contributed by Eric Yang.
2019-03-01 11:06:09 -08:00
dcaca19871
YARN-9139. Simplify initializer code of GpuDiscoverer. Contributed by Szilard Nemeth.
2019-03-01 19:24:35 +05:30
538bb4880d
YARN-9323. FSLeafQueue#computeMaxAMResource does not override zero values for custom resources
...
(Contributed by Szilard Nemeth via Daniel Templeton)
Change-Id: Id844ccf09488f367c0c7de0a3b2d4aca1bba31cc
2019-02-27 19:59:48 -08:00
7b928f19a4
YARN-9322. Store metrics for custom resource types into FSQueueMetrics and query them in FairSchedulerQueueInfo
...
(Contributed by Szilard Nemeth via Daniel Templeton)
Change-Id: I14c12f1265999d62102f2ec5506d90015efeefe8
2019-02-27 19:43:50 -08:00
1779fc57a1
YARN-9324. TestSchedulingRequestContainerAllocation(Async) fails with junit-4.11. Contributed by Prabhu Joseph.
2019-02-28 09:56:29 +08:00
ea3cdc60b3
YARN-3841 [atsv2 Storage implementation] Adding retry semantics to HDFS backing storage. Contributed by Abhishek Modi.
2019-02-27 14:55:35 -08:00
0ec962ac8f
YARN-5336 Limit the flow name size & consider cleanup for hex chars. Contributed by Sushil Ks
2019-02-27 14:43:39 -08:00
fbc7bb315f
YARN-9245. Added query docker image command ability to node manager.
...
Contributed by Chandni Singh
2019-02-27 14:57:24 -05:00
8c30114b00
YARN-9248. RMContainerImpl:Invalid event: ACQUIRED at KILLED. Contributed by lujie.
2019-02-27 17:29:02 +08:00
6c96f5e4b6
YARN-8378. ApplicationHistoryManagerImpl#getApplications doesn't honor filters. Contributed by Lantao Jin.
2019-02-27 10:32:58 +05:30
8eae260af5
YARN-9311. Fix TestRMRestart hangs. Contributed by Prabhu Joseph.
2019-02-27 10:28:16 +05:30
c6ea28c480
YARN-9331. [YARN-8851] Fix a bug that lacking cgroup initialization when bootstrap DeviceResourceHandlerImpl. Contributed by Zhankun Tang.
2019-02-26 10:05:31 +08:00
95372657fc
YARN-9287. Consecutive StringBuilder append should be reuse. Contributed by Ayush Saxena.
2019-02-25 11:45:37 -08:00
3e1739d589
YARN-9329. updatePriority is blocked when using FairScheduler. Contributed by Jiandan Yang.
2019-02-26 00:08:13 +08:00
5e91ebd91a
YARN-9121. Replace GpuDiscoverer.getInstance() to a readable object for easy access control. Contributed by Szilard Nemeth.
2019-02-25 11:30:46 +05:30
9cd5c5447f
YARN-9316. TestPlacementConstraintsUtil#testInterAppConstraintsByAppID fails intermittently. Contributed by Prabhu Joseph.
2019-02-24 22:42:27 +08:00
50094d7fef
YARN-9300. Lazy preemption should trigger an update on queue preemption metrics for CapacityScheduler. Contributed by Tao Yang.
2019-02-24 22:17:29 +08:00
dddcfa4d9f
YARN-8821. [YARN-8851] GPU hierarchy/topology scheduling support based on pluggable device framework. Contributed by Zhankun Tang.
2019-02-24 14:37:06 +05:30
ed13cf8406
YARN-9317. Avoid repeated YarnConfiguration#timelineServiceV2Enabled check. Contributed by Prabhu Joseph
2019-02-22 21:01:34 +05:30
95fbbfed75
YARN-9118. Handle exceptions with parsing user defined GPU devices in GpuDiscoverer. Contributed by Szilard Nemeth.
2019-02-22 20:22:17 +05:30
9c88695bcd
YARN-9238. Avoid allocating opportunistic containers to previous/removed/non-exist application attempt. Contributed by lujie.
2019-02-22 20:58:33 +08:00
632d5e8a98
YARN-9156. Improve debug message in device plugin method compatibility check of ResourcePluginManager. Contributed by Zhankun Tang.
2019-02-22 11:27:30 +08:00
b17a2602d1
YARN-9319. Fix compilation issue of handling typedef an existing name by gcc compiler. (Zhankun Tang via wangda)
...
Change-Id: Ie8293cc5b710ff355fb6a1ea7201be53ed0385bf
2019-02-21 11:17:32 -08:00
eedcc8e26e
YARN-9315. TestCapacitySchedulerMetrics fails intermittently. Contributed by Prabhu Joseph.
2019-02-21 17:21:21 +08:00
a63c358b78
YARN-8132. Final Status of applications shown as UNDEFINED in ATS app queries. Contributed by Prabhu Joseph.
2019-02-21 13:29:01 +05:30
b8de78c570
YARN-9286. [Timeline Server] Sorting based on FinalStatus shows pop-up message. Contributed by Bilwa S T.
2019-02-20 01:19:20 +05:30
db4d1a1e2f
YARN-9060. [YARN-8851] Phase 1 - Support device isolation and use the Nvidia GPU plugin as an example. Contributed by Zhankun Tang.
2019-02-18 15:58:04 +05:30
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR.
2019-02-18 15:21:46 +09:00
ba56bc24f5
YARN-9213. RM Web UI v1 does not show custom resource allocations for containers page. Contributed by Szilard Nemeth.
2019-02-17 20:08:20 +05:30
7c1b561e33
YARN-8927. Added support for top level Dockerhub images to trusted registry using library keyword.
...
Contributed by Zhankun Tang
2019-02-15 11:18:07 -05:00
9385ec45d7
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-02-15 18:47:10 +09:00
3dc2523266
YARN-9184. Add a system flag to allow update to latest docker images.
...
Contributed by Zhaohui Xin
2019-02-12 16:16:35 -05:00
7536488bbd
YARN-996. REST API support for node resource configuration. Contributed by Inigo Goiri.
2019-02-11 15:27:43 -08:00
0a1637c750
YARN-8555. Parameterize TestSchedulingRequestContainerAllocation(Async) to cover both PC handler options. Contributed by Prabhu Joseph.
2019-02-11 15:53:50 +08:00
e0ab1bdece
YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri.
2019-02-09 00:26:50 +09:00
d1ca9432dd
YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein
2019-02-07 16:38:11 +00:00
3c96a03dea
YARN-8498. Yarn NodeManager OOM Listener Fails Compilation on Ubuntu 18.04. Contributed by Ayush Saxena.
2019-02-07 13:02:35 +05:30
7fa62e150c
YARN-8219. Add application launch time to ATSV2. Contributed by Abhishek Modi.
2019-02-07 09:58:53 +05:30
1e5e08d83b
YARN-7627. [ATSv2] When passing a non-number as metricslimit, the error message is wrong. Contributed by Charan Hebri.
2019-02-07 09:58:53 +05:30
e3ec18b0c4
YARN-6735. Have a way to turn off container metrics from NMs. Contributed by Abhishek Modi.
2019-02-05 13:48:04 +05:30
604b2489a9
YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla.
2019-02-04 12:48:28 +05:30
f20b043a02
YARN-9263. TestConfigurationNodeAttributesProvider fails after Mockito updated. Contributed by Weiwei Yang.
2019-02-02 23:04:34 +08:00
28ad20a711
YARN-9262. TestRMAppAttemptTransitions is failing with an NPE. Contributed by lujie.
2019-02-02 23:02:45 +08:00
2c13513040
YARN-8549 Adding a NoOp timeline writer and reader plugin classes for ATSv2. Contributed by Prabha Manepalli.
2019-02-01 14:26:50 -08:00
71c49fa60f
YARN-9099. GpuResourceAllocator#getReleasingGpus calculates number of GPUs in a wrong way. Contributed by Szilard Nemeth.
2019-01-31 09:25:29 +05:30
a3a9ae3cea
YARN-9251. Build failure for -Dhbase.profile=2.0. Contributed by Rohith Sharma K S.
2019-01-30 05:17:04 -08:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
04105bbfdb
YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph
2019-01-29 17:22:24 +00:00
4f63ffe444
YARN-9237. NM should ignore sending finished apps to RM during RM fail-over. Contributed by Jiandan Yang.
2019-01-29 10:34:26 +08:00
2e636dd3c4
YARN-9074. Consolidate docker removal logic in ContainerCleanup.
...
Contributed by Zhaohui Xin
2019-01-28 18:05:53 -05:00
1ab69a9543
YARN-9221. Added flag to disable dynamic auxiliary service feature.
...
Contributed by Billie Rinaldi
2019-01-25 19:05:36 -05:00
6cace58e21
YARN-9222. Print launchTime in ApplicationSummary
2019-01-25 13:17:00 -08:00
a33ef4fd31
YARN-8867. Added resource localization status to YARN service status call.
...
Contributed by Chandni Singh
2019-01-24 18:43:21 -05:00
a4bd64e724
YARN-9116. Capacity Scheduler: implements queue level maximum-allocation inheritance. Contributed by Aihua Xu.
2019-01-24 17:53:43 +08:00
721d5c2a5f
YARN-8101. Add UT to verify node-attributes in RM nodes rest API. Contributed by Prabhu Joseph.
2019-01-23 17:59:36 +08:00
e3e076dc78
YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang.
2019-01-23 14:14:49 +08:00
2fa9389c2e
YARN-9146. Added REST API to configure auxiliary service.
...
Contributed by Billie Rinaldi
2019-01-22 18:24:43 -05:00
d43df31751
YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang.
2019-01-22 10:23:32 +08:00
abde1e1f58
YARN-9204. RM fails to start if absolute resource is specified for partition capacity in CS queues. Contributed by Jiandan Yang.
2019-01-21 16:54:14 +08:00
6d7eedfd28
YARN-9194. Invalid event: REGISTERED and LAUNCH_FAILED at FAILED, and NullPointerException happens in RM while shutdown a NM. (lujie via wangda)
...
Change-Id: I4359f59a73a278a941f4bb9d106dd38c9cb471fe
2019-01-17 14:20:10 -08:00
713ded6b15
YARN-9150 Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2. Contributed by Sushil Ks
2019-01-15 21:28:10 -08:00
6be39230a6
YARN-9150 Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2. Contributed by Sushil Ks
2019-01-15 21:25:37 -08:00
614af50625
YARN-9179. Fix NPE in AbstractYarnScheduler#updateNewContainerInfo.
2019-01-15 12:59:59 +09:00
6a923464af
YARN-6523. Optimize system credentials sent in node heartbeat responses. Contributed by Manikandan R
2019-01-08 16:54:05 -06:00
0921b706f7
YARN-9037. [CSI] Ignore volume resource in resource calculators based on tags. Contributed by Sunil Govindan.
2019-01-08 14:30:53 +08:00
489411579c
YARN-9169. Add metrics for queued opportunistic and guaranteed containers. Contributed by Abhishek Modi.
2019-01-07 15:16:55 -08:00
0a01d49917
YARN-8822. Nvidia-docker v2 support for YARN GPU feature. (Charo Zhang via wangda)
...
Change-Id: Id8af27134d3286a7a10d85eda9be25df9689d0e7
2019-01-07 12:07:26 -08:00
944cf87223
YARN-9173. FairShare calculation broken for large values after YARN-8833. Contributed by Wilfred Spiegelenburg.
2019-01-07 15:57:31 +08:00
6e35f7130f
YARN-9166. Fix logging for preemption of Opportunistic containers for Guaranteed containers. Contributed by Abhishek Modi.
2019-01-04 11:16:41 -08:00
8c6978c3ba
YARN-6149. Allow port range to be specified while starting NM Timeline collector manager. Contributed by Abhishek Modi.
2019-01-04 14:07:02 +05:30
f4906ac019
YARN-9038. [CSI] Add ability to publish/unpublish volumes on node managers. Contributed by Weiwei Yang.
2019-01-04 12:10:00 +05:30
dfceffa70d
YARN-9147. Rmove auxiliary services when manifest file is removed.
...
Contributed by Billie Rinaldi
2019-01-03 12:57:21 -05:00
cfe89e6f96
YARN-9164. Shutdown NM may cause NPE when opportunistic container scheduling is enabled. Contributed by lujie.
2019-01-03 23:56:28 +08:00
128f340f4f
YARN-8894. Improve InMemoryPlan#toString. Contributed by BELUGA BEHR.
2018-12-28 17:43:47 +09:00
15fe1b34dd
YARN-8878. Remove StringBuffer from ManagedParentQueue.java. Contributed by BELUGA BEHR.
2018-12-28 17:29:22 +09:00
091ad974cd
HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun.
2018-12-24 09:33:59 -08:00
657aa433e2
YARN 9108. Fix FederationIntercepter merge home and secondary allocate response typo. Contributed by Abhishek Modi.
2018-12-22 12:41:49 -08:00
f82922dcfa
YARN-5168. Added exposed port information for Docker container.
...
Contributed by Xun Liu
2018-12-21 19:44:07 -05:00
ea724181d6
YARN-9132. Added file permission check for auxiliary services manifest file.
...
Contributed by Billie Rinaldi
2018-12-21 14:56:39 -05:00
f659485ee8
YARN-8925. Updating distributed node attributes only when necessary. Contributed by Tao Yang.
2018-12-21 10:56:42 +08:00
a80d321074
YARN-9152. Improved AuxServices REST API output.
...
Contributed by Billie Rinaldi
2018-12-20 19:21:55 -05:00
5df9fb16b9
YARN-9130. Add Bind_HOST configuration for Yarn Web Proxy. Contributed by Rong Tang.
2018-12-19 18:58:00 -08:00
2e544dc921
YARN-9129. Ensure flush after printing to log plus additional cleanup. Contributed by Eric Yang
2018-12-19 16:45:23 -08:00
e815fd9c49
YARN-9126. Fix container clean up for reinitialization.
...
Contributed by Chandni Singh
2018-12-19 14:55:56 -05:00
c7a5a4435e
YARN-9075. Add ability to register/remove auxiliary service without restart node manager.
...
Contributed by Billie Rinaldi
2018-12-18 17:05:51 -05:00
a6e9d27c29
YARN-9117. Add a check for insecure setup for container terminal. Contributed by Eric Yang
2018-12-18 12:49:10 -08:00
c5c73182e5
YARN-9072. Send exit command to terminate docker exec on connection close. Contributed by Eric Yang
2018-12-18 10:06:33 -08:00
71e0b0d800
YARN-9040. Fixed memory leak in LevelDBCacheTimelineStore and DBIterator.
...
Contributed by Tarun Parimi
2018-12-17 12:04:25 -05:00
4f622337ee
YARN-9091. Improve terminal message when connection is refused. Contributed by Eric Yang
2018-12-14 19:55:50 -08:00
42297f7d99
YARN-8963. Add flag to disable interactive shell. Contributed by Eric Yang
2018-12-14 18:44:36 -08:00
b2d7204ed0
YARN-9125. Fixed Carriage Return detection in Docker container launch command.
...
Contributed by Billie Rinaldi
2018-12-14 17:52:26 -05:00
ca90b2480e
YARN-9032. Support sh shell for interactive container shell at command line. Contributed by Eric Yang
2018-12-14 14:08:22 -08:00
72e7c6a489
YARN-8962. Add ability to use interactive shell with normal yarn container. Contributed by Eric Yang
2018-12-12 18:18:57 -08:00
37eb919c59
YARN-8885. [DevicePlugin] Support NM APIs to query device resource allocation. (Zhankun Tang via wangda)
...
Change-Id: I2a9870709b512af1ac6c09c9701d0b3c0791ff32
2018-12-12 11:45:47 -08:00
61bdcb7b2b
YARN-9015. [DevicePlugin] Add an interface for device plugin to provide customized scheduler. (Zhankun Tang via wangda)
...
Change-Id: Ib2e4ae47a6f29bb3082c1f8520cf5a52ca720979
2018-12-12 11:44:22 -08:00
881230da21
YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth)
2018-12-11 11:41:32 -08:00
ac578c0e82
YARN-9087. Improve logging for initialization of Resource plugins. (Contributed by Szilard Nemeth)
2018-12-10 11:12:54 -08:00
64411a6ff7
YARN-8738. FairScheduler should not parse negative maxResources or minResources values as positive. (Contributed by Szilard Nemeth)
2018-12-10 11:06:50 -08:00
1c09a10e96
YARN-9009. Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs. Contributed by OrDTesters.
2018-12-10 11:55:24 +08:00
154449fbd8
YARN-8914. Add xtermjs to YARN UI2. Contributed by Eric Yang and Akhil PB
2018-12-07 10:56:17 -08:00
cc51607ccd
YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics
2018-12-07 10:45:37 -08:00
5d4a432660
YARN-9025. TestFairScheduler#testChildMaxResources is flaky. (Contributed by Szilard Nemeth)
2018-12-05 16:36:13 -08:00
b3c75c1f1d
YARN-8985. Improve debug log in FSParentQueue when assigning container. (Contributed by Wilfred Spiegelenburg)
2018-12-05 15:02:57 -08:00
2bba329c14
YARN-8994. Fix race condition between move app and queue cleanup in Fair Scheduler. (Contributed by Wilfred Spiegelenburg)
2018-12-05 14:38:15 -08:00
1b790f4dd1
YARN-9071. Improved status update for reinitialized containers.
...
Contributed by Chandni Singh
2018-12-05 17:00:56 -05:00
e89941fdbb
YARN-9041. Performance Optimization of method FSPreemptionThread#identifyContainersToPreempt. Contributed by Wanqiang Ji.
2018-12-04 10:08:45 -08:00
6d7b44c489
YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu
2018-11-30 10:13:52 -08:00
bad12031f6
YARN-9010. Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl. (Zhankun Tang via wangda)
...
Change-Id: Iaecc66d57781cc10f19ead4647e47fc9556676da
2018-11-29 14:56:07 -08:00
07142f54a8
YARN-9069. Fix SchedulerInfo#getSchedulerType for custom schedulers. Contributed by Bilwa S T.
2018-11-29 22:02:59 +05:30
a68d766e87
YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt.
2018-11-29 21:43:34 +05:30
fe7dab8ef5
YARN-8989. [YARN-8851] Move DockerCommandPlugin volume related APIs' invocation from DockerLinuxContainerRuntime#prepareContainer to #launchContainer. (Zhankun Tang via wangda)
...
Change-Id: Ia6d532c687168448416dfdf46f0ac34bff20e6ca
2018-11-28 15:03:06 -08:00
8ebeda98a9
YARN-8974. Improve the assertion message in TestGPUResourceHandler. (Zhankun Tang via wangda)
...
Change-Id: I4eb58e9d251d5f54e7feffc4fbb813b4f5ae4b1b
2018-11-28 14:36:30 -08:00
9ed87567ad
YARN-9061. Improve the GPU/FPGA module log message of container-executor. (Zhankun Tang via wangda)
...
Change-Id: Iece9b47438357077a53984a820d4d6423f480518
2018-11-28 14:31:31 -08:00
579ef4be06
YARN-8882. [YARN-8851] Add a shared device mapping manager (scheduler) for device plugins. (Zhankun Tang via wangda)
...
Change-Id: I9435136642c3d556971a357bf687f69df90bb45e
2018-11-28 14:09:52 -08:00
3ce99e32f7
YARN-8812. Containers fail during creating a symlink which started with hyphen for a resource file. Contributed by Oleksandr Shevchenko
2018-11-28 08:46:11 -06:00
33e0df4b35
YARN-8986. Added port publish for Docker container running with bridge network.
...
Contributed by Charo Zhang
2018-11-26 19:45:05 -05:00
0d8406135f
YARN-9054. Fix FederationStateStoreFacade#buildGetSubClustersCacheRequest. Contributed by Bibin A Chundatt.
2018-11-26 10:57:38 -08:00
e223a790a7
YARN-9042. Fix javadoc error in deviceplugin package. Contributed by Zhankun Tang.
2018-11-22 20:04:30 +05:30
a41b648e98
YARN-8992. Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue. (Contributed by Wilfred Spiegelenburg)
2018-11-20 14:12:37 -08:00
49824ed260
YARN-8838. Check that container user is same as websocket user for interactive shell. Contributed by Eric Yang
2018-11-20 11:12:24 -08:00
5fb14e0635
YARN-8953. [CSI] CSI driver adaptor module support in NodeManager. Contributed by Weiwei Yang.
2018-11-20 08:02:11 +05:30
6357803645
YARN-8881. [YARN-8851] Add basic pluggable device plugin framework. (Zhankun Tang via wangda)
...
Change-Id: If9a2f68cd4713b4ec932cdeda68106f17437c3d3
2018-11-19 08:54:31 -08:00
d027a24f03
YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun.
2018-11-18 23:18:26 +08:00
abd6d48c46
YARN-8860. Federation client intercepter class contains unwanted character. Contributed by Abhishek Modi.
2018-11-16 10:23:06 -08:00
ee3355be3c
YARN-8303. YarnClient should contact TimelineReader for application/attempt/container report.
2018-11-16 17:58:19 +05:30
eeaf8edaa7
YARN-8856. TestTimelineReaderWebServicesHBaseStorage tests failing with NoClassDefFoundError. Contributed by Sushil Ks.
2018-11-15 09:25:25 -08:00
21ec4bdaef
YARN-8672. Improve token filename management for localization.
...
Contributed by Chandni Singh
2018-11-14 15:22:01 -05:00
1f9c4f32e8
YARN-8776. Implement Container Exec feature in LinuxContainerExecutor. Contributed by Eric Yang
2018-11-12 10:42:30 -08:00
c741109522
YARN-8987. Usability improvements node-attributes CLI. Contributed by Bibin A Chundatt.
2018-11-12 18:18:23 +08:00
4e7284443e
YARN-8902. [CSI] Add volume manager that manages CSI volume lifecycle. Contributed by Weiwei Yang.
2018-11-12 11:57:02 +05:30
b5ec85d966
YARN-8933. [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor. Contributed by Botong Huang.
2018-11-11 11:12:53 -08:00
524a7523c4
YARN-8990. Fix fair scheduler race condition in app submit and queue cleanup. (Contributed by Wilfred Spiegelenburg)
2018-11-08 16:02:48 -08:00
e1bbf7dcdf
YARN-8988. Reduce the verbose log on RM heartbeat path when distributed node-attributes is enabled. Contributed by Tao Yang.
2018-11-08 17:47:18 +08:00
f8c72d7b3a
YARN-8880. Add configurations for pluggable plugin framework. Contributed by Zhankun Tang.
2018-11-08 12:23:00 +08:00
c96cbe8659
YARN-8977. Remove unnecessary type casting when calling AbstractYarnScheduler#getSchedulerNode. Contributed by Wanqiang Ji.
2018-11-07 22:42:22 +08:00
951c98f890
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang.
2018-11-07 11:17:35 +09:00
ab6aa4c726
YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
2018-11-06 08:40:59 -06:00
5d6554c722
YARN-8970. Improve the debug message in CS#allocateContainerOnSingleNode. Contributed by Zhankun Tang.
2018-11-06 14:50:09 +08:00
c7fcca0d7e
YARN-8969. AbstractYarnScheduler#getNodeTracker should return generic type to avoid type casting. Contributed by Wanqiang Ji.
2018-11-06 13:14:57 +08:00
f84a278baa
YARN-8905. [Router] Add JvmMetricsInfo and pause monitor. Contributed by Bilwa S T.
2018-11-03 20:35:31 +05:30
989715ec50
YARN-8893. [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client. Contributed by Botong Huang.
2018-11-02 15:30:08 -07:00
aed836efbf
YARN-8897. LoadBasedRouterPolicy throws NPE in case of sub cluster unavailability. Contributed by Bilwa S T.
2018-11-02 11:27:11 -07:00
babc946d40
YARN-8954. Reservations list field in ReservationListInfo is not accessible. Contributed by Oleksandr Shevchenko.
2018-11-02 11:10:08 -07:00
2ab611d48b
YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne
2018-11-01 14:22:00 -07:00
4ec4ec6971
YARN-8950. Fix compilation issue due to dependency convergence error for hbase.profile=2.0.
2018-10-30 11:30:08 +05:30
1d90a0dd23
YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg.
2018-10-29 11:53:10 +08:00
d07e873b7d
YARN-8569. Create an interface to provide cluster information to application. Contributed by Eric Yang
2018-10-26 17:57:05 -07:00
f76e3c3db7
YARN-8930. CGroup-based strict container memory enforcement does not work with CGroupElasticMemoryController (haibochen via rkanter)
2018-10-25 11:09:47 -07:00
c16c49b8c3
YARN-8587. Added retries for fetching docker exit code.
...
Contributed by Charo Zhang
2018-10-24 17:29:46 -04:00
69b328943e
YARN-8929. DefaultOOMHandler should only pick running containers to kill upon oom events (haibochen via rkanter)
2018-10-24 13:15:50 -07:00
766b78ee07
YARN-8911. ContainerScheduler incorrectly uses percentage number as the cpu resource utlization.
2018-10-24 07:58:26 -07:00
93fb3b4b9c
YARN-8904. TestRMDelegationTokens can fail in testRMDTMasterKeyStateOnRollingMasterKey. Contributed by Wilfred Spiegelenburg
2018-10-23 12:49:15 -05:00
0b62983c5a
YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli.
2018-10-23 12:58:41 +05:30
66e7a2c787
YARN-8922. Fixed test-container-executor test setup and clean up.
...
Contributed by Robert Kanter
2018-10-22 15:28:58 -04:00
e2a9fa8448
HADOOP-15821. Move YARN Registry to Hadoop Registry.
...
Contributed by Íñigo Goiri
2018-10-19 19:46:48 -04:00
47ad98b2e1
YARN-8910. Fixed misleading log statement when container max retries is infinite.
...
Contributed by Chandni Singh
2018-10-19 13:49:04 -04:00
5e02b4915b
YARN-8916. Define a constant docker string in ContainerRuntimeConstants.java for better maintainability. (Zhankun Tang via wangda)
...
Change-Id: I1349e740037f81afdbe30edbe741f20e88fd0a90
2018-10-19 09:49:26 -07:00
285d2c0753
YARN-8449. RM HA for AM web server HTTPS Support. (Contributed by Robert Kanter)
2018-10-18 21:24:36 -07:00
13cc0f50ea
YARN-8907. Fix incorrect logging message in TestCapacityScheduler. Contributed by Zhankun Tang.
2018-10-19 09:55:59 +08:00
780be14f07
YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda)
...
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
2018-10-18 10:58:21 -07:00
a457a8951a
YARN-8456. Fix a configuration handling bug when user leave FPGA discover executable path configuration default but set OpenCL SDK path environment variable. (Zhankun Tang via wangda)
...
Change-Id: Iff150ea98ba0c60d448474fd940eb121afce6965
2018-10-18 10:57:11 -07:00
32fe351bb6
YARN-8864. NM incorrectly logs container user as the user who sent a start/stop container request in its audit log. (Contributed by Wilfred Spiegelenburg)
2018-10-18 08:28:07 -07:00
2202e00ba8
YARN-8868. Set HTTPOnly attribute to Cookie. Contributed by Chandni Singh.
2018-10-18 15:23:20 +05:30
5085e5fa9e
YARN-8759. Copy of resource-types.xml is not deleted if test fails, causes other test failures. Contributed by Antal Bálint Steinbach.
2018-10-17 16:05:08 +05:30
84e22a6af4
YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth)
2018-10-16 14:14:30 -07:00
c2288ac45b
YARN-8448. AM HTTPS Support for AM communication with RMWeb proxy. (Contributed by Robert Kanter)
2018-10-16 13:36:26 -07:00
f880ff418c
YARN-8775. TestDiskFailures.testLocalDirsFailures sometimes can fail on concurrent File modifications. (Contributed by Antal Bálint Steinbach)
2018-10-15 09:37:20 -07:00
bca928d3c7
YARN-3879 [Storage implementation] Create HDFS backing storage implementation for ATS reads. Contributed by Abhishek Modi.
2018-10-11 21:14:06 -07:00
8d1981806f
YARN-5742 Serve aggregated logs of historical apps from timeline service. Contributed by Rohith Sharma KS
2018-10-11 16:26:07 -07:00
96d28b4750
YARN-8777. Container Executor C binary change to execute interactive docker command. Contributed by Eric Yang
2018-10-11 09:25:21 -07:00
e787d65a08
YARN-8861. executorLock is misleading in ContainerLaunch. Contributed by Chandni Singh
2018-10-11 10:54:57 -05:00
045069efec
YARN-6989 Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way. Contributed by Abhishek Modi
2018-10-10 15:17:05 -07:00
5ce70e1211
YARN-7644. NM gets backed up deleting docker containers. Contributed by Chandni Singh
2018-10-10 09:52:19 -05:00
edce866489
YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan.
2018-10-10 09:32:27 +08:00
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.
2018-10-10 10:16:57 +09:00
c3d22d3b45
YARN-7652. Handle AM register requests asynchronously in FederationInterceptor. Contributed by Botong Huang.
2018-10-09 10:29:40 -07:00
347ea38581
YARN-8763. Added node manager websocket API for accessing containers.
...
Contributed by Zian Chen
2018-10-08 13:56:47 -04:00
7c13872cbb
YARN-8659. RMWebServices returns only RUNNING apps when filtered with queue. (Contributed by Szilard Nemeth)
2018-10-08 09:40:37 -07:00
9bb2801e8c
YARN-4254. ApplicationAttempt stuck for ever due to UnknownHostException. Contributed by Bibin A Chundatt
2018-10-05 15:52:46 -05:00
c968365650
YARN-8644. Improve unit test for RMAppImpl.FinalTransition. (Contributed by Szilard Nemeth)
2018-10-05 09:32:30 -07:00
e60b797c88
YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth)
2018-10-04 13:00:31 -07:00
b6d5d84e07
YARN-8732. Add unit tests of min/max allocation for custom resource types in FairScheduler. (Contributed by Szilard Nemeth)
2018-10-04 12:47:31 -07:00
2e9913caf2
YARN-8844. TestNMProxy unit test is failing. (Eric Yang via wangda)
...
Change-Id: I241fa8701b6f1dbcad87fd2e9a429e32e7aa40f5
(cherry picked from commit b3ac886933
2018-10-04 10:53:39 -07:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
5edb9d3b97
YARN-8785. Improve the error message when a bind mount is not whitelisted. Contributed by Simon Prewo
2018-10-02 07:16:29 -06:00
d0ee6fbe28
YARN-8621. Add test coverage of custom Resource Types for the apps/<appId> REST API endpoint. (Contributed by Szilard Nemeth)
2018-10-01 14:46:42 -07:00
59d5af21b7
YARN-8760. [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer. Contributed by Botong Huang.
2018-10-01 13:12:38 -07:00
fd6be5898a
YARN-8468. Enable the use of queue based maximum container allocation limit and implement it in FairScheduler. Contributed by Antal Bálint Steinbach.
2018-09-29 17:47:12 +08:00
8598b498bc
YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang.
2018-09-28 15:32:07 +00:00
7093afd874
YARN-8829. Cluster metrics can fail with IndexOutOfBoundsException. Contributed by Akshay Agarwal.
2018-09-28 12:35:33 +05:30
90e2e493b3
YARN-8270 Adding JMX Metrics for Timeline Collector and Reader. Contributed by Sushil Ks.
2018-09-27 15:53:39 -07:00
b237a0dd44
YARN-6456. Added config to set default container runtimes.
...
Contributed by Craig Condit
2018-09-27 15:31:18 -04:00
6b988d821e
YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang
2018-09-26 14:43:00 -07:00
913f87dada
YARN-8665. Added Yarn service cancel upgrade option.
...
Contributed by Chandni Singh
2018-09-26 14:51:35 -04:00
e5287a4fe0
YARN-8824. App Nodelabel missed after RM restart for finished apps. Contributed by Bibin A Chundatt.
2018-09-26 12:30:26 +05:30
44edcdfd6a
YARN-8745. Misplaced the TestRMWebServicesFairScheduler.java file. Contributed by Y. SREENIVASULU REDDY.
2018-09-26 10:09:11 +09:00
50bc7746d7
YARN-8815. RM fails to recover finished unmanaged AM. Contributed by Bibin A Chundatt.
2018-09-25 11:31:14 +05:30
29dad7d258
YARN-8616. systemClock should be used in RMAppImpl instead of System.currentTimeMills(), to be consistent. (Contributed by Szilard Nemeth)
2018-09-24 16:04:28 -07:00
3090922805
YARN-8696. [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async. Contributed by Botong Huang.
2018-09-24 11:37:05 -07:00
aa4bd493c3
YARN-8801. Fixed header comments for docker utility functions.
...
Contributed by Zian Chen
2018-09-20 13:08:59 -04:00
6b5838ed32
YARN-8784. DockerLinuxContainerRuntime prevents access to distributed cache entries on a full disk. Contributed by Eric Badger
2018-09-19 16:44:51 -05:00
Jonathan Hung
bibinchundatt
Abhishek Modi
Rohith Sharma K S
Weiwei Yang
Eric E Payne
Eric Yang
Szilard Nemeth
Sunil G
Adam Antal
HUAN-PING SU
Eric Badger
hunshenshi
Haibo Chen
Vidura Mudalige
Akira Ajisaka
Giovanni Matteo Fumarola
Zhankun Tang
Suma Shivaprasad
Ahmed Hussein
Sean Mackrory
Inigo Goiri
Prabhu Joseph
Siyao Meng
Szilard Nemeth
Gabor Bota
Igor Rudenko
Vrushali C
Yufei Gu
Devaraj K
Zoltan Siegl
Shweta Yakkali
Eric Badger
Yufei Gu
Wangda Tan
Masatake Iwasaki
Vinayakumar B
Jason Lowe
Erik Krogen
Botong Huang
Billie Rinaldi
Jonathan Hung
Jonathan Hung
Robert Kanter
Shane Kumpf
Eric E Payne