Commit Graph

6048 Commits

Author SHA1 Message Date
bibinchundatt 81da221c75 YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 2020-08-07 08:36:52 +05:30
Prabhu Joseph c7e71a6c0b YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver.
Contributed by Bilwa ST.
2020-08-06 13:19:52 +05:30
Eric Badger 5edd8b925e YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource.
Contributed by Bibin Chundatt and Eric Payne.
2020-08-05 18:47:49 +00:00
Prabhu Joseph 0277856738 YARN-10377. Fix filter index to show apps while clicking on queue in RM UI
Contributed by Tarun Parimi.
2020-08-04 23:10:45 +05:30
Prabhu Joseph aa5afa72c5 YARN-10381. Add application attempt state in AppAttempts RM REST API
Contributed by Siddharth Ahuja. Reviewed by Bilwa ST.
2020-08-04 23:07:28 +05:30
kevinzhao1661 ab2b3df2de
YARN-10383. YarnCommands.md is inconsistent with the source code (#2177) 2020-08-04 12:02:16 +09:00
Brahma Reddy Battula eac558380f YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T. 2020-08-03 12:54:36 +05:30
bibinchundatt 5323e83edf YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 2020-08-01 13:03:46 +05:30
Eric E Payne e0c9653166 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 15:30:22 +00:00
bibinchundatt 5d8600e80a YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 2020-07-29 23:26:08 +05:30
Jonathan Hung 3eaf62726f YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:43:19 -07:00
bibinchundatt 5dadf963d3 YARN-10208. Add capacityScheduler metric for NODE_UPDATE interval. Contributed by Pranjal Protim Borah. 2020-07-28 11:55:47 +05:30
Prabhu Joseph 60a254621a YARN-10366. Fix Yarn rmadmin help message shows two labels for one node for --replaceLabelsOnNode.
Contributed by Tanu Ajmera.
2020-07-27 12:23:21 +05:30
Akira Ajisaka d02be17a26
YARN-10362. Javadoc for TimelineReaderAuthenticationFilterInitializer is broken. Contributed by Xieming Li. 2020-07-27 01:55:04 +09:00
Eric Badger ac5f21dbef YARN-4771. Some containers can be skipped during log aggregation after NM
restart. Contributed by Jason Lowe and Jim Brennan.
2020-07-24 22:35:16 +00:00
Prabhu Joseph 247eb0979b YARN-10319. Record Last N Scheduler Activities from ActivitiesManager
Reviewed by Tao Yang and Adam Antal.
2020-07-24 11:43:31 +05:30
bibinchundatt bfcd775381 YARN-10315. Avoid sending RMNodeResourceupdate event if resource is same. Contributed by Sushil Ks. 2020-07-23 16:41:35 +05:30
Masatake Iwasaki 1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) 2020-07-22 13:40:20 +09:00
Eric Badger 736bed6d6d [YARN-10353] Log vcores used and cumulative cpu in containers monitor.
Contributed by Jim Brennan
2020-07-20 18:08:27 +00:00
Ayush Saxena 6bcb24d269 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-18 14:33:43 +05:30
Prabhu Joseph cc71d50b21 YARN-10339. Fix TimelineClient in NodeManager failing when Simple Http Auth used in Secure Cluster
Contributed by Tarun Parimi.
2020-07-16 23:36:49 +05:30
Ahmed Hussein 1f71c4ae71 HADOOP-17099. Replace Guava Predicate with Java8+ Predicate
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 11:39:06 -05:00
Ahmed Hussein 98fcffe93f HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 09:53:18 -05:00
Bilwa S T 8980c23fc2
YARN-10350. TestUserGroupMappingPlacementRule fails
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-07-15 14:31:34 +09:00
Eric Badger 48f90115b5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by
Jim Brennan
2020-07-13 23:10:39 +00:00
Eric E Payne 0427100b75 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 2020-07-13 18:57:50 +00:00
Xiaoyu Yao f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. (#2085) 2020-07-09 11:33:37 -07:00
Sunil G 5dd270e208 YARN-10333. YarnClient obtain Delegation Token for Log Aggregation Path. Contributed by Prabhu Joseph. 2020-07-09 12:50:25 +05:30
Brahma Reddy Battula dfe60392c9 YARN-10341. Yarn Service Container Completed event doesn't get processed. Contributed by Bilwa S T. 2020-07-09 12:34:52 +05:30
Akira Ajisaka 10d218934c
YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126) 2020-07-09 10:29:47 +05:30
Prabhu Joseph 3a4d05b850 YARN-8047. RMWebApp make external class pluggable.
Contributed by Bilwa S T.
2020-07-08 12:43:20 +05:30
Prabhu Joseph 2bbd00dff4 YARN-10337. Fix failing testcase TestRMHATimelineCollectors.
Contributed by Bilwa S T.
2020-07-07 17:10:59 +05:30
Akira Ajisaka 639acb6d89
HADOOP-17111. Replace Guava Optional with Java8+ Optional. Contributed by Ahmed Hussein. 2020-07-06 16:08:36 +09:00
Szilard Nemeth 04abd0eb17 YARN-10330. Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule. Contributed by Peter Bacsko 2020-07-01 14:10:55 +02:00
Szilard Nemeth 9b5557a9e8 YARN-10325. Document max-parallel-apps for Capacity Scheduler. Contributed by Peter Bacsko 2020-07-01 13:41:30 +02:00
Eric Yang e8dc862d38 YARN-9809. Added node manager health status to resource manager registration call.
Contributed by Eric Badger via eyang
2020-06-30 11:39:16 -07:00
Akira Ajisaka cd188ea9f0
YARN-10331. Upgrade node.js to 10.21.0. (#2106) 2020-06-30 16:52:57 +09:00
Eric E Payne 74fc13cf91 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 2020-06-29 18:39:53 +00:00
Eric Yang 0be26811f3 YARN-10328. Fixed ZK Curator NodeExists exception in YARN service AM logs
Contributed by Bilwa S T via eyang
2020-06-29 09:21:24 -07:00
Brahma Reddy Battula 2c03524fa4 YARN-6526. Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call. COntributed by Bilwa S T. 2020-06-26 20:43:27 +05:30
Szilard Nemeth e0c1d8a969 YARN-10327. Remove duplication of checking for invalid application ID in TestLogsCLI. Contributed by Marton Hudaky 2020-06-26 13:21:55 +02:00
Szilard Nemeth 6459cc73e6 YARN-10318. ApplicationHistory Web UI incorrect column indexing. Contributed by Andras Gyori 2020-06-26 13:15:14 +02:00
Szilard Nemeth 33d3df7ee2 YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy. Contributed by Szilard Nemeth 2020-06-26 11:14:24 +02:00
Szilard Nemeth 6a8fd73b27 YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed by Marton Hudaky 2020-06-25 17:27:58 +02:00
Szilard Nemeth 03f855e3e7 YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps settings. Contributed by Peter Bacsko 2020-06-23 12:13:04 +02:00
Prabhu Joseph ce1008fe61 YARN-10321. Break down TestUserGroupMappingPlacementRule#testMapping into test scenarios
Contributed by Szilard Nemeth. Reviewed by Gergely Pollak.
2020-06-22 12:49:14 +05:30
Surendra Singh Lilhore b2facc84a1 YARN-9460. QueueACLsManager and ReservationsACLManager should not use instanceof checks. Contributed by Bilwa S T. 2020-06-20 20:09:31 +05:30
Szilard Nemeth 469841446f YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko 2020-06-19 14:50:24 +02:00
Eric Yang 89689c52c3 YARN-10308. Update javadoc and variable names for YARN service.
Contributed by Bilwa S T via eyang
2020-06-17 09:04:26 -07:00
Szilard Nemeth 5b1a56f9f1 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:34:40 +02:00
Vinayakumar B e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:16:33 +05:30
Tao Yang 7c4de59fc1 YARN-10293. Reserved Containers not allocated from available space of other nodes in CandidateNodeSet for MultiNodePlacement. Contributed by Prabhu Joseph. 2020-06-12 22:10:25 +08:00
Eric Badger fed6fecd3a YARN-10312. Add support for yarn logs -logFile to retain backward compatibility.
Contributed by Jim Brennan.
2020-06-11 21:02:41 +00:00
Eric E Payne 56247db302 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 2020-06-09 18:43:16 +00:00
Szilard Nemeth 8abff5151a YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-05 15:00:39 +02:00
Szilard Nemeth 8b146c17b3 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-05 11:38:32 +02:00
Szilard Nemeth a4835db95a YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-05 10:11:40 +02:00
Szilard Nemeth b5efdea4fd YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:29:15 +02:00
Szilard Nemeth aa6d13455b YARN-10284. Add lazy initialization of LogAggregationFileControllerFactory in LogServlet. Contributed by Adam Antal 2020-06-02 15:53:48 +02:00
Szilard Nemeth e0a0741ac8 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-02 15:42:39 +02:00
Szilard Nemeth 593af878c0 YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-27 13:05:52 +02:00
Szilard Nemeth fb030f3239 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak"
This reverts commit ac4540dd8e.
Reverting because of missing Jira-ID in the commit message.
2020-05-27 13:04:48 +02:00
Jonathan Hung c30c23cb66 YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 2020-05-26 16:29:42 -07:00
Szilard Nemeth ac4540dd8e FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 2020-05-21 13:11:44 +02:00
Eric Yang 726b8e324b YARN-10228. Relax restriction of file path character in yarn.service.am.java.opts.
Contributed by Bilwa S T via eyang
2020-05-20 09:23:56 -07:00
Prabhu Joseph cef0756929 YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient.
Contributed by Bilwa S T. Reviewed by Akira Ajisaka.
2020-05-20 11:36:52 +05:30
Akira Ajisaka b65815d691
Revert "YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient."
This reverts commit 7836bc4c35.
2020-05-18 16:29:07 +09:00
Prabhu Joseph 7836bc4c35 YARN-9606. Set sslfactory for AuthenticatedURL() while creating LogsCLI#webServiceClient.
Contributed by Bilwa S T. Reviewed by Brahma Reddy Battula.
2020-05-14 19:40:42 +05:30
Prabhu Joseph 6ce295b787 YARN-10259. Fix reservation logic in Multi Node Placement.
Reviewed by Wangda Tan.
2020-05-14 16:52:11 +05:30
Surendra Singh Lilhore 1958cb7c2b YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:09:34 +05:30
Ayush Saxena 0918433b4d YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:36:20 +05:30
Inigo Goiri 108ecf992f YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 2020-05-13 10:04:12 -07:00
Prabhu Joseph 450e5aa9dd YARN-10154. Addendum Patch which fixes below bugs
1. RM fails to start when LeafQueueTemplate max capacity is not specified.
2. Job stuck in ACCEPTED state with DominantResourceCalculator as Queue
   Capacity is set to NaN during RM startup with clusterResource is zero.

Reviewed by Sunil G and Manikandan R.
2020-05-13 14:35:37 +05:30
Jonathan Hung fff1d2c122 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 2020-05-12 10:48:54 -07:00
Elixir Kook a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) (#2007)
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
Inigo Goiri 35010120fb YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 2020-05-06 13:22:54 -07:00
Ahmed Hussein 92e3ebb401 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-05-06 12:27:17 -05:00
Szilard Nemeth cb6399c109 YARN-10257. FS-CS converter: skip increment properties for mem/vcores and fix DRF check. Contributed by Peter Bacsko 2020-05-05 16:00:43 +02:00
Szilard Nemeth 0debe55d6c YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-05 13:11:28 +02:00
Ahmed Hussein f5081a9a5d YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-05-04 10:13:51 -05:00
Surendra Singh Lilhore 257b04ce4e YARN-6553. Replace MockResourceManagerFacade with MockRM for AMRMProxy/Router tests. Contributed by Bilwa S T. 2020-05-02 12:43:20 +05:30
Inigo Goiri d125d39108 YARN-6973. Adding RM Cluster Id in ApplicationReport. Contributed by Bilwa S T. 2020-04-29 11:23:58 -07:00
Szilard Nemeth 410c605aec YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 2020-04-29 15:53:30 +02:00
Szilard Nemeth ab36429559 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 2020-04-28 18:53:28 +02:00
Akira Ajisaka 6ef01646ba
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 2020-04-28 05:35:36 +09:00
Prabhu Joseph 9224568b0e YARN-10156. Destroy Jersey Client in TimelineConnector.
Contributed by Tanu Ajmera.
2020-04-27 18:49:15 +05:30
Szilard Nemeth f91e21ac10 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 2020-04-27 08:26:11 +02:00
Szilard Nemeth d4874585f4 YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-24 17:04:47 +02:00
Akira Ajisaka 9827ff2961
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 2020-04-24 09:32:25 +09:00
Prabhu Joseph 60fa15366e YARN-10240. Prevent Fatal CancelledException in TimelineV2Client when stopping
Contributed by Tarun Parimi.
2020-04-21 12:01:33 +05:30
Szilard Nemeth 92b28e38b7 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-18 10:13:37 +02:00
Szilard Nemeth 76900b4f5b YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-18 09:57:27 +02:00
Jonathan Hung 3601054822 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:44 -07:00
Sunil G 2fe122e322 YARN-10154. CS Dynamic Queues cannot be configured with absolute resources. Contributed by Manikandan R. 2020-04-17 18:51:19 +05:30
Szilard Nemeth 4db598e0e6 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-15 13:18:44 +02:00
Prabhu Joseph 94f7470c71 YARN-10233. Fix YARN UI2 Daemon Logs
Contributed by Akhil PB.
2020-04-15 13:39:37 +05:30
Szilard Nemeth 55fcbcb5c2 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 2020-04-15 07:24:04 +02:00
Prabhu Joseph c791b0e90e YARN-10219. Fix YARN Native Service Placement Constraints with Node Attributes.
Contributed by Eric Yang.
2020-04-14 12:34:53 +05:30
Jonathan Hung 23481ad378 YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 2020-04-09 11:06:11 -07:00
Szilard Nemeth c04a484aed YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. 2020-04-09 15:46:25 +02:00
Sunil G 6fb29d0f17 YARN-10226. NPE in Capacity Scheduler while using %primary_group queue mapping. Contributed by Peter Bacsko. 2020-04-09 18:51:42 +05:30
Szilard Nemeth 8e4517f251 YARN-5277. When localizers fail due to resource timestamps being out, provide more diagnostics. Contributed by Siddharth Ahuja 2020-04-09 12:31:01 +02:00
Akira Ajisaka e53d472bb0
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 2020-04-09 01:47:31 +09:00
Wilfred Spiegelenburg 2214005c0f
YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja 2020-04-08 11:30:03 +10:00
Szilard Nemeth bffb43b00e YARN-10207. CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UI. Contributed by Siddharth Ahuja 2020-04-07 17:03:17 +02:00
Szilard Nemeth 45362a9f4c YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 2020-04-07 15:51:55 +02:00
Prabhu Joseph 7a39da3537 YARN-10120. Amendment fix for Java Doc.
Contributed by Bilwa S T.
2020-04-03 13:07:41 +05:30
Prabhu Joseph 25361b077b YARN-10120. Https Support in Router WebServiceClient.
Contributed by Bilwa S T.
2020-04-02 18:11:43 +05:30
Akira Ajisaka c162648aff
YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen Zhao. 2020-04-01 16:05:47 +09:00
Brahma Reddy Battula 8914cf9167 Preparing for 3.4.0 development 2020-03-29 23:24:25 +05:30
RogPodge 50f7f6dfd1
YARN-10210. Add a RMFailoverProxyProvider that does DNS resolution on failover. 2020-03-26 08:14:30 -07:00
Szilard Nemeth 3d5ade1839 YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R 2020-03-26 13:23:43 +01:00
Szilard Nemeth 9a297ff31d YARN-10197. FS-CS converter: fix emitted ordering policy string and max-am-resource percent value. Contributed by Peter Bacsko 2020-03-26 12:35:10 +01:00
Szilard Nemeth 348685dcb9 YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-03-26 12:18:15 +01:00
Jonathan Hung 6ce189c621 YARN-10200. Add number of containers to RMAppManager summary 2020-03-25 10:41:59 -07:00
RogPodge 2d294bd575
HADOOP-16938. Make non-HA proxy providers pluggable 2020-03-25 08:06:58 -07:00
Sunil G cdb2107066 YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
Prabhu Joseph d312991ead YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue
Contributed by Peter Bacsko. Reviewed by Manikandan R.
2020-03-23 18:01:14 +05:30
Prabhu Joseph f2d3ac2a3f YARN-10034. Remove Allocation Tags from released container from Decommission node
Contributed by Kyungwan Nam. Reviewed by Adam Antal.
2020-03-19 12:55:14 +05:30
Eric Badger 1975479285 YARN-2710. RM HA tests failed intermittently on trunk. Contributed by Ahmed
Hussein.
2020-03-16 22:24:02 +00:00
Weiwei Yang b9d825f178 YARN-9567. Add diagnostics for outstanding resource requests on app attempts page. Contributed by Tao Yang. 2020-03-13 14:39:36 -07:00
Weiwei Yang d273f6a2b4 YARN-9538. Document scheduler/app activities and REST APIs. Contributed by Tao Yang. 2020-03-13 14:36:52 -07:00
Bilwa c83644deac YARN-10110. Adding RouterPolicyProvider for RM Federation.
Contributed by Bilwa S T.
2020-03-13 16:56:44 +05:30
Szilard Nemeth 5ead9c15ca YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-03-12 12:29:03 +01:00
Szilard Nemeth 38d87883b6 YARN-10193. FS-CS converter: fix incorrect capacity conversion. Contributed by Peter Bacsko 2020-03-12 12:18:37 +01:00
Sunil G 0fd8bf5f6b YARN-10191. FS-CS converter: call System.exit function call for every code path in main method. Contributed by Peter Bacsko. 2020-03-12 15:03:12 +05:30
Szilard Nemeth cf9cf83a43 YARN-9354. Resources should be created with ResourceTypesTestHelper instead of TestUtils. Contributed by Andras Gyori 2020-03-10 16:44:48 +01:00
Szilard Nemeth 61f4cf3055 YARN-10002. Code cleanup and improvements in ConfigurationStoreBaseTest. Contributed by Benjamin Teke 2020-03-10 16:35:04 +01:00
Szilard Nemeth 9314ef947f YARN-10168. FS-CS Converter: tool doesn't handle min/max resource conversion correctly. Contributed by Peter Bacsko 2020-03-10 16:07:46 +01:00
Eric E Payne ede05b19d1 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 2020-03-10 14:17:45 +00:00
Szilard Nemeth 44afe1154d YARN-9419. Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled. Contribued by Andras Gyori 2020-03-09 16:08:24 +01:00
bibinchundatt 3859fa76d0 YARN-6924. Metrics for Federation AMRMProxy. Contributed by Young Chen 2020-03-07 09:34:42 +05:30
Szilard Nemeth ea0444851d YARN.10003. YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore. Contributed by Benjamin Teke 2020-03-06 17:49:45 +01:00
Szilard Nemeth 004e955348 YARN-10167. FS-CS Converter: Need to validate c-s.xml after converting. Contributed by Peter Bacsko 2020-03-05 09:56:42 +01:00
Eric Badger 2649f8b327 YARN-10173. Make pid file generation timeout configurable in case of reacquired
container. Contributed by Adam Antal.
2020-03-04 23:31:57 +00:00
Szilard Nemeth c0d084247c YARN-10175. FS-CS converter: only convert placement rules if a cmd line switch is defined. Contributed by Peter Bacsko 2020-03-03 09:46:32 +01:00
Szilard Nemeth e9eecedf69 YARN-10148. addendum: Fix method call parameter order of setAdminAndSubmitACL in TestCapacitySchedulerQueueACLs. Contributed by Kinga Marton 2020-03-02 16:25:14 +01:00
Inigo Goiri a43510e21d YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 2020-02-27 13:18:30 -08:00
Inigo Goiri b420ddeada YARN-10155. TestDelegationTokenRenewer.testTokenThreadTimeout fails in trunk. Contributed by Manikandan R. 2020-02-27 12:53:20 -08:00
Szilard Nemeth 10461e0193 YARN-10148. Add Unit test for queue ACL for both FS and CS. Contributed by Kinga Marton 2020-02-27 20:10:32 +01:00
Akira Ajisaka 7dfa37e8f0
YARN-10152. Fix findbugs warnings in hadoop-yarn-applications-mawo-core module (#1852) 2020-02-27 10:36:49 +09:00
Brahma Reddy Battula 3a9ccf7f6d YARN-10141.Interceptor in FederationInterceptorREST doesnt update on RM switchover. Contributed by D M Murali Krishna Reddy. 2020-02-26 23:24:00 +05:30
tanu.ajmera b5df1da3a1 YARN-9593. Support Comma in the value of Scheduler Configuration Mutation API. 2020-02-26 15:04:29 +05:30
Elixir Kook d608e94f92
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 2020-02-26 17:29:26 +09:00
Szilard Nemeth d68616b045 YARN-10130. FS-CS converter: Do not allow output dir to be the same as input dir. Contributed by Adam Antal 2020-02-25 21:30:04 +01:00
Wei-Chiu Chuang dda00d3ff5
YARN-10074. Update netty to 4.1.42Final in yarn-csi. Contributed by Wei-Chiu Chuang. (#1807) 2020-02-25 13:47:52 +09:00
Szilard Nemeth 72fa9c7f1b YARN-10135. FS-CS converter tool: issue warning on dynamic auto-create mapping rules. Contributed by Peter Bacsko 2020-02-24 21:54:07 +01:00
Szilard Nemeth 34cf63c7d0 YARN-10157. FS-CS converter: initPropertyActions() is not called without rules file. Contributed by Peter Bacsko 2020-02-24 21:45:49 +01:00
Prabhu Joseph 181e6d033e YARN-10119. Option to reset AM failure count for YARN Service
Contributed by kyungwan nam.
2020-02-20 20:34:06 +05:30
Szilard Nemeth 4af2556b48 YARN-10143. YARN-10101 broke Yarn logs CLI. Contributed by Adam Antal 2020-02-20 15:27:15 +01:00
Szilard Nemeth 3f1aad05f0 YARN-10147. FPGA plugin can't find the localized aocx file. Contributed by Peter Bacsko 2020-02-19 15:54:25 +01:00
Sunil G 6526f95bd2 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 2020-02-19 11:17:22 +05:30
Akira Ajisaka d6d7f8d8c5
YARN-8374. Upgrade objenesis to 2.6 (#1798) 2020-02-19 09:50:37 +09:00
Brahma Reddy Battula 20add89718 YARN-10136. [Router] : Application metrics are hardcode as N/A in UI. Contributed by Bilwa S T. 2020-02-14 16:50:29 +05:30
Szilard Nemeth da99ac7e93 YARN-10137. UIv2 build is broken in trunk. Contributed by Adam Antal 2020-02-13 16:31:35 +01:00
Szilard Nemeth f1b1b332f5 YARN-10029. Add option to UIv2 to get container logs from the new JHS API. Contributed by Adam Antal 2020-02-13 12:08:54 +01:00
Prabhu Joseph fe7d67a8a2 YARN-9521. Handle FileSystem close in ApiServiceClient
Contributed by kyungwan nam. Reviewed by Eric Yang.
2020-02-13 09:39:13 +05:30
Prabhu Joseph e637797211 YARN-10127. Remove setting App Ordering Policy to ParentQueue in FSQueueConverter
Contributed by Peter Bacsko.
2020-02-11 22:01:58 +05:30
Sunil G 28f730b317 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 2020-02-09 21:14:53 +05:30
Ayush Saxena 3f0a7cd17a YARN-9624. Use switch case for ProtoUtils#convertFromProtoFormat containerState. Contributed by Bilwa S T 2020-02-09 19:14:18 +05:30
Vinayakumar B 7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Szilard Nemeth 71b2c2ffe9 YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-06 12:25:06 +01:00
Jonathan Hung 314e2f9d2e YARN-10116. Expose diagnostics in RMAppManager summary 2020-02-04 17:44:05 -08:00
Giovanni Matteo Fumarola bf8686f43f YARN-8982. [Router] Add locality policy. Contributed by Young Chen. 2020-01-30 16:59:36 -08:00
Szilard Nemeth a7d72c523a YARN-10099. FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues. Contributed by Peter Bacsko 2020-01-30 16:03:38 +01:00
Akira Ajisaka a5ef08b619
YARN-9743. [JDK11] TestTimelineWebServices.testContextFactory fails. (#1824) Contributed by Akira Ajisaka and Kinga Marton. 2020-01-30 14:10:31 +09:00
Prabhu Joseph 825db8fe2a YARN-10107. Fix GpuResourcePlugin#getNMResourceInfo to honor Auto Discovery Enabled
Contributed by Szilard Nemeth.
2020-01-29 13:30:00 +05:30
Eric Badger e578e52aae YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 03:54:43 +00:00
Inigo Goiri 5abd0148eb YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-28 10:06:37 -08:00
Prabhu Joseph 1ab9c692fa YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change
Contributed by Kinga Marton.
2020-01-28 23:16:04 +05:30
Szilard Nemeth 94f060242e YARN-10104. FS-CS converter: dry run should work without output defined. Contributed by Peter Bacsko 2020-01-28 15:36:04 +01:00
Szilard Nemeth ca29768035 YARN-10085. FS-CS converter: remove mixed ordering policy check. Contributed by Peter Bacsko 2020-01-28 15:22:48 +01:00
Szilard Nemeth 9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
Inigo Goiri b4870bce3a Revert "YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R."
This reverts commit 0696828a09.
2020-01-21 17:45:17 -08:00
Inigo Goiri 0696828a09 YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-21 13:41:01 -08:00
Akira Ajisaka 1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 2020-01-21 10:58:32 +09:00
Szilard Nemeth 581072a8f0 YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-20 13:10:32 +01:00
Szilard Nemeth 6d52bbbfcf YARN-9525. IFile format is not working against s3a remote folder. Contributed by Adam Antal 2020-01-20 12:36:55 +01:00
Szilard Nemeth 8b3ee2f7e9 YARN-9462. TestResourceTrackerService.testNodeRemovalGracefully fails sporadically. Contributed by Prabhu Joseph 2020-01-20 09:54:22 +01:00
Szilard Nemeth 57aad0f43a YARN-10081. Exception message from ClientRMProxy#getRMAddress is misleading. Contributed by Ravuri Sushma Sree 2020-01-20 09:41:06 +01:00
Szilard Nemeth 14d0f9a775 YARN-8148. Update decimal values for queue capacities shown on queue status CLI. Contributed by Prabhu Joseph 2020-01-20 09:28:23 +01:00
Prabhu Joseph a0ff42d761 YARN-10070. Fix NPE if no queue mapping defined for proxy user when using application tag based placement.
Contributed by Kinga Marton. Reviewed by Adam Antal.
2020-01-16 12:55:27 +05:30
Akira Ajisaka 14c2c3d69d
YARN-9512. [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of ClassCastException. (#1806) 2020-01-16 12:21:50 +09:00
Szilard Nemeth 2aa065d98f YARN-10082. FS-CS converter: disable terminal placement rule checking. Contributed by Peter Bacsko 2020-01-15 12:42:13 +01:00
Szilard Nemeth 7c5cecc3b3 YARN-9970. Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping. Contributed by Manikandan R 2020-01-15 09:55:41 +01:00
Szilard Nemeth 1c51f36be7 YARN-9788. Queue Management API does not support parallel updates. Contributed by Prabhu Joseph 2020-01-14 12:26:03 +01:00
Szilard Nemeth 13cea0412c YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-14 11:00:08 +01:00
Szilard Nemeth 621c5eac38 YARN-9912. Capacity scheduler: support u:user2:%secondary_group queue mapping. Contributed by Manikandan R 2020-01-13 17:23:00 +01:00
Szilard Nemeth b7ef8a333f YARN-9868. Validate %primary_group queue in CS queue manager. Contributed by Manikandan R 2020-01-13 17:15:09 +01:00
Szilard Nemeth 2576c31644 YARN-9989. Typo in CapacityScheduler documentation: Runtime Configuration. Contributed by Kevin Su 2020-01-13 16:50:07 +01:00
Szilard Nemeth d842dfffa5 YARN-9866. u:user2:%primary_group is not working as expected. Contributed by Manikandan R 2020-01-12 14:04:15 +01:00
Szilard Nemeth 24e6a9e43a YARN-10067. Add dry-run feature to FS-CS converter tool. Contributed by Peter Bacsko 2020-01-12 13:48:39 +01:00
Eric Yang 2301b25899 YARN-10019. Improved container-executor exec() calls.
Contributed by Peter Bacsko
2020-01-10 19:04:04 -05:00
Adam Antal 20a90c0b5b
YARN-10071. Sync Mockito version with other modules
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2020-01-10 17:41:04 +09:00
Eric E Payne 93233a7d6e YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 2020-01-09 17:18:44 +00:00
Eric E Payne b1e07d27cc YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 2020-01-08 19:26:01 +00:00
Eric E Payne 6899be5a17 YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 2020-01-08 17:29:56 +00:00
Prabhu Joseph 571795cd18 YARN-10068. Fix TimelineV2Client leaking File Descriptors.
Contributed by Anand Srinivasan. Reviewed by Adam Antal.
2020-01-08 12:01:30 +05:30
Eric Yang d81d45ff2f YARN-9956. Improved connection error message for YARN ApiServerClient.
Contributed by Prabhu Joseph
2020-01-06 13:24:16 -05:00
Szilard Nemeth dd2607e3ec YARN-10026. Pull out common code pieces from ATS v1.5 and v2. Contributed by Adam Antal 2020-01-06 17:16:11 +01:00
Szilard Nemeth 768ee22e9e YARN-10035. Add ability to filter the Cluster Applications API request by name. Contributed by Adam Antal 2020-01-06 16:26:33 +01:00
Prabhu Joseph eca7e14c2f YARN-10053. Use Shared Group Mapping Service in Placement Rules.
Contributed by Wilfred Spiegelenburg.
2020-01-02 14:13:57 +05:30
Prabhu Joseph 21ada4d1b0 Revert "YARN-10053. Use Shared Group Mapping Service in Placement Rules."
This reverts commit 217b56ffdd.
2020-01-02 14:12:43 +05:30
Prabhu Joseph 217b56ffdd YARN-10053. Use Shared Group Mapping Service in Placement Rules.
Contributed by Wilfred Spiegelenburg.
2020-01-02 14:07:49 +05:30
Liu sheng 0fed874adf YARN-10041. Should not use AbstractPath to create unix domain socket (#1771) 2019-12-27 16:50:15 +05:30
Akira Ajisaka 40887c9b12
YARN-10057. Upgrade the dependencies managed by yarnpkg. (#1780) 2019-12-25 14:10:44 +09:00
Akira Ajisaka 34ff7dbaf5
YARN-10055. bower install fails. (#1778) 2019-12-23 19:45:47 +09:00
Eric Badger 412035b47a YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 18:07:00 +00:00
Zhankun Tang 12722ab0c7 YARN-10042. Upgrade grpc-xxx depdencies to 1.26.0. Contributed by Sheng Liu. 2019-12-20 11:10:27 +08:00
Giovanni Matteo Fumarola ef59ffd362 YARN-10038. [UI] Finish Time is not correctly parsed in the RM Apps page. Contributed by Inigo Goiri. 2019-12-19 11:37:17 -08:00
Masatake Iwasaki 7868da894a YARN-10037. Upgrade build tools for YARN Web UI v2. 2019-12-19 19:27:31 +09:00
Jonathan Hung 7b93575b92 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 2019-12-18 13:18:11 -08:00
Jonathan Hung fddc3d55c3 YARN-10039. Allow disabling app submission from REST endpoints 2019-12-18 10:48:05 -08:00
Masatake Iwasaki 92c8962f98
YARN-10020. Fix build instruction of hadoop-yarn-ui. (#1750) 2019-12-18 13:34:12 +09:00
Eric Badger f47dcf2d4c YARN-10033. TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources. Contributed by Eric Payne. 2019-12-17 17:11:23 +00:00
Szilard Nemeth 631dbbc6f2 YARN-9923. Introduce HealthReporter interface to support multiple health checker files. Contributed by Adam Antal 2019-12-15 17:28:04 +01:00
prabhujoseph d4bde134e3 YARN-7721. Fix TestContinuousScheduling fails sporadically with NPE.
Contributed by Wilfred Spiegelenburg.
2019-12-10 15:00:23 +05:30
Akira Ajisaka dc66de7448
YARN-9985. Unsupported transitionToObserver option displaying for rmadmin command. Contributed by Ayush Saxena. 2019-12-09 18:37:34 +09:00
dependabot[bot] e9c5bb8078 Bump bower from 1.7.7 to 1.8.8 (#1683)
Bumps [bower](https://github.com/bower/bower) from 1.7.7 to 1.8.8.
- [Release notes](https://github.com/bower/bower/releases)
- [Changelog](https://github.com/bower/bower/blob/master/CHANGELOG.md)
- [Commits](https://github.com/bower/bower/compare/v1.7.7...v1.8.8)

Signed-off-by: dependabot[bot] <support@github.com>
2019-12-09 15:46:08 +09:00
Eric Badger 8dffd8dc89 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-09 01:25:10 +00:00
Eric Badger ab2cc5ac92 Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger"
This reverts commit 76bb297db9.
2019-12-09 01:25:10 +00:00
Jonathan Hung 6d970d8100 YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 2019-12-08 16:35:02 -08:00
Jonathan Hung cd4779f317 Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources"
This reverts commit 92bce918dc.
2019-12-08 16:34:46 -08:00
Jonathan Hung 92bce918dc YARN-10012. Guaranteed and max capacity queue metrics for custom resources 2019-12-08 15:52:17 -08:00
Eric Badger 76bb297db9 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-12-06 18:33:58 +00:00
prabhujoseph e575df5ca6 YARN-10006. IOException used in place of YARNException in CapaitySheduler
Contributed by Adam Antal. Reviewed by Szilard Nemeth.
2019-12-06 12:51:28 +05:30
Szilard Nemeth 4f758dd4c6 YARN-9607. Auto-configuring rollover-size of IFile format for non-appendable filesystems. Contributed by Adam Antal 2019-12-05 21:49:00 +01:00
Szilard Nemeth cf68857631 YARN-9938. Validate Parent Queue for QueueMapping contains dynamic group as parent queue. Contributed by Manikandan R 2019-12-05 21:04:42 +01:00
Szilard Nemeth 0ab9c0398d YARN-9969. Improve yarn.scheduler.capacity.queue-mappings documentation. Contributed by Manikandan R 2019-12-05 20:53:43 +01:00
Szilard Nemeth 5cc6f945da YARN-9789. Disable Option for Write Ahead Logs of LogMutation. Contributed by Prabhu Joseph 2019-12-05 20:43:26 +01:00
Szilard Nemeth 4627dd6708 YARN-9780. SchedulerConf Mutation API does not Allow Stop and Remove Queue in a single call. Contributed by Prabhu Joseph 2019-12-05 20:38:37 +01:00
Szilard Nemeth c71befaf8f YARN-9781. SchedConfCli to get current stored scheduler configuration. Contributed by Prabhu Joseph 2019-12-05 20:31:24 +01:00
Szilard Nemeth 520fe2c99b YARN-5106. Provide a builder interface for FairScheduler allocations for use in tests. Contributed by Adam Antal 2019-12-05 17:37:40 +01:00
Sunil G 682e6fdeda YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth. 2019-12-05 13:26:23 +05:30
Wanqiang Ji c48de9aa2d YARN-9958. Remove the invalid lock in ContainerExecutor (#1704) 2019-12-04 14:35:41 +09:00
Abhishek Modi a2dadac790 YARN-9990. Testcase fails with Insufficient configured threads: required=16 < max=10. Contributed by Prabhu Joseph. 2019-11-29 11:54:33 +05:30
Szilard Nemeth ef950b0863 YARN-9290. Invalid SchedulingRequest not rejected in Scheduler PlacementConstraintsHandler. Contributed by Prabhu Joseph 2019-11-26 22:04:07 +01:00
Szilard Nemeth 828ab400ee YARN-9362. Code cleanup in TestNMLeveldbStateStoreService. Contributed by Denes Gerencser 2019-11-26 21:45:12 +01:00
prabhujoseph aa7ab2719f YARN-9991. Fix Application Tag prefix to userid. Contributed by Szilard Nemeth. 2019-11-27 01:59:12 +05:30
Szilard Nemeth 8c9018d5c7 YARN-9899. Migration tool that help to generate CS config based on FS config [Phase 2]. Contributed by Peter Bacsko 2019-11-26 21:22:35 +01:00
Szilard Nemeth 52e9ee39a1 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 2019-11-26 16:42:59 +01:00
Szilard Nemeth 448ffb12ec YARN-9937. addendum: Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2019-11-26 13:41:41 +01:00
Eric Badger 6c20512b68 Revert "YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger"
This reverts commit 289bbca870.
2019-11-25 17:46:42 +00:00
HUAN-PING SU f8e36e03b4 YARN-9966. Code duplication in UserGroupMappingPlacementRule (#1709) 2019-11-25 15:28:53 +09:00
Szilard Nemeth 4c1a1287bc YARN-9968. Public Localizer is exiting in NodeManager due to NullPointerException. Contributed by Tarun Parimi 2019-11-22 12:58:10 +01:00
Tao Yang 8afabe41e4 YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 2019-11-22 16:06:14 +08:00
Szilard Nemeth dfdc6d6dd9 YARN-9886. Queue mapping based on userid passed through application tag. Contributed by Julia Kinga Marton 2019-11-19 21:14:15 +01:00
Eric Yang ea8ffac121 YARN-9983. Fixed typo in YARN Service overview.
Contributed by Denes Gerencser
2019-11-19 14:16:51 -05:00
Prabhu Joseph 30b2096933 YARN-9986. Allow app owner to signal container via RMWebServices. Contributed by kyungwan nam. 2019-11-19 11:35:00 +00:00
Sunil G ea68756c0c YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:10:41 +05:30
Sunil G 215f2052fc YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 2019-11-19 14:03:02 +05:30
Abhishek Modi dc3f4fc2f4 YARN-9965. Fix NodeManager failing to start on subsequent times when Hdfs Auxillary Jar is set (addendum). Contributed by Prabhu Joseph. 2019-11-19 08:08:40 +05:30
Eric Badger 0e22e9ab83 YARN-9562. Add Java changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-11-18 21:56:31 +00:00
Eric Badger 289bbca870 YARN-9561. Add C changes for the new RuncContainerRuntime. Contributed by Eric Badger 2019-11-18 21:54:16 +00:00
Szilard Nemeth 27642367ef YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko 2019-11-18 16:29:39 +01:00
prabhujoseph bd454348b0 YARN-9982. Fix Container API example link in NodeManager REST API doc. Contributed by Charan Hebri. 2019-11-18 15:19:47 +05:30
Yufei Gu b83b9ab418 YARN-9537. Add configuration to disable AM preemption. Contributed by ZhouKang 2019-11-12 09:40:16 -08:00
Abhishek Modi fb512f5087 YARN-9697. Efficient allocation of Opportunistic containers. Contributed by Abhishek Modi. 2019-11-12 16:34:04 +05:30
Szilard Nemeth 30b93f914b YARN-9865. Capacity scheduler: add support for combined %user + %secondary_group mapping. Contributed by Manikandan R 2019-11-11 13:28:02 +01:00
Abhishek Modi 516377bfa6 YARN-9965. Fix NodeManager failing to start when Hdfs Auxillary Jar is set. Contributed by Prabhu Joseph. 2019-11-11 16:01:44 +05:30
Szilard Nemeth ceb9c6175e YARN-9890. [UI2] Add Application tag to the app table and app detail page. Contributed by Julia Kinga Marton 2019-11-09 14:12:04 +01:00
Szilard Nemeth 31f172fd96 YARN-9677. Make FpgaDevice and GpuDevice classes more similar to each other. Contributed by kevin su 2019-11-09 14:04:14 +01:00
zhoukang 7d0adddf09 YARN-9689: Support proxy user for Router to support kerberos (#1654) 2019-11-04 17:13:29 -08:00
Sunil G 51e7d1b37e YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 2019-11-04 23:27:49 +05:30
Sunil G d462308e04 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 2019-11-03 08:47:27 +05:30
Jonathan Hung e6137d0e50 Make upstream aware of 2.10.0 release 2019-10-30 20:58:32 -07:00
Sunil G 9a2e43e29e YARN-9937. Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph. 2019-10-31 00:34:27 +05:30
Szilard Nemeth e3e7daa4f5 YARN-9889. [UI] Add Application Tag column to RM All Applications table. Contributed by Kinga Marton 2019-10-30 16:26:56 +01:00
Abhishek Modi ed45c13f67 YARN-2442. ResourceManager JMX UI does not give HA State. Contributed by Rohith Sharma and Cyrus Jackson. 2019-10-29 15:54:55 +05:30
Eric Badger eef34f2d87 YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 2019-10-25 17:13:04 +00:00
Eric Yang ac6b6a6a85 HADOOP-16614. Add aarch64 support for dependent leveldbjni.
Contributed by liusheng
close #1546
2019-10-24 11:45:57 -04:00
Zhankun Tang fd84ca5161 YARN-9921. Issue in PlacementConstraint when YARN Service AM retries allocation on component failure. Contributed by Tarun Parimi 2019-10-24 10:05:00 +08:00
Eric E Payne 83d148074f YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 2019-10-21 20:30:50 +00:00
Eric Yang 3f7756dc6c YARN-9922. Fixed YARN javadoc errors from YARN-9699.
Contributed by Peter Bacsko
2019-10-21 16:16:49 -04:00
Eric Yang 155864da00 YARN-9875. Improve fair scheduler configuration store on HDFS.
Contributed by Prabhu Joseph
2019-10-18 20:30:11 -04:00
Eric Yang 72b1bed998 YARN-9884. Make container-executor mount logic modular
Contributed by Eric Badger
2019-10-18 19:10:32 -04:00
Mate Szalay-Beko 6d92aa7c30 HADOOP-16579. Upgrade to Curator 4.2.0 and ZooKeeper 3.5.5 (#1656). Contributed by Norbert Kalmár, Mate Szalay-Beko
* HADOOP-16579 - Upgrade to Apache Curator 4.2.0 and ZooKeeper 3.5.5

- Add a static initializer for the unit tests using ZooKeeper to enable
the four-letter-words diagnostic telnet commands. (this is an interface
that become disabled by default, so to keep the ZooKeeper 3.4.x behavior
we enabled it for the tests)
- Also fix ZKFailoverController to look for relevant fail-over ActiveAttempt
records. The new ZooKeeper seems to respond quicker during the fail-over
tests than the ZooKeeper, so we made sure to catch all the relevant records
by adding a new parameter to ZKFailoverontroller.waitForActiveAttempt().

Co-authored-by: Norbert Kalmár <nkalmar@cloudera.com>
2019-10-18 13:26:20 -07:00
Yuming Wang 3d41f33018 HADOOP-16152. Upgrade Eclipse Jetty version to 9.4.x. Contributed by Yuming Wang, Siyao Meng.
Co-authored-By: Siyao Meng <smeng@cloudera.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-10-18 08:25:02 -07:00
Szilard Nemeth f0699a7406 YARN-9841. Capacity scheduler: add support for combined %user + %primary_group mapping. Contributed by Manikandan R 2019-10-18 11:19:49 +02:00
Sunil G 3990ffa083 YARN-9909. Offline format of YarnConfigurationStore. Contributed by Prabhu Joseph. 2019-10-17 06:43:06 +05:30
Eric E Payne a5034c7988 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 2019-10-16 21:10:08 +00:00
Sunil G 090f73a9aa YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 2019-10-16 18:14:31 +05:30
Szilard Nemeth a8ef03e961 YARN-9840. Capacity scheduler: add support for Secondary Group rule mapping. Contributed by Manikandan R 2019-10-15 17:22:52 +02:00
Szilard Nemeth 5cc7873a47 YARN-9699. Migration tool that help to generate CS config based on FS config [Phase 1]. Contributed by Peter Bacsko 2019-10-14 17:19:39 +02:00
Szilard Nemeth ec86f42e40 YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-11 14:01:19 +02:00
Eric Yang 31e0122f4d YARN-9860. Enable service mode for Docker containers on YARN
Contributed by Prabhu Joseph and Shane Kumpf
2019-10-10 19:02:02 -04:00
Szilard Nemeth 6f1ab95168 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 2019-10-09 13:26:26 +02:00
Szilard Nemeth 35f093f5b3 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 2019-10-09 13:09:09 +02:00
Sunil G be901f4962 YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 2019-10-09 15:53:14 +05:30
Jonathan Hung 5462d2176f YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:20:13 -07:00
Sunil G fb1ecff6a2 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph"
This reverts commit 4510970e2f.
2019-10-05 09:15:17 +05:30
Sunil G 4510970e2f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 2019-10-04 21:49:07 +05:30
Sunil G 0d2d6f9267 YARN-9792. Document examples of SchedulerConf with Node Labels. Contributed by Prabhu Joseph 2019-10-02 23:43:47 +05:30
Abhishek Modi 61a8436004 YARN-9870. Remove unused function from OpportunisticContainerAllocatorAMService. Contributed by Abhishek Modi. 2019-10-02 17:55:07 +05:30
Eric Yang d1ddba60e5 YARN-9862. Increase yarn-services-core test timeout value.
Contributed by Prabhu Joseph
2019-10-01 21:15:26 -04:00
Sunil G 99cd7572f1 YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 2019-10-01 20:06:21 +05:30
bibinchundatt 425a6c89ae YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:02:26 +05:30
Sunil G 137546a78a YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 09:08:41 +05:30
Abhishek Modi 4d3c580b03 YARN-9859. Refactoring of OpportunisticContainerAllocator. Contributed by Abhishek Modi. 2019-09-30 23:40:15 +05:30
Eric Badger 18a8c2404e YARN-9857. TestDelegationTokenRenewer throws NPE but tests pass. Contributed by Ahmed Hussein 2019-09-26 16:15:08 +00:00
Jonathan Hung 606e341c1a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-25 17:54:53 -07:00
Szilard Nemeth c72457787d YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-09-25 10:36:22 +02:00
Szilard Nemeth bec0864394 YARN-9808. Zero length files in container log output haven't got a header. Contributed by Adam Antal 2019-09-25 10:28:34 +02:00
Jonathan Hung c2731d4b63 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 2019-09-24 13:51:32 -07:00
Duo Zhang bfe1dac29d
HADOOP-16560. [YARN] use protobuf-maven-plugin to generate protobuf classes (#1496)
Signed-off-by:  Vinayakumar B <vinayakumarb@apache.org>
2019-09-24 08:50:36 +08:00
Jonathan Hung 3d78b1223d YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 2019-09-23 13:12:16 -07:00
Vinayakumar B 1654497f98
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1 (#1432)
HADOOP-16557. [pb-upgrade] Upgrade protobuf.version to 3.7.1. Contributed by Vinayakumar B.
2019-09-20 16:08:30 +05:30
Malcolm Taylor 56248f9d87
HADOOP-16556. Fix some alerts raised by LGTM.
Contributed by Malcolm Taylor.

Change-Id: Ic60c3f4681dd9d48b3afcba7520bd1e4d3cc4231
2019-09-19 16:00:05 +01:00
Sunil G 01d7924473 YARN-9814. JobHistoryServer can't delete aggregated files, if remote app root directory is created by NodeManager. Contributed by Adam Antal. 2019-09-18 09:27:41 +05:30
Weiwei Yang f8c14326ee YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 2019-09-18 09:12:28 +08:00
Eric Yang eefe9bc85c YARN-9837. Fixed reading YARN Service JSON spec file larger than 128k.
Contributed by Tarun Parimi
2019-09-17 13:13:35 -04:00
Sunil G c474e24c0b YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 2019-09-17 19:41:38 +05:30
Abhishek Modi da2b4fe204 YARN-9794. RM crashes due to runtime errors in TimelineServiceV2Publisher. Contributed by Tarun Parimi. 2019-09-15 14:16:18 +05:30
HUAN-PING SU 4239007349 YARN-9787. Typo in analysesErrorMsg. Contributed by kevin su.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Surendra Singh Lilhore <surendralilhore@apache.org>
2019-09-13 10:36:45 -07:00
Abhishek Modi 44850f6784 YARN-9816. EntityGroupFSTimelineStore#scanActiveLogs fails when undesired files are present under /ats/active. Contribued by Prabhu Joseph. 2019-09-12 12:50:10 +05:30
Abhishek Modi 3b06f0bf9e YARN-9819. Make TestOpportunisticContainerAllocatorAMService more resilient. Contribued by Abhishek Modi 2019-09-12 08:09:57 +05:30
Eric Badger 56b7571131 YARN-9815 ReservationACLsTestBase fails with NPE. Contributed by Ahmed Hussein 2019-09-11 21:59:28 +00:00
Jonathan Hung f8f8598ea5 YARN-9824. Fall back to configured queue ordering policy class name 2019-09-10 15:19:07 -07:00
Eric Yang 10144a580e YARN-9728. Bugfix for escaping illegal xml characters for Resource Manager REST API.
Contributed by Prabhu Joseph
2019-09-10 17:04:39 -04:00
Abhishek Modi 60af8793b4 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. 2019-09-09 14:26:00 +05:30
Jonathan Hung 387c332b64 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:23:04 -07:00
Jonathan Hung 43e389b980 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:07:11 -07:00
Abhishek Modi 162af6f554 YARN-9784. org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue is flaky. Contributed by Julia Kinga Marton. 2019-09-07 18:08:04 +05:30
Wangda Tan 34b82e6da0 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda)
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
2019-09-06 19:18:13 -07:00
Tao Yang 73575701ab YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:52:39 +08:00
Jonathan Hung b71a7f195c YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 11:00:07 -07:00
Jonathan Hung e7d44e48f7 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:17:07 -07:00
Jonathan Hung 0ccf4b0fe1 YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 2019-09-05 14:01:42 -07:00
Billie Rinaldi 2e2e5401f2 YARN-9718. Fixed yarn.service.am.java.opts shell injection. Contributed by Eric Yang 2019-09-05 12:49:58 -07:00
Tao Yang 172bcd8e01 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Contributed by zhuqi. 2019-09-05 16:20:05 +08:00
Rohith Sharma K S 3db7184082 YARN-9804. Update ATSv2 document for latest feature supports. 2019-09-05 08:42:17 +05:30
Zhankun Tang bb26514ba9 YARN-9785. Fix DominantResourceCalculator when one resource is zero. Contributed by Bibin A Chundatt, Sunil Govindan, Bilwa S T. 2019-09-03 14:55:15 +08:00
bibinchundatt 03489124ea YARN-9797. LeafQueue#activateApplications should use resourceCalculator#fitsIn. Contributed by Bilwa S T. 2019-09-03 11:53:34 +05:30
Abhishek Modi f4d6e82075 YARN-9400. Remove unnecessary if at EntityGroupFSTimelineStore#parseApplicationId. Contributed by Prabhu Joseph. 2019-09-02 10:58:23 +05:30
Abhishek Modi b939ae98e6 YARN-8174. Add containerId to ResourceLocalizationService fetch failure log statement. Contributed by Prabhu Joseph. 2019-09-02 10:45:59 +05:30
Abhishek Modi bc27f7fb6d YARN-7982. Do ACLs check while retrieving entity-types per application. Contribued by Prabhu Joseph. 2019-09-02 10:15:59 +05:30
Abhishek Modi 751b5a1ac8 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 2019-09-01 22:40:40 +05:30
Abhishek Modi fef65b4c2b YARN-9793. Remove duplicate sentence from TimelineServiceV2.md. Contributed by Julia Kinga Marton. 2019-09-01 12:43:26 +05:30
Abhishek Modi c7ef4fb74a YARN-8678. Queue Management API - rephrase error messages. Contributed by Prabhu Joseph. 2019-09-01 10:00:04 +05:30
Abhishek Modi d2d963f3d4 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 2019-09-01 09:54:46 +05:30
Abhishek Modi 7d998cb627 YARN-9800. TestRMDelegationTokens can fail in testRemoveExpiredMasterKeyInRMStateStore. Contributed by Adam Antal. 2019-08-31 10:03:09 +05:30
Abhishek Modi eeccd2f6f6 YARN-9798. ApplicationMasterServiceTestBase#testRepeatedFinishApplicationMaster fails intermittently. Contributed by Tao Yang. 2019-08-31 09:57:45 +05:30
Abhishek Modi c929b383f8 YARN-9540. TestRMAppTransitions fails intermittently. Contributed by Tao Yang. 2019-08-30 17:17:43 +05:30
Rohith Sharma K S 5b557de721 YARN-9714. ZooKeeper connection in ZKRMStateStore leaks after RM transitioned to standby. Contributed by Tao Yang. 2019-08-30 10:35:09 +05:30
Weiwei Yang 8c0759d02a YARN-9664. Improve response of scheduler/app activities for better understanding. Contributed by Tao Yang. 2019-08-29 18:14:39 +08:00
Eric E Payne 6f2226a013 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 2019-08-28 20:40:34 +00:00
Jonathan Hung 8ef46595da YARN-9438. launchTime not written to state store for running applications 2019-08-27 15:53:43 -07:00
Eric Yang 717c853873 YARN-9755. Fixed RM failing to start when FileSystemBasedConfigurationProvider is configured.
Contributed by Prabhu Joseph
2019-08-27 13:14:59 -04:00
Rohith Sharma K S d70f5231a7 YARN-9640. Slow event processing could cause too many attempt unregister events. Contributed by Bibin A Chundatt. 2019-08-27 08:38:12 +05:30
Jonathan Hung 8660e48ca1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 2019-08-26 15:50:33 -07:00