Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,344 Commits 326 Branches 371 Tags
Commit Graph

5108 Commits

Author SHA1 Message Date
Sunil G
933d81f0ce YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 
(cherry picked from commit 99cd7572f113cbbb8f5ef89af60af3734fe8cfa7)
 2019-10-01 20:08:12 +05:30
Sunil G
71792f2122 YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 2019-10-01 20:03:35 +05:30
bibinchundatt
6529a30d9e YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:06:38 +05:30
Jonathan Hung
95ec7050b5 Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
(cherry picked from commit f4f210d2e57e49999579efb7db039cc623798267)
 2019-09-25 17:49:52 -07:00
Jonathan Hung
783cbced1d YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
(cherry picked from commit dd094d79023f6598e47146166aa8c213e03d41b7)
 2019-09-24 13:52:14 -07:00
Jonathan Hung
6a1d2d56bd YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d3fdc29d500803cefd2931b54f44928)
(cherry picked from commit a1fa9a8a7f79a1a711cd881b526724b502e03456)
 2019-09-23 13:13:09 -07:00
Sunil G
45cf3de2e9 YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 
(cherry picked from commit c474e24c0b73c0f52a7d5af2495355f4a0799344)
 2019-09-18 09:23:46 +05:30
Weiwei Yang
1978317068 YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 
(cherry picked from commit f8c14326ee0026f2c57a4d061bff5fadec74b74d)
 2019-09-18 09:20:37 +08:00
Eric Yang
dae22c962d YARN-9837. Fixed reading YARN Service JSON spec file larger than 128k. 
Contributed by Tarun Parimi
 2019-09-17 13:17:13 -04:00
Jonathan Hung
d75693bd6e YARN-9824. Fall back to configured queue ordering policy class name 
(cherry picked from commit f8f8598ea5e86330fa5b553963f5632da286409d)
(cherry picked from commit 1dbf87c9ff33b5e2f6cbb0ce8efd348c4314a0e2)
 2019-09-10 15:31:58 -07:00
Jonathan Hung
80735a15a5 YARN-8541 (branch-3.1 addendum): RM startup failure on recovery after user deletion 2019-09-09 20:15:42 -07:00
bibinchundatt
aee8fb567b YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt. 
(cherry picked from commit a68d766e876631d7ee2e1a6504d4120ba628d178)
(cherry picked from commit e10050678d0bcd0e4384f2a684daf8c18e30ecb9)
 2019-09-09 20:00:46 -07:00
Wangda Tan
9e8ff94d16 YARN-8361. Change App Name Placement Rule to use App Name instead of App Id for configuration. (Zian Chen via wangda) 
Change-Id: I17e5021f8f611a9c5e3bd4b38f25e08585afc6b1
(cherry picked from commit a2e49f41a8bcc03ce0a85b294d0b86fee7e86f31)
 2019-09-09 20:00:33 -07:00
Wangda Tan
81d63d5ea1 YARN-8016. Refine PlacementRule interface and add a app-name queue mapping rule as an example. (Zian Chen via wangda) 
Change-Id: I35caf1480e0f76f5f3a53528af09312e39414bbb
(cherry picked from commit a90471b3e65326cc18ed31fe21aef654833b5883)
 2019-09-09 19:59:50 -07:00
Jonathan Hung
0e88bcd8e6 YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:27:33 -07:00
Jonathan Hung
080fc6d943 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 
(cherry picked from commit 43e389b9801e09741fdf78fef067b8ac60f691c8)
(cherry picked from commit 45220d115797663b8749980b78a61bafcb2344f1)
 2019-09-08 19:15:12 -07:00
Wangda Tan
0ee7d09138 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda) 
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
(cherry picked from commit 34b82e6da0a471010cdae613ba39487889d79369)
 2019-09-06 19:19:59 -07:00
Tao Yang
36c1caced5 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time(addendum). Contributed by Jonathan Hung. 2019-09-07 08:19:02 +08:00
Tao Yang
1f6f4a2457 YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:54:30 +08:00
Jonathan Hung
980a922481 YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 10:52:37 -07:00
Jonathan Hung
11f6e3bc41 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:02:20 -07:00
Billie Rinaldi
619bd1e876 YARN-9718. Fixed yarn.service.am.java.opts shell injection. Contributed by Eric Yang 
(cherry picked from commit 2e2e5401f297545181323b126a69eaa2239afb02)
 2019-09-05 14:19:05 -07:00
Jonathan Hung
37d1f8c81e YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 
(cherry picked from commit 0ccf4b0fe16a8c879a560f2a612a3185eb2df72b)
(cherry picked from commit cb806988d72bde1f9837c9e0fb82a3a6c032542c)
 2019-09-05 14:06:09 -07:00
Tao Yang
d74c069427 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time. Contributed by zhuqi. 2019-09-05 16:54:55 +08:00
Zhankun Tang
ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Zhankun Tang
fff4fbc957 YARN-9785. Fix DominantResourceCalculator when one resource is zero. Contributed by Bibin A Chundatt, Sunil Govindan, Bilwa S T. 2019-09-04 12:05:29 +08:00
bibinchundatt
3210d1e993 YARN-9797. LeafQueue#activateApplications should use resourceCalculator#fitsIn. Contributed by Bilwa S T. 
(cherry picked from commit 03489124ea1b8d5648ade5e3563e39b5bc323384)
 2019-09-03 11:56:19 +05:30
Akira Ajisaka
3c9d2f5317 YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. 
(cherry picked from commit 5db7c4906235dd1070b9706ca4971f1eb5d74139)
(cherry picked from commit a453f38015fec454f63e9f5459b0330ac3bc7eb3)
 2019-08-30 17:55:35 -07:00
Eric E Payne
51896ff7e6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)
 2019-08-28 21:05:23 +00:00
Jonathan Hung
f73842780e YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
 2019-08-27 15:45:42 -07:00
Jonathan Hung
6baa0d1e4d YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
(cherry picked from commit e4249c320257586384035ea3fc286fe54cc699a1)
 2019-08-26 15:55:11 -07:00
bibinchundatt
eb618e4f22 YARN-9642. Fix Memory Leak in AbstractYarnScheduler caused by timer. Contributed by Bibin A Chundatt. 
(cherry picked from commit d3ce53e5073e35e162f1725836282e4268cd26a5)
 2019-08-26 23:25:16 +05:30
Szilard Nemeth
fd2e353236 YARN-9100. Add tests for GpuResourceAllocator and do minor code cleanup. Contributed by Peter Bacsko 2019-08-16 15:27:10 +02:00
Szilard Nemeth
0a379e94ba YARN-8586. Extract log aggregation related fields and methods from RMAppImpl. Contributed by Peter Bacsko 2019-08-16 12:15:27 +02:00
Szilard Nemeth
94114378ce YARN-9488. Skip YARNFeatureNotEnabledException from ClientRMService. Contributed by Prabhu Joseph 
(cherry picked from commit 1845a83cec6563482523d8c34b38c4e36c0aa9df)
 2019-08-15 17:16:32 +02:00
Szilard Nemeth
aa0631a042 YARN-9140. Code cleanup in ResourcePluginManager.initialize and in TestResourcePluginManager. Contributed by Peter Bacsko 2019-08-14 19:04:09 +02:00
Eric Badger
a995e6352f YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)
 2019-08-13 17:16:57 +00:00
Szilard Nemeth
cb91ab73b0 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 
(cherry picked from commit 8b3c6791b13fc57891cf81e83d4b626b4f2932e6)
 2019-08-13 15:47:57 +02:00
Szilard Nemeth
a762a6be29 Revert "YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko" 
This reverts commit b20fd9e21295add7e80f07b471bba5c76e433aed.
Commit is reverted since unnecessary files were added, accidentally.
 2019-08-13 15:47:57 +02:00
Szilard Nemeth
9da9b6d58e YARN-9723. ApplicationPlacementContext is not required for terminated jobs during recovery. Contributed by Prabhu Joseph 
(cherry picked from commit e4b538bbda6dc25d7f45bffd6a4ce49f3f84acdc)
 2019-08-12 15:16:49 +02:00
Szilard Nemeth
148121d889 YARN-9451. AggregatedLogsBlock shows wrong NM http port. Contributed by Prabhu Joseph 
(cherry picked from commit b91099efd6e1fdcb31ec4ca7142439443c9ae536)
 2019-08-12 15:06:48 +02:00
Szilard Nemeth
6b4ded7647 YARN-9135. NM State store ResourceMappings serialization are tested with Strings instead of real Device objects. Contributed by Peter Bacsko 2019-08-12 14:03:50 +02:00
Sunil G
58ad5ad493 YARN-9729. [UI2] Fix error message for logs when ATSv2 is offline. Contributed by Zoltan Siegl. 
(cherry picked from commit 1c5b28659fe1310030d14e0be40dcd77b25056d6)
 2019-08-11 11:49:57 +05:30
Szilard Nemeth
be9ac8adf9 Logging fileSize of log files under NM Local Dir. Contributed by Prabhu Joseph 
(cherry picked from commit 54ac80176e8487b7a18cd9e16a11efa289d0b7df)
 2019-08-09 13:23:49 +02:00
Sunil G
1c4364cb9a YARN-9715. [UI2] yarn-container-log URI need to be encoded to avoid potential misuses. Contributed by Akhil PB. 
(cherry picked from commit acffec7a92be540aa8531dbe06a3ea7bb813ab93)
 2019-08-09 16:06:07 +05:30
Adam Antal
600a61f410 YARN-9124. Resolve contradiction in ResourceUtils: addMandatoryResources / checkMandatoryResources work differently (#1121) 
(cherry picked from commit cbcada804d119b837ad99de71d7f44cb4629026e)
 2019-08-09 11:44:22 +02:00
Szilard Nemeth
410f7a3069 YARN-9092. Create an object for cgroups mount enable and cgroups mount path as they belong together. Contributed by Gergely Pollak 
(cherry picked from commit e0c21c6da91776caf661661a19c368939c81fcc4)
 2019-08-09 10:25:12 +02:00
Szilard Nemeth
b2f39f81fe YARN-9096: Some GpuResourcePlugin and ResourcePluginManager methods are synchronized unnecessarily. Contributed by Gergely Pollak 
(cherry picked from commit 742e30b47381ad63e2b2fe63738cd0fe6cbce106)
 2019-08-09 10:05:40 +02:00
Szilard Nemeth
943dfc78d1 YARN-9094: Remove unused interface method: NodeResourceUpdaterPlugin#handleUpdatedResourceFromRM. Contributed by Gergely Pollak 
(cherry picked from commit 72d7e570a73989aa18b737c0e642d570a55c6781)
 2019-08-09 09:53:14 +02:00
Eric E Payne
b131214685 YARN-9685: NPE when rendering the info table of leaf queue in non-accessible partitions. Contributed by Tao Yang. 
(cherry picked from commit 3b38f2019e4f8d056580f3ed67ecef591011d7a6)
 2019-08-08 13:08:05 +00:00