Commit Graph

3769 Commits

Author SHA1 Message Date
Akira Ajisaka b658ae002f
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari.
(cherry picked from commit c9b525bd94)
2017-10-04 07:02:31 +09:00
Akira Ajisaka 2fa1d39952
YARN-6916. Moving logging APIs over to slf4j in hadoop-yarn-server-common. Contributed by Bibin A Chundatt and Akira Ajisaka.
(cherry picked from commit 4a87773718)
2017-10-04 06:08:00 +09:00
Jian He d2c3b93ee5 YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. Contributed by Wangda Tan
(cherry picked from commit 4111e6c781)
2017-10-03 12:23:03 -07:00
Sidharta S dcc454b4a0 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe
(cherry picked from commit 7eb8499996)
2017-10-02 19:32:19 -07:00
Wangda Tan c2f751cb0f YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda)
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
(cherry picked from commit 280080fad0)
2017-10-02 16:37:27 -07:00
Junping Du aef3e4b6af YARN-7194. Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat. Contributed by Xuan Gong.
(cherry picked from commit c92c1d521e)
2017-10-02 16:35:27 -07:00
Wangda Tan d58d75ca50 YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda)
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
(cherry picked from commit ec2ae3060a)
2017-09-29 16:49:59 -07:00
Junping Du c3eade44d0 YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong.
(cherry picked from commit 3fddabc2fe)
2017-09-29 16:49:44 -07:00
Wangda Tan cdaab892d8 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda)
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
(cherry picked from commit febeead5f9)
2017-09-29 16:46:34 -07:00
Eric Payne 7a9479597f YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe.
(cherry picked from commit 7f6118f918)
2017-09-29 10:39:01 -05:00
Wangda Tan 091fc32ce8 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda)
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
(cherry picked from commit d3b1c63195)
2017-09-28 16:46:23 -07:00
Chris Trezzo 3d23522110 YARN-7250. Update Shared cache client api to use URLs.
(cherry picked from commit c114da5e64)
2017-09-28 15:31:23 -07:00
Jason Lowe a1e9739730 YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh
(cherry picked from commit 85d81ae58e)
2017-09-28 14:14:14 -05:00
Andrew Wang 5a4f37019a Preparing for 3.0.0 GA development 2017-09-28 11:32:01 -07:00
Arun Suresh 59453dad8c YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh)
(cherry picked from commit df800f6cf3)
2017-09-28 11:29:26 -07:00
Arun Suresh ed855e2eef YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh)
(cherry picked from commit 66ca0a6540)
2017-09-28 11:27:43 -07:00
Arun Suresh 8eebee5094 YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh)
(cherry picked from commit 4f8194430f)
2017-09-28 11:24:20 -07:00
Arun Suresh dd51a74601 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh)
(cherry picked from commit 864fbacd45)
2017-09-28 11:22:58 -07:00
Daniel Templeton 7b49de5e30 YARN-6622. Document Docker work as experimental (Contributed by Varun Vasudev)
(cherry picked from commit 6651cbcc72)
2017-09-28 10:16:45 -07:00
Xuan 81e7872793 YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad
'start' value. Contributed by Jason Lowe

(cherry picked from commit 28c4957fcc)
2017-09-27 15:47:45 -07:00
Chris Trezzo ad7ed9d044 YARN-7253. Shared Cache Manager daemon command listed as admin subcmd in yarn script.
(cherry picked from commit c87db8d154)
2017-09-27 11:36:42 -07:00
Jason Lowe 8526bfa19d Revert "YARN-6570. No logs were found for running application, running"
This reverts commit e91447ccc9.
2017-09-26 10:41:44 -05:00
Rohith Sharma K S 52571d9fa3 YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R.
(cherry picked from commit 06e5a7b5cf)
2017-09-26 14:46:47 +05:30
Akira Ajisaka 149d3ad6ba
YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. Contributed by Sen Zhao.
(cherry picked from commit e928ee583c)
2017-09-26 02:04:23 +09:00
Junping Du b58cc16171 YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi.
(cherry picked from commit 4002bf0a9e)
2017-09-22 14:00:40 -07:00
Jason Lowe a294bfd80e YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger.
(cherry picked from commit bfd1a72ba8)
2017-09-21 17:43:51 -05:00
Junping Du 9e3313f8a1 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi.
(cherry picked from commit e5e1851d80)
2017-09-21 14:06:45 -07:00
Wangda Tan 4dbf7a501a YARN-6991. "Kill application" button does not show error if other user tries to kill the application for secure cluster. (Suma Shivaprasad via wangda)
Change-Id: I7e7894b24609709f89064ee5882f055dbb09080b
(cherry picked from commit 263e2c692a)
2017-09-21 12:27:18 -07:00
Jason Lowe 1e55972273 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger
(cherry picked from commit 10d7493587)
2017-09-21 08:44:16 -05:00
Sangjin Lee f919bcadb7 YARN-6771. Use classloader inside configuration class to make new
classes. Contributed by Jongyoul Lee.

(cherry picked from commit 8b336632ac)
2017-09-20 22:54:30 -07:00
Junping Du 974605c025 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh.
(cherry picked from commit 647b7527a9)
2017-09-19 18:31:54 -07:00
Arun Suresh 2ccfe92a4e YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh)
(cherry picked from commit 56ef5279c1)
2017-09-18 15:50:39 -07:00
Andrew Wang 531290c2c8 HADOOP-14835. mvn site build throws SAX errors. Contributed by Andrew Wang and Sean Mackrory.
(cherry picked from commit 3cf3540f19)
2017-09-18 15:13:45 -07:00
Xuan e91447ccc9 YARN-6570. No logs were found for running application, running
container. Contributed by Junping Du

(cherry picked from commit 7c732924a8)
2017-09-18 14:05:02 -07:00
Jason Lowe 7e72be9456 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh
(cherry picked from commit a4f9c7c924)
2017-09-18 10:23:22 -05:00
Rohith Sharma K S 508e93e48c YARN-7157. Add admin configuration to filter per-user's apps in secure cluster. Contributed by Sunil G. 2017-09-18 22:56:52 +10:00
Wangda Tan e3a0104980 YARN-7149. Cross-queue preemption sometimes starves an underserved queue. (Eric Payne via wangda)
Change-Id: Ib269991dbebce160378e8372ee6d24849c4a5ed6
(cherry picked from commit 3dfa937a1fadfc62947755872515f549b3b15e6a)
2017-09-17 21:19:20 -07:00
Wangda Tan 03b6e95fa5 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda)
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
2017-09-17 21:19:20 -07:00
Junping Du 21f6aa77ab YARN-7174. Add retry logic in LogsCLI when fetch running application logs. Contributed by Xuan Gong.
(cherry picked from commit 1a84c24b0c)
2017-09-15 15:34:30 -07:00
Yufei Gu 6362a9503d YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu)
(cherry picked from commit 5ab21dfe9c)
2017-09-15 09:51:50 -07:00
Robert Kanter d3d32d3580 YARN-7162. Remove XML excludes file format (rkanter) 2017-09-15 09:48:59 -07:00
Yufei Gu 8d49c2fc3b YARN-7057. FSAppAttempt#getResourceUsage doesn't need to consider resources queued for preemption. (Contributed by Karthik via Yufei Gu)
(cherry picked from commit 82c5dd1d50)
2017-09-14 15:48:48 -07:00
Robert Kanter aa21b01fcb YARN-7146. Many RM unit tests failing with FairScheduler (rkanter)
(cherry picked from commit bb34ae9554)
2017-09-13 15:45:28 -07:00
Eric Payne 4f6e8e08dc YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe.
(cherry picked from commit 3860be7961)
2017-09-13 14:46:04 -05:00
Arun Suresh 783db4792c YARN-7185. [Addendum patch] Minor javadoc and checkstyle fix.
(cherry picked from commit a1c398e0dc)
2017-09-12 17:04:25 -07:00
Arun Suresh 1230b60fa6 YARN-7185. ContainerScheduler should only look at availableResource for GUARANTEED containers when OPPORTUNISTIC container queuing is enabled. (Wangda Tan via asuresh)
(cherry picked from commit 2ae72692fc)
2017-09-12 16:38:58 -07:00
Wangda Tan d2e3818823 YARN-7173. Container update RM-NM communication fix for backward compatibility. (Arun Suresh via wangda)
Change-Id: I1c39ed5c59dee739ba5044b61b3ef5ed203b79c1
(cherry picked from commit e74d1be04b)
2017-09-12 08:09:47 -07:00
Haibo Chen 13310f91a5 YARN-7132. FairScheduler.initScheduler() contains a surprising unary plus. (Yeliang Cang via Haibo Chen)
(cherry picked from commit fa531788fd)
2017-09-11 20:49:33 -07:00
Haibo Chen 4f58533389 YARN-7128. The error message in TimelineSchemaCreator is not enough to find out the error. (Jinjiang Ling via Haibo Chen)
(cherry picked from commit 661f5eb0c6)
2017-09-11 16:22:06 -07:00
Sunil G 07cbb81e8b YARN-7163. RMContext need not to be injected to webapp and other Always Running services. Contributed by Rohith Sharma K S.
(cherry picked from commit 722ee84194)
2017-09-11 10:36:29 +05:30
Sunil G ad09c8f299 YARN-7104. Improve Nodes Heatmap in new YARN UI with better color coding. Contributed by Da Ding.
(cherry picked from commit ab8368d2e0)
2017-09-09 06:54:26 +05:30
Sunil G 6b54eec371 YARN-7022. Improve click interaction in queue topology in new YARN UI. Contributed by Abdullah Yousufi.
(cherry picked from commit ed162b7aa0)
2017-09-09 06:42:55 +05:30
Sunil G fd63f8112b YARN-7075. Better styling for donut charts in new YARN UI. Contributed by Da Ding.
(cherry picked from commit 275980bb1e)
2017-09-09 06:33:14 +05:30
Sunil G f3c9b2e40e YARN-6992. Kill application button is visible even if the application is FINISHED in RM UI. Contributed by Suma Shivaprasad.
(cherry picked from commit 6f101e7df1)
2017-09-09 06:25:53 +05:30
Haibo Chen 8ff2523687 HADOOP-14847. Remove Guava Supplier and change to java Supplier in AMRMClient and AMRMClientAysnc. (Bharat Viswanadham via Haibo Chen)
(cherry picked from commit 8edc60531f)
2017-09-08 14:35:03 -07:00
Rohith Sharma K S 720718eecc YARN-7130. ATSv2 documentation changes post merge. Contributed by Varun Saxena.
(cherry picked from commit 0bfb3a2566)
2017-09-08 22:42:02 +05:30
Rohith Sharma K S e8e1a73cf5 YARN-7140. CollectorInfo should have Public visibility. Contributed by Varun Saxena.
(cherry picked from commit 4a83170be4)
2017-09-08 22:41:52 +05:30
Arun Suresh abe3085f23 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh)
(cherry picked from commit 1f53ae7972)
2017-09-08 09:25:14 -07:00
Sunil G dfc174b747 YARN-6600. Introduce default and max lifetime of application at LeafQueue level. Contributed by Rohith Sharma K S.
(cherry picked from commit 56d93d2e39)
2017-09-08 20:06:06 +05:30
Jason Lowe 7287932397 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf
(cherry picked from commit b0b535d9d5)
2017-09-07 16:25:41 -05:00
Arun Suresh 05b4abf537 YARN-6978. Add updateContainer API to NMClient. (Kartheek Muthyala via asuresh)
(cherry picked from commit c41118a7f8)
2017-09-07 10:28:18 -07:00
Junping Du 2608b53910 YARN-7144. Log Aggregation controller should not swallow the exceptions when it calls closeWriter and closeReader. Contributed by Xuan Gong.
(cherry picked from commit 22de9449f8)
2017-09-06 15:02:23 -07:00
Junping Du 00594eb686 YARN-7148. TestLogsCLI fails in trunk and branch-2 and javadoc error. Contributed by Xuan Gong.
(cherry picked from commit 63231a2a30)
2017-09-06 15:01:35 -07:00
Jason Lowe 6ec611d8c1 YARN-7164. TestAMRMClientOnRMRestart fails sporadically with bind address in use. Contributed by Jason Lowe
(cherry picked from commit 704267cb49)
2017-09-06 15:28:45 -05:00
Varun Saxena 5323335fca YARN-7152. [ATSv2] Registering timeline client before AMRMClient service init throw exception (Rohith Sharma K S via Varun Saxena)
(cherry picked from commit ef87d34a0e)
2017-09-04 20:21:08 +05:30
Oleg Danilov 305528768c
YARN-7023. Incorrect ReservationId.compareTo() implementation.
This closes #262

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c5281a85e1)
2017-09-01 21:35:03 +02:00
Jason Lowe 99a7f5d451 YARN-7120. CapacitySchedulerPage NPE in "Aggregate scheduler counts" section. Contributed by Eric Payne 2017-09-01 11:09:14 -05:00
Sunil G a756704f5a YARN-7071. Add vcores and number of containers in new YARN UI node heat map. Contributed by Abdullah Yousufi. 2017-09-01 20:46:39 +05:30
bibinchundatt dcd0bedcc8 YARN-7141. Move logging APIs to slf4j in timelineservice after ATSv2 merge. Contributed by Varun Saxena 2017-09-01 11:29:16 +05:30
Allen Wittenauer 0adc3a0533
YARN-6721. container-executor should have stack checking
Signed-off-by: Chris Douglas <cdouglas@apache.org>
2017-08-31 19:39:31 -07:00
Allen Wittenauer 190410085b
HADOOP-14364. refresh changelog/release notes with newer Apache Yetus build
Signed-off-by: Andrew Wang <wang@apache.org>
2017-08-31 19:06:49 -07:00
Junping Du 91cc070d67 YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-08-31 16:41:43 -07:00
Subru Krishnan bac4e8cca8 YARN-7095. Federation: routing getNode/getNodes/getMetrics REST invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 2017-08-31 15:05:41 -07:00
Daniel Templeton d4417dae4f YARN-6780. ResourceWeights.toString() cleanup (Contributed by weiyuan via Daniel Templeton) 2017-08-31 09:35:01 -07:00
Sunil G f9e0cc8cdc YARN-7116. CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage. Contributed by Wangda Tan. 2017-08-31 18:42:01 +05:30
Allen Wittenauer 71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms
Signed-off-by: Chris Douglas <cdouglas@apache.org>
2017-08-30 21:47:24 -07:00
Robert Kanter ce79f7be29 YARN-7094. Document the current known issue with server-side NM graceful decom (rkanter) 2017-08-30 18:04:55 -07:00
Daniel Templeton cc23514aba YARN-7115. Move BoundedAppender to org.hadoop.yarn.util pacakge
(Contributed by Jian He via Daniel Templeton)
2017-08-30 17:26:13 -07:00
Haibo Chen a20e7105ea YARN-6868. Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml. (Ray Chiang via Haibo Chen) 2017-08-30 09:14:59 -07:00
Varun Saxena 3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Varun Saxena 512068a01f YARN-6905. Addendum to fix TestTimelineReaderWebServicesHBaseStorage due to missing FastNumberFormat 2017-08-30 11:29:54 +05:30
Varun Saxena 16ba4f544f MAPREDUCE-6838. Addendum to fix code comment 2017-08-30 11:29:54 +05:30
Jian He 08f40bcc7f MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Vrushali C bea3e4df76 YARN-6047 Documentation updates for TimelineService v2 (Contributed by Rohith Sharma) 2017-08-30 11:29:54 +05:30
Haibo Chen 9b08f365d7 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-08-30 11:29:54 +05:30
Varun Saxena b2efebdd07 YARN-6861. Reader API for sub application entities (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:54 +05:30
Jian He 7fd6ae2479 YARN-6134. [ATSv2 Security] Regenerate delegation token for app just before token expires if app collector is active. Contributed by Varun Saxena 2017-08-30 11:29:54 +05:30
Rohith Sharma K S e276c75ec1 YARN-7041. Nodemanager NPE running jobs with security off. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Rohith Sharma K S 32188d3295 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena. 2017-08-30 11:29:54 +05:30
Jian He b664569586 YARN-7006. [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol. Contributed by Varun Saxena 2017-08-30 11:29:53 +05:30
Jason Lowe d5ff965fee YARN-6820. Restrict read access to timelineservice v2 data. Contributed by Vrushali C 2017-08-30 11:29:53 +05:30
Varun Saxena 60765aff9b YARN-6874. Supplement timestamp for min start/max end time columns in flow run table to avoid overwrite (Vrushali C via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 354be99dbf YARN-6133. [ATSv2 Security] Renew delegation token for app automatically if an app collector is active. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 7594d1de7b YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Varun Saxena 9f6540535d YARN-6734. Ensure sub-application user is extracted & sent to timeline service (Rohith Sharma K S via Varun Saxena) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 3fb71b1393 YARN-6888. Refactor AppLevelTimelineCollector such that RM does not have aggregator threads created. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S a990ff70c2 YARN-6733. Add table for storing sub-application entities. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30
Vrushali C 61136d03f2 YARN-6850 Ensure that supplemented timestamp is stored only for flow run metrics (Contributed by Varun Saxena via Vrushali C) 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 70078e91e3 YARN-4455. Support fetching metrics by time range. Contributed by Varun Saxena. 2017-08-30 11:29:53 +05:30
Rohith Sharma K S 660413165a YARN-6801. NPE in RM while setting collectors map in NodeHeartbeatResponse. Contributed by Vrushali C. 2017-08-30 11:29:53 +05:30