Commit Graph

1667 Commits

Author SHA1 Message Date
Jian He b2568d69fc YARN-3522. Fixed DistributedShell to instantiate TimeLineClient as the correct user. Contributed by Zhijie Shen
(cherry picked from commit aa4a192feb)
2015-04-23 11:08:16 -07:00
Steve Loughran b3f2826bfb YARN-2605. [RM HA] Rest api endpoints doing redirect incorrectly. (Xuan Gong via stevel) 2015-04-23 09:22:09 +01:00
Vinod Kumar Vavilapalli 7dd5f42a76 Set the release date for 2.7.0
(cherry picked from commit d52de61544)
2015-04-20 20:18:38 -07:00
Jian He b493ec28a8 YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe
(cherry picked from commit f967fd2f21)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-04-20 10:41:17 -07:00
Wangda Tan d6105d944c YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda)
(cherry picked from commit f47a5763ac)
2015-04-17 17:23:34 -07:00
Harsh J d4a462c02e YARN-3462. Patches applied for YARN-2424 are inconsistent between trunk and branch-2. Contributed by Naganarasimha G R. 2015-04-15 09:42:20 +05:30
Jason Lowe 70ac8cda41 YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column. (Jason Lowe via wangda)
(cherry picked from commit 73d9e4c77b)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
2015-04-10 13:52:34 +00:00
Vinod Kumar Vavilapalli 2b44fcaa53 Adding release 2.7.1 to CHANGES.txt
(cherry picked from commit 623fd46c1e)
2015-04-09 14:01:23 -07:00
Vinod Kumar Vavilapalli 4c6867ff01 YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp.
(cherry picked from commit 9c5911294e)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-04-09 13:11:26 -07:00
Zhijie Shen 64cf079858 YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong.
(cherry picked from commit 8366a36ad3)
2015-04-06 13:18:16 -07:00
Jian He edc3cb736e YARN-3273. Improve scheduler UI to facilitate scheduling analysis and debugging. Contributed Rohith Sharmaks
(cherry picked from commit 658097d6da)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/webapp/AppAttemptBlock.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java
2015-04-06 13:17:31 -07:00
Zhijie Shen cf33bc1050 Revert "YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong."
This reverts commit e4059b9cce.
2015-04-01 16:41:25 -07:00
Zhijie Shen e4059b9cce YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong.
(cherry picked from commit 8366a36ad3)
2015-04-01 13:49:51 -07:00
Vinod Kumar Vavilapalli 35af6f1802 YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. Contributed by Junping Du and Karthik Kambatla.
(cherry picked from commit c358368f51)
2015-03-30 10:11:49 -07:00
Xuan 646710318c YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG.
Contributed by Varun Saxena

(cherry picked from commit e556198e71)
2015-03-25 04:53:31 -07:00
Xuan 4dfd84ec08 YARN-3393. Getting application(s) goes wrong when app finishes before
starting the attempt. Contributed by Zhijie Shen

(cherry picked from commit 9fae455e26)
2015-03-23 20:34:59 -07:00
cnauroth 57e297208d YARN-3336. FileSystem memory leak in DelegationTokenRenewer.
(cherry picked from commit 6ca1f12024)
(cherry picked from commit 342c525eaa)
2015-03-23 10:46:22 -07:00
Tsuyoshi Ozawa d2e19160dc YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. Contributed by Naganarasimha G R.
(cherry picked from commit 82eda771e0)
2015-03-24 00:26:08 +09:00
Wangda Tan 57a0382f4b YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda)
(cherry picked from commit 6bc7710ec7)
2015-03-20 10:44:34 -07:00
Xuan b63ff4c36d YARN-2777. Mark the end of individual log in aggregated log. Contributed
by Varun Saxena

(cherry picked from commit 1a4b528691)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-03-20 10:28:24 -07:00
Jian He e67bd31ac0 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong
(cherry picked from commit 4e886eb9cb)
2015-03-19 22:34:34 -07:00
Jian He 54e63c52c9 Backport part of YARN-3273 to rename CapacitySchedulerLeafQueueInfo#aMResourceLimit to AMResourceLimit. Contributed by Rohith Sharmaks 2015-03-17 22:08:10 -07:00
Karthik Kambatla 47e6fc2bf9 Revert "YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)"
This reverts commit c2b185def8.

(cherry picked from commit 32b4330456)
2015-03-17 12:32:14 -07:00
Tsuyoshi Ozawa f15a2b23ca YARN-3349. Treat all exceptions as failure in TestFSRMStateStore#testFSRMStateStoreClientRetry. Contributed by Zhihai Xu.
(cherry picked from commit 7522a643fa)
2015-03-17 08:10:57 +09:00
Zhijie Shen d702816e7d YARN-2854. Updated the documentation of the timeline service and the generic history service. Contributed by Naganarasimha G R.
(cherry picked from commit fbe811d904)
2015-03-16 11:02:06 -07:00
Tsuyoshi Ozawa 2b2f7f2b90 YARN-1453. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA, Andrew Purtell, and Allen Wittenauer.
(cherry picked from commit 3da9a97cfb)
2015-03-16 23:20:34 +09:00
Xuan 6ee45ae3a8 YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't
work in ATS / RM web ui. Contributed by Naganarasimha G R

(cherry picked from commit 3ff1ba2a7b)
2015-03-15 20:28:09 -07:00
Jonathan Eagles e4d8dddb49 YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles)
(cherry picked from commit 8180e676ab)
2015-03-13 12:12:40 -05:00
Vinod Kumar Vavilapalli 0cd9eb9987 YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. Contributed by Xuan Gong.
(cherry picked from commit 863079bb87)
2015-03-12 13:34:33 -07:00
Xuan 8b008c8714 YARN-3338. Exclude jline dependency from YARN. Contributed by Zhijie
Shen

(cherry picked from commit 06ce1d9a6c)
2015-03-12 10:26:38 -07:00
Zhijie Shen ec9816b2b3 YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM web page. Contributed by Xuan Gong.
(cherry picked from commit 85f6d67fa7)
2015-03-11 19:41:20 -07:00
Tsuyoshi Ozawa a5a933a9a0 YARN-3295. Fix documentation nits found in markdown conversion. Contributed by Masatake Iwasaki.
(cherry picked from commit 30c428a858)
2015-03-11 15:02:59 +09:00
Jian He 8c5642296d YARN-3187. Documentation of Capacity Scheduler Queue mapping based on user or group. Contributed by Gururaj Shetty
(cherry picked from commit a380643d20)
2015-03-10 11:17:16 -07:00
Jian He d579fddf24 YARN-3300. Outstanding_resource_requests table should not be shown in AHS. Contributed by Xuan Gong
(cherry picked from commit c3003eba6f)
2015-03-09 20:48:09 -07:00
Zhijie Shen 6527be374b YARN-3287. Made TimelineClient put methods do as the correct login context. Contributed by Daryn Sharp and Jonathan Eagles.
(cherry picked from commit d6e05c5ee2)
2015-03-09 15:52:06 -07:00
Junping Du b91e1b0feb YARN-3296. Mark ResourceCalculatorProcessTree class as Public for configurable resource monitoring. Contributed by Hitesh Shah
(cherry picked from commit 7ce3c76353)
2015-03-08 14:48:42 -07:00
Jason Lowe ef3d9bdf6b YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne
(cherry picked from commit 27e8ea820f)
2015-03-06 22:45:09 +00:00
Jian He 6cef2c16de YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu
(cherry picked from commit 21101c01f2)
2015-03-06 14:18:56 -08:00
Xuan 56c2050ab7 YARN-3227. Timeline renew delegation token fails when RM user's TGT is
expired. Contributed by Zhijie Shen

(cherry picked from commit d1abc5d4fc)
2015-03-06 13:33:18 -08:00
Jian He a5f3fb4dc1 YARN-1809. Synchronize RM and TimeLineServer Web-UIs. Contributed by Zhijie Shen and Xuan Gong
(cherry picked from commit 95bfd087dc)
2015-03-05 21:27:03 -08:00
Vinod Kumar Vavilapalli 9c8882546a YARN-2786. Created a yarn cluster CLI and seeded with one command for listing node-labels collection. Contributed by Wangda Tan. 2015-03-05 10:56:28 -08:00
Tsuyoshi Ozawa 5680b41597 Update CHANGES.txt for YARN-2616 to fix indentation.
(cherry picked from commit 28b85a2116)
2015-03-06 00:53:20 +09:00
Tsuyoshi Ozawa 6660c2f83b YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. Contributed by Ryu Kobayashi.
(cherry picked from commit 1b67209612)
2015-03-05 19:56:59 +09:00
Karthik Kambatla 0d62e94887 YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client. (Zhihai Xu via kasha)
(cherry picked from commit 8d88691d16)
2015-03-04 19:49:21 -08:00
Karthik Kambatla 721d7b5741 YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending jobs. (Siqi Li via kasha)
(cherry picked from commit 22426a1c9f)
2015-03-04 18:07:33 -08:00
Karthik Kambatla 1749094fab YARN-3122. Metrics for container's actual CPU usage. (Anubhav Dhoot via kasha)
(cherry picked from commit 53947f37c7)
2015-03-04 17:39:39 -08:00
Jason Lowe ab397194c3 YARN-3131. YarnClientImpl should check FAILED and KILLED state in submitApplication. Contributed by Chang Li
(cherry picked from commit 03cc22945e)
2015-03-04 18:05:20 +00:00
Jian He aafe571398 YARN-3285. (Backport YARN-3168) Convert branch-2 .apt.vm files of YARN to markdown. Contributed by Masatake Iwasaki 2015-03-03 16:44:59 -08:00
Wangda Tan 0a502c665b YARN-3272. Surface container locality info in RM web UI (Jian He via wangda)
(cherry picked from commit e17e5ba9d7)
2015-03-03 11:49:45 -08:00
Vinod Kumar Vavilapalli 253c78548b YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. Contributed by Wangda Tan.
(cherry picked from commit 14dd647c55)
2015-03-02 17:54:01 -08:00
Wangda Tan 19a8e48230 YARN-3270. Fix node label expression not getting set in ApplicationSubmissionContext (Rohit Agarwal via wangda)
(cherry picked from commit abac6eb9d5)
2015-03-02 17:24:55 -08:00
Jian He 76b7283fa1 YARN-3281. Added RMStateStore to StateMachine visualization list. Contributed by Chengbing Liu
(cherry picked from commit 5d0bae550f)
2015-03-02 14:43:12 -08:00
Wangda Tan 0b0be0056b YARN-3262. Surface application outstanding resource requests table in RM web UI. (Jian He via wangda)
(cherry picked from commit edcecedc1c)
2015-02-27 16:14:35 -08:00
Tsuyoshi Ozawa 79f73f4613 YARN-2820. Retry in FileSystemRMStateStore when FS's operations fail due to IOException. Contributed by Zhihai Xu.
(cherry picked from commit 01a1621930)
2015-02-28 00:57:01 +09:00
Konstantin V Shvachko 9e67f2cb0d YARN-3255. RM, NM, JobHistoryServer, and WebAppProxyServer's main() should support generic options. Contributed by Konstantin Shvachko. 2015-02-26 17:26:08 -08:00
Tsuyoshi Ozawa dc348f4894 YARN-3217. Remove httpclient dependency from hadoop-yarn-server-web-proxy. Contributed by Brahma Reddy Battula.
(cherry picked from commit 773b6515ac)
2015-02-27 00:25:12 +09:00
Devaraj K f83d2e4410 YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against
all Schedulers even when using ParameterizedSchedulerTestBase. Contributed
by Anubhav Dhoot.

(cherry picked from commit 0d4296f0e0)
2015-02-26 15:47:32 +05:30
Jason Lowe 257087417e YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He
(cherry picked from commit 1a68fc4346)
2015-02-25 16:15:35 +00:00
Tsuyoshi Ozawa 13e213c41a YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. Contributed by Zhihai Xu.
(cherry picked from commit 6cbd9f1113)
2015-02-25 11:23:34 +09:00
Devaraj K b9e82e27f5 YARN-2797. Add -help to yarn logs and nodes CLI command. Contributed by
Jagadesh Kiran N.

(cherry picked from commit b610c68d44)
2015-02-24 12:08:50 +05:30
Xuan 0ca974d0b0 YARN-2797. TestWorkPreservingRMRestart should use
ParametrizedSchedulerTestBase. Contributed by Karthik Kambatla

(cherry picked from commit fe7a302473)
2015-02-21 19:19:08 -08:00
Xuan f8435c5d3d YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY.
Contributed by zhihai xu

(cherry picked from commit e3d290244c)
2015-02-21 16:20:33 -08:00
Xuan fefeba4ac8 YARN-3238. Connection timeouts to nodemanagers are retried at multiple
levels. Contributed by Jason Lowe

(cherry picked from commit 92d67ace32)
2015-02-21 16:07:28 -08:00
Xuan 9cbe9bcbaa YARN-3237. AppLogAggregatorImpl fails to log error cause. Contributed by
Rushabh S Shah

(cherry picked from commit f56c65bb3e)
2015-02-20 14:03:46 -08:00
Wangda Tan a1963968d2 YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
(cherry picked from commit ce5bf927c3)
2015-02-20 10:40:04 -08:00
Junping Du 4a87a61fe9 YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. Contributed by Zhihai Xu
(cherry picked from commit c33ae271c2)
2015-02-20 09:44:31 -08:00
Jason Lowe 3e8dfd1299 YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node. Contributed by Rohith
(cherry picked from commit a64dd3d24b)
2015-02-20 15:12:08 +00:00
Jian He eaeaf80d39 YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in RMApp. Contributed by Rohith Sharmaks
(cherry picked from commit c0d9b93953)
2015-02-19 15:43:14 -08:00
Wangda Tan cd5eb9c1de YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda)
(cherry picked from commit d49ae725d5)
2015-02-19 11:01:47 -08:00
Tsuyoshi Ozawa fef5efa0cb YARN-1615. Fix typos in delay scheduler's description. Contributed by Akira Ajisaka.
(cherry picked from commit b8a14efdf5)
2015-02-19 10:24:40 +09:00
Jian He b8fe1a747e YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA
(cherry picked from commit 1c03376300)
2015-02-18 16:34:42 -08:00
Jian He 8f06eaa251 YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. Contributed by Wangda Tan
(cherry picked from commit f5da5566d9)
2015-02-18 11:52:28 -08:00
Xuan ba18adbb27 YARN-3207. Secondary filter matches entites which do not have the key
being filtered for. Contributed by Zhijie Shen

(cherry picked from commit 57db50cbe3)
2015-02-17 18:18:56 -08:00
Tsuyoshi Ozawa fabac2ff30 YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and Devaraj K.
(cherry picked from commit 556386a070)
2015-02-17 01:54:09 +09:00
Tsuyoshi Ozawa 42368b91c4 Revert "YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and devaraj."
This reverts commit 67e3a77720.
2015-02-17 01:53:58 +09:00
Tsuyoshi Ozawa 67e3a77720 YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and devaraj.
(cherry picked from commit 9aae81c934)
2015-02-17 01:48:53 +09:00
Tsuyoshi Ozawa fd8f7e2c7d YARN-3203. Correct a log message in AuxServices. Contributed by Brahma Reddy Battula.
(cherry picked from commit 447bd7b5a6)
2015-02-16 23:56:32 +09:00
Junping Du eee89bd18c YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. (Contributed by Xuan Gong)
(cherry picked from commit ab0b958a52)
2015-02-15 06:48:29 -08:00
cnauroth 48302e687a YARN-2899. Run TestDockerContainerExecutorWithMocks on Linux only. Contributed by Ming Ma.
(cherry picked from commit 6804d68901)
2015-02-13 21:59:14 -08:00
Jason Lowe c9cd58d711 YARN-2847. Linux native container executor segfaults if default banned user detected. Contributed by Olaf Flebbe
(cherry picked from commit 1a0f508b63)
2015-02-13 20:22:12 +00:00
Xuan 3c4b5606dc YARN-3164. RMAdmin command usage prints incorrect command name.
Contributed by Bibin A Chundatt

(cherry picked from commit 253035491c)
2015-02-12 21:40:42 -08:00
Tsuyoshi Ozawa 613065d75a YARN-3182. Cleanup switch statement in ApplicationMasterLauncher#handle(). Contributed by Ray Chiang.
(cherry picked from commit 110cf6b9a3)
2015-02-13 14:21:43 +09:00
Tsuyoshi Ozawa 43429cd33a YARN-2994. Document work-preserving RM restart. Contributed by Jian He.
(cherry picked from commit b0d81e05ab)
2015-02-13 13:08:37 +09:00
Akira Ajisaka 9d8dbf8ea4 YARN-3191. Log object should be initialized with its own class. Contributed by Rohith.
(cherry picked from commit 99c7f2b8fa417303e8f42ea3641c302135879280)
2015-02-12 17:59:04 -08:00
Xuan df6bee5a75 YARN-3179. Update use of Iterator to Iterable in RMAdminCLI and
CommonNodeLabelsManager. Contributed by Ray Chiang

(cherry picked from commit 2586915bb3)
2015-02-12 17:32:58 -08:00
Xuan 5f0495689f YARN-3158. Correct log messages in ResourceTrackerService. Contributed
by Varun Saxena

(cherry picked from commit 99f6bd4f7a)
2015-02-12 16:43:28 -08:00
Jian He b5d6f76c6a YARN-3104. Fixed RM to not generate new AMRM tokens on every heartbeat between rolling and activation. Contributed by Jason Lowe
(cherry picked from commit 18297e0972)
2015-02-12 16:03:12 -08:00
Karthik Kambatla fcfbdf6440 YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)
(cherry picked from commit c2b185def8)
2015-02-12 15:39:50 -08:00
Jian He 1be2d64ddd YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan
(cherry picked from commit 18a594257e)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-02-12 15:01:14 -08:00
Junping Du 380cc4dbed YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager restart. (Contributed by Jason Lowe)
(cherry picked from commit 04f5ef18f7)
2015-02-12 11:48:24 -08:00
Xuan 58d9ce2ea1 YARN-3147. Clean up RM web proxy code. Contributed by Steve Loughran
(cherry picked from commit 83be450acc)
2015-02-12 10:59:18 -08:00
Tsuyoshi Ozawa dc14f2772f YARN-3157. Refactor the exception handling in ConverterUtils#to*Id. Contributed by Bibin A Chundatt.
(cherry picked from commit 7bc492adab)
2015-02-12 23:49:27 +09:00
Junping Du 634cc297ba YARN-1580. Documentation error regarding container-allocation.expiry-interval-ms (Brahma Reddy Battula via junping_du)
(cherry picked from commit 46c7577b98)
2015-02-12 03:40:57 -08:00
Tsuyoshi Ozawa fd047702e4 YARN-1237. Description for yarn.nodemanager.aux-services in yarn-default.xml is misleading. Contributed by Brahma Reddy Battula.
(cherry picked from commit b3bcbaf277)
2015-02-12 15:19:54 +09:00
Xuan cac6fe217a YARN-3151. On Failover tracking url wrong in application cli for KILLED
application. Contributed by Rohith

(cherry picked from commit 65c69e296e)
2015-02-11 21:21:06 -08:00
Jason Lowe 38333c8f29 YARN-3074. Nodemanager dies when localizer runner tries to write to a full disk. Contributed by Varun Saxena
(cherry picked from commit b379972ab3)
2015-02-11 16:34:42 +00:00
Steve Loughran a8eaec396f YARN-2683. [YARN-913] registry config options: document and move to core-default. (stevel) 2015-02-11 14:26:22 +00:00
Steve Loughran 4241c2be3d YARN-2616 [YARN-913] Add CLI client to the registry to list, view and manipulate entries. (Akshay Radia via stevel) 2015-02-11 14:18:46 +00:00
Junping Du f2c3058b8d YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Contributed by Chengbing Liu)
(cherry picked from commit c541a374d8)
2015-02-11 03:46:40 -08:00
Zhijie Shen fd75b8c9ca YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. Contributed by Devaraj K.
(cherry picked from commit d5855c0e46)
2015-02-10 15:29:09 -08:00