Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,664 Commits 326 Branches 371 Tags
Commit Graph

1458 Commits

Author SHA1 Message Date
subru
cbfbdf60d6 YARN-2080. Integrating reservation system with ResourceManager and client-RM protocol. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 8baeaead8532898163f1006276b731a237b1a559)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
(cherry picked from commit 6261f7cc69a0eb3eebc9898c7599c7c20f432b4e)
 2014-10-06 10:21:06 -07:00
subru
fec639cda1 YARN-1712. Plan follower that synchronizes the current state of reservation subsystem with the scheduler. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 169085319b8b76641f8b9f6840a3fef06d221e2b)
(cherry picked from commit 3418c56bcf4bbddaf483bdaa1a15a8bbc4039bfe)
 2014-10-06 10:21:06 -07:00
carlo curino
6bfdaf06c4 YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit c4918cb4cb5a267a8cfd6eace28fcfe7ad6174e8)
(cherry picked from commit b6df0dddcdafd7ec67c76ea92aea3ff3e94db247)
 2014-10-06 10:21:05 -07:00
carlo curino
6a3c167175 YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit aef7928899b37262773f3dc117157bb746bf8918)
(cherry picked from commit f66ffcf832235e0da0bb050fff08e248b547c360)
 2014-10-06 10:21:05 -07:00
subru
63250ef9d6 YARN-1709. In-memory data structures used to track resources over time to enable reservations. 
(cherry picked from commit 0d8b2cd88b958b1e602fd4ea4078ef8d4742a7c3)
(cherry picked from commit cf4b34282aafee9f6b09d3433c4de1ae4b359168)
 2014-10-06 10:21:05 -07:00
subru
3a167056d1 YARN-1708. Public YARN APIs for creating/updating/deleting reservations. 
(cherry picked from commit 3f2e3b275bcf29264a112c5d2f4eae289fe7f52c)
(cherry picked from commit c9266df40434fdde05bd3a4c75f7a6bc29db760f)
 2014-10-06 10:21:05 -07:00
carlo curino
b81f571e60 YARN-2475. Logic for responding to capacity drops for the ReservationSystem. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit f83a07f266f2c5e6eead554d8a331ed7e75e10d5)
(cherry picked from commit 1c6950354f3c35a7824770dc251d5aec3be4876a)
 2014-10-06 10:21:05 -07:00
carlo curino
cf5ef00b96 YARN-1707. Introduce APIs to add/remove/resize queues in the CapacityScheduler. Contributed by Carlo Curino and Subru Krishnan 
(cherry picked from commit aac47fda7fecda9fc18ade34d633eca895865a70)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
(cherry picked from commit eb3e40b833b9d82c1556843f960194dc42e482f3)
 2014-10-06 10:21:04 -07:00
Vinod Kumar Vavilapalli
7ed61e150c YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for use by long running services. Contributed by Xuan Gong. 
(cherry picked from commit 34cdcaad71cad76c0874a4e5266b4074009d2ffc)
 2014-10-03 12:17:03 -07:00
Karthik Kambatla
b6ce0a1f69 YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 
(cherry picked from commit 80d11eb68e60f88e16d7d41edecbddfc935a6b10)
 2014-10-03 11:52:05 -07:00
Jian He
1c58ab9206 YARN-2562. Changed ContainerId#toString() to be more readable. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit eb6ce5e97c6a379caf9de0b34cc7770c4447bb1a)
 2014-10-03 11:04:46 -07:00
Jian He
6f4c77409b YARN-2628. Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free. Contributed by Varun Vasudev 
(cherry picked from commit 054f28552687e9b9859c0126e16a2066e20ead3f)
 2014-10-02 15:16:07 -07:00
Zhijie Shen
ecaae67ab3 YARN-2527. Fixed the potential NPE in ApplicationACLsManager and added test cases for it. Contributed by Benoy Antony. 
(cherry picked from commit 1c93025a1b370db46e345161dbc15e03f829823f)
 2014-10-02 14:57:27 -07:00
Jason Lowe
531c1fd00a YARN-2624. Resource Localization fails on a cluster due to existing cache directories. Contributed by Anubhav Dhoot 
(cherry picked from commit 29f520052e2b02f44979980e446acc0dccd96d54)
 2014-10-02 17:40:44 +00:00
Karthik Kambatla
b6f4d1ef42 YARN-2254. TestRMWebServicesAppsModification should run against both CS and FS. (Zhihai Xu via kasha) 
(cherry picked from commit 5e0b49da9caa53814581508e589f3704592cf335)
 2014-10-02 10:15:41 -07:00
Jian He
61c7ceaf82 YARN-2617. Fixed NM to not send duplicate container status whose app is not running. Contributed by Jun Gong 
(cherry picked from commit 3ef1cf187faeb530e74606dd7113fd1ba08140d7)
 2014-10-02 10:04:42 -07:00
junping_du
6483342a61 YARN-1979. TestDirectoryCollection fails when the umask is unusual. (Contributed by Vinod Kumar Vavilapalli and Tsuyoshi OZAWA) 
(cherry picked from commit c7cee9b4551918d5d35bf4e9dc73982a050c73ba)
 2014-10-02 08:04:25 -07:00
Vinod Kumar Vavilapalli
34dd2d1d1c YARN-2446. Augmented Timeline service APIs to start taking in domains as a parameter while posting entities and events. Contributed by Zhijie Shen. 
(cherry picked from commit 9e40de6af7959ac7bb5f4e4d2833ca14ea457614)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-01 17:54:12 -07:00
Vinod Kumar Vavilapalli
3326fba382 YARN-1972. Added a secure container-executor for Windows. Contributed by Remus Rusanu. 
commit ba7f31c2ee8d23ecb183f88920ef06053c0b9769 is the corresponding trunk commit, this is a slightly different patch for branch-2.
 2014-10-01 17:07:21 -07:00
junping_du
625456746c YARN-2613. Support retry in NMClient for rolling-upgrades. (Contributed by Jian He) 2014-10-01 17:08:55 -07:00
Zhijie Shen
4b50e23271 YARN-2630. Prevented previous AM container status from being acquired by the current restarted AM. Contributed by Jian He. 
(cherry picked from commit 52bbe0f11bc8e97df78a1ab9b63f4eff65fd7a76)
 2014-10-01 15:39:36 -07:00
Vinod Kumar Vavilapalli
64bed0460e YARN-1063. Augmented Hadoop common winutils to have the ability to create containers as domain users. Contributed by Remus Rusanu. 
Committed as a YARN patch even though all the code changes are in common.

(cherry picked from commit 5ca97f1e60b8a7848f6eadd15f6c08ed390a8cda)
 2014-10-01 09:55:57 -07:00
Karthik Kambatla
3664659592 YARN-2179. Fix version in sharedcachemanager-pom. 2014-10-01 07:34:24 -07:00
Karthik Kambatla
b616c69835 YARN-2179. [YARN-1492] Initial cache manager structure and context. (Chris Trezzo via kasha) 
(cherry picked from commit 17d1202c35a1992eab66ea05dfd2baf219a17aec)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/bin/yarn
 2014-09-30 19:05:10 -07:00
Jian He
823f02725f YARN-2602. Fixed possible NPE in ApplicationHistoryManagerOnTimelineStore. Contributed by Zhijie Shen 
(cherry picked from commit bbff96be48119774688981d04baf444639135977)
 2014-09-30 16:44:44 -07:00
Karthik Kambatla
5a43b795b7 YARN-2594. Potential deadlock in RM when querying ApplicationResourceUsageReport. (Wangda Tan via kasha) 
(cherry picked from commit 14d60dadc25b044a2887bf912ba5872367f2dffb)
 2014-09-30 16:17:38 -07:00
Jason Lowe
a0305ba3fc YARN-2387. Resource Manager crashes with NPE due to lack of synchronization. Contributed by Mit Desai 
(cherry picked from commit feaf139b4f327d33011e5a4424c06fb44c630955)
 2014-09-30 22:38:27 +00:00
Zhijie Shen
d977205b9b YARN-2627. Added the info logs of attemptFailuresValidityInterval and number of previous failed attempts. Contributed by Xuan Gong. 
(cherry picked from commit 9582a50176800433ad3fa8829a50c28b859812a3)
 2014-09-30 12:59:34 -07:00
Karthik Kambatla
21f920ab86 YARN-2610. Hamlet should close table tags. (Ray Chiang via kasha) 
(cherry picked from commit f7743dd07dfbe0dde9be71acfaba16ded52adba7)
 2014-09-30 11:47:01 -07:00
Arun C. Murthy
6a60fa279a Preparing to branch hadoop-2.6 2014-09-30 10:37:03 -07:00
Jonathan Eagles
9bc676aa6e YARN-2606. Application History Server tries to access hdfs before doing secure login (Mit Desai via jeagles) 2014-09-29 16:40:46 -05:00
Jason Lowe
4f6a367166 YARN-1769. CapacityScheduler: Improve reservations. Contributed by Thomas Graves 
(cherry picked from commit 9c22065109a77681bc2534063eabe8692fbcb3cd)
 2014-09-29 14:13:21 +00:00
Jian He
cb08ed1484 YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier to use protobuf object as the payload. Contributed by Junping Du. 
(cherry picked from commit 5391919b09ce9549d13c897aa89bb0a0536760fe)
 2014-09-26 17:53:35 -07:00
Allen Wittenauer
6f5d555ad0 YARN-2372. There are Chinese Characters in the FairScheduler's document (Fengdong Yu via aw) 2014-09-26 12:47:26 -07:00
Allen Wittenauer
f694b44036 YARN-2577. Clarify ACL delimiter and how to configure ACL groups only (Mikos Christine via aw) 2014-09-26 11:06:04 -07:00
Karthik Kambatla
b923c291b4 YARN-2608. FairScheduler: Potential deadlocks in loading alloc files and clock access. (Wei Yan via kasha) 
(cherry picked from commit c9811af09a3d3f9f2f1b86fc9d6f2763d3225e44)
 2014-09-25 17:43:08 -07:00
Jason Lowe
e4d46e5ff6 YARN-2523. ResourceManager UI showing negative value for "Decommissioned Nodes" field. Contributed by Rohith 
(cherry picked from commit 8269bfa613999f71767de3c0369817b58cfe1416)
 2014-09-25 22:39:57 +00:00
Zhijie Shen
83926b3c57 YARN-2546. Made REST API for application creation/submission use numeric and boolean types instead of the string of them. Contributed by Varun Vasudev. 
(cherry picked from commit 72b0881ca641fa830c907823f674a5c5e39aa15a)
 2014-09-24 17:58:48 -07:00
Zhijie Shen
3a2e400377 YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier. Contributed by Xuan Gong. 
(cherry picked from commit c86674a3a4d99aa56bb8ed3f6df51e3fef215eba)
 2014-09-24 17:51:54 -07:00
Karthik Kambatla
30165d4cdd YARN-2596. TestWorkPreservingRMRestart fails with FairScheduler. (kasha) 
(cherry picked from commit 39c87344e16a08ab69e25345b3bce92aec92db47)
 2014-09-24 15:24:17 -07:00
Vinod Kumar Vavilapalli
5d251d99d6 YARN-2102. Added the concept of a Timeline Domain to handle read/write ACLs on Timeline service event data. Contributed by Zhijie Shen. 
(cherry picked from commit d78b452a4f413c6931a494c33df0666ce9b44973)
 2014-09-24 11:29:01 -07:00
Allen Wittenauer
e5a531a8a7 YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) 2014-09-24 08:48:42 -07:00
Zhijie Shen
19cfe2f800 YARN-2569. Added the log handling APIs for the long running services. Contributed by Xuan Gong. 
(cherry picked from commit 5338ac416ab8ab3e7e0a7bfb4a53151fc457f673)
 2014-09-23 10:38:27 -07:00
Karthik Kambatla
024bcfdc0d YARN-2252. Intermittent failure of TestFairScheduler.testContinuousScheduling. (Ratandeep Ratti and kasha via kasha) 
(cherry picked from commit f5578207d2d20aa71adf12aa9f6c37b00ee9b8dc)
 2014-09-23 00:03:47 -07:00
Karthik Kambatla
8ba8521de5 YARN-1959. Fix headroom calculation in FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit 568d3dc2bbe43b7d2833d5da2b0e6d75eb86e5dd)
 2014-09-22 23:51:45 -07:00
junping_du
34ee084c3d YARN-2584. TestContainerManagerSecurity fails on trunk. (Contributed by Jian He) 
(cherry picked from commit a9a55db0650b5e6b7d88afb883b88aba7a13cd44)
 2014-09-22 22:47:59 -07:00
Karthik Kambatla
3984435661 YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5. (Wei Yan via kasha) 
(cherry picked from commit 43efdd30b52f4ee8967a03dac4d548095d6cafdb)
 2014-09-22 16:10:20 -07:00
Karthik Kambatla
24a814771e YARN-2540. FairScheduler: Queue filters not working on scheduler page in RM UI. (Ashwin Shankar via kasha) 
(cherry picked from commit 23e17ce0ffba11a08c033b753573a756532cd1c2)
 2014-09-22 15:42:36 -07:00
Zhijie Shen
b8d645539d MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN scheduler resource type is memory plus cpu. Contributed by Peng Zhang and Varun Vasudev. 2014-09-22 10:41:55 -07:00
Jian He
3ce97a9efd YARN-1372. Ensure all completed containers are reported to the AMs across RM restart. Contributed by Anubhav Dhoot 
(cherry picked from commit 0a641496c706fc175e7bf66d69ebf71c7d078e84)
 2014-09-22 10:32:44 -07:00