Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,459 Commits 326 Branches 371 Tags
Commit Graph

1390 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli
76cf2250de YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan. 
(cherry picked from commit b3056c266a628a65cf7ceb61b55ab3bd0a09baf2)
 2014-10-15 18:48:36 -07:00
Vinod Kumar Vavilapalli
e8e3a36213 YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan. 
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.

(cherry picked from commit f2ea555ac6c06a3f2f6559731f48711fff05d3f1)
 2014-10-15 18:34:19 -07:00
Jian He
f93d2ea27e YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:28:26 -07:00
Zhijie Shen
bec165e163 YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. 
(cherry picked from commit 1220bb72d452521c6f09cebe1dd77341054ee9dd)
 2014-10-14 21:56:44 -07:00
Zhijie Shen
8aeda20b25 HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(cherry picked from commit cdce88376a60918dfe2f3bcd82a7666d74992a19)
 2014-10-14 11:37:28 -07:00
Karthik Kambatla
88455173e8 YARN-2566. DefaultContainerExecutor should pick a working directory randomly. (Zhihai Xu via kasha) 
(cherry picked from commit cc93e7e683fa74eb1a7aa2b357a36667bd21086a)
 2014-10-13 16:32:42 -07:00
Karthik Kambatla
a1116b56a4 YARN-2641. Decommission nodes on -refreshNodes instead of next NM-RM heartbeat. (Zhihai Xu via kasha) 
(cherry picked from commit da709a2eac7110026169ed3fc4d0eaf85488d3ef)
 2014-10-13 16:24:00 -07:00
Jian He
31a4bf7321 Missing Changes.txt for YARN-2308 
(cherry picked from commit 178bc505da5d06d591a19aac13c040c6a9cf28ad)
 2014-10-13 14:20:18 -07:00
Jian He
536254f5e7 YARN-2308. Changed CapacityScheduler to explicitly throw exception if the queue 
to which the apps were submitted is changed across RM restart. Contributed by Craig Welch & Chang Li
(cherry picked from commit f9680d9a160ee527c8f2c1494584abf1a1f70f82)
 2014-10-13 14:16:12 -07:00
Jason Lowe
6b5741d423 YARN-2377. Localization exception stack traces are not passed as diagnostic info. Contributed by Gera Shegalov 
(cherry picked from commit a56ea0100215ecf2e1471a18812b668658197239)
 2014-10-13 18:32:38 +00:00
Zhijie Shen
e51ae64761 YARN-2651. Spun off LogRollingInterval from LogAggregationContext. Contributed by Xuan Gong. 
(cherry picked from commit 4aed2d8e91c7dccc78fbaffc409d3076c3316289)
 2014-10-13 10:55:09 -07:00
Jason Lowe
9ed9ea8c52 YARN-2667. Fix the release audit warning caused by hadoop-yarn-registry. Contributed by Yi Liu 
(cherry picked from commit 344a10ad5e26c25abd62eda65eec2820bb808a74)
 2014-10-13 15:40:16 +00:00
Steve Loughran
38dedc2f07 YARN-2668 yarn-registry JAR won't link against ZK 3.4.5. (stevel) 2014-10-11 18:33:29 -07:00
Vinod Kumar Vavilapalli
fff6a1f77e YARN-2501. Enhanced AMRMClient library to support requests against node labels. Contributed by Wangda Tan. 
(cherry picked from commit a5ec3d080978a67837946a991843a081ea712539)
 2014-10-10 19:58:42 -07:00
Vinod Kumar Vavilapalli
8ef83a9480 YARN-2494. Added NodeLabels Manager internal API and implementation. Contributed by Wangda Tan. 
(cherry picked from commit db7f1653198b950e89567c06898d64f6b930a0ee)
 2014-10-10 11:45:15 -07:00
Zhijie Shen
1e6d81a886 YARN-2583. Modified AggregatedLogDeletionService to be able to delete rolling aggregated logs. Contributed by Xuan Gong. 
(cherry picked from commit cb81bac0029fce3a9726df3523f0b692cd3375b8)
 2014-10-10 00:16:34 -07:00
cnauroth
b81641a310 YARN-2662. TestCgroupsLCEResourcesHandler leaks file descriptors. Contributed by Chris Nauroth. 
(cherry picked from commit d3afd730acfa380ab5032be5ee296c5d73744518)
 2014-10-09 22:47:04 -07:00
Zhijie Shen
79204dced9 YARN-2671. Fix the Jira number in the change log. 
(cherry picked from commit 5b12df6587eb4f37d09c9ffc35a0ea59694df831)
 2014-10-09 20:59:37 -07:00
Zhijie Shen
ad47a27dbf YARN-2617. Fixed ApplicationSubmissionContext to still set resource for backward compatibility. Contributed by Wangda Tan. 
(cherry picked from commit e532ed8faa8db4b008a5b8d3f82b48a1b314fa6c)
 2014-10-09 18:00:01 -07:00
Karthik Kambatla
7e20187f91 YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) 
(cherry picked from commit 4f426fe2232ed90d8fdf8619fbdeae28d788b5c8)
 2014-10-09 17:16:45 -07:00
Vinod Kumar Vavilapalli
d3125f3de5 YARN-2544. Added admin-API objects for using node-labels. Contributed by Wangda Tan. 
(cherry picked from commit 596702a02501e9cb09aabced168027189eaf02ba)
 2014-10-09 15:47:20 -07:00
Mayank Bansal
c1a86d105e YARN-2598 GHS should show N/A instead of null for the inaccessible information (Zhijie Shen via mayank) 2014-10-09 13:43:36 -07:00
Zhijie Shen
4d170d3db6 YARN-2629. Made the distributed shell use the domain-based timeline ACLs. Contributed by Zhijie Shen. 
(cherry picked from commit 1d4612f5ad9678c952b416e798dccd20c88f96ef)
 2014-10-09 13:01:44 -07:00
Vinod Kumar Vavilapalli
b42cd54d7a YARN-2493. Added user-APIs for using node-labels. Contributed by Wangda Tan. 
(cherry picked from commit 180afa2f86f9854c536c3d4ff7476880e41ac48d)
 2014-10-09 11:27:17 -07:00
Steve Loughran
d218ab58fc YARN-913 service registry: YARN-2652 add hadoop-yarn-registry package under hadoop-yarn 2014-10-08 12:55:47 -07:00
Jian He
5e6f86e3aa YARN-2649. Fixed TestAMRMRPCNodeUpdates test failure. Contributed by Ming Ma 
(cherry picked from commit e16e25ab1beac89c8d8be4e9f2a7fbefe81d35f3)
 2014-10-08 11:03:00 -07:00
Jian He
9ff7c0c96c YARN-1857. CapacityScheduler headroom doesn't account for other AM's running. Contributed by Chen He and Craig Welch 
(cherry picked from commit 30d56fdbb40d06c4e267d6c314c8c767a7adc6a3)
 2014-10-07 13:48:03 -07:00
Jian He
5c33e91229 YARN-2644. Fixed CapacityScheduler to return up-to-date headroom when AM allocates. Contributed by Craig Welch 
(cherry picked from commit 519e5a7dd2bd540105434ec3c8939b68f6c024f8)
 2014-10-06 15:51:38 -07:00
Jian He
43358be60b YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier to use protobuf as payload. Contributed by Junping Du 
(cherry picked from commit ea26cc0b4ac02b8af686dfda80f540e5aa70c358)
 2014-10-06 10:48:27 -07:00
Chris Douglas
4ba102bdc3 YARN-1051. Add a system for creating reservations of cluster capacity. 
Contributed by Subru Krishnan and Carlo Curino.
(cherry picked from commit c8212bacb1b2a7e6ee83cc56f72297465ce99390)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-06 10:30:53 -07:00
subru
fb5e9df7fd YARN-2611. Fixing jenkins findbugs warning and TestRMWebServicesCapacitySched for branch YARN-1051. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit c47464aba407d1dafe10be23fe454f0489cc4367)
(cherry picked from commit a2986234be4e02f9ccb589f9ff5f7ffb28bc6400)
 2014-10-06 10:21:07 -07:00
subru
ae0f16ccc8 YARN-2576. Making test patch pass in branch. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 90ac0be86b898aefec5471db4027554c8e1b310c)
(cherry picked from commit 5e10a13bb4759984494c6a870c7f08fb6693c9c0)
 2014-10-06 10:21:06 -07:00
subru
cbfbdf60d6 YARN-2080. Integrating reservation system with ResourceManager and client-RM protocol. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 8baeaead8532898163f1006276b731a237b1a559)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
(cherry picked from commit 6261f7cc69a0eb3eebc9898c7599c7c20f432b4e)
 2014-10-06 10:21:06 -07:00
subru
fec639cda1 YARN-1712. Plan follower that synchronizes the current state of reservation subsystem with the scheduler. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 169085319b8b76641f8b9f6840a3fef06d221e2b)
(cherry picked from commit 3418c56bcf4bbddaf483bdaa1a15a8bbc4039bfe)
 2014-10-06 10:21:06 -07:00
carlo curino
6bfdaf06c4 YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit c4918cb4cb5a267a8cfd6eace28fcfe7ad6174e8)
(cherry picked from commit b6df0dddcdafd7ec67c76ea92aea3ff3e94db247)
 2014-10-06 10:21:05 -07:00
carlo curino
6a3c167175 YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit aef7928899b37262773f3dc117157bb746bf8918)
(cherry picked from commit f66ffcf832235e0da0bb050fff08e248b547c360)
 2014-10-06 10:21:05 -07:00
subru
63250ef9d6 YARN-1709. In-memory data structures used to track resources over time to enable reservations. 
(cherry picked from commit 0d8b2cd88b958b1e602fd4ea4078ef8d4742a7c3)
(cherry picked from commit cf4b34282aafee9f6b09d3433c4de1ae4b359168)
 2014-10-06 10:21:05 -07:00
subru
3a167056d1 YARN-1708. Public YARN APIs for creating/updating/deleting reservations. 
(cherry picked from commit 3f2e3b275bcf29264a112c5d2f4eae289fe7f52c)
(cherry picked from commit c9266df40434fdde05bd3a4c75f7a6bc29db760f)
 2014-10-06 10:21:05 -07:00
carlo curino
b81f571e60 YARN-2475. Logic for responding to capacity drops for the ReservationSystem. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit f83a07f266f2c5e6eead554d8a331ed7e75e10d5)
(cherry picked from commit 1c6950354f3c35a7824770dc251d5aec3be4876a)
 2014-10-06 10:21:05 -07:00
carlo curino
cf5ef00b96 YARN-1707. Introduce APIs to add/remove/resize queues in the CapacityScheduler. Contributed by Carlo Curino and Subru Krishnan 
(cherry picked from commit aac47fda7fecda9fc18ade34d633eca895865a70)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
(cherry picked from commit eb3e40b833b9d82c1556843f960194dc42e482f3)
 2014-10-06 10:21:04 -07:00
Vinod Kumar Vavilapalli
7ed61e150c YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for use by long running services. Contributed by Xuan Gong. 
(cherry picked from commit 34cdcaad71cad76c0874a4e5266b4074009d2ffc)
 2014-10-03 12:17:03 -07:00
Karthik Kambatla
b6ce0a1f69 YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 
(cherry picked from commit 80d11eb68e60f88e16d7d41edecbddfc935a6b10)
 2014-10-03 11:52:05 -07:00
Jian He
1c58ab9206 YARN-2562. Changed ContainerId#toString() to be more readable. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit eb6ce5e97c6a379caf9de0b34cc7770c4447bb1a)
 2014-10-03 11:04:46 -07:00
Jian He
6f4c77409b YARN-2628. Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free. Contributed by Varun Vasudev 
(cherry picked from commit 054f28552687e9b9859c0126e16a2066e20ead3f)
 2014-10-02 15:16:07 -07:00
Zhijie Shen
ecaae67ab3 YARN-2527. Fixed the potential NPE in ApplicationACLsManager and added test cases for it. Contributed by Benoy Antony. 
(cherry picked from commit 1c93025a1b370db46e345161dbc15e03f829823f)
 2014-10-02 14:57:27 -07:00
Jason Lowe
531c1fd00a YARN-2624. Resource Localization fails on a cluster due to existing cache directories. Contributed by Anubhav Dhoot 
(cherry picked from commit 29f520052e2b02f44979980e446acc0dccd96d54)
 2014-10-02 17:40:44 +00:00
Karthik Kambatla
b6f4d1ef42 YARN-2254. TestRMWebServicesAppsModification should run against both CS and FS. (Zhihai Xu via kasha) 
(cherry picked from commit 5e0b49da9caa53814581508e589f3704592cf335)
 2014-10-02 10:15:41 -07:00
Jian He
61c7ceaf82 YARN-2617. Fixed NM to not send duplicate container status whose app is not running. Contributed by Jun Gong 
(cherry picked from commit 3ef1cf187faeb530e74606dd7113fd1ba08140d7)
 2014-10-02 10:04:42 -07:00
junping_du
6483342a61 YARN-1979. TestDirectoryCollection fails when the umask is unusual. (Contributed by Vinod Kumar Vavilapalli and Tsuyoshi OZAWA) 
(cherry picked from commit c7cee9b4551918d5d35bf4e9dc73982a050c73ba)
 2014-10-02 08:04:25 -07:00
Vinod Kumar Vavilapalli
34dd2d1d1c YARN-2446. Augmented Timeline service APIs to start taking in domains as a parameter while posting entities and events. Contributed by Zhijie Shen. 
(cherry picked from commit 9e40de6af7959ac7bb5f4e4d2833ca14ea457614)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-01 17:54:12 -07:00