Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,371 Commits 326 Branches 371 Tags
Commit Graph

961 Commits

Author SHA1 Message Date
carlo curino
6a3c167175 YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit aef7928899b37262773f3dc117157bb746bf8918)
(cherry picked from commit f66ffcf832235e0da0bb050fff08e248b547c360)
 2014-10-06 10:21:05 -07:00
subru
63250ef9d6 YARN-1709. In-memory data structures used to track resources over time to enable reservations. 
(cherry picked from commit 0d8b2cd88b958b1e602fd4ea4078ef8d4742a7c3)
(cherry picked from commit cf4b34282aafee9f6b09d3433c4de1ae4b359168)
 2014-10-06 10:21:05 -07:00
carlo curino
b81f571e60 YARN-2475. Logic for responding to capacity drops for the ReservationSystem. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit f83a07f266f2c5e6eead554d8a331ed7e75e10d5)
(cherry picked from commit 1c6950354f3c35a7824770dc251d5aec3be4876a)
 2014-10-06 10:21:05 -07:00
carlo curino
cf5ef00b96 YARN-1707. Introduce APIs to add/remove/resize queues in the CapacityScheduler. Contributed by Carlo Curino and Subru Krishnan 
(cherry picked from commit aac47fda7fecda9fc18ade34d633eca895865a70)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
(cherry picked from commit eb3e40b833b9d82c1556843f960194dc42e482f3)
 2014-10-06 10:21:04 -07:00
Vinod Kumar Vavilapalli
7ed61e150c YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for use by long running services. Contributed by Xuan Gong. 
(cherry picked from commit 34cdcaad71cad76c0874a4e5266b4074009d2ffc)
 2014-10-03 12:17:03 -07:00
Karthik Kambatla
b6ce0a1f69 YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 
(cherry picked from commit 80d11eb68e60f88e16d7d41edecbddfc935a6b10)
 2014-10-03 11:52:05 -07:00
Jian He
6f4c77409b YARN-2628. Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free. Contributed by Varun Vasudev 
(cherry picked from commit 054f28552687e9b9859c0126e16a2066e20ead3f)
 2014-10-02 15:16:07 -07:00
Jason Lowe
531c1fd00a YARN-2624. Resource Localization fails on a cluster due to existing cache directories. Contributed by Anubhav Dhoot 
(cherry picked from commit 29f520052e2b02f44979980e446acc0dccd96d54)
 2014-10-02 17:40:44 +00:00
Karthik Kambatla
b6f4d1ef42 YARN-2254. TestRMWebServicesAppsModification should run against both CS and FS. (Zhihai Xu via kasha) 
(cherry picked from commit 5e0b49da9caa53814581508e589f3704592cf335)
 2014-10-02 10:15:41 -07:00
Jian He
61c7ceaf82 YARN-2617. Fixed NM to not send duplicate container status whose app is not running. Contributed by Jun Gong 
(cherry picked from commit 3ef1cf187faeb530e74606dd7113fd1ba08140d7)
 2014-10-02 10:04:42 -07:00
junping_du
6483342a61 YARN-1979. TestDirectoryCollection fails when the umask is unusual. (Contributed by Vinod Kumar Vavilapalli and Tsuyoshi OZAWA) 
(cherry picked from commit c7cee9b4551918d5d35bf4e9dc73982a050c73ba)
 2014-10-02 08:04:25 -07:00
Vinod Kumar Vavilapalli
34dd2d1d1c YARN-2446. Augmented Timeline service APIs to start taking in domains as a parameter while posting entities and events. Contributed by Zhijie Shen. 
(cherry picked from commit 9e40de6af7959ac7bb5f4e4d2833ca14ea457614)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-01 17:54:12 -07:00
Vinod Kumar Vavilapalli
3326fba382 YARN-1972. Added a secure container-executor for Windows. Contributed by Remus Rusanu. 
commit ba7f31c2ee8d23ecb183f88920ef06053c0b9769 is the corresponding trunk commit, this is a slightly different patch for branch-2.
 2014-10-01 17:07:21 -07:00
junping_du
625456746c YARN-2613. Support retry in NMClient for rolling-upgrades. (Contributed by Jian He) 2014-10-01 17:08:55 -07:00
Zhijie Shen
4b50e23271 YARN-2630. Prevented previous AM container status from being acquired by the current restarted AM. Contributed by Jian He. 
(cherry picked from commit 52bbe0f11bc8e97df78a1ab9b63f4eff65fd7a76)
 2014-10-01 15:39:36 -07:00
Karthik Kambatla
3664659592 YARN-2179. Fix version in sharedcachemanager-pom. 2014-10-01 07:34:24 -07:00
Karthik Kambatla
b616c69835 YARN-2179. [YARN-1492] Initial cache manager structure and context. (Chris Trezzo via kasha) 
(cherry picked from commit 17d1202c35a1992eab66ea05dfd2baf219a17aec)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/bin/yarn
 2014-09-30 19:05:10 -07:00
Jian He
823f02725f YARN-2602. Fixed possible NPE in ApplicationHistoryManagerOnTimelineStore. Contributed by Zhijie Shen 
(cherry picked from commit bbff96be48119774688981d04baf444639135977)
 2014-09-30 16:44:44 -07:00
Karthik Kambatla
5a43b795b7 YARN-2594. Potential deadlock in RM when querying ApplicationResourceUsageReport. (Wangda Tan via kasha) 
(cherry picked from commit 14d60dadc25b044a2887bf912ba5872367f2dffb)
 2014-09-30 16:17:38 -07:00
Zhijie Shen
d977205b9b YARN-2627. Added the info logs of attemptFailuresValidityInterval and number of previous failed attempts. Contributed by Xuan Gong. 
(cherry picked from commit 9582a50176800433ad3fa8829a50c28b859812a3)
 2014-09-30 12:59:34 -07:00
Jonathan Eagles
9bc676aa6e YARN-2606. Application History Server tries to access hdfs before doing secure login (Mit Desai via jeagles) 2014-09-29 16:40:46 -05:00
Jason Lowe
4f6a367166 YARN-1769. CapacityScheduler: Improve reservations. Contributed by Thomas Graves 
(cherry picked from commit 9c22065109a77681bc2534063eabe8692fbcb3cd)
 2014-09-29 14:13:21 +00:00
Jian He
cb08ed1484 YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier to use protobuf object as the payload. Contributed by Junping Du. 
(cherry picked from commit 5391919b09ce9549d13c897aa89bb0a0536760fe)
 2014-09-26 17:53:35 -07:00
Karthik Kambatla
b923c291b4 YARN-2608. FairScheduler: Potential deadlocks in loading alloc files and clock access. (Wei Yan via kasha) 
(cherry picked from commit c9811af09a3d3f9f2f1b86fc9d6f2763d3225e44)
 2014-09-25 17:43:08 -07:00
Jason Lowe
e4d46e5ff6 YARN-2523. ResourceManager UI showing negative value for "Decommissioned Nodes" field. Contributed by Rohith 
(cherry picked from commit 8269bfa613999f71767de3c0369817b58cfe1416)
 2014-09-25 22:39:57 +00:00
Zhijie Shen
83926b3c57 YARN-2546. Made REST API for application creation/submission use numeric and boolean types instead of the string of them. Contributed by Varun Vasudev. 
(cherry picked from commit 72b0881ca641fa830c907823f674a5c5e39aa15a)
 2014-09-24 17:58:48 -07:00
Zhijie Shen
3a2e400377 YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier. Contributed by Xuan Gong. 
(cherry picked from commit c86674a3a4d99aa56bb8ed3f6df51e3fef215eba)
 2014-09-24 17:51:54 -07:00
Karthik Kambatla
30165d4cdd YARN-2596. TestWorkPreservingRMRestart fails with FairScheduler. (kasha) 
(cherry picked from commit 39c87344e16a08ab69e25345b3bce92aec92db47)
 2014-09-24 15:24:17 -07:00
Vinod Kumar Vavilapalli
5d251d99d6 YARN-2102. Added the concept of a Timeline Domain to handle read/write ACLs on Timeline service event data. Contributed by Zhijie Shen. 
(cherry picked from commit d78b452a4f413c6931a494c33df0666ce9b44973)
 2014-09-24 11:29:01 -07:00
Allen Wittenauer
e5a531a8a7 YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) 2014-09-24 08:48:42 -07:00
Karthik Kambatla
024bcfdc0d YARN-2252. Intermittent failure of TestFairScheduler.testContinuousScheduling. (Ratandeep Ratti and kasha via kasha) 
(cherry picked from commit f5578207d2d20aa71adf12aa9f6c37b00ee9b8dc)
 2014-09-23 00:03:47 -07:00
Karthik Kambatla
8ba8521de5 YARN-1959. Fix headroom calculation in FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit 568d3dc2bbe43b7d2833d5da2b0e6d75eb86e5dd)
 2014-09-22 23:51:45 -07:00
junping_du
34ee084c3d YARN-2584. TestContainerManagerSecurity fails on trunk. (Contributed by Jian He) 
(cherry picked from commit a9a55db0650b5e6b7d88afb883b88aba7a13cd44)
 2014-09-22 22:47:59 -07:00
Karthik Kambatla
3984435661 YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5. (Wei Yan via kasha) 
(cherry picked from commit 43efdd30b52f4ee8967a03dac4d548095d6cafdb)
 2014-09-22 16:10:20 -07:00
Karthik Kambatla
24a814771e YARN-2540. FairScheduler: Queue filters not working on scheduler page in RM UI. (Ashwin Shankar via kasha) 
(cherry picked from commit 23e17ce0ffba11a08c033b753573a756532cd1c2)
 2014-09-22 15:42:36 -07:00
Jian He
3ce97a9efd YARN-1372. Ensure all completed containers are reported to the AMs across RM restart. Contributed by Anubhav Dhoot 
(cherry picked from commit 0a641496c706fc175e7bf66d69ebf71c7d078e84)
 2014-09-22 10:32:44 -07:00
Karthik Kambatla
c1a3819a4d YARN-2453. TestProportionalCapacityPreemptionPolicy fails with FairScheduler. (Zhihai Xu via kasha) 
(cherry picked from commit 9721e2c1feb5aecea3a6dab5bda96af1cd0f8de3)
 2014-09-21 23:14:27 -07:00
Karthik Kambatla
1714af0618 YARN-2452. TestRMApplicationHistoryWriter fails with FairScheduler. (Zhihai Xu via kasha) 
(cherry picked from commit c50fc92502934aa2a8f84ea2466d4da1e3eace9d)
 2014-09-21 13:15:45 -07:00
Jian He
07981c8f60 YARN-2565. Fixed RM to not use FileSystemApplicationHistoryStore unless explicitly set. Contributed by Zhijie Shen 
(cherry picked from commit 444acf8ea795e4bc782f1ce3b5ef7a1a47d1d27d)
 2014-09-19 11:28:22 -07:00
Zhijie Shen
9333ee3068 YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He. 
(cherry picked from commit 6fe5c6b746a40019b9a43676c33efec0f971c4b9)
 2014-09-18 22:02:53 -07:00
Jason Lowe
d9273a9547 YARN-2561. MR job client cannot reconnect to AM after NM restart. Contributed by Junping Du 
(cherry picked from commit a337f0e3549351344bce70cb23ddc0a256c894b0)
 2014-09-18 21:36:20 +00:00
Jason Lowe
44c22c3d8d YARN-2363. Submitted applications occasionally lack a tracking URL. Contributed by Jason Lowe 
(cherry picked from commit 9ea7b6c063c0bdd4551962e21d0173f671e9df03)
 2014-09-18 20:15:46 +00:00
Vinod Kumar Vavilapalli
d24ef142b7 YARN-2001. Added a time threshold for RM to wait before starting container allocations after restart/failover. Contributed by Jian He. 
(cherry picked from commit 485c96e3cb9b0b05d6e490b4773506da83ebc61d)
 2014-09-18 11:04:27 -07:00
Jian He
d61cdd66db YARN-2559. Fixed NPE in SystemMetricsPublisher when retrieving FinalApplicationStatus. Contributed by Zhijie Shen 
(cherry picked from commit ee21b13cbd4654d7181306404174329f12193613)
 2014-09-17 21:44:49 -07:00
Jian He
3746b1e905 YARN-2558. Updated ContainerTokenIdentifier#read/write to use ContainerId#getContainerId. Contributed by Tsuyoshi OZAWA. 
(cherry picked from commit f4886111aa573ec928de69e8ca9328d480bf673e)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-09-17 15:22:02 -07:00
junping_du
1e1f4c2fdf YARN-1250. Generic history service should support application-acls. (Contributed by Zhijie Shen) 
(cherry picked from commit 90a0c03f0a696d32e871a5da4560828edea8cfa9)
 2014-09-16 18:30:06 -07:00
Vinod Kumar Vavilapalli
9d34dc87e1 YARN-2531. Added a configuration for admins to be able to override app-configs and enforce/not-enforce strict control of per-container cpu usage. Contributed by Varun Vasudev. 
(cherry picked from commit 9f6891d9ef7064d121305ca783eb62586c8aa018)
 2014-09-16 10:15:37 -07:00
Jian He
1c3f837835 YARN-2529. Generic history service RPC interface doesn't work when service authorization is enabled. Contributed by Zhijie Shen 
(cherry picked from commit 88e329fdfb27c11f1964c0345188bcf328c95fd6)
 2014-09-15 13:15:07 -07:00
cnauroth
aaef460e7a YARN-2549. TestContainerLaunch fails due to classpath problem with hamcrest classes. Contributed by Chris Nauroth. 
(cherry picked from commit 9d4ec97c95b51c6f99112030da91950551b22b9c)
 2014-09-15 11:54:41 -07:00
Vinod Kumar Vavilapalli
5cdb24d4b1 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 
(cherry picked from commit 14e2639fd0d53f7e0b58f2f4744af44983d4e867)
 2014-09-13 18:05:02 -07:00