Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,652 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1683 Commits

Author SHA1 Message Date
Xuan 9fae455e26 YARN-3393. Getting application(s) goes wrong when app finishes before 
starting the attempt. Contributed by Zhijie Shen
 2015-03-23 20:33:16 -07:00
Karthik Kambatla 972f1f1ab9 YARN-2868. FairScheduler: Metric for latency to allocate first container for an application. (Ray Chiang via kasha) 2015-03-23 14:07:05 -07:00
Karthik Kambatla 2bc097cd14 YARN-3241. FairScheduler handles invalid queue names inconsistently. (Zhihai Xu via kasha) 2015-03-23 13:22:03 -07:00
cnauroth 6ca1f12024 YARN-3336. FileSystem memory leak in DelegationTokenRenewer. 2015-03-23 10:45:50 -07:00
Tsuyoshi Ozawa 82eda771e0 YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. Contributed by Naganarasimha G R. 2015-03-24 00:25:30 +09:00
Jian He e1feb4ea1a YARN-3345. Add non-exclusive node label API. Contributed by Wangda Tan 2015-03-20 19:04:38 -07:00
Jian He 586348e4cb YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 2015-03-20 13:54:01 -07:00
Junping Du d81109e588 YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path. Contributed by Xuan Gong 2015-03-20 13:41:22 -07:00
Wangda Tan 6bc7710ec7 YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 2015-03-20 10:42:05 -07:00
Jian He 4e886eb9cb YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong 2015-03-19 22:27:21 -07:00
Devaraj K 93d0f4acc8 YARN-3357. Move TestFifoScheduler to FIFO package. Contributed by Rohith 
Sharmaks.
 2015-03-19 12:16:52 +05:30
Karthik Kambatla 20b49224eb YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) 2015-03-18 16:30:33 -07:00
Jian He 658097d6da YARN-3273. Improve scheduler UI to facilitate scheduling analysis and debugging. Contributed Rohith Sharmaks 2015-03-17 21:30:23 -07:00
Tsuyoshi Ozawa 3bc72cc16d YARN-3205. FileSystemRMStateStore should disable FileSystem Cache to avoid get a Filesystem with an old configuration. Contributed by Zhihai Xu. 2015-03-18 11:53:19 +09:00
Jian He 968425e9f7 YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks 2015-03-17 13:49:59 -07:00
Karthik Kambatla 32b4330456 Revert "YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)" 
This reverts commit c2b185def8.
 2015-03-17 12:31:15 -07:00
Jian He 487374b7fe YARN-3243. CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits. Contributed by Wangda Tan. 2015-03-17 10:24:23 -07:00
Devaraj K 7179f94f9d YARN-3197. Confusing log generated by CapacityScheduler. Contributed by 
Varun Saxena.
 2015-03-17 15:57:57 +05:30
Ravi Prakash 56085203c4 YARN-3339. TestDockerContainerExecutor should pull a single image and not the entire centos repository. (Ravindra Kumar Naik via raviprak) 2015-03-16 16:17:58 -07:00
Tsuyoshi Ozawa 7522a643fa YARN-3349. Treat all exceptions as failure in TestFSRMStateStore#testFSRMStateStoreClientRetry. Contributed by Zhihai Xu. 2015-03-17 08:09:55 +09:00
Tsuyoshi Ozawa 3da9a97cfb YARN-1453. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA, Andrew Purtell, and Allen Wittenauer. 2015-03-16 23:19:05 +09:00
Xuan 3ff1ba2a7b YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't 
work in ATS / RM web ui. Contributed by Naganarasimha G R
 2015-03-15 20:26:10 -07:00
Jonathan Eagles 8180e676ab YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 2015-03-13 12:04:30 -05:00
Vinod Kumar Vavilapalli 863079bb87 YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. Contributed by Xuan Gong. 2015-03-12 13:32:29 -07:00
Zhijie Shen 85f6d67fa7 YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM web page. Contributed by Xuan Gong. 2015-03-11 19:35:19 -07:00
Allen Wittenauer a5cf985bf5 YARN-2280. Resource manager web service fields are not accessible (Krisztian Horvath via aw) 2015-03-10 16:32:20 -07:00
Jian He c3003eba6f YARN-3300. Outstanding_resource_requests table should not be shown in AHS. Contributed by Xuan Gong 2015-03-09 20:46:48 -07:00
Zhijie Shen d6e05c5ee2 YARN-3287. Made TimelineClient put methods do as the correct login context. Contributed by Daryn Sharp and Jonathan Eagles. 2015-03-09 13:54:36 -07:00
Jason Lowe 27e8ea820f YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne 2015-03-06 22:37:26 +00:00
Jian He 21101c01f2 YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. Contributed by Chuan Liu 2015-03-06 14:18:11 -08:00
Jian He 95bfd087dc YARN-1809. Synchronize RM and TimeLineServer Web-UIs. Contributed by Zhijie Shen and Xuan Gong 2015-03-05 21:20:09 -08:00
Tsuyoshi Ozawa 1b67209612 YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. Contributed by Ryu Kobayashi. 2015-03-05 19:55:56 +09:00
Karthik Kambatla 8d88691d16 YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client. (Zhihai Xu via kasha) 2015-03-04 19:49:05 -08:00
Karthik Kambatla 22426a1c9f YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending jobs. (Siqi Li via kasha) 2015-03-04 18:06:58 -08:00
Karthik Kambatla 53947f37c7 YARN-3122. Metrics for container's actual CPU usage. (Anubhav Dhoot via kasha) 2015-03-04 17:33:30 -08:00
Jian He b2f1ec312e YARN-3222. Fixed NPE on RMNodeImpl#ReconnectNodeTransition when a node is reconnected with a different port. Contributed by Rohith Sharmaks 2015-03-03 16:28:28 -08:00
Wangda Tan e17e5ba9d7 YARN-3272. Surface container locality info in RM web UI (Jian He via wangda) 2015-03-03 11:49:01 -08:00
Tsuyoshi Ozawa d1c6accb6f HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-03-03 14:17:52 +09:00
Vinod Kumar Vavilapalli 14dd647c55 YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. Contributed by Wangda Tan. 2015-03-02 17:52:47 -08:00
Jian He 5d0bae550f YARN-3281. Added RMStateStore to StateMachine visualization list. Contributed by Chengbing Liu 2015-03-02 14:39:49 -08:00
Wangda Tan edcecedc1c YARN-3262. Surface application outstanding resource requests table in RM web UI. (Jian He via wangda) 2015-02-27 16:13:32 -08:00
Tsuyoshi Ozawa 01a1621930 YARN-2820. Retry in FileSystemRMStateStore when FS's operations fail due to IOException. Contributed by Zhihai Xu. 2015-02-28 00:56:44 +09:00
Konstantin V Shvachko 8ca0d957c4 YARN-3255. RM, NM, JobHistoryServer, and WebAppProxyServer's main() should support generic options. Contributed by Konstantin Shvachko. 2015-02-26 17:12:19 -08:00
Tsuyoshi Ozawa 773b6515ac YARN-3217. Remove httpclient dependency from hadoop-yarn-server-web-proxy. Contributed by Brahma Reddy Battula. 2015-02-27 00:24:29 +09:00
Devaraj K 0d4296f0e0 YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against 
all Schedulers even when using ParameterizedSchedulerTestBase. Contributed
by Anubhav Dhoot.
 2015-02-26 15:45:41 +05:30
Jason Lowe 1a68fc4346 YARN-3239. WebAppProxy does not support a final tracking url which has query fragments and params. Contributed by Jian He 2015-02-25 16:14:34 +00:00
Tsuyoshi Ozawa 6cbd9f1113 YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. Contributed by Zhihai Xu. 2015-02-25 10:38:11 +09:00
Allen Wittenauer d4ac6822e1 YARN-2980. Move health check script related functionality to hadoop-common (Varun Saxena via aw) 2015-02-24 11:25:26 -08:00
Tsuyoshi Ozawa 9cedad11d8 Revert "HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)" 
This reverts commit 946456c6d8.

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/QuotaByStorageTypeEntry.java
 2015-02-25 00:32:04 +09:00
Xuan fe7a302473 YARN-2797. TestWorkPreservingRMRestart should use 
ParametrizedSchedulerTestBase. Contributed by Karthik Kambatla
 2015-02-21 19:17:29 -08:00
Xuan e3d290244c YARN-3236. Cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY. 
Contributed by zhihai xu
 2015-02-21 16:18:40 -08:00
Xuan f56c65bb3e YARN-3237. AppLogAggregatorImpl fails to log error cause. Contributed by 
Rushabh S Shah
 2015-02-20 14:02:40 -08:00
Wangda Tan ce5bf927c3 YARN-3230. Clarify application states on the web UI. (Jian He via wangda) 2015-02-20 10:39:28 -08:00
Junping Du c33ae271c2 YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. Contributed by Zhihai Xu 2015-02-20 09:43:39 -08:00
Jason Lowe a64dd3d24b YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node. Contributed by Rohith 2015-02-20 15:10:10 +00:00
Jian He c0d9b93953 YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in RMApp. Contributed by Rohith Sharmaks 2015-02-19 15:42:39 -08:00
Wangda Tan d49ae725d5 YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda) 2015-02-19 11:00:57 -08:00
Tsuyoshi Ozawa 946456c6d8 HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) 2015-02-19 13:06:53 +09:00
Tsuyoshi Ozawa b8a14efdf5 YARN-1615. Fix typos in delay scheduler's description. Contributed by Akira Ajisaka. 2015-02-19 10:24:07 +09:00
Jian He 1c03376300 YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA 2015-02-18 16:06:55 -08:00
Jian He f5da5566d9 YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. Contributed by Wangda Tan 2015-02-18 11:51:51 -08:00
Xuan 57db50cbe3 YARN-3207. Secondary filter matches entites which do not have the key 
being filtered for. Contributed by Zhijie Shen
 2015-02-17 18:17:29 -08:00
Tsuyoshi Ozawa 556386a070 YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and Devaraj K. 2015-02-17 01:53:31 +09:00
Tsuyoshi Ozawa 3f32357c36 Revert "YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and devaraj." 
This reverts commit 9aae81c934.
 2015-02-17 01:52:22 +09:00
Tsuyoshi Ozawa 9aae81c934 YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and devaraj. 2015-02-17 01:48:22 +09:00
Tsuyoshi Ozawa 447bd7b5a6 YARN-3203. Correct a log message in AuxServices. Contributed by Brahma Reddy Battula. 2015-02-16 23:55:58 +09:00
Junping Du ab0b958a52 YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. (Contributed by Xuan Gong) 2015-02-15 06:46:32 -08:00
cnauroth 6804d68901 YARN-2899. Run TestDockerContainerExecutorWithMocks on Linux only. Contributed by Ming Ma. 2015-02-13 21:58:54 -08:00
Jason Lowe 1a0f508b63 YARN-2847. Linux native container executor segfaults if default banned user detected. Contributed by Olaf Flebbe 2015-02-13 20:20:55 +00:00
Tsuyoshi Ozawa 110cf6b9a3 YARN-3182. Cleanup switch statement in ApplicationMasterLauncher#handle(). Contributed by Ray Chiang. 2015-02-13 14:21:21 +09:00
Akira Ajisaka 6a49e58cb8 YARN-3191. Log object should be initialized with its own class. Contributed by Rohith. 2015-02-12 17:58:54 -08:00
Xuan 99f6bd4f7a YARN-3158. Correct log messages in ResourceTrackerService. Contributed 
by Varun Saxena
 2015-02-12 16:42:15 -08:00
Jian He 18297e0972 YARN-3104. Fixed RM to not generate new AMRM tokens on every heartbeat between rolling and activation. Contributed by Jason Lowe 2015-02-12 16:02:45 -08:00
Jian He 18a594257e YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan 2015-02-12 14:58:09 -08:00
Karthik Kambatla c2b185def8 YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha) 2015-02-12 13:44:47 -08:00
Junping Du 04f5ef18f7 YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager restart. (Contributed by Jason Lowe) 2015-02-12 11:46:47 -08:00
Xuan 83be450acc YARN-3147. Clean up RM web proxy code. Contributed by Steve Loughran 2015-02-12 10:57:55 -08:00
Xuan 65c69e296e YARN-3151. On Failover tracking url wrong in application cli for KILLED 
application. Contributed by Rohith
 2015-02-11 21:19:48 -08:00
Jason Lowe b379972ab3 YARN-3074. Nodemanager dies when localizer runner tries to write to a full disk. Contributed by Varun Saxena 2015-02-11 16:33:43 +00:00
Junping Du c541a374d8 YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Contributed by Chengbing Liu) 2015-02-11 03:44:31 -08:00
Zhijie Shen d5855c0e46 YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. Contributed by Devaraj K. 2015-02-10 15:24:01 -08:00
Jason Lowe 3f5431a22f YARN-2809. Implement workaround for linux kernel panic when removing cgroup. Contributed by Nathan Roberts 2015-02-10 17:27:21 +00:00
Jason Lowe 4eb5f7fa32 YARN-3090. DeletionService can silently ignore deletion task failures. Contributed by Varun Saxena 2015-02-10 16:54:21 +00:00
Zhijie Shen 23bf6c7207 YARN-3100. Made YARN authorization pluggable. Contributed by Jian He. 2015-02-09 20:34:56 -08:00
Jian He 0af6a99a3f YARN-3094. Reset timer for liveness monitors after RM recovery. Contributed by Jun Gong 2015-02-09 13:47:08 -08:00
Karthik Kambatla 7e42088abf YARN-2990. FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests. (kasha) 2015-02-08 22:48:10 -08:00
Jason Lowe da2fb2bc46 YARN-3143. RM Apps REST API can return NPE or entries missing id and other fields. Contributed by Jason Lowe 2015-02-06 21:47:32 +00:00
Jason Lowe 4c484320b4 YARN-3089. LinuxContainerExecutor does not handle file arguments to deleteAsUser. Contributed by Eric Payne 2015-02-06 20:39:01 +00:00
Jian He c1957fef29 YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan 2015-02-06 11:34:20 -08:00
Arun C. Murthy 02f154a001 YARN-1537. Fix race condition in TestLocalResourcesTrackerImpl.testLocalResourceCache. Contributed by Xuan Gong. 2015-02-05 23:58:52 -08:00
Arun C. Murthy 18b2507eda YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen. 2015-02-05 23:48:55 -08:00
Jian He 4641196fe0 YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA 2015-02-05 16:12:53 -08:00
Jason Lowe 69c8a7f45b YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves 2015-02-05 19:28:49 +00:00
Sandy Ryza b6466deac6 YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza) 2015-02-05 09:39:28 -08:00
Tsuyoshi Ozawa 26dee1486b YARN-3058. Fix error message of tokens' activation delay configuration. Contributed by Yi Liu. 2015-02-04 11:41:14 +09:00
Zhijie Shen e5b56e2af6 YARN-2543. Made resource usage be published to the timeline server too. Contributed by Naganarasimha G R. 2015-02-03 17:34:22 -08:00
Xuan b73e776abc YARN-3056. Add verification for containerLaunchDuration in 
TestNodeManagerMetrics. Contributed by zhihai Xu
 2015-02-03 15:12:54 -08:00
Wangda Tan 5bd984691b YARN-3075. NodeLabelsManager implementation to retrieve label to node mapping (Varun Saxena via wangda) 2015-02-03 12:52:42 -08:00
Jian He 21d80b3dd9 YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan 2015-02-03 11:43:12 -08:00
Robert Kanter f7a77819a1 YARN-3022. Expose Container resource information from NodeManager for monitoring (adhoot via ranter) 2015-02-03 10:39:41 -08:00
Jason Lowe a761bf8726 YARN-3085. Application summary should include the application type. Contributed by Rohith 2015-02-03 14:56:34 +00:00
Jian He 054a947989 YARN-3077. Fixed RM to create zk root path recursively. Contributed by Chun Chen 2015-01-30 17:34:49 -08:00
Jian He 86358221fc YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 2015-01-30 15:15:20 -08:00
Jonathan Eagles 30a8778c63 YARN-3108. ApplicationHistoryServer doesn't process -D arguments (Chang Li via jeagles) 2015-01-29 16:51:38 -06:00
Akira Ajisaka 342efa110a HADOOP-9907. Webapp http://hostname:port/metrics link is not working. (aajisaka) 2015-01-30 02:49:10 +09:00
Allen Wittenauer 9dd0b7a2ab YARN-2428. LCE default banned user list should have yarn (Varun Saxena via aw) 2015-01-29 09:30:26 -08:00
Wangda Tan 7882bc0f14 YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) 2015-01-28 21:54:38 -08:00
Wangda Tan 18741adf97 YARN-2932. Add entry for preemptable status (enabled/disabled) to scheduler web UI and queue initialize/refresh logging. (Eric Payne via wangda) 2015-01-27 15:36:09 -08:00
Jonathan Eagles a8ad1e8089 YARN-2897. CrossOriginFilter needs more log statements (Mit Desai via jeagles) 2015-01-27 16:06:39 -06:00
Jian He 4e15fc0841 YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService might lead NM to crash. Contributed by Varun Saxena 2015-01-27 13:31:22 -08:00
Tsuyoshi Ozawa f56da3ce04 YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. 2015-01-28 00:15:34 +09:00
Jian He 6f9fe76918 YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan 2015-01-26 15:38:00 -08:00
Jason Lowe 902c6ea7e4 YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native executor returns an error. Contributed by Eric Payne 2015-01-26 15:40:21 +00:00
Xuan 0d6bd62102 YARN-3024. LocalizerRunner should give DIE action when all resources are 
localized. Contributed by Chengbing Liu
 2015-01-25 19:37:57 -08:00
Tsuyoshi Ozawa 24aa462673 YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. Contributed by Wangda Tan. 2015-01-23 20:37:05 +09:00
Tsuyoshi Ozawa 3aab354e66 YARN-3082. Non thread safe access to systemCredentials in NodeHeartbeatResponse processing. Contributed by Anubhav Dhoot. 2015-01-23 16:04:18 +09:00
Wangda Tan 0a2d3e717d YARN-2933. Capacity Scheduler preemption policy should only consider capacity without labels temporarily. Contributed by Mayank Bansal 2015-01-19 16:48:50 -08:00
Zhijie Shen 43302f6f44 YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-server-common. Contributed by Ferdinand Xu. 2015-01-16 16:37:42 -08:00
Karthik Kambatla 84198564ba YARN-2984. Metrics for container's actual memory usage. (kasha) 2015-01-17 05:44:04 +05:30
Junping Du 5d1cca34fa YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout. (Contributed by Jian He) 2015-01-16 00:08:36 -08:00
Jian He 9e33116d1d YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. Contributed by Zhijie Shen 2015-01-15 11:07:43 -08:00
Jian He c53420f583 YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch 2015-01-13 17:32:07 -08:00
Robert Kanter 51881535e6 YARN-2643. Don't create a new DominantResourceCalculator on every FairScheduler.allocate call. (kasha via rkanter) 2015-01-12 17:51:00 -08:00
Robert Kanter ae7bf31fe1 YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter) 2015-01-12 10:47:52 -08:00
Jian He a260406268 YARN-3014. Replaces labels on a host should update all NM's labels on that host. Contributed by Wangda Tan 2015-01-09 17:49:53 -08:00
Jian He cc2a745f7e YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed container statuses on heartbeat. Contributed by Chengbing Liu 2015-01-08 11:12:54 -08:00
Zhijie Shen dc2eaa26b2 YARN-2996. Improved synchronization and I/O operations of FS- and Mem- RMStateStore. Contributed by Yi Liu. 2015-01-08 09:47:02 -08:00
Jian He e13a484a2b YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. Contributed by Yi Liu 2015-01-07 14:12:20 -08:00
Zhijie Shen 60103fca04 YARN-2427. Added the API of moving apps between queues in RM web services. Contributed by Varun Vasudev. 2015-01-06 14:37:44 -08:00
Jian He dd57c2047b YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena 2015-01-06 14:15:57 -08:00
Karthik Kambatla 0c4b112677 YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler. (Anubhav Dhoot via kasha) 2015-01-06 04:42:10 +05:30
Zhijie Shen 562a701945 YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. Contributed by Varun Saxena. 2015-01-05 13:33:07 -08:00
Tsuyoshi Ozawa ddc5be48fc YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed by Rohith Sharmaks. 2015-01-05 00:08:31 +09:00
Karthik Kambatla e7257acd8a YARN-2998. Abstract out scheduler independent PlanFollower components. (Anubhav Dhoot via kasha) 2014-12-30 19:55:24 -08:00
Jian He e2351c7ae2 YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and app ACLs. Contributed by Varun Saxena 2014-12-30 17:15:37 -08:00
Jian He b7442bf92e YARN-2493. Added node-labels page on RM web UI. Contributed by Wangda Tan 2014-12-30 16:49:01 -08:00
Jian He 746ad6e989 Revert "YARN-2492(wrong jira number). Added node-labels page on RM web UI. Contributed by Wangda Tan" 
This reverts commit 5f57b904f5.
 2014-12-30 16:48:49 -08:00
Jian He 5f57b904f5 YARN-2492. Added node-labels page on RM web UI. Contributed by Wangda Tan 2014-12-30 15:38:28 -08:00
Zhijie Shen 241d3b3a50 YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. Contributed by Varun Saxena. 2014-12-29 09:59:35 -08:00
Jian He 1454efe5d4 YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla 2014-12-26 17:59:58 -08:00
Junping Du 40ee4bff65 YARN-2993. Several fixes (missing acl check, error log msg ...) and some refinement in AdminService. (Contributed by Yi Liu) 2014-12-26 06:59:25 -08:00
Jian He 4f18018b7a YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith Sharmaks 2014-12-23 22:14:29 -08:00
Zhijie Shen 41a548a916 YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. 2014-12-23 20:32:36 -08:00
Junping Du d468c9aaf1 YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) 2014-12-23 20:04:33 -08:00
Junping Du 856b46cf47 Revert ca3381de61 and 20674255ad as missing JIRA number in commit message 2014-12-23 20:02:33 -08:00
Junping Du ca3381de61 Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) 2014-12-23 18:57:16 -08:00
Jian He 149512a837 YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen 2014-12-23 18:25:37 -08:00
Jian He 0d89859b51 YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharmaks 2014-12-22 21:53:22 -08:00
Jian He fdf042dfff YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 2014-12-22 16:51:15 -08:00
Junping Du a696fbb001 YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) 2014-12-22 03:06:37 -08:00
Karthik Kambatla 24ee9e3431 YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) 2014-12-20 12:17:50 -08:00
Jian He 808cba3821 YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks 2014-12-19 16:56:30 -08:00
Karthik Kambatla 954fb8581e YARN-2675. containersKilled metrics is not updated when the container is killed during localization. (Zhihai Xu via kasha) 2014-12-19 16:02:20 -08:00
Karthik Kambatla a22ffc3188 YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler. (Anubhav Dhoot via kasha) 2014-12-19 15:37:12 -08:00
Jason Lowe 0402bada19 YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by Jian He 2014-12-18 23:28:18 +00:00
Junping Du 2b4b0e8847 YARN-2972. DelegationTokenRenewer thread pool never expands. Contributed by Jason Lowe 2014-12-17 17:40:00 -08:00
Karthik Kambatla b7f64823e1 YARN-2203. [YARN-1492] Web UI for cache manager. (Chris Trezzo via kasha) 2014-12-17 16:32:21 -08:00
Karthik Kambatla a1bd140964 YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. (Chris Trezzo via kasha) 2014-12-17 16:23:52 -08:00
Karthik Kambatla f2d150ea12 YARN-2964. FSLeafQueue#assignContainer - document the reason for using both write and read locks. (Tsuyoshi Ozawa via kasha) 2014-12-17 12:03:16 -08:00
cnauroth e996a1bfd4 HADOOP-11321. copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions. Contributed by Chris Nauroth. 2014-12-16 15:29:22 -08:00
Karthik Kambatla e597249d36 YARN-2914. [YARN-1492] Potential race condition in Singleton implementation of SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics. (Varun Saxena via kasha) 2014-12-15 12:00:12 -08:00
Jian He a4f2995b9e Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-12-15 11:04:09 -08:00
Steve Loughran 3681de2039 YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) 2014-12-12 17:11:07 +00:00
Devaraj K bda748ac3a YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in 
SchedulerApplicationAttempt ctor. Contributed by Devaraj K.
 2014-12-12 12:34:43 +05:30
Jian He 2ed90a57fd YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. Contributed by Wangda Tan 2014-12-09 16:48:04 -08:00
Karthik Kambatla a2e07a5456 YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg via kasha) 2014-12-09 14:00:31 -08:00
Karthik Kambatla db73cc9124 YARN-2931. PublicLocalizer may fail until directory is initialized by LocalizeRunner. (Anubhav Dhoot via kasha) 2014-12-08 22:26:18 -08:00
Jian He e69af836f3 YARN-2869. CapacityScheduler should trim sub queue names when parse configuration. Contributed by Wangda Tan 2014-12-05 17:33:39 -08:00
Jason Lowe 4b13082199 YARN-2056. Disable preemption at Queue level. Contributed by Eric Payne 2014-12-05 21:06:48 +00:00
Karthik Kambatla 78968155d7 YARN-2189. [YARN-1492] Admin service for cache manager. (Chris Trezzo via kasha) 2014-12-04 17:36:32 -08:00
Jian He 258623ff8b YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R 2014-12-04 12:53:18 -08:00
Jian He 73fbb3c66b YARN-2880. Added a test to make sure node labels will be recovered if RM restart is enabled. Contributed by Rohith Sharmaks 2014-12-03 17:14:52 -08:00
Harsh J a31e016491 YARN-2891. Failed Container Executor does not provide a clear error message. Contributed by Dustin Cote. (harsh) 2014-12-04 03:17:15 +05:30
Karthik Kambatla 799353e2c7 YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps. (Naganarasimha G R via kasha) 2014-12-03 13:44:41 -08:00
Jason Lowe 03ab24aa01 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 2014-12-03 17:02:14 +00:00
Junping Du e65b7c5ff6 YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values. (Contributed by Tsuyoshi OZAWA) 2014-12-03 04:11:18 -08:00
Jian He 392c3aaea8 YARN-2894. Fixed a bug regarding application view acl when RM fails over. Contributed by Rohith Sharmaks 2014-12-02 17:16:35 -08:00
Jian He 52bcefca8b YARN-2136. Changed RMStateStore to ignore store opearations when fenced. Contributed by Varun Saxena 2014-12-02 10:54:48 -08:00
Zhijie Shen 51af8d367d YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 2014-12-01 19:04:04 -08:00
Jian He a7fba0bc28 YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe 2014-12-01 16:38:25 -08:00
Junping Du c732ed760e YARN-2907. SchedulerNode#toString should print all resource detail instead of only memory. (Contributed by Rohith) 2014-12-01 05:38:22 -08:00
Zhijie Shen 8f1454cc6d YARN-2165. Added the sanity check for the numeric configuration values of the timeline service. Contributed by Vasanth kumar RJ. 2014-11-26 12:09:34 -08:00
Karthik Kambatla fe1f2db5ee YARN-2188. [YARN-1492] Client service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 2014-11-25 16:21:46 -08:00
Jian He 8a7ca13b13 YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. Contributed by Jason Lowe 2014-11-25 16:16:56 -08:00
Jian He 5805a81efb YARN-2404. Removed ApplicationAttemptState and ApplicationState class in RMStateStore. Contributed by Tsuyoshi OZAWA 2014-11-25 12:48:22 -08:00
Zhijie Shen e37a4ff0c1 YARN-2697. Remove useless RMAuthenticationHandler. Contributed by Haosong Huang. 2014-11-24 14:58:59 -08:00
Jason Lowe 1ce4d33c2d YARN-1984. LeveldbTimelineStore does not handle db exceptions properly. Contributed by Varun Saxena 2014-11-24 22:36:59 +00:00
Sandy Ryza a128cca305 YARN-2669. FairScheduler: queue names shouldn't allow periods (Wei Yan via Sandy Ryza) 2014-11-21 16:06:41 -08:00
Karthik Kambatla 233b61e495 YARN-2679. Add metric for container launch duration. (Zhihai Xu via kasha) 2014-11-21 14:22:21 -08:00
Karthik Kambatla 3114d4731d YARN-2604. Scheduler should consider max-allocation-* in conjunction with the largest node. (Robert Kanter via kasha) 2014-11-21 10:32:28 -08:00
Karthik Kambatla a9a0cc3679 YARN-2315. FairScheduler: Set current capacity in addition to capacity. (Zhihai Xu via kasha) 2014-11-19 20:15:40 -08:00
Karthik Kambatla c90fb84aaa YARN-2802. ClusterMetrics to include AM launch and register delays. (Zhihai Xu via kasha) 2014-11-19 19:50:12 -08:00
Jian He 9cb8b75ba5 YARN-2865. Fixed RM to always create a new RMContext when transtions from StandBy to Active. Contributed by Rohith Sharmaks 2014-11-19 19:48:52 -08:00
Karthik Kambatla 2fce6d6141 YARN-2690. [YARN-2574] Make ReservationSystem and its dependent classes independent of Scheduler type. (Anubhav Dhoot via kasha) 2014-11-17 16:45:57 -08:00
Jason Lowe 81c9d17af8 YARN-2414. RM web UI: app page will crash if app is failed before any attempt has been created. Contributed by Wangda Tan 2014-11-17 21:15:48 +00:00
Vinod Kumar Vavilapalli 198fb58776 YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. Contributed by Rohith Sharma. 2014-11-15 12:05:25 -08:00
Jian He 9b860661a5 YARN-2432. RMStateStore should process the pending events before close. Contributed by Varun Saxena 2014-11-14 17:24:09 -08:00
Sandy Ryza 1a47f890ba YARN-2811. In Fair Scheduler, reservation fulfillments shouldn't ignore max share (Siqi Li via Sandy Ryza) 2014-11-14 15:18:56 -08:00
Jason Lowe 49c38898b0 YARN-2816. NM fail to start with NPE during container recovery. Contributed by Zhihai Xu 2014-11-14 21:25:59 +00:00
Jian He d005404ef7 YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state on app recovery. Contributed by Rohith Sharmaks 2014-11-13 15:34:26 -08:00
Zhijie Shen 3648cb57c9 YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, attempts and containers. Contributed by Robert Kanter. 2014-11-13 13:47:42 -08:00
Vinod Kumar Vavilapalli 3651fe1b08 YARN-2853. Fixed a bug in ResourceManager causing apps to hang when the user kill request races with ApplicationMaster finish. Contributed by Jian He. 2014-11-13 08:13:03 -08:00
Jason Lowe 33ea5ae92b YARN-2846. Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart. Contributed by Junping Du 2014-11-13 16:11:04 +00:00
Zhijie Shen be7bf956e9 YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. 2014-11-12 11:07:57 -08:00
Karthik Kambatla a04143039e YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) 2014-11-12 09:31:05 -08:00
Jason Lowe f8aefa5e9c YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne 2014-11-12 17:01:15 +00:00
Ravi Prakash 53f64ee516 YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN 2014-11-11 21:28:11 -08:00
Karthik Kambatla 061bc293c8 YARN-2735. diskUtilizationPercentageCutoff and diskUtilizationSpaceCutoff are initialized twice in DirectoryCollection. (Zhihai Xu via kasha) 2014-11-11 10:31:39 -08:00
Xuan 5c9a51f140 YARN-2841. RMProxy should retry EOFException. Contributed by Jian He 2014-11-10 18:25:01 -08:00
Vinod Kumar Vavilapalli e76faebc95 YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery consistent with the (somewhat incorrect) behaviour in the non-recovery case. Contributed by Jian He. 2014-11-09 18:56:58 -08:00
Arun C. Murthy 43cd07b408 YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 2014-11-09 14:57:37 -08:00
Zhijie Shen 9a4e0d343e YARN-2505. Supported get/add/remove/change labels in RM REST API. Contributed by Craig Welch. 2014-11-07 20:35:46 -08:00
Vinod Kumar Vavilapalli df36edf751 YARN-2826. Fixed user-groups mappings' refresh bug caused by YARN-2826. Contributed by Wangda Tan. 2014-11-07 19:43:55 -08:00
Xuan 4a114dd67a YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen 2014-11-07 16:11:30 -08:00
Jason Lowe c3d475070a YARN-2825. Container leak on NM. Contributed by Jian He 2014-11-07 23:16:37 +00:00
Vinod Kumar Vavilapalli 4cfd5bc7c1 YARN-2753. Fixed a bunch of bugs in the NodeLabelsManager classes. Contributed by Zhihai xu. 2014-11-07 14:15:53 -08:00
cnauroth 06b797947c YARN-2803. MR distributed cache not working correctly on Windows after NodeManager privileged account changes. Contributed by Craig Welch. 2014-11-07 12:29:39 -08:00
Vinod Kumar Vavilapalli 2ac1be7dec YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. 2014-11-07 10:39:37 -08:00
Xuan 1e97f2f094 YARN-2810. TestRMProxyUsersConf fails on Windows VMs. Contributed by Varun Vasudev 2014-11-07 09:44:43 -08:00
Vinod Kumar Vavilapalli a5657182a7 YARN-2823. Fixed ResourceManager app-attempt state machine to inform schedulers about previous finished attempts of a running appliation to avoid expectation mismatch w.r.t transferred containers. Contributed by Jian He. 2014-11-07 09:28:36 -08:00
Vinod Kumar Vavilapalli a3839a9fbf YARN-2744. Fixed CapacityScheduler to validate node-labels correctly against queues. Contributed by Wangda Tan. 2014-11-06 17:28:12 -08:00
Vinod Kumar Vavilapalli f5b19bed7d YARN-2818. Removed the now unnecessary user entity injection from Timeline service given we now have domains. Contributed by Zhijie Shen. 2014-11-06 11:48:40 -08:00
Xuan b0b52c4e11 YARN-2812. TestApplicationHistoryServer is likely to fail on less powerful machine. Contributed by Zhijie Shen 2014-11-05 20:42:38 -08:00
Xuan e4b4901d36 YARN-2813. Fixed NPE from MemoryTimelineStore.getDomains. Contributed by Zhijie Shen 2014-11-05 18:24:41 -08:00
Jian He 395275af86 YARN-2579. Fixed a deadlock issue when EmbeddedElectorService and FatalEventDispatcher try to transition RM to StandBy at the same time. Contributed by Rohith Sharmaks 2014-11-05 16:59:54 -08:00
Vinod Kumar Vavilapalli 834e931d8e YARN-2805. Fixed ResourceManager to load HA configs correctly before kerberos login. Contributed by Wangda Tan. 2014-11-05 15:30:33 -08:00
Zhijie Shen b4c951ab83 YARN-2767. Added a test case to verify that http static user cannot kill or submit apps in the secure mode. Contributed by Varun Vasudev. 2014-11-05 10:57:38 -08:00
Vinod Kumar Vavilapalli b76179895d YARN-2804. Fixed Timeline service to not fill the logs with JAXB bindings exceptions. Contributed by Zhijie Shen. 2014-11-04 18:00:06 -08:00
Haohui Mai d794f785de HADOOP-11230. Add missing dependency of bouncycastle for kms, httpfs, hdfs, MR and YARN. Contributed by Robert Kanter. 2014-11-04 17:52:03 -08:00
Karthik Kambatla b2cd269802 YARN-2010. Handle app-recovery failures gracefully. (Jian He and Karthik Kambatla via kasha) 2014-11-04 17:45:24 -08:00
Vinod Kumar Vavilapalli c5a46d4c8c YARN-1922. Fixed NodeManager to kill process-trees correctly in the presence of races between the launch and the stop-container call and when root processes crash. Contributed by Billie Rinaldi. 2014-11-03 16:38:55 -08:00
Vinod Kumar Vavilapalli ec6cbece8e YARN-2795. Fixed ResourceManager to not crash loading node-label data from HDFS in secure mode. Contributed by Wangda Tan. 2014-11-03 13:44:06 -08:00
Vinod Kumar Vavilapalli 58e9f24e0f YARN-2788. Fixed backwards compatiblity issues with log-aggregation feature that were caused when adding log-upload-time via YARN-2703. Contributed by Xuan Gong. 2014-11-03 13:16:29 -08:00
Jason Lowe 6157ace547 YARN-2730. DefaultContainerExecutor runs only one localizer at a time. Contributed by Siqi Li 2014-11-03 20:37:47 +00:00
Zhijie Shen 27715ec63b YARN-2785. Fixed intermittent TestContainerResourceUsage failure. Contributed by Varun Vasudev. 2014-11-02 15:20:40 -08:00
Vinod Kumar Vavilapalli 5c0381c96a YARN-2790. Fixed a NodeManager bug that was causing log-aggregation to fail beyond HFDS delegation-token expiry even when RM is a proxy-user (YARN-2704). Contributed by Jian He. 2014-11-01 16:32:35 -07:00
Zhijie Shen 1cd088fd9d YARN-2711. Fixed TestDefaultContainerExecutor#testContainerLaunchError failure on Windows. Contributed by Varun Vasudev. 2014-10-31 17:43:30 -07:00
Vinod Kumar Vavilapalli 6ce32f593b YARN-2789. Re-instated the NodeReport.newInstance private unstable API modified in YARN-2698 so that tests in YARN frameworks don't break. Contributed by Wangda Tan. 2014-10-31 15:32:21 -07:00
Xuan 86ff28dea0 YARN-2701. Addendum patch. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 2014-10-31 14:36:25 -07:00
Jian He 1b4be91866 YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. Contributed by Zhijie Shen 2014-10-31 13:16:45 -07:00
Karthik Kambatla 256697acd5 YARN-2186. [YARN-1492] Node Manager uploader service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 2014-10-31 13:00:42 -07:00
Vinod Kumar Vavilapalli d1828d9443 YARN-2779. Fixed ResourceManager to not require delegation tokens for communicating with Timeline Service. Contributed by Zhijie Shen. 2014-10-30 23:16:19 -07:00
Vinod Kumar Vavilapalli e0233c16eb YARN-2698. Moved some node label APIs to be correctly placed in client protocol. Contributed by Wangda Tan. 2014-10-30 22:59:31 -07:00
Jason Lowe 73e626ad91 YARN-2755. NM fails to clean up usercache_DEL_<timestamp> dirs after YARN-661. Contributed by Siqi Li 2014-10-30 15:10:27 +00:00
Karthik Kambatla 179cab81e0 YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks. (Tsuyoshi Ozawa via kasha) 2014-10-30 00:29:07 -07:00
Karthik Kambatla 782971ae7a YARN-2742. FairSchedulerConfiguration should allow extra spaces between value and unit. (Wei Yan via kasha) 2014-10-29 10:24:57 -07:00
Jian He d5e0a09721 YARN-2503. Added node lablels in web UI. Contributed by Wangda Tan 2014-10-28 17:57:54 -07:00
Zhijie Shen 8984e9b177 YARN-2741. Made NM web UI serve logs on the drive other than C: on Windows. Contributed by Craig Welch. 2014-10-28 14:11:19 -07:00
Xuan 69f79bee8b YARN-2758. Update TestApplicationHistoryClientService to use the new generic history store. Contributed by Zhijie Shen 2014-10-28 13:28:42 -07:00
Xuan 0d3e7e2bd6 RN-2279. Add UTs to cover timeline server authentication. Contributed by Zhijie Shen 2014-10-28 11:16:49 -07:00
Vinod Kumar Vavilapalli a16d022ca4 YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the sake of localization and log-aggregation for long-running services. Contributed by Jian He. 2014-10-27 15:49:47 -07:00
Jian He c05b581a55 YARN-2591. Fixed AHSWebServices to return FORBIDDEN(403) if the request user doesn't have access to the history data. Contributed by Zhijie Shen 2014-10-27 10:45:53 -07:00
Xuan ce1a4419a6 YARN-2726. CapacityScheduler should explicitly log when an accessible label has no capacity. Contributed by Wangda Tan 2014-10-27 10:26:21 -07:00
Vinod Kumar Vavilapalli 0186645505 YARN-2743. Fixed a bug in ResourceManager that was causing RMDelegationToken identifiers to be tampered and thus causing app submission failures in secure mode. Contributed by Jian He. 2014-10-26 11:14:34 -07:00
Karthik Kambatla c51e53d7aa YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 2014-10-25 10:31:06 -07:00
Jian He 5864dd99a4 YARN-1915. Fixed a race condition that client could use the ClientToAMToken to contact with AM before AM actually receives the ClientToAMTokenMasterKey. Contributed by Jason Lowe 2014-10-24 22:47:56 -07:00
Zhijie Shen f81dc3f995 YARN-2703. Added logUploadedTime into LogValue for better display. Contributed by Xuan Gong. 2014-10-24 14:10:46 -07:00
Zhijie Shen 0f3b6900be YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He. 2014-10-23 21:56:03 -07:00
Jian He 3b12fd6cfb YARN-2198. Remove the need to run NodeManager as privileged account for Windows Secure Container Executor. Contributed by Remus Rusanu 2014-10-22 15:57:46 -07:00
Zhijie Shen 7e3b5e6f5c YARN-2721. Suppress NodeExist exception thrown by ZKRMStateStore when it retries creating znode. Contributed by Jian He. 2014-10-21 21:49:17 -07:00
Vinod Kumar Vavilapalli c0e034336c YARN-2715. Fixed ResourceManager to respect common configurations for proxy users/groups beyond just the YARN level config. Contributed by Zhijie Shen. 2014-10-21 20:09:40 -07:00
cnauroth 6637e3cf95 YARN-2720. Windows: Wildcard classpath variables not expanded against resources contained in archives. Contributed by Craig Welch. 2014-10-21 12:33:21 -07:00
Jason Lowe 6f2028bd15 YARN-90. NodeManager should identify failed disks becoming good again. Contributed by Varun Vasudev 2014-10-21 17:31:13 +00:00
Jian He 4fa1fb3193 Missing file for YARN-2701 2014-10-20 19:57:45 -07:00
Jian He 2839365f23 YARN-2701. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 2014-10-20 18:45:47 -07:00
Vinod Kumar Vavilapalli 8256766498 YARN-2504. Enhanced RM Admin CLI to support management of node-labels. Contribyted by Wangda Tan. 2014-10-18 12:07:40 -07:00
Vinod Kumar Vavilapalli 39063cd36f YARN-2676. Enhanced Timeline auth-filter to support proxy users. Contributed by Zhijie Shen. 2014-10-17 22:02:50 -07:00
Vinod Kumar Vavilapalli e9c66e8fd2 YARN-2705. Fixed bugs in ResourceManager node-label manager that were causing test-failures: added a dummy in-memory labels-manager. Contributed by Wangda Tan. 2014-10-17 18:26:12 -07:00
Jian He c3de2412eb YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over. Contributed by Tsuyoshi OZAWA 2014-10-17 16:35:27 -07:00
Jian He a6aa6e42ca YARN-2588. Standby RM fails to transitionToActive if previous transitionToActive failed with ZK exception. Contributed by Rohith Sharmaks 2014-10-17 10:54:24 -07:00
Vinod Kumar Vavilapalli abae63caf9 YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail when using ephemeral ports on NodeIDs. Contributed by Wangda Tan. 2014-10-17 08:58:08 -07:00
Jian He 0fd0ebae64 YARN-2682. Updated WindowsSecureContainerExecutor to not use DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir() instead. Contributed by Zhihai Xu 2014-10-16 18:14:34 -07:00
Jian He 233d446be1 YARN-2621. Simplify the output when the user doesn't have the access for getDomain(s). Contributed by Zhijie Shen 2014-10-16 16:37:21 -07:00
Vinod Kumar Vavilapalli b3056c266a YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan. 2014-10-15 18:47:26 -07:00
Vinod Kumar Vavilapalli f2ea555ac6 YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan. 
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.
 2014-10-15 18:33:06 -07:00
Jian He 0af1a2b5bc YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:22:07 -07:00
Zhijie Shen 1220bb72d4 YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. 2014-10-14 21:50:46 -07:00
Zhijie Shen cdce88376a HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 2014-10-14 11:35:38 -07:00
Karthik Kambatla cc93e7e683 YARN-2566. DefaultContainerExecutor should pick a working directory randomly. (Zhihai Xu via kasha) 2014-10-13 16:32:01 -07:00
Karthik Kambatla da709a2eac YARN-2641. Decommission nodes on -refreshNodes instead of next NM-RM heartbeat. (Zhihai Xu via kasha) 2014-10-13 16:23:04 -07:00
Jian He f9680d9a16 YARN-2308. Changed CapacityScheduler to explicitly throw exception if the queue 
to which the apps were submitted is changed across RM restart. Contributed by Craig Welch & Chang Li
 2014-10-13 14:09:04 -07:00
Jason Lowe a56ea01002 YARN-2377. Localization exception stack traces are not passed as diagnostic info. Contributed by Gera Shegalov 2014-10-13 18:31:16 +00:00
Zhijie Shen 4aed2d8e91 YARN-2651. Spun off LogRollingInterval from LogAggregationContext. Contributed by Xuan Gong. 2014-10-13 10:54:09 -07:00
Vinod Kumar Vavilapalli db7f165319 YARN-2494. Added NodeLabels Manager internal API and implementation. Contributed by Wangda Tan. 2014-10-10 11:44:21 -07:00
Zhijie Shen cb81bac002 YARN-2583. Modified AggregatedLogDeletionService to be able to delete rolling aggregated logs. Contributed by Xuan Gong. 2014-10-10 00:11:30 -07:00
cnauroth d3afd730ac YARN-2662. TestCgroupsLCEResourcesHandler leaks file descriptors. Contributed by Chris Nauroth. 2014-10-09 22:44:44 -07:00
Karthik Kambatla 4f426fe223 YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) 2014-10-09 17:16:06 -07:00
Mayank Bansal df3becf080 YARN-2598 GHS should show N/A instead of null for the inaccessible information (Zhijie Shen via mayank) 2014-10-08 17:27:53 -07:00
Jian He e16e25ab1b YARN-2649. Fixed TestAMRMRPCNodeUpdates test failure. Contributed by Ming Ma 2014-10-08 10:58:51 -07:00
Jian He 30d56fdbb4 YARN-1857. CapacityScheduler headroom doesn't account for other AM's running. Contributed by Chen He and Craig Welch 2014-10-07 13:45:04 -07:00
Jian He 519e5a7dd2 YARN-2644. Fixed CapacityScheduler to return up-to-date headroom when AM allocates. Contributed by Craig Welch 2014-10-06 15:48:46 -07:00
Jian He ea26cc0b4a YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier to use protobuf as payload. Contributed by Junping Du 2014-10-06 10:47:43 -07:00
Chris Douglas c8212bacb1 YARN-1051. Add a system for creating reservations of cluster capacity. 
Contributed by Subru Krishnan and Carlo Curino.
 2014-10-03 17:09:02 -07:00
subru a2986234be YARN-2611. Fixing jenkins findbugs warning and TestRMWebServicesCapacitySched for branch YARN-1051. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit c47464aba4)
 2014-10-03 15:43:23 -07:00
subru 5e10a13bb4 YARN-2576. Making test patch pass in branch. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 90ac0be86b)
 2014-10-03 15:43:13 -07:00
subru 6261f7cc69 YARN-2080. Integrating reservation system with ResourceManager and client-RM protocol. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 8baeaead85)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
 2014-10-03 15:42:43 -07:00
subru 3418c56bcf YARN-1712. Plan follower that synchronizes the current state of reservation subsystem with the scheduler. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 169085319b)
 2014-10-03 15:42:10 -07:00
carlo curino b6df0dddcd YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit c4918cb4cb)
 2014-10-03 15:42:03 -07:00
carlo curino f66ffcf832 YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit aef7928899)
 2014-10-03 15:41:57 -07:00
subru cf4b34282a YARN-1709. In-memory data structures used to track resources over time to enable reservations. 
(cherry picked from commit 0d8b2cd88b)
 2014-10-03 15:41:51 -07:00
carlo curino 1c6950354f YARN-2475. Logic for responding to capacity drops for the ReservationSystem. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit f83a07f266)
 2014-10-03 15:41:21 -07:00
carlo curino eb3e40b833 YARN-1707. Introduce APIs to add/remove/resize queues in the CapacityScheduler. Contributed by Carlo Curino and Subru Krishnan 
(cherry picked from commit aac47fda7f)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
 2014-10-03 15:41:02 -07:00
Vinod Kumar Vavilapalli 34cdcaad71 YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for use by long running services. Contributed by Xuan Gong. 2014-10-03 12:15:40 -07:00
Karthik Kambatla 80d11eb68e YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 2014-10-03 11:49:49 -07:00
Jian He 054f285526 YARN-2628. Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free. Contributed by Varun Vasudev 2014-10-02 15:13:33 -07:00
Jason Lowe 29f520052e YARN-2624. Resource Localization fails on a cluster due to existing cache directories. Contributed by Anubhav Dhoot 2014-10-02 17:39:34 +00:00
Karthik Kambatla 5e0b49da9c YARN-2254. TestRMWebServicesAppsModification should run against both CS and FS. (Zhihai Xu via kasha) 2014-10-02 10:15:04 -07:00
Jian He 3ef1cf187f YARN-2617. Fixed NM to not send duplicate container status whose app is not running. Contributed by Jun Gong 2014-10-02 10:04:09 -07:00
junping_du c7cee9b455 YARN-1979. TestDirectoryCollection fails when the umask is unusual. (Contributed by Vinod Kumar Vavilapalli and Tsuyoshi OZAWA) 2014-10-02 08:02:59 -07:00
Vinod Kumar Vavilapalli 9e40de6af7 YARN-2446. Augmented Timeline service APIs to start taking in domains as a parameter while posting entities and events. Contributed by Zhijie Shen. 2014-10-01 17:52:03 -07:00
junping_du 0708827a93 YARN-2613. Support retry in NMClient for rolling-upgrades. (Contributed by Jian He) 2014-10-01 16:50:30 -07:00
Zhijie Shen 52bbe0f11b YARN-2630. Prevented previous AM container status from being acquired by the current restarted AM. Contributed by Jian He. 2014-10-01 15:38:11 -07:00
Vinod Kumar Vavilapalli ba7f31c2ee YARN-1972. Added a secure container-executor for Windows. Contributed by Remus Rusanu. 2014-10-01 10:14:41 -07:00
Karthik Kambatla 17d1202c35 YARN-2179. [YARN-1492] Initial cache manager structure and context. (Chris Trezzo via kasha) 2014-09-30 19:01:47 -07:00
Jian He bbff96be48 YARN-2602. Fixed possible NPE in ApplicationHistoryManagerOnTimelineStore. Contributed by Zhijie Shen 2014-09-30 16:44:17 -07:00
Karthik Kambatla 14d60dadc2 YARN-2594. Potential deadlock in RM when querying ApplicationResourceUsageReport. (Wangda Tan via kasha) 2014-09-30 16:12:27 -07:00
Zhijie Shen 9582a50176 YARN-2627. Added the info logs of attemptFailuresValidityInterval and number of previous failed attempts. Contributed by Xuan Gong. 2014-09-30 12:57:45 -07:00
Jonathan Eagles e10eeaabce YARN-2606. Application History Server tries to access hdfs before doing secure login (Mit Desai via jeagles) 2014-09-29 16:37:23 -05:00
Jason Lowe 9c22065109 YARN-1769. CapacityScheduler: Improve reservations. Contributed by Thomas Graves 2014-09-29 14:12:18 +00:00
Jian He 5391919b09 YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier to use protobuf object as the payload. Contributed by Junping Du. 2014-09-26 17:48:41 -07:00
Karthik Kambatla f4357240a6 YARN-2608. FairScheduler: Potential deadlocks in loading alloc files and clock access. (Wei Yan via kasha) 2014-09-25 17:42:51 -07:00
Jason Lowe 8269bfa613 YARN-2523. ResourceManager UI showing negative value for "Decommissioned Nodes" field. Contributed by Rohith 2014-09-25 22:37:05 +00:00
Zhijie Shen 72b0881ca6 YARN-2546. Made REST API for application creation/submission use numeric and boolean types instead of the string of them. Contributed by Varun Vasudev. 2014-09-24 17:57:32 -07:00
Zhijie Shen c86674a3a4 YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier. Contributed by Xuan Gong. 2014-09-24 17:50:26 -07:00
Karthik Kambatla 39c87344e1 YARN-2596. TestWorkPreservingRMRestart fails with FairScheduler. (kasha) 2014-09-24 15:23:31 -07:00
Vinod Kumar Vavilapalli d78b452a4f YARN-2102. Added the concept of a Timeline Domain to handle read/write ACLs on Timeline service event data. Contributed by Zhijie Shen. 2014-09-24 11:27:53 -07:00
Allen Wittenauer 034df0e2eb YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) 2014-09-24 08:47:55 -07:00
Karthik Kambatla f5578207d2 YARN-2252. Intermittent failure of TestFairScheduler.testContinuousScheduling. (Ratandeep Ratti and kasha via kasha) 2014-09-23 00:03:16 -07:00
Karthik Kambatla 568d3dc2bb YARN-1959. Fix headroom calculation in FairScheduler. (Anubhav Dhoot via kasha) 2014-09-22 23:49:39 -07:00
junping_du a9a55db065 YARN-2584. TestContainerManagerSecurity fails on trunk. (Contributed by Jian He) 2014-09-22 22:45:06 -07:00
Karthik Kambatla 43efdd30b5 YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5. (Wei Yan via kasha) 2014-09-22 16:09:52 -07:00
Karthik Kambatla 23e17ce0ff YARN-2540. FairScheduler: Queue filters not working on scheduler page in RM UI. (Ashwin Shankar via kasha) 2014-09-22 15:40:47 -07:00
Jian He 0a641496c7 YARN-1372. Ensure all completed containers are reported to the AMs across RM restart. Contributed by Anubhav Dhoot 2014-09-22 10:30:53 -07:00
Karthik Kambatla 9721e2c1fe YARN-2453. TestProportionalCapacityPreemptionPolicy fails with FairScheduler. (Zhihai Xu via kasha) 2014-09-21 23:13:45 -07:00
Karthik Kambatla c50fc92502 YARN-2452. TestRMApplicationHistoryWriter fails with FairScheduler. (Zhihai Xu via kasha) 2014-09-21 13:15:04 -07:00
Jian He 444acf8ea7 YARN-2565. Fixed RM to not use FileSystemApplicationHistoryStore unless explicitly set. Contributed by Zhijie Shen 2014-09-19 11:26:29 -07:00
Zhijie Shen 6fe5c6b746 YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He. 2014-09-18 21:56:56 -07:00
Jason Lowe a337f0e354 YARN-2561. MR job client cannot reconnect to AM after NM restart. Contributed by Junping Du 2014-09-18 21:34:40 +00:00
Jason Lowe 9ea7b6c063 YARN-2363. Submitted applications occasionally lack a tracking URL. Contributed by Jason Lowe 2014-09-18 20:13:16 +00:00
Vinod Kumar Vavilapalli 485c96e3cb YARN-2001. Added a time threshold for RM to wait before starting container allocations after restart/failover. Contributed by Jian He. 2014-09-18 11:03:12 -07:00
Jian He ee21b13cbd YARN-2559. Fixed NPE in SystemMetricsPublisher when retrieving FinalApplicationStatus. Contributed by Zhijie Shen 2014-09-17 21:44:15 -07:00
Jian He f4886111aa YARN-2558. Updated ContainerTokenIdentifier#read/write to use ContainerId#getContainerId. Contributed by Tsuyoshi OZAWA. 2014-09-17 15:13:59 -07:00
junping_du 90a0c03f0a YARN-1250. Generic history service should support application-acls. (Contributed by Zhijie Shen) 2014-09-16 18:20:49 -07:00
Vinod Kumar Vavilapalli 9f6891d9ef YARN-2531. Added a configuration for admins to be able to override app-configs and enforce/not-enforce strict control of per-container cpu usage. Contributed by Varun Vasudev. 2014-09-16 10:14:46 -07:00
Jian He 88e329fdfb YARN-2529. Generic history service RPC interface doesn't work when service authorization is enabled. Contributed by Zhijie Shen 2014-09-15 13:13:54 -07:00
cnauroth 9d4ec97c95 YARN-2549. TestContainerLaunch fails due to classpath problem with hamcrest classes. Contributed by Chris Nauroth. 2014-09-15 11:53:57 -07:00
Vinod Kumar Vavilapalli 14e2639fd0 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 2014-09-13 18:04:05 -07:00
Zhijie Shen 98588cf044 YARN-2528. Relaxed http response split vulnerability protection for the origins header and made it accept multiple origins in CrossOriginFilter. Contributed by Jonathan Eagles. 2014-09-12 21:33:01 -07:00
XuanGong e65ae575a0 YARN-2456. Possible livelock in CapacityScheduler when RM is recovering 
apps. Contributed by Jian He
 2014-09-12 15:21:46 -07:00
Jonathan Eagles 54e5794806 YARN-2547. Cross Origin Filter throws UnsupportedOperationException upon destroy (Mit Desai via jeagles) 2014-09-12 15:35:17 -05:00
Jian He 3122daa802 YARN-2229. Changed the integer field of ContainerId to be long type. Contributed by Tsuyoshi OZAWA 2014-09-12 10:33:33 -07:00
Jason Lowe 78b048393a YARN-2484. FileSystemRMStateStore#readFile/writeFile should close FSData(In|Out)putStream in final block. Contributed by Tsuyoshi OZAWA 2014-09-12 14:09:14 +00:00
Zhijie Shen 469ea3dcef YARN-2538. Added logs when RM sends roll-overed AMRMToken to AM. Contributed by Xuan Gong. 2014-09-11 22:19:00 -07:00
junping_du 6b8b1608e6 YARN-2033. Merging generic-history into the Timeline Store (Contributed by Zhijie Shen) 2014-09-12 10:04:51 +08:00
Karthik Kambatla c11ada5ea6 YARN-2534. FairScheduler: Potential integer overflow calculating totalMaxShare. (Zhihai Xu via kasha) 2014-09-11 12:06:06 -07:00
Vinod Kumar Vavilapalli 4be95175cd YARN-2440. Enabled Nodemanagers to limit the aggregate cpu usage across all containers to a preconfigured limit. Contributed by Varun Vasudev. 2014-09-10 19:22:52 -07:00
Jian He 83be3ad444 YARN-415. Capture aggregate memory allocation at the app-level for chargeback. Contributed by Eric Payne & Andrey Klochkov 2014-09-10 18:20:54 -07:00
Jian He cbfe26370b YARN-2158. Fixed TestRMWebServicesAppsModification#testSingleAppKill test failure. Contributed by Varun Vasudev 2014-09-10 12:47:34 -07:00
XUAN 47bdfa044a YARN-2459. RM crashes if App gets rejected for any reason and HA is enabled. Contributed by Jian He 2014-09-10 11:44:41 -07:00
Vinod Kumar Vavilapalli b67d5ba784 YARN-2448. Changed ApplicationMasterProtocol to expose RM-recognized resource types to the AMs. Contributed by Varun Vasudev. 2014-09-10 10:15:47 -07:00
Karthik Kambatla 3072c83b38 YARN-1458. FairScheduler: Zero weight can lead to livelock. (Zhihai Xu via kasha) 2014-09-10 08:26:14 -07:00
Zhijie Shen a092cdf32d YARN-2512. Allowed pattern matching for origins in CrossOriginFilter. Contributed by Jonathan Eagles. 2014-09-07 17:49:06 -07:00
Jonathan Eagles e6420fec0a YARN-2508. Cross Origin configuration parameters prefix are not honored (Mit Desai via jeagles) 2014-09-05 19:42:40 -05:00
Zhijie Shen 51a4faf521 YARN-2511. Allowed all origins by default when CrossOriginFilter is enabled. Contributed by Jonathan Eagles. 2014-09-04 15:01:01 -07:00
Jason Lowe 3fa5f728c4 YARN-2431. NM restart: cgroup is not removed for reacquired containers. Contributed by Jason Lowe 2014-09-04 21:11:27 +00:00
Jonathan Eagles 91d45f0f01 YARN-2509. Enable Cross Origin Filter for timeline server only and not all Yarn servers (Mit Desai via jeagles) 2014-09-04 11:53:52 -05:00
Karthik Kambatla 1dcaba9a7a YARN-2394. FairScheduler: Configure fairSharePreemptionThreshold per queue. (Wei Yan via kasha) 2014-09-03 10:27:36 -07:00
Karthik Kambatla 0f34e6f387 YARN-2395. FairScheduler: Preemption timeout should be configurable per queue. (Wei Yan via kasha) 2014-08-30 01:17:13 -07:00
Jian He 5c14bc426b YARN-1506. Changed RMNode/SchedulerNode to update resource with event notification. Contributed by Junping Du 2014-08-29 23:05:51 -07:00
Karthik Kambatla 270a271f53 YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the scheduler page. (Ashwin Shankar and Wei Yan via kasha) 2014-08-29 17:15:38 -07:00
Jason Lowe b03653f9a5 YARN-2462. TestNodeManagerResync#testBlockNewContainerRequestsOnStartAndResync should have a test timeout. Contributed by Eric Payne 2014-08-29 20:15:40 +00:00
Jian He c686aa3533 YARN-2447. RM web service app submission doesn't pass secrets correctly. Contributed by Varun Vasudev 2014-08-29 11:40:47 -07:00
Hitesh Shah 3de66011c2 YARN-2450. Fix typos in log messages. Contributed by Ray Chiang. 2014-08-29 11:16:36 -07:00
Zhijie Shen 4bd0194e6b YARN-2449. Fixed the bug that TimelineAuthenticationFilterInitializer is not automatically added when hadoop.http.filter.initializers is not configured. Contributed by Varun Vasudev. 2014-08-29 09:40:39 -07:00
Karthik Kambatla fa80ca49bd YARN-2405. NPE in FairSchedulerAppsBlock. (Tsuyoshi Ozawa via kasha) 2014-08-28 23:21:37 -07:00
Jian 7b3e27ab73 YARN-2406. Move RM recovery related proto to yarn_server_resourcemanager_recovery.proto. Contributed by Tsuyoshi OZAWA 2014-08-28 21:47:26 -07:00
Zhijie Shen d778abf022 YARN-2035. FileSystemApplicationHistoryStore should not make working dir when it already exists. Contributed by Jonathan Eagles. 2014-08-27 02:01:00 -07:00
Karthik Kambatla d16bfd1d0f YARN-1326. RM should log using RMStore at startup time. (Tsuyoshi Ozawa via kasha) 2014-08-27 01:43:58 -07:00
Karthik Kambatla 0097b15e21 YARN-2393. FairScheduler: Add the notion of steady fair share. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619845 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 15:44:47 +00:00
Jason Darrell Lowe 4236c6600e YARN-2434. RM should not recover containers from previously failed attempt when AM restart is not enabled. Contributed by Jian He 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619614 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 22:41:34 +00:00
Allen Wittenauer 7e75226e68 YARN-2424. LCE should support non-cgroups, non-secure mode (Chris Douglas via aw) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619421 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 14:57:11 +00:00
Karthik Kambatla e1dd210fa5 YARN-1919. Potential NPE in EmbeddedElectorService#stop. (Tsuyoshi Ozawa via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619251 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-20 22:10:52 +00:00
Jian He e4539e88e3 YARN-2174. Enable HTTPs for the writer REST API of TimelineServer. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1619160 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-20 17:05:07 +00:00
Zhijie Shen f6a778c372 YARN-2249. Avoided AM release requests being lost on work preserving RM restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618972 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-19 20:33:49 +00:00
Jian He 375c221960 YARN-2409. RM ActiveToStandBy transition missing stoping previous rmDispatcher. Contributed by Rohith 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618915 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-19 17:49:39 +00:00
Jian He 519c4be95a YARN-2411. Support simple user and group mappings to queues. Contributed by Ram Venkatesh 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618542 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 06:08:45 +00:00
Jian He c3084d6c16 YARN-2389. Added functionality for schedulers to kill all applications in a queue. Contributed by Subramaniam Venkatraman Krishnan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618294 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-15 23:53:57 +00:00
Jian He 7360cec692 YARN-2378. Added support for moving applications across queues in CapacityScheduler. Contributed by Subramaniam Venkatraman Krishnan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618106 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-15 06:00:31 +00:00
Zhijie Shen a9023c2736 YARN-2397. Avoided loading two authentication filters for RM and TS web interfaces. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1618054 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-14 21:17:20 +00:00
Zhijie Shen 41aa4badf8 YARN-2277. Added cross-origin support for the timeline server web services. Contributed by Jonathan Eagles. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617832 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-13 20:29:23 +00:00
Karthik Kambatla 5197f8c3c5 YARN-1370. Fair scheduler to re-populate container allocation state. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617645 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-13 01:38:59 +00:00
Karthik Kambatla 4239695588 YARN-2399. Delete old versions of files. FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617619 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-12 22:51:57 +00:00
Karthik Kambatla 486e718fc1 YARN-2399. FairScheduler: Merge AppSchedulable and FSSchedulerApp into FSAppAttempt. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617600 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-12 21:43:27 +00:00
Junping Du c2febdcbaa YARN-1337. Recover containers upon nodemanager restart. (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617448 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-12 10:56:13 +00:00
Jian He c4dc685343 YARN-2138. Cleaned up notifyDone* APIs in RMStateStore. Contributed by Varun Saxena 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617341 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-11 18:24:24 +00:00
Xuan Gong 946be75704 YARN-2400: Addendum fix for TestAMRestart failure. Contributed by Jian He 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617333 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-11 17:42:53 +00:00
Karthik Kambatla da7b508ffc YARN-2361. RMAppAttempt state machine entries for KILLED state has duplicate event entries. (Zhihai Xu via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617190 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-11 01:42:26 +00:00
Karthik Kambatla bdd3e2ce49 YARN-2337. ResourceManager sets ClientRMService in RMContext multiple times. (Zhihai Xu via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617183 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-11 00:13:27 +00:00
Junping Du e91d099c4a YARN-2302. Refactor TimelineWebServices. (Contributed by Zhijie Shen) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617055 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-10 07:21:15 +00:00
Xuan Gong 743f7f30da YARN-2400. Fixed TestAMRestart fails intermittently. Contributed by Jian He: 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1617028 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-09 23:31:11 +00:00
Karthik Kambatla a7643f4de7 YARN-2026. Fair scheduler: Consider only active queues for computing fairshare. (Ashwin Shankar via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616915 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-09 02:10:00 +00:00
Xuan Gong eeb4acd955 YARN-2212: ApplicationMaster needs to find a way to update the AMRMToken periodically. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616892 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-08 21:38:24 +00:00
Karthik Kambatla 6d39525f6b YARN-2352. Add missing file. FairScheduler: Collect metrics on duration of critical methods that affect performance. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616784 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-08 14:38:18 +00:00
Karthik Kambatla 14864e9c7c YARN-2352. FairScheduler: Collect metrics on duration of critical methods that affect performance. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616769 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-08 14:17:54 +00:00
Jian He 8437df8ba9 YARN-2008. Fixed CapacityScheduler to calculate headroom based on max available capacity instead of configured max capacity. Contributed by Craig Welch 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616580 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-07 20:00:04 +00:00
Zhijie Shen 3bfdb0091d YARN-2288. Made persisted data in LevelDB timeline store be versioned. Contributed by Junping Du. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616540 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-07 17:02:26 +00:00
Zhijie Shen be6360593b YARN-2388. Fixed TestTimelineWebServices failure due to HADOOP-10791. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616405 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-07 05:36:50 +00:00
Karthik Kambatla 8feddc4c84 YARN-2359. Application hangs when it fails to launch AM container. (Zhihai Xu via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1616375 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-07 00:06:17 +00:00
Junping Du b8f151231b YARN-1354. Recover applications upon nodemanager restart. (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1615550 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-04 13:25:37 +00:00
Junping Du 04ae560370 YARN-2370. Fix comment in o.a.h.y.server.resourcemanager.schedulerAppSchedulingInfo (Contributed by Wenwu Peng) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1615469 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-04 01:15:46 +00:00
Jian He a41c314373 YARN-2343. Improve NMToken expire exception message. Contributed by Li Lu 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1615270 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-01 23:44:48 +00:00
Xuan Gong e52f67e389 YARN-1994. Expose YARN/MR endpoints on multiple interfaces. Contributed by Craig Welch, Milan Potocnik,and Arpit Agarwal 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614981 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 20:06:02 +00:00
Zhijie Shen 1d6e178144 YARN-2347. Consolidated RMStateVersion and NMDBSchemaVersion into Version in yarn-server-common. Contributed by Junping Du. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614838 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 09:27:43 +00:00
Karthik Kambatla c0b49ff107 YARN-2328. FairScheduler: Verify update and continuous scheduling threads are stopped when the scheduler is stopped. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1614432 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-29 17:41:52 +00:00
Zhijie Shen d6532d3a77 YARN-2247. Made RM web services authenticate users via kerberos and delegation token. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613821 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-27 17:55:06 +00:00
Aaron Myers 5d4677b57b YARN-1796. container-executor shouldn't require o-r permissions. Contributed by Aaron T. Myers. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613548 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-26 01:51:35 +00:00
Jian He d4fec34933 YARN-2211. Persist AMRMToken master key in RMStateStore for RM recovery. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613515 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-25 20:42:37 +00:00
Karthik Kambatla 1e553858f9 YARN-2214. FairScheduler: preemptContainerPreCheck() in FSParentQueue delays convergence towards fairness. (Ashwin Shankar via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1613459 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-25 16:13:07 +00:00
Devarajulu K 2050e0dad6 YARN-1342. Recover container tokens upon nodemanager restart. Contributed by Jason Lowe. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612995 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-24 05:02:00 +00:00
Jason Darrell Lowe 28fca92521 YARN-2147. client lacks delegation token exception details when application submit fails. Contributed by Chen He 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612950 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-23 21:40:57 +00:00
Sanford Ryza c88402f36d YARN-2313. Livelock can occur in FairScheduler when there are lots of running apps (Tsuyoshi Ozawa via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612769 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-23 05:00:52 +00:00
Karthik Kambatla ff77582991 YARN-2273. NPE in ContinuousScheduling thread when we lose a node. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612720 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 22:44:38 +00:00
Zhijie Shen eac0701c96 YARN-2319. Made the MiniKdc instance start/close before/after the class of TestRMWebServicesDelegationTokens. Contributed by Wenwu Peng. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612588 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 15:15:29 +00:00
Junping Du afb9394c91 YARN-2242. Addendum patch. Improve exception information on AM launch crashes. (Contributed by Li Lu) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612565 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 13:07:23 +00:00
Junping Du 537c361f5b YARN-2013. The diagnostics is always the ExitCodeException stack when the container crashes. (Contributed by Tsuyoshi OZAWA) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612449 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 03:01:58 +00:00
Karthik Kambatla 266a4ff6a5 YARN-2131. Addendum. Add a way to format the RMStateStore. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612443 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-22 01:30:36 +00:00
Jason Darrell Lowe 1ad2d7b405 YARN-2321. NodeManager web UI can incorrectly report Pmem enforcement. Contributed by Leitao Guo 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612411 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-21 21:55:06 +00:00
Jason Darrell Lowe 8a87085820 YARN-2045. Data persisted in NM should be versioned. Contributed by Junping Du 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612285 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-21 14:43:59 +00:00
Sanford Ryza 7c71a3b876 YARN-2323. FairShareComparator creates too many Resource objects (Hong Zhiguo via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1612187 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-21 05:00:27 +00:00
Karthik Kambatla 8871d8ed9f YARN-2244. FairScheduler missing handling of containers for unknown application attempts. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611840 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-19 00:12:05 +00:00
Xuan Gong f1b831ccfb YARN-2208. AMRMTokenManager need to have a way to roll over AMRMToken. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611820 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-18 21:46:29 +00:00
Junping Du 403ec8ea80 YARN-1341. Recover NMTokens upon nodemanager restart. (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611512 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 23:33:22 +00:00
Jian He 3c193811ca YARN-2219. Addendum patch for YARN-2219 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611240 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 03:28:39 +00:00
Vinod Kumar Vavilapalli bda23181bf YARN-2219. Changed ResourceManager to avoid AMs and NMs getting exceptions after RM recovery but before scheduler learns about apps and app-attempts. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1611222 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 00:14:56 +00:00
Vinod Kumar Vavilapalli 030580387a YARN-2233. Implemented ResourceManager web-services to create, renew and cancel delegation tokens. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610876 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-15 23:00:17 +00:00
Mayank Bansal 43589a8df7 YARN-1408 Preemption caused Invalid State Event: ACQUIRED at KILLED and caused a task timeout for 30mins. (Sunil G via mayank) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610860 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-15 21:48:58 +00:00
Vinod Kumar Vavilapalli a6cd712ca7 YARN-2228. Augmented TimelineServer to load pseudo authentication filter when authentication = simple. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610575 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-15 04:33:48 +00:00
Vinod Kumar Vavilapalli c6cc6a6a8e YARN-2260. Fixed ResourceManager's RMNode to correctly remember containers when nodes resync during work-preserving RM restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1610557 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-14 23:32:03 +00:00
Karthik Kambatla 4d211d5fde YARN-2274. FairScheduler: Add debug information about cluster capacity, availability and reservations. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1609942 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-12 15:10:19 +00:00
Jian He c9fb040c87 YARN-2181. Added preemption info to logs and RM web UI. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1609561 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-10 20:03:35 +00:00
Karthik Kambatla 8fbca62a90 YARN-2131. Add a way to format the RMStateStore. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1609278 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-09 19:58:43 +00:00
Zhijie Shen 12c4197b35 YARN-2158. Improved assertion messages of TestRMWebServicesAppsModification. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1608667 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-08 05:50:04 +00:00
Jian He 6d7dbd4fed YARN-1367. Changed NM to not kill containers on NM resync if RM work-preserving restart is enabled. Contributed by Anubhav Dhoot 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1608334 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-07 04:37:59 +00:00
Sanford Ryza 5644f529f3 YARN-2250. FairScheduler.findLowestCommonAncestorQueue returns null when queues not identical (Krisztian Horvath via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607872 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-04 15:16:43 +00:00
Junping Du 5cb489f9d3 YARN-2242. Improve exception information on AM launch crashes. (Contributed by Li Lu) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607655 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-03 14:15:19 +00:00
Vinod Kumar Vavilapalli 45b191e38c YARN-2232. Fixed ResourceManager to allow DelegationToken owners to be able to cancel their own tokens in secure mode. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607484 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 21:36:42 +00:00
Karthik Kambatla 447c1c233b YARN-2241. ZKRMStateStore: On startup, show nicer messages if znodes already exist. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607473 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 20:33:26 +00:00
Steve Loughran d1f54f4f4b YARN-2065 AM cannot create new containers after restart 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607441 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 18:35:10 +00:00
Karthik Kambatla 58e398f31d YARN-2204. Explicitly enable vmem check in TestContainersMonitor#testContainerKillOnMemoryOverflow. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607231 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 02:05:37 +00:00
Mayank Bansal 03a25d2cc1 YARN-2022 Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy (Sunil G via mayank) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607227 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 01:54:47 +00:00
Vinod Kumar Vavilapalli 075ff276ca YARN-1713. Added get-new-app and submit-app functionality to RM web services. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1607216 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-02 00:23:07 +00:00
Xuan Gong e5ae7c55d1 TestRMApplicationHistoryWriter sometimes fails in trunk. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606835 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-30 16:51:22 +00:00
Jian He b0c51504c4 YARN-2052. Embedded an epoch number in container id to ensure the uniqueness of container id after RM restarts. Contributed by Tsuyoshi OZAWA 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606557 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-29 18:24:03 +00:00
Jian He b717d44b52 YARN-614. Changed ResourceManager to not count disk failure, node loss and RM restart towards app failures. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606407 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-28 23:37:46 +00:00
Zhijie Shen 55a0aa0bad YARN-2201. Made TestRMWebServicesAppsModification be independent of the changes on yarn-default.xml. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606285 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-28 03:30:44 +00:00
Jason Darrell Lowe 73927ba695 YARN-2104. Scheduler queue filter failed to work because index of queue column changed. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606265 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-27 23:11:12 +00:00
Karthik Kambatla f911f5495b YARN-2204. Addendum patch. TestAMRestart#testAMRestartWithExistingContainers assumes CapacityScheduler. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1606168 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-27 18:09:41 +00:00
Ravi Prakash f084170a0a YARN 2163. WebUI: Order of AppId in apps table should be consistent with ApplicationId.compareTo() 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605964 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-27 06:43:46 +00:00
Vinod Kumar Vavilapalli 9571db19eb YARN-2171. Improved CapacityScheduling to not lock on nodemanager-count when AMs heartbeat in. Contributed by Jason Lowe. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605616 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-25 21:56:42 +00:00
Karthik Kambatla 1a3a7e0c1a YARN-2204. TestAMRestart#testAMRestartWithExistingContainers assumes CapacityScheduler. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605548 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-25 18:50:53 +00:00
Jian He c3f1c30e65 YARN-1365. Changed ApplicationMasterService to allow an app to re-register after RM restart. Contributed by Anubhav Dhoot 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605263 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-25 04:42:39 +00:00
Vinod Kumar Vavilapalli e285b98f0f YARN-2152. Added missing information into ContainerTokenIdentifier so that NodeManagers can report the same to RM when RM restarts. Contributed Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605205 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 21:43:22 +00:00
Thomas Graves 1f9a0fd927 YARN-2072. RM/NM UIs and webservices are missing vcore information. (Nathan Roberts via tgraves) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605162 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 19:34:34 +00:00
Karthik Kambatla c0991d11eb YARN-2109. Fix TestRM to work with both schedulers. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605142 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 17:30:53 +00:00
Karthik Kambatla db4d277117 YARN-2192. TestRMHA fails when run with a mix of Schedulers. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605138 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 17:05:35 +00:00
Sanford Ryza 29c102cad0 YARN-2111. In FairScheduler.attemptScheduling, we don't count containers as assigned if they have 0 memory but non-zero cores (Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605113 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 15:40:39 +00:00
Vinod Kumar Vavilapalli d16470025a YARN-2074. Changed ResourceManager to not count AM preemptions towards app failures. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1605106 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 15:15:12 +00:00
Jian He 59b5e9fa15 YARN-2191. Added a new test to ensure NM will clean up completed applications in the case of RM restart. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1604949 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-23 22:52:38 +00:00
Karthik Kambatla 6fcbf9b848 YARN-2187. FairScheduler: Disable max-AM-share check by default. (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1604321 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-21 07:30:07 +00:00
Haohui Mai 22cbcd16cb HADOOP-10717. Revert r1603571. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603620 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-18 20:37:25 +00:00
Steve Loughran bae5837d3b HADOOP-10717. Missing JSP support in Jetty 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603534 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-18 17:08:45 +00:00
Jason Darrell Lowe 98238a8d4a YARN-2167. LeveldbIterator should get closed in NMLeveldbStateStoreService#loadLocalizationState() within finally block. Contributed by Junping Du 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603039 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-17 02:12:03 +00:00
Junping Du 072360d128 YARN-1339. Recover DeletionService state upon nodemanager restart. (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603036 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-17 01:02:16 +00:00
Jian He 95897ca14b YARN-1885. Fixed a bug that RM may not send application-clean-up signal to NMs where the completed applications previously ran in case of RM restart. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603028 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-16 23:56:12 +00:00
Karthik Kambatla 082c2ab8b8 YARN-2159. Better logging in SchedulerNode#allocateContainer. (Ray Chiang via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1603003 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-16 20:45:38 +00:00
Vinod Kumar Vavilapalli dc7dd1fa19 YARN-1702. Added kill app functionality to RM web services. Contributed by Varun Vasudev. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1602298 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-12 21:31:52 +00:00
Karthik Kambatla 4bc91b44c9 YARN-2155. FairScheduler: Incorrect threshold check for preemption. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1602295 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-12 21:23:32 +00:00
Jian He c6e29a9f06 YARN-2125. Changed ProportionalCapacityPreemptionPolicy to log CSV in debug level. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601980 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-11 18:31:28 +00:00
Jian He 710a8693e5 YARN-2124. Fixed NPE in ProportionalCapacityPreemptionPolicy. Contributed by Wangda Tan 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601964 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-11 17:30:18 +00:00
Bikas Saha ecfd43a2f1 YARN-2091. Add more values to ContainerExitStatus and pass it from NM to RM and then to app masters (Tsuyoshi OZAWA via bikas) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601762 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-10 20:08:33 +00:00
Karthik Kambatla 5de6f72054 YARN-1424. RMAppAttemptImpl should return the DummyApplicationResourceUsageReport for all invalid accesses. (Ray Chiang via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601742 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-10 19:03:06 +00:00
Jian He c94f2cec3a Augmented RMStateStore with state machine. Contributed by Binglin Chang. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601491 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-09 19:44:31 +00:00
Vinod Kumar Vavilapalli 424fd9494f YARN-1368. Added core functionality of recovering container state into schedulers after ResourceManager Restart so as to preserve running work in the cluster. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601303 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-09 03:09:21 +00:00
Karthik Kambatla 9958072d53 YARN-2132. ZKRMStateStore.ZKAction#runWithRetries doesn't log the exception it encounters. (Vamsee Yarlagadda via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601065 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-07 06:12:59 +00:00
Karthik Kambatla 85d4c787e0 YARN-2128. FairScheduler: Incorrect calculation of amResource usage. (Wei Yan via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601050 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-07 01:21:33 +00:00
Karthik Kambatla 969bef3c77 YARN-2122. In AllocationFileLoaderService, the reloadThread should be created in init() and started in start(). (Robert Kanter via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1601045 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-07 00:49:20 +00:00
Zhijie Shen 08b4aa699a YARN-2117. Fixed the issue that secret file reader is potentially not closed in TimelineAuthenticationFilterInitializer. Contributed by Chen He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600994 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-06 20:07:58 +00:00
Zhijie Shen d7f5d76869 YARN-2118. Fixed the type mismatch in Map#containsKey check of TimelineWebServices#injectOwnerInfo. Contributed by Ted Yu. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600967 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-06 17:47:03 +00:00
Junping Du 0ceb742549 YARN-1977. Add tests on getApplicationRequest with filtering start time range. (Contributed by Junping Du) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600644 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-05 13:15:44 +00:00
Karthik Kambatla d1d6bb70f4 YARN-2061. Revisit logging levels in ZKRMStateStore. (Ray Chiang via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600498 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-04 21:11:09 +00:00
Karthik Kambatla 731f417f67 YARN-2119. DEFAULT_PROXY_ADDRESS should use DEFAULT_PROXY_PORT. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1600484 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-04 20:12:47 +00:00
Sanford Ryza 16caa3fd18 YARN-1913. With Fair Scheduler, cluster can logjam when all resources are consumed by AMs (Wei Yan via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1599400 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-03 00:56:48 +00:00
Karthik Kambatla 0aad2d56df YARN-1550. NPE in FairSchedulerAppsBlock#render. (Anubhav Dhoot via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1599345 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-02 20:22:52 +00:00
Karthik Kambatla a4ba451802 YARN-1474. Make schedulers services. (Tsuyoshi Ozawa via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598908 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-31 19:33:09 +00:00
Vinod Kumar Vavilapalli 23c325ad47 YARN-2115. Replaced RegisterNodeManagerRequest's ContainerStatus with a new NMContainerStatus which has more information that is needed for work-preserving RM-restart. Contributed by Jian He. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598790 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-31 00:20:50 +00:00
Junping Du 66598697a6 YARN-1338. Recover localized resource cache state upon nodemanager restart (Contributed by Jason Lowe) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598640 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-30 15:37:27 +00:00
Karthik Kambatla 49a3a0cd0c YARN-2054. Better defaults for YARN ZK configs for retries and retry-inteval when HA is enabled. (kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598630 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-30 15:24:49 +00:00
Arpit Agarwal 4a4868e523 HADOOP-10448. Support pluggable mechanism to specify proxy user settings (Contributed by Benoy Antony) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598396 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-29 20:52:01 +00:00
Vinod Kumar Vavilapalli cd24e955cf YARN-2112. Fixed yarn-common's pom.xml to include jackson dependencies so that both Timeline Server and client can access them. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598373 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-29 19:06:36 +00:00
Sanford Ryza 342da5b4d3 YARN-596. Use scheduling policies throughout the queue hierarchy to decide which containers to preempt (Wei Yan via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1598197 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-29 04:01:24 +00:00