Commit Graph

2144 Commits

Author SHA1 Message Date
Varun Vasudev fb9c1519b3 YARN-4252. Log container-executor invocation details when exit code is non-zero. Contributed by Sidharta Seethana. 2015-10-14 15:18:08 +05:30
Varun Vasudev 8d59293089 YARN-4253. Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor. Contributed by Sidharta Seethana. 2015-10-14 14:58:10 +05:30
Steve Loughran c59af2fdf8 HADOOP-10775. Shell operations to fail with meaningful errors on windows if winutils.exe not found. (stevel) 2015-10-13 21:49:28 +01:00
Varun Vasudev 5b6bae0094 YARN-4017. container-executor overuses PATH_MAX. Contributed by Sidharta Seethana 2015-10-13 12:48:30 +05:30
Jian He 9849c8b386 YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left. Contributed by Meng Ding 2015-10-12 11:51:33 -07:00
Zhihai Xu 049c6e8dc0 YARN-4201. AMBlacklist does not work for minicluster. Contributed by Jun Gong. 2015-10-12 00:14:25 -07:00
Devaraj K db93047881 YARN-3964. Support NodeLabelsProvider at Resource Manager side.
Contributed by Dian Fu.
2015-10-11 11:21:29 +05:30
Wangda Tan def374e666 YARN-4140. RM container allocation delayed incase of app submitted to Nodelabel partition. (Bibin A Chundatt via wangda) 2015-10-09 16:38:59 -07:00
Karthik Kambatla 4aa9b3e75c MAPREDUCE-6302. Incorrect headroom can lead to a deadlock between map and reduce allocations. (kasha) 2015-10-09 07:37:39 -07:00
Jason Lowe a0bca2b5ad YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S 2015-10-09 14:17:38 +00:00
Rohith Sharma K S 8f195387a4 YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned for a user. (Anubhav Dhoot via rohithsharmaks) 2015-10-09 10:09:26 +05:30
Jason Lowe 8d226225d0 YARN-3943. Use separate threshold configurations for disk-full detection and disk-not-full detection. Contributed by Zhihai Xu 2015-10-08 22:25:34 +00:00
Rohith Sharma K S 3793cbe4c3 YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (Bibin A Chundatt via rohithsharmaks) 2015-10-07 10:12:14 +05:30
Rohith Sharma K S 9156fc60c6 YARN-4209. RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition. (Zhihai Xu via rohithsharmaks) 2015-10-07 09:34:59 +05:30
Wangda Tan 29a582ada0 YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request. (Naganarasimha G R via wangda) 2015-10-06 11:56:04 -07:00
Wangda Tan 30ac69c6bd YARN-4176. Resync NM nodelabels with RM periodically for distributed nodelabels. (Bibin A Chundatt via wangda) 2015-10-05 15:47:21 -07:00
Vinod Kumar Vavilapalli f1c19b9365 HADOOP-12441. Fixed shell-kill command behaviour to work correctly on some Linux distributions after HADOOP-12317. Contributed by Wangda Tan. 2015-10-05 10:56:55 -07:00
Harsh J c918f7be5e HADOOP-12458. Retries is typoed to spell Retires in parts of hadoop-yarn and hadoop-common. Contributed by Neelesh Srinivas Salian. 2015-10-03 18:37:58 +05:30
Xuan 8f08532bde YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma 2015-10-02 18:50:47 -07:00
Jason Lowe fdf02d1f26 YARN-3619. ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException. Contributed by Zhihai Xu 2015-10-02 20:20:31 +00:00
Jason Lowe 854d25b0c3 YARN-3727. For better error recovery, check if the directory exists before using it for localization. Contributed by Zhihai Xu 2015-09-30 14:59:44 +00:00
Karthik Kambatla a0b5a0a419 YARN-4066. Large number of queues choke fair scheduler. (Johan Gustavsson via kasha) 2015-09-29 07:55:34 -07:00
Anubhav Dhoot 9735afe967 YARN-4180. AMLauncher does not retry on failures when talking to NM. (adhoot) 2015-09-28 16:13:41 -07:00
Jason Lowe 9f53a95ff6 YARN-4141. Runtime Application Priority change should not throw exception for applications at finishing states. Contributed by Sunil G 2015-09-28 22:55:20 +00:00
Anubhav Dhoot fb2e525c07 YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot) 2015-09-28 09:05:45 -07:00
Xuan d1b9b85244 YARN-3624. ApplicationHistoryServer should not reverse the order of the
filters it gets. Contributed by Mit Desai
2015-09-24 15:01:17 -07:00
Rohith Sharma K S a9aafad12b YARN-4044. Running applications information changes such as movequeue is not published to TimeLine server. (Sunil G via rohithsharmaks) 2015-09-24 12:13:22 +05:30
Rohith Sharma K S 8ed0d4b744 YARN-4152. NodeManager crash with NPE when LogAggregationService#stopContainer called for absent container. (Bibin A Chundatt via rohithsharmaks) 2015-09-24 11:24:14 +05:30
Jian He b3f6b641dc YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan 2015-09-23 13:29:38 -07:00
Jian He 89cab1ba5f YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 2015-09-23 13:29:38 -07:00
Jian He c57eac5dfe YARN-3868. Recovery support for container resizing. Contributed by Meng Ding 2015-09-23 13:29:38 -07:00
Jian He c3dc1af072 YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding 2015-09-23 13:29:37 -07:00
Jian He c59ae4eeb1 YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan 2015-09-23 13:29:37 -07:00
Jian He 5f5a968d65 YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding 2015-09-23 13:29:37 -07:00
Jian He ffd820c27a YARN-1645. ContainerManager implementation to support container resizing. Contributed by Meng Ding & Wangda Tan 2015-09-23 13:29:37 -07:00
Jian He 83a18add10 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan) 2015-09-23 13:29:36 -07:00
Jian He dfe2cb849f YARN-3866. AM-RM protocol changes to support container resizing. Contributed by Meng Ding 2015-09-23 13:29:36 -07:00
Jason Lowe 692d51c09d YARN-3975. WebAppProxyServlet should not redirect to RM page if AHS is enabled. Contributed by Mit Desai 2015-09-23 16:38:55 +00:00
Jason Lowe c890c51a91 YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu 2015-09-23 15:42:01 +00:00
Tsuyoshi Ozawa dfd807afab HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 2015-09-22 12:54:29 +09:00
Chris Douglas 8e01b0d97a YARN-4188. Make MoveApplicationAcrossQueues abstract, newInstance static
Contributed by Giovanni Matteo Fumarola
2015-09-21 17:51:48 -07:00
Wangda Tan b00392dd9c YARN-4113. RM should respect retry-interval when uses RetryPolicies.RETRY_FOREVER. (Sunil G via wangda) 2015-09-21 11:04:03 -07:00
Rohith Sharma K S c9cb6a5960 YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks) 2015-09-21 09:59:30 +05:30
Arun Suresh 94dec5a916 YARN-3920. FairScheduler container reservation on a node should be configurable to limit it to large containers (adhoot via asuresh) 2015-09-18 14:02:55 -07:00
Wangda Tan 9bc913a35c YARN-3212. RMNode State Transition Update with DECOMMISSIONING state. (Junping Du via wangda) 2015-09-18 10:04:17 -07:00
Steve Loughran a7201d635f YARN-2597 MiniYARNCluster should propagate reason for AHS not starting 2015-09-18 09:45:35 +01:00
Rohith Sharma K S 723c31d45b YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state.(Nijel S F via rohithsharmaks) 2015-09-18 08:44:10 +05:30
Jian He 6c6e734f0b YARN-4034. Render cluster Max Priority in scheduler metrics in RM web UI. Contributed by Rohith Sharma K S 2015-09-17 14:55:50 +08:00
Steve Loughran bf2f2b4fc4 YARN-3433. Jersey tests failing with Port in Use -again. (Brahma Reddy Battula) 2015-09-16 10:31:02 +01:00
Jian He 452079af8b YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface. Contributed by Naganarasimha G R 2015-09-16 14:59:20 +08:00
Xuan 34ef1a092b YARN-4149. yarn logs -am should provide an option to fetch all the log
files. Contributed by Varun Vasudev
2015-09-15 14:36:30 -07:00
Jason Lowe 8c1cdb17a0 YARN-4158. Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainers. Contributed by Zhihai Xu 2015-09-15 20:21:33 +00:00
Wangda Tan ae5308fe1d YARN-3717. Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. (Naganarasimha G R via wangda) 2015-09-15 11:40:50 -07:00
Junping Du 73e3a49eb0 YARN-313. Add Admin API for supporting node resource configuration in command line. (Contributed by Inigo Goiri, Kenji Kikushima and Junping Du) 2015-09-15 07:56:47 -07:00
Jian He 5468baa80a YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan 2015-09-15 15:39:20 +08:00
Wangda Tan e2a0270217 YARN-4151. Fix findbugs errors in hadoop-yarn-server-common module. (Meng Ding via wangda) 2015-09-14 16:00:43 -07:00
Jian He e1b1d7e4ae YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt 2015-09-14 14:09:19 +08:00
Karthik Kambatla 332b520a48 YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. (Zhihai Xu via kasha) 2015-09-13 18:07:43 -07:00
Karthik Kambatla 81df7b586a YARN-2005. Blacklisting support for scheduling AMs. (Anubhav Dhoot via kasha) 2015-09-13 17:03:15 -07:00
Steve Loughran 7269906254 HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:26 +01:00
Robert Kanter ea4bb2749f YARN-4145. Make RMHATestBase abstract so its not run when running all tests under that namespace (adhoot via rkanter) 2015-09-11 11:46:10 -07:00
Robert Kanter b84fb41bb6 YARN-4115. Reduce loglevel of ContainerManagementProtocolProxy to Debug (adhoot via rkanter) 2015-09-11 11:46:10 -07:00
Varun Vasudev 486d5cb803 YARN-4136. LinuxContainerExecutor loses info when forwarding ResourceHandlerException. Contributed by Bibin A Chundatt. 2015-09-11 14:37:48 +05:30
Wangda Tan 77666105b4 YARN-4106. NodeLabels for NM in distributed mode is not updated even after clusterNodelabel addition in RM. (Bibin A Chundatt via wangda) 2015-09-10 09:30:09 -07:00
Robert Kanter 6dd6ca442a YARN-4086. Allow Aggregated Log readers to handle HAR files (rkanter) 2015-09-09 18:03:04 -07:00
Zhihai Xu 16b9037dc1 YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe. 2015-09-08 12:29:54 -07:00
Varun Vasudev 435f935ba7 YARN-4121. Fix typos in capacity scheduler documentation. Contributed by Kai Sasaki. 2015-09-08 12:33:43 +05:30
Jian He 6f72f1e600 YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil 2015-09-08 09:35:46 +08:00
Xuan 9b78e6e33d YARN-4087. Followup fixes after YARN-2019 regarding RM behavior when
state-store error occurs. Contributed by Jian He
2015-09-07 17:45:47 -07:00
Varun Vasudev 1dbd8e34a7 YARN-3591. Resource localization on a bad disk causes subsequent containers failure. Contributed by Lavkesh Lahngir. 2015-09-07 11:32:12 +05:30
Wangda Tan bcc85e3bab YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) 2015-09-04 15:13:53 -07:00
Jason Lowe 6eaca2e363 YARN-4105. Capacity Scheduler headroom for DRF is wrong. Contributed by Chang Li 2015-09-04 15:30:53 +00:00
Varun Vasudev 40d222e862 YARN-4103. RM WebServices missing scheme for appattempts logLinks. Contributed by Jonathan Eagles. 2015-09-04 14:31:51 +05:30
Varun Vasudev b469ac531a YARN-3970. Add REST api support for Application Priority. Contributed by Naganarasimha G R. 2015-09-03 16:40:10 +05:30
Jian He 09c64ba1ba YARN-4101. RM should print alert messages if Zookeeper and Resourcemanager gets connection issue. Contributed by Xuan Gong 2015-09-02 17:45:23 -07:00
Rohith Sharma K S 7d6687fe76 YARN-3893. Both RM in active state when Admin#transitionToActive failure from refeshAll() (Bibin A Chundatt via rohithsharmaks) 2015-09-02 15:22:48 +05:30
Rohith Sharma K S 095ab9ab5f YARN-4073. Removed unused ApplicationACLsManager in ContainerManagerImpl constructor. (Naganarasimha G R via rohithsharmaks) 2015-09-02 14:13:33 +05:30
Andrew Wang 0eb9b19325 HADOOP-12369. Point hadoop-project/pom.xml java.security.krb5.conf within target folder. 2015-09-01 10:57:32 -07:00
Tsuyoshi Ozawa faa38e1aa4 YARN-2801. Add documentation for node labels feature. Contributed by Wangda Tan and Naganarasimha G R. 2015-09-01 17:54:49 +09:00
Varun Vasudev bf669b6d9f YARN-4082. Container shouldn't be killed when node's label updated. Contributed by Wangda Tan. 2015-09-01 14:19:11 +05:30
Jian He a3fd2ccc86 YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. Contributed by Xuan Gong 2015-08-31 17:33:24 -07:00
Junping Du beb65c9465 YARN-1556. NPE getting application report with a null appId. Contributed by Weiwei Yang. 2015-08-28 05:57:34 -07:00
Jian He a9c8ea71aa YARN-3250. Support admin cli interface in for Application Priority. Contributed by Rohith Sharma K S 2015-08-27 13:25:53 -07:00
Andrew Wang 90fe7bcc3b HADOOP-12362. Set hadoop.tmp.dir and hadoop.log.dir in pom. Contributed by Charlie Helin. 2015-08-27 10:17:00 -07:00
Jian He 57c7ae1aff YARN-4014. Support user cli interface in for Application Priority. Contributed by Rohith Sharma K S 2015-08-24 20:36:44 -07:00
Rohith Sharma K S feaf034994 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 2015-08-24 11:25:07 +05:30
Xuan b71c6006f5 YARN-221. Addendum patch to compilation issue which is caused by missing
AllContainerLogAggregationPolicy. Contributed by Xuan Gong
2015-08-23 16:46:30 -07:00
Xuan 37e1c3d82a YARN-221. NM should provide a way for AM to tell it not to aggregate
logs. Contributed by Ming Ma
2015-08-22 16:25:24 -07:00
Rohith Sharma K S 22de7c1dca YARN-3986. getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface 2015-08-21 10:51:11 +05:30
Wangda Tan fc07464d1a YARN-2923. Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup. (Naganarasimha G R) 2015-08-20 11:51:03 -07:00
Xuan 22dc5fc209 YARN-4028. AppBlock page key update and diagnostics value null on
recovery. Contributed by Bibin A Chundatt
2015-08-18 22:53:03 -07:00
Zhihai Xu 14215c8ef8 YARN-4057. If ContainersMonitor is not enabled, only print related log info one time. Contributed by Jun Gong. 2015-08-18 11:36:02 -07:00
Zhihai Xu 3a76a010b8 YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. 2015-08-18 10:36:40 -07:00
Karthik Kambatla 13604bd5f1 YARN-4055. Report node resource utilization in heartbeat. (Inigo Goiri via kasha) 2015-08-16 15:08:53 -07:00
Karthik Kambatla def12933b3 YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) 2015-08-16 06:24:16 -07:00
Jian He 0a030546e2 YARN-3987. Send AM container completed msg to NM once AM finishes. Contributed by sandflee 2015-08-13 16:22:53 -07:00
Jian He 7a445fcfab YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe 2015-08-13 16:02:57 -07:00
Jian He 38aed1a94e YARN-4005. Completed container whose app is finished is possibly not removed from NMStateStore. Contributed by Jun Gong 2015-08-13 14:46:08 -07:00
Robert Kanter dc2340c60e YARN-4031. Add JvmPauseMonitor to ApplicationHistoryServer and WebAppProxyServer (djp via rkanter) 2015-08-12 15:47:47 -07:00
Jian He e5003be907 YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan 2015-08-12 15:07:50 -07:00