Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,226 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1385 Commits

Author SHA1 Message Date
Jason Lowe 0f30913ea8 YARN-3552. RM Web UI shows -1 running containers for completed apps. Contributed by Rohith 
(cherry picked from commit 9356cf8676)
 2015-05-05 13:43:57 +00:00
Jian He f71c49c5e3 YARN-2725. Added test cases of retrying creating znode in ZKRMStateStore. Contributed by Tsuyoshi Ozawa 
(cherry picked from commit d701acc9c6)
 2015-05-04 16:17:01 -07:00
Wangda Tan 4e77ee2f6f YARN-3375. NodeHealthScriptRunner.shouldRun() check is performing 3 times for starting NodeHealthScriptRunner (Devaraj K via wangda) 
(cherry picked from commit 71f4de220c)
 2015-05-04 16:02:57 -07:00
Allen Wittenauer 02e650248d YARN-2980. Move health check script related functionality to hadoop-common (Varun Saxena via aw) 
(cherry picked from commit d4ac6822e1)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
 2015-05-04 16:00:20 -07:00
Jason Lowe 99b49a8181 YARN-3097. Logging of resource recovery on NM restart has redundancies. Contributed by Eric Payne 
(cherry picked from commit 8f65c793f2)
 2015-05-04 10:32:33 -05:00
Gera Shegalov 9325dd863c YARN-2893. AMLaucher: sporadic job failures due to EOFException in readTokenStorageStream. (Zhihai Xu via gera) 
(cherry picked from commit f8204e241d)
 2015-05-01 19:04:48 -07:00
Robert Kanter 32b3b8e2ef YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter) 
(cherry picked from commit ac7d152901)
 2015-05-01 16:39:59 -07:00
Vinod Kumar Vavilapalli 90f6e8c90f YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan. 
(cherry picked from commit 1b3b9e5c31)
 2015-04-30 21:43:22 -07:00
Zhijie Shen c9ee316045 YARN-3544. Got back AM logs link on the RM web UI for a completed app. Contributed by Xuan Gong. 
(cherry picked from commit 21bf2cdcb77f69abc906e6cd401a8fb221f250e9)
 2015-04-30 13:06:46 -07:00
Wangda Tan 100b41add6 YARN-3564. Fix TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable fails randomly. (Jian He via wangda) 
(cherry picked from commit e2e8f77118)
 2015-04-30 11:06:47 -07:00
Jian He a7a5737bc4 YARN-3533. Test: Fix launchAM in MockRM to wait for attempt to be scheduled. Contributed by Anubhav Dhoot 
(cherry picked from commit 4c1af156ae)
 2015-04-29 14:51:04 -07:00
tgraves 2e13183f60 YARN-3517. RM web ui for dumping scheduler logs should be for admins only (Varun Vasudev via tgraves) 
(cherry picked from commit 2e215484bd)
 2015-04-29 21:27:16 +00:00
Karthik Kambatla 4cc38df7de YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) 
(cherry picked from commit 8f82970e0c)
 2015-04-28 21:03:12 -07:00
Wangda Tan 8ee632caa7 YARN-2740. Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled. (Naganarasimha G R via wangda) 
(cherry picked from commit db1b674b50)
 2015-04-27 16:42:36 -07:00
Xuan 0e8af401d7 YARN-3530. ATS throws exception on trying to filter results without 
otherinfo. Contributed by zhijie shen

(cherry picked from commit 7f07c4d810)
 2015-04-27 10:37:40 -07:00
Karthik Kambatla 4045c41afe YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha) 
(cherry picked from commit 47279c3228)
 2015-04-26 09:14:12 -07:00
Jian He 9bf09b334d YARN-2498. Respect labels in preemption policy of capacity scheduler for inter-queue preemption. Contributed by Wangda Tan 
(cherry picked from commit d497f6ea2b)
 2015-04-24 17:03:57 -07:00
Tsuyoshi Ozawa 3884948d6c YARN-3406. Display count of running containers in the RM's Web UI. Contributed by Ryu Kobayashi. 
(cherry picked from commit 4a3dabd94f)
 2015-04-25 07:17:42 +09:00
Jason Lowe cf4154676b YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked. Contributed by Brahma Reddy Battula 
(cherry picked from commit 5e093f0d40)
 2015-04-24 22:04:03 +00:00
Jian He 0583c27fb1 YARN-3387. Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time. Contributed by Sandflee 
(cherry picked from commit d03dcb9635)
 2015-04-24 12:14:17 -07:00
Xuan c184466939 YARN-3511. Add errors and warnings page to ATS. Contributed by Varun Vasudev 
(cherry picked from commit eee9facbba)
 2015-04-24 09:43:35 -07:00
Xuan 8f6053ae51 YARN-3516. killing ContainerLocalizer action doesn't take effect when 
private localizer receives FETCH_FAILURE status. Contributed by zhihai
xu

(cherry picked from commit 0b3f8957a8)
 2015-04-23 16:41:52 -07:00
Vinod Kumar Vavilapalli 0ec6e7e7f5 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv) 
(cherry picked from commit f5fe35e297)
 2015-04-23 11:25:30 -07:00
Wangda Tan 9692bcc2c8 YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) 
(cherry picked from commit 395205444e)
 2015-04-23 10:48:00 -07:00
tgraves 1cd2fcf25d YARN-3434. Interaction between reservations and userlimit can result in significant ULF violation 
(cherry picked from commit 189a63a719)
 2015-04-23 14:49:24 +00:00
Steve Loughran 63ccb9167e YARN-2605. [RM HA] Rest api endpoints doing redirect incorrectly. (Xuan Gong via stevel) 2015-04-23 09:22:23 +01:00
Vinod Kumar Vavilapalli 04783b0402 YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. 
(cherry picked from commit a100be685c)
 2015-04-22 17:36:55 -07:00
Junping Du 8de6045842 New parameter of CLI for decommissioning node gracefully in RMAdmin CLI. Contributed by Devaraj K 
(cherry picked from commit fad9d7e85b)
 2015-04-22 10:08:16 -07:00
Jian He 613a783380 YARN-3503. Expose disk utilization percentage and bad local and log dir counts in NM metrics. Contributed by Varun Vasudev 
(cherry picked from commit 674c7ef649)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
 2015-04-21 21:06:06 -07:00
Jian He dc4698bb33 YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. Contributed by Rohith Sharmaks 
(cherry picked from commit bdd90110e6)
 2015-04-21 20:34:30 -07:00
Wangda Tan 8a9599fb9e YARN-3410. YARN admin should be able to remove individual application records from RMStateStore. (Rohith Sharmaks via wangda) 2015-04-21 17:47:27 -07:00
Tsuyoshi Ozawa 0b87ae27aa YARN-3495. Confusing log generated by FairScheduler. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 105afd5477)
 2015-04-22 05:48:16 +09:00
Wangda Tan d0ea982e64 YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) 
(cherry picked from commit 44872b76fc)
 2015-04-20 17:13:01 -07:00
Jian He 6f70f1b1af YARN-3136. Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers. Contributed by Sunil G 
(cherry picked from commit 497c86b485)
 2015-04-18 12:47:30 -07:00
Wangda Tan 3316cd4357 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763ac)
 2015-04-17 17:23:00 -07:00
Wangda Tan e7cbecddc3 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d)
 2015-04-17 17:15:16 -07:00
Jian He 9827da95bc YARN-3451. Display attempt start time and elapsed time on the web UI. Contributed by Rohith Sharmaks 
(cherry picked from commit 6779467ab6)
 2015-04-17 15:57:36 -07:00
Jian He 57eb07d34b YARN-2696. Queue sorting in CapacityScheduler should consider node label. Contributed by Wangda Tan 
(cherry picked from commit d573f09fb9)
 2015-04-17 13:37:34 -07:00
Junping Du b244701fad YARN-1402. Update related Web UI and CLI with exposing client API to check log aggregation status. Contributed by Xuan Gong. 
(cherry picked from commit 1db355a875)
 2015-04-17 13:20:27 -07:00
Jian He 42cf8e0554 YARN-3021. YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp. Contributed by Yongjun Zhang 
(cherry picked from commit bb6dde68f1)
 2015-04-16 19:44:32 -07:00
Akira Ajisaka 51c458a5c4 MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 1fa8075794)
 2015-04-16 10:06:03 -05:00
Jian He 6fed2c2a79 YARN-3354. Add node label expression in ContainerTokenIdentifier to support RM recovery. Contributed by Wangda Tan 
(cherry picked from commit 1b89a3e173)
 2015-04-15 14:03:29 -07:00
Jian He 58f99b740a YARN-3404. Display queue name on application page. Contributed by Ryu Kobayashi 
(cherry picked from commit b2e6cf607f)
 2015-04-15 13:53:24 -07:00
Tsuyoshi Ozawa e977247f3e YARN-3326. Support RESTful API for getLabelsToNodes. Contributed by Naganarasimha G R. 
(cherry picked from commit e48cedc663)
 2015-04-15 14:04:24 -05:00
Wangda Tan 89a7c98436 YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. (Craig Welch via wangda) 
(cherry picked from commit 5004e75332)
 2015-04-15 09:57:52 -07:00
Harsh J 01af29106a YARN-3462. Patches applied for YARN-2424 are inconsistent between trunk and branch-2. Contributed by Naganarasimha G R. 2015-04-15 09:41:43 +05:30
Jian He 9ebbf1bfce YARN-3361. CapacityScheduler side changes to support non-exclusive node labels. Contributed by Wangda Tan 
(cherry picked from commit 0fefda645b)
 2015-04-14 11:46:35 -07:00
Jian He 81bbee6852 YARN-3266. RMContext#inactiveNodes should have NodeId as map key. Contributed by Chengbing Liu 
(cherry picked from commit b46ee1e7a3)
 2015-04-14 11:31:42 -07:00
Junping Du 7c072bf092 YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. 
(cherry picked from commit 838b06ac87)
 2015-04-13 18:37:39 -07:00
Jian He d58f5c8894 YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. Contributed by Rohith Sharmaks 
(cherry picked from commit a1afbc48b5)
 2015-04-13 14:07:51 -07:00
Jian He 32b01263d3 YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root queue metrics. Contributed by Rohit Agarwal 
(cherry picked from commit 944a16579f)
 2015-04-13 10:54:15 -07:00
Karthik Kambatla ad99d268e5 YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. (Jun Hong via kasha) 
(cherry picked from commit e516706b89)
 2015-04-10 11:21:30 -07:00
Jian He d3daf9665c YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev 2015-04-10 10:40:31 -07:00
Junping Du 0891de0712 YARN-1376. NM need to notify the log aggregation status to RM through Node heartbeat. Contributed by Xuan Gong. 
(cherry picked from commit 92431c9617)
 2015-04-10 08:58:07 -07:00
Xuan f5b49160d9 YARN-3293. Track and display capacity scheduler health metrics in web 
UI. Contributed by Varun Vasudev

(cherry picked from commit afa5d4715a)
 2015-04-09 23:40:02 -07:00
Vinod Kumar Vavilapalli 1ff3fd33ed YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e)
 2015-04-09 13:10:06 -07:00
Wangda Tan 73d9e4c77b YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column. (Jason Lowe via wangda) 
(cherry picked from commit 1885141e90)
 2015-04-09 10:36:10 -07:00
Karthik Kambatla 53e0bf5c17 YARN-3465. Use LinkedHashMap to preserve order of resource requests. (Zhihai Xu via kasha) 
(cherry picked from commit 6495940eae)
 2015-04-09 00:10:23 -07:00
Hitesh Shah 55b794e7fa YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 
(cherry picked from commit 265ed1fe80)
 2015-04-08 14:14:06 -07:00
Tsuyoshi Ozawa ac32fa187c YARN-3457. NPE when NodeManager.serviceInit fails and stopRecoveryStore called. Contributed by Bibin A Chundatt. 
(cherry picked from commit dd852f5b8c)
 2015-04-08 15:56:33 +09:00
Xuan 0522d6970d YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for 
a fixed time period. Contributed by Varun Vasudev

(cherry picked from commit d27e9241e8)
 2015-04-07 09:54:29 -07:00
Xuan 7941cc554b YARN-3110. Few issues in ApplicationHistory web ui. Contributed by Naganarasimha G R 
(cherry picked from commit 19a4feaf6f)
 2015-04-07 08:24:09 -07:00
Robert Kanter 354d2c75cf YARN-2429. TestAMRMTokens.testTokenExpiry fails Intermittently with error message:Invalid AMRMToken (zxu via rkanter) 
(cherry picked from commit 99b08a748e)
 2015-04-06 14:11:50 -07:00
Tsuyoshi Ozawa d2b2d76cce TestFairScheduler.testContinuousScheduling fails Intermittently. Contributed by Zhihai Xu. 
(cherry picked from commit 53959e69f7)
 2015-04-06 20:19:47 +09:00
Xuan 1eb6d74967 YARN-3435. AM container to be allocated Appattempt AM container shown as 
null. Contributed by Bibin A Chundatt

(cherry picked from commit 96d72118f5)
 2015-04-05 00:43:22 -07:00
Wangda Tan 405aead879 YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) 
(cherry picked from commit bad070fe15)
 2015-04-02 17:23:44 -07:00
Vinod Kumar Vavilapalli d8e17c58bc YARN-3365. Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control. Contributed by Sidharta Seethana. 
(cherry picked from commit b21c72777a)
 2015-04-02 16:55:00 -07:00
Sandy Ryza 6286cfd711 YARN-3415. Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue (Zhihai Xu via Sandy Ryza) 2015-04-02 13:57:53 -07:00
Zhijie Shen 6d0bebc912 YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong. 
(cherry picked from commit 8366a36ad3)
 2015-04-01 13:49:00 -07:00
Xuan e26b6e55e9 YARN-3248. Display count of nodes blacklisted by apps in the web UI. 
Contributed by Varun Vasudev

(cherry picked from commit 4728bdfa15)
 2015-04-01 04:20:37 -07:00
Tsuyoshi Ozawa 80755edb7a YARN-3424. Change logs for ContainerMonitorImpl's resourse monitoring from info to debug. Contributed by Anubhav Dhoot. 
(cherry picked from commit c69ba81497)
 2015-04-01 17:44:40 +09:00
Karthik Kambatla 93b29d962d YARN-3428. Debug log resources to be localized for a container. (kasha) 
(cherry picked from commit 2daa478a64)
 2015-03-31 17:35:11 -07:00
Karthik Kambatla 85c420a4d4 YARN-3412. RM tests should use MockRM where possible. (kasha) 
(cherry picked from commit 79f7f2aabf)
 2015-03-31 09:22:16 -07:00
Tsuyoshi Ozawa 99b8255693 YARN-3258. FairScheduler: Need to add more logging to investigate allocations. Contributed by Anubhav Dhoot. 
(cherry picked from commit b5a22e9838)
 2015-03-31 17:43:07 +09:00
Wangda Tan cba4ed1678 YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) 
(cherry picked from commit 2a945d24f7)
 2015-03-30 12:05:54 -07:00
Vinod Kumar Vavilapalli c5bc48946d YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. Contributed by Junping Du and Karthik Kambatla. 
(cherry picked from commit c358368f51)
 2015-03-30 10:11:12 -07:00
Ravi Prakash b1b4951452 YARN-3288. Document and fix indentation in the DockerContainerExecutor code 
(cherry picked from commit e0ccea33c9)
 2015-03-28 08:01:26 -07:00
Haohui Mai 17f17dac4e HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. 2015-03-25 11:13:04 -07:00
Xuan 0c4fa872be YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. 
Contributed by Varun Saxena

(cherry picked from commit e556198e71)
 2015-03-25 04:51:55 -07:00
Wangda Tan 29629020c1 YARN-3383. AdminService should use warn instead of info to log exception when operation fails. (Li Lu via wangda) 
(cherry picked from commit 97a7277a2d)
 2015-03-24 10:39:45 -07:00
Xuan cbdcdfad6d YARN-3393. Getting application(s) goes wrong when app finishes before 
starting the attempt. Contributed by Zhijie Shen

(cherry picked from commit 9fae455e26)
 2015-03-23 20:34:29 -07:00
Karthik Kambatla 4e0c48703e YARN-2868. FairScheduler: Metric for latency to allocate first container for an application. (Ray Chiang via kasha) 
(cherry picked from commit 972f1f1ab9)
 2015-03-23 14:10:23 -07:00
Karthik Kambatla 75591e4131 YARN-3241. FairScheduler handles invalid queue names inconsistently. (Zhihai Xu via kasha) 
(cherry picked from commit 2bc097cd14)
 2015-03-23 13:24:22 -07:00
cnauroth 342c525eaa YARN-3336. FileSystem memory leak in DelegationTokenRenewer. 
(cherry picked from commit 6ca1f12024)
 2015-03-23 10:46:06 -07:00
Tsuyoshi Ozawa cbacf20755 YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. Contributed by Naganarasimha G R. 
(cherry picked from commit 82eda771e0)
 2015-03-24 00:25:52 +09:00
Jian He 9c494cedaf YARN-3345. Add non-exclusive node label API. Contributed by Wangda Tan 
(cherry picked from commit e1feb4ea1a)
 2015-03-20 19:05:14 -07:00
Jian He a3a155a34e YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 
(cherry picked from commit 586348e4cb)
 2015-03-20 13:54:46 -07:00
Junping Du f40f17489c YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path. Contributed by Xuan Gong 
(cherry picked from commit d81109e588)
 2015-03-20 13:42:31 -07:00
Wangda Tan 8e142d27cb YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 
(cherry picked from commit 6bc7710ec7)
 2015-03-20 10:42:51 -07:00
Jian He 3f0c9e5fe3 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong 
(cherry picked from commit 4e886eb9cb)
 2015-03-19 22:27:57 -07:00
Devaraj K a888dac344 YARN-3357. Move TestFifoScheduler to FIFO package. Contributed by Rohith 
Sharmaks.

(cherry picked from commit 93d0f4acc8)
 2015-03-19 12:18:08 +05:30
Karthik Kambatla 05627dd96d YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) 
(cherry picked from commit 20b49224eb)
 2015-03-18 16:33:57 -07:00
Jian He 15ebacf03a YARN-3273. Improve scheduler UI to facilitate scheduling analysis and debugging. Contributed Rohith Sharmaks 
(cherry picked from commit 658097d6da)
 2015-03-17 21:31:20 -07:00
Tsuyoshi Ozawa b00b216a91 YARN-3205. FileSystemRMStateStore should disable FileSystem Cache to avoid get a Filesystem with an old configuration. Contributed by Zhihai Xu. 
(cherry picked from commit 3bc72cc16d)
 2015-03-18 11:53:40 +09:00
Jian He ab34e6975b YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks 
(cherry picked from commit 968425e9f7)
 2015-03-17 13:50:44 -07:00
Karthik Kambatla 1e77d92d62 Revert "YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)" 
This reverts commit c2b185def8.

(cherry picked from commit 32b4330456)
 2015-03-17 12:31:44 -07:00
Jian He 1c601e492f YARN-3243. CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits. Contributed by Wangda Tan. 
(cherry picked from commit 487374b7fe)
 2015-03-17 10:25:07 -07:00
Devaraj K 895588b439 YARN-3197. Confusing log generated by CapacityScheduler. Contributed by 
Varun Saxena.

(cherry picked from commit 7179f94f9d)
 2015-03-17 15:59:19 +05:30
Ravi Prakash 9f227ad696 YARN-3339. TestDockerContainerExecutor should pull a single image and not the entire centos repository. (Ravindra Kumar Naik via raviprak) 
(cherry picked from commit 56085203c4)
 2015-03-16 16:18:42 -07:00
Tsuyoshi Ozawa a20427d339 YARN-3349. Treat all exceptions as failure in TestFSRMStateStore#testFSRMStateStoreClientRetry. Contributed by Zhihai Xu. 
(cherry picked from commit 7522a643fa)
 2015-03-17 08:10:12 +09:00