Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,117 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1917 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli db1a33e956 Moving YARN-3351, YARN-3382, YARN-3472, MAPREDUCE-6238 to the 2.7.1 CHANGES.txt 
sections given the recent merge into branch-2.7.

(cherry picked from commit 2f82ae042a)
 2015-04-24 17:20:02 -07:00
Jian He 9bf09b334d YARN-2498. Respect labels in preemption policy of capacity scheduler for inter-queue preemption. Contributed by Wangda Tan 
(cherry picked from commit d497f6ea2b)
 2015-04-24 17:03:57 -07:00
Tsuyoshi Ozawa 3884948d6c YARN-3406. Display count of running containers in the RM's Web UI. Contributed by Ryu Kobayashi. 
(cherry picked from commit 4a3dabd94f)
 2015-04-25 07:17:42 +09:00
Jason Lowe bd750f160b Fix commit version for YARN-3537 
(cherry picked from commit 78fe6e57c7)
 2015-04-24 22:09:01 +00:00
Jason Lowe cf4154676b YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked. Contributed by Brahma Reddy Battula 
(cherry picked from commit 5e093f0d40)
 2015-04-24 22:04:03 +00:00
Akira Ajisaka 73ba3ebe7c YARN-3444. Fix typo capabililty. Contributed by Gabor Liptak. 
(cherry picked from commit 5ce3a77f3c)
 2015-04-25 06:09:03 +09:00
Jian He 0583c27fb1 YARN-3387. Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time. Contributed by Sandflee 
(cherry picked from commit d03dcb9635)
 2015-04-24 12:14:17 -07:00
Xuan c184466939 YARN-3511. Add errors and warnings page to ATS. Contributed by Varun Vasudev 
(cherry picked from commit eee9facbba)
 2015-04-24 09:43:35 -07:00
Xuan 8f6053ae51 YARN-3516. killing ContainerLocalizer action doesn't take effect when 
private localizer receives FETCH_FAILURE status. Contributed by zhihai
xu

(cherry picked from commit 0b3f8957a8)
 2015-04-23 16:41:52 -07:00
Vinod Kumar Vavilapalli 0ec6e7e7f5 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv) 
(cherry picked from commit f5fe35e297)
 2015-04-23 11:25:30 -07:00
Jian He 48d8d88245 YARN-3522. Fixed DistributedShell to instantiate TimeLineClient as the correct user. Contributed by Zhijie Shen 
(cherry picked from commit aa4a192feb)
 2015-04-23 11:07:54 -07:00
Wangda Tan 9692bcc2c8 YARN-3319. Implement a FairOrderingPolicy. (Craig Welch via wangda) 
(cherry picked from commit 395205444e)
 2015-04-23 10:48:00 -07:00
tgraves 1cd2fcf25d YARN-3434. Interaction between reservations and userlimit can result in significant ULF violation 
(cherry picked from commit 189a63a719)
 2015-04-23 14:49:24 +00:00
Steve Loughran 63ccb9167e YARN-2605. [RM HA] Rest api endpoints doing redirect incorrectly. (Xuan Gong via stevel) 2015-04-23 09:22:23 +01:00
Vinod Kumar Vavilapalli 04783b0402 YARN-3366. Enhanced NodeManager to support classifying/shaping outgoing network bandwidth traffic originating from YARN containers Contributed by Sidharta Seethana. 
(cherry picked from commit a100be685c)
 2015-04-22 17:36:55 -07:00
Junping Du 8de6045842 New parameter of CLI for decommissioning node gracefully in RMAdmin CLI. Contributed by Devaraj K 
(cherry picked from commit fad9d7e85b)
 2015-04-22 10:08:16 -07:00
Akira Ajisaka f469f9af2d YARN-3410. Addendum fix for compilation error. Contributed by Rohith. 
(cherry picked from commit b08908ae5e)
 2015-04-22 17:48:55 +09:00
Jian He 613a783380 YARN-3503. Expose disk utilization percentage and bad local and log dir counts in NM metrics. Contributed by Varun Vasudev 
(cherry picked from commit 674c7ef649)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
 2015-04-21 21:06:06 -07:00
Jian He dc4698bb33 YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. Contributed by Rohith Sharmaks 
(cherry picked from commit bdd90110e6)
 2015-04-21 20:34:30 -07:00
Wangda Tan 8a9599fb9e YARN-3410. YARN admin should be able to remove individual application records from RMStateStore. (Rohith Sharmaks via wangda) 2015-04-21 17:47:27 -07:00
Tsuyoshi Ozawa 0b87ae27aa YARN-3495. Confusing log generated by FairScheduler. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 105afd5477)
 2015-04-22 05:48:16 +09:00
Vinod Kumar Vavilapalli 27fc4fb998 Set the release date for 2.7.0 
(cherry picked from commit d52de61544)
 2015-04-20 20:18:01 -07:00
Wangda Tan d0ea982e64 YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) 
(cherry picked from commit 44872b76fc)
 2015-04-20 17:13:01 -07:00
Jian He 6b454aacc3 YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe 
(cherry picked from commit f967fd2f21)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-04-20 10:40:09 -07:00
Jian He 6f70f1b1af YARN-3136. Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers. Contributed by Sunil G 
(cherry picked from commit 497c86b485)
 2015-04-18 12:47:30 -07:00
Wangda Tan 3316cd4357 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763ac)
 2015-04-17 17:23:00 -07:00
Wangda Tan e7cbecddc3 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d)
 2015-04-17 17:15:16 -07:00
Jian He 9827da95bc YARN-3451. Display attempt start time and elapsed time on the web UI. Contributed by Rohith Sharmaks 
(cherry picked from commit 6779467ab6)
 2015-04-17 15:57:36 -07:00
Jian He 57eb07d34b YARN-2696. Queue sorting in CapacityScheduler should consider node label. Contributed by Wangda Tan 
(cherry picked from commit d573f09fb9)
 2015-04-17 13:37:34 -07:00
Junping Du b244701fad YARN-1402. Update related Web UI and CLI with exposing client API to check log aggregation status. Contributed by Xuan Gong. 
(cherry picked from commit 1db355a875)
 2015-04-17 13:20:27 -07:00
Jian He 42cf8e0554 YARN-3021. YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp. Contributed by Yongjun Zhang 
(cherry picked from commit bb6dde68f1)
 2015-04-16 19:44:32 -07:00
Akira Ajisaka 51c458a5c4 MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 1fa8075794)
 2015-04-16 10:06:03 -05:00
Jian He 6fed2c2a79 YARN-3354. Add node label expression in ContainerTokenIdentifier to support RM recovery. Contributed by Wangda Tan 
(cherry picked from commit 1b89a3e173)
 2015-04-15 14:03:29 -07:00
Jian He 58f99b740a YARN-3404. Display queue name on application page. Contributed by Ryu Kobayashi 
(cherry picked from commit b2e6cf607f)
 2015-04-15 13:53:24 -07:00
Tsuyoshi Ozawa e977247f3e YARN-3326. Support RESTful API for getLabelsToNodes. Contributed by Naganarasimha G R. 
(cherry picked from commit e48cedc663)
 2015-04-15 14:04:24 -05:00
Wangda Tan 89a7c98436 YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. (Craig Welch via wangda) 
(cherry picked from commit 5004e75332)
 2015-04-15 09:57:52 -07:00
Harsh J 01af29106a YARN-3462. Patches applied for YARN-2424 are inconsistent between trunk and branch-2. Contributed by Naganarasimha G R. 2015-04-15 09:41:43 +05:30
Tsuyoshi Ozawa 3daca4f040 YARN-3436. Fix URIs in documantion of YARN web service REST APIs. Contributed by Bibin A Chundatt. 
(cherry picked from commit 05007b45e5)
 2015-04-14 14:14:17 -05:00
Jian He 9ebbf1bfce YARN-3361. CapacityScheduler side changes to support non-exclusive node labels. Contributed by Wangda Tan 
(cherry picked from commit 0fefda645b)
 2015-04-14 11:46:35 -07:00
Jian He 81bbee6852 YARN-3266. RMContext#inactiveNodes should have NodeId as map key. Contributed by Chengbing Liu 
(cherry picked from commit b46ee1e7a3)
 2015-04-14 11:31:42 -07:00
Junping Du 7c072bf092 YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. 
(cherry picked from commit 838b06ac87)
 2015-04-13 18:37:39 -07:00
Junping Du a64b8915de YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. 
(cherry picked from commit a77d628339)
 2015-04-13 17:41:02 -07:00
Jian He 1afea8a15a YARN-3394. Enrich WebApplication proxy documentation. Contributed by Naganarasimha G R 
(cherry picked from commit 7b4671490a)
 2015-04-13 17:05:58 -07:00
Jian He d58f5c8894 YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. Contributed by Rohith Sharmaks 
(cherry picked from commit a1afbc48b5)
 2015-04-13 14:07:51 -07:00
Karthik Kambatla fa12700379 Fix up author name to Jun Gong in CHANGES.txt for YARN-3469 
(cherry picked from commit 5cda6fffd3)
 2015-04-13 13:52:44 -07:00
Jian He 32b01263d3 YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root queue metrics. Contributed by Rohit Agarwal 
(cherry picked from commit 944a16579f)
 2015-04-13 10:54:15 -07:00
Karthik Kambatla ad99d268e5 YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. (Jun Hong via kasha) 
(cherry picked from commit e516706b89)
 2015-04-10 11:21:30 -07:00
Jian He d3daf9665c YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev 2015-04-10 10:40:31 -07:00
Junping Du 0891de0712 YARN-1376. NM need to notify the log aggregation status to RM through Node heartbeat. Contributed by Xuan Gong. 
(cherry picked from commit 92431c9617)
 2015-04-10 08:58:07 -07:00
Xuan f5b49160d9 YARN-3293. Track and display capacity scheduler health metrics in web 
UI. Contributed by Varun Vasudev

(cherry picked from commit afa5d4715a)
 2015-04-09 23:40:02 -07:00
Vinod Kumar Vavilapalli aec1c5ec7d Adding release 2.7.1 to CHANGES.txt 
(cherry picked from commit 623fd46c1e)
 2015-04-09 13:56:14 -07:00
Vinod Kumar Vavilapalli 1ff3fd33ed YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e)
 2015-04-09 13:10:06 -07:00
Wangda Tan 73d9e4c77b YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column. (Jason Lowe via wangda) 
(cherry picked from commit 1885141e90)
 2015-04-09 10:36:10 -07:00
Karthik Kambatla 53e0bf5c17 YARN-3465. Use LinkedHashMap to preserve order of resource requests. (Zhihai Xu via kasha) 
(cherry picked from commit 6495940eae)
 2015-04-09 00:10:23 -07:00
Hitesh Shah 55b794e7fa YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 
(cherry picked from commit 265ed1fe80)
 2015-04-08 14:14:06 -07:00
Wangda Tan 5f59e621ba YARN-2901 addendum: Fixed findbugs warning caused by previously patch 
(cherry picked from commit ba9ee22ca4)
 2015-04-08 11:02:54 -07:00
Wangda Tan 97244b143c YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. (Varun Vasudev via wangda) 
(cherry picked from commit 7af086a515)
 2015-04-08 10:58:38 -07:00
Tsuyoshi Ozawa ac32fa187c YARN-3457. NPE when NodeManager.serviceInit fails and stopRecoveryStore called. Contributed by Bibin A Chundatt. 
(cherry picked from commit dd852f5b8c)
 2015-04-08 15:56:33 +09:00
Robert Kanter b584ce6c0a YARN-3429. Fix incorrect CHANGES.txt 
(cherry picked from commit 5b8a3ae366)
 2015-04-07 16:16:08 -07:00
Xuan 0522d6970d YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for 
a fixed time period. Contributed by Varun Vasudev

(cherry picked from commit d27e9241e8)
 2015-04-07 09:54:29 -07:00
Xuan 7941cc554b YARN-3110. Few issues in ApplicationHistory web ui. Contributed by Naganarasimha G R 
(cherry picked from commit 19a4feaf6f)
 2015-04-07 08:24:09 -07:00
Zhijie Shen 735e4f15ab Move YARN-3273 from 2.8 to 2.7. 
(cherry picked from commit 5e9f4099de4b949c838a6224442f335c67d80936)
 2015-04-06 14:15:23 -07:00
Robert Kanter 354d2c75cf YARN-2429. TestAMRMTokens.testTokenExpiry fails Intermittently with error message:Invalid AMRMToken (zxu via rkanter) 
(cherry picked from commit 99b08a748e)
 2015-04-06 14:11:50 -07:00
Tsuyoshi Ozawa d2b2d76cce TestFairScheduler.testContinuousScheduling fails Intermittently. Contributed by Zhihai Xu. 
(cherry picked from commit 53959e69f7)
 2015-04-06 20:19:47 +09:00
Xuan 1eb6d74967 YARN-3435. AM container to be allocated Appattempt AM container shown as 
null. Contributed by Bibin A Chundatt

(cherry picked from commit 96d72118f5)
 2015-04-05 00:43:22 -07:00
Wangda Tan 405aead879 YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) 
(cherry picked from commit bad070fe15)
 2015-04-02 17:23:44 -07:00
Vinod Kumar Vavilapalli d8e17c58bc YARN-3365. Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control. Contributed by Sidharta Seethana. 
(cherry picked from commit b21c72777a)
 2015-04-02 16:55:00 -07:00
Sandy Ryza 6286cfd711 YARN-3415. Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue (Zhihai Xu via Sandy Ryza) 2015-04-02 13:57:53 -07:00
Zhijie Shen 6d0bebc912 YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong. 
(cherry picked from commit 8366a36ad3)
 2015-04-01 13:49:00 -07:00
Wangda Tan 865be70b02 YARN-3425. NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed. (Bibin A Chundatt via wangda) 
(cherry picked from commit 492239424a)
 2015-04-01 10:15:47 -07:00
Xuan 5e4d457674 YARN-3248. Correct fix version from branch-2.7 to branch-2.8 in the change log. 
(cherry picked from commit 2e79f1c212)
 2015-04-01 04:33:32 -07:00
Xuan e26b6e55e9 YARN-3248. Display count of nodes blacklisted by apps in the web UI. 
Contributed by Varun Vasudev

(cherry picked from commit 4728bdfa15)
 2015-04-01 04:20:37 -07:00
Tsuyoshi Ozawa 80755edb7a YARN-3424. Change logs for ContainerMonitorImpl's resourse monitoring from info to debug. Contributed by Anubhav Dhoot. 
(cherry picked from commit c69ba81497)
 2015-04-01 17:44:40 +09:00
Karthik Kambatla 93b29d962d YARN-3428. Debug log resources to be localized for a container. (kasha) 
(cherry picked from commit 2daa478a64)
 2015-03-31 17:35:11 -07:00
Vinod Kumar Vavilapalli c980e34bc5 YARN-3304. Addendum patch. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. (Junping Du and Karthik Kambatla via vinodkv) 
(cherry picked from commit 7610925e90)
 2015-03-31 17:28:48 -07:00
Karthik Kambatla 85c420a4d4 YARN-3412. RM tests should use MockRM where possible. (kasha) 
(cherry picked from commit 79f7f2aabf)
 2015-03-31 09:22:16 -07:00
Tsuyoshi Ozawa 99b8255693 YARN-3258. FairScheduler: Need to add more logging to investigate allocations. Contributed by Anubhav Dhoot. 
(cherry picked from commit b5a22e9838)
 2015-03-31 17:43:07 +09:00
Wangda Tan cba4ed1678 YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) 
(cherry picked from commit 2a945d24f7)
 2015-03-30 12:05:54 -07:00
Vinod Kumar Vavilapalli c5bc48946d YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. Contributed by Junping Du and Karthik Kambatla. 
(cherry picked from commit c358368f51)
 2015-03-30 10:11:12 -07:00
Ravi Prakash b1b4951452 YARN-3288. Document and fix indentation in the DockerContainerExecutor code 
(cherry picked from commit e0ccea33c9)
 2015-03-28 08:01:26 -07:00
Robert Kanter 6f0e864d61 YARN-3400. [JDK 8] Build Failure due to unreported exceptions in RPCUtil (rkanter) 
(cherry picked from commit 87130bf6b2)
 2015-03-26 11:00:37 -07:00
Haohui Mai 17f17dac4e HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. 2015-03-25 11:13:04 -07:00
Karthik Kambatla 2158cd2bef YARN-3397. yarn rmadmin should skip -failover. (J.Andreina via kasha) 
(cherry picked from commit c906a1de72)
 2015-03-25 07:42:45 -07:00
Xuan 0c4fa872be YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. 
Contributed by Varun Saxena

(cherry picked from commit e556198e71)
 2015-03-25 04:51:55 -07:00
Wangda Tan 29629020c1 YARN-3383. AdminService should use warn instead of info to log exception when operation fails. (Li Lu via wangda) 
(cherry picked from commit 97a7277a2d)
 2015-03-24 10:39:45 -07:00
Harsh J e7702ac6a1 YARN-1880. Cleanup TestApplicationClientProtocolOnHA. Contributed by ozawa. 
(cherry picked from commit 8a9bbf548dad82598354c24fcd4dc80a715da7ac)
 2015-03-24 11:58:16 +05:30
Xuan cbdcdfad6d YARN-3393. Getting application(s) goes wrong when app finishes before 
starting the attempt. Contributed by Zhijie Shen

(cherry picked from commit 9fae455e26)
 2015-03-23 20:34:29 -07:00
Karthik Kambatla 4e0c48703e YARN-2868. FairScheduler: Metric for latency to allocate first container for an application. (Ray Chiang via kasha) 
(cherry picked from commit 972f1f1ab9)
 2015-03-23 14:10:23 -07:00
Karthik Kambatla 75591e4131 YARN-3241. FairScheduler handles invalid queue names inconsistently. (Zhihai Xu via kasha) 
(cherry picked from commit 2bc097cd14)
 2015-03-23 13:24:22 -07:00
cnauroth 342c525eaa YARN-3336. FileSystem memory leak in DelegationTokenRenewer. 
(cherry picked from commit 6ca1f12024)
 2015-03-23 10:46:06 -07:00
Tsuyoshi Ozawa cbacf20755 YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. Contributed by Naganarasimha G R. 
(cherry picked from commit 82eda771e0)
 2015-03-24 00:25:52 +09:00
Jian He 9c494cedaf YARN-3345. Add non-exclusive node label API. Contributed by Wangda Tan 
(cherry picked from commit e1feb4ea1a)
 2015-03-20 19:05:14 -07:00
Junping Du 9aedb3d688 YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 7f1e2f9969)
 2015-03-20 18:22:23 -07:00
Jian He a3a155a34e YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 
(cherry picked from commit 586348e4cb)
 2015-03-20 13:54:46 -07:00
Junping Du f40f17489c YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path. Contributed by Xuan Gong 
(cherry picked from commit d81109e588)
 2015-03-20 13:42:31 -07:00
Wangda Tan 8e142d27cb YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 
(cherry picked from commit 6bc7710ec7)
 2015-03-20 10:42:51 -07:00
Xuan 7990e1d80d YARN-2777. Mark the end of individual log in aggregated log. Contributed 
by Varun Saxena

(cherry picked from commit 1a4b528691)
 2015-03-20 10:25:26 -07:00
Jian He 3f0c9e5fe3 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong 
(cherry picked from commit 4e886eb9cb)
 2015-03-19 22:27:57 -07:00
Devaraj K a888dac344 YARN-3357. Move TestFifoScheduler to FIFO package. Contributed by Rohith 
Sharmaks.

(cherry picked from commit 93d0f4acc8)
 2015-03-19 12:18:08 +05:30
Karthik Kambatla 05627dd96d YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) 
(cherry picked from commit 20b49224eb)
 2015-03-18 16:33:57 -07:00