Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,075 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1777 Commits

Author SHA1 Message Date
Junping Du 8de6045842 New parameter of CLI for decommissioning node gracefully in RMAdmin CLI. Contributed by Devaraj K 
(cherry picked from commit fad9d7e85b)
 2015-04-22 10:08:16 -07:00
Akira Ajisaka f469f9af2d YARN-3410. Addendum fix for compilation error. Contributed by Rohith. 
(cherry picked from commit b08908ae5e)
 2015-04-22 17:48:55 +09:00
Jian He 613a783380 YARN-3503. Expose disk utilization percentage and bad local and log dir counts in NM metrics. Contributed by Varun Vasudev 
(cherry picked from commit 674c7ef649)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
 2015-04-21 21:06:06 -07:00
Jian He dc4698bb33 YARN-3494. Expose AM resource limit and usage in CS QueueMetrics. Contributed by Rohith Sharmaks 
(cherry picked from commit bdd90110e6)
 2015-04-21 20:34:30 -07:00
Wangda Tan 8a9599fb9e YARN-3410. YARN admin should be able to remove individual application records from RMStateStore. (Rohith Sharmaks via wangda) 2015-04-21 17:47:27 -07:00
Tsuyoshi Ozawa 0b87ae27aa YARN-3495. Confusing log generated by FairScheduler. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 105afd5477)
 2015-04-22 05:48:16 +09:00
Wangda Tan d0ea982e64 YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) 
(cherry picked from commit 44872b76fc)
 2015-04-20 17:13:01 -07:00
Jian He 6b454aacc3 YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. Contributed by Jason Lowe 
(cherry picked from commit f967fd2f21)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-04-20 10:40:09 -07:00
Jian He 6f70f1b1af YARN-3136. Fixed a synchronization problem of AbstractYarnScheduler#getTransferredContainers. Contributed by Sunil G 
(cherry picked from commit 497c86b485)
 2015-04-18 12:47:30 -07:00
Wangda Tan 3316cd4357 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763ac)
 2015-04-17 17:23:00 -07:00
Wangda Tan e7cbecddc3 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d)
 2015-04-17 17:15:16 -07:00
Jian He 9827da95bc YARN-3451. Display attempt start time and elapsed time on the web UI. Contributed by Rohith Sharmaks 
(cherry picked from commit 6779467ab6)
 2015-04-17 15:57:36 -07:00
Jian He 57eb07d34b YARN-2696. Queue sorting in CapacityScheduler should consider node label. Contributed by Wangda Tan 
(cherry picked from commit d573f09fb9)
 2015-04-17 13:37:34 -07:00
Junping Du b244701fad YARN-1402. Update related Web UI and CLI with exposing client API to check log aggregation status. Contributed by Xuan Gong. 
(cherry picked from commit 1db355a875)
 2015-04-17 13:20:27 -07:00
Jian He 42cf8e0554 YARN-3021. YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp. Contributed by Yongjun Zhang 
(cherry picked from commit bb6dde68f1)
 2015-04-16 19:44:32 -07:00
Akira Ajisaka 51c458a5c4 MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 1fa8075794)
 2015-04-16 10:06:03 -05:00
Jian He 6fed2c2a79 YARN-3354. Add node label expression in ContainerTokenIdentifier to support RM recovery. Contributed by Wangda Tan 
(cherry picked from commit 1b89a3e173)
 2015-04-15 14:03:29 -07:00
Jian He 58f99b740a YARN-3404. Display queue name on application page. Contributed by Ryu Kobayashi 
(cherry picked from commit b2e6cf607f)
 2015-04-15 13:53:24 -07:00
Tsuyoshi Ozawa e977247f3e YARN-3326. Support RESTful API for getLabelsToNodes. Contributed by Naganarasimha G R. 
(cherry picked from commit e48cedc663)
 2015-04-15 14:04:24 -05:00
Wangda Tan 89a7c98436 YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. (Craig Welch via wangda) 
(cherry picked from commit 5004e75332)
 2015-04-15 09:57:52 -07:00
Harsh J 01af29106a YARN-3462. Patches applied for YARN-2424 are inconsistent between trunk and branch-2. Contributed by Naganarasimha G R. 2015-04-15 09:41:43 +05:30
Tsuyoshi Ozawa 3daca4f040 YARN-3436. Fix URIs in documantion of YARN web service REST APIs. Contributed by Bibin A Chundatt. 
(cherry picked from commit 05007b45e5)
 2015-04-14 14:14:17 -05:00
Jian He 9ebbf1bfce YARN-3361. CapacityScheduler side changes to support non-exclusive node labels. Contributed by Wangda Tan 
(cherry picked from commit 0fefda645b)
 2015-04-14 11:46:35 -07:00
Jian He 81bbee6852 YARN-3266. RMContext#inactiveNodes should have NodeId as map key. Contributed by Chengbing Liu 
(cherry picked from commit b46ee1e7a3)
 2015-04-14 11:31:42 -07:00
Junping Du 7c072bf092 YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. 
(cherry picked from commit 838b06ac87)
 2015-04-13 18:37:39 -07:00
Junping Du a64b8915de YARN-3347. Improve YARN log command to get AMContainer logs as well as running containers logs. Contributed by Xuan Gong. 
(cherry picked from commit a77d628339)
 2015-04-13 17:41:02 -07:00
Jian He 1afea8a15a YARN-3394. Enrich WebApplication proxy documentation. Contributed by Naganarasimha G R 
(cherry picked from commit 7b4671490a)
 2015-04-13 17:05:58 -07:00
Jian He d58f5c8894 YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. Contributed by Rohith Sharmaks 
(cherry picked from commit a1afbc48b5)
 2015-04-13 14:07:51 -07:00
Jian He 32b01263d3 YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root queue metrics. Contributed by Rohit Agarwal 
(cherry picked from commit 944a16579f)
 2015-04-13 10:54:15 -07:00
Karthik Kambatla ad99d268e5 YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. (Jun Hong via kasha) 
(cherry picked from commit e516706b89)
 2015-04-10 11:21:30 -07:00
Jian He d3daf9665c YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev 2015-04-10 10:40:31 -07:00
Junping Du 0891de0712 YARN-1376. NM need to notify the log aggregation status to RM through Node heartbeat. Contributed by Xuan Gong. 
(cherry picked from commit 92431c9617)
 2015-04-10 08:58:07 -07:00
Xuan f5b49160d9 YARN-3293. Track and display capacity scheduler health metrics in web 
UI. Contributed by Varun Vasudev

(cherry picked from commit afa5d4715a)
 2015-04-09 23:40:02 -07:00
Vinod Kumar Vavilapalli 1ff3fd33ed YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e)
 2015-04-09 13:10:06 -07:00
Wangda Tan 73d9e4c77b YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column. (Jason Lowe via wangda) 
(cherry picked from commit 1885141e90)
 2015-04-09 10:36:10 -07:00
Karthik Kambatla 53e0bf5c17 YARN-3465. Use LinkedHashMap to preserve order of resource requests. (Zhihai Xu via kasha) 
(cherry picked from commit 6495940eae)
 2015-04-09 00:10:23 -07:00
Hitesh Shah 55b794e7fa YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 
(cherry picked from commit 265ed1fe80)
 2015-04-08 14:14:06 -07:00
Wangda Tan 5f59e621ba YARN-2901 addendum: Fixed findbugs warning caused by previously patch 
(cherry picked from commit ba9ee22ca4)
 2015-04-08 11:02:54 -07:00
Wangda Tan 97244b143c YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. (Varun Vasudev via wangda) 
(cherry picked from commit 7af086a515)
 2015-04-08 10:58:38 -07:00
Tsuyoshi Ozawa ac32fa187c YARN-3457. NPE when NodeManager.serviceInit fails and stopRecoveryStore called. Contributed by Bibin A Chundatt. 
(cherry picked from commit dd852f5b8c)
 2015-04-08 15:56:33 +09:00
Xuan 0522d6970d YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for 
a fixed time period. Contributed by Varun Vasudev

(cherry picked from commit d27e9241e8)
 2015-04-07 09:54:29 -07:00
Xuan 7941cc554b YARN-3110. Few issues in ApplicationHistory web ui. Contributed by Naganarasimha G R 
(cherry picked from commit 19a4feaf6f)
 2015-04-07 08:24:09 -07:00
Robert Kanter 354d2c75cf YARN-2429. TestAMRMTokens.testTokenExpiry fails Intermittently with error message:Invalid AMRMToken (zxu via rkanter) 
(cherry picked from commit 99b08a748e)
 2015-04-06 14:11:50 -07:00
Tsuyoshi Ozawa d2b2d76cce TestFairScheduler.testContinuousScheduling fails Intermittently. Contributed by Zhihai Xu. 
(cherry picked from commit 53959e69f7)
 2015-04-06 20:19:47 +09:00
Xuan 1eb6d74967 YARN-3435. AM container to be allocated Appattempt AM container shown as 
null. Contributed by Bibin A Chundatt

(cherry picked from commit 96d72118f5)
 2015-04-05 00:43:22 -07:00
Wangda Tan 405aead879 YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda) 
(cherry picked from commit bad070fe15)
 2015-04-02 17:23:44 -07:00
Vinod Kumar Vavilapalli d8e17c58bc YARN-3365. Enhanced NodeManager to support using the 'tc' tool via container-executor for outbound network traffic control. Contributed by Sidharta Seethana. 
(cherry picked from commit b21c72777a)
 2015-04-02 16:55:00 -07:00
Sandy Ryza 6286cfd711 YARN-3415. Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue (Zhihai Xu via Sandy Ryza) 2015-04-02 13:57:53 -07:00
Zhijie Shen 6d0bebc912 YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong. 
(cherry picked from commit 8366a36ad3)
 2015-04-01 13:49:00 -07:00
Wangda Tan 865be70b02 YARN-3425. NPE from RMNodeLabelsManager.serviceStop when NodeLabelsManager.serviceInit failed. (Bibin A Chundatt via wangda) 
(cherry picked from commit 492239424a)
 2015-04-01 10:15:47 -07:00
Xuan e26b6e55e9 YARN-3248. Display count of nodes blacklisted by apps in the web UI. 
Contributed by Varun Vasudev

(cherry picked from commit 4728bdfa15)
 2015-04-01 04:20:37 -07:00
Tsuyoshi Ozawa 80755edb7a YARN-3424. Change logs for ContainerMonitorImpl's resourse monitoring from info to debug. Contributed by Anubhav Dhoot. 
(cherry picked from commit c69ba81497)
 2015-04-01 17:44:40 +09:00
Karthik Kambatla 93b29d962d YARN-3428. Debug log resources to be localized for a container. (kasha) 
(cherry picked from commit 2daa478a64)
 2015-03-31 17:35:11 -07:00
Vinod Kumar Vavilapalli c980e34bc5 YARN-3304. Addendum patch. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. (Junping Du and Karthik Kambatla via vinodkv) 
(cherry picked from commit 7610925e90)
 2015-03-31 17:28:48 -07:00
Karthik Kambatla 85c420a4d4 YARN-3412. RM tests should use MockRM where possible. (kasha) 
(cherry picked from commit 79f7f2aabf)
 2015-03-31 09:22:16 -07:00
Tsuyoshi Ozawa 99b8255693 YARN-3258. FairScheduler: Need to add more logging to investigate allocations. Contributed by Anubhav Dhoot. 
(cherry picked from commit b5a22e9838)
 2015-03-31 17:43:07 +09:00
Wangda Tan cba4ed1678 YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda) 
(cherry picked from commit 2a945d24f7)
 2015-03-30 12:05:54 -07:00
Vinod Kumar Vavilapalli c5bc48946d YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. Contributed by Junping Du and Karthik Kambatla. 
(cherry picked from commit c358368f51)
 2015-03-30 10:11:12 -07:00
Ravi Prakash b1b4951452 YARN-3288. Document and fix indentation in the DockerContainerExecutor code 
(cherry picked from commit e0ccea33c9)
 2015-03-28 08:01:26 -07:00
Robert Kanter 6f0e864d61 YARN-3400. [JDK 8] Build Failure due to unreported exceptions in RPCUtil (rkanter) 
(cherry picked from commit 87130bf6b2)
 2015-03-26 11:00:37 -07:00
Haohui Mai 17f17dac4e HADOOP-10670. Allow AuthenticationFilters to load secret from signature secret files. Contributed by Kai Zheng. 2015-03-25 11:13:04 -07:00
Karthik Kambatla 2158cd2bef YARN-3397. yarn rmadmin should skip -failover. (J.Andreina via kasha) 
(cherry picked from commit c906a1de72)
 2015-03-25 07:42:45 -07:00
Xuan 0c4fa872be YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. 
Contributed by Varun Saxena

(cherry picked from commit e556198e71)
 2015-03-25 04:51:55 -07:00
Wangda Tan 29629020c1 YARN-3383. AdminService should use warn instead of info to log exception when operation fails. (Li Lu via wangda) 
(cherry picked from commit 97a7277a2d)
 2015-03-24 10:39:45 -07:00
Harsh J e7702ac6a1 YARN-1880. Cleanup TestApplicationClientProtocolOnHA. Contributed by ozawa. 
(cherry picked from commit 8a9bbf548dad82598354c24fcd4dc80a715da7ac)
 2015-03-24 11:58:16 +05:30
Xuan cbdcdfad6d YARN-3393. Getting application(s) goes wrong when app finishes before 
starting the attempt. Contributed by Zhijie Shen

(cherry picked from commit 9fae455e26)
 2015-03-23 20:34:29 -07:00
Karthik Kambatla 4e0c48703e YARN-2868. FairScheduler: Metric for latency to allocate first container for an application. (Ray Chiang via kasha) 
(cherry picked from commit 972f1f1ab9)
 2015-03-23 14:10:23 -07:00
Karthik Kambatla 75591e4131 YARN-3241. FairScheduler handles invalid queue names inconsistently. (Zhihai Xu via kasha) 
(cherry picked from commit 2bc097cd14)
 2015-03-23 13:24:22 -07:00
cnauroth 342c525eaa YARN-3336. FileSystem memory leak in DelegationTokenRenewer. 
(cherry picked from commit 6ca1f12024)
 2015-03-23 10:46:06 -07:00
Tsuyoshi Ozawa cbacf20755 YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. Contributed by Naganarasimha G R. 
(cherry picked from commit 82eda771e0)
 2015-03-24 00:25:52 +09:00
Jian He 9c494cedaf YARN-3345. Add non-exclusive node label API. Contributed by Wangda Tan 
(cherry picked from commit e1feb4ea1a)
 2015-03-20 19:05:14 -07:00
Junping Du 9aedb3d688 YARN-3350. YARN RackResolver spams logs with messages at info level. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 7f1e2f9969)
 2015-03-20 18:22:23 -07:00
Jian He a3a155a34e YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 
(cherry picked from commit 586348e4cb)
 2015-03-20 13:54:46 -07:00
Junping Du f40f17489c YARN-3269. Yarn.nodemanager.remote-app-log-dir could not be configured to fully qualified path. Contributed by Xuan Gong 
(cherry picked from commit d81109e588)
 2015-03-20 13:42:31 -07:00
Wangda Tan 8e142d27cb YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 
(cherry picked from commit 6bc7710ec7)
 2015-03-20 10:42:51 -07:00
Xuan 7990e1d80d YARN-2777. Mark the end of individual log in aggregated log. Contributed 
by Varun Saxena

(cherry picked from commit 1a4b528691)
 2015-03-20 10:25:26 -07:00
Jian He 3f0c9e5fe3 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong 
(cherry picked from commit 4e886eb9cb)
 2015-03-19 22:27:57 -07:00
Devaraj K a888dac344 YARN-3357. Move TestFifoScheduler to FIFO package. Contributed by Rohith 
Sharmaks.

(cherry picked from commit 93d0f4acc8)
 2015-03-19 12:18:08 +05:30
Karthik Kambatla 05627dd96d YARN-3351. AppMaster tracking URL is broken in HA. (Anubhav Dhoot via kasha) 
(cherry picked from commit 20b49224eb)
 2015-03-18 16:33:57 -07:00
Jian He 15ebacf03a YARN-3273. Improve scheduler UI to facilitate scheduling analysis and debugging. Contributed Rohith Sharmaks 
(cherry picked from commit 658097d6da)
 2015-03-17 21:31:20 -07:00
Tsuyoshi Ozawa b00b216a91 YARN-3205. FileSystemRMStateStore should disable FileSystem Cache to avoid get a Filesystem with an old configuration. Contributed by Zhihai Xu. 
(cherry picked from commit 3bc72cc16d)
 2015-03-18 11:53:40 +09:00
Jian He ab34e6975b YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks 
(cherry picked from commit 968425e9f7)
 2015-03-17 13:50:44 -07:00
Karthik Kambatla 1e77d92d62 Revert "YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)" 
This reverts commit c2b185def8.

(cherry picked from commit 32b4330456)
 2015-03-17 12:31:44 -07:00
Jian He 1c601e492f YARN-3243. CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits. Contributed by Wangda Tan. 
(cherry picked from commit 487374b7fe)
 2015-03-17 10:25:07 -07:00
Devaraj K 895588b439 YARN-3197. Confusing log generated by CapacityScheduler. Contributed by 
Varun Saxena.

(cherry picked from commit 7179f94f9d)
 2015-03-17 15:59:19 +05:30
Ravi Prakash 9f227ad696 YARN-3339. TestDockerContainerExecutor should pull a single image and not the entire centos repository. (Ravindra Kumar Naik via raviprak) 
(cherry picked from commit 56085203c4)
 2015-03-16 16:18:42 -07:00
Tsuyoshi Ozawa a20427d339 YARN-3349. Treat all exceptions as failure in TestFSRMStateStore#testFSRMStateStoreClientRetry. Contributed by Zhihai Xu. 
(cherry picked from commit 7522a643fa)
 2015-03-17 08:10:12 +09:00
Harsh J 2684b0b440 MAPREDUCE-4414. Add main methods to JobConf and YarnConfiguration, for debug purposes. Contributed by Plamen Jeliazkov. 
(cherry picked from commit 571f75b468a4412be2178466cb5d3888b05cd076)
 2015-03-17 01:02:21 +05:30
Zhijie Shen fbe811d904 YARN-2854. Updated the documentation of the timeline service and the generic history service. Contributed by Naganarasimha G R. 2015-03-16 10:52:32 -07:00
Tsuyoshi Ozawa d6e3ef2506 YARN-1453. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA, Andrew Purtell, and Allen Wittenauer. 
(cherry picked from commit 3da9a97cfb)
 2015-03-16 23:20:17 +09:00
Xuan b6eb36dbdc YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't 
work in ATS / RM web ui. Contributed by Naganarasimha G R

(cherry picked from commit 3ff1ba2a7b)
 2015-03-15 20:27:44 -07:00
Jonathan Eagles 44aedad5dd YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 
(cherry picked from commit 8180e676ab)
 2015-03-13 12:05:21 -05:00
Vinod Kumar Vavilapalli 53aa3a4d1f YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. Contributed by Xuan Gong. 
(cherry picked from commit 863079bb87)
 2015-03-12 13:33:42 -07:00
Zhijie Shen 426535007b YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM web page. Contributed by Xuan Gong. 
(cherry picked from commit 85f6d67fa7)
 2015-03-11 19:39:46 -07:00
Tsuyoshi Ozawa d29cf296cb YARN-3295. Fix documentation nits found in markdown conversion. Contributed by Masatake Iwasaki. 
(cherry picked from commit 30c428a858)
 2015-03-11 15:02:45 +09:00
Jian He 2690c72522 YARN-3187. Documentation of Capacity Scheduler Queue mapping based on user or group. Contributed by Gururaj Shetty 
(cherry picked from commit a380643d20)
 2015-03-10 11:16:51 -07:00
Jian He 4021451746 YARN-3300. Outstanding_resource_requests table should not be shown in AHS. Contributed by Xuan Gong 
(cherry picked from commit c3003eba6f)
 2015-03-09 20:47:33 -07:00
Zhijie Shen a94d23762e YARN-3287. Made TimelineClient put methods do as the correct login context. Contributed by Daryn Sharp and Jonathan Eagles. 
(cherry picked from commit d6e05c5ee2)
 2015-03-09 13:55:51 -07:00
Tsuyoshi Ozawa b46f9e72db HADOOP-11602. Backport 'Fix toUpperCase/toLowerCase to use Locale.ENGLISH.' (ozawa) 2015-03-09 19:53:55 +09:00
Vinod Kumar Vavilapalli d9416317a9 Preparing for 2.8.0 development. Changing SNAPSHOT version to be 2.8 on branch-2. 2015-03-08 20:32:51 -07:00