Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,941 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1861 Commits

Author SHA1 Message Date
Robert Kanter e5059b91bb YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter) 
(cherry picked from commit ae7bf31fe1)
 2015-01-12 10:48:45 -08:00
Jian He 36b3dcaab2 YARN-3014. Replaces labels on a host should update all NM's labels on that host. Contributed by Wangda Tan 
(cherry picked from commit a260406268)
 2015-01-09 17:50:18 -08:00
Jian He 68709cb9be YARN-2956. Added missing links in YARN documentation. Contributed by Masatake Iwasaki 
(cherry picked from commit 27a489cba6)
 2015-01-09 17:47:00 -08:00
cnauroth 9a7c763b3f HADOOP-11464. Reinstate support for launching Hadoop processes on Windows using Cygwin. Contributed by Chris Nauroth. 2015-01-09 15:01:33 -08:00
Jian He e7e6173049 YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed container statuses on heartbeat. Contributed by Chengbing Liu 
(cherry picked from commit cc2a745f7e)
 2015-01-08 11:28:24 -08:00
Zhijie Shen d2fbba790a YARN-2996. Improved synchronization and I/O operations of FS- and Mem- RMStateStore. Contributed by Yi Liu. 
(cherry picked from commit dc2eaa26b2)
 2015-01-08 09:49:27 -08:00
Jian He 9eb7611e81 Moved YARN-2880 to improvement section in CHANGES.txt 
(cherry picked from commit ef237bd52f)
 2015-01-07 16:21:12 -08:00
Jian He fe664b8b8d YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on getProto method. Contributed by Varun Saxena 
(cherry picked from commit 2638f4d0f0)
 2015-01-07 16:15:03 -08:00
Jian He d3b26454e4 YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. Contributed by Yi Liu 
(cherry picked from commit e13a484a2b)
 2015-01-07 14:12:54 -08:00
Jian He 492717cbf5 YARN-2230. Fixed few configs description in yarn-default.xml. Contributed by Vijay Bhat 
(cherry picked from commit fe8d2bd741)
 2015-01-07 10:57:41 -08:00
Zhijie Shen 8a81312062 YARN-2427. Added the API of moving apps between queues in RM web services. Contributed by Varun Vasudev. 
(cherry picked from commit 60103fca04)
 2015-01-06 14:39:44 -08:00
Jian He c61e8a7bfa YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena 
(cherry picked from commit dd57c2047b)
 2015-01-06 14:16:43 -08:00
Karthik Kambatla 7adffad2bb YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit 0c4b112677)
 2015-01-06 04:42:55 +05:30
Karthik Kambatla d070597a0e YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the scheduler page. (Ashwin Shankar and Wei Yan via kasha) 
(cherry picked from commit 270a271f53)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-01-06 03:28:48 +05:30
Zhijie Shen c6cf748985 YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. Contributed by Varun Saxena. 
(cherry picked from commit 562a701945)
 2015-01-05 13:35:09 -08:00
Tsuyoshi Ozawa c116743bdd YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed by Rohith Sharmaks. 
(cherry picked from commit ddc5be48fc)
 2015-01-05 00:11:01 +09:00
Zhijie Shen efe6260762 YARN-2991. Fixed DrainDispatcher to reuse the draining code path in AsyncDispatcher. Contributed by Rohith Sharmaks. 
(cherry picked from commit 947578c1c1)
 2015-01-02 11:10:46 -08:00
Karthik Kambatla 798ab51289 YARN-2998. Abstract out scheduler independent PlanFollower components. (Anubhav Dhoot via kasha) 
(cherry picked from commit e7257acd8a)
 2014-12-30 19:55:50 -08:00
Jian He cde5bfe3ec YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and app ACLs. Contributed by Varun Saxena 
(cherry picked from commit e2351c7ae2)
 2014-12-30 17:16:05 -08:00
Jian He 143e48c25a YARN-2493. Added node-labels page on RM web UI. Contributed by Wangda Tan 
(cherry picked from commit b7442bf92e)
 2014-12-30 16:49:59 -08:00
Zhijie Shen e5cb52ec1f YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. Contributed by Varun Saxena. 
(cherry picked from commit 241d3b3a50)
 2014-12-29 10:07:54 -08:00
Jian He ca0349b87a YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla 
(cherry picked from commit 1454efe5d4)
 2014-12-26 18:00:32 -08:00
Junping Du fc3e810a19 YARN-2993. Several fixes (missing acl check, error log msg ...) and some refinement in AdminService. (Contributed by Yi Liu) 
(cherry picked from commit 40ee4bff65)
 2014-12-26 07:01:44 -08:00
Tsuyoshi Ozawa 4ca49c8634 YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) 2014-12-25 13:09:51 +09:00
Jian He 60530a6c4f YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith Sharmaks 
(cherry picked from commit 4f18018b7a)
 2014-12-23 22:15:28 -08:00
Zhijie Shen f02bd6683a YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. 
(cherry picked from commit 41a548a916)
 2014-12-23 20:35:32 -08:00
Junping Du 361fab14be YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) 
(cherry picked from commit d468c9aaf1)
 2014-12-23 20:06:53 -08:00
Jian He e3a54b1b8b YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen 
(cherry picked from commit 149512a837)
 2014-12-23 18:26:37 -08:00
Junping Du ae3faab13e YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command. (Contributed by Varun Saxena) 
(cherry picked from commit e1ee0d45ea)
 2014-12-23 18:24:45 -08:00
Jian He 8b398a66ca YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharmaks 
(cherry picked from commit 0d89859b51)
 2014-12-22 21:55:06 -08:00
Jian He 411836b74c YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 
(cherry picked from commit fdf042dfff)
 2014-12-22 16:52:15 -08:00
Junping Du 6f252f7f36 YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) 
(cherry picked from commit a696fbb001)
 2014-12-22 03:08:25 -08:00
Tsuyoshi Ozawa d5d13bbe0f CHANGES.txt: move YARN-2977 from 2.6.1 to 2.7.0 2014-12-21 12:53:25 +09:00
Karthik Kambatla 2abec14ec6 YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) 
(cherry picked from commit 24ee9e3431)
 2014-12-20 12:20:29 -08:00
Tsuyoshi Ozawa 7b6a0badc2 CHANGES.txt: add YARN-2977 2014-12-20 22:57:58 +09:00
Tsuyoshi Ozawa 31753da961 YARN-2977. Fixed intermittent TestNMClient failure. (Contributed by Junping Du) 2014-12-20 22:43:47 +09:00
Jian He 9180d11b3b YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks 
(cherry picked from commit 808cba3821)
 2014-12-19 16:56:55 -08:00
Karthik Kambatla b4e8ae591d YARN-2675. containersKilled metrics is not updated when the container is killed during localization. (Zhihai Xu via kasha) 
(cherry picked from commit 954fb8581e)
 2014-12-19 16:03:02 -08:00
Karthik Kambatla 8ee40a1580 YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit a22ffc3188)
 2014-12-19 15:38:27 -08:00
Jason Lowe 173664d70f YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by Jian He 
(cherry picked from commit 0402bada19)
 2014-12-18 23:31:11 +00:00
Junping Du cec9300201 YARN-2949. Add documentation for CGroups. (Contributed by Varun Vasudev) 
(cherry picked from commit 389f881d42)
 2014-12-18 06:26:53 -08:00
Junping Du 11e4ec52a1 YARN-2972. DelegationTokenRenewer thread pool never expands. Contributed by Jason Lowe 
(cherry picked from commit 2b4b0e8847)
 2014-12-17 17:41:21 -08:00
Karthik Kambatla d0d42c4e32 YARN-2203. [YARN-1492] Web UI for cache manager. (Chris Trezzo via kasha) 
(cherry picked from commit b7f64823e1)
 2014-12-17 16:34:33 -08:00
Karthik Kambatla 375a1e5971 YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. (Chris Trezzo via kasha) 
(cherry picked from commit a1bd140964)
 2014-12-17 16:25:02 -08:00
Karthik Kambatla 2d832ad2eb YARN-2964. FSLeafQueue#assignContainer - document the reason for using both write and read locks. (Tsuyoshi Ozawa via kasha) 
(cherry picked from commit f2d150ea12)
 2014-12-17 16:24:17 -08:00
Karthik Kambatla 300a665307 YARN-2189. Admin service for cache manager. Addendum to fix merge conflicts 2014-12-17 11:47:24 -08:00
cnauroth 36068768d8 HADOOP-11321. copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions. Contributed by Chris Nauroth. 
(cherry picked from commit e996a1bfd4)
 2014-12-16 15:32:23 -08:00
Jian He 878350a0e8 YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments before sending to RM. Contributed by Rohith Sharmaks 
(cherry picked from commit c65f1b382e)
 2014-12-16 11:01:24 -08:00
Karthik Kambatla b1ccf68db3 YARN-2914. [YARN-1492] Potential race condition in Singleton implementation of SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics. (Varun Saxena via kasha) 
(cherry picked from commit e597249d36)
 2014-12-15 12:00:46 -08:00
Jian He d2356a552a Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
(cherry picked from commit a4f2995b9e)
 2014-12-15 11:08:44 -08:00
Devaraj K 4c6760bc6b YARN-2356. yarn status command for non-existent application/application 
attempt/container is too verbose. Contributed by Sunil G.

(cherry picked from commit fae3e8614f)
 2014-12-15 14:46:27 +05:30
Harsh J da36252077 YARN-2950. Change message to mandate, not suggest JS requirement on UI. Contributed by Dustin Cote. 
(cherry picked from commit 0e37bbc8e3)
 2014-12-13 07:27:37 +05:30
Steve Loughran a858d726c8 YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) 2014-12-12 17:10:54 +00:00
Devaraj K 10ff600b35 YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in 
SchedulerApplicationAttempt ctor. Contributed by Devaraj K.

(cherry picked from commit bda748ac3a)
 2014-12-12 12:37:13 +05:30
Jian He 41f0d20fcb YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks 
(cherry picked from commit 614b6afea4)
 2014-12-11 11:18:45 -08:00
Jian He b31dfb17d8 YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. Contributed by Wangda Tan 
(cherry picked from commit 437322afca)
 2014-12-09 17:56:41 -08:00
Jian He 0f47e4e745 YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. Contributed by Wangda Tan 
(cherry picked from commit 2ed90a57fd)
 2014-12-09 16:48:46 -08:00
Karthik Kambatla 1986ea8dd2 YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg via kasha) 
(cherry picked from commit a2e07a5456)
 2014-12-09 14:00:52 -08:00
Karthik Kambatla 9d72b0282f YARN-2931. PublicLocalizer may fail until directory is initialized by LocalizeRunner. (Anubhav Dhoot via kasha) 
(cherry picked from commit db73cc9124)
 2014-12-08 22:26:44 -08:00
Karthik Kambatla d02cb9c512 YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent. (Ray Chiang via kasha) 
(cherry picked from commit 120e1decd7)
 2014-12-07 22:28:57 -08:00
Jian He 4cc0abe4fe YARN-2869. CapacityScheduler should trim sub queue names when parse configuration. Contributed by Wangda Tan 
(cherry picked from commit e69af836f3)
 2014-12-05 17:34:15 -08:00
Jason Lowe b72fb6c774 YARN-2056. Disable preemption at Queue level. Contributed by Eric Payne 
(cherry picked from commit 4b13082199)
 2014-12-05 21:08:14 +00:00
Robert Kanter 2ebd1bc654 YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YarnConfiguration. (rchiang via rkanter) 
(cherry picked from commit 3c72f54ef5)
 2014-12-05 12:10:20 -08:00
Karthik Kambatla aadd0c392b YARN-2189. [YARN-1492] Admin service for cache manager. (Chris Trezzo via kasha) 
(cherry picked from commit 78968155d7)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/bin/yarn
 2014-12-04 19:48:18 -08:00
Jian He 1d1e7682c9 YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R 
(cherry picked from commit 258623ff8b)
 2014-12-04 12:53:52 -08:00
Jian He 58c971164c YARN-2880. Added a test to make sure node labels will be recovered if RM restart is enabled. Contributed by Rohith Sharmaks 
(cherry picked from commit 73fbb3c66b)
 2014-12-03 17:17:40 -08:00
Harsh J ee25c0d890 YARN-2891. Failed Container Executor does not provide a clear error message. Contributed by Dustin Cote. (harsh) 
(cherry picked from commit 4b13318dea7a1cbbbfc1f84207af829cbe2f720e)
 2014-12-04 03:19:01 +05:30
Karthik Kambatla 25be97808b YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps. (Naganarasimha G R via kasha) 
(cherry picked from commit 799353e2c7)
 2014-12-03 13:45:03 -08:00
Jason Lowe db723a8499 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 
(cherry picked from commit 03ab24aa01)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2014-12-03 17:12:47 +00:00
Junping Du 86535ff65f YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values. (Contributed by Tsuyoshi OZAWA) 
(cherry picked from commit e65b7c5ff6)
 2014-12-03 04:12:35 -08:00
Jian He d6f3d4893d YARN-2894. Fixed a bug regarding application view acl when RM fails over. Contributed by Rohith Sharmaks 
(cherry picked from commit 392c3aaea8)
 2014-12-02 17:17:16 -08:00
Jian He 5067ac098b YARN-2136. Changed RMStateStore to ignore store opearations when fenced. Contributed by Varun Saxena 
(cherry picked from commit 52bcefca8b)
 2014-12-02 11:02:00 -08:00
Zhijie Shen d21ef79707 YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 
(cherry picked from commit 51af8d367d)
 2014-12-01 19:08:31 -08:00
Jian He d208c90148 YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe 
(cherry picked from commit a7fba0bc28)
 2014-12-01 16:45:17 -08:00
Jason Lowe 38ea1419f6 YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted. Contributed by Varun Saxena 
(cherry picked from commit 0f9528b99a)
 2014-12-01 22:30:20 +00:00
Junping Du c7bd22974a YARN-2907. SchedulerNode#toString should print all resource detail instead of only memory. (Contributed by Rohith) 
(cherry picked from commit c732ed760e)
 2014-12-01 05:41:56 -08:00
Zhijie Shen a9adc1295c YARN-2165. Added the sanity check for the numeric configuration values of the timeline service. Contributed by Vasanth kumar RJ. 
(cherry picked from commit 8f1454cc6d)
 2014-11-26 12:12:42 -08:00
Karthik Kambatla 3d4536af82 YARN-2188. [YARN-1492] Client service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 
(cherry picked from commit 834b3b98e7ecc4bfe93c7dd1a9e928e9653cf138)
 2014-11-25 16:22:14 -08:00
Jian He ae35b0e14d YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. Contributed by Jason Lowe 
(cherry picked from commit 8a7ca13b13)
 2014-11-25 16:18:02 -08:00
Jian He 2863056530 YARN-2404. Removed ApplicationAttemptState and ApplicationState class in RMStateStore. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit 5805a81efb)
 2014-11-25 12:51:03 -08:00
Zhijie Shen 57d62d4ded YARN-2697. Remove useless RMAuthenticationHandler. Contributed by Haosong Huang. 
(cherry picked from commit e37a4ff0c1)
 2014-11-24 15:06:44 -08:00
Jason Lowe 89ef49fb08 YARN-1984. LeveldbTimelineStore does not handle db exceptions properly. Contributed by Varun Saxena 
(cherry picked from commit 1ce4d33c2d)
 2014-11-24 22:38:09 +00:00
Sandy Ryza 3e2e8eac22 YARN-2669. FairScheduler: queue names shouldn't allow periods (Wei Yan via Sandy Ryza) 2014-11-21 16:08:22 -08:00
Karthik Kambatla e0df934ed1 Update release date for 2.5.2 
(cherry picked from commit 1e9a3f42ff)
 2014-11-21 14:34:47 -08:00
Karthik Kambatla af0b54a4ee YARN-2679. Add metric for container launch duration. (Zhihai Xu via kasha) 
(cherry picked from commit 233b61e495)
 2014-11-21 14:22:53 -08:00
Karthik Kambatla e9db0aa35c YARN-2604. Scheduler should consider max-allocation-* in conjunction with the largest node. (Robert Kanter via kasha) 
(cherry picked from commit 3114d4731d)
 2014-11-21 10:42:47 -08:00
Jonathan Eagles 6570f7fd72 YARN-2375. Allow enabling/disabling timeline server per framework. (Mit Desai via jeagles) 
(cherry picked from commit c298a9a845)
 2014-11-20 23:40:26 -06:00
Karthik Kambatla 7ba5bb0c5b YARN-2315. FairScheduler: Set current capacity in addition to capacity. (Zhihai Xu via kasha) 
(cherry picked from commit a9a0cc3679)
 2014-11-19 20:19:54 -08:00
Karthik Kambatla 564deb75a8 YARN-2802. ClusterMetrics to include AM launch and register delays. (Zhihai Xu via kasha) 
(cherry picked from commit f9f8e750edbe6db453f0a845e2ed49ede66e0e8a)
 2014-11-19 19:50:35 -08:00
Jian He db31ef7e7f YARN-2865. Fixed RM to always create a new RMContext when transtions from StandBy to Active. Contributed by Rohith Sharmaks 
(cherry picked from commit 9cb8b75ba5)
 2014-11-19 19:49:44 -08:00
Jian He c359e2a6a4 YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. Contributed by Abin Shahab 
(cherry picked from commit bc4ee5e06f)
 2014-11-19 14:45:01 -08:00
Zhijie Shen 610e123772 YARN-2870. Updated the command to run the timeline server it the document. Contributed by Masatake Iwasaki. 
(cherry picked from commit ef38fb9758)
 2014-11-18 16:17:07 -08:00
Jian He 1412e712b4 YARN-2157. Added YARN metrics in the documentaion. Contributed by Akira AJISAKA 
(cherry picked from commit 90a968d675)
 2014-11-18 16:13:08 -08:00
Karthik Kambatla 8df4c04c3a YARN-2690. [YARN-2574] Make ReservationSystem and its dependent classes independent of Scheduler type. (Anubhav Dhoot via kasha) 
(cherry picked from commit 2fce6d6141)
 2014-11-17 16:46:18 -08:00
Jason Lowe 242fd0e39a YARN-2414. RM web UI: app page will crash if app is failed before any attempt has been created. Contributed by Wangda Tan 
(cherry picked from commit 81c9d17af8)
 2014-11-17 21:16:54 +00:00
Vinod Kumar Vavilapalli 41275755c7 YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. Contributed by Rohith Sharma. 
(cherry picked from commit 198fb58776)
 2014-11-15 12:06:53 -08:00
Jian He 21054c5d14 YARN-2432. RMStateStore should process the pending events before close. Contributed by Varun Saxena 
(cherry picked from commit 9b860661a5)
 2014-11-14 17:25:21 -08:00
Sandy Ryza c4c77669f0 YARN-2811. In Fair Scheduler, reservation fulfillments shouldn't ignore max share (Siqi Li via Sandy Ryza) 2014-11-14 15:20:37 -08:00
Jason Lowe ad140d1fc8 YARN-2816. NM fail to start with NPE during container recovery. Contributed by Zhihai Xu 
(cherry picked from commit 49c38898b0)
 2014-11-14 21:27:16 +00:00
Jason Lowe 00d135d75a YARN-2857. ConcurrentModificationException in ContainerLogAppender. Contributed by Mohammad Kamrul Islam 
(cherry picked from commit f2fe8a800e)
 2014-11-14 20:48:51 +00:00
Jian He beb184ac58 YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state on app recovery. Contributed by Rohith Sharmaks 
(cherry picked from commit d005404ef7)
 2014-11-13 15:34:56 -08:00
Zhijie Shen 53d6c91df9 YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, attempts and containers. Contributed by Robert Kanter. 
(cherry picked from commit 3648cb57c9)
 2014-11-13 13:54:30 -08:00
Vinod Kumar Vavilapalli d35bcf2074 Revert "YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw)" 
This reverts commit b0a9cd37ab.

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 4ae9780e6a)
 2014-11-13 13:21:46 -08:00
Arun C. Murthy fc324d7062 Preparing to release hadoop-2.6.0-rc1. 
(cherry picked from commit 9ff7315cbe)
 2014-11-13 13:03:13 -08:00
Arun C. Murthy d0285d1ad6 YARN-2635. Merging to branch-2.6 for hadoop-2.6.0-rc1. 
(cherry picked from commit 81dc0ac6dc)
 2014-11-13 10:28:10 -08:00
Arun C. Murthy 906e0a1862 YARN-2853. Merging to branch-2.6 for hadoop-2.6.0-rc1. 
(cherry picked from commit d648e60eba)
 2014-11-13 10:05:07 -08:00
Jason Lowe 6fd547da8f YARN-2846. Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart. Contributed by Junping Du 
(cherry picked from commit 33ea5ae92b)
 2014-11-13 16:17:08 +00:00
Vinod Kumar Vavilapalli ed3e5cb164 YARN-2853. Fixed a bug in ResourceManager causing apps to hang when the user kill request races with ApplicationMaster finish. Contributed by Jian He. 
(cherry picked from commit 3651fe1b08)
 2014-11-13 08:13:57 -08:00
cnauroth 69a7780ee5 HADOOP-7984. Add hadoop --loglevel option to change log level. Contributed by Aikira AJISAKA. 2014-11-12 21:44:05 -08:00
Allen Wittenauer a6de8f2e30 YARN-2603. ApplicationConstants missing HADOOP_MAPRED_HOME (Ray Chiang via aw) 2014-11-12 17:04:24 -08:00
Zhijie Shen 8405ec0895 YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. 
(cherry picked from commit be7bf956e9)
 2014-11-12 11:09:34 -08:00
Arun C. Murthy f7f78cf934 YARN-2843. Merging to branch-2.6 for hadoop-2.6.0-rc1. 2014-11-12 10:41:44 -08:00
Arun C. Murthy cd8559e099 YARN-2841. Merging to branch-2.6 for hadoop-2.6.0-rc1. 2014-11-12 10:37:40 -08:00
Karthik Kambatla ff1b13ded5 YARN-2236. [YARN-1492] Shared Cache uploader service on the Node Manager. (Chris Trezzo and Sanjin Lee via kasha) 
(cherry picked from commit a04143039e)
 2014-11-12 09:31:30 -08:00
Jason Lowe 6984d899e3 YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne 
(cherry picked from commit f8aefa5e9c)
 2014-11-12 17:02:30 +00:00
Ravi Prakash d863f54f57 YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN 2014-11-11 21:29:27 -08:00
Karthik Kambatla 6cbd3aa2e8 YARN-570. Time strings are formated in different timezone. (Akira Ajisaka and Peng Zhang via kasha) 
(cherry picked from commit 840b13110d63d7e062cefd4b993cc7dea44aed8d)
 2014-11-11 13:23:29 -08:00
Vinod Kumar Vavilapalli 9d6c0d7c9f YARN-2843. Fixed NodeLabelsManager to trim inputs for hosts and labels so as to make them work correctly. Contributed by Wangda Tan. 
(cherry picked from commit 0fd97f9c19)
 2014-11-11 12:45:59 -08:00
Karthik Kambatla f030959849 YARN-2735. diskUtilizationPercentageCutoff and diskUtilizationSpaceCutoff are initialized twice in DirectoryCollection. (Zhihai Xu via kasha) 
(cherry picked from commit 061bc293c8)
 2014-11-11 10:31:56 -08:00
Xuan b390579c15 YARN-2841: Correct fix version from branch-2.6 to branch-2.7 in the 
change log.

(cherry picked from commit 58e9bf4b90)
 2014-11-10 18:35:59 -08:00
Xuan b3badf935a YARN-2841. RMProxy should retry EOFException. Contributed by Jian He 
(cherry picked from commit 5c9a51f140)
 2014-11-10 18:26:32 -08:00
Karthik Kambatla 92de44c2c5 Set the release date for 2.5.2 
(cherry picked from commit 68a0508728)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-10 15:23:43 -08:00
Arun C. Murthy adfb830a2b Preparing to release hadoop-2.6.0: Set version in branch-2 to 2.7.0-SNAPSHOT. 2014-11-09 19:19:02 -08:00
Karthik Kambatla 569b7dae6c Fix broken download of directories 
(cherry picked from commit 745c9a0124)
 2014-11-09 19:03:31 -08:00
Vinod Kumar Vavilapalli d76fc94b21 YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery consistent with the (somewhat incorrect) behaviour in the non-recovery case. Contributed by Jian He. 
(cherry picked from commit e76faebc95)
 2014-11-09 18:57:48 -08:00
Arun C. Murthy cff08dab5e Updated CHANGES.txt to reflect hadoop-2.6.0 release dates. 
(cherry picked from commit 770cc14442)
 2014-11-09 18:48:01 -08:00
Arun C. Murthy 175d222bfc YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 
(cherry picked from commit 43cd07b408)
 2014-11-09 15:03:59 -08:00
Vinod Kumar Vavilapalli 0a5d95f705 YARN-2607. Fixed issues in TestDistributedShell. Contributed by Wangda Tan. 
(cherry picked from commit 737d9284c1)
 2014-11-08 11:02:17 -08:00
Zhijie Shen 1880a5a7c3 YARN-2505. Supported get/add/remove/change labels in RM REST API. Contributed by Craig Welch. 
(cherry picked from commit 9a4e0d343e)
 2014-11-07 20:39:53 -08:00
Vinod Kumar Vavilapalli b865a95f06 YARN-2826. Fixed user-groups mappings' refresh bug caused by YARN-2826. Contributed by Wangda Tan. 
(cherry picked from commit df36edf751)
 2014-11-07 19:44:52 -08:00
Xuan 7d26cffb0c YARN-2819. NPE in ATS Timeline Domains when upgrading from 2.4 to 2.6. Contributed by Zhijie Shen 
(cherry picked from commit 4a114dd67a)
 2014-11-07 16:12:45 -08:00
Jason Lowe 9447235527 YARN-2632. Document NM Restart feature. Contributed by Junping Du and Vinod Kumar Vavilapalli 
(cherry picked from commit 1e215e8ba2)
 2014-11-07 23:41:29 +00:00
Jason Lowe a5764cb783 YARN-2825. Container leak on NM. Contributed by Jian He 
(cherry picked from commit c3d475070a)
 2014-11-07 23:17:34 +00:00
Vinod Kumar Vavilapalli a50345f654 YARN-2753. Fixed a bunch of bugs in the NodeLabelsManager classes. Contributed by Zhihai xu. 
(cherry picked from commit 4cfd5bc7c1)
 2014-11-07 14:16:54 -08:00
cnauroth c16f718293 YARN-2803. MR distributed cache not working correctly on Windows after NodeManager privileged account changes. Contributed by Craig Welch. 
(cherry picked from commit 06b797947c)
 2014-11-07 12:36:42 -08:00
Vinod Kumar Vavilapalli 7b19079657 YARN-2827. Fixed bugs in "yarn queue" CLI. Contributed by Wangda Tan. 2014-11-07 12:00:49 -08:00
Vinod Kumar Vavilapalli d5d2905262 YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. 
(cherry picked from commit 2ac1be7dec)
 2014-11-07 10:44:25 -08:00
Xuan bf79541868 YARN-2810. TestRMProxyUsersConf fails on Windows VMs. Contributed by Varun Vasudev 
(cherry picked from commit 1e97f2f094)
 2014-11-07 09:46:04 -08:00
Vinod Kumar Vavilapalli eb01c602ed YARN-2823. Fixed ResourceManager app-attempt state machine to inform schedulers about previous finished attempts of a running appliation to avoid expectation mismatch w.r.t transferred containers. Contributed by Jian He. 
(cherry picked from commit a5657182a7)
 2014-11-07 09:30:04 -08:00
Vinod Kumar Vavilapalli b1521c7c3c YARN-2647. Added a queue CLI for getting queue information. Contributed by Sunil Govind. 
(cherry picked from commit 61effcbebc)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/bin/yarn
 2014-11-06 20:57:55 -08:00
Vinod Kumar Vavilapalli 52e57a95d9 YARN-2744. Fixed CapacityScheduler to validate node-labels correctly against queues. Contributed by Wangda Tan. 
(cherry picked from commit a3839a9fbf)
 2014-11-06 17:29:12 -08:00
Steve Loughran 5924e74d55 YARN-2768 Improved Yarn Registry service record structure (stevel) 2014-11-06 20:21:25 +00:00
Vinod Kumar Vavilapalli 0f9199fb07 YARN-2818. Removed the now unnecessary user entity injection from Timeline service given we now have domains. Contributed by Zhijie Shen. 
(cherry picked from commit f5b19bed7d)
 2014-11-06 11:49:44 -08:00
Xuan 4aa98d5991 YARN-2812. TestApplicationHistoryServer is likely to fail on less powerful machine. Contributed by Zhijie Shen 
(cherry picked from commit b0b52c4e11)
 2014-11-05 20:44:07 -08:00
Xuan e26be1d85c YARN-2813. Fixed NPE from MemoryTimelineStore.getDomains. Contributed by Zhijie Shen 2014-11-05 18:26:07 -08:00
Jian He f92ff24f5e YARN-2579. Fixed a deadlock issue when EmbeddedElectorService and FatalEventDispatcher try to transition RM to StandBy at the same time. Contributed by Rohith Sharmaks 
(cherry picked from commit 395275af86)
 2014-11-05 17:03:01 -08:00
Vinod Kumar Vavilapalli 3bc31e35a6 YARN-2805. Fixed ResourceManager to load HA configs correctly before kerberos login. Contributed by Wangda Tan. 
(cherry picked from commit 834e931d8e)
 2014-11-05 15:31:41 -08:00
Zhijie Shen f3ed97de3a YARN-2767. Added a test case to verify that http static user cannot kill or submit apps in the secure mode. Contributed by Varun Vasudev. 
(cherry picked from commit 7a4c92a9d55fcecef066053ac30dff0fcd4ec90c)
 2014-11-05 10:59:05 -08:00
Vinod Kumar Vavilapalli 56ab333161 YARN-2804. Fixed Timeline service to not fill the logs with JAXB bindings exceptions. Contributed by Zhijie Shen. 
(cherry picked from commit b76179895d)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-11-04 18:03:17 -08:00
Haohui Mai fa92b40f09 HADOOP-11230. Add missing dependency of bouncycastle for kms, httpfs, hdfs, MR and YARN. Contributed by Robert Kanter. 2014-11-04 17:53:00 -08:00
Karthik Kambatla 3e4b280de7 YARN-2010. Handle app-recovery failures gracefully. (Jian He and Karthik Kambatla via kasha) 
(cherry picked from commit b2cd269802)
 2014-11-04 17:48:49 -08:00
Karthik Kambatla 6f9484221e Adding release 2.5.2 to CHANGES.txt 
(cherry picked from commit 4de56d27fb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-11-04 16:51:54 -08:00
Zhijie Shen e06c23a6c9 YARN-2752. Made ContainerExecutor append "nice -n" arg only when priority adjustment flag is set. Contributed by Xuan Gong. 2014-11-04 15:50:10 -08:00
Haohui Mai aa09c4c342 HADOOP-10563. Remove the dependency of jsp in trunk. Contributed by Haohui Mai. 2014-11-04 14:52:51 -08:00
Vinod Kumar Vavilapalli 9c76dcadaf YARN-1922. Fixed NodeManager to kill process-trees correctly in the presence of races between the launch and the stop-container call and when root processes crash. Contributed by Billie Rinaldi. 
(cherry picked from commit c5a46d4c8c)
 2014-11-03 16:40:37 -08:00
Vinod Kumar Vavilapalli 277141b82d YARN-2795. Fixed ResourceManager to not crash loading node-label data from HDFS in secure mode. Contributed by Wangda Tan. 
(cherry picked from commit ec6cbece8e)
 2014-11-03 13:45:31 -08:00
Vinod Kumar Vavilapalli 715c81ef6d YARN-2788. Fixed backwards compatiblity issues with log-aggregation feature that were caused when adding log-upload-time via YARN-2703. Contributed by Xuan Gong. 
(cherry picked from commit 58e9f24e0f)
 2014-11-03 13:19:34 -08:00
Vinod Kumar Vavilapalli 52c58115d2 YARN-2798. Fixed YarnClient to populate the renewer correctly for Timeline delegation tokens. Contributed by Zhijie Shen. 
(cherry picked from commit 71fbb474f5)
 2014-11-03 12:50:39 -08:00
Jason Lowe 8091ea54d5 YARN-2730. DefaultContainerExecutor runs only one localizer at a time. Contributed by Siqi Li 
(cherry picked from commit 6157ace547)
 2014-11-03 20:39:14 +00:00
Zhijie Shen caae0a0a08 YARN-2785. Fixed intermittent TestContainerResourceUsage failure. Contributed by Varun Vasudev. 
(cherry picked from commit 27715ec63b)
 2014-11-02 15:22:54 -08:00
Vinod Kumar Vavilapalli 6627f67bf5 YARN-2790. Fixed a NodeManager bug that was causing log-aggregation to fail beyond HFDS delegation-token expiry even when RM is a proxy-user (YARN-2704). Contributed by Jian He. 
(cherry picked from commit 5c0381c96a)
 2014-11-01 16:33:35 -07:00
Zhijie Shen d9ac25454c YARN-2711. Fixed TestDefaultContainerExecutor#testContainerLaunchError failure on Windows. Contributed by Varun Vasudev. 
(cherry picked from commit 1cd088fd9d)
 2014-10-31 17:45:05 -07:00
Jason Lowe ada40aa933 YARN-2707. Potential null dereference in FSDownload. Contributed by Gera Shegalov 
(cherry picked from commit 7bc179f9f5)
 2014-10-31 22:54:49 +00:00
Vinod Kumar Vavilapalli 6d4a04a033 YARN-2789. Re-instated the NodeReport.newInstance private unstable API modified in YARN-2698 so that tests in YARN frameworks don't break. Contributed by Wangda Tan. 
(cherry picked from commit 6ce32f593b)
 2014-10-31 15:33:41 -07:00
Xuan 902566741e YARN-2701. Addendum patch. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 2014-10-31 14:39:06 -07:00
Jian He a87f827ae4 YARN-2770. Added functionality to renew/cancel TimeLineDelegationToken. Contributed by Zhijie Shen 
(cherry picked from commit 1b4be91866)
 2014-10-31 13:17:19 -07:00
Karthik Kambatla 268af259b5 YARN-2186. [YARN-1492] Node Manager uploader service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 
(cherry picked from commit 256697acd5)
 2014-10-31 13:01:25 -07:00
Vinod Kumar Vavilapalli 9b879d713a YARN-2778. Moved node-lables' reports to the yarn nodes CLI from the admin CLI. Contributed by Wangda Tan. 
(cherry picked from commit b6c1188b85)
 2014-10-31 10:19:14 -07:00
Vinod Kumar Vavilapalli 97694e78ad YARN-2779. Fixed ResourceManager to not require delegation tokens for communicating with Timeline Service. Contributed by Zhijie Shen. 
(cherry picked from commit d1828d9443)
 2014-10-30 23:17:16 -07:00
Vinod Kumar Vavilapalli 0ba8d8c19c YARN-2698. Moved some node label APIs to be correctly placed in client protocol. Contributed by Wangda Tan. 
(cherry picked from commit e0233c16eb)
 2014-10-30 23:02:06 -07:00
Steve Loughran cd24075343 YARN-2677 registry punycoding of usernames doesn't fix all usernames to be DNS-valid (stevel) 2014-10-30 22:32:20 +00:00
Jason Lowe 3e8544c5f2 YARN-2755. NM fails to clean up usercache_DEL_<timestamp> dirs after YARN-661. Contributed by Siqi Li 
(cherry picked from commit 73e626ad91)
 2014-10-30 15:11:57 +00:00
Karthik Kambatla d2ba115f06 YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks. (Tsuyoshi Ozawa via kasha) 
(cherry picked from commit 179cab81e0)
 2014-10-30 00:39:34 -07:00
Zhijie Shen d2a5c386c1 YARN-2769. Fixed the problem that timeline domain is not set in distributed shell AM when using shell_command on Windows. Contributed by Varun Vasudev. 
(cherry picked from commit 72496311def320987c31822776e8fbb73dfcc726)
 2014-10-29 12:14:48 -07:00
Karthik Kambatla e88832dfb3 YARN-2742. FairSchedulerConfiguration should allow extra spaces between value and unit. (Wei Yan via kasha) 
(cherry picked from commit 782971ae7a)
 2014-10-29 10:25:29 -07:00
Zhijie Shen 734cbe7de2 YARN-2747. Fixed the test failure of TestAggregatedLogFormat when native I/O is enabled. Contributed by Xuan Gong. 
(cherry picked from commit ec63a3ffbd)
 2014-10-28 22:45:20 -07:00
Jian He 720de7eb4c YARN-2503. Added node lablels in web UI. Contributed by Wangda Tan 
(cherry picked from commit d5e0a09721)
 2014-10-28 18:00:59 -07:00
Zhijie Shen f40389ae08 YARN-2741. Made NM web UI serve logs on the drive other than C: on Windows. Contributed by Craig Welch. 
(cherry picked from commit 8984e9b177)
 2014-10-28 14:12:09 -07:00
Xuan 43b1878f86 YARN-2758. Update TestApplicationHistoryClientService to use the new generic history store. Contributed by Zhijie Shen 2014-10-28 13:30:55 -07:00
Xuan 43a53f2b65 YARN-2279. Add UTs to cover timeline server authentication. Contributed by Zhijie Shen 2014-10-28 11:18:35 -07:00
Karthik Kambatla 3f7edeb055 YARN-2760. Remove 'experimental' from FairScheduler docs. (Harsh J via kasha) 
(cherry picked from commit ade3727ecb)
 2014-10-28 08:49:34 -07:00
Jian He a1c7a707ca YARN-2502. Changed DistributedShell to support node labels. Contributed by Wangda Tan 
(cherry picked from commit f6b963fdfc)
 2014-10-27 20:13:53 -07:00
Vinod Kumar Vavilapalli 0ad33e1483 YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the sake of localization and log-aggregation for long-running services. Contributed by Jian He. 
(cherry picked from commit a16d022ca4)
 2014-10-27 15:50:51 -07:00
Jian He 2d3c64943d YARN-2591. Fixed AHSWebServices to return FORBIDDEN(403) if the request user doesn't have access to the history data. Contributed by Zhijie Shen 
(cherry picked from commit c05b581a55)
 2014-10-27 10:46:37 -07:00
Xuan 3a77838535 YARN-2726. CapacityScheduler should explicitly log when an accessible label has no capacity. Contributed by Wangda Tan 2014-10-27 10:28:45 -07:00
Zhijie Shen cb3974524c YARN-2734. Skipped sub-folders in the local log dir when aggregating logs. Contributed by Xuan Gong. 
(cherry picked from commit caecd9fffe)
 2014-10-26 12:59:29 -07:00
Vinod Kumar Vavilapalli 36d3f1fcca YARN-2743. Fixed a bug in ResourceManager that was causing RMDelegationToken identifiers to be tampered and thus causing app submission failures in secure mode. Contributed by Jian He. 
(cherry picked from commit 0186645505)
 2014-10-26 11:15:45 -07:00
Xuan f862999a7e YARN-2723. Fix rmadmin -replaceLabelsOnNode does not correctly parse 
port. Contributed by Naganarasimha G R
 2014-10-25 12:47:09 -07:00
Karthik Kambatla acc6f622e0 YARN-2183. [YARN-1492] Cleaner service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 
(cherry picked from commit c51e53d7aa)
 2014-10-25 11:50:06 -07:00
Jian He 9bd149978d YARN-2314. Disable ContainerManagementProtocolProxy cache by default to prevent creating thousands of threads in a large cluster. Contributed by Jason Lowe 
(cherry picked from commit f44cf99599)
 2014-10-24 23:06:00 -07:00
Jian He 96a6e02d16 YARN-1915. Fixed a race condition that client could use the ClientToAMToken to contact with AM before AM actually receives the ClientToAMTokenMasterKey. Contributed by Jason Lowe 
(cherry picked from commit 5864dd99a4)
 2014-10-24 22:48:48 -07:00
Karthik Kambatla 9af3cfc7eb YARN-2713. "RM Home" link in NM should point to one of the RMs in an HA setup. (kasha) 
(cherry picked from commit 683897fd02)
 2014-10-24 15:00:37 -07:00
Zhijie Shen 1b81105143 YARN-2703. Added logUploadedTime into LogValue for better display. Contributed by Xuan Gong. 
(cherry picked from commit f81dc3f995)
 2014-10-24 14:12:17 -07:00
Zhijie Shen 13c60bac57 YARN-2724. Skipped uploading a local log file to HDFS if exception is raised when opening it. Contributed by Xuan Gong. 
(cherry picked from commit e31f0a6558)
 2014-10-24 11:15:19 -07:00
Jian He d57f2b9dea Updated CHANGES.txt to move YARN-2682 to branch-2.6 
(cherry picked from commit 071c925c7d)
 2014-10-23 22:28:23 -07:00
Zhijie Shen 3b03ea6b50 YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He. 
(cherry picked from commit 0f3b6900be)
 2014-10-23 22:03:30 -07:00
Zhijie Shen 36515435ff YARN-2732. Fixed syntax error in SecureContainer.apt.vm. Contributed by Jian He. 
(cherry picked from commit b94b8b30f2)
 2014-10-22 18:41:05 -07:00
Jian He 1c235a4448 YARN-2198. Remove the need to run NodeManager as privileged account for Windows Secure Container Executor. Contributed by Remus Rusanu 
(cherry picked from commit 3b12fd6cfb)
 2014-10-22 15:58:26 -07:00
Steve Loughran 5e6494ea95 YARN-2700 TestSecureRMRegistryOperations failing on windows: auth problems 2014-10-22 21:13:32 +01:00
Steve Loughran a64ac5501c YARN-2692 ktutil test hanging on some machines/ktutil versions (stevel) 2014-10-22 14:35:13 +01:00
Zhijie Shen 4b9de2cdfe YARN-2721. Suppress NodeExist exception thrown by ZKRMStateStore when it retries creating znode. Contributed by Jian He. 
(cherry picked from commit 7e3b5e6f5c)
 2014-10-21 21:52:01 -07:00
Vinod Kumar Vavilapalli 77dc9afe86 YARN-2715. Fixed ResourceManager to respect common configurations for proxy users/groups beyond just the YARN level config. Contributed by Zhijie Shen. 
(cherry picked from commit c0e034336c)
 2014-10-21 20:10:45 -07:00
Zhijie Shen 296bbd8971 YARN-2709. Made timeline client getDelegationToken API retry if ConnectException happens. Contributed by Li Lu. 
(cherry picked from commit b2942762d7)
 2014-10-21 16:08:18 -07:00
cnauroth a066134277 YARN-2720. Windows: Wildcard classpath variables not expanded against resources contained in archives. Contributed by Craig Welch. 
(cherry picked from commit 6637e3cf95)
 2014-10-21 12:34:58 -07:00
Jason Lowe 3820bf055e YARN-90. NodeManager should identify failed disks becoming good again. Contributed by Varun Vasudev 
(cherry picked from commit 6f2028bd15)
 2014-10-21 17:33:34 +00:00
Zhijie Shen fd7ba56f6a YARN-2717. Avoided duplicate logging when container logs are not found. Contributed by Xuan Gong. 
(cherry picked from commit 171f2376d2)
 2014-10-20 22:23:12 -07:00
Jian He b52928a6ea Missing file for YARN-1879 
(cherry picked from commit 4a78a75228)
 2014-10-20 21:14:57 -07:00
Jian He e9564e729f Missing file for YARN-2701 
(cherry picked from commit 4fa1fb3193)
 2014-10-20 19:58:21 -07:00
Jian He 3c8ae89050 YARN-2701. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 
(cherry picked from commit 2839365f23)
 2014-10-20 19:54:10 -07:00
Zhijie Shen f75570734b YARN-2582. Fixed Log CLI and Web UI for showing aggregated logs of LRS. Contributed Xuan Gong. 
(cherry picked from commit e90718fa5a)
 2014-10-20 13:42:22 -07:00
Zhijie Shen 42ceed8f20 YARN-2673. Made timeline client put APIs retry if ConnectException happens. Contributed by Li Lu. 
(cherry picked from commit 89427419a3)
 2014-10-20 12:23:10 -07:00
Vinod Kumar Vavilapalli 9915d52185 YARN-2504. Enhanced RM Admin CLI to support management of node-labels. Contribyted by Wangda Tan. 
(cherry picked from commit 8256766498)
 2014-10-18 12:08:28 -07:00
Vinod Kumar Vavilapalli ea841eb25d YARN-2676. Enhanced Timeline auth-filter to support proxy users. Contributed by Zhijie Shen. 
(cherry picked from commit 39063cd36f)
 2014-10-17 22:03:43 -07:00
Vinod Kumar Vavilapalli 1c52b6551b YARN-2705. Fixed bugs in ResourceManager node-label manager that were causing test-failures: added a dummy in-memory labels-manager. Contributed by Wangda Tan. 
(cherry picked from commit e9c66e8fd2)
 2014-10-17 18:27:04 -07:00
Jian He 78a57087bf YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit c3de2412eb)
 2014-10-17 16:36:15 -07:00
Jian He 79428e021f YARN-2588. Standby RM fails to transitionToActive if previous transitionToActive failed with ZK exception. Contributed by Rohith Sharmaks 
(cherry picked from commit a6aa6e42ca)
 2014-10-17 10:55:04 -07:00
Vinod Kumar Vavilapalli 99ce4277a8 YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail when using ephemeral ports on NodeIDs. Contributed by Wangda Tan. 
(cherry picked from commit abae63caf9)
 2014-10-17 08:59:22 -07:00
Jian He 10110b6eae Moved YARN-2682 from 2.6 to 2.7 in CHANGES.txt 
(cherry picked from commit 72093fd8cb)
 2014-10-16 19:03:29 -07:00
Jian He a78f1f8da8 YARN-2682. Updated WindowsSecureContainerExecutor to not use DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir() instead. Contributed by Zhihai Xu 
(cherry picked from commit 0fd0ebae64)
 2014-10-16 18:15:43 -07:00
Jian He caba212f12 YARN-2621. Simplify the output when the user doesn't have the access for getDomain(s). Contributed by Zhijie Shen 
(cherry picked from commit 233d446be1)
 2014-10-16 16:38:00 -07:00
Steve Loughran fddbf52caa YARN-2689 TestSecureRMRegistryOperations failing on windows: secure ZK won't start 2014-10-16 14:21:49 -07:00
Vinod Kumar Vavilapalli 76cf2250de YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan. 
(cherry picked from commit b3056c266a)
 2014-10-15 18:48:36 -07:00
Vinod Kumar Vavilapalli e8e3a36213 YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan. 
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.

(cherry picked from commit f2ea555ac6)
 2014-10-15 18:34:19 -07:00
Jian He f93d2ea27e YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:28:26 -07:00
Zhijie Shen bec165e163 YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. 
(cherry picked from commit 1220bb72d4)
 2014-10-14 21:56:44 -07:00
Zhijie Shen 8aeda20b25 HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(cherry picked from commit cdce88376a)
 2014-10-14 11:37:28 -07:00
Karthik Kambatla 88455173e8 YARN-2566. DefaultContainerExecutor should pick a working directory randomly. (Zhihai Xu via kasha) 
(cherry picked from commit cc93e7e683)
 2014-10-13 16:32:42 -07:00
Karthik Kambatla a1116b56a4 YARN-2641. Decommission nodes on -refreshNodes instead of next NM-RM heartbeat. (Zhihai Xu via kasha) 
(cherry picked from commit da709a2eac)
 2014-10-13 16:24:00 -07:00
Jian He 31a4bf7321 Missing Changes.txt for YARN-2308 
(cherry picked from commit 178bc505da)
 2014-10-13 14:20:18 -07:00
Jian He 536254f5e7 YARN-2308. Changed CapacityScheduler to explicitly throw exception if the queue 
to which the apps were submitted is changed across RM restart. Contributed by Craig Welch & Chang Li
(cherry picked from commit f9680d9a16)
 2014-10-13 14:16:12 -07:00
Jason Lowe 6b5741d423 YARN-2377. Localization exception stack traces are not passed as diagnostic info. Contributed by Gera Shegalov 
(cherry picked from commit a56ea01002)
 2014-10-13 18:32:38 +00:00
Zhijie Shen e51ae64761 YARN-2651. Spun off LogRollingInterval from LogAggregationContext. Contributed by Xuan Gong. 
(cherry picked from commit 4aed2d8e91)
 2014-10-13 10:55:09 -07:00
Jason Lowe 9ed9ea8c52 YARN-2667. Fix the release audit warning caused by hadoop-yarn-registry. Contributed by Yi Liu 
(cherry picked from commit 344a10ad5e)
 2014-10-13 15:40:16 +00:00
Steve Loughran 38dedc2f07 YARN-2668 yarn-registry JAR won't link against ZK 3.4.5. (stevel) 2014-10-11 18:33:29 -07:00
Vinod Kumar Vavilapalli fff6a1f77e YARN-2501. Enhanced AMRMClient library to support requests against node labels. Contributed by Wangda Tan. 
(cherry picked from commit a5ec3d0809)
 2014-10-10 19:58:42 -07:00
Vinod Kumar Vavilapalli 8ef83a9480 YARN-2494. Added NodeLabels Manager internal API and implementation. Contributed by Wangda Tan. 
(cherry picked from commit db7f165319)
 2014-10-10 11:45:15 -07:00
Zhijie Shen 1e6d81a886 YARN-2583. Modified AggregatedLogDeletionService to be able to delete rolling aggregated logs. Contributed by Xuan Gong. 
(cherry picked from commit cb81bac002)
 2014-10-10 00:16:34 -07:00
cnauroth b81641a310 YARN-2662. TestCgroupsLCEResourcesHandler leaks file descriptors. Contributed by Chris Nauroth. 
(cherry picked from commit d3afd730ac)
 2014-10-09 22:47:04 -07:00
Zhijie Shen 79204dced9 YARN-2671. Fix the Jira number in the change log. 
(cherry picked from commit 5b12df6587)
 2014-10-09 20:59:37 -07:00
Zhijie Shen ad47a27dbf YARN-2617. Fixed ApplicationSubmissionContext to still set resource for backward compatibility. Contributed by Wangda Tan. 
(cherry picked from commit e532ed8faa)
 2014-10-09 18:00:01 -07:00
Karthik Kambatla 7e20187f91 YARN-2180. [YARN-1492] In-memory backing store for cache manager. (Chris Trezzo via kasha) 
(cherry picked from commit 4f426fe223)
 2014-10-09 17:16:45 -07:00
Vinod Kumar Vavilapalli d3125f3de5 YARN-2544. Added admin-API objects for using node-labels. Contributed by Wangda Tan. 
(cherry picked from commit 596702a025)
 2014-10-09 15:47:20 -07:00
Mayank Bansal c1a86d105e YARN-2598 GHS should show N/A instead of null for the inaccessible information (Zhijie Shen via mayank) 2014-10-09 13:43:36 -07:00
Zhijie Shen 4d170d3db6 YARN-2629. Made the distributed shell use the domain-based timeline ACLs. Contributed by Zhijie Shen. 
(cherry picked from commit 1d4612f5ad)
 2014-10-09 13:01:44 -07:00
Vinod Kumar Vavilapalli b42cd54d7a YARN-2493. Added user-APIs for using node-labels. Contributed by Wangda Tan. 
(cherry picked from commit 180afa2f86)
 2014-10-09 11:27:17 -07:00
Steve Loughran d218ab58fc YARN-913 service registry: YARN-2652 add hadoop-yarn-registry package under hadoop-yarn 2014-10-08 12:55:47 -07:00
Jian He 5e6f86e3aa YARN-2649. Fixed TestAMRMRPCNodeUpdates test failure. Contributed by Ming Ma 
(cherry picked from commit e16e25ab1b)
 2014-10-08 11:03:00 -07:00
Jian He 9ff7c0c96c YARN-1857. CapacityScheduler headroom doesn't account for other AM's running. Contributed by Chen He and Craig Welch 
(cherry picked from commit 30d56fdbb4)
 2014-10-07 13:48:03 -07:00
Jian He 5c33e91229 YARN-2644. Fixed CapacityScheduler to return up-to-date headroom when AM allocates. Contributed by Craig Welch 
(cherry picked from commit 519e5a7dd2)
 2014-10-06 15:51:38 -07:00
Jian He 43358be60b YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier to use protobuf as payload. Contributed by Junping Du 
(cherry picked from commit ea26cc0b4a)
 2014-10-06 10:48:27 -07:00
Chris Douglas 4ba102bdc3 YARN-1051. Add a system for creating reservations of cluster capacity. 
Contributed by Subru Krishnan and Carlo Curino.
(cherry picked from commit c8212bacb1)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-06 10:30:53 -07:00
subru fb5e9df7fd YARN-2611. Fixing jenkins findbugs warning and TestRMWebServicesCapacitySched for branch YARN-1051. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit c47464aba4)
(cherry picked from commit a2986234be)
 2014-10-06 10:21:07 -07:00
subru ae0f16ccc8 YARN-2576. Making test patch pass in branch. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 90ac0be86b)
(cherry picked from commit 5e10a13bb4)
 2014-10-06 10:21:06 -07:00
subru cbfbdf60d6 YARN-2080. Integrating reservation system with ResourceManager and client-RM protocol. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 8baeaead85)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
(cherry picked from commit 6261f7cc69)
 2014-10-06 10:21:06 -07:00
subru fec639cda1 YARN-1712. Plan follower that synchronizes the current state of reservation subsystem with the scheduler. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 169085319b)
(cherry picked from commit 3418c56bcf)
 2014-10-06 10:21:06 -07:00
carlo curino 6bfdaf06c4 YARN-1711. Policy to enforce instantaneous and over-time quotas on user reservation. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit c4918cb4cb)
(cherry picked from commit b6df0dddcd)
 2014-10-06 10:21:05 -07:00
carlo curino 6a3c167175 YARN-1710. Logic to find allocations within a Plan that satisfy user ReservationRequest(s). Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit aef7928899)
(cherry picked from commit f66ffcf832)
 2014-10-06 10:21:05 -07:00
subru 63250ef9d6 YARN-1709. In-memory data structures used to track resources over time to enable reservations. 
(cherry picked from commit 0d8b2cd88b)
(cherry picked from commit cf4b34282a)
 2014-10-06 10:21:05 -07:00
subru 3a167056d1 YARN-1708. Public YARN APIs for creating/updating/deleting reservations. 
(cherry picked from commit 3f2e3b275b)
(cherry picked from commit c9266df404)
 2014-10-06 10:21:05 -07:00
carlo curino b81f571e60 YARN-2475. Logic for responding to capacity drops for the ReservationSystem. Contributed by Carlo Curino and Subru Krishnan. 
(cherry picked from commit f83a07f266)
(cherry picked from commit 1c6950354f)
 2014-10-06 10:21:05 -07:00
carlo curino cf5ef00b96 YARN-1707. Introduce APIs to add/remove/resize queues in the CapacityScheduler. Contributed by Carlo Curino and Subru Krishnan 
(cherry picked from commit aac47fda7f)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
(cherry picked from commit eb3e40b833)
 2014-10-06 10:21:04 -07:00
Vinod Kumar Vavilapalli 7ed61e150c YARN-2468. Enhanced NodeManager to support log handling APIs (YARN-2569) for use by long running services. Contributed by Xuan Gong. 
(cherry picked from commit 34cdcaad71)
 2014-10-03 12:17:03 -07:00
Karthik Kambatla b6ce0a1f69 YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 
(cherry picked from commit 80d11eb68e)
 2014-10-03 11:52:05 -07:00
Jian He 1c58ab9206 YARN-2562. Changed ContainerId#toString() to be more readable. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit eb6ce5e97c)
 2014-10-03 11:04:46 -07:00
Jian He 6f4c77409b YARN-2628. Capacity scheduler with DominantResourceCalculator carries out reservation even though slots are free. Contributed by Varun Vasudev 
(cherry picked from commit 054f285526)
 2014-10-02 15:16:07 -07:00
Zhijie Shen ecaae67ab3 YARN-2527. Fixed the potential NPE in ApplicationACLsManager and added test cases for it. Contributed by Benoy Antony. 
(cherry picked from commit 1c93025a1b)
 2014-10-02 14:57:27 -07:00
Jason Lowe 531c1fd00a YARN-2624. Resource Localization fails on a cluster due to existing cache directories. Contributed by Anubhav Dhoot 
(cherry picked from commit 29f520052e)
 2014-10-02 17:40:44 +00:00
Karthik Kambatla b6f4d1ef42 YARN-2254. TestRMWebServicesAppsModification should run against both CS and FS. (Zhihai Xu via kasha) 
(cherry picked from commit 5e0b49da9c)
 2014-10-02 10:15:41 -07:00
Jian He 61c7ceaf82 YARN-2617. Fixed NM to not send duplicate container status whose app is not running. Contributed by Jun Gong 
(cherry picked from commit 3ef1cf187f)
 2014-10-02 10:04:42 -07:00
junping_du 6483342a61 YARN-1979. TestDirectoryCollection fails when the umask is unusual. (Contributed by Vinod Kumar Vavilapalli and Tsuyoshi OZAWA) 
(cherry picked from commit c7cee9b455)
 2014-10-02 08:04:25 -07:00
Vinod Kumar Vavilapalli 34dd2d1d1c YARN-2446. Augmented Timeline service APIs to start taking in domains as a parameter while posting entities and events. Contributed by Zhijie Shen. 
(cherry picked from commit 9e40de6af7)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-01 17:54:12 -07:00
Vinod Kumar Vavilapalli 3326fba382 YARN-1972. Added a secure container-executor for Windows. Contributed by Remus Rusanu. 
commit ba7f31c2ee is the corresponding trunk commit, this is a slightly different patch for branch-2.
 2014-10-01 17:07:21 -07:00
junping_du 625456746c YARN-2613. Support retry in NMClient for rolling-upgrades. (Contributed by Jian He) 2014-10-01 17:08:55 -07:00
Zhijie Shen 4b50e23271 YARN-2630. Prevented previous AM container status from being acquired by the current restarted AM. Contributed by Jian He. 
(cherry picked from commit 52bbe0f11b)
 2014-10-01 15:39:36 -07:00
Vinod Kumar Vavilapalli 64bed0460e YARN-1063. Augmented Hadoop common winutils to have the ability to create containers as domain users. Contributed by Remus Rusanu. 
Committed as a YARN patch even though all the code changes are in common.

(cherry picked from commit 5ca97f1e60)
 2014-10-01 09:55:57 -07:00
Karthik Kambatla 3664659592 YARN-2179. Fix version in sharedcachemanager-pom. 2014-10-01 07:34:24 -07:00
Karthik Kambatla b616c69835 YARN-2179. [YARN-1492] Initial cache manager structure and context. (Chris Trezzo via kasha) 
(cherry picked from commit 17d1202c35)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/bin/yarn
 2014-09-30 19:05:10 -07:00
Jian He 823f02725f YARN-2602. Fixed possible NPE in ApplicationHistoryManagerOnTimelineStore. Contributed by Zhijie Shen 
(cherry picked from commit bbff96be48)
 2014-09-30 16:44:44 -07:00
Karthik Kambatla 5a43b795b7 YARN-2594. Potential deadlock in RM when querying ApplicationResourceUsageReport. (Wangda Tan via kasha) 
(cherry picked from commit 14d60dadc2)
 2014-09-30 16:17:38 -07:00
Jason Lowe a0305ba3fc YARN-2387. Resource Manager crashes with NPE due to lack of synchronization. Contributed by Mit Desai 
(cherry picked from commit feaf139b4f)
 2014-09-30 22:38:27 +00:00
Zhijie Shen d977205b9b YARN-2627. Added the info logs of attemptFailuresValidityInterval and number of previous failed attempts. Contributed by Xuan Gong. 
(cherry picked from commit 9582a50176)
 2014-09-30 12:59:34 -07:00
Karthik Kambatla 21f920ab86 YARN-2610. Hamlet should close table tags. (Ray Chiang via kasha) 
(cherry picked from commit f7743dd07d)
 2014-09-30 11:47:01 -07:00
Arun C. Murthy 6a60fa279a Preparing to branch hadoop-2.6 2014-09-30 10:37:03 -07:00
Jonathan Eagles 9bc676aa6e YARN-2606. Application History Server tries to access hdfs before doing secure login (Mit Desai via jeagles) 2014-09-29 16:40:46 -05:00
Jason Lowe 4f6a367166 YARN-1769. CapacityScheduler: Improve reservations. Contributed by Thomas Graves 
(cherry picked from commit 9c22065109)
 2014-09-29 14:13:21 +00:00
Jian He cb08ed1484 YARN-668. Changed NMTokenIdentifier/AMRMTokenIdentifier/ContainerTokenIdentifier to use protobuf object as the payload. Contributed by Junping Du. 
(cherry picked from commit 5391919b09)
 2014-09-26 17:53:35 -07:00
Allen Wittenauer 6f5d555ad0 YARN-2372. There are Chinese Characters in the FairScheduler's document (Fengdong Yu via aw) 2014-09-26 12:47:26 -07:00
Allen Wittenauer f694b44036 YARN-2577. Clarify ACL delimiter and how to configure ACL groups only (Mikos Christine via aw) 2014-09-26 11:06:04 -07:00
Karthik Kambatla b923c291b4 YARN-2608. FairScheduler: Potential deadlocks in loading alloc files and clock access. (Wei Yan via kasha) 
(cherry picked from commit c9811af09a3d3f9f2f1b86fc9d6f2763d3225e44)
 2014-09-25 17:43:08 -07:00
Jason Lowe e4d46e5ff6 YARN-2523. ResourceManager UI showing negative value for "Decommissioned Nodes" field. Contributed by Rohith 
(cherry picked from commit 8269bfa613)
 2014-09-25 22:39:57 +00:00
Zhijie Shen 83926b3c57 YARN-2546. Made REST API for application creation/submission use numeric and boolean types instead of the string of them. Contributed by Varun Vasudev. 
(cherry picked from commit 72b0881ca6)
 2014-09-24 17:58:48 -07:00
Zhijie Shen 3a2e400377 YARN-2581. Passed LogAggregationContext to NM via ContainerTokenIdentifier. Contributed by Xuan Gong. 
(cherry picked from commit c86674a3a4)
 2014-09-24 17:51:54 -07:00
Karthik Kambatla 30165d4cdd YARN-2596. TestWorkPreservingRMRestart fails with FairScheduler. (kasha) 
(cherry picked from commit 39c87344e1)
 2014-09-24 15:24:17 -07:00
Vinod Kumar Vavilapalli 5d251d99d6 YARN-2102. Added the concept of a Timeline Domain to handle read/write ACLs on Timeline service event data. Contributed by Zhijie Shen. 
(cherry picked from commit d78b452a4f)
 2014-09-24 11:29:01 -07:00
Allen Wittenauer e5a531a8a7 YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) 2014-09-24 08:48:42 -07:00
Zhijie Shen 19cfe2f800 YARN-2569. Added the log handling APIs for the long running services. Contributed by Xuan Gong. 
(cherry picked from commit 5338ac416a)
 2014-09-23 10:38:27 -07:00
Karthik Kambatla 024bcfdc0d YARN-2252. Intermittent failure of TestFairScheduler.testContinuousScheduling. (Ratandeep Ratti and kasha via kasha) 
(cherry picked from commit f5578207d2)
 2014-09-23 00:03:47 -07:00
Karthik Kambatla 8ba8521de5 YARN-1959. Fix headroom calculation in FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit 568d3dc2bb)
 2014-09-22 23:51:45 -07:00
junping_du 34ee084c3d YARN-2584. TestContainerManagerSecurity fails on trunk. (Contributed by Jian He) 
(cherry picked from commit a9a55db065)
 2014-09-22 22:47:59 -07:00
Karthik Kambatla 3984435661 YARN-2539. FairScheduler: Set the default value for maxAMShare to 0.5. (Wei Yan via kasha) 
(cherry picked from commit 43efdd30b5)
 2014-09-22 16:10:20 -07:00