Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,339 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1570 Commits

Author SHA1 Message Date
Tsuyoshi Ozawa 04ad500fac YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. Contributed by Robert Metzger. 
(cherry picked from commit f56da3ce04)
 2015-01-28 00:16:07 +09:00
Jian He 61b4116b4b YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan 
(cherry picked from commit 6f9fe76918)
 2015-01-26 15:38:27 -08:00
Jason Lowe 07fe6a36cb YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native executor returns an error. Contributed by Eric Payne 
(cherry picked from commit 902c6ea7e4)
 2015-01-26 15:41:23 +00:00
Xuan a7696b3fbf YARN-3024. LocalizerRunner should give DIE action when all resources are 
localized. Contributed by Chengbing Liu

(cherry picked from commit 0d6bd62102)
 2015-01-25 19:39:52 -08:00
Tsuyoshi Ozawa 25f2cf8a4e YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. Contributed by Wangda Tan. 
(cherry picked from commit 24aa462673)
 2015-01-23 20:37:26 +09:00
Tsuyoshi Ozawa ff627d94e7 YARN-3082. Non thread safe access to systemCredentials in NodeHeartbeatResponse processing. Contributed by Anubhav Dhoot. 
(cherry picked from commit 3aab354e66)
 2015-01-23 16:05:05 +09:00
Tsuyoshi Ozawa 316eedb55f YARN-3078. LogCLIHelpers lacks of a blank space before string 'does not exist'. Contributed by Sam Liu. 
(cherry picked from commit 5712c9f96a)
 2015-01-22 12:25:29 +09:00
Wangda Tan 610aa71a10 YARN-2731. Fixed RegisterApplicationMasterResponsePBImpl to properly invoke maybeInitBuilder. (Contributed by Carlo Curino) 
(cherry picked from commit f250ad1773)
 2015-01-20 14:49:06 -08:00
Wangda Tan ef6fc24dfb YARN-2933. Capacity Scheduler preemption policy should only consider capacity without labels temporarily. Contributed by Mayank Bansal 
(cherry picked from commit 0a2d3e717d)
 2015-01-19 16:57:17 -08:00
cnauroth 2c4abe9208 YARN-3015. yarn classpath command should support same options as hadoop classpath. Contributed by Varun Saxena. 2015-01-19 15:54:41 -08:00
Akira Ajisaka 401c7d72f5 YARN-3071. Remove invalid char from sample conf in doc of FairScheduler. (Contributed by Masatake Iwasaki) 
(cherry picked from commit 4a5c3a4cfe)
 2015-01-19 21:22:48 +09:00
Junping Du ef4d7b73b9 YARN-3070. TestRMAdminCLI#testHelp fails for transitionToActive command. (Contributed by Junping Du) 
(cherry picked from commit 19cbce3898)
 2015-01-18 23:09:24 -08:00
Karthik Kambatla 4d8fa9615f YARN-2984. Metrics for container's actual memory usage. (kasha) 
(cherry picked from commit 84198564ba)
 2015-01-17 06:26:47 +05:30
Zhijie Shen dcd6eed13a YARN-2815. Excluded transitive dependency of JLine in hadoop-yarn-server-common. Contributed by Ferdinand Xu. 
(cherry picked from commit 43302f6f44)
 2015-01-16 16:38:36 -08:00
Junping Du 7cddec31d7 YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout. (Contributed by Jian He) 
(cherry picked from commit 5d1cca34fa)
 2015-01-16 00:10:36 -08:00
Jian He ed5a0d0aa8 YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. Contributed by Zhijie Shen 
(cherry picked from commit 9e33116d1d)
 2015-01-15 11:08:19 -08:00
Akira Ajisaka b349cd9d0b YARN-3005. [JDK7] Use switch statement for String instead of if-else statement in RegistrySecurity.java (Contributed by Kengo Seki) 
(cherry picked from commit 533e551eb4)
 2015-01-15 21:30:26 +09:00
Karthik Kambatla d98b8f4bd3 YARN-2217. [YARN-1492] Shared cache client side changes. (Chris Trezzo via kasha) 
(cherry picked from commit ba5116ec8e)
 2015-01-15 14:32:35 +05:30
Xuan 79c8f4dcba YARN-2807. Option "--forceactive" not works as described in usage of 
"yarn rmadmin -transitionToActive". Contributed by Masatake Iwasaki

(cherry picked from commit d15cbae73c)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/YarnCommands.apt.vm
 2015-01-14 11:18:10 -08:00
Junping Du 0119e717ae YARN-3019. Make work-preserving-recovery the default mechanism for RM recovery. (Contributed by Jian He) 
(cherry picked from commit f92e503800)
 2015-01-13 18:23:32 -08:00
Jian He 4931600030 YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch 
(cherry picked from commit c53420f583)
 2015-01-13 17:32:36 -08:00
Robert Kanter 13cdcf28e0 YARN-2643. Don't create a new DominantResourceCalculator on every FairScheduler.allocate call. (kasha via rkanter) 
(cherry picked from commit 51881535e6)
 2015-01-12 17:51:45 -08:00
Robert Kanter 7c4e5ff660 YARN-2957. Create unit test to automatically compare YarnConfiguration and yarn-default.xml. (rchiang via rkanter) 
(cherry picked from commit f451631915)
 2015-01-12 12:00:52 -08:00
Robert Kanter e5059b91bb YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter) 
(cherry picked from commit ae7bf31fe1)
 2015-01-12 10:48:45 -08:00
Jian He 36b3dcaab2 YARN-3014. Replaces labels on a host should update all NM's labels on that host. Contributed by Wangda Tan 
(cherry picked from commit a260406268)
 2015-01-09 17:50:18 -08:00
Jian He 68709cb9be YARN-2956. Added missing links in YARN documentation. Contributed by Masatake Iwasaki 
(cherry picked from commit 27a489cba6)
 2015-01-09 17:47:00 -08:00
cnauroth 9a7c763b3f HADOOP-11464. Reinstate support for launching Hadoop processes on Windows using Cygwin. Contributed by Chris Nauroth. 2015-01-09 15:01:33 -08:00
Jian He e7e6173049 YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed container statuses on heartbeat. Contributed by Chengbing Liu 
(cherry picked from commit cc2a745f7e)
 2015-01-08 11:28:24 -08:00
Zhijie Shen d2fbba790a YARN-2996. Improved synchronization and I/O operations of FS- and Mem- RMStateStore. Contributed by Yi Liu. 
(cherry picked from commit dc2eaa26b2)
 2015-01-08 09:49:27 -08:00
Jian He fe664b8b8d YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on getProto method. Contributed by Varun Saxena 
(cherry picked from commit 2638f4d0f0)
 2015-01-07 16:15:03 -08:00
Jian He d3b26454e4 YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. Contributed by Yi Liu 
(cherry picked from commit e13a484a2b)
 2015-01-07 14:12:54 -08:00
Jian He 492717cbf5 YARN-2230. Fixed few configs description in yarn-default.xml. Contributed by Vijay Bhat 
(cherry picked from commit fe8d2bd741)
 2015-01-07 10:57:41 -08:00
Zhijie Shen 8a81312062 YARN-2427. Added the API of moving apps between queues in RM web services. Contributed by Varun Vasudev. 
(cherry picked from commit 60103fca04)
 2015-01-06 14:39:44 -08:00
Jian He c61e8a7bfa YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena 
(cherry picked from commit dd57c2047b)
 2015-01-06 14:16:43 -08:00
Karthik Kambatla 7adffad2bb YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit 0c4b112677)
 2015-01-06 04:42:55 +05:30
Karthik Kambatla d070597a0e YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the scheduler page. (Ashwin Shankar and Wei Yan via kasha) 
(cherry picked from commit 270a271f53)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-01-06 03:28:48 +05:30
Zhijie Shen c6cf748985 YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. Contributed by Varun Saxena. 
(cherry picked from commit 562a701945)
 2015-01-05 13:35:09 -08:00
Tsuyoshi Ozawa c116743bdd YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed by Rohith Sharmaks. 
(cherry picked from commit ddc5be48fc)
 2015-01-05 00:11:01 +09:00
Zhijie Shen efe6260762 YARN-2991. Fixed DrainDispatcher to reuse the draining code path in AsyncDispatcher. Contributed by Rohith Sharmaks. 
(cherry picked from commit 947578c1c1)
 2015-01-02 11:10:46 -08:00
Karthik Kambatla 798ab51289 YARN-2998. Abstract out scheduler independent PlanFollower components. (Anubhav Dhoot via kasha) 
(cherry picked from commit e7257acd8a)
 2014-12-30 19:55:50 -08:00
Jian He cde5bfe3ec YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and app ACLs. Contributed by Varun Saxena 
(cherry picked from commit e2351c7ae2)
 2014-12-30 17:16:05 -08:00
Jian He 143e48c25a YARN-2493. Added node-labels page on RM web UI. Contributed by Wangda Tan 
(cherry picked from commit b7442bf92e)
 2014-12-30 16:49:59 -08:00
Zhijie Shen e5cb52ec1f YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. Contributed by Varun Saxena. 
(cherry picked from commit 241d3b3a50)
 2014-12-29 10:07:54 -08:00
Jian He ca0349b87a YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla 
(cherry picked from commit 1454efe5d4)
 2014-12-26 18:00:32 -08:00
Junping Du fc3e810a19 YARN-2993. Several fixes (missing acl check, error log msg ...) and some refinement in AdminService. (Contributed by Yi Liu) 
(cherry picked from commit 40ee4bff65)
 2014-12-26 07:01:44 -08:00
Tsuyoshi Ozawa 4ca49c8634 YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) 2014-12-25 13:09:51 +09:00
Jian He 60530a6c4f YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith Sharmaks 
(cherry picked from commit 4f18018b7a)
 2014-12-23 22:15:28 -08:00
Zhijie Shen f02bd6683a YARN-2937. Fixed new findbugs warnings in hadoop-yarn-nodemanager. Contributed by Varun Saxena. 
(cherry picked from commit 41a548a916)
 2014-12-23 20:35:32 -08:00
Junping Du 361fab14be YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Contributed by Li Lu) 
(cherry picked from commit d468c9aaf1)
 2014-12-23 20:06:53 -08:00
Jian He e3a54b1b8b YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen 
(cherry picked from commit 149512a837)
 2014-12-23 18:26:37 -08:00
Junping Du ae3faab13e YARN-2970. NodeLabel operations in RMAdmin CLI get missing in help command. (Contributed by Varun Saxena) 
(cherry picked from commit e1ee0d45ea)
 2014-12-23 18:24:45 -08:00
Jian He 8b398a66ca YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharmaks 
(cherry picked from commit 0d89859b51)
 2014-12-22 21:55:06 -08:00
Jian He 411836b74c YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 
(cherry picked from commit fdf042dfff)
 2014-12-22 16:52:15 -08:00
Junping Du 6f252f7f36 YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) 
(cherry picked from commit a696fbb001)
 2014-12-22 03:08:25 -08:00
Karthik Kambatla 2abec14ec6 YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) 
(cherry picked from commit 24ee9e3431)
 2014-12-20 12:20:29 -08:00
Tsuyoshi Ozawa 31753da961 YARN-2977. Fixed intermittent TestNMClient failure. (Contributed by Junping Du) 2014-12-20 22:43:47 +09:00
Jian He 9180d11b3b YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks 
(cherry picked from commit 808cba3821)
 2014-12-19 16:56:55 -08:00
Karthik Kambatla b4e8ae591d YARN-2675. containersKilled metrics is not updated when the container is killed during localization. (Zhihai Xu via kasha) 
(cherry picked from commit 954fb8581e)
 2014-12-19 16:03:02 -08:00
Karthik Kambatla 8ee40a1580 YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit a22ffc3188)
 2014-12-19 15:38:27 -08:00
Jason Lowe 173664d70f YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by Jian He 
(cherry picked from commit 0402bada19)
 2014-12-18 23:31:11 +00:00
Junping Du cec9300201 YARN-2949. Add documentation for CGroups. (Contributed by Varun Vasudev) 
(cherry picked from commit 389f881d42)
 2014-12-18 06:26:53 -08:00
Junping Du 11e4ec52a1 YARN-2972. DelegationTokenRenewer thread pool never expands. Contributed by Jason Lowe 
(cherry picked from commit 2b4b0e8847)
 2014-12-17 17:41:21 -08:00
Karthik Kambatla d0d42c4e32 YARN-2203. [YARN-1492] Web UI for cache manager. (Chris Trezzo via kasha) 
(cherry picked from commit b7f64823e1)
 2014-12-17 16:34:33 -08:00
Karthik Kambatla 375a1e5971 YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. (Chris Trezzo via kasha) 
(cherry picked from commit a1bd140964)
 2014-12-17 16:25:02 -08:00
Karthik Kambatla 2d832ad2eb YARN-2964. FSLeafQueue#assignContainer - document the reason for using both write and read locks. (Tsuyoshi Ozawa via kasha) 
(cherry picked from commit f2d150ea12)
 2014-12-17 16:24:17 -08:00
Karthik Kambatla 300a665307 YARN-2189. Admin service for cache manager. Addendum to fix merge conflicts 2014-12-17 11:47:24 -08:00
cnauroth 36068768d8 HADOOP-11321. copyToLocal cannot save a file to an SMB share unless the user has Full Control permissions. Contributed by Chris Nauroth. 
(cherry picked from commit e996a1bfd4)
 2014-12-16 15:32:23 -08:00
Jian He 878350a0e8 YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments before sending to RM. Contributed by Rohith Sharmaks 
(cherry picked from commit c65f1b382e)
 2014-12-16 11:01:24 -08:00
Karthik Kambatla b1ccf68db3 YARN-2914. [YARN-1492] Potential race condition in Singleton implementation of SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics. (Varun Saxena via kasha) 
(cherry picked from commit e597249d36)
 2014-12-15 12:00:46 -08:00
Jian He d2356a552a Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
(cherry picked from commit a4f2995b9e)
 2014-12-15 11:08:44 -08:00
Devaraj K 4c6760bc6b YARN-2356. yarn status command for non-existent application/application 
attempt/container is too verbose. Contributed by Sunil G.

(cherry picked from commit fae3e8614f)
 2014-12-15 14:46:27 +05:30
Harsh J da36252077 YARN-2950. Change message to mandate, not suggest JS requirement on UI. Contributed by Dustin Cote. 
(cherry picked from commit 0e37bbc8e3)
 2014-12-13 07:27:37 +05:30
Steve Loughran a858d726c8 YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel) 2014-12-12 17:10:54 +00:00
Devaraj K 10ff600b35 YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in 
SchedulerApplicationAttempt ctor. Contributed by Devaraj K.

(cherry picked from commit bda748ac3a)
 2014-12-12 12:37:13 +05:30
Jian He 41f0d20fcb YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher. Contributed by Rohith Sharmaks 
(cherry picked from commit 614b6afea4)
 2014-12-11 11:18:45 -08:00
Jian He b31dfb17d8 YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. Contributed by Wangda Tan 
(cherry picked from commit 437322afca)
 2014-12-09 17:56:41 -08:00
Jian He 0f47e4e745 YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. Contributed by Wangda Tan 
(cherry picked from commit 2ed90a57fd)
 2014-12-09 16:48:46 -08:00
Karthik Kambatla 1986ea8dd2 YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg via kasha) 
(cherry picked from commit a2e07a5456)
 2014-12-09 14:00:52 -08:00
Karthik Kambatla 9d72b0282f YARN-2931. PublicLocalizer may fail until directory is initialized by LocalizeRunner. (Anubhav Dhoot via kasha) 
(cherry picked from commit db73cc9124)
 2014-12-08 22:26:44 -08:00
Karthik Kambatla d02cb9c512 YARN-2927. [YARN-1492] InMemorySCMStore properties are inconsistent. (Ray Chiang via kasha) 
(cherry picked from commit 120e1decd7)
 2014-12-07 22:28:57 -08:00
Jian He 4cc0abe4fe YARN-2869. CapacityScheduler should trim sub queue names when parse configuration. Contributed by Wangda Tan 
(cherry picked from commit e69af836f3)
 2014-12-05 17:34:15 -08:00
Jason Lowe b72fb6c774 YARN-2056. Disable preemption at Queue level. Contributed by Eric Payne 
(cherry picked from commit 4b13082199)
 2014-12-05 21:08:14 +00:00
Robert Kanter 2ebd1bc654 YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YarnConfiguration. (rchiang via rkanter) 
(cherry picked from commit 3c72f54ef5)
 2014-12-05 12:10:20 -08:00
Karthik Kambatla aadd0c392b YARN-2189. [YARN-1492] Admin service for cache manager. (Chris Trezzo via kasha) 
(cherry picked from commit 78968155d7)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/bin/yarn
 2014-12-04 19:48:18 -08:00
Jian He 1d1e7682c9 YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R 
(cherry picked from commit 258623ff8b)
 2014-12-04 12:53:52 -08:00
Jian He 58c971164c YARN-2880. Added a test to make sure node labels will be recovered if RM restart is enabled. Contributed by Rohith Sharmaks 
(cherry picked from commit 73fbb3c66b)
 2014-12-03 17:17:40 -08:00
Harsh J ee25c0d890 YARN-2891. Failed Container Executor does not provide a clear error message. Contributed by Dustin Cote. (harsh) 
(cherry picked from commit 4b13318dea7a1cbbbfc1f84207af829cbe2f720e)
 2014-12-04 03:19:01 +05:30
Karthik Kambatla 25be97808b YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps. (Naganarasimha G R via kasha) 
(cherry picked from commit 799353e2c7)
 2014-12-03 13:45:03 -08:00
Jason Lowe db723a8499 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 
(cherry picked from commit 03ab24aa01)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2014-12-03 17:12:47 +00:00
Junping Du 86535ff65f YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values. (Contributed by Tsuyoshi OZAWA) 
(cherry picked from commit e65b7c5ff6)
 2014-12-03 04:12:35 -08:00
Jian He d6f3d4893d YARN-2894. Fixed a bug regarding application view acl when RM fails over. Contributed by Rohith Sharmaks 
(cherry picked from commit 392c3aaea8)
 2014-12-02 17:17:16 -08:00
Jian He 5067ac098b YARN-2136. Changed RMStateStore to ignore store opearations when fenced. Contributed by Varun Saxena 
(cherry picked from commit 52bcefca8b)
 2014-12-02 11:02:00 -08:00
Zhijie Shen d21ef79707 YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 
(cherry picked from commit 51af8d367d)
 2014-12-01 19:08:31 -08:00
Jian He d208c90148 YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe 
(cherry picked from commit a7fba0bc28)
 2014-12-01 16:45:17 -08:00
Jason Lowe 38ea1419f6 YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted. Contributed by Varun Saxena 
(cherry picked from commit 0f9528b99a)
 2014-12-01 22:30:20 +00:00
Junping Du c7bd22974a YARN-2907. SchedulerNode#toString should print all resource detail instead of only memory. (Contributed by Rohith) 
(cherry picked from commit c732ed760e)
 2014-12-01 05:41:56 -08:00
Zhijie Shen a9adc1295c YARN-2165. Added the sanity check for the numeric configuration values of the timeline service. Contributed by Vasanth kumar RJ. 
(cherry picked from commit 8f1454cc6d)
 2014-11-26 12:12:42 -08:00
Karthik Kambatla 3d4536af82 YARN-2188. [YARN-1492] Client service for cache manager. (Chris Trezzo and Sangjin Lee via kasha) 
(cherry picked from commit 834b3b98e7ecc4bfe93c7dd1a9e928e9653cf138)
 2014-11-25 16:22:14 -08:00
Jian He ae35b0e14d YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. Contributed by Jason Lowe 
(cherry picked from commit 8a7ca13b13)
 2014-11-25 16:18:02 -08:00
Jian He 2863056530 YARN-2404. Removed ApplicationAttemptState and ApplicationState class in RMStateStore. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit 5805a81efb)
 2014-11-25 12:51:03 -08:00