Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,942 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

683 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Junping Du 628c782870 Tests in mapreduce-client-app are writing outside of target. Contributed by Akira AJISAKA. 
(cherry picked from commit 15d577bfbb)
 2015-11-25 09:16:22 -08:00
Akira Ajisaka c6cf7910a4 MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job configuration. Contributed by Gabor Liptak. 
(cherry picked from commit 5ec44fc35c)
 2015-11-25 16:33:18 +09:00
Jason Lowe 2f18218508 MAPREDUCE-5870. Support for passing Job priority through Application Submission Context in Mapreduce Side. Contributed by Sunil G 
(cherry picked from commit f634505d48)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
 2015-11-24 22:15:37 +00:00
Jian He b339a4b8d6 YARN-4349. Support CallerContext in YARN. Contributed by Wangda Tan 
(cherry picked from commit 8676a118a1)
 2015-11-23 17:20:24 -08:00
Jian He 2cf084ac76 Revert "YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan" 
This reverts commit a4ff03b200.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-11-16 20:19:31 -08:00
Jian He f562c155d2 MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API. Contributed by Junping Du 2015-11-16 17:08:22 -08:00
Jian He a4ff03b200 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan 
(cherry picked from commit 8fbea531d7)
 2015-11-09 11:05:12 -08:00
Tsuyoshi Ozawa 1c80473a41 YARN-4320. TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188. Contributed by Varun Saxena. 
(cherry picked from commit ce31b22739)
 2015-10-30 18:00:35 +09:00
Jason Lowe 145058003a MAPREDUCE-6515. Update Application priority in AM side from AM-RM heartbeat. Contributed by Sunil G 
(cherry picked from commit cf953b6258)
 2015-10-29 18:05:56 +00:00
Karthik Kambatla 2d8f9e3fd9 MAPREDUCE-6302. Incorrect headroom can lead to a deadlock between map and reduce allocations. (kasha) 
(cherry picked from commit 4aa9b3e75c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2015-10-09 07:54:47 -07:00
rohithsharmaks 7dd9ebdebc MAPREDUCE-6485. Create a new task attempt with failed map task priority if in-progress attempts are unassigned. (Xianyin Xin via rohithsharmaks) 
(cherry picked from commit 439f43ad3d)
 2015-10-02 20:36:20 +05:30
Akira Ajisaka e42f576232 MAPREDUCE-6497. Fix wrong value of JOB_FINISHED event in JobHistoryEventHandler. Contributed by Shinichi Yamashita. 
(cherry picked from commit 195793c6f3)
 2015-10-01 10:26:00 +02:00
Jason Lowe 3c4c52a095 MAPREDUCE-6492. AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask. Contributed by Bibin A Chundatt 
(cherry picked from commit ab11085b81)
 2015-09-28 22:13:03 +00:00
Jian He d458805690 YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 
(cherry picked from commit 89cab1ba5f)

Conflicts:
	hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
 2015-09-24 10:30:04 -07:00
Jian He c19e8ffdc7 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan) 
(cherry picked from commit 83a18add10)
 2015-09-24 10:23:25 -07:00
Jian He 89dc37450c YARN-3866. AM-RM protocol changes to support container resizing. Contributed by Meng Ding 
(cherry picked from commit dfe2cb849f)
 2015-09-24 10:23:13 -07:00
Tsuyoshi Ozawa 52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 
(cherry picked from commit dfd807afab)
 2015-09-22 12:54:44 +09:00
Zhihai Xu 88a3a07636 MAPREDUCE-6460. TestRMContainerAllocator.testAttemptNotFoundCausesRMCommunicatorException fails. Contributed by Zhihai Xu. 
(cherry picked from commit e3ace31e2b)
 2015-09-19 00:12:15 -07:00
Jason Lowe f27f1ef44c MAPREDUCE-5982. Task attempts that fail from the ASSIGNED state can disappear. Contributed by Chang Li 
(cherry picked from commit ee4ee6af6a)
 2015-09-17 21:38:44 +00:00
Jason Lowe 4861fadb17 MAPREDUCE-5002. AM could potentially allocate a reduce container to a map attempt. Contributed by Chang Li 
(cherry picked from commit 3f82f582e5)
 2015-09-17 18:18:28 +00:00
Vinod Kumar Vavilapalli 9da81a3955 MAPREDUCE-6454. Fixed MapReduce to modify HADOOP_CLASSPATH to have distributed cache files so that child processes running hadoop scripts can access these files. Contributed by Junping Du. 2015-08-20 17:35:24 -07:00
Karthik Kambatla 20734320a1 MAPREDUCE-6439. AM may fail instead of retrying if RM shuts down during the allocate call. (Anubhav Dhoot via kasha) 
(cherry picked from commit 8dfec7a197)
 2015-08-15 00:53:13 -07:00
Karthik Kambatla b826168173 MAPREDUCE-5817. Mappers get rescheduled on node transition even after all reducers are completed. (Sangjin Lee via kasha) 
(cherry picked from commit 27d24f96ab)
 2015-08-14 12:40:38 -07:00
Jason Lowe 508da4d461 MAPREDUCE-6394. Speed up Task processing loop in HsTasksBlock#render(). Contributed by Ray Chiang 
(cherry picked from commit d0e0ba8010)
 2015-07-31 18:19:52 +00:00
Zhihai Xu 9d40eead81 MAPREDUCE-6433. launchTime may be negative. Contributed by Zhihai Xu 
(cherry picked from commit 93d50b7824)
 2015-07-30 23:12:30 -07:00
Chris Douglas 2a0e551cd8 MAPREDUCE-6427. Fix typo in JobHistoryEventHandler. Contributed by Ray Chiang 2015-07-14 14:56:21 -07:00
Akira Ajisaka 0ba2defb9e YARN-3381. Fix typo InvalidStateTransitonException. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 19295b36d9)
 2015-07-13 17:53:27 +09:00
Jason Lowe 8a4970e845 MAPREDUCE-6418. MRApp should not shutdown LogManager during shutdown. Contributed by Chang Li 
(cherry picked from commit eac1d18943)
 2015-07-01 18:02:17 +00:00
Jason Lowe f3195eda27 MAPREDUCE-6420. Interrupted Exception in LocalContainerLauncher should be logged in warn/info level. Contributed by Chang Li 
(cherry picked from commit 826715622e)
 2015-07-01 17:50:47 +00:00
Jason Lowe df4e1e4965 MAPREDUCE-6376. Add avro binary support for jhist files. Contributed by Ray Chiang 
(cherry picked from commit 2ac87df578)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2015-07-01 16:07:54 +00:00
Jason Lowe fdfd5be2b3 MAPREDUCE-6384. Add the last reporting reducer info for too many fetch failure diagnostics. Contributed by Chang Li 
(cherry picked from commit b6ba56457c)
 2015-06-30 21:24:38 +00:00
Gera Shegalov d8cc8f54dd MAPREDUCE-6408. Queue name and user name should be printed on the job page. (Siqi Li via gera) 
(cherry picked from commit 11ac848207)
 2015-06-22 14:42:31 -07:00
Gera Shegalov 34e744ce1a MAPREDUCE-6316. Task Attempt List entries should link to the task overview. (Siqi Li via gera) 
(cherry picked from commit 63d40d5286)
 2015-06-20 11:52:11 -07:00
Devaraj K 5c3070dc16 MAPREDUCE-6405. NullPointerException in App Attempts page. Contributed by 
Siqi Li and Gera Shegalov.

(cherry picked from commit 311a4179cc)
 2015-06-20 10:37:42 +05:30
Gera Shegalov 01991876ed MAPREDUCE-6395. Improve the commit failure messages in MRAppMaster recovery. (Brahma Reddy Battula via gera) 
(cherry picked from commit 10107243be)
 2015-06-19 02:29:31 -07:00
Devaraj K 5426e3e929 MAPREDUCE-6350. JobHistory doesn't support fully-functional search. 
Contributed by Siqi Li.
 2015-06-10 12:43:31 +05:30
Tsuyoshi Ozawa c6cdecc6b3 MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes. Contributed by Ray Chiang. 
(cherry picked from commit 6afe20a7a4)
 2015-06-08 15:06:23 -07:00
Vinod Kumar Vavilapalli 6044b5e5f6 MAPREDUCE-5232. Add a configuration to be able to log classpath and other system properties on mapreduce JVMs startup. Contributed by Sangjin Lee. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1482643 13f79535-47bb-0310-9956-ffa450edef68
(cherry picked from commit 4d8e350750)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java
 2015-06-04 11:15:30 -07:00
Wangda Tan 7cd0874dfd MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) 
(cherry picked from commit 3164e7d838)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2015-05-27 14:43:20 -07:00
cnauroth 6ed8989a6f YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch. 
(cherry picked from commit 7bc33b63c78c6ee9a04c0c1511ef16e63f7f3074)
 2015-05-27 14:21:15 -07:00
Tsuyoshi Ozawa e694c3339e MAPREDUCE-6364. Add a Kill link to Task Attempts page. Contributed by Ryu Kobayashi. 
(cherry picked from commit 022f49d59e)
 2015-05-26 23:59:20 +09:00
Vinod Kumar Vavilapalli 87862970f1 Fixing MR intermediate spills. Contributed by Arun Suresh. 
(cherry picked from commit 6b710a42e0)
 2015-05-14 16:56:56 -07:00
Devaraj K 454236ec19 MAPREDUCE-5708. Duplicate String.format in 
YarnOutputFiles.getSpillFileForWrite. Contributed by Konstantin Weitz.

(cherry picked from commit 05ff54c66c)
 2015-05-14 22:11:15 +05:30
Vinayakumar B cda1962996 MAPREDUCE-6360. TestMapreduceConfigFields is placed in wrong dir, introducing compile error (Contributed by Arshad Mohammad) 
(cherry picked from commit 360dff5903)
 2015-05-12 12:50:44 +05:30
Jason Lowe 53e9973afc MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma 2015-05-11 22:40:05 +00:00
Karthik Kambatla 52469b10dd MAPREDUCE-6353. Divide by zero error in MR AM when calculating available containers. (Anubhav Dhoot via kasha) 
(cherry picked from commit 1773aac780)
 2015-05-09 14:43:57 -07:00
Karthik Kambatla e8dd0bc4fd YARN-1287. Consolidate MockClocks. (Sebastian Wong and Anubhav Dhoot via kasha) 
(cherry picked from commit 70fb37cd79)
 2015-05-09 14:35:13 -07:00
Junping Du 39c188449e MAPREDUCE-6359. In RM HA setup, Cluster tab links populated with AM hostname instead of RM. Contributed by zhaoyunjiong. 
(cherry picked from commit df36ad0a08)
 2015-05-09 06:12:01 -07:00
Devaraj K ade5165ecf MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. 2015-05-08 15:20:35 +05:30
Akira Ajisaka 5cc036d473 MAPREDUCE-6079. Rename JobImpl#username to reporterUserName. Contributed by Tsuyoshi Ozawa. 
(cherry picked from commit ecfa052274)
 2015-05-08 18:17:35 +09:00
Tsuyoshi Ozawa 1ae8bf5b21 MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. 
(cherry picked from commit d18f10ad1b)
 2015-05-08 16:00:41 +09:00
Jason Lowe 976a3c1f9d MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne 
(cherry picked from commit f30065c8b6)
 2015-05-07 22:06:54 +00:00
Akira Ajisaka f7c011cbaa MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. 
(cherry picked from commit d335071398)
 2015-05-07 19:41:44 +09:00
Robert Kanter bd207b6cc0 MAPREDUCE-6192. Create unit test to automatically compare MR related classes and mapred-default.xml (rchiang via rkanter) 2015-05-05 14:47:52 -07:00
Xuan 02cdcaec58 YARN-2123. Progress bars in Web UI always at 100% due to non-US locale. 
Contributed by Akira AJISAKA

(cherry picked from commit b7dd3a4f04)
 2015-05-05 10:45:18 -07:00
Jason Lowe 81f128f293 MAPREDUCE-6259. IllegalArgumentException due to missing job submit time. Contributed by zhihai xu 
(cherry picked from commit bf70c5ae28)
 2015-05-04 20:40:16 +00:00
Devaraj K c3d083a9ff MAPREDUCE-6339. Job history file is not flushed correctly because 
isTimerActive flag is not set true when flushTimerTask is scheduled.
Contributed by zhihai xu.

(cherry picked from commit f5b38477f9)
 2015-04-30 12:36:00 +05:30
Vinod Kumar Vavilapalli 32dc13d907 MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM tokens when they roll-over. Contributed by Jason Lowe. 
(cherry picked from commit 9fc32c5c4d)
 2015-04-27 14:59:57 -07:00
Gera Shegalov fa915f73e2 MAPREDUCE-6333. TestEvents,TestAMWebServicesTasks,TestAppController are broken due to MAPREDUCE-6297. (Siqi Li via gera) 
(cherry picked from commit 78c6b46241)
 2015-04-24 17:38:16 -07:00
Gera Shegalov 6f5dc9f5af MAPREDUCE-6293. Set job classloader on uber-job's LocalContainerLauncher event thread. (Sangjin Lee via gera) 
(cherry picked from commit 725eb52ddc)
 2015-04-21 14:21:56 -07:00
Akira Ajisaka 51c458a5c4 MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 1fa8075794)
 2015-04-16 10:06:03 -05:00
Akira Ajisaka 4801b0b478 Revert "MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li." 
This reverts commit 3de0bf9a35.
 2015-04-11 12:37:44 +09:00
Akira Ajisaka 3de0bf9a35 MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li. 
(cherry picked from commit 4cec9975c0e2c2fcca090888dc2c5e149c5be1e9)
 2015-04-11 01:31:44 +09:00
Hitesh Shah 55b794e7fa YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai. 
(cherry picked from commit 265ed1fe80)
 2015-04-08 14:14:06 -07:00
Jason Lowe 24d8c6f355 MAPREDUCE-6277. Job can post multiple history files if attempt loses connection to the RM. Contributed by Chang Li 
(cherry picked from commit 30da99cbaf)
 2015-03-18 19:31:32 +00:00
Tsuyoshi Ozawa 10745fcecd MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers. Contributed by Zhihai Xu 
(cherry picked from commit 9d38520c8e)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2015-03-14 16:45:23 +09:00
Jonathan Eagles 44aedad5dd YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles) 
(cherry picked from commit 8180e676ab)
 2015-03-13 12:05:21 -05:00
Tsuyoshi Ozawa b46f9e72db HADOOP-11602. Backport 'Fix toUpperCase/toLowerCase to use Locale.ENGLISH.' (ozawa) 2015-03-09 19:53:55 +09:00
Vinod Kumar Vavilapalli d9416317a9 Preparing for 2.8.0 development. Changing SNAPSHOT version to be 2.8 on branch-2. 2015-03-08 20:32:51 -07:00
Tsuyoshi Ozawa ccde4aed18 MAPREDUCE-5657. Backport '[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments.' Contributed by Akira AJISAKA. 2015-03-09 12:00:04 +09:00
Tsuyoshi Ozawa ac874764d5 MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 348208014b)
 2015-03-05 14:13:13 +09:00
Junping Du a9613bcc94 Ability to limit running map and reduce tasks. Contributed by Jason Lowe. 
(cherry picked from commit 4228de9402)

Conflicts:
    hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2015-03-03 02:23:00 -08:00
Akira Ajisaka 5e686dae44 MAPREDUCE-6264. Remove httpclient dependency from hadoop-mapreduce-client. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 3f665aa4d5ac2e737e29ff63b3837c083a49ea80)
 2015-02-24 11:33:05 -08:00
Tsuyoshi Ozawa 7386323735 MAPREDUCE-6255. Fix JobCounter's format to use grouping separator. Contributed by Ryu Kobayashi. 
(cherry picked from commit ba3c80a5ca)
 2015-02-13 16:11:16 +09:00
Devaraj K a00c9362e6 MAPREDUCE-6253. Update use of Iterator to Iterable. Contributed by Ray 
Chiang.

(cherry picked from commit 76e309ead0)
 2015-02-12 13:47:19 +05:30
Jason Lowe eb927a1d6b MAPREDUCE-6186. Redundant call to requireJob() while displaying the conf page. Contributed by Rohit Agarwal 
(cherry picked from commit e1990ab427)
 2015-02-05 20:14:13 +00:00
Robert Kanter a3a7efe748 MAPREDUCE-6143. add configuration for mapreduce speculative execution in MR2 (zxu via rkanter) 
(cherry picked from commit 8acc5e9b4b)
 2015-02-02 13:52:29 -08:00
Akira Ajisaka 1081b4c4a7 HADOOP-9907. Webapp http://hostname:port/metrics link is not working. (aajisaka) 
(cherry picked from commit 342efa110a)
 2015-01-30 02:50:46 +09:00
Jian He 43b3b43cea MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken service name properly. Contributed by Jason Lowe 
(cherry picked from commit cff05bff1f)
 2015-01-28 15:52:11 -08:00
Akira Ajisaka 2e5eef9da2 MAPREDUCE-6210. Use getApplicationAttemptId() instead of getApplicationID() for logging AttemptId in RMContainerAllocator.java (Contributed by Leitao Guo) 
(cherry picked from commit d336d13678)
 2015-01-14 17:43:25 +09:00
Jian He d2356a552a Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
(cherry picked from commit a4f2995b9e)
 2014-12-15 11:08:44 -08:00
Devaraj K 25f4a0193b MAPREDUCE-6046. Change the class name for logs in RMCommunicator. 
Contributed by Sahil Takiar.

(cherry picked from commit 0bd0229110)
 2014-12-12 11:44:00 +05:30
Haohui Mai 759179e631 HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, non-core directories. Contributed by Li Lu. 2014-12-09 10:46:25 -08:00
Jason Lowe db723a8499 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 
(cherry picked from commit 03ab24aa01)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2014-12-03 17:12:47 +00:00
Zhijie Shen d21ef79707 YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 
(cherry picked from commit 51af8d367d)
 2014-12-01 19:08:31 -08:00
Jason Lowe b6d7b789bd MAPREDUCE-6160. Potential NullPointerException in MRClientProtocol interface implementation. Contributed by Rohith 
(cherry picked from commit 0c588904f8)
 2014-12-01 22:40:38 +00:00
Jonathan Eagles 6570f7fd72 YARN-2375. Allow enabling/disabling timeline server per framework. (Mit Desai via jeagles) 
(cherry picked from commit c298a9a845)
 2014-11-20 23:40:26 -06:00
Devaraj K 09df35e51d MAPREDUCE-6049. AM JVM does not exit if MRClientService graceful shutdown 
fails. Contributed by Rohith.
(cherry picked from commit 7250b0bf91)
 2014-11-18 22:07:26 +05:30
Jason Lowe 0b9a1a3e03 MAPREDUCE-6162. mapred hsadmin fails on a secure cluster. Contributed by Jason Lowe 
(cherry picked from commit 09b3dee122)
 2014-11-17 23:01:16 +00:00
Arun C. Murthy adfb830a2b Preparing to release hadoop-2.6.0: Set version in branch-2 to 2.7.0-SNAPSHOT. 2014-11-09 19:19:02 -08:00
Arun C. Murthy 175d222bfc YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 
(cherry picked from commit 43cd07b408)
 2014-11-09 15:03:59 -08:00
Jason Lowe c0bbb72912 MAPREDUCE-5960. JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar path. Contributed by Gera Shegalov 
(cherry picked from commit 10f9f5101c)
 2014-11-06 15:12:06 +00:00
Zhijie Shen c4aaa4db6c MAPREDUCE-6052. Supported overriding the default container-log4j.properties file per job. Contributed by Junping Du. 
(cherry picked from commit ed63b11646)
 2014-11-01 00:49:38 -07:00
Vinod Kumar Vavilapalli 063bb0508b MAPREDUCE-6142. Fixed test failures in TestJobHistoryEventHandler and TestMRTimelineEventHandling. Contributed by Zhijie Shen. 
(cherry picked from commit 3f1441245d)
 2014-10-29 09:58:50 -07:00
Zhijie Shen 52695d38fc MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. 
(cherry picked from commit 971e91c8c0)
 2014-10-27 21:08:38 -07:00
Zhijie Shen 733380fa70 MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. 
(cherry picked from commit 6b2f11b54b)
 2014-10-27 20:41:02 -07:00
Zhijie Shen 3b03ea6b50 YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He. 
(cherry picked from commit 0f3b6900be)
 2014-10-23 22:03:30 -07:00
Jian He f93d2ea27e YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:28:26 -07:00
Jason Lowe e17e3bdbc4 MAPREDUCE-6125. TestContainerLauncherImpl sometimes fails. Contributed by Mit Desai 
(cherry picked from commit bbe80cdc7b)
 2014-10-13 15:09:26 +00:00
Karthik Kambatla 7bfd9e068d MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha) 
(cherry picked from commit ed0e0ef9748ce5b231de677ab41c3035137bbde4)
 2014-10-11 22:49:46 -07:00
Jason Lowe 979b915ecc MAPREDUCE-6029. TestCommitterEventHandler fails in trunk. Contributed by Mit Desai 
(cherry picked from commit 2e789eb226)
 2014-10-07 14:11:48 +00:00
Zhijie Shen b8d645539d MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN scheduler resource type is memory plus cpu. Contributed by Peng Zhang and Varun Vasudev. 2014-09-22 10:41:55 -07:00
Jason Lowe f4534746fb MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs. Contributed by Gera Shegalov 
(cherry picked from commit 7039b98e1c)
 2014-09-22 15:22:58 +00:00
Zhijie Shen 9333ee3068 YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He. 
(cherry picked from commit 6fe5c6b746)
 2014-09-18 22:02:53 -07:00
Vinod Kumar Vavilapalli 5cdb24d4b1 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 
(cherry picked from commit 14e2639fd0)
 2014-09-13 18:05:02 -07:00
Jason Lowe 0b3f918b3b MAPREDUCE-6071. JobImpl#makeUberDecision doesn't log that Uber mode is disabled because of too much CPUs. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit 45efc966ee)
 2014-09-05 13:48:16 +00:00
Chris Douglas 4f95017602 Fix typos in log messages. Contributed by Ray Chiang 2014-08-28 16:30:13 -07:00
Zhijie Shen 7cad25ec94 MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir creation on Windows. Contributed by Zhijie Shen. 
svn merge --ignore-ancestry -c 1619863 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619864 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 17:08:17 +00:00
Jason Darrell Lowe e06838038e svn merge -c 1618691 FIXES: MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2. Contributed by chang li 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618692 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 18:19:09 +00:00
Zhijie Shen 570183a437 MAPREDUCE-6024. Shortened the time when Fetcher is stuck in retrying before concluding the failure by configuration. Contributed by Yunjiong Zhao. 
svn merge --ignore-ancestry -c 1618677 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618679 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 17:59:32 +00:00
Eric Yang 0b20086da1 MAPREDUCE-6033. Updated access check for displaying job information 
(Yu Gao via Eric Yang)



git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618649 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-18 16:14:27 +00:00
Zhijie Shen a7a79a198b MAPREDUCE-6032. Made MR jobs write job history files on the default FS when the current context’s FS is different. Contributed by Benjamin Zhitomirsky. 
svn merge --ignore-ancestry -c 1618269 ../../trunk/
svn merge --ignore-ancestry -c 1618270 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618271 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-15 20:33:55 +00:00
Xuan Gong 4365c4530b YARN-2212 ApplicationMaster needs to find a way to update the AMRMToken periodically. Contributed by Xuan Gong 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1616891 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-08 21:37:35 +00:00
Jason Darrell Lowe 24346da6b6 svn merge -c 1616018 FIXES: MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1616020 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-05 21:59:01 +00:00
Xuan Gong ecc257e384 YARN-1994. Expose YARN/MR endpoints on multiple interfaces. Contributed by Craig Welch, Milan Potocnik and Arpit Agarwal 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1615010 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-31 21:29:01 +00:00
Zhijie Shen 8cde949397 MAPREDUCE-6002. Made MR task avoid reporting error to AM when the task process is shutting down. Contributed by Wangda Tan. 
svn merge --ignore-ancestry -c 1613743 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1613744 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-27 01:41:44 +00:00
Jason Darrell Lowe 7c44536678 svn merge -c 1612358 FIXES: MAPREDUCE-5957. AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used. Contributed by Sangjin Lee 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1612371 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-21 18:22:45 +00:00
Jian He 99805f5eac Merge r1611434 from trunk. MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving RM-restart. Contributed by Rohith 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1611436 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-17 19:01:27 +00:00
Karthik Kambatla 4c7db3c6a3 MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1611197 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-16 21:30:19 +00:00
Zhijie Shen bba266610c MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current attempt is the last retry. Contributed by Wangda Tan. 
svn merge --ignore-ancestry -c 1609649 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1609650 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-11 08:49:54 +00:00
Vinod Kumar Vavilapalli fd6f4f72e4 MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side resource configuration for deciding uber-mode on map-only jobs. Contributed by Siqi Li. 
svn merge --ignore-ancestry -c 1608595 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1608596 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-07 20:14:33 +00:00
Karthik Kambatla 9133e3142c Preparing for 2.6.0 development. Updated versions. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1607541 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-03 06:11:57 +00:00
Zhijie Shen 38bb24538d MAPREDUCE-5900. Changed to the interpret container preemption exit code as a task attempt killing event. Contributed by Mayank Bansal. 
svn merge --ignore-ancestry -c 1607512 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1607513 13f79535-47bb-0310-9956-ffa450edef68
 2014-07-03 01:45:53 +00:00
Vinod Kumar Vavilapalli a2e2c8ad97 YARN-2152. Added missing information into ContainerTokenIdentifier so that NodeManagers can report the same to RM when RM restarts. Contributed Jian He. 
svn merge --ignore-ancestry -c 1605205 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1605206 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-24 21:44:00 +00:00
Karthik Kambatla ff2886e909 MAPREDUCE-5844. Add a configurable delay to reducer-preemption. (Maysam Yabandeh via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1604032 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-19 21:13:14 +00:00
Jian He d3462755bb Merge r1603338 from trunk. MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event at COMMIT_PENDING state. Contributed by Zhijie Shen 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1603339 13f79535-47bb-0310-9956-ffa450edef68
 2014-06-18 00:53:23 +00:00
Junping Du 6d3bfa33c9 Merge r1594795 from trunk: MAPREDUCE-5861. finishedSubMaps field in LocalContainerLauncher does not need to be volatile. (Contributed by Tsuyoshi OZAWA) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1594797 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-15 07:10:42 +00:00
Jonathan Turner Eagles f8dad8a751 MAPREDUCE-5888. Failed job leaves hung AM after it unregisters (Jason Lowe via jeagles) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1594318 13f79535-47bb-0310-9956-ffa450edef68
 2014-05-13 18:28:07 +00:00
Jason Darrell Lowe 3434d7b78b svn merge -c 1590668 FIXES: MAPREDUCE-5812. Make job context available to OutputCommitter.isRecoverySupported(). Contributed by Mohammad Kamrul Islam 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590675 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-28 15:27:01 +00:00
Eric Yang fef656527a MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob. 
(Jinghui Wang via Eric Yang)


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590267 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-26 18:02:19 +00:00
Jason Darrell Lowe 2a76cade32 svn merge -c 1590168 FIXES: MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state. Contributed by Ming Ma 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590174 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-25 22:34:44 +00:00
Jason Darrell Lowe aa3d6ed6ea svn merge -c 1590046 FIXES: MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed. Contributed by Jason Lowe 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590047 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-25 14:56:35 +00:00
Haohui Mai 1566fff15f MAPREDUCE-5837. Merge r1589903 from trunk. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589904 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-24 22:55:25 +00:00
Jason Darrell Lowe c02ab95ee6 svn merge -c 1589524 FIXES: MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill. Contributed by Sangjin Lee 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589526 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-23 21:57:55 +00:00
Chris Nauroth 2d8f62f5c1 MAPREDUCE-5833. TestRMContainerAllocator fails ocassionally. Contributed by Zhijie Shen. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589259 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-22 19:32:17 +00:00
Chris Nauroth f0d3664d83 MAPREDUCE-5852. Merging change r1589006 from trunk to branch-2. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589007 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-21 23:49:28 +00:00
Jason Darrell Lowe c8d5d1f82a svn merge -c 1588559 FIXES: MAPREDUCE-4937. MR AM handles an oversized split metainfo file poorly. Contributed by Eric Payne 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1588561 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-18 20:40:00 +00:00
Vinod Kumar Vavilapalli 7eff6cc305 MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka. 
svn merge --ignore-ancestry -c 1586559 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1586560 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-11 04:02:19 +00:00
Jason Darrell Lowe 58aefafe76 svn merge -c 1586479 FIXES: MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down. Contributed by Gera Shegalov 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1586482 13f79535-47bb-0310-9956-ffa450edef68
 2014-04-10 22:09:16 +00:00
Arun Murthy e052a4579f Merge -c 1583050 from trunk to branch-2 to fix MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1583051 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-29 19:16:03 +00:00
Vinod Kumar Vavilapalli e51fbb6a9e MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong. 
svn merge --ignore-ancestry -c 1581180 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1581181 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-25 02:01:13 +00:00
Vinod Kumar Vavilapalli 751bba3d14 MAPREDUCE-5806. Fixed a bug in MRAppMaster so as to enable users to properly override HADOOP_ROOT_LOGGER or HADOOP_CLIENT_OPTS. Contributed by Varun Vasudev. 
svn merge --ignore-ancestry -c 1580100 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1580101 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-22 00:33:21 +00:00
Vinod Kumar Vavilapalli d470c7b71a YARN-1824. Improved NodeManager and clients to be able to handle cross platform application submissions. Contributed by Jian He. 
MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to handle cross platform application submissions. Contributed by Jian He.
svn merge --ignore-ancestry -c 1578135 ../../trunk/ with a couple of minor edits for working in branch-2.


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1578139 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-16 19:13:16 +00:00
Jason Darrell Lowe c7a6775d94 svn merge -c 1577692 FIXES: MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt finish time. Contributed by Rushabh S Shah 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1577696 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-14 20:37:40 +00:00
Jason Darrell Lowe fd118493c6 svn merge -c 1577647 FIXES: MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed before registering with RM. Contributed by Rohith 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1577648 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-14 18:20:04 +00:00
Jason Darrell Lowe 838b4d3f84 MAPREDUCE-5751. MR app master fails to start in some cases if mapreduce.job.classloader is true. Contributed by Sangjin Lee 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1577555 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-14 14:47:37 +00:00
Arun Murthy 9e695db974 Merge -c 1576946 from trunk to branch-2 to fix MAPREDUCE-5553. Allow users to easily access completed/pending/successful/failed tasks on MR AM web-ui. Contributed by Paul Han. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1576947 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-12 22:05:54 +00:00
Jason Darrell Lowe 804f86f42c svn merge -c 1576940 FIXES: MAPREDUCE-5778. JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1576943 13f79535-47bb-0310-9956-ffa450edef68
 2014-03-12 21:59:50 +00:00
Vinod Kumar Vavilapalli 04474afc81 MAPREDUCE-5770. Fixed MapReduce ApplicationMaster to correctly redirect to the YARN's web-app proxy with the correct scheme prefix. Contributed by Jian He. 
svn merge --ignore-ancestry -c 1572711 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1572712 13f79535-47bb-0310-9956-ffa450edef68
 2014-02-27 19:24:49 +00:00