b8d645539d
MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN scheduler resource type is memory plus cpu. Contributed by Peng Zhang and Varun Vasudev.
2014-09-22 10:41:55 -07:00
f4534746fb
MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs. Contributed by Gera Shegalov
...
(cherry picked from commit 7039b98e1c
2014-09-22 15:22:58 +00:00
9333ee3068
YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He.
...
(cherry picked from commit 6fe5c6b746
2014-09-18 22:02:53 -07:00
5cdb24d4b1
YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong.
...
(cherry picked from commit 14e2639fd0
2014-09-13 18:05:02 -07:00
0b3f918b3b
MAPREDUCE-6071. JobImpl#makeUberDecision doesn't log that Uber mode is disabled because of too much CPUs. Contributed by Tsuyoshi OZAWA
...
(cherry picked from commit 45efc966ee
2014-09-05 13:48:16 +00:00
4f95017602
Fix typos in log messages. Contributed by Ray Chiang
2014-08-28 16:30:13 -07:00
7cad25ec94
MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir creation on Windows. Contributed by Zhijie Shen.
...
svn merge --ignore-ancestry -c 1619863 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619864 13f79535-47bb-0310-9956-ffa450edef68
2014-08-22 17:08:17 +00:00
e06838038e
svn merge -c 1618691 FIXES: MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2. Contributed by chang li
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618692 13f79535-47bb-0310-9956-ffa450edef68
2014-08-18 18:19:09 +00:00
570183a437
MAPREDUCE-6024. Shortened the time when Fetcher is stuck in retrying before concluding the failure by configuration. Contributed by Yunjiong Zhao.
...
svn merge --ignore-ancestry -c 1618677 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618679 13f79535-47bb-0310-9956-ffa450edef68
2014-08-18 17:59:32 +00:00
0b20086da1
MAPREDUCE-6033. Updated access check for displaying job information
...
(Yu Gao via Eric Yang)
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618649 13f79535-47bb-0310-9956-ffa450edef68
2014-08-18 16:14:27 +00:00
a7a79a198b
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when the current context’s FS is different. Contributed by Benjamin Zhitomirsky.
...
svn merge --ignore-ancestry -c 1618269 ../../trunk/
svn merge --ignore-ancestry -c 1618270 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1618271 13f79535-47bb-0310-9956-ffa450edef68
2014-08-15 20:33:55 +00:00
4365c4530b
YARN-2212 ApplicationMaster needs to find a way to update the AMRMToken periodically. Contributed by Xuan Gong
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1616891 13f79535-47bb-0310-9956-ffa450edef68
2014-08-08 21:37:35 +00:00
24346da6b6
svn merge -c 1616018 FIXES: MAPREDUCE-6014. New task status field in task attempts table can lead to an empty web page. Contributed by Mit Desai
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1616020 13f79535-47bb-0310-9956-ffa450edef68
2014-08-05 21:59:01 +00:00
ecc257e384
YARN-1994. Expose YARN/MR endpoints on multiple interfaces. Contributed by Craig Welch, Milan Potocnik and Arpit Agarwal
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1615010 13f79535-47bb-0310-9956-ffa450edef68
2014-07-31 21:29:01 +00:00
8cde949397
MAPREDUCE-6002. Made MR task avoid reporting error to AM when the task process is shutting down. Contributed by Wangda Tan.
...
svn merge --ignore-ancestry -c 1613743 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1613744 13f79535-47bb-0310-9956-ffa450edef68
2014-07-27 01:41:44 +00:00
7c44536678
svn merge -c 1612358 FIXES: MAPREDUCE-5957. AM throws ClassNotFoundException with job classloader enabled if custom output format/committer is used. Contributed by Sangjin Lee
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1612371 13f79535-47bb-0310-9956-ffa450edef68
2014-07-21 18:22:45 +00:00
99805f5eac
Merge r1611434 from trunk. MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving RM-restart. Contributed by Rohith
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1611436 13f79535-47bb-0310-9956-ffa450edef68
2014-07-17 19:01:27 +00:00
4c7db3c6a3
MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly assumes a single dir for mapOutIndex. (Gera Shegalov via kasha)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1611197 13f79535-47bb-0310-9956-ffa450edef68
2014-07-16 21:30:19 +00:00
bba266610c
MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current attempt is the last retry. Contributed by Wangda Tan.
...
svn merge --ignore-ancestry -c 1609649 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1609650 13f79535-47bb-0310-9956-ffa450edef68
2014-07-11 08:49:54 +00:00
fd6f4f72e4
MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side resource configuration for deciding uber-mode on map-only jobs. Contributed by Siqi Li.
...
svn merge --ignore-ancestry -c 1608595 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1608596 13f79535-47bb-0310-9956-ffa450edef68
2014-07-07 20:14:33 +00:00
9133e3142c
Preparing for 2.6.0 development. Updated versions.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1607541 13f79535-47bb-0310-9956-ffa450edef68
2014-07-03 06:11:57 +00:00
38bb24538d
MAPREDUCE-5900. Changed to the interpret container preemption exit code as a task attempt killing event. Contributed by Mayank Bansal.
...
svn merge --ignore-ancestry -c 1607512 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1607513 13f79535-47bb-0310-9956-ffa450edef68
2014-07-03 01:45:53 +00:00
a2e2c8ad97
YARN-2152. Added missing information into ContainerTokenIdentifier so that NodeManagers can report the same to RM when RM restarts. Contributed Jian He.
...
svn merge --ignore-ancestry -c 1605205 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1605206 13f79535-47bb-0310-9956-ffa450edef68
2014-06-24 21:44:00 +00:00
ff2886e909
MAPREDUCE-5844. Add a configurable delay to reducer-preemption. (Maysam Yabandeh via kasha)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1604032 13f79535-47bb-0310-9956-ffa450edef68
2014-06-19 21:13:14 +00:00
d3462755bb
Merge r1603338 from trunk. MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event at COMMIT_PENDING state. Contributed by Zhijie Shen
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1603339 13f79535-47bb-0310-9956-ffa450edef68
2014-06-18 00:53:23 +00:00
6d3bfa33c9
Merge r1594795 from trunk: MAPREDUCE-5861. finishedSubMaps field in LocalContainerLauncher does not need to be volatile. (Contributed by Tsuyoshi OZAWA)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1594797 13f79535-47bb-0310-9956-ffa450edef68
2014-05-15 07:10:42 +00:00
f8dad8a751
MAPREDUCE-5888. Failed job leaves hung AM after it unregisters (Jason Lowe via jeagles)
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1594318 13f79535-47bb-0310-9956-ffa450edef68
2014-05-13 18:28:07 +00:00
3434d7b78b
svn merge -c 1590668 FIXES: MAPREDUCE-5812. Make job context available to OutputCommitter.isRecoverySupported(). Contributed by Mohammad Kamrul Islam
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590675 13f79535-47bb-0310-9956-ffa450edef68
2014-04-28 15:27:01 +00:00
fef656527a
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
...
(Jinghui Wang via Eric Yang)
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590267 13f79535-47bb-0310-9956-ffa450edef68
2014-04-26 18:02:19 +00:00
2a76cade32
svn merge -c 1590168 FIXES: MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state. Contributed by Ming Ma
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590174 13f79535-47bb-0310-9956-ffa450edef68
2014-04-25 22:34:44 +00:00
aa3d6ed6ea
svn merge -c 1590046 FIXES: MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed. Contributed by Jason Lowe
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1590047 13f79535-47bb-0310-9956-ffa450edef68
2014-04-25 14:56:35 +00:00
1566fff15f
MAPREDUCE-5837. Merge r1589903 from trunk.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589904 13f79535-47bb-0310-9956-ffa450edef68
2014-04-24 22:55:25 +00:00
c02ab95ee6
svn merge -c 1589524 FIXES: MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill. Contributed by Sangjin Lee
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589526 13f79535-47bb-0310-9956-ffa450edef68
2014-04-23 21:57:55 +00:00
2d8f62f5c1
MAPREDUCE-5833. TestRMContainerAllocator fails ocassionally. Contributed by Zhijie Shen.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589259 13f79535-47bb-0310-9956-ffa450edef68
2014-04-22 19:32:17 +00:00
f0d3664d83
MAPREDUCE-5852. Merging change r1589006 from trunk to branch-2.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1589007 13f79535-47bb-0310-9956-ffa450edef68
2014-04-21 23:49:28 +00:00
c8d5d1f82a
svn merge -c 1588559 FIXES: MAPREDUCE-4937. MR AM handles an oversized split metainfo file poorly. Contributed by Eric Payne
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1588561 13f79535-47bb-0310-9956-ffa450edef68
2014-04-18 20:40:00 +00:00
7eff6cc305
MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster gracefully handle empty-queue names. Contributed by Akira Ajisaka.
...
svn merge --ignore-ancestry -c 1586559 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1586560 13f79535-47bb-0310-9956-ffa450edef68
2014-04-11 04:02:19 +00:00
58aefafe76
svn merge -c 1586479 FIXES: MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down. Contributed by Gera Shegalov
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1586482 13f79535-47bb-0310-9956-ffa450edef68
2014-04-10 22:09:16 +00:00
e052a4579f
Merge -c 1583050 from trunk to branch-2 to fix MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the local-filesystem, rather than rely on the classpath. Contributed by Gera Shegalov.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1583051 13f79535-47bb-0310-9956-ffa450edef68
2014-03-29 19:16:03 +00:00
e51fbb6a9e
MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it recovers from a commit during a previous attempt. Contributed by Xuan Gong.
...
svn merge --ignore-ancestry -c 1581180https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1581181 13f79535-47bb-0310-9956-ffa450edef68
2014-03-25 02:01:13 +00:00
751bba3d14
MAPREDUCE-5806. Fixed a bug in MRAppMaster so as to enable users to properly override HADOOP_ROOT_LOGGER or HADOOP_CLIENT_OPTS. Contributed by Varun Vasudev.
...
svn merge --ignore-ancestry -c 1580100https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1580101 13f79535-47bb-0310-9956-ffa450edef68
2014-03-22 00:33:21 +00:00
d470c7b71a
YARN-1824. Improved NodeManager and clients to be able to handle cross platform application submissions. Contributed by Jian He.
...
MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to handle cross platform application submissions. Contributed by Jian He.
svn merge --ignore-ancestry -c 1578135 ../../trunk/ with a couple of minor edits for working in branch-2.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1578139 13f79535-47bb-0310-9956-ffa450edef68
2014-03-16 19:13:16 +00:00
c7a6775d94
svn merge -c 1577692 FIXES: MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt finish time. Contributed by Rushabh S Shah
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1577696 13f79535-47bb-0310-9956-ffa450edef68
2014-03-14 20:37:40 +00:00
fd118493c6
svn merge -c 1577647 FIXES: MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed before registering with RM. Contributed by Rohith
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1577648 13f79535-47bb-0310-9956-ffa450edef68
2014-03-14 18:20:04 +00:00
838b4d3f84
MAPREDUCE-5751. MR app master fails to start in some cases if mapreduce.job.classloader is true. Contributed by Sangjin Lee
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1577555 13f79535-47bb-0310-9956-ffa450edef68
2014-03-14 14:47:37 +00:00
9e695db974
Merge -c 1576946 from trunk to branch-2 to fix MAPREDUCE-5553. Allow users to easily access completed/pending/successful/failed tasks on MR AM web-ui. Contributed by Paul Han.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1576947 13f79535-47bb-0310-9956-ffa450edef68
2014-03-12 22:05:54 +00:00
804f86f42c
svn merge -c 1576940 FIXES: MAPREDUCE-5778. JobSummary does not escape newlines in the job name. Contributed by Akira AJISAKA
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1576943 13f79535-47bb-0310-9956-ffa450edef68
2014-03-12 21:59:50 +00:00
04474afc81
MAPREDUCE-5770. Fixed MapReduce ApplicationMaster to correctly redirect to the YARN's web-app proxy with the correct scheme prefix. Contributed by Jian He.
...
svn merge --ignore-ancestry -c 1572711 ../../trunk/
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1572712 13f79535-47bb-0310-9956-ffa450edef68
2014-02-27 19:24:49 +00:00
c5af93cd47
MAPREDUCE-5766. Moved ping messages from TaskAttempts to be at DEBUG level inside the ApplicationMaster log. Contributed by Jian He.
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1572381 13f79535-47bb-0310-9956-ffa450edef68
2014-02-27 01:30:07 +00:00
58d84af922
svn merge -c 1572269 FIXES: Preserve Job diagnostics in history. Contributed by Gera Shegalov
...
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1572276 13f79535-47bb-0310-9956-ffa450edef68
2014-02-26 21:42:47 +00:00
Zhijie Shen
Jason Lowe
Vinod Kumar Vavilapalli
Chris Douglas
Eric Yang
Xuan Gong
Jian He
Karthik Kambatla
Junping Du
Jonathan Turner Eagles
Haohui Mai
Chris Nauroth
Arun Murthy