Commit Graph

493 Commits

Author SHA1 Message Date
Akira Ajisaka 7d665f0a98 MAPREDUCE-6520. Migrate MR Client test cases part 1.
(cherry picked from commit ef68b44195)
2016-03-11 22:25:29 +09:00
Robert Kanter d9f01f28a0 MAPREDUCE-6640. mapred job -history command should be able to take Job ID (rkanter)
(cherry picked from commit 4343a4cf77)
2016-02-23 17:38:25 -08:00
Tsuyoshi Ozawa 929e869ae0 YARN-4708. Missing default mapper type in TimelineServer performance test tool usage. Contributed by Kai Sasaki.
(cherry picked from commit b68901d7dd)
2016-02-22 07:55:35 +09:00
Robert Kanter a33109a0b8 MAPREDUCE-6627. Add machine-readable output to mapred job -history command (rkanter)
(cherry picked from commit 8eee59ce6b)
2016-02-18 17:58:40 -08:00
Jason Lowe 303c650d0d MAPREDUCE-6191. Improve clearing stale state of Java serialization
testcase.  (Sam Liu via Eric Yang)
(cherry picked from commit c379e102dd)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
2016-02-10 16:44:12 +00:00
Wangda Tan 9591363ad8 YARN-4340. Add list API to reservation system. (Sean Po via wangda)
(cherry picked from commit 9875325d5c)
2016-02-02 10:19:19 +08:00
Jason Lowe f5f6652137 MAPREDUCE-6618. YarnClientProtocolProvider leaking the YarnClient thread. Contributed by Xuan Gong
(cherry picked from commit 59a212b6e1)
2016-02-01 16:06:42 +00:00
Karthik Kambatla ef59521094 YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha)
(cherry picked from commit d40859fab1)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
2016-01-18 10:59:34 +01:00
Akira Ajisaka 6efee8907a MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Vlad Sharanhovich and Bibin A Chundatt.
(cherry picked from commit 7b0964f354)
2016-01-14 10:41:22 +09:00
Sangjin Lee 2dfa0613a1 MAPREDUCE-6577. MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set (sjlee)
(cherry picked from commit f6f16118d3)
2016-01-05 15:25:47 -08:00
Jian He 1147b086d9 YARN-4164. Changed updateApplicationPriority API to return the updated application priority. Contributed by Rohith Sharma K S 2015-12-18 14:14:31 -08:00
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Jason Lowe 2f18218508 MAPREDUCE-5870. Support for passing Job priority through Application Submission Context in Mapreduce Side. Contributed by Sunil G
(cherry picked from commit f634505d48)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
2015-11-24 22:15:37 +00:00
Haohui Mai 469fef2a4f HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai Zheng. 2015-11-22 16:18:16 -08:00
Sangjin Lee 3214c2331f MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
(cherry picked from commit 9ad708a0df)
2015-11-11 16:22:28 -08:00
Xuan b505bf9e58 YARN-4313. Race condition in MiniMRYarnCluster when getting history
server address. Contributed by Jian He

(cherry picked from commit 7412ff48ee)
2015-10-29 17:37:24 -07:00
Tsuyoshi Ozawa 3138b43f16 YARN-3573. MiniMRYarnCluster constructor that starts the timeline server using a boolean should be marked deprecated. Contributed by Brahma Reddy Battula.
(cherry picked from commit 96677bef00)
2015-10-27 12:35:20 +09:00
Junping Du 801ce47e28 MAPREDUCE-6508. TestNetworkedJob fails consistently due to delegation token changes on RM. Contributed by Akira AJISAKA
(cherry picked from commit eb6379ca25)
2015-10-23 10:07:41 -07:00
Sangjin Lee 180efe6677 YARN-2556. Tool to measure the performance of the timeline server (Chang Li via sjlee)
(cherry picked from commit 58590fef49)
2015-10-16 16:57:32 -07:00
Jason Lowe 0139517811 YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S
(cherry picked from commit a0bca2b5ad)
2015-10-09 14:19:41 +00:00
Xuan 53bddc410f YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma
(cherry picked from commit 8f08532bde)
2015-10-02 18:53:04 -07:00
Haohui Mai 0971503522 MAPREDUCE-6483. Replace deprecated method NameNode.getUri() with DFSUtilClient.getNNUri() in TestMRCredentials. Contributed by Mingliang Liu. 2015-09-18 13:36:22 -07:00
Haohui Mai ac2ed09be7 MAPREDUCE-6477. Replace usage of deprecated NameNode.DEFAULT_PORT in TestFileSystem. Contributed by Mingliang Liu. 2015-09-16 13:06:38 -07:00
Jason Lowe 52f56524d4 MAPREDUCE-6472. MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks. Contributed by Naganarasimha G R
(cherry picked from commit 62943b8e3a)
2015-09-15 19:44:17 +00:00
Andrew Wang 355398e441 HADOOP-12358. Add -safely flag to rm to prompt when deleting many files. Contributed by Xiaoyu Yao.
(cherry picked from commit e1feaf6db0)
2015-09-04 13:48:42 -07:00
Akira Ajisaka a61c0f5ca1 Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula."
This reverts commit 98446ee695.
2015-09-02 12:26:25 +09:00
Zhihai Xu 297bda600e MAPREDUCE-6452. NPE when intermediate encrypt enabled for LocalRunner. Contributed by Zhihai Xu
(cherry picked from commit cbb249534a)
2015-08-28 12:20:49 -07:00
Jian He 387076894f YARN-4014. Support user cli interface in for Application Priority. Contributed by Rohith Sharma K S
(cherry picked from commit 57c7ae1aff)
2015-08-24 20:37:18 -07:00
Jason Lowe 31fd7a51d8 MAPREDUCE-6413. TestLocalJobSubmission is failing with unknown host. Contributed by zhihai xu
(cherry picked from commit aa5b15b03b)
2015-06-25 19:52:23 +00:00
Akira Ajisaka c4747158b7 MAPREDUCE-6403. Fix typo in the usage of NNBench. Contributed by Jagadesh Kiran N.
(cherry picked from commit 445b13240c)
2015-06-22 15:34:34 +09:00
Akira Ajisaka 98446ee695 MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula.
(cherry picked from commit 75a2560e51)
2015-06-15 14:53:28 -07:00
Zhijie Shen 2fcd3fe6ba Revert "YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong."
This reverts commit 4e2ed34d20.
2015-06-03 14:16:51 -07:00
Zhijie Shen 4e2ed34d20 YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong.
(cherry picked from commit a9f986bc5a41ab74171da1076d10e83c5adc284f)
2015-05-30 21:06:32 -07:00
Wangda Tan 7cd0874dfd MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda)
(cherry picked from commit 3164e7d838)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
2015-05-27 14:43:20 -07:00
Tsuyoshi Ozawa 6042b53a74 MAPREDUCE-6204. TestJobCounters should use new properties instead of JobConf.MAPRED_TASK_JAVA_OPTS.
(cherry picked from commit b27ce3c531)
2015-05-22 23:30:12 +09:00
Wangda Tan b5ba6979b7 YARN-3583. Support of NodeLabel object instead of plain String in YarnClient side. (Sunil G via wangda)
(cherry picked from commit 563eb1ad2a)
(cherry picked from commit b0d22b0c606fad6b4ab5443c0aed07c829b46726)
2015-05-19 17:37:24 -07:00
Vinod Kumar Vavilapalli 87862970f1 Fixing MR intermediate spills. Contributed by Arun Suresh.
(cherry picked from commit 6b710a42e0)
2015-05-14 16:56:56 -07:00
Vinod Kumar Vavilapalli d5755ba241 MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch.
(cherry picked from commit f24452d14e)
2015-05-12 12:13:23 -07:00
Jason Lowe 53e9973afc MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma 2015-05-11 22:40:05 +00:00
Ravi Prakash 23065e6051 MAPREDUCE-4750. Enable NNBenchWithoutMR in MapredTestDriver (Liang Xie and Jason Lowe via raviprak)
(cherry picked from commit 5aab014340)
2015-05-08 15:58:29 -07:00
Jason Lowe 0f6c971282 MAPREDUCE-5248. Let NNBenchWithoutMR specify the replication factor for its test. Contributed by Erik Paulson
(cherry picked from commit 30099a36c6)
2015-05-08 22:30:21 +00:00
Devaraj K ade5165ecf MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. 2015-05-08 15:20:35 +05:30
Akira Ajisaka f7c011cbaa MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula.
(cherry picked from commit d335071398)
2015-05-07 19:41:44 +09:00
Tsuyoshi Ozawa 4e96175b33 MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA.
(cherry picked from commit 551615fa13)
2015-05-05 10:23:25 +09:00
Vinod Kumar Vavilapalli 0ec6e7e7f5 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv)
(cherry picked from commit f5fe35e297)
2015-04-23 11:25:30 -07:00
Gera Shegalov 6f5dc9f5af MAPREDUCE-6293. Set job classloader on uber-job's LocalContainerLauncher event thread. (Sangjin Lee via gera)
(cherry picked from commit 725eb52ddc)
2015-04-21 14:21:56 -07:00
Robert Kanter 756c254293 MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter)
(cherry picked from commit d50e8f0928)
2015-04-20 14:14:49 -07:00
Jian He d3daf9665c YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev 2015-04-10 10:40:31 -07:00
Harsh J b817f0924d MAPREDUCE-6314. TestPipeApplication fails on trunk. Contributed by Varun Vasudev.
(cherry picked from commit c91b2de68a)
2015-04-10 14:03:37 +05:30
Hitesh Shah 55b794e7fa YARN-2890. MiniYarnCluster should turn on timeline service if configured to do so. Contributed by Mit Desai.
(cherry picked from commit 265ed1fe80)
2015-04-08 14:14:06 -07:00
Akira Ajisaka 2e509908ee MAPREDUCE-6292. Use org.junit package instead of junit.framework in TestCombineFileInputFormat. (aajisaka)
(cherry picked from commit c770df49b4)
2015-03-25 19:01:38 +09:00
Tsuyoshi Ozawa 94eb8e5b3d MAPREDUCE-6285. ClientServiceDelegate should not retry upon AuthenticationException. Contributed by Jonathan Eagles.
(cherry picked from commit 4170c99147)
2015-03-25 00:57:08 +09:00
Harsh J 97da36777f MAPREDUCE-6239. Consolidate TestJobConf classes in hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core. Contributed by Varun Saxena. 2015-03-22 02:20:11 +05:30
Jonathan Eagles 44aedad5dd YARN-3267. Timelineserver applies the ACL rules after applying the limit on the number of records (Chang Li via jeagles)
(cherry picked from commit 8180e676ab)
2015-03-13 12:05:21 -05:00
Tsuyoshi Ozawa 482504befd MAPREDUCE-4742. Fix typo in nnbench#displayUsage. Contributed by Liang Xie.
(cherry picked from commit 20b8ee1350)
2015-03-11 00:41:37 +09:00
Junping Du dbcdcb0d3c Configurable timeout between YARNRunner terminate the application and forcefully kill. Contributed by Eric Payne.
(cherry picked from commit d39bc903a0)
2015-03-10 06:23:24 -07:00
Tsuyoshi Ozawa b46f9e72db HADOOP-11602. Backport 'Fix toUpperCase/toLowerCase to use Locale.ENGLISH.' (ozawa) 2015-03-09 19:53:55 +09:00
Vinod Kumar Vavilapalli d9416317a9 Preparing for 2.8.0 development. Changing SNAPSHOT version to be 2.8 on branch-2. 2015-03-08 20:32:51 -07:00
Tsuyoshi Ozawa ccde4aed18 MAPREDUCE-5657. Backport '[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments.' Contributed by Akira AJISAKA. 2015-03-09 12:00:04 +09:00
Wangda Tan cd5eb9c1de YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda)
(cherry picked from commit d49ae725d5)
2015-02-19 11:01:47 -08:00
Plamen Jeliazkov 32d215dc89 MAPREDUCE-6228. Add truncate operation to SLive. Contributed by Plamen Jeliazkov. 2015-02-19 00:14:11 -08:00
Tsuyoshi Ozawa 9523b52da5 MAPREDUCE-4286. TestClientProtocolProviderImpls passes on failure conditions. Contributed by Devaraj K.
(cherry picked from commit 7c782047c6)
2015-02-18 15:46:06 +09:00
yliu 6933975143 MAPREDUCE-6227. DFSIO for truncate. (shv via yliu) 2015-02-08 02:42:10 +08:00
Robert Kanter 46a2027226 MAPREDUCE-6233. org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort failed in trunk (zxu via rkanter)
(cherry picked from commit e2ee2ff7d7)
2015-02-05 14:31:44 -08:00
Akira Ajisaka 8d98e5c960 MAPREDUCE-5800. Use Job#getInstance instead of deprecated constructors. (aajisaka)
(cherry picked from commit bd69fb2d44)

Conflicts:
	hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/mapreduce/TestStreamXmlRecordReader.java
2015-02-03 14:35:15 -08:00
Brandon Li d847363821 HADOOP-9992. Modify the NN loadGenerator to optionally run as a MapReduce job. Contributed by Akshay Radia
(cherry picked from commit 09d31bc630)
2015-01-09 17:25:19 -08:00
Robert Kanter f9341c1e2c MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)
(cherry picked from commit 390a7c12f5)
2014-12-19 15:44:01 -08:00
Robert Kanter 22b38cf29a MAPREDUCE-6045. need close the DataInputStream after open it in TestMapReduce.java (zxu via rkanter)
(cherry picked from commit d9e4d67d18)
2014-12-19 12:00:00 -08:00
Jian He d2356a552a Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai."
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
(cherry picked from commit a4f2995b9e)
2014-12-15 11:08:44 -08:00
Harsh J 6e06a51e30 MAPREDUCE-5420. Remove mapreduce.task.tmp.dir from mapred-default.xml. Contributed by James Carman. (harsh)
(cherry picked from commit cb99f43305)
2014-12-11 09:46:08 +05:30
Jason Lowe db723a8499 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov
(cherry picked from commit 03ab24aa01)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
2014-12-03 17:12:47 +00:00
Zhijie Shen d21ef79707 YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai.
(cherry picked from commit 51af8d367d)
2014-12-01 19:08:31 -08:00
Arun C. Murthy adfb830a2b Preparing to release hadoop-2.6.0: Set version in branch-2 to 2.7.0-SNAPSHOT. 2014-11-09 19:19:02 -08:00
Arun C. Murthy 175d222bfc YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles.
(cherry picked from commit 43cd07b408)
2014-11-09 15:03:59 -08:00
Jason Lowe c0bbb72912 MAPREDUCE-5960. JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar path. Contributed by Gera Shegalov
(cherry picked from commit 10f9f5101c)
2014-11-06 15:12:06 +00:00
Jian He 944fb1c3a9 MAPREDUCE-6048. Fixed TestJavaSerialization failure. Contributed by Varun Vasudev
(cherry picked from commit 73068f677b)
2014-11-04 20:20:33 -08:00
Haohui Mai fa92b40f09 HADOOP-11230. Add missing dependency of bouncycastle for kms, httpfs, hdfs, MR and YARN. Contributed by Robert Kanter. 2014-11-04 17:53:00 -08:00
Zhijie Shen c4aaa4db6c MAPREDUCE-6052. Supported overriding the default container-log4j.properties file per job. Contributed by Junping Du.
(cherry picked from commit ed63b11646)
2014-11-01 00:49:38 -07:00
Vinod Kumar Vavilapalli 0ba8d8c19c YARN-2698. Moved some node label APIs to be correctly placed in client protocol. Contributed by Wangda Tan.
(cherry picked from commit e0233c16eb)
2014-10-30 23:02:06 -07:00
Vinod Kumar Vavilapalli 063bb0508b MAPREDUCE-6142. Fixed test failures in TestJobHistoryEventHandler and TestMRTimelineEventHandling. Contributed by Zhijie Shen.
(cherry picked from commit 3f1441245d)
2014-10-29 09:58:50 -07:00
Zhijie Shen 52695d38fc MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter.
(cherry picked from commit 971e91c8c0)
2014-10-27 21:08:38 -07:00
Zhijie Shen 733380fa70 MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter.
(cherry picked from commit 6b2f11b54b)
2014-10-27 20:41:02 -07:00
Jason Lowe 8885b75d78 MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in client. Contributed by Rohith
(cherry picked from commit 209b1699fc)
2014-10-17 19:53:17 +00:00
Jian He f93d2ea27e YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:28:26 -07:00
Jason Lowe 025b6c13e9 MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling. Contributed by Gera Shegalov
(cherry picked from commit f19771a24c)
2014-10-15 17:51:53 +00:00
Jason Lowe 942fb67a04 MAPREDUCE-6115. TestPipeApplication#testSubmitter fails in trunk. Contributed by Binglin Chang
(cherry picked from commit 793dbf2465)
2014-10-13 16:05:29 +00:00
Karthik Kambatla 7bfd9e068d MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)
(cherry picked from commit ed0e0ef9748ce5b231de677ab41c3035137bbde4)
2014-10-11 22:49:46 -07:00
cnauroth f6278cbac5 MAPREDUCE-6123. TestCombineFileInputFormat incorrectly starts 2 MiniDFSCluster instances. Contributed by Chris Nauroth.
(cherry picked from commit 777877d079)
2014-10-09 22:38:25 -07:00
subru 30a370e705 MAPREDUCE-6103.Adding reservation APIs to MR resource manager delegate. Contributed by Subru Krishnan and Carlo Curino.
(cherry picked from commit aa92dd45f2)
(cherry picked from commit 3f282762d1)
2014-10-06 10:21:06 -07:00
Jason Lowe 7b8f2f5b1d MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk. Contributed by Akira AJISAKA
(cherry picked from commit 23fbfe36c3)
2014-09-29 18:14:11 +00:00
Jason Lowe 0cfff28f57 HADOOP-11049. javax package system class default is too broad. Contributed by Sangjin Lee
(cherry picked from commit cf4631d7c9)
2014-09-29 14:37:28 +00:00
Allen Wittenauer 67d6fc3f01 MAPREDUCE-5945. Update the description of GenericOptionsParser -jt option (Akira AJISAKA via aw) 2014-09-26 14:26:36 -07:00
Jason Lowe f4534746fb MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs. Contributed by Gera Shegalov
(cherry picked from commit 7039b98e1c)
2014-09-22 15:22:58 +00:00
Jason Lowe bdb864e5e1 MAPREDUCE-6091. YARNRunner.getJobStatus() fails with ApplicationNotFoundException if the job rolled off the RM view. Contributed by Sangjin Lee
(cherry picked from commit 951847ba94)
2014-09-19 20:17:44 +00:00
Karthik Kambatla 372bf54072 MAPREDUCE-6086. mapreduce.job.credentials.binary should allow all URIs. (Zhihai Xu via kasha)
(cherry picked from commit 52945a33cc)
2014-09-18 15:47:21 -07:00
Vinod Kumar Vavilapalli 5cdb24d4b1 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong.
(cherry picked from commit 14e2639fd0)
2014-09-13 18:05:02 -07:00
Jason Lowe 27086f5942 MAPREDUCE-5931. Validate SleepJob command line parameters. Contributed by Gera Shegalov
(cherry picked from commit 15366d9227)
2014-08-29 19:52:10 +00:00
Chris Douglas 4f95017602 Fix typos in log messages. Contributed by Ray Chiang 2014-08-28 16:30:13 -07:00
Alejandro Abdelnur c77bd85b62 HDFS-6134 and HADOOP-10150 subtasks. 2014-08-28 15:03:08 -07:00
Jason Lowe 38df629367 MAPREDUCE-5885. build/test/test.mapred.spill causes release audit warnings. Contributed by Chen He
(cherry picked from commit 812bd0c0e5)
2014-08-27 15:22:06 +00:00