Commit Graph

2648 Commits

Author SHA1 Message Date
Sean Mackrory 1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Akira Ajisaka ceacadc51e
MAPREDUCE-6921. TestUmbilicalProtocolWithJobToken#testJobTokenRpc fails. 2017-08-01 14:56:42 +09:00
Junping Du fbb7d6bcbb Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)"
This reverts commit e8a31f2e1c.
2017-07-31 14:09:16 -07:00
Junping Du 713349a9af Revert "MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)"
This reverts commit 390a7c12f5.
2017-07-28 14:21:04 -07:00
Junping Du e15f928295 Revert "MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu."
This reverts commit 433542904a.
2017-07-28 14:06:59 -07:00
Akira Ajisaka 38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier. 2017-07-28 09:10:52 +09:00
Varun Saxena f66fd11e51 YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-27 23:14:50 +05:30
Akira Ajisaka cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Sunil G fb3b5d33ff MAPREDUCE-6889. Add Job#close API to shutdown MR client services. Contributed by Rohith Sharma K S. 2017-07-17 13:35:15 +05:30
Ravi Prakash 43f0503286 MAPREDUCE-6910. MapReduceTrackingUriPlugin can not return the right URI of history server with HTTPS. Contributed by Lantao Jin 2017-07-13 16:16:45 -07:00
Junping Du f484a6ff60 MAPREDUCE-6246. DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2. Contributed by ramtin and Gergely Novák. 2017-07-07 13:26:16 -07:00
Naganarasimha bf1f59937d MAPREDUCE-6905. HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh. Contributed by LiXin Ge. 2017-07-02 15:56:27 +05:30
Andrew Wang af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Akira Ajisaka a5c0476a99
MAPREDUCE-6697. Concurrent task limits should only be applied when necessary. Contributed by Nathan Roberts. 2017-06-28 10:50:09 +09:00
Robert Kanter 2b87faf166 MAPREDUCE-6904. HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh (rkanter) 2017-06-26 17:35:55 -07:00
Ravi Prakash 82bbcbf37f MAPREDUCE-6897. Add Unit Test to ensure Job end notification gets sent even when appMaster stop gets YarnRuntimeException. Contributed by Gergely Novák 2017-06-16 14:25:44 -07:00
Jason Lowe 9ef2743fa1 Revert "MAPREDUCE-6898. TestKill.testKillTask is flaky. Contributed by Peter Bacsko"
This reverts commit e78542f4be.
2017-06-16 14:48:51 -05:00
Jason Lowe e78542f4be MAPREDUCE-6898. TestKill.testKillTask is flaky. Contributed by Peter Bacsko 2017-06-16 14:24:54 -05:00
Xiao Chen 4ebc23ba7b HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively. 2017-06-16 09:49:33 -07:00
Junping Du 6ed54f3439 MAPREDUCE-6895. Job end notification not send due to YarnRuntimeException. Contributed by yunjiong zhao. 2017-06-13 15:21:04 -07:00
Daniel Templeton 036a24bab3 MAPREDUCE-6896. Document wrong spelling in usage of MapredTestDriver tools.
(Contributed by LiXin Ge via Daniel Templeton)
2017-06-13 13:21:23 -07:00
Brahma Reddy Battula 24181f559f MAPREDUCE-6676. NNBench should Throw IOException when rename and delete fails. Contributed by Brahma Reddy Battula. 2017-06-07 16:34:47 +08:00
Sean Busbey f5517a8200
HADOOP-13921. Remove log4j classes from JobConf.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-06-01 13:35:14 +09:00
Akira Ajisaka d4015f8628
MAPREDUCE-6887. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin. 2017-05-30 14:48:58 +09:00
Andrew Wang 16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Sunil G b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Akira Ajisaka 035d46836b
MAPREDUCE-6459. Native task crashes when merging spilled file on ppc64. Contributed by Tao Jie and Ayappan. 2017-05-17 07:50:29 -04:00
Jason Lowe 1e71fe8c42 MAPREDUCE-6882. Increase MapReduce test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 2017-05-10 09:57:41 -05:00
Chris Douglas 45382bf047 AuditLogger and TestAuditLogger are dead code. Contributed by Vrushali C 2017-05-09 10:31:52 -07:00
Chris Douglas e4f34ecb04 HADOOP-14390. Correct spelling of 'succeed' and variants. Contributed by Dongtao Zhang 2017-05-05 12:10:50 -07:00
Akira Ajisaka 3ed3062fe3
MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. Contributed by Weiwei Yang. 2017-04-27 15:45:33 +09:00
Akira Ajisaka 8a99eba96d
Revert "YARN-5007. Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster. Contributed by Andras Bokor."
This reverts commit 34ab8e73d4.
2017-04-26 15:34:23 +09:00
Robert Kanter 3721cfe1fb MAPREDUCE-6871. Allow users to specify racks and nodes for strict locality for AMs (rkanter) 2017-04-21 16:12:01 -07:00
Daniel Templeton dd43b895c2 YARN-6202. Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded
(Contributed by Yufei Gu via Daniel Templeton)
2017-04-19 11:44:55 -07:00
Haibo Chen f1de2c8560 MAPREDUCE-6875. Rename mapred-site.xml.template to mapred-site.xml. (Yuanbo Liu via Haibo Chen) 2017-04-17 12:25:30 -07:00
Haibo Chen ac3cfdf3e5 MAPREDUCE-6673. Add a test example job that grows in memory usage over time (Karthik Kambatla via Haibo Chen)
Change-Id: Iccfc8c67c38c526cc61726d87bfcbcf69ac36fea
2017-04-17 11:15:39 -05:00
Allen Wittenauer 96cbb4fce2 HADOOP-14202. fix jsvc/secure user var inconsistencies
Signed-off-by: John Zhuge <jzhuge@apache.org>
2017-04-07 08:59:21 -07:00
Robert Kanter e7167e4a13 MAPREDUCE-6201. TestNetworkedJob fails on trunk (pbacsko via rkanter) 2017-04-06 17:44:47 -07:00
Daniel Templeton fc0885da29 MAPREDUCE-6846. Fragments specified for libjar paths are not handled correctly
(Contributed by Chris Trezzo via Daniel Templeton)
2017-04-05 17:25:40 -07:00
Andrew Wang 539ef5aa2e HFDS-11596. hadoop-hdfs-client jar is in the wrong directory in release tarball. Contributed by Yuanbo Liu. 2017-04-05 16:04:09 -07:00
Akira Ajisaka 34ab8e73d4
YARN-5007. Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster. Contributed by Andras Bokor. 2017-04-05 17:33:07 +09:00
Chris Douglas 6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Akira Ajisaka 845529b3ab MAPREDUCE-6824. TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines. Contributed by Chris Trezzo. 2017-04-03 13:06:54 +09:00
Jason Lowe 4d1fac5df2 YARN-6411. Clean up the overwrite of createDispatcher() in subclass of MockRM. Contributed by Yufei Gu 2017-03-31 10:05:34 -05:00
Jason Lowe c9b7ce9273 MAPREDUCE-6836. exception thrown when accessing the job configuration web UI. Contributed by Haibo Chen 2017-03-30 12:55:32 -05:00
Chris Douglas 7297e82cb4 HADOOP-14250. Correct spelling of 'separate' and variants. Contributed by Doris Gu 2017-03-30 10:17:11 -07:00
Jason Lowe c8bd5fc7a8 MAPREDUCE-6850. Shuffle Handler keep-alive connections are closed from the server side. Contributed by Jonathan Eagles 2017-03-30 10:57:19 -05:00
Zhe Zhang de7efd2687 MAPREDUCE-6873. MR Job Submission Fails if MR framework application path not on defaultFS. Contributed by Erik Krogen. 2017-03-29 22:10:55 -07:00
Ming Ma ceab00ac62 MAPREDUCE-6862. Fragments are not handled correctly by resource limit checking. (Chris Trezzo via mingma) 2017-03-29 17:41:58 -07:00
Robert Kanter 9bae6720cb YARN-6050. AMs can't be scheduled on racks or nodes (rkanter) 2017-03-27 17:23:13 -07:00
Junping Du 945b006fe8 MAPREDUCE-6868. License check for jdiff output files should be ignored. Contributed by Akira Ajisaka. 2017-03-26 23:18:05 -07:00
Joe Mészáros e0a2eb71e3
MAPREDUCE-6866. Fix getNumMapTasks() documentation in JobConf.
This closes #205

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-03-27 13:51:03 +09:00
Junping Du 6df029db36 Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 release. 2017-03-25 14:28:48 -07:00
Naganarasimha 34a931c0fa MAPREDUCE-6865. Fix typo in javadoc for DistributedCache. Contributed by Attila Sasvari. 2017-03-20 00:04:23 +05:30
Jason Lowe 4ebe8a6a23 MAPREDUCE-6859. hadoop-mapreduce-client-jobclient.jar sets a main class that isn't in the JAR. Contributed by Daniel Templeton 2017-03-08 10:27:57 -06:00
Robert Kanter 38d75dfd3a MAPREDUCE-6839. TestRecovery.testCrashed failed (pairg via rkanter) 2017-03-07 13:34:46 -08:00
Akira Ajisaka f01a69f84f
Treat encrypted files as private. Contributed by Daniel Templeton. 2017-03-07 13:22:11 +09:00
Akira Ajisaka 14413989ca
MAPREDUCE-6855. Specify charset when create String in CredentialsTestJob. Contributed by Kai Sasaki. 2017-03-07 13:10:59 +09:00
Akira Ajisaka 209ecd1a5c
HADOOP-14108. CLI MiniCluster: add an option to specify NameNode HTTP port. Contributed by Takanobu Asanuma. 2017-03-07 01:34:51 +09:00
Akira Ajisaka 490abfb10f HADOOP-14057. Fix package.html to compile with Java 9. 2017-03-04 00:25:22 +09:00
Akira Ajisaka 19a1fc6373 MAPREDUCE-6753. Variable in byte printed directly in mapreduce client. Contributed by Kai Sasaki. 2017-03-03 16:11:10 +09:00
Jian He 747bafaf96 MAPREDUCE-6852. Job#updateStatus() failed with NPE due to race condition. Contributed by Junping Du 2017-03-02 10:58:56 -08:00
Rakesh Radhakrishnan 82ef9accaf HDFS-11428. Change setErasureCodingPolicy to take a required string EC policy name. Contributed by Andrew Wang. 2017-03-01 16:06:02 +05:30
Akira Ajisaka 4d33683882 MAPREDUCE-6841. Fix dead link in MapReduce tutorial document. Contributed by Victor Nee. 2017-02-27 19:46:37 +09:00
Daniel Templeton 732ee6f0b5 MAPREDUCE-6825. YARNRunner#createApplicationSubmissionContext method is longer than 150 lines (Contributed by Gergely Novák via Daniel Templeton) 2017-02-22 15:38:11 -08:00
Wangda Tan 1a6ca75f38 YARN-6143. Fix incompatible issue caused by YARN-3583. (Sunil G via wangda) 2017-02-22 11:17:09 -08:00
Eric Yang 172b23af33 HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu. 2017-02-18 18:34:13 -08:00
Sangjin Lee 4fa1afdb88 YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R. 2017-02-16 11:41:04 -08:00
Daniel Templeton deb368b1b6 MAPREDUCE-6842. Update the links in PiEstimator document (Contributed by Jung Yoo via Daniel Templeton) 2017-02-07 13:05:40 -08:00
Junping Du 7afe5b1a31 MAPREDUCE-6404. Allow AM to specify a port range for starting its webapp. Contributed by Varun Saxena. 2017-02-07 01:40:49 -08:00
Junping Du d401e63b6c MAPREDUCE-6338. MR AppMaster does not honor ephemeral port range. Contributed by Frank Nguyen. 2017-02-05 19:28:01 -08:00
Masatake Iwasaki 3619ae32be MAPREDUCE-6644. Use doxia macro to generate in-page TOC of MapReduce site documentation. (iwasakims) 2017-02-02 01:19:36 +09:00
Karthik Kambatla c65f884fc7 MAPREDUCE-6829. Add peak memory usage counter for each task. (Miklos Szegedi via kasha) 2017-01-26 11:08:13 -08:00
Daniel Templeton a7463b6c88 MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log (Contributed by Gergő Pásztor via Daniel Templeton) 2017-01-25 14:32:40 -08:00
Jason Lowe 69fa81679f YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 2017-01-23 17:12:51 +00:00
Andrew Wang 5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Allen Wittenauer 0eb4b513b7 HADOOP-13673. Update scripts to be smarter when running with privilege
Signed-off-by: Andrew Wang <wang@apache.org>
Signed-off-by: Ravi Prakash <raviprak@apache.org>
2017-01-18 14:39:05 -08:00
Jason Lowe 78b487bde1 MAPREDUCE-6831. Flaky test TestJobImpl.testKilledDuringKillAbort. Contributed by Peter Bacsko 2017-01-17 16:01:42 +00:00
Akira Ajisaka 7ee8be1aa3 MAPREDUCE-6819. Replace UTF8 with Text in MRBench. Contributed by Peter Bacsko. 2017-01-17 17:56:10 +09:00
Arun Suresh 85826f6ca5 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh)
(cherry picked from commit 4985217de4)
2017-01-08 09:04:20 -08:00
Junping Du 679478d0c6 MAPREDUCE-6711. JobImpl fails to handle preemption events on state COMMITTING. Contributed by Prabhu Joseph. 2017-01-07 22:04:18 -08:00
Daniel Templeton 0b8a7c18dd MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu via Daniel Templeton) 2017-01-05 17:56:08 -08:00
Wangda Tan 0a55bd841e YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 2017-01-05 10:31:05 -08:00
Jason Lowe a0a2761621 HADOOP-13578. Add Codec for ZStandard Compression. Contributed by churro morales 2017-01-04 14:46:25 +00:00
Akira Ajisaka 01d31fe938 HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 2017-01-04 14:10:36 +09:00
Brahma Reddy Battula 6938b67711 HADOOP-13883. Addendum patch to correct the message and alphabetized with the earlier patch. Contributed by Yiqun Lin. 2016-12-31 16:16:02 +05:30
Daniel Templeton 95c2c24cd0 MAPREDUCE-5155. Race condition in test case TestFetchFailure cause it to fail (Contributed by Haibo Chen via Daniel Templeton) 2016-12-29 10:50:47 -08:00
Daniel Templeton ded2d08f33 HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton) 2016-12-27 07:18:08 -08:00
Akira Ajisaka ae401539ea YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 2016-12-22 14:32:24 +09:00
Jian He 523411d69b YARN-4844 (Addendum). Change JobStatus(usedMem,reservedMem,neededMem)
back to int for compatibility. Contributed by Wangda Tan
2016-12-20 11:28:44 -08:00
Arun Suresh 2273a74c1f YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 2016-12-16 08:14:34 -08:00
Akira Ajisaka b0b033ea2e MAPREDUCE-6821. Fix javac warning related to the deprecated APIs after upgrading Jackson. Contributed by Yiqin Lin. 2016-12-13 14:22:07 +09:00
Akira Ajisaka 92a8917ca5 MAPREDUCE-6820. Fix dead links in Job relevant classes. Contributed by Yiqun Lin. 2016-12-10 10:55:41 +09:00
Robert Kanter 4b149a1e77 YARN-4457. Cleanup unchecked types for EventHandler (templedf via rkanter) 2016-12-09 17:28:25 -08:00
Akira Ajisaka f54afdb83b MAPREDUCE-4683. Create and distribute hadoop-mapreduce-client-core-tests.jar. 2016-12-08 11:27:58 +09:00
Karthik Kambatla a793cec2c9 MAPREDUCE-6817. The format of job start time in JHS is different from those of submit and finish time. (Haibo Chen via kasha) 2016-12-07 13:36:49 -08:00
Akira Ajisaka da4ecc9f82 MAPREDUCE-6816. Progress bars in Web UI always at 100%. Contributed by Shen Yinjie. 2016-12-07 06:41:08 +09:00
Robert Kanter 6b4a9970f1 MAPREDUCE-6571. JobEndNotification info logs are missing in AM container syslog (haibochen via rkanter) 2016-12-06 13:11:03 -08:00
Akira Ajisaka b2a3d6c519 HADOOP-13835. Move Google Test Framework code from mapreduce to hadoop-common. Contributed by Varun Vasudev. 2016-12-06 14:01:47 +09:00
Jason Lowe 0cfd7ad21f MAPREDUCE-6815. Fix flaky TestKill.testKillTask(). Contributed by Haibo Chen 2016-12-02 17:22:11 +00:00