7afe5b1a31
MAPREDUCE-6404. Allow AM to specify a port range for starting its webapp. Contributed by Varun Saxena.
2017-02-07 01:40:49 -08:00
d401e63b6c
MAPREDUCE-6338. MR AppMaster does not honor ephemeral port range. Contributed by Frank Nguyen.
2017-02-05 19:28:01 -08:00
3619ae32be
MAPREDUCE-6644. Use doxia macro to generate in-page TOC of MapReduce site documentation. (iwasakims)
2017-02-02 01:19:36 +09:00
c65f884fc7
MAPREDUCE-6829. Add peak memory usage counter for each task. (Miklos Szegedi via kasha)
2017-01-26 11:08:13 -08:00
a7463b6c88
MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log (Contributed by Gergő Pásztor via Daniel Templeton)
2017-01-25 14:32:40 -08:00
69fa81679f
YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He
2017-01-23 17:12:51 +00:00
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
0eb4b513b7
HADOOP-13673. Update scripts to be smarter when running with privilege
...
Signed-off-by: Andrew Wang <wang@apache.org>
Signed-off-by: Ravi Prakash <raviprak@apache.org>
2017-01-18 14:39:05 -08:00
78b487bde1
MAPREDUCE-6831. Flaky test TestJobImpl.testKilledDuringKillAbort. Contributed by Peter Bacsko
2017-01-17 16:01:42 +00:00
7ee8be1aa3
MAPREDUCE-6819. Replace UTF8 with Text in MRBench. Contributed by Peter Bacsko.
2017-01-17 17:56:10 +09:00
85826f6ca5
YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh)
...
(cherry picked from commit 4985217de4
2017-01-08 09:04:20 -08:00
679478d0c6
MAPREDUCE-6711. JobImpl fails to handle preemption events on state COMMITTING. Contributed by Prabhu Joseph.
2017-01-07 22:04:18 -08:00
0b8a7c18dd
MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu via Daniel Templeton)
2017-01-05 17:56:08 -08:00
0a55bd841e
YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda)
2017-01-05 10:31:05 -08:00
a0a2761621
HADOOP-13578. Add Codec for ZStandard Compression. Contributed by churro morales
2017-01-04 14:46:25 +00:00
01d31fe938
HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
2017-01-04 14:10:36 +09:00
6938b67711
HADOOP-13883. Addendum patch to correct the message and alphabetized with the earlier patch. Contributed by Yiqun Lin.
2016-12-31 16:16:02 +05:30
95c2c24cd0
MAPREDUCE-5155. Race condition in test case TestFetchFailure cause it to fail (Contributed by Haibo Chen via Daniel Templeton)
2016-12-29 10:50:47 -08:00
ded2d08f33
HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton)
2016-12-27 07:18:08 -08:00
ae401539ea
YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor.
2016-12-22 14:32:24 +09:00
523411d69b
YARN-4844 (Addendum). Change JobStatus(usedMem,reservedMem,neededMem)
...
back to int for compatibility. Contributed by Wangda Tan
2016-12-20 11:28:44 -08:00
2273a74c1f
YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh)
2016-12-16 08:14:34 -08:00
b0b033ea2e
MAPREDUCE-6821. Fix javac warning related to the deprecated APIs after upgrading Jackson. Contributed by Yiqin Lin.
2016-12-13 14:22:07 +09:00
92a8917ca5
MAPREDUCE-6820. Fix dead links in Job relevant classes. Contributed by Yiqun Lin.
2016-12-10 10:55:41 +09:00
4b149a1e77
YARN-4457. Cleanup unchecked types for EventHandler (templedf via rkanter)
2016-12-09 17:28:25 -08:00
f54afdb83b
MAPREDUCE-4683. Create and distribute hadoop-mapreduce-client-core-tests.jar.
2016-12-08 11:27:58 +09:00
a793cec2c9
MAPREDUCE-6817. The format of job start time in JHS is different from those of submit and finish time. (Haibo Chen via kasha)
2016-12-07 13:36:49 -08:00
da4ecc9f82
MAPREDUCE-6816. Progress bars in Web UI always at 100%. Contributed by Shen Yinjie.
2016-12-07 06:41:08 +09:00
6b4a9970f1
MAPREDUCE-6571. JobEndNotification info logs are missing in AM container syslog (haibochen via rkanter)
2016-12-06 13:11:03 -08:00
b2a3d6c519
HADOOP-13835. Move Google Test Framework code from mapreduce to hadoop-common. Contributed by Varun Vasudev.
2016-12-06 14:01:47 +09:00
0cfd7ad21f
MAPREDUCE-6815. Fix flaky TestKill.testKillTask(). Contributed by Haibo Chen
2016-12-02 17:22:11 +00:00
c87b3a448a
MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job overview page in JHS (haibochen via rkanter)
2016-12-01 17:29:38 -08:00
7c848719de
MAPREDUCE-6810. Fix hadoop-mapreduce-client-nativetask compilation with GCC-6.2.1. Contributed by Ravi Prakash.
2016-11-30 10:47:41 -08:00
8f6e14399a
MAPREDUCE-6565. Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution. Contributed by Li Lu.
2016-11-29 15:51:27 -08:00
6d8b4f6c27
YARN-4395. Typo in comment in ClientServiceDelegate (Contributed by Alison Yu via Daniel Templeton)
2016-11-29 15:30:22 -08:00
be88d574a9
MAPREDUCE-6743. nativetask unit tests need to provide usable output; fix link errors during mvn test
2016-11-28 15:24:44 -08:00
209e805430
HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki.
2016-11-28 14:34:57 +09:00
683e0c71fe
MAPREDUCE-6793. io.sort.factor code default and mapred-default.xml values inconsistent. Contributed by Gera Shegalov.
2016-11-21 22:10:57 +05:30
7584fbf4cb
MAPREDUCE-6801. Fix flaky TestKill.testKillJob (Haibo Chen via Varun Saxena)
2016-11-19 01:37:59 +05:30
b8690a9d25
MAPREDUCE-6811. TestPipeApplication#testSubmitter fails after HADOOP-13802 (Brahma Reddy Battula via Varun Saxena)
2016-11-17 00:51:07 +05:30
7ef290cfef
MAPREDUCE-6804. Add timeout when starting JobHistoryServer in MiniMRYarnCluster. Contributed by Andras Bokor
2016-11-16 15:23:12 +00:00
5af572b644
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
2016-11-15 10:57:00 -08:00
3219b7b4ac
YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh)
2016-11-15 07:56:25 -08:00
99c2bbd337
MAPREDUCE-6797. Job history server scans can become blocked on a single, slow entry. Contributed by Prabhu Joseph
2016-11-14 20:20:50 +00:00
bcc15c6290
YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S
2016-11-09 16:08:05 -08:00
e1c6ef2efa
HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey.
2016-11-08 16:33:55 -08:00
026b39ad9d
MAPREDUCE-6782. JHS task page search based on each individual column not working (Ajith S via Varun Saxena)
2016-11-08 14:57:56 +05:30
ca33bdd5c6
MAPREDUCE-6790. Update jackson from 1.9.13 to 2.x in hadoop-mapreduce.
2016-11-07 11:19:21 +09:00
de0132748d
MAPREDUCE-6796. Remove unused properties from JTConfig.java. Contributed by Haibo Chen.
2016-11-05 03:49:19 +09:00
0aafc122d4
YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh)
2016-11-04 07:31:54 -07:00
fc2b69eba1
MAPREDUCE-6765. MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported (haibochen via rkanter)
2016-11-01 20:47:25 -07:00
235f39b7c4
MAPREDUCE-6795. Update the document for JobConf#setNumReduceTasks. Contributed by Yiqun Lin.
2016-11-01 20:52:12 +09:00
f646fe34be
MAPREDUCE-6799. Document mapreduce.jobhistory.webapp.https.address in mapred-default.xml. Contributed by Yiqun Lin.
2016-10-31 15:17:20 +09:00
28660f51af
MAPREDUCE-2631. Potential resource leaks in BinaryProtocol$TeeOutputStream.java. Contributed by Sunil G.
2016-10-28 05:50:13 +05:30
5877f20f9c
HADOOP-10075. Update jetty dependency to version 9 (rkanter)
2016-10-27 16:09:00 -07:00
dd4ed6a587
YARN-4388. Cleanup mapreduce.job.hdfs-servers from yarn-default.xml (Junping Du via Varun Saxena)
2016-10-28 02:22:25 +05:30
060558c6f2
MAPREDUCE-6541. Exclude scheduled reducer memory when calculating available mapper slots from headroom to avoid deadlock. Contributed by Varun Saxena
2016-10-27 18:03:13 +05:30
8bd6d77a41
YARN-5757. RM REST API documentation is not up to date (Miklos Szegedi via Varun Saxena)
2016-10-27 17:21:01 +05:30
4e403def80
MAPREDUCE-6798. Fix intermittent failure of TestJobHistoryParsing.testJobHistoryMethods (Haibo Chen via Varun Saxena)
2016-10-27 15:36:59 +05:30
d3bb69a667
YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton.
2016-10-26 15:32:07 +09:00
9b7ce948a5
MAPREDUCE-6792. Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir(). Contributed by Santhosh G Nayak.
2016-10-25 11:20:23 -07:00
d4725bfcb2
MAPREDUCE-6728. Give fetchers hint when ShuffleHandler rejects a shuffling connection (haibochen via rkanter)
2016-10-21 17:46:17 -07:00
29caf6d7df
MAPREDUCE-6791. remove unnecessary dependency from hadoop-mapreduce-client-jobclient to hadoop-mapreduce-client-shuffle (haibochen via rkanter)
2016-10-18 18:00:29 -07:00
bea004eaeb
MAPREDUCE-6780. Add support for HDFS directory with erasure code policy to TeraGen and TeraSort. Contributed by Sammi Chen
2016-10-09 15:33:26 +06:00
f3f37e6fb8
MAPREDUCE-6776. yarn.app.mapreduce.client.job.max-retries should have a more useful default (miklos.szegedi@cloudera.com via rkanter)
2016-10-07 14:47:06 -07:00
272a21747e
MAPREDUCE-6789. Fix TestAMWebApp failure. Contributed by Daniel Templeton.
2016-10-06 15:57:15 +09:00
de7a0a92ca
MAPREDUCE-6638. Do not attempt to recover progress from previous job attempts if spill encryption is enabled. (Haibo Chen via kasha)
2016-10-03 10:30:22 -07:00
a1b8251bf7
MAPREDUCE-6771. RMContainerAllocator sends container diagnostics event after corresponding completion event. Contributed by Haibo Chen
2016-09-29 15:27:17 +00:00
0d6778d800
MAPREDUCE-6718. add progress log to JHS during startup (haibochen via rkanter)
2016-09-28 15:41:40 -07:00
875062b5bc
HADOOP-13544. JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs. (vinodkv via wangda)
2016-09-27 11:26:45 -07:00
4815d024c5
YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh
2016-09-26 08:46:54 -07:00
2f163cd5cf
Revert "YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh"
...
This reverts commit fe644bafe7
2016-09-26 08:36:59 -07:00
fe644bafe7
YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh
2016-09-26 22:41:16 +08:00
d0372dc613
YARN-3692. Allow REST API to set a user generated message when killing an application. Contributed by Rohith Sharma K S
2016-09-23 06:30:49 +05:30
4fc632ae19
MAPREDUCE-6632. Master.getMasterAddress() should be updated to use YARN-4629 (templedf via rkanter)
2016-09-22 16:12:56 -07:00
537095d13c
MAPREDUCE-6740. Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval. (Haibo Chen via kasha)
2016-09-21 18:30:11 -07:00
7d21c280a8
MAPREDUCE-6777. Typos in 4 log messages. Contributed by Mehran Hassani
2016-09-17 10:19:59 +05:30
501a77856d
MAPREDUCE-6774. Add support for HDFS erasure code policy to TestDFSIO. Contributed by Sammi Chen
2016-09-18 09:03:15 +08:00
ea0c2b8b05
HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng
2016-09-15 10:46:00 +08:00
58ed4fa544
HADOOP-13341. Deprecate HADOOP_SERVERNAME_OPTS; replace with (command)_(subcommand)_OPTS
...
This commit includes the following changes:
HADOOP-13356. Add a function to handle command_subcommand_OPTS
HADOOP-13355. Handle HADOOP_CLIENT_OPTS in a function
HADOOP-13554. Add an equivalent of hadoop_subcmd_opts for secure opts
HADOOP-13562. Change hadoop_subcommand_opts to use only uppercase
HADOOP-13358. Modify HDFS to use hadoop_subcommand_opts
HADOOP-13357. Modify common to use hadoop_subcommand_opts
HADOOP-13359. Modify YARN to use hadoop_subcommand_opts
HADOOP-13361. Modify hadoop_verify_user to be consistent with hadoop_subcommand_opts (ie more granularity)
HADOOP-13564. modify mapred to use hadoop_subcommand_opts
HADOOP-13563. hadoop_subcommand_opts should print name not actual content during debug
HADOOP-13360. Documentation for HADOOP_subcommand_OPTS
This closes apache/hadoop#126
2016-09-12 11:10:00 -07:00
9f192cc5ac
MAPREDUCE-6628. Potential memory leak in CryptoOutputStream. Contributed by Mariappan Asokan
2016-09-09 11:12:44 -07:00
d355573f56
Revert "HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng"
...
This reverts commit 62a9667136
2016-09-08 05:50:17 +08:00
62a9667136
HADOOP-13218. Migrate other Hadoop side tests to prepare for removing WritableRPCEngine. Contributed by Wei Zhou and Kai Zheng
2016-09-07 17:05:33 +08:00
d6d9cff21b
YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh)
2016-08-30 15:52:29 -07:00
af508605a9
MAPREDUCE-4784. TestRecovery occasionally fails. Contributed by Haibo Chen
2016-08-30 13:59:57 +00:00
69f7277625
MAPREDUCE-6768. TestRecovery.testSpeculative failed with NPE. Contributed by Haibo Chen
2016-08-29 19:56:09 +00:00
9ef632f3b0
YARN-5557. Add localize API to the ContainerManagementProtocol. Contributed by Jian He.
2016-08-26 09:04:44 -07:00
1360bd2d54
MAPREDUCE-6764. Teragen LOG initialization bug. Contributed by Yufei Gu.
2016-08-25 09:50:12 -07:00
5a6fc5f483
MAPREDUCE-6767. TestSlive fails after a common change. Contributed by Daniel Templeton.
2016-08-25 10:55:00 +09:00
6fa9bf4407
MAPREDUCE-6761. Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure. Contributed by Peter Vary
2016-08-24 14:40:51 +00:00
0ce1ab95cc
MAPREDUCE-6578. Add support for HDFS heterogeneous storage testing to TestDFSIO. Contributed by Wei Zhou and Sammi Chen
2016-08-24 22:17:05 +08:00
143c59e4c5
YARN-4491. yarn list command to support filtering by tags. Contributed by Varun Saxena
2016-08-24 01:53:02 +05:30
8cc4a67059
MAPREDUCE-6587. Remove unused params in connection-related methods of Fetcher. Contributed by Yiqun Lin.
2016-08-23 17:04:55 +09:00
d37b45d613
MAPREDUCE-6762. ControlledJob#toString failed with NPE when job status is not successfully updated (Weiwei Yang via Varun Saxena)
2016-08-21 21:46:17 +05:30
3d937457ee
MAPREDUCE-6310. Add jdiff support to MapReduce. (Li Lu/vinodkv via wangda)
2016-08-19 16:26:29 -07:00
723facfa40
MAPREDUCE-6763. Shuffle server listen queue is too small (Jason Lowe via Varun Saxena)
2016-08-20 03:58:16 +05:30
f80a729832
MAPREDUCE-6690. Limit the number of resources a single map reduce job can submit for localization. Contributed by Chris Trezzo
2016-08-17 16:22:31 +00:00
6c154abd33
MAPREDUCE-6751. Add debug log message when splitting is not possible due to unsplittable compression. (Peter Vary via rchiang)
2016-08-16 12:13:22 -07:00
c4b77aed7e
MAPREDUCE-6750. Fix TestHSAdminServer#testRefreshSuperUserGroups (Kihwal Lee via Varun Saxena)
2016-08-09 23:35:44 +05:30
4aba858750
MAPREDUCE-6741. Add MR support to redact job conf properties. Contributed by Haibo Chen
2016-08-09 15:55:35 +00:00
82c9e06101
YARN-5457. Refactor DistributedScheduling framework to pull out common functionality. (asuresh)
2016-08-09 00:42:29 -07:00
6529fe9345
MAPREDUCE-6748. Enhance logging for Cluster.java around InetSocketAddress (Vrushali C via Varun Saxena)
2016-08-08 21:49:54 +05:30
70c2781152
MAPREDUCE-6730. Use StandardCharsets instead of String overload in TextOutputFormat. Contributed by Sahil Kang.
...
This closes #114
2016-08-04 20:13:11 +09:00
8f1c374bec
MAPREDUCE-6682. TestMRCJCFileOutputCommitter fails intermittently Contributed by Akira Ajisaka.
2016-08-04 11:21:47 +05:30
6890d5b472
MAPREDUCE-6724. Single shuffle to memory must not exceed Integer#MAX_VALUE. (Haibo Chen via gera)
2016-08-01 23:35:47 -07:00
ce93595d7a
MAPREDUCE-6729. Accurately compute the test execute time in DFSIO. Contributed by mingleizhang.
...
This closes #112
2016-07-30 22:13:19 +09:00
3f0bffea2d
MAPREDUCE-6746. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon.
2016-07-30 11:45:12 +09:00
54fe17a607
HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák.
2016-07-27 19:41:09 +09:00
703fdf86c6
MAPREDUCE-6744. Increase timeout on TestDFSIO tests. Contributed by Eric Badger
2016-07-25 21:54:48 +00:00
ecff7d0359
MAPREDUCE-6738. TestJobListCache.testAddExisting failed intermittently in slow VM testbed. Contributed by Junping Du.
2016-07-21 11:34:48 -07:00
521f34317a
HADOOP-12928. Update netty to 3.10.5.Final to sync with zookeeper. (lei)
2016-07-20 20:38:20 -07:00
8f0d3d69d6
MAPREDUCE-6365. Refactor JobResourceUploader#uploadFilesInternal (Chris Trezzo via sjlee)
2016-07-19 20:15:37 -07:00
da456ffd62
Preparing for 3.0.0-alpha2 development
2016-07-15 19:04:17 -07:00
0881ed3fc3
MAPREDUCE-6652. Add configuration property to prevent JHS from loading jobs with a task count greater than X (haibochen via rkanter)
2016-07-15 13:30:03 -07:00
1d053836b1
MAPREDUCE-6733. MapReduce JerseyTest tests failing with java.net.BindException: Address already in use. Contributed by Vinod Kumar Vavilapalli.
2016-07-15 11:23:32 -07:00
ad256beb50
MAPREDUCE-6625. TestCLI#testGetJob fails occasionally (haibochen via rkanter)
2016-07-12 15:51:18 -07:00
d14e729a55
MAPREDUCE-6731. TestMRTimelineEventHandling#testMRNewTimelineServiceEventHandling fails randomly for concurrent tests. (Sangjin Lee via Varun Saxena).
2016-07-12 20:45:53 +05:30
819224dcf9
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:36:20 -07:00
6cf6ab7b78
Made a number of miscellaneous fixes for javac, javadoc, and checstyle warnings.
2016-07-10 08:46:05 -07:00
8bf87eede2
MAPREDUCE-6720. Inconsistent values of counters across tasks and job reported to timeline service. Contributed by Varun Saxena
2016-07-10 08:46:04 -07:00
1ff6833bba
YARN-5243. fix several rebase and other miscellaneous issues before merge. (Sangjin Lee via Varun Saxena)
2016-07-10 08:46:03 -07:00
0a9b085f05
YARN-5189. Make HBaseTimeline[Reader|Writer]Impl default and move FileSystemTimeline*Impl. (Joep Rottinghuis and Sangjin Lee via gtcarrera9)
2016-07-10 08:46:01 -07:00
000a4d8e13
MAPREDUCE-6688. Store job configurations in Timeline Service v2 (Varun Saxena via sjlee)
2016-07-10 08:45:57 -07:00
f0dbd7a40f
MAPREDUCE-6424. Store MR counters as timeline metrics instead of event. (Naganarasimha G R via varunsaxena)
2016-07-10 08:45:57 -07:00
a3cf40e532
YARN-3461. Consolidate flow name/version/run defaults. (Sangjin Lee via Varun Saxena)
2016-07-10 08:45:55 -07:00
4ba6354717
MAPREDUCE-6546. reconcile the two versions of the timeline service performance tests. (Sangjin Lee via Naganarasimha G R)
2016-07-10 08:45:54 -07:00
0d02ab8729
YARN-3367. Replace starting a separate thread for post entity with event loop in TimelineClient (Naganarasimha G R via sjlee)
2016-07-10 08:45:53 -07:00
960af7d471
YARN-4409. Fix javadoc and checkstyle issues in timelineservice code (Varun Saxena via sjlee)
2016-07-10 08:45:53 -07:00
6934b05c71
YARN-4238. createdTime and modifiedTime is not reported while publishing entities to ATSv2. (Varun Saxena via Naganarasimha G R)
2016-07-10 08:45:52 -07:00
8ef546c1ee
YARN-4445. Unify the term flowId and flowName in timeline v2 codebase.
...
Contributed by Zhan Zhang.
2016-07-10 08:45:49 -07:00
89e5c44f9e
YARN-4356. Ensure the timeline service v.2 is disabled cleanly and has no
...
impact when it's turned off. Contributed by Sangjin Lee.
2016-07-10 08:45:48 -07:00
ae72f1dc77
YARN-4297. TestJobHistoryEventHandler and TestRMContainerAllocator failing on YARN-2928 branch (Varun Saxena via sjlee)
2016-07-10 08:45:47 -07:00
561c746cbe
MAPREDUCE-6370. Made the timeline service v2 test driver write event ID. Contributed by Li Lu.
...
(cherry picked from commit 827633ee9fee26e3e15343cbccb0b7905ae02170)
2016-07-10 08:45:41 -07:00
5088f6c76a
MAPREDUCE-6337. Added a mode to replay MR job history files and put them into the timeline service v2. Contributed by Sangjin Lee.
...
(cherry picked from commit 463e070a8e7c882706a96eaa20ea49bfe9982875)
2016-07-10 08:45:41 -07:00
fc8485d893
YARN-3551. Consolidate data model change according to the backend implementation (Zhijie Shen via sale)
...
(cherry picked from commit 557a3950bddc837469244835f5577899080115d8)
2016-07-10 08:45:35 -07:00
8c7b6dd2c7
MAPREDUCE-6335. Created MR job based performance test driver for the timeline service v2. Contributed by Sangjin Lee.
...
(cherry picked from commit b689f5d43d3f5434a30fe52f1a7e12e1fc5c71f4)
2016-07-10 08:45:34 -07:00
b50a6d78f5
MAPREDUCE-6327. Made MR AM use timeline service v2 API to write history events and counters. Contributed by Junping Du.
2016-07-10 08:45:34 -07:00
9d46a49c74
HADOOP-13320. Fix arguments check in documentation for WordCount v2.0. Contributed by niccolo becchi.
...
This closes #108
2016-07-07 15:30:39 -07:00
96fa0f848b
HADOOP-12709 Cut s3:// from trunk. Contributed by Mingliang Liu.
2016-06-29 16:04:50 +01:00
23c3ff85a9
HADOOP-13209. replace slaves with workers (John Smith via aw)
2016-06-28 05:53:03 -07:00
4fd37eed90
MAPREDUCE-5803. Counters page display all task nevertheless of task type( Map or Reduce). Contributed by Kai Sasaki.
2016-06-28 09:42:42 +05:30
0f23cd497b
MAPREDUCE-6725. Javadoc for CLI#listEvents() contains no-existent param. Contributed by shenyinjie.
2016-06-25 00:26:24 +09:00
69c21d2db0
MAPREDUCE-6641. TestTaskAttempt fails in trunk. Contributed by Haibo Chen.
2016-06-23 13:26:10 +09:00
79a7289165
MAPREDUCE-6721. mapreduce.reduce.shuffle.memory.limit.percent=0.0 should be legal to enforce shuffle to disk. (Gera Shegalov via ozawa)
...
This closes #102
2016-06-22 17:20:13 -07:00
d8107fcd1c
MAPREDUCE-6197. Cache MapOutputLocations in ShuffleHandler. Contributed by Junping Du
2016-06-21 14:25:58 -07:00
605b4b6136
MAPREDUCE-6719. The list of -libjars archives should be replaced with a wildcard in the distributed cache to reduce the application footprint in the state store (Daniel Templeton via sjlee)
2016-06-21 11:25:11 -07:00
5d58858bb6
HADOOP-9613. [JDK8] Update jersey version to latest 1.x release.
2016-06-21 08:05:32 +09:00
51d497fa93
MAPREDUCE-6542. HistoryViewer uses SimpleDateFormat, but SimpleDateFormat is not threadsafe. Contributed by zhangyubiao.
2016-06-17 11:03:54 +09:00
c77a1095dc
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:06:38 -07:00
e3ba9ad3f1
HADOOP-13270. BZip2CompressionInputStream finds the same compression marker twice in corner case, causing duplicate data blocks. Contributed by Kai Sasaki.
2016-06-14 10:18:41 +09:00
0b7b8a3776
MAPREDUCE-6741. Refactor UncompressedSplitLineReader.fillBuffer(). Contributed by Daniel Templeton.
2016-06-10 19:15:36 +09:00
0af96a1c08
MAPREDUCE-6240. Hadoop client displays confusing error message. (gera)
2016-06-08 12:59:37 -07:00
733f3f18d5
MAPREDUCE-6702. TestMiniMRChildTask.testTaskEnv and TestMiniMRChildTask.testTaskOldEnv are failing (ajisakaa via rkanter)
2016-06-07 15:46:06 -07:00
4a1cedc010
MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma)
2016-06-06 14:30:51 -07:00
dc26601d8f
YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh)
2016-06-02 09:01:02 -07:00
93d8a7f2a2
Revert "HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng"
...
This reverts commit a6c79f92d5
2016-06-01 08:41:15 +08:00
42f90ab885
YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 21:24:16 +05:30
013532a95e
YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino)
2016-05-25 16:55:49 -07:00
a6c79f92d5
HADOOP-12579. Deprecate and remove WriteableRPCEngine. Contributed by Kai Zheng
2016-05-27 04:29:42 +08:00
ae353ea969
MAPREDUCE-6703. Add flag to allow MapReduce AM to request for OPPORTUNISTIC containers. Contributed by Arun Suresh
2016-05-24 19:47:50 -07:00
3b3b63081b
MAPREDUCE-6607. Enable regex pattern matching when mapreduce.task.files.preserve.filepattern is set. Contributed by Kai Sasaki.
2016-05-23 07:24:31 +09:00
21d2b90213
MAPREDUCE-6696. Add a configuration to limit the number of map tasks allowed per job. Contributed by Zhihai Xu
2016-05-19 19:25:16 -07:00
22fcd819f0
YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron.
...
(cherry picked from commit 2703ec6871
2016-05-19 14:15:21 -07:00
992a49353f
MAPREDUCE-6686. Add a way to download the job config from the mapred CLI (rkanter)
2016-05-18 12:16:21 -07:00
f6ef876fe1
MAPREDUCE-6657. Job history server can fail on startup when NameNode is in start phase. Contributed by Haibo Chen.
2016-05-17 14:41:47 -07:00
34fddd1e91
MAPREDUCE-6698. Increase timeout on TestUnnecessaryBlockingOnHistoryFileInfo.testTwoThreadsQueryingDifferentJobOfSameUser. Contributed by Haibo Chen
2016-05-17 15:01:33 +00:00
12fa4ec141
MAPREDUCE-6701. application master log can not be available when clicking jobhistory's am logs link. Contributed by Haibo Chen
2016-05-17 14:54:13 +00:00
b357930526
MAPREDUCE-6693. ArrayIndexOutOfBoundsException occurs when the length of the job name is equal to mapreduce.jobhistory.jobname.limit. Contributed by Ajith S.
2016-05-17 20:19:01 +09:00
730bc746f9
HADOOP-12930. Dynamic subcommands for hadoop shell scripts (aw)
...
This commit contains the following JIRA issues:
HADOOP-12931. bin/hadoop work for dynamic subcommands
HADOOP-12932. bin/yarn work for dynamic subcommands
HADOOP-12933. bin/hdfs work for dynamic subcommands
HADOOP-12934. bin/mapred work for dynamic subcommands
HADOOP-12935. API documentation for dynamic subcommands
HADOOP-12936. modify hadoop-tools to take advantage of dynamic subcommands
HADOOP-13086. enable daemonization of dynamic commands
HADOOP-13087. env var doc update for dynamic commands
HADOOP-13088. fix shellprofiles in hadoop-tools to allow replacement
HADOOP-13089. hadoop distcp adds client opts twice when dynamic
HADOOP-13094. hadoop-common unit tests for dynamic commands
HADOOP-13095. hadoop-hdfs unit tests for dynamic commands
HADOOP-13107. clean up how rumen is executed
HADOOP-13108. dynamic subcommands need a way to manipulate arguments
HADOOP-13110. add a streaming subcommand to mapred
HADOOP-13111. convert hadoop gridmix to be dynamic
HADOOP-13115. dynamic subcommand docs should talk about exit vs. continue program flow
HADOOP-13117. clarify daemonization and security vars for dynamic commands
HADOOP-13120. add a --debug message when dynamic commands have been used
HADOOP-13121. rename sub-project shellprofiles to match the rest of Hadoop
HADOOP-13129. fix typo in dynamic subcommand docs
HADOOP-13151. Underscores should be escaped in dynamic subcommands document
HADOOP-13153. fix typo in debug statement for dynamic subcommands
2016-05-16 17:54:45 -07:00
9227dfc25f
MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg
2016-05-13 14:33:48 +00:00
3c5c57af28
HADOOP-13142. Change project version from 3.0.0 to 3.0.0-alpha1.
2016-05-12 18:27:28 -07:00
ca5613af91
Revert "Update project version to 3.0.0-alpha1-SNAPSHOT."
...
This reverts commit 6b53802cba
2016-05-12 15:32:45 -07:00
6b53802cba
Update project version to 3.0.0-alpha1-SNAPSHOT.
2016-05-12 11:05:05 -07:00
7eddecd357
MAPREDUCE-6639 Process hangs in LocatedFileStatusFetcher if FileSystem.get throws. Ryan Blue via stevel
2016-05-12 19:00:32 +01:00
5ffb54694b
MAPREDUCE-6684. High contention on scanning of user directory under immediate_done in Job History Server. Contributed by Haibo Chen
2016-05-10 16:03:49 +00:00
cd35b692de
MAPREDUCE-6678. Allow ShuffleHandler readahead without drop-behind. Contributed by Nathan Roberts.
2016-05-10 15:45:01 +00:00
c9bb96fa81
MAPREDUCE-6689. MapReduce job can infinitely increase number of reducer resource requests. Contributed by Wangda Tan
2016-05-06 22:25:47 +00:00
8d48266720
MAPREDUCE-6514. Fixed MapReduce ApplicationMaster to properly updated resources ask after ramping down of all reducers avoiding job hangs. Contributed by Varun Saxena and Wangda Tan.
2016-05-05 19:03:27 -07:00
9d3fcdfbb3
MAPREDUCE-6675. TestJobImpl.testUnusableNode failed (haibochen via rkanter)
2016-05-04 22:42:21 -07:00
ecce3b7d53
MAPREDUCE-6677. LocalContainerAllocator doesn't specify resource of the containers allocated (haibochen via rkanter)
2016-05-04 22:19:20 -07:00
4ee4e5ca2b
MAPREDUCE-6526. Remove usage of metrics v1 from hadoop-mapreduce. (aajisaka)
2016-05-03 10:46:11 +09:00
26ac0f8f0b
MAPREDUCE-6672. TestTeraSort fails on Windows. Contributed by Tibor Kiss.
2016-04-29 11:33:32 -07:00
63e5412f1a
HDFS-9427. HDFS should not default to ephemeral ports. Contributed by Xiaobing Zhou.
2016-04-22 15:14:40 -07:00
7da5847cf1
MAPREDUCE-2398. MRBench: setting the baseDir parameter has no effect. Contributed by Michael Noll and Wilfred Spiegelenburg.
2016-04-21 17:53:51 +05:30
1e48eefe58
MAPREDUCE-6680. JHS UserLogDir scan algorithm sometime could skip directory with update in CloudFS (Azure FileSystem, S3, etc. Contributed by Junping Du
2016-04-20 19:02:10 -07:00
e6c0742012
YARN-4965. Distributed shell AM failed due to ClientHandlerException thrown by jersey. Contributed by Junping Du
2016-04-16 19:39:18 -07:00
cc8b83a8e8
MAPREDUCE-6649. getFailureInfo not returning any failure info. Contributed by Eric Badger
2016-04-16 22:07:27 +00:00
8b2880c0b6
MAPREDUCE-6513. MR job got hanged forever when one NM unstable for some time. (Varun Saxena via wangda)
2016-04-14 11:00:53 -07:00
1fec06e037
MAPREDUCE-6633. AM should retry map attempts if the reduce task encounters commpression related errors. Contributed by Rushabh Shah
2016-04-09 16:51:57 +00:00
3be1ab485f
MAPREDUCE-6647. MR usage counters use the resources requested instead of the resources allocated (haibochen via rkanter)
2016-04-06 17:15:43 -07:00
de96d7c88a
MAPREDUCE-6670. TestJobListCache#testEviction sometimes fails on Windows with timeout. Contributed by Gergely Novák.
2016-04-06 08:32:35 -07:00
0a74610d1c
HADOOP-11393. Revert HADOOP_PREFIX, go back to HADOOP_HOME (aw)
2016-03-31 07:51:05 -07:00
690d8a368d
MAPREDUCE-6663. [NNBench] Refactor nnbench as a Tool implementation. Contributed by Brahma Reddy Battula.
2016-03-30 11:42:54 +09:00
0050fa5f1c
MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote.
2016-03-29 18:17:52 +09:00
0ef8bbfd87
MAPREDUCE-6662. Clear ASF Warnings on test data files. Contributed by Vinayakumar B
2016-03-29 10:32:21 +08:00
90fcb16c0f
MAPREDUCE-6655. Fix a typo (STRICT_IE6) in Encrypted Shuffle. Contributed by Wei-Chiu Chuang.
2016-03-28 05:30:42 -07:00
115be193df
Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote."
...
This reverts commit 2c268cc936
2016-03-28 16:05:35 +09:00
3f622a143c
MAPREDUCE-6535. TaskID default constructor results in NPE on toString(). Contributed by Daniel Templeton
2016-03-25 20:04:32 +00:00
2c268cc936
MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote.
2016-03-25 15:17:27 +09:00
738155063e
HADOOP-12857. rework hadoop-tools (aw)
2016-03-23 13:46:38 -07:00
368c77376a
MAPREDUCE-6656. [NNBench] OP_DELETE operation isn't working after MAPREDUCE-6363. Contributed by J.Andreina.
2016-03-23 23:22:43 +09:00
0bfe5a0167
MAPREDUCE-6580. Test failure: TestMRJobsWithProfiler. Contributed by Eric Badger.
2016-03-23 13:18:06 +09:00
57ead18a85
MAPREDUCE-6110. JobHistoryServer CLI throws NullPointerException with job ids that do not exist. (Kai Sasaki via gtcarrera9)
2016-03-22 14:27:32 -07:00
7fae4c68e6
YARN-4732. *ProcessTree classes have too many whitespace issues
2016-03-20 10:47:38 -07:00
80fa70c4e1
MAPREDUCE-6645. TestWordStats outputs logs under directories other than target/test-dir. Contributed by Gabor Liptak.
2016-03-17 18:08:31 +09:00
6529c87551
MAPREDUCE-6579. JobStatus#getFailureInfo should not output diagnostic information when the job is running. (Akira AJISAKA via wangda)
2016-03-15 17:28:40 -07:00
247a790609
MAPREDUCE-6520. Migrate MR Client test cases part 1. Contributed by Dustin Cote.
2016-03-11 22:51:20 +09:00
6876b9f9f9
Revert "MAPREDUCE-6520. Migrate MR Client test cases part 1."
...
This reverts commit ef68b44195
2016-03-11 22:50:57 +09:00
ef68b44195
MAPREDUCE-6520. Migrate MR Client test cases part 1.
2016-03-11 22:20:55 +09:00
89b16d27e2
HADOOP-12905. Clean up CHANGES.txt RAT exclusions from pom.xml files. Contributed by Chris Nauroth.
2016-03-08 21:22:06 -08:00
e51a8c1056
YARN-4737. Add CSRF filter support in YARN. Contributed by Jonathan Maron.
2016-03-07 15:26:44 +05:30
4f9fe3ac03
MAPREDUCE-6648. Add yarn.app.mapreduce.am.log.level to mapred-default.xml (harsh)
2016-03-07 13:09:57 +05:30
cbd31328a6
HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims)
2016-03-04 14:11:36 +09:00
1bc454615e
HADOOP-11792. Remove all of the CHANGES.txt files.
2016-03-03 21:02:31 -08:00
ff0ee84d77
MAPREDUCE-4785. TestMRApp occasionally fails (haibochen via rkanter)
2016-03-03 16:38:07 -08:00
0f72da7e28
MAPREDUCE-6622. Add capability to set JHS job cache to a task-based limit (rchiang via rkanter)
2016-02-26 17:57:17 -08:00
c4d4df8de0
YARN-4701. When task logs are not available, port 8041 is referenced instead of port 8042 (haibochen via rkanter)
2016-02-25 12:48:02 -08:00
4343a4cf77
MAPREDUCE-6640. mapred job -history command should be able to take Job ID (rkanter)
2016-02-23 17:37:49 -08:00
c6f2d761d5
MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.
2016-02-23 13:05:18 +05:30
b68901d7dd
YARN-4708. Missing default mapper type in TimelineServer performance test tool usage. Contributed by Kai Sasaki.
2016-02-22 07:54:53 +09:00
6eae4337d1
MAPREDUCE-6613. Change mapreduce.jobhistory.jhist.format default from json to binary (rchiang via rkanter)
2016-02-19 17:15:54 -08:00
37577852ba
MAPREDUCE-6637. Testcase Failure : TestFileInputFormat.testSplitLocationInfo. Contributed by Brahma Reddy Battula.
2016-02-19 16:33:17 -08:00
8eee59ce6b
MAPREDUCE-6627. Add machine-readable output to mapred job -history command (rkanter)
2016-02-18 17:58:26 -08:00
a365a3941c
HADOOP-12817. Enable TLS v1.1 and 1.2 (rkanter)
2016-02-18 11:09:50 -08:00
2440671a11
MAPREDUCE-6634. Log uncaught exceptions/errors in various thread pools in mapreduce. Contributed by Sidharta Seethana.
2016-02-18 14:18:36 +05:30
453e7e0d10
MAPREDUCE-6341. addendum patch to fix typos. Contributed by John Michael Luy.
2016-02-17 14:17:32 +09:00
ec12ce8f48
Set the release date for 2.6.4.
2016-02-13 01:42:50 -08:00
a429f857b2
Update CHANGES.txt for commit of MAPREDUCE-6191 to other branches.
2016-02-10 16:52:13 +00:00
5e565ce7a7
Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta."
...
This reverts commit fe124da5ff
2016-02-10 23:43:33 +09:00
a0b1f10a30
MAPREDUCE-6626. Reuse ObjectMapper instance in MapReduce. Contributed by Lin Yiqun.
2016-02-10 03:05:07 +09:00
fba6e9f5ef
Update CHANGES.txt to reflect recent commits to branch-2.7 and branch-2.6.
2016-02-09 16:54:45 +00:00
fe124da5ff
HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
2016-02-06 19:31:01 +09:00
414cf07650
Adding release 2.6.5 to CHANGES.txt
2016-02-02 18:06:10 -08:00
ccbba4a4de
MAPREDUCE-6620. Jobs that did not start are shown as starting in 1969 in the JHS web UI (haibochen via rkanter)
2016-02-02 15:19:35 -08:00
43e669b22d
MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal(). Contributed by Xuan Gong
2016-02-02 19:30:45 +00:00
9875325d5c
YARN-4340. Add list API to reservation system. (Sean Po via wangda)
2016-02-02 10:17:33 +08:00
59a212b6e1
MAPREDUCE-6618. YarnClientProtocolProvider leaking the YarnClient thread. Contributed by Xuan Gong
2016-02-01 16:05:06 +00:00
df99ea8a92
MAPREDUCE-6616. Fail to create jobhistory file if there are some multibyte characters in the job name. Contributed by Kousuke Saruta.
2016-01-29 16:20:29 +09:00
41da9a0773
MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth.
2016-01-28 14:46:01 -08:00
86560a4978
MAPREDUCE-6595. Fix findbugs warnings in OutputCommitter and FileOutputCommitter. Contributed by Akira AJISAKA.
2016-01-28 06:14:21 -08:00
dca0dc8ac2
MAPREDUCE-6431. JobClient should be an AutoClosable (haibochen via rkanter)
2016-01-27 17:11:07 -08:00
d0d7c22168
MAPREDUCE-6610. JobHistoryEventHandler should not swallow timeline response. Contributed by Li Lu
2016-01-25 23:01:03 -08:00
ec4d2d9f40
Release process for 2.7.2: Set the release date for 2.7.2
2016-01-25 15:45:57 -08:00
2b833297ce
Added MAPREDUCE-6614 to 2.8.0 in CHANGES.txt.
2016-01-25 11:41:19 +09:00
f5c8c857be
MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki.
2016-01-22 18:35:43 +09:00
d40859fab1
YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha)
2016-01-18 10:58:14 +01:00
5af5f9575a
HADOOP-12714. Fix hadoop-mapreduce-client-nativetask unit test which fails because it is not able to open the "glibc bug spill" file. (cmccabe)
2016-01-15 13:53:05 -08:00
9fbd579ab5
MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory. Contributed by Bibin A Chundatt
2016-01-15 16:52:53 +00:00
b1ed28fa77
HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe)
2016-01-14 11:02:34 -08:00
5cc44d18aa
MAPREDUCE-6601. Fix typo in Job#setUseNewAPI. Contributed by Kai Sasaki.
2016-01-15 00:44:24 +09:00
Junping Du
Masatake Iwasaki
Karthik Kambatla
Daniel Templeton
Jason Lowe
Andrew Wang
Allen Wittenauer
Akira Ajisaka
Arun Suresh
Wangda Tan
Brahma Reddy Battula
Jian He
Robert Kanter
Ravi Prakash
Rohith Sharma K S
Varun Saxena
Mingliang Liu
Naganarasimha
Kai Zheng
Karthik Kambatla
Chris Douglas
Wei-Chiu Chuang
Ray Chiang
Gera Shegalov
Lei Xu
Sangjin Lee
Li Lu
naganarasimha
Zhijie Shen
Steve Loughran
Tsuyoshi Ozawa
Sangjin Lee
Ming Ma
Varun Vasudev
Carlo Curino
Eric Payne
Vinod Kumar Vavilapalli
Chris Nauroth
Jing Zhao
Harsh J
Xuan
Robert Kanter
Colin Patrick Mccabe