0afc036deb
MAPREDUCE-7103. Fix TestHistoryViewerPrinter on windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
2018-06-05 12:24:57 -07:00
6d5e87aec2
HADOOP-15507. Add MapReduce counters about EC bytes read.
2018-06-04 21:13:47 -07:00
dad1bb868f
MAPREDUCE-7105. Fix TestNativeCollectorOnlyHandler.testOnCall on Windows because of the path format. Contributed by Giovanni Matteo Fumarola.
2018-06-04 12:55:54 -07:00
a804b7c9d2
MAPREDUCE-7102. Fix TestJavaSerialization for Windows due a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
2018-06-02 20:25:42 -07:00
d5e69d8994
MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user(addendum). Contributed by Sunil Govindan.
2018-06-02 09:27:03 +05:30
d1e2b80980
MAPREDUCE-7098. Upgrade common-langs version to 3.7 in hadoop-mapreduce-project
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-31 18:31:10 +09:00
88cbe57c06
MAPREDUCE-7097. MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user. Contributed by Sunil Govindan.
2018-05-28 14:05:49 +05:30
a2cdffb95a
MAPREDUCE-7094. LocalDistributedCacheManager leaves classloaders open, which leaks FDs. Contributed by Adam Szita.
2018-05-17 11:16:04 -07:00
274eee3284
MAPREDUCE-7095. Race conditions in closing FadvisedChunkedFile. (Miklos Szegedi via Haibo Chen)
2018-05-10 10:46:55 -07:00
c8b53c4364
YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter)
2018-05-10 09:31:59 -07:00
e9159db623
MAPREDUCE-7093. Use assertEquals instead of assertTrue(a == b) in TestMapReduceJobControlWithMocks
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-07 01:15:14 +09:00
8cdb032aff
YARN-8223. Improved yarn auxiliary service to load jar file from HDFS.
...
Contributed by Zian Chen
2018-05-04 12:36:31 -04:00
1a95a4524a
MAPREDUCE-7073. Optimize TokenCache#obtainTokensForNamenodesInternal
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-02 16:14:28 +09:00
68c6ec719d
MAPREDUCE-7086. Add config to allow FileInputFormat to ignore directories when recursive=false. Contributed by Sergey Shelukhin
2018-05-01 16:19:53 -05:00
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-27 19:21:37 +09:00
83e60cd2db
MAPREDUCE-7042. Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled. Contributed by Rohith Sharma K S.
2018-04-26 13:54:40 -07:00
42e248372c
Revert "HDFS-12515. Ozone: mvn package compilation fails on HDFS-7240. Contributed by Anu Engineer."
...
This fix was need because of change in Trunk. That patch has been reverted and change proposed by this
patch will be applied in trunk. So reverting now to match the changes in trunk. Please see the JIRA for
details.
This reverts commit 244e7a5f65
2018-04-26 05:36:04 -07:00
c14e8b17ec
HDFS-12515. Ozone: mvn package compilation fails on HDFS-7240. Contributed by Anu Engineer.
2018-04-26 05:36:04 -07:00
1b9ecc264a
MAPREDUCE-7072. mapred job -history prints duplicate counter in human output (wilfreds via rkanter)
2018-04-24 11:30:38 -07:00
e4c39f3247
HADOOP-15396. Some java source files are executable
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-19 08:35:38 +09:00
bb92bfb4ef
YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)
2018-04-17 11:15:20 -07:00
86dfb70875
Revert "YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)"
...
This reverts commit 0c14b6f0d8
2018-04-17 06:04:38 -07:00
0c14b6f0d8
YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)
2018-04-17 05:59:03 -07:00
995cba65fe
MAPREDUCE-7077. Pipe mapreduce job fails with Permission denied for jobTokenPassword. (Akira Ajisaka via wangda)
...
Change-Id: Ie8f01425d58409fa3661f768205b7616128c8aa4
(cherry picked from commit 035e0f97ea
2018-04-13 13:52:10 -07:00
1a407bc990
HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen
2018-04-12 23:27:51 -07:00
4571351ccc
MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan
2018-04-12 11:12:46 -05:00
6bb128dfb8
YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe.
2018-04-12 16:04:23 +00:00
6cc59a09e7
MAPREDUCE-7062. Update mapreduce.job.tags description for making use for ATSv2 purpose. Contributed by Charan Hebri.
2018-04-11 09:45:39 +05:30
6cf023f9b7
Added CHANGES/RELEASES/Jdiff for 3.1.0 release
...
Change-Id: Ied5067a996151c04d15cad46c46ac98b60c37b39
(cherry picked from commit 2d96570452a72569befdf9cfe9b90c9fa2e0e261)
2018-04-05 15:52:39 -07:00
345e7624d5
YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S
2018-04-04 15:08:03 -07:00
dde1579096
HADOOP-15146. Remove DataOutputByteBuffer.
...
Contributed by BELUGA BEHR.
2018-04-02 15:52:40 +01:00
edb202e493
MAPREDUCE-6441. Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes (wattsinabox, rchiang, haibochen via rkanter)
2018-03-26 14:55:53 -07:00
86816da5b4
MAPREDUCE-7066. TestQueue fails on Java9
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-19 09:39:49 +09:00
2626ec35e9
MAPREDUCE-7064. Flaky test TestTaskAttempt#testReducerCustomResourceTypes. Contributed by Peter Bacsko
2018-03-14 15:05:24 -05:00
3a8dade9b1
Revert "HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu."
...
This reverts commit 172b23af33
2018-03-09 22:46:30 -08:00
32fa3a63e0
MAPREDUCE-6930. mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml. Contributed by Sen Zhao
2018-03-09 10:41:16 -06:00
4971276631
HADOOP-15271. Remove unicode multibyte characters from JavaDoc
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-05 15:07:58 -08:00
6e6945cd78
MAPREDUCE-7059. Downward Compatibility issue: MR job fails because of unknown setErasureCodingPolicy method from 3.x client to HDFS 2.x cluster. Contributed by Jiandan Yang.
2018-03-01 10:18:53 +08:00
7dd385098c
MAPREDUCE-7010. Make Job History File Permissions configurable. Contributed by Gergely Novák
2018-02-26 14:32:46 -08:00
2fa7963c3d
HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi.
2018-02-24 14:41:55 -08:00
2bc3351eaf
HADOOP-6852. apparent bug in concatenated-bzip2 support (decoding). Contributed by Zsolt Venczel.
2018-02-21 12:57:14 -07:00
0898ff42e9
HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka
2018-02-16 14:49:00 -06:00
82f029f7b5
MAPREDUCE-7053: Timed out tasks can fail to produce thread dump. Contributed by Jason Lowe.
2018-02-16 08:15:09 -06:00
a53d62ab26
MAPREDUCE-7052. TestFixedLengthInputFormat#testFormatCompressedIn is flaky. Contributed by Peter Bacsko
2018-02-15 15:12:57 -06:00
87e2570a14
MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko
2018-02-12 13:21:09 -06:00
60f9e60b3b
Preparing for 3.2.0 development
...
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
4304fcd5bd
HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen.
2018-02-06 13:43:45 -08:00
5a725bb886
MAPREDUCE-7033: Map outputs implicitly rely on permissive umask for shuffle. Contributed by Jason Lowe
2018-01-31 15:45:30 -06:00
38af237969
YARN-6599. Support anti-affinity constraint via AppPlacementAllocator. (Wangda Tan via asuresh)
2018-01-31 01:30:17 -08:00
56feaa40bb
MAPREDUCE-7036. ASF License warning in hadoop-mapreduce-client
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-01-30 00:39:39 +09:00
6eef3d7f1a
MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko
2018-01-26 15:31:43 -06:00
a37e7f0ad8
MAPREDUCE-7022. Fast fail rogue jobs based on task scratch dir size. Contributed by Johan Gustavsson
2018-01-26 14:36:45 -06:00
cc10852252
MAPREDUCE-7041. MR should not try to clean up at first job attempt. (Gergo Repas via Haibo Chen)
2018-01-25 16:11:30 -08:00
cff9edd4b5
MAPREDUCE-7015. Possible race condition in JHS if the job is not loaded. Contributed by Peter Bacsko
2018-01-24 14:44:07 -06:00
836643d793
MAPREDUCE-6995. Uploader tool for Distributed Cache Deploy documentation (miklos.szegedi@cloudera.com via rkanter)
2018-01-19 17:57:54 -08:00
c191538ed1
HADOOP-15166 CLI MiniCluster fails with ClassNotFoundException o.a.h.yarn.server.timelineservice.collector.TimelineCollectorManager. Contributed by Gera Shegalov
2018-01-19 16:15:55 -08:00
cce71dceef
MAPREDUCE-6984. MR AM to clean up temporary files from previous attempt in case of no recovery. (Gergo Repas via Haibo Chen)
2018-01-19 12:56:17 -08:00
6e42d05829
MAPREDUCE-7029. FileOutputCommitter is slow on filesystems lacking recursive delete. Contributed by Karthik Palaniappan
2018-01-17 08:14:11 -06:00
d716084f45
MAPREDUCE-7032. Add the ability to specify a delayed replication count (miklos.szegedi@cloudera.com via rkanter)
2018-01-16 10:45:45 -08:00
1a9c5d479e
MAPREDUCE-7034. Moving logging APIs over to slf4j the rest of all in hadoop-mapreduce
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-01-15 15:38:03 +09:00
e404650f48
MAPREDUCE-7030. Uploader tool should ignore symlinks to the same directory (miklos.szegedi@cloudera.com via rkanter)
2018-01-12 14:18:01 -08:00
fe35103591
MAPREDUCE-7028. Concurrent task progress updates causing NPE in Application Master. Contributed by Gergo Repas
2018-01-03 11:01:38 -06:00
a7f8caf58e
Add 2.8.3 release jdiff files.
...
(cherry picked from commit c89f99aade
2017-12-18 22:39:01 -08:00
c7a4dda3c5
HADOOP-15109. TestDFSIO -read -random doesn't work on file sized 4GB. Contributed by Ajay Kumar.
2017-12-18 13:25:47 -08:00
2316f52690
MAPREDUCE-7018. Apply erasure coding properly to framework tarball and support plain tar (miklos.szegedi@cloudera.com via rkanter)
2017-12-11 14:00:42 -08:00
c2e8a5c229
MAPREDUCE-7000. Moving logging APIs over to slf4j in hadoop-mapreduce-client-nativetask. Contributed by Jinjiang Ling.
2017-12-07 16:27:08 +09:00
d4cae977a2
MAPREDUCE-6998. Moving logging APIs over to slf4j in hadoop-mapreduce-client-jobclient. Contributed by Gergely Novák.
2017-12-07 16:21:25 +09:00
e00c7f78c1
HADOOP-14976. Set HADOOP_SHELL_EXECNAME explicitly in scripts.
2017-12-04 21:02:04 -08:00
3b78607a02
MAPREDUCE-6994. Uploader tool for Distributed Cache Deploy code changes (miklos.szegedi@cloudera.com via rkanter)
2017-12-01 12:12:15 -08:00
21d3627355
MAPREDUCE-5124. AM lacks flow control for task events. Contributed by Peter Bacsko
2017-12-01 14:04:25 -06:00
3cd75845da
MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran.
2017-11-27 22:01:00 +05:30
de8b6ca5ef
HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
...
Contributed by Steve Loughran and Ryan Blue.
2017-11-22 15:28:12 +00:00
782ba3bf9d
MAPREDUCE-7011. TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
2017-11-21 20:42:28 -08:00
5eb7dbe9b3
Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh)
2017-11-09 15:15:51 -08:00
e4c220ee4f
MAPREDUCE-7001. Moving logging APIs over to slf4j in hadoop-mapreduce-client-shuffle. Contributed by Jinjiang Ling.
2017-11-08 19:28:08 +09:00
ffee10b68e
MAPREDUCE-6997. Moving logging APIs over to slf4j in hadoop-mapreduce-client-hs. Contributed by Gergely Novák.
2017-11-08 19:21:43 +09:00
61bbdc511e
MAPREDUCE-6975. Logging task counters. Contributed by Prabhu Joseph.
2017-11-06 12:58:38 +05:30
e565b5277d
HDFS-12682. ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED.
2017-11-02 21:27:35 -07:00
178751ed8c
MAPREDUCE-6983. Moving logging APIs over to slf4j in hadoop-mapreduce-client-core. Contributed by Jinjiang Ling.
2017-11-02 17:43:08 +09:00
2fae63aa60
HADOOP-14919. BZip2 drops records when reading data in splits. Contributed by Jason Lowe
2017-10-31 09:30:13 -05:00
9a7e810838
YARN-6927. Add support for individual resource types requests in MapReduce
...
(Contributed by Gergo Repas via Daniel Templeton)
2017-10-30 11:04:22 -07:00
665bb147aa
YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla
2017-10-27 16:56:05 -05:00
de09716fb8
MAPREDUCE-6977 Move logging APIs over to slf4j in hadoop-mapreduce-client-common.
...
Contributed by Jinjiang Ling.
2017-10-27 10:45:03 +01:00
a25b5aa0cf
Add several jdiff xml files for 2.8.2 release.
2017-10-26 19:30:48 -07:00
2523e1cce5
MAPREDUCE-6972. Enable try-with-resources for RecordReader. Contributed by Zoltan Haindrich.
2017-10-18 11:18:39 +09:00
e46d5bb962
MAPREDUCE-5951. Add support for the YARN Shared Cache.
2017-10-12 10:59:20 -07:00
639f98cc9d
MAPREDUCE-6951. Improve exception message when mapreduce.jobhistory.webapp.address is in wrong format. Contributed by Prabhu Joseph.
2017-10-11 14:27:38 +05:30
453d48bdfb
MAPREDUCE-6971. Moving logging APIs over to slf4j in hadoop-mapreduce-client-app. Contributed by Jinjiang Ling.
2017-10-03 12:14:54 +09:00
9df05005ac
MAPREDUCE-6968. Staging directory erasure coding config property has a typo. Contributed by Jason Lowe
2017-09-26 14:52:27 -05:00
908d8e914e
MAPREDUCE-6965. QuasiMonteCarlo should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari
2017-09-22 09:34:53 -05:00
c71d137960
MAPREDUCE-6966. DistSum should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari.
2017-09-22 15:07:59 +09:00
53be075241
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergely Novák.
2017-09-22 13:27:59 +09:00
b9db0afa1e
MAPREDUCE-6964. BaileyBorweinPlouffe should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari
2017-09-21 09:37:19 -05:00
53047f934e
Revert "MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák."
...
This reverts commit 2018538fdb
2017-09-21 11:16:05 +09:00
51edaacd09
HDFS-12445. Correct spellings of choosen to chosen. Contributed by hu xiaodong.
2017-09-19 13:48:23 -07:00
595d478408
MAPREDUCE-6960. Shuffle Handler prints disk error stack traces for every read failure.
2017-09-19 10:35:15 -05:00
3a20debdde
MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe
2017-09-19 09:13:17 -05:00
ea845ba58c
Revert "MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe"
...
This reverts commit b3d61304f2
2017-09-19 08:45:05 -05:00
2018538fdb
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák.
2017-09-19 11:05:54 +09:00
3cf3540f19
HADOOP-14835. mvn site build throws SAX errors. Contributed by Andrew Wang and Sean Mackrory.
2017-09-18 15:13:42 -07:00
b3d61304f2
MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe
2017-09-18 17:04:43 -05:00
0adc0471d0
MAPREDUCE-6954. Disable erasure coding for files that are uploaded to the MR staging area (pbacsko via rkanter)
2017-09-18 10:40:06 -07:00
5f496683fb
Revert "YARN-7162. Remove XML excludes file format (rkanter)" - wrong commit message
...
This reverts commit 3a8d57a0a2
2017-09-18 10:32:08 -07:00
3a8d57a0a2
YARN-7162. Remove XML excludes file format (rkanter)
2017-09-15 12:00:26 -07:00
11390c2d11
MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter.
...
Contributed by Steve Loughran
2017-09-15 16:59:04 +01:00
c3f35c422b
HADOOP-14089. Automated checking for malformed client. Contributed by Sean Busbey.
2017-09-13 16:57:50 -07:00
4d98936eec
MAPREDUCE-6957. shuffle hangs after a node manager connection timeout. Contributed by Jooseong Kim
2017-09-13 17:21:13 -05:00
556812c179
HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)
2017-09-12 10:19:34 -07:00
df3855541a
YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda)
...
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
2017-09-12 09:19:12 -07:00
c2032e251e
YARN-5708. Implement APIs to get resource profiles from the RM. Contributed by Varun Vasudev.
2017-09-12 09:19:10 -07:00
a323f73bae
MAPREDUCE-6953. Skip the testcase testJobWithChangePriority if FairScheduler is used (pbacsko via rkanter)
2017-09-08 13:14:36 -07:00
c35510a465
HADOOP-14849. some wrong spelling words update. Contributed by Chen Hongfei.
2017-09-08 10:02:34 -07:00
0d419c984f
Preparing for 3.1.0 development
2017-09-01 11:53:48 -07:00
190410085b
HADOOP-14364. refresh changelog/release notes with newer Apache Yetus build
...
Signed-off-by: Andrew Wang <wang@apache.org>
2017-08-31 19:06:49 -07:00
71bbb86d69
HADOOP-14670. Increase minimum cmake version for all platforms
...
Signed-off-by: Chris Douglas <cdouglas@apache.org>
2017-08-30 21:47:24 -07:00
3e0e2033cc
MAPREDUCE-6931. Remove TestDFSIO "Total Throughput" calculation. Contributed by Dennis Huo.
2017-08-30 14:07:23 -07:00
d04f85f387
MAPREDUCE-6892. Issues with the count of failed/killed tasks in the jhist file. (Peter Bacsko via Haibo Chen)
2017-08-30 10:07:48 -07:00
3d00c8f394
Made fixes for whitespace errors and checstyle warnings before merge.
2017-08-30 11:29:54 +05:30
08f40bcc7f
MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena
2017-08-30 11:29:54 +05:30
7594d1de7b
YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena.
2017-08-30 11:29:53 +05:30
684de1a902
YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R.
2017-08-30 11:29:52 +05:30
8499299740
MAPREDUCE-6818. Remove direct reference to TimelineClientImpl. Contributed by Li Lu.
2017-08-30 11:29:51 +05:30
092fead5d9
YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena.
2017-08-30 11:29:51 +05:30
ae7abad3d9
MAPREDUCE-6945. TestMapFileOutputFormat missing @after annotation. Contributed by Ajay Kumar
2017-08-27 15:19:55 -07:00
8b7cbe3840
HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar.
2017-08-25 09:55:46 -07:00
dd7916d3cd
HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas.
2017-08-17 09:35:36 -07:00
ab051bd42e
MAPREDUCE-6936. Remove unnecessary dependency of hadoop-yarn-server-common from hadoop-mapreduce-client-common (haibochen via rkanter)
2017-08-16 16:14:04 -07:00
0acc5e0036
MAPREDUCE-6940. Copy-paste error in the TaskAttemptUnsuccessfulCompletionEvent constructor. Contributed by Oleg Danilov
2017-08-16 16:34:06 -05:00
645a8f2a4d
HADOOP-14726. Mark FileStatus::isDir as final
2017-08-14 21:57:20 -07:00
ce797a1706
HDFS-12221. Replace xcerces in XmlEditsVisitor. (Ajay Kumar via lei)
2017-08-14 10:27:47 -07:00
4222c97108
HADOOP-10392. Use FileSystem#makeQualified(Path) instead of Path#makeQualified(FileSystem) (ajisakaa via aw)
2017-08-11 09:25:56 -07:00
a32e0138fb
MAPREDUCE-6870. Add configuration for MR job to finish when all reducers are complete. (Peter Bacsko via Haibo Chen)
2017-08-10 15:17:36 -07:00
ac7d0604bc
MAPREDUCE-6923. Optimize MapReduce Shuffle I/O for small partitions. Contributed by Robert Schmidtke.
2017-08-09 15:39:52 -07:00
735fce5bec
MAPREDUCE-6927. MR job should only set tracking url if history was successfully written. Contributed by Eric Badger
2017-08-08 14:46:47 -05:00
35dc782923
YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh)
2017-08-03 21:15:40 -07:00
b8e8241854
MAPREDUCE-6914. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu)
2017-08-03 11:44:34 -07:00
1a1bf6b7d0
HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer.
2017-08-02 12:25:05 -06:00
ceacadc51e
MAPREDUCE-6921. TestUmbilicalProtocolWithJobToken#testJobTokenRpc fails.
2017-08-01 14:56:42 +09:00
fbb7d6bcbb
Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)"
...
This reverts commit e8a31f2e1c
2017-07-31 14:09:16 -07:00
713349a9af
Revert "MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)"
...
This reverts commit 390a7c12f5
2017-07-28 14:21:04 -07:00
e15f928295
Revert "MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu."
...
This reverts commit 433542904a
2017-07-28 14:06:59 -07:00
38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier.
2017-07-28 09:10:52 +09:00
f66fd11e51
YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena)
2017-07-27 23:14:50 +05:30
cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor.
2017-07-25 15:24:56 +09:00
fb3b5d33ff
MAPREDUCE-6889. Add Job#close API to shutdown MR client services. Contributed by Rohith Sharma K S.
2017-07-17 13:35:15 +05:30
43f0503286
MAPREDUCE-6910. MapReduceTrackingUriPlugin can not return the right URI of history server with HTTPS. Contributed by Lantao Jin
2017-07-13 16:16:45 -07:00
f484a6ff60
MAPREDUCE-6246. DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2. Contributed by ramtin and Gergely Novák.
2017-07-07 13:26:16 -07:00
bf1f59937d
MAPREDUCE-6905. HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh. Contributed by LiXin Ge.
2017-07-02 15:56:27 +05:30
af2773f609
Updating version for 3.0.0-beta1 development
2017-06-29 17:57:40 -07:00
a5c0476a99
MAPREDUCE-6697. Concurrent task limits should only be applied when necessary. Contributed by Nathan Roberts.
2017-06-28 10:50:09 +09:00
2b87faf166
MAPREDUCE-6904. HADOOP_JOB_HISTORY_OPTS should be HADOOP_JOB_HISTORYSERVER_OPTS in mapred-config.sh (rkanter)
2017-06-26 17:35:55 -07:00
82bbcbf37f
MAPREDUCE-6897. Add Unit Test to ensure Job end notification gets sent even when appMaster stop gets YarnRuntimeException. Contributed by Gergely Novák
2017-06-16 14:25:44 -07:00
9ef2743fa1
Revert "MAPREDUCE-6898. TestKill.testKillTask is flaky. Contributed by Peter Bacsko"
...
This reverts commit e78542f4be
2017-06-16 14:48:51 -05:00
e78542f4be
MAPREDUCE-6898. TestKill.testKillTask is flaky. Contributed by Peter Bacsko
2017-06-16 14:24:54 -05:00
4ebc23ba7b
HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively.
2017-06-16 09:49:33 -07:00
6ed54f3439
MAPREDUCE-6895. Job end notification not send due to YarnRuntimeException. Contributed by yunjiong zhao.
2017-06-13 15:21:04 -07:00
036a24bab3
MAPREDUCE-6896. Document wrong spelling in usage of MapredTestDriver tools.
...
(Contributed by LiXin Ge via Daniel Templeton)
2017-06-13 13:21:23 -07:00
24181f559f
MAPREDUCE-6676. NNBench should Throw IOException when rename and delete fails. Contributed by Brahma Reddy Battula.
2017-06-07 16:34:47 +08:00
f5517a8200
HADOOP-13921. Remove log4j classes from JobConf.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-06-01 13:35:14 +09:00
d4015f8628
MAPREDUCE-6887. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin.
2017-05-30 14:48:58 +09:00
16ad896d5c
Update maven version for 3.0.0-alpha4 development
2017-05-26 14:09:44 -07:00
b6f66b0da1
YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang.
2017-05-22 14:10:06 +05:30
035d46836b
MAPREDUCE-6459. Native task crashes when merging spilled file on ppc64. Contributed by Tao Jie and Ayappan.
2017-05-17 07:50:29 -04:00
1e71fe8c42
MAPREDUCE-6882. Increase MapReduce test timeouts from 1 second to 10 seconds. Contributed by Eric Badger
2017-05-10 09:57:41 -05:00
45382bf047
AuditLogger and TestAuditLogger are dead code. Contributed by Vrushali C
2017-05-09 10:31:52 -07:00
e4f34ecb04
HADOOP-14390. Correct spelling of 'succeed' and variants. Contributed by Dongtao Zhang
2017-05-05 12:10:50 -07:00
3ed3062fe3
MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. Contributed by Weiwei Yang.
2017-04-27 15:45:33 +09:00
8a99eba96d
Revert "YARN-5007. Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster. Contributed by Andras Bokor."
...
This reverts commit 34ab8e73d4
2017-04-26 15:34:23 +09:00
3721cfe1fb
MAPREDUCE-6871. Allow users to specify racks and nodes for strict locality for AMs (rkanter)
2017-04-21 16:12:01 -07:00
dd43b895c2
YARN-6202. Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded
...
(Contributed by Yufei Gu via Daniel Templeton)
2017-04-19 11:44:55 -07:00
f1de2c8560
MAPREDUCE-6875. Rename mapred-site.xml.template to mapred-site.xml. (Yuanbo Liu via Haibo Chen)
2017-04-17 12:25:30 -07:00
ac3cfdf3e5
MAPREDUCE-6673. Add a test example job that grows in memory usage over time (Karthik Kambatla via Haibo Chen)
...
Change-Id: Iccfc8c67c38c526cc61726d87bfcbcf69ac36fea
2017-04-17 11:15:39 -05:00
96cbb4fce2
HADOOP-14202. fix jsvc/secure user var inconsistencies
...
Signed-off-by: John Zhuge <jzhuge@apache.org>
2017-04-07 08:59:21 -07:00
e7167e4a13
MAPREDUCE-6201. TestNetworkedJob fails on trunk (pbacsko via rkanter)
2017-04-06 17:44:47 -07:00
fc0885da29
MAPREDUCE-6846. Fragments specified for libjar paths are not handled correctly
...
(Contributed by Chris Trezzo via Daniel Templeton)
2017-04-05 17:25:40 -07:00
539ef5aa2e
HFDS-11596. hadoop-hdfs-client jar is in the wrong directory in release tarball. Contributed by Yuanbo Liu.
2017-04-05 16:04:09 -07:00
34ab8e73d4
YARN-5007. Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster. Contributed by Andras Bokor.
2017-04-05 17:33:07 +09:00
6eba79232f
HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang
2017-04-03 20:13:14 -07:00
845529b3ab
MAPREDUCE-6824. TaskAttemptImpl#createCommonContainerLaunchContext is longer than 150 lines. Contributed by Chris Trezzo.
2017-04-03 13:06:54 +09:00
4d1fac5df2
YARN-6411. Clean up the overwrite of createDispatcher() in subclass of MockRM. Contributed by Yufei Gu
2017-03-31 10:05:34 -05:00
c9b7ce9273
MAPREDUCE-6836. exception thrown when accessing the job configuration web UI. Contributed by Haibo Chen
2017-03-30 12:55:32 -05:00
7297e82cb4
HADOOP-14250. Correct spelling of 'separate' and variants. Contributed by Doris Gu
2017-03-30 10:17:11 -07:00
c8bd5fc7a8
MAPREDUCE-6850. Shuffle Handler keep-alive connections are closed from the server side. Contributed by Jonathan Eagles
2017-03-30 10:57:19 -05:00
de7efd2687
MAPREDUCE-6873. MR Job Submission Fails if MR framework application path not on defaultFS. Contributed by Erik Krogen.
2017-03-29 22:10:55 -07:00
ceab00ac62
MAPREDUCE-6862. Fragments are not handled correctly by resource limit checking. (Chris Trezzo via mingma)
2017-03-29 17:41:58 -07:00
9bae6720cb
YARN-6050. AMs can't be scheduled on racks or nodes (rkanter)
2017-03-27 17:23:13 -07:00
945b006fe8
MAPREDUCE-6868. License check for jdiff output files should be ignored. Contributed by Akira Ajisaka.
2017-03-26 23:18:05 -07:00
e0a2eb71e3
MAPREDUCE-6866. Fix getNumMapTasks() documentation in JobConf.
...
This closes #205
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-03-27 13:51:03 +09:00
6df029db36
Add COMMON/MAPREDUCE/YARN jdiff of 2.8.0 release.
2017-03-25 14:28:48 -07:00
34a931c0fa
MAPREDUCE-6865. Fix typo in javadoc for DistributedCache. Contributed by Attila Sasvari.
2017-03-20 00:04:23 +05:30
4ebe8a6a23
MAPREDUCE-6859. hadoop-mapreduce-client-jobclient.jar sets a main class that isn't in the JAR. Contributed by Daniel Templeton
2017-03-08 10:27:57 -06:00
38d75dfd3a
MAPREDUCE-6839. TestRecovery.testCrashed failed (pairg via rkanter)
2017-03-07 13:34:46 -08:00
f01a69f84f
Treat encrypted files as private. Contributed by Daniel Templeton.
2017-03-07 13:22:11 +09:00
14413989ca
MAPREDUCE-6855. Specify charset when create String in CredentialsTestJob. Contributed by Kai Sasaki.
2017-03-07 13:10:59 +09:00
209ecd1a5c
HADOOP-14108. CLI MiniCluster: add an option to specify NameNode HTTP port. Contributed by Takanobu Asanuma.
2017-03-07 01:34:51 +09:00
490abfb10f
HADOOP-14057. Fix package.html to compile with Java 9.
2017-03-04 00:25:22 +09:00
19a1fc6373
MAPREDUCE-6753. Variable in byte printed directly in mapreduce client. Contributed by Kai Sasaki.
2017-03-03 16:11:10 +09:00
747bafaf96
MAPREDUCE-6852. Job#updateStatus() failed with NPE due to race condition. Contributed by Junping Du
2017-03-02 10:58:56 -08:00
82ef9accaf
HDFS-11428. Change setErasureCodingPolicy to take a required string EC policy name. Contributed by Andrew Wang.
2017-03-01 16:06:02 +05:30
4d33683882
MAPREDUCE-6841. Fix dead link in MapReduce tutorial document. Contributed by Victor Nee.
2017-02-27 19:46:37 +09:00
732ee6f0b5
MAPREDUCE-6825. YARNRunner#createApplicationSubmissionContext method is longer than 150 lines (Contributed by Gergely Novák via Daniel Templeton)
2017-02-22 15:38:11 -08:00
1a6ca75f38
YARN-6143. Fix incompatible issue caused by YARN-3583. (Sunil G via wangda)
2017-02-22 11:17:09 -08:00
172b23af33
HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu.
2017-02-18 18:34:13 -08:00
4fa1afdb88
YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R.
2017-02-16 11:41:04 -08:00
deb368b1b6
MAPREDUCE-6842. Update the links in PiEstimator document (Contributed by Jung Yoo via Daniel Templeton)
2017-02-07 13:05:40 -08:00
7afe5b1a31
MAPREDUCE-6404. Allow AM to specify a port range for starting its webapp. Contributed by Varun Saxena.
2017-02-07 01:40:49 -08:00
d401e63b6c
MAPREDUCE-6338. MR AppMaster does not honor ephemeral port range. Contributed by Frank Nguyen.
2017-02-05 19:28:01 -08:00
3619ae32be
MAPREDUCE-6644. Use doxia macro to generate in-page TOC of MapReduce site documentation. (iwasakims)
2017-02-02 01:19:36 +09:00
c65f884fc7
MAPREDUCE-6829. Add peak memory usage counter for each task. (Miklos Szegedi via kasha)
2017-01-26 11:08:13 -08:00
a7463b6c88
MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log (Contributed by Gergő Pásztor via Daniel Templeton)
2017-01-25 14:32:40 -08:00
69fa81679f
YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He
2017-01-23 17:12:51 +00:00
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
0eb4b513b7
HADOOP-13673. Update scripts to be smarter when running with privilege
...
Signed-off-by: Andrew Wang <wang@apache.org>
Signed-off-by: Ravi Prakash <raviprak@apache.org>
2017-01-18 14:39:05 -08:00
78b487bde1
MAPREDUCE-6831. Flaky test TestJobImpl.testKilledDuringKillAbort. Contributed by Peter Bacsko
2017-01-17 16:01:42 +00:00
7ee8be1aa3
MAPREDUCE-6819. Replace UTF8 with Text in MRBench. Contributed by Peter Bacsko.
2017-01-17 17:56:10 +09:00
85826f6ca5
YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh)
...
(cherry picked from commit 4985217de4
2017-01-08 09:04:20 -08:00
679478d0c6
MAPREDUCE-6711. JobImpl fails to handle preemption events on state COMMITTING. Contributed by Prabhu Joseph.
2017-01-07 22:04:18 -08:00
0b8a7c18dd
MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu via Daniel Templeton)
2017-01-05 17:56:08 -08:00
0a55bd841e
YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda)
2017-01-05 10:31:05 -08:00
a0a2761621
HADOOP-13578. Add Codec for ZStandard Compression. Contributed by churro morales
2017-01-04 14:46:25 +00:00
01d31fe938
HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
2017-01-04 14:10:36 +09:00
6938b67711
HADOOP-13883. Addendum patch to correct the message and alphabetized with the earlier patch. Contributed by Yiqun Lin.
2016-12-31 16:16:02 +05:30
95c2c24cd0
MAPREDUCE-5155. Race condition in test case TestFetchFailure cause it to fail (Contributed by Haibo Chen via Daniel Templeton)
2016-12-29 10:50:47 -08:00
ded2d08f33
HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton)
2016-12-27 07:18:08 -08:00
ae401539ea
YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor.
2016-12-22 14:32:24 +09:00
523411d69b
YARN-4844 (Addendum). Change JobStatus(usedMem,reservedMem,neededMem)
...
back to int for compatibility. Contributed by Wangda Tan
2016-12-20 11:28:44 -08:00
2273a74c1f
YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh)
2016-12-16 08:14:34 -08:00
b0b033ea2e
MAPREDUCE-6821. Fix javac warning related to the deprecated APIs after upgrading Jackson. Contributed by Yiqin Lin.
2016-12-13 14:22:07 +09:00
92a8917ca5
MAPREDUCE-6820. Fix dead links in Job relevant classes. Contributed by Yiqun Lin.
2016-12-10 10:55:41 +09:00
4b149a1e77
YARN-4457. Cleanup unchecked types for EventHandler (templedf via rkanter)
2016-12-09 17:28:25 -08:00
f54afdb83b
MAPREDUCE-4683. Create and distribute hadoop-mapreduce-client-core-tests.jar.
2016-12-08 11:27:58 +09:00
a793cec2c9
MAPREDUCE-6817. The format of job start time in JHS is different from those of submit and finish time. (Haibo Chen via kasha)
2016-12-07 13:36:49 -08:00
da4ecc9f82
MAPREDUCE-6816. Progress bars in Web UI always at 100%. Contributed by Shen Yinjie.
2016-12-07 06:41:08 +09:00
6b4a9970f1
MAPREDUCE-6571. JobEndNotification info logs are missing in AM container syslog (haibochen via rkanter)
2016-12-06 13:11:03 -08:00
b2a3d6c519
HADOOP-13835. Move Google Test Framework code from mapreduce to hadoop-common. Contributed by Varun Vasudev.
2016-12-06 14:01:47 +09:00
0cfd7ad21f
MAPREDUCE-6815. Fix flaky TestKill.testKillTask(). Contributed by Haibo Chen
2016-12-02 17:22:11 +00:00
c87b3a448a
MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job overview page in JHS (haibochen via rkanter)
2016-12-01 17:29:38 -08:00
7c848719de
MAPREDUCE-6810. Fix hadoop-mapreduce-client-nativetask compilation with GCC-6.2.1. Contributed by Ravi Prakash.
2016-11-30 10:47:41 -08:00
8f6e14399a
MAPREDUCE-6565. Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution. Contributed by Li Lu.
2016-11-29 15:51:27 -08:00
6d8b4f6c27
YARN-4395. Typo in comment in ClientServiceDelegate (Contributed by Alison Yu via Daniel Templeton)
2016-11-29 15:30:22 -08:00
be88d574a9
MAPREDUCE-6743. nativetask unit tests need to provide usable output; fix link errors during mvn test
2016-11-28 15:24:44 -08:00
209e805430
HADOOP-13506. Redundant groupid warning in child projects. Contributed by Kai Sasaki.
2016-11-28 14:34:57 +09:00
683e0c71fe
MAPREDUCE-6793. io.sort.factor code default and mapred-default.xml values inconsistent. Contributed by Gera Shegalov.
2016-11-21 22:10:57 +05:30
7584fbf4cb
MAPREDUCE-6801. Fix flaky TestKill.testKillJob (Haibo Chen via Varun Saxena)
2016-11-19 01:37:59 +05:30
b8690a9d25
MAPREDUCE-6811. TestPipeApplication#testSubmitter fails after HADOOP-13802 (Brahma Reddy Battula via Varun Saxena)
2016-11-17 00:51:07 +05:30
7ef290cfef
MAPREDUCE-6804. Add timeout when starting JobHistoryServer in MiniMRYarnCluster. Contributed by Andras Bokor
2016-11-16 15:23:12 +00:00
5af572b644
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
2016-11-15 10:57:00 -08:00
3219b7b4ac
YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh)
2016-11-15 07:56:25 -08:00
99c2bbd337
MAPREDUCE-6797. Job history server scans can become blocked on a single, slow entry. Contributed by Prabhu Joseph
2016-11-14 20:20:50 +00:00
bcc15c6290
YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S
2016-11-09 16:08:05 -08:00
Inigo Goiri
Xiao Chen
Rohith Sharma K S
Takanobu Asanuma
Miklos Szegedi
Haibo Chen
Robert Kanter
Abhishek Modi
Eric Yang
Bibin A Chundatt
Jason Lowe
Ewan Higgs
Sunil G
Anu Engineer
Shashikant Banerjee
Wangda Tan
Erik Krogen
Eric E Payne
Vrushali C
Steve Loughran
Owen O'Malley
Weiwei Yang
Billie Rinaldi
Arpit Agarwal
Sean Mackrory
Eric Payne
Arun Suresh
Junping Du
Chen Liang
Akira Ajisaka
Chris Douglas
Naganarasimha
Daniel Templeton
Chris Trezzo
Andrew Wang
Ray Chiang
Varun Vasudev
Allen Wittenauer
Konstantin V Shvachko
Varun Saxena
Jian He
Sangjin Lee
Mingliang Liu
Lei Xu
Ravi Prakash
Jason Lowe
Yufei Gu
Ravi Prakash
Brahma Reddy Battula
Sean Busbey
Haibo Chen
Zhe Zhang
Ming Ma
Joe Mészáros
Rakesh Radhakrishnan
Masatake Iwasaki
Karthik Kambatla