9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
159a715eef
MAPREDUCE-7192. JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable. Contributed by Jiandan Yang.
2019-03-13 17:00:35 +08:00
f0605146b3
MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang.
2019-03-11 15:02:54 +08:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
c991e2ca5a
MAPREDUCE-7177. Disable speculative execution in TestDFSIO. Contributed by Zhaohui Xin.
2019-02-04 14:03:34 +09:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
1ab69a9543
YARN-9221. Added flag to disable dynamic auxiliary service feature.
...
Contributed by Billie Rinaldi
2019-01-25 19:05:36 -05:00
a33ef4fd31
YARN-8867. Added resource localization status to YARN service status call.
...
Contributed by Chandni Singh
2019-01-24 18:43:21 -05:00
2fa9389c2e
YARN-9146. Added REST API to configure auxiliary service.
...
Contributed by Billie Rinaldi
2019-01-22 18:24:43 -05:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
f82922dcfa
YARN-5168. Added exposed port information for Docker container.
...
Contributed by Xun Liu
2018-12-21 19:44:07 -05:00
7affa3053c
YARN-9131. Updated document usage for dynamic auxiliary service.
...
Contributed by Billie Rinaldi
2018-12-20 18:45:49 -05:00
499c70eda5
MAPREDUCE-7166. map-only job should ignore node lost event when task is already succeeded. Contributed by Lei Li.
2018-12-20 10:09:50 +09:00
c7a5a4435e
YARN-9075. Add ability to register/remove auxiliary service without restart node manager.
...
Contributed by Billie Rinaldi
2018-12-18 17:05:51 -05:00
e0d75088f2
MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. Contributed by Zhaohui Xin.
2018-12-13 16:26:25 +09:00
881230da21
YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth)
2018-12-11 11:41:32 -08:00
9886c27c71
MAPREDUCE-7159. FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used. Contributed by Peter Bacsko
2018-12-06 14:48:17 -06:00
4d8de7ab69
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla
2018-11-28 14:54:59 -06:00
13a21f6607
MAPREDUCE-6190. If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck. Contributed by Zhaohui Xin.
2018-11-28 17:57:42 +09:00
1aad99a718
MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin.
2018-11-27 20:01:11 +09:00
27ffec7ba7
YARN-8778. Add command line interface to invoke interactive docker shell. Contributed by Eric Yang
2018-11-19 17:59:12 -08:00
1e15c7e855
HADOOP-15936. [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier. Contributed by Zsolt Venczel.
2018-11-15 12:42:31 -08:00
762a56cc64
MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)
...
Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a
2018-11-13 11:25:41 -08:00
0b6625a973
MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan
2018-11-07 08:20:49 -06:00
ba1f9d66d9
MAPREDUCE-7156. NullPointerException when reaching max shuffle connections. Contributed by Peter Bacsko
2018-11-06 17:55:51 -06:00
ab6aa4c726
YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
2018-11-06 08:40:59 -06:00
74a5e683fe
MAPREDUCE-7151. RMContainerAllocator#handleJobPriorityChange expects application_priority always. Contributed by Bilwa S T.
2018-10-24 23:22:05 +05:30
823bb5dda8
MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter)
2018-10-23 15:28:13 -07:00
babd1449bf
MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its memory. (Contributed by Misha Dmitriev)
2018-10-16 13:44:41 -07:00
25f8fcb064
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.
2018-10-16 10:23:31 -07:00
735a36afd5
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection.
...
Contributed by Steve Loughran.
2018-10-11 20:01:42 +01:00
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.
2018-10-10 10:16:57 +09:00
5b1cfcaeff
MAPREDUCE-7035. Skip javadoc build for auto-generated sources in hadoop-mapreduce-client. Contributed by Mukul Kumar Singh.
2018-10-09 21:13:04 +09:00
31fa9bb799
HADOOP-15767. [JDK10] Building native package on JDK10 fails due to missing javah. Contributed by Takanobu Asanuma.
2018-10-03 11:29:18 +09:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
230d90c5b9
MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda)
...
Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a
2018-09-24 13:52:53 -07:00
8382b860d4
MAPREDUCE-7138. ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler. Contributed by Oleksandr Shevchenko
2018-09-18 17:06:32 -05:00
34b2237e42
MAPREDUCE-7137. MRAppBenchmark.benchmark1() fails with NullPointerException. Contributed by Oleksandr Shevchenko
2018-09-18 16:56:31 -05:00
2cf8927683
MAPREDUCE-3801: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently. Contributed by Jason Lowe
2018-09-18 20:26:28 +00:00
488806baca
MAPREDUCE-7140. Refactoring TaskAttemptInfo to separate Map and Reduce tasks. Contributed by Oleksandr Shevchenko
2018-09-14 15:10:27 -05:00
2886024ac3
MAPREDUCE-7133. History Server task attempts REST API returns invalid data. Contributed by Oleksandr Shevchenko
2018-09-13 14:41:38 -05:00
8cf6a9a2bd
YARN-7892. Revisit NodeAttribute class structure. Contributed by Naganarasimha G R.
2018-09-12 16:01:00 +05:30
5dc7d6e0f3
YARN-8104. Add API to fetch node to attribute mapping. Contributed by Bibin A Chundatt.
2018-09-12 16:01:00 +05:30
0a01b1350d
YARN-8100. Support API interface to query cluster attributes and attribute to nodes. Contributed by Bibin A Chundatt.
2018-09-12 16:01:00 +05:30
598380ad5a
MAPREDUCE-7136. TestMRAppMetrics should shutdown DefaultMetricsSystem after completion. Contributed by Oleksandr Shevchenko
2018-09-11 13:10:50 -05:00
7ce997af97
MAPREDUCE-7135. TestTaskAttemptContainerRequest should reset UserGroupInformation. Contributed by Oleksandr Shevchenko
2018-09-11 11:16:33 -05:00
eb0b5a844f
MAPREDUCE-7131. Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediate. Contributed by Anthony Hsu
2018-09-06 14:13:29 -05:00
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs.
...
Contributed by Steve Loughran.
2018-08-30 14:49:53 +01:00
84973d1049
MAPREDUCE-6861. Add metrics tags for ShuffleClientMetrics. (Contributed by Zoltan Siegl)
2018-08-27 16:53:06 +02:00
b3b4d4ccb5
MAPREDUCE-7118. Distributed cache conflicts breaks backwards compatability. (Jason Lowe via wangda)
...
Change-Id: I89ab4852b4ad305fec19812e8931c59d96581376
2018-07-19 14:26:05 -07:00
88625f5cd9
HADOOP-15531. Use commons-text instead of commons-lang in some classes to fix deprecation warnings. Contributed by Takanobu Asanuma.
2018-07-13 11:42:12 -04:00
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
ae055622ed
MAPREDUCE-7114. Make FrameworkUploader symlink ignore improvement. Contributed by Gergo Repas.
2018-06-22 13:05:41 -07:00
d6ee4290df
MAPREDUCE-7113. Typos in test names in TestTaskAttempt: "testAppDiognostic". Contributed by Szilard Nemeth.
2018-06-20 12:04:44 -07:00
1e94e5977f
MAPREDUCE-7063. Fix log level inconsistency in CombineFileInputFormat.java
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-06-18 14:25:11 -07:00
5670e89b2e
MAPREDUCE-7101. Add config parameter to allow JHS to alway scan user dir irrespective of modTime. (Thomas Marquardt via asuresh)
2018-06-12 15:36:52 -07:00
04b74eddc6
MAPREDUCE-7108. TestFileOutputCommitter fails on Windows. (Zuoming Zhang via gera)
2018-06-12 11:21:51 -07:00
0afc036deb
MAPREDUCE-7103. Fix TestHistoryViewerPrinter on windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
2018-06-05 12:24:57 -07:00
6d5e87aec2
HADOOP-15507. Add MapReduce counters about EC bytes read.
2018-06-04 21:13:47 -07:00
dad1bb868f
MAPREDUCE-7105. Fix TestNativeCollectorOnlyHandler.testOnCall on Windows because of the path format. Contributed by Giovanni Matteo Fumarola.
2018-06-04 12:55:54 -07:00
a804b7c9d2
MAPREDUCE-7102. Fix TestJavaSerialization for Windows due a mismatch line separator. Contributed by Giovanni Matteo Fumarola.
2018-06-02 20:25:42 -07:00
d5e69d8994
MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user(addendum). Contributed by Sunil Govindan.
2018-06-02 09:27:03 +05:30
d1e2b80980
MAPREDUCE-7098. Upgrade common-langs version to 3.7 in hadoop-mapreduce-project
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-31 18:31:10 +09:00
88cbe57c06
MAPREDUCE-7097. MapReduce JHS should honor yarn.webapp.filter-entity-list-by-user. Contributed by Sunil Govindan.
2018-05-28 14:05:49 +05:30
a2cdffb95a
MAPREDUCE-7094. LocalDistributedCacheManager leaves classloaders open, which leaks FDs. Contributed by Adam Szita.
2018-05-17 11:16:04 -07:00
274eee3284
MAPREDUCE-7095. Race conditions in closing FadvisedChunkedFile. (Miklos Szegedi via Haibo Chen)
2018-05-10 10:46:55 -07:00
c8b53c4364
YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter)
2018-05-10 09:31:59 -07:00
e9159db623
MAPREDUCE-7093. Use assertEquals instead of assertTrue(a == b) in TestMapReduceJobControlWithMocks
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-07 01:15:14 +09:00
8cdb032aff
YARN-8223. Improved yarn auxiliary service to load jar file from HDFS.
...
Contributed by Zian Chen
2018-05-04 12:36:31 -04:00
1a95a4524a
MAPREDUCE-7073. Optimize TokenCache#obtainTokensForNamenodesInternal
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-02 16:14:28 +09:00
68c6ec719d
MAPREDUCE-7086. Add config to allow FileInputFormat to ignore directories when recursive=false. Contributed by Sergey Shelukhin
2018-05-01 16:19:53 -05:00
84ecfe3ceb
HADOOP-14188. Remove the usage of org.mockito.internal.util.reflection.Whitebox
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-27 19:21:37 +09:00
83e60cd2db
MAPREDUCE-7042. Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled. Contributed by Rohith Sharma K S.
2018-04-26 13:54:40 -07:00
1b9ecc264a
MAPREDUCE-7072. mapred job -history prints duplicate counter in human output (wilfreds via rkanter)
2018-04-24 11:30:38 -07:00
e4c39f3247
HADOOP-15396. Some java source files are executable
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-04-19 08:35:38 +09:00
bb92bfb4ef
YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)
2018-04-17 11:15:20 -07:00
86dfb70875
Revert "YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)"
...
This reverts commit 0c14b6f0d8
2018-04-17 06:04:38 -07:00
0c14b6f0d8
YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen)
2018-04-17 05:59:03 -07:00
995cba65fe
MAPREDUCE-7077. Pipe mapreduce job fails with Permission denied for jobTokenPassword. (Akira Ajisaka via wangda)
...
Change-Id: Ie8f01425d58409fa3661f768205b7616128c8aa4
(cherry picked from commit 035e0f97ea
2018-04-13 13:52:10 -07:00
1a407bc990
HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen
2018-04-12 23:27:51 -07:00
4571351ccc
MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan
2018-04-12 11:12:46 -05:00
6bb128dfb8
YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe.
2018-04-12 16:04:23 +00:00
6cc59a09e7
MAPREDUCE-7062. Update mapreduce.job.tags description for making use for ATSv2 purpose. Contributed by Charan Hebri.
2018-04-11 09:45:39 +05:30
345e7624d5
YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S
2018-04-04 15:08:03 -07:00
dde1579096
HADOOP-15146. Remove DataOutputByteBuffer.
...
Contributed by BELUGA BEHR.
2018-04-02 15:52:40 +01:00
edb202e493
MAPREDUCE-6441. Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes (wattsinabox, rchiang, haibochen via rkanter)
2018-03-26 14:55:53 -07:00
86816da5b4
MAPREDUCE-7066. TestQueue fails on Java9
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-19 09:39:49 +09:00
2626ec35e9
MAPREDUCE-7064. Flaky test TestTaskAttempt#testReducerCustomResourceTypes. Contributed by Peter Bacsko
2018-03-14 15:05:24 -05:00
3a8dade9b1
Revert "HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu."
...
This reverts commit 172b23af33
2018-03-09 22:46:30 -08:00
32fa3a63e0
MAPREDUCE-6930. mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml. Contributed by Sen Zhao
2018-03-09 10:41:16 -06:00
6e6945cd78
MAPREDUCE-7059. Downward Compatibility issue: MR job fails because of unknown setErasureCodingPolicy method from 3.x client to HDFS 2.x cluster. Contributed by Jiandan Yang.
2018-03-01 10:18:53 +08:00
7dd385098c
MAPREDUCE-7010. Make Job History File Permissions configurable. Contributed by Gergely Novák
2018-02-26 14:32:46 -08:00
2fa7963c3d
HADOOP-15254. Correct the wrong word spelling 'intialize'. Contributed by fang zhenyi.
2018-02-24 14:41:55 -08:00
2bc3351eaf
HADOOP-6852. apparent bug in concatenated-bzip2 support (decoding). Contributed by Zsolt Venczel.
2018-02-21 12:57:14 -07:00
0898ff42e9
HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka
2018-02-16 14:49:00 -06:00
82f029f7b5
MAPREDUCE-7053: Timed out tasks can fail to produce thread dump. Contributed by Jason Lowe.
2018-02-16 08:15:09 -06:00
a53d62ab26
MAPREDUCE-7052. TestFixedLengthInputFormat#testFormatCompressedIn is flaky. Contributed by Peter Bacsko
2018-02-15 15:12:57 -06:00
87e2570a14
MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko
2018-02-12 13:21:09 -06:00
Steve Loughran
Weiwei Yang
Akira Ajisaka
Eric Yang
Haibo Chen
Jason Lowe
Billie Rinaldi
Arpit Agarwal
Wangda Tan
bibinchundatt
Xiao Chen
Sunil G
Eric E Payne
Naganarasimha
Todd Lipcon
Miklos Szegedi
Vidura Mudalige
Arun Suresh
Gera Shegalov
Inigo Goiri
Rohith Sharma K S
Takanobu Asanuma
Miklos Szegedi
Robert Kanter
Abhishek Modi
Ewan Higgs
Shashikant Banerjee
Erik Krogen
Vrushali C
Owen O'Malley
Sean Mackrory
Eric Payne