5969922305
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f
2020-07-15 09:57:36 -05:00
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:20:10 +05:30
890617c7ac
Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)."
...
This reverts commit 741fcf2c63
2020-06-08 20:25:02 +00:00
586d9427e3
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
...
Contributed by Tarun Parimi.
(cherry picked from commit 10db97df1c
2020-05-28 13:36:21 +10:00
dfa7f160a5
Preparing for 3.3.1 development
2020-04-30 13:33:42 +09:00
e45faae1d4
YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori
...
(cherry picked from commit ab36429559
2020-04-28 18:56:29 +02:00
741fcf2c63
MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles).
...
(cherry picked from commit e2322e1117
2020-04-27 19:34:38 +00:00
b397a3a875
MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein)
...
(cherry picked from commit 11d17417ce
2020-04-13 18:51:00 +00:00
ef01e9d40e
MAPREDUCE-7266. historyContext doesn't need to be a class attribute inside JobHistoryServer. Contributed by Siddharth Ahuja
...
(cherry picked from commit e2c87b7917
2020-04-09 12:40:22 +02:00
cb806221e9
MAPREDUCE-7269. TestNetworkedJob fails ( #1940 )
...
(cherry picked from commit ab7495dc7d
2020-04-06 17:58:25 +09:00
215a62e99f
MAPREDUCE-7268. Fix TestMapreduceConfigFields ( #1935 )
...
(cherry picked from commit f8e598f278
2020-04-03 11:00:02 +09:00
ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters ( #1397 )
2020-03-16 11:28:36 +09:00
8d6ff87c18
MAPREDUCE-7263. Remove obsolete validateTargetPath() from FrameworkUploader. Contributed by Marton Hudaky
2020-02-12 15:53:33 +01:00
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency ( #1635 ). Contributed by Vinayakumar B.
2020-02-07 14:51:24 +05:30
71b2c2ffe9
YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal
2020-02-06 12:25:06 +01:00
b897f6834b
MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein)
2020-01-29 16:54:45 +00:00
7f3e1e0c07
MAPREDUCE-7260. Cross origin request support for Job history server web UI. Contributed by Adam Antal
2020-01-29 14:42:52 +01:00
08251538fe
MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-28 10:57:33 -06:00
3f01c48106
MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-27 15:50:13 -06:00
9520b2ad79
YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal
2020-01-22 16:25:07 +01:00
b8518e6dab
MAPREDUCE-7247. Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int ( #1684 )
2020-01-16 12:53:56 +09:00
a2fdd7c2b5
MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. ( #1804 )
2020-01-15 12:09:09 +09:00
13cea0412c
YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal
2020-01-14 11:00:08 +01:00
93233a7d6e
YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla)
2020-01-09 17:18:44 +00:00
cdd6efd3ab
MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-08 11:08:13 -06:00
b343e1533b
MAPREDUCE-7255. Fix typo in MapReduce documentaion example ( #1793 )
2020-01-06 12:36:11 +09:00
682e6fdeda
YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth.
2019-12-05 13:26:23 +05:30
da9aa9c909
MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0.
...
Contributed by Peter Bacsko.
2019-12-05 12:32:57 +05:30
304e75a48b
Revert "MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0."
...
This reverts commit 8d82c44102
2019-12-05 12:31:13 +05:30
8d82c44102
MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0.
...
Contributed by Peter Bacsko.
2019-12-04 20:20:56 +05:30
013c6e51b6
MAPREDUCE-7249. Fix Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure.
...
Contributed by Wilfred Spiegelenburg.
2019-11-28 10:57:15 +05:30
7f2ea2ac46
MAPREDUCE-7240. Fix Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER.
...
Contributed by Huachao and Peter Bacsko. Reviewed by Wilfred Spiegelenburg.
2019-11-27 14:27:27 +05:30
ed302f1fed
MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2019-11-05 14:55:20 -06:00
ac6b6a6a85
HADOOP-16614. Add aarch64 support for dependent leveldbjni.
...
Contributed by liusheng
close #1546
2019-10-24 11:45:57 -04:00
5462d2176f
YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena
2019-10-08 11:20:13 -07:00
1921e94292
HADOOP-16458. LocatedFileStatusFetcher.getFileStatuses failing intermittently with S3
...
Contributed by Steve Loughran.
Includes
-S3A glob scans don't bother trying to resolve symlinks
-stack traces don't get lost in getFileStatuses() when exceptions are wrapped
-debug level logging of what is up in Globber
-Contains HADOOP-13373. Add S3A implementation of FSMainOperationsBaseTest.
-ITestRestrictedReadAccess tests incomplete read access to files.
This adds a builder API for constructing globbers which other stores can use
so that they too can skip symlink resolution when not needed.
Change-Id: I23bcdb2783d6bd77cf168fdc165b1b4b334d91c7
2019-10-01 18:11:05 +01:00
e8e7d7b4cd
HADOOP-16561. [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes ( #1500 )
...
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
2019-09-24 08:52:12 +08:00
bc2d3a71d6
HADOOP-16549. Remove Unsupported SSL/TLS Versions from Docs/Properties. Contributed by Daisuke Kobayashi.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2019-09-10 10:51:47 +08:00
5d497abe21
YARN-9817. Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero. Contributed by Prabhu Joseph.
2019-09-07 07:39:02 +08:00
3f4f097f1f
MAPREDUCE-7230. TestHSWebApp.testLogsViewSingle fails. Contributed by Prabhu Joseph
2019-08-15 17:07:43 +02:00
ac6c4f0b29
MAPREDUCE-7197. Fix order of actual and expected expression in assert statements. Contributed by Adam Antal
2019-08-12 13:54:28 +02:00
a7371a779c
MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko.
2019-08-01 13:01:30 +02:00
8a59cd1b8a
HDFS-12967. NNBench should support multi-cluster access. Contributed by Chen Zhang.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-07-29 05:55:12 -07:00
c0a0c353e8
MAPREDUCE-6973. Fix comments on creating _SUCCESS file.
...
This closes #280
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-26 21:21:26 +09:00
ee87e9a42e
MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build
...
This closes #1089
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-07-23 10:34:10 +09:00
79f6118dcc
MAPREDUCE-6521. MiniMRYarnCluster should not create /tmp/hadoop-yarn/staging on local filesystem in unit test.
...
Contributed by Masatake Iwasaki.
Change-Id: Id74b90eb7cfb9d676188fc5aa47249ee8904a3d5
2019-07-18 13:15:18 +01:00
b417a4c854
MAPREDUCE-7214. Remove unused pieces related to `mapreduce.job.userlog.retain.hours`
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-11 18:40:35 +09:00
3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java
2019-06-07 10:27:41 +09:00
e7e30a5f8b
MAPREDUCE-7210. Replace `mapreduce.job.counters.limit` with `mapreduce.job.counters.max` in mapred-default.xml
...
This closes #878
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-06-05 10:45:23 +09:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
5565f2c532
MAPREDUCE-7198. mapreduce.task.timeout=0 configuration used to disable timeout doesn't work.
2019-05-23 10:21:11 +09:00
67f9a7b165
MAPREDUCE-7205. Treat container scheduler kill exit code as a task attempt killing event.
...
This closes #821
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2019-05-22 14:23:13 +09:00
d7979079ea
HADOOP-16210. Update guava to 27.0-jre in hadoop-project trunk. Contributed by Gabor Bota.
2019-04-03 12:59:39 -06:00
856cbf62d3
MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-04-01 10:25:38 -07:00
246ab77f28
HADOOP-16196. Path Parameterize Comparable.
...
Author: David Mollitor <david.mollitor@cloudera.com>
2019-03-22 10:26:24 +00:00
9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
159a715eef
MAPREDUCE-7192. JobHistoryServer attempts page support jump to containers log page in NM when logAggregation is disable. Contributed by Jiandan Yang.
2019-03-13 17:00:35 +08:00
f0605146b3
MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang.
2019-03-11 15:02:54 +08:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
c991e2ca5a
MAPREDUCE-7177. Disable speculative execution in TestDFSIO. Contributed by Zhaohui Xin.
2019-02-04 14:03:34 +09:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
1ab69a9543
YARN-9221. Added flag to disable dynamic auxiliary service feature.
...
Contributed by Billie Rinaldi
2019-01-25 19:05:36 -05:00
a33ef4fd31
YARN-8867. Added resource localization status to YARN service status call.
...
Contributed by Chandni Singh
2019-01-24 18:43:21 -05:00
2fa9389c2e
YARN-9146. Added REST API to configure auxiliary service.
...
Contributed by Billie Rinaldi
2019-01-22 18:24:43 -05:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
f82922dcfa
YARN-5168. Added exposed port information for Docker container.
...
Contributed by Xun Liu
2018-12-21 19:44:07 -05:00
7affa3053c
YARN-9131. Updated document usage for dynamic auxiliary service.
...
Contributed by Billie Rinaldi
2018-12-20 18:45:49 -05:00
499c70eda5
MAPREDUCE-7166. map-only job should ignore node lost event when task is already succeeded. Contributed by Lei Li.
2018-12-20 10:09:50 +09:00
c7a5a4435e
YARN-9075. Add ability to register/remove auxiliary service without restart node manager.
...
Contributed by Billie Rinaldi
2018-12-18 17:05:51 -05:00
e0d75088f2
MAPREDUCE-7170. Doc typo in PluggableShuffleAndPluggableSort.md. Contributed by Zhaohui Xin.
2018-12-13 16:26:25 +09:00
881230da21
YARN-9051. Integrate multiple CustomResourceTypesConfigurationProvider implementations into one. (Contributed by Szilard Nemeth)
2018-12-11 11:41:32 -08:00
9886c27c71
MAPREDUCE-7159. FrameworkUploader: ensure proper permissions of generated framework tar.gz if restrictive umask is used. Contributed by Peter Bacsko
2018-12-06 14:48:17 -06:00
4d8de7ab69
MAPREDUCE-7164. FileOutputCommitter does not report progress while merging paths. Contributed by Kuhu Shukla
2018-11-28 14:54:59 -06:00
13a21f6607
MAPREDUCE-6190. If a task stucks before its first heartbeat, it never timeouts and the MR job becomes stuck. Contributed by Zhaohui Xin.
2018-11-28 17:57:42 +09:00
1aad99a718
MAPREDUCE-7162. TestEvents#testEvents fails. Contributed by Zhaohui Xin.
2018-11-27 20:01:11 +09:00
27ffec7ba7
YARN-8778. Add command line interface to invoke interactive docker shell. Contributed by Eric Yang
2018-11-19 17:59:12 -08:00
1e15c7e855
HADOOP-15936. [JDK 11] MiniDFSClusterManager & MiniHadoopClusterManager compilation fails due to the usage of '_' as identifier. Contributed by Zsolt Venczel.
2018-11-15 12:42:31 -08:00
762a56cc64
MAPREDUCE-7158. Inefficient Flush Logic in JobHistory EventWriter. (Zichen Sun via wangda)
...
Change-Id: I99ace87980da03bb35a8012cea7218d602a8817a
2018-11-13 11:25:41 -08:00
0b6625a973
MAPREDUCE-7148. Fast fail jobs when exceeds dfs quota limitation. Contributed by Wang Yan
2018-11-07 08:20:49 -06:00
ba1f9d66d9
MAPREDUCE-7156. NullPointerException when reaching max shuffle connections. Contributed by Peter Bacsko
2018-11-06 17:55:51 -06:00
ab6aa4c726
YARN-8865. RMStateStore contains large number of expired RMDelegationToken. Contributed by Wilfred Spiegelenburg
2018-11-06 08:40:59 -06:00
74a5e683fe
MAPREDUCE-7151. RMContainerAllocator#handleJobPriorityChange expects application_priority always. Contributed by Bilwa S T.
2018-10-24 23:22:05 +05:30
823bb5dda8
MAPREDUCE-4669. MRAM web UI does not work with HTTPS. (Contributed by Robert Kanter)
2018-10-23 15:28:13 -07:00
babd1449bf
MAPREDUCE-7150. Optimize collections used by MR JHS to reduce its memory. (Contributed by Misha Dmitriev)
2018-10-16 13:44:41 -07:00
25f8fcb064
MAPREDUCE-7132. JobSplitWriter prints unnecessary warnings if EC(RS10,4) is used. Contributed by Peter Bacsko.
2018-10-16 10:23:31 -07:00
735a36afd5
MAPREDUCE-7149. Javadocs for FileInputFormat and OutputFormat to mention DT collection.
...
Contributed by Steve Loughran.
2018-10-11 20:01:42 +01:00
6fa3feb577
HADOOP-15832. Upgrade BouncyCastle to 1.60. Contributed by Robert Kanter.
2018-10-10 10:16:57 +09:00
5b1cfcaeff
MAPREDUCE-7035. Skip javadoc build for auto-generated sources in hadoop-mapreduce-client. Contributed by Mukul Kumar Singh.
2018-10-09 21:13:04 +09:00
31fa9bb799
HADOOP-15767. [JDK10] Building native package on JDK10 fails due to missing javah. Contributed by Takanobu Asanuma.
2018-10-03 11:29:18 +09:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
230d90c5b9
MAPREDUCE-7125. JobResourceUploader creates LocalFileSystem when it's not necessary. (Peter Cseh via wangda)
...
Change-Id: I1aa720ed03739f6f4abeec46f6068e2ab332987a
2018-09-24 13:52:53 -07:00
8382b860d4
MAPREDUCE-7138. ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler. Contributed by Oleksandr Shevchenko
2018-09-18 17:06:32 -05:00
34b2237e42
MAPREDUCE-7137. MRAppBenchmark.benchmark1() fails with NullPointerException. Contributed by Oleksandr Shevchenko
2018-09-18 16:56:31 -05:00
2cf8927683
MAPREDUCE-3801: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently. Contributed by Jason Lowe
2018-09-18 20:26:28 +00:00
488806baca
MAPREDUCE-7140. Refactoring TaskAttemptInfo to separate Map and Reduce tasks. Contributed by Oleksandr Shevchenko
2018-09-14 15:10:27 -05:00
2886024ac3
MAPREDUCE-7133. History Server task attempts REST API returns invalid data. Contributed by Oleksandr Shevchenko
2018-09-13 14:41:38 -05:00
8cf6a9a2bd
YARN-7892. Revisit NodeAttribute class structure. Contributed by Naganarasimha G R.
2018-09-12 16:01:00 +05:30
Ahmed Hussein
Vinayakumar B
Eric Badger
Wilfred Spiegelenburg
Akira Ajisaka
Szilard Nemeth
Eric E Payne
Wanqiang Ji
kevinzhao1661
Masatake Iwasaki
Sergey Pogorelov
Sunil G
prabhujoseph
prabhujoseph
Eric Yang
Jonathan Hung
Steve Loughran
Duo Zhang
Daisuke Kobayashi
Tao Yang
Chen Zhang
Mehul Garnara
pingsutw
Gabor Bota
Adam Antal
David Mollitor
Steve Loughran
Weiwei Yang
Haibo Chen
Jason Lowe
Billie Rinaldi
Arpit Agarwal
Wangda Tan
bibinchundatt
Xiao Chen
Eric E Payne