Commit Graph

2030 Commits

Author SHA1 Message Date
Jungtaek Lim ebdacedc83
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. (#2624)
Contributed by Jungtaek Lim.

Change-Id: Iaff2f55e5378c22ce8a92ae776f5aba3f0fc304e
2021-01-27 19:08:54 +00:00
Steve Loughran 5be450393c
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. (#2579)
Part of the HADOOP-16830 IOStatistics API feature.

If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null.

Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.

Contributed by Steve Loughran.

Change-Id: Iff1485297c2c7e181b54eaf1d2c4f80faeee7cfa
2021-01-14 13:20:38 +00:00
lzx404243 b3f85576e9
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499)
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad3)
2021-01-12 15:45:54 +09:00
dengzh bc0c3ece0f HADOOP-17390. Skip license check on lz4 code files (#2478)
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license

Contributed by Zhihua Deng.
2020-12-29 13:32:26 -08:00
Liang-Chi Hsieh 87064df1f2 HADOOP-17292. Using lz4-java in Lz4Codec (#2350)
Contributed by Liang-Chi Hsieh.
2020-12-29 13:17:26 -08:00
Ayush Saxena 8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Akira Ajisaka efd74cc8ae
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136)
(cherry picked from commit 3e7b65c3e5)
2020-12-04 10:07:58 +09:00
Akira Ajisaka db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287)
2020-11-26 11:34:39 +09:00
Peter Bacsko 8ed565382f MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan. 2020-11-25 11:36:58 +01:00
dengzh abc87aef18
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. (#2469)
Contributed by Zhihua Deng.

Change-Id: Iee62539d02bd8f8a928171d8258e640487050a05
2020-11-23 16:33:41 +00:00
Peter Bacsko ced08fd87f MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:14:49 +01:00
Akira Ajisaka 71c106b412
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. (#2463)
(cherry picked from commit 2045a9df13)
2020-11-18 14:16:45 +09:00
Akira Ajisaka c40f0f1eb3
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko.
(cherry picked from commit 7bc305db5d)
2020-10-23 04:41:37 +09:00
Swaroopa Kadam 05a73ded93
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-10-12 11:09:08 -07:00
Liang-Chi Hsieh 8f60a90688 HADOOP-17125. Use snappy-java in SnappyCodec (#2297)
This switches the SnappyCodec to use the java-snappy codec, rather than the native one.

To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.

This comesin as an avro dependency,  so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7

Contributed by DB Tsai and Liang-Chi Hsieh

Change-Id: Id52a404a0005480e68917cd17f0a27b7744aea4e
2020-10-06 17:15:17 +01:00
zz e5e91397de
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223)
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-09-19 23:26:37 -07:00
ywheel 2efa28cb79
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338)
(cherry picked from commit cf4eb75608)
2020-07-30 13:28:35 +09:00
Ayush Saxena e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Ahmed Hussein 5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
2020-07-15 09:57:36 -05:00
Vinayakumar B 534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Eric Badger 890617c7ac Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)."
This reverts commit 741fcf2c63.
2020-06-08 20:25:02 +00:00
Wilfred Spiegelenburg 586d9427e3
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c)
2020-05-28 13:36:21 +10:00
Akira Ajisaka dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
Szilard Nemeth e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori
(cherry picked from commit ab36429559)
2020-04-28 18:56:29 +02:00
Eric E Payne 741fcf2c63 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles).
(cherry picked from commit e2322e1117)
2020-04-27 19:34:38 +00:00
Eric E Payne b397a3a875 MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein)
(cherry picked from commit 11d17417ce)
2020-04-13 18:51:00 +00:00
Szilard Nemeth ef01e9d40e MAPREDUCE-7266. historyContext doesn't need to be a class attribute inside JobHistoryServer. Contributed by Siddharth Ahuja
(cherry picked from commit e2c87b7917)
2020-04-09 12:40:22 +02:00
Akira Ajisaka cb806221e9
MAPREDUCE-7269. TestNetworkedJob fails (#1940)
(cherry picked from commit ab7495dc7d)
2020-04-06 17:58:25 +09:00
Wanqiang Ji 215a62e99f
MAPREDUCE-7268. Fix TestMapreduceConfigFields (#1935)
(cherry picked from commit f8e598f278)
2020-04-03 11:00:02 +09:00
Wanqiang Ji ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters (#1397) 2020-03-16 11:28:36 +09:00
Szilard Nemeth 8d6ff87c18 MAPREDUCE-7263. Remove obsolete validateTargetPath() from FrameworkUploader. Contributed by Marton Hudaky 2020-02-12 15:53:33 +01:00
Vinayakumar B 7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Szilard Nemeth 71b2c2ffe9 YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-06 12:25:06 +01:00
Eric E Payne b897f6834b MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein) 2020-01-29 16:54:45 +00:00
Szilard Nemeth 7f3e1e0c07 MAPREDUCE-7260. Cross origin request support for Job history server web UI. Contributed by Adam Antal 2020-01-29 14:42:52 +01:00
Ahmed Hussein 08251538fe MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-28 10:57:33 -06:00
Ahmed Hussein 3f01c48106 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-27 15:50:13 -06:00
Szilard Nemeth 9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
kevinzhao1661 b8518e6dab MAPREDUCE-7247. Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int (#1684) 2020-01-16 12:53:56 +09:00
Masatake Iwasaki a2fdd7c2b5 MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. (#1804) 2020-01-15 12:09:09 +09:00
Szilard Nemeth 13cea0412c YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-14 11:00:08 +01:00
Eric E Payne 93233a7d6e YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 2020-01-09 17:18:44 +00:00
Ahmed Hussein cdd6efd3ab MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-08 11:08:13 -06:00
Sergey Pogorelov b343e1533b MAPREDUCE-7255. Fix typo in MapReduce documentaion example (#1793) 2020-01-06 12:36:11 +09:00
Sunil G 682e6fdeda YARN-9052. Replace all MockRM submit method definitions with a builder. Contributed by Szilard Nemeth. 2019-12-05 13:26:23 +05:30
prabhujoseph da9aa9c909 MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0.
Contributed by Peter Bacsko.
2019-12-05 12:32:57 +05:30
prabhujoseph 304e75a48b Revert "MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0."
This reverts commit 8d82c44102.
2019-12-05 12:31:13 +05:30
prabhujoseph 8d82c44102 MAPREDUCE-7250. FrameworkUploader skip replication check if timeout is 0.
Contributed by Peter Bacsko.
2019-12-04 20:20:56 +05:30
prabhujoseph 013c6e51b6 MAPREDUCE-7249. Fix Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure.
Contributed by Wilfred Spiegelenburg.
2019-11-28 10:57:15 +05:30
prabhujoseph 7f2ea2ac46 MAPREDUCE-7240. Fix Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER.
Contributed by Huachao and Peter Bacsko. Reviewed by Wilfred Spiegelenburg.
2019-11-27 14:27:27 +05:30