61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 ( #3990 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-26 20:31:16 +09:00
7328c34ba5
MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS
...
This is a mapreduce/spark output committer optimized for
performance and correctness on Azure ADLS Gen 2 storage
(via the abfs connector) and Google Cloud Storage
(via the external gcs connector library).
* It is safe to use with HDFS, however it has not been optimized
for that use.
* It is *not* safe for use with S3, and will fail if an attempt
is made to do so.
Contributed by Steve Loughran
Change-Id: I6f3502e79c578b9fd1a8c1485f826784b5421fca
2022-03-17 11:24:13 +00:00
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies ( #4000 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-03-08 17:27:04 +09:00
08c803ea30
MAPREDUCE-7371. DistributedCache alternative APIs should not use DistributedCache APIs internally ( #3855 )
2022-01-09 00:18:10 +09:00
bface2ac6c
MAPREDUCE-7368. DBOutputFormat.DBRecordWriter#write must throw exception when it fails. ( #3671 ). Contributed by Stamatis Zampetakis.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-08 16:40:11 +05:30
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages ( #3712 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom ( #3555 )
2021-10-21 16:54:25 +09:00
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
b1ad4eab9a
HADOOP-17959. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-cloud-storage-project and hadoop-mapreduce-project modules ( #3537 )
...
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
2021-10-11 16:22:50 +09:00
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
f5148ca542
MAPREDUCE-7363. Rename JobClientUnitTest to TestJobClients ( #3487 )
2021-09-28 09:50:01 -07:00
2ee294b1b1
Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko."
...
This reverts commit 7bc305db5d
2021-09-25 09:29:33 -07:00
6187f76f11
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter ( #2500 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
2021-09-01 13:47:09 +09:00
7b5be74228
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() ( #2968 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-30 08:39:33 +09:00
e31169c864
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup ( #1808 )
...
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-02 15:36:53 +09:00
e95c3259de
MAPREDUCE-7356. Remove some duplicate dependencies from mapreduce-client's child poms ( #3193 ). Contributed by Viraj Jasani.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-07-13 19:30:13 +05:30
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
7581413156
MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST)
2021-07-07 20:43:44 +00:00
3f4221ec34
MAPREDUCE-7351 - CleanupJob during handle of SIGTERM signal ( #3176 )
...
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
2021-07-07 09:08:15 +05:30
7c7d02edbd
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
2021-06-25 20:32:08 +00:00
6e11461eaa
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects ( #3123 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-21 16:46:06 +09:00
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom ( #3087 )
2021-06-11 12:15:52 +09:00
207c92753f
MAPREDUCE-7350. Replace Guava Lists usage by Hadoop's own Lists in hadoop-mapreduce-project ( #3074 )
2021-06-07 11:51:29 +09:00
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom ( #3049 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-26 17:14:31 +09:00
8a489ce78e
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. ( #3053 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-05-26 15:47:56 +09:00
2a206c20cc
HADOOP-17722. Replace Guava Sets usage by Hadoop's own Sets in hadoop-mapreduce-project ( #3032 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 10:07:22 +09:00
0d59500e8c
HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar.
2021-05-21 01:36:26 +00:00
5404ab4bca
MAPREDUCE-7343. Increase the job name max length in mapred job -list. ( #2995 ). Contributed by Ayush Saxena.
2021-05-14 00:15:33 +05:30
1448756505
MAPREDUCE-7334. TestJobEndNotifier fails. ( #2877 )
2021-04-08 23:53:51 -07:00
663ca14a76
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x ( #2775 )
2021-04-09 11:58:53 +09:00
46a5979805
MAPREDUCE-7270. TestHistoryViewerPrinter could be failed when the locale isn't English. ( #1942 )
...
Contributed by Sungpeo Kook.
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2021-04-07 23:58:10 -07:00
03e42efa30
YARN-10544. AMParams.java having un-necessary access identifier static final. Contributed by ANANDA G B.
2021-03-31 08:25:20 +05:30
ede490d131
MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein
2021-03-22 18:41:25 +00:00
299b8062f1
MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
2021-03-15 20:13:17 +00:00
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
a17dc4e821
[MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file.
...
Contributed by Eric Badger.
2021-03-02 21:47:31 +00:00
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
2021-02-26 13:42:33 -06:00
8bca105655
MAPREDUCE-7323. Remove job_history_summary.py. ( #2712 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-02-22 23:15:04 +09:00
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )"
...
This reverts commit 4cf3531583
2021-02-19 11:09:10 +09:00
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )
...
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
462561654b
MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log.
...
Contributed by Jim Brennan.
2021-02-09 19:01:38 +00:00
2a38ed0e0c
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. ( #2624 )
...
Contributed by Jungtaek Lim.
2021-01-27 19:08:08 +00:00
ca7dd5fad3
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest ( #2499 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-01-12 15:45:13 +09:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
9b2956e254
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. ( #2579 )
...
Part of the HADOOP-16830 IOStatistics API feature.
If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null.
Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.
Contributed by Steve Loughran.
2020-12-31 16:02:10 +00:00
3234e5eaf3
YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori
2020-12-12 09:42:22 +01:00
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:32:10 +09:00
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs ( #2488 )" because JIRA ID is missing.
...
This reverts commit ce5b3d7975
2020-11-26 11:30:53 +09:00
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:27:30 +09:00
f813f14b5b
MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan
2020-11-24 17:02:35 +01:00
f13c7b1b02
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. ( #2469 )
...
Contributed by Zhihua Deng.
2020-11-23 15:40:22 +00:00
740399a602
HADOOP-17390. Skip license check on lz4 code files ( #2478 )
...
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license
Contributed by Zhihua Deng.
2020-11-20 14:12:20 +00:00
fb92aa4012
MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann
2020-11-20 13:13:51 +01:00
34aa6137bd
HADOOP-17292. Using lz4-java in Lz4Codec ( #2350 )
...
Contributed by Liang-Chi Hsieh.
2020-11-18 12:03:25 -08:00
2045a9df13
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. ( #2463 )
2020-11-18 14:15:11 +09:00
da93cd962c
MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka.
2020-10-28 10:35:30 +01:00
7bc305db5d
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko.
2020-10-23 04:34:49 +09:00
42e7e0380f
HADOOP-17144. Update Hadoop's lz4 to v1.9.2. Contributed by Hemanth Boyina.
2020-10-18 18:37:46 +05:30
1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. ( #2342 ). Contributed by Ayush Saxena.
2020-10-17 12:01:18 +05:30
2e46ef9417
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-10-12 10:57:02 -07:00
c9ea344f98
HADOOP-17125. Use snappy-java in SnappyCodec ( #2297 )
...
This switches the SnappyCodec to use the java-snappy codec, rather than the native one.
To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.
This comesin as an avro dependency, so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7
Contributed by DB Tsai and Liang-Chi Hsieh
2020-10-06 17:07:54 +01:00
4347a5c955
HADOOP-17294. Fix typos existance to existence ( #2357 )
2020-10-06 10:10:44 +09:00
143bdd4188
MAPREDUCE-7289. Fix wrong comment in LongLong.java ( #2338 )
2020-09-29 23:06:25 +09:00
95dfc875d3
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache ( #2223 )
...
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-09-19 23:10:05 -07:00
4ffe26f9b8
[UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori
2020-08-31 16:00:39 +02:00
82a7505646
YARN-10304. Create an endpoint for remote application log directory path query. Contributed by Andras Gyori
2020-08-25 13:29:12 +02:00
dc5470ae86
MAPREDUCE-7288. Fix TestLongLong#testRightShift ( #2183 )
2020-08-06 09:35:15 +09:00
cf4eb75608
MAPREDUCE-7051. Fix typo in MultipleOutputFormat ( #338 )
2020-07-30 13:01:22 +09:00
1b29c9bfee
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 )
2020-07-22 13:40:20 +09:00
6bcb24d269
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-18 14:33:43 +05:30
98fcffe93f
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-07-15 09:53:18 -05:00
80046d1c8a
MAPREDUCE-7285. Junit class missing from hadoop-mapreduce-client-jobclient-*-tests jar. ( #2139 )
2020-07-15 15:24:34 +09:00
3e7b65c3e5
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails ( #2136 )
2020-07-15 13:46:20 +09:00
f91a8ad88b
HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet. ( #2085 )
2020-07-09 11:33:37 -07:00
7e73cad974
MAPREDUCE-7280. MiniMRYarnCluster has hard-coded timeout waiting to start history server, with no way to disable. ( #2065 )
2020-06-30 10:52:25 +09:00
8fd0fdf889
MAPREDUCE-7281. Fix NoClassDefFoundError on 'mapred minicluster'. ( #2077 )
2020-06-20 07:37:55 +09:00
e154084770
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:16:33 +05:30
fbb8775430
Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)."
...
This reverts commit e2322e1117
2020-06-08 20:35:27 +00:00
10db97df1c
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
...
Contributed by Tarun Parimi.
2020-05-28 13:23:01 +10:00
d4e36409d4
MAPREDUCE-6826. Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING. Contributed by Bilwa S T.
2020-05-19 11:06:36 +05:30
a3f945fb84
HADOOP-17035. fixed typos (timeout, interruped) ( #2007 )
...
Co-authored-by: Sungpeo Kook <elixir.kook@kakaocorp.com>
2020-05-12 10:50:04 -05:00
263c76b678
HADOOP-17011. Tolerate leading and trailing spaces in fs.defaultFS. Contributed by Ctest
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2020-04-30 14:15:28 -07:00
ab36429559
YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori
2020-04-28 18:53:28 +02:00
e2322e1117
MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles).
2020-04-27 19:10:00 +00:00
a1b0697d37
MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T
2020-04-18 19:42:20 +05:30
053e39e1e4
MAPREDUCE-7273. Fixed Job History server token renewal.
...
Contributed by Peter Bacsko via eyang
2020-04-16 20:17:21 -04:00
11d17417ce
MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein)
2020-04-13 18:20:07 +00:00
e2c87b7917
MAPREDUCE-7266. historyContext doesn't need to be a class attribute inside JobHistoryServer. Contributed by Siddharth Ahuja
2020-04-09 12:38:54 +02:00
ab7495dc7d
MAPREDUCE-7269. TestNetworkedJob fails ( #1940 )
2020-04-06 17:57:37 +09:00
f8e598f278
MAPREDUCE-7268. Fix TestMapreduceConfigFields ( #1935 )
2020-04-03 10:59:07 +09:00
c613296dc8
MAPREDUCE-7241. FileInputFormat listStatus with less memory footprint. Contributed by Zhihua Deng
2020-04-01 07:46:33 -05:00
8914cf9167
Preparing for 3.4.0 development
2020-03-29 23:24:25 +05:30
ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters ( #1397 )
2020-03-16 11:28:36 +09:00
8d6ff87c18
MAPREDUCE-7263. Remove obsolete validateTargetPath() from FrameworkUploader. Contributed by Marton Hudaky
2020-02-12 15:53:33 +01:00
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency ( #1635 ). Contributed by Vinayakumar B.
2020-02-07 14:51:24 +05:30
71b2c2ffe9
YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal
2020-02-06 12:25:06 +01:00
b897f6834b
MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein)
2020-01-29 16:54:45 +00:00
7f3e1e0c07
MAPREDUCE-7260. Cross origin request support for Job history server web UI. Contributed by Adam Antal
2020-01-29 14:42:52 +01:00
08251538fe
MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
2020-01-28 10:57:33 -06:00
PJ Fanning
Steve Loughran
Viraj Jasani
Stamatis Zampetakis
Dongjoon Hyun
Chao Sun
lzx404243
jenny
Eric Payne
Shubham Gupta
Jim Brennan
Akira Ajisaka
Masatake Iwasaki
Ayush Saxena
lichaojacobs
Sungpeo Kook
Brahma Reddy Battula
Eric Badger
Ahmed Hussein
Jungtaek Lim
He Xiaoqiao
adamantal
Szilard Nemeth
dengzh
Peter Bacsko
Liang-Chi Hsieh
hemanthboyina
Swaroopa Kadam
Ikko Ashimine
Wanqiang Ji
zz
Adam Antal
ywheel
Ahmed Hussein
Xiaoyu Yao
Vinayakumar B
Wilfred Spiegelenburg
Surendra Singh Lilhore
Mingliang Liu
Eric E Payne
Eric Yang
Wanqiang Ji
Jason Lowe