fb4e8172a0
MAPREDUCE-7391. TestLocalDistributedCacheManager failing after HADOOP-16202 ( #4472 )
...
Fixing a mockito-based test which broke when HADOOP-16202
changed the methods being invoked.
Contributed by Steve Loughran
2022-06-22 13:13:24 +01:00
53a530aa88
MAPREDUCE-7371. DistributedCache alternative APIs should not use DistributedCache APIs internally ( #3855 )
...
Contributed by Viraj Jasani
2022-06-22 13:13:05 +01:00
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 ( #4482 )
...
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.
Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.
Contributed by Steve Loughran
2022-06-22 13:09:50 +01:00
aeb2a2f860
HADOOP-17833. Improve Magic Committer performance ( #3289 ) ( #4470 )
...
Speed up the magic committer with key changes being
* Writes under __magic always retain directory markers
* File creation under __magic skips all overwrite checks,
including the LIST call intended to stop files being
created over dirs.
* mkdirs under __magic probes the path for existence
but does not look any further.
Extra parallelism in task and job commit directory scanning
Use of createFile and openFile with parameters which all for
HEAD checks to be skipped.
The committer can write the summary _SUCCESS file to the path
`fs.s3a.committer.summary.report.directory`, which can be in a
different file system/bucket if desired, using the job id as
the filename.
Also: HADOOP-15460. S3A FS to add `fs.s3a.create.performance`
Application code can set the createFile() option
fs.s3a.create.performance to true to disable the same
safety checks when writing under magic directories.
Use with care.
The createFile option prefix `fs.s3a.create.header.`
can be used to add custom headers to S3 objects when
created.
Contributed by Steve Loughran.
2022-06-21 10:49:37 +01:00
4f860f8ac2
MAPREDUCE-7369. Fixed MapReduce tasks timing out when spends more time on MultipleOutputs#close ( #4247 )
...
Contributed by Ravuri Sushma sree.
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
(cherry picked from commit 36c4be819f
2022-06-20 08:02:58 +00:00
43f4a0e92d
MAPREDUCE-7387. Fix TestJHSSecurity#testDelegationToken AssertionError due to HDFS-16563 ( #4428 ). Contributed by fanshilun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-20 12:16:33 +05:30
fe306ce57e
HADOOP-18198. Release 3.3.3: release notes and jdiff files.
...
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version
Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
2022-05-17 19:00:09 +01:00
e123de9f19
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2)
...
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.
As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.
This commit depends on the associated hadoop-common patch,
which must be committed first.
Contributed by Steve Loughran.
Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
2022-04-27 19:23:25 +01:00
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-27 10:30:07 +01:00
f4290055c6
MAPREDUCE-7246. In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, updating the datatype of appId to "string". ( #4223 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fb13c1e4a8
2022-04-25 14:31:15 +09:00
44e662272f
HADOOP-18198. Preparing for 3.3.4 development
...
Change-Id: I2bf19beb541739af22fced38c2545f09c4e1bd53
2022-04-12 14:09:08 +01:00
160b6d106d
HADOOP-18088. Replace log4j 1.x with reload4j. ( #4052 )
...
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-04-07 08:33:13 +09:00
85843f2158
MAPREDUCE-7373. Building MapReduce NativeTask fails on Fedora 34+ ( #4120 )
...
(cherry picked from commit dc4a680da8
2022-03-30 13:49:45 +00:00
1cc83f0f45
MAPREDUCE-7341. Add an intermediate manifest committer for Azure and GCS
...
This is a mapreduce/spark output committer optimized for
performance and correctness on Azure ADLS Gen 2 storage
(via the abfs connector) and Google Cloud Storage
(via the external gcs connector library).
* It is safe to use with HDFS, however it has not been optimized
for that use.
* It is *not* safe for use with S3, and will fail if an attempt
is made to do so.
Contributed by Steve Loughran
Change-Id: I6f3502e79c578b9fd1a8c1485f826784b5421fca
2022-03-17 11:46:41 +00:00
b0c1158829
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3764 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-13 13:52:44 +09:00
e079fa6577
Preparing for 3.3.3 development
2021-11-16 16:02:34 -08:00
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6
2021-10-07 11:00:19 +09:00
9fd0832a99
Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko."
...
This reverts commit c40f0f1eb3
2021-09-23 15:04:26 -07:00
d2c02f5afc
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter ( #2500 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
(cherry picked from commit 6187f76f11
2021-09-01 17:15:31 +09:00
4a93ca78f9
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() ( #2968 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7b5be74228
2021-08-30 08:41:46 +09:00
3645a13586
HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar.
...
(cherry picked from commit 0d59500e8c
2021-08-25 01:54:36 +00:00
b8a8821735
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup ( #1808 )
...
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e31169c864
2021-08-02 15:39:38 +09:00
e395711164
MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST)
...
(cherry picked from commit 7581413156
2021-07-07 20:57:32 +00:00
75f8198aa8
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
...
(cherry picked from commit 7c7d02edbd
2021-06-25 20:36:41 +00:00
4825c7c28d
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects ( #3123 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa
2021-06-21 16:47:37 +09:00
e14d00469a
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. ( #3053 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 8a489ce78e
2021-05-26 15:48:51 +09:00
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )"
...
This reverts commit 4ffe5eb1dd
2021-05-24 17:37:18 +08:00
fa4915fdbb
Preparing for 3.3.2 development
2021-05-19 21:52:37 +08:00
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )
...
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb
* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661 ). Contributed by Vinayakumar B.
(cherry picked from commit c4c0683dff
2021-05-17 18:57:46 -07:00
930f384e30
MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka.
...
(cherry picked from commit da93cd962c
2021-04-23 18:42:50 +00:00
068f114066
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x ( #2775 )
...
(cherry picked from commit 663ca14a76
2021-04-09 12:00:38 +09:00
e079aaa820
MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T
...
(cherry picked from commit a1b0697d37
2021-04-02 21:31:45 +05:30
c70f5eb8fa
MAPREDUCE-6826. Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING. Contributed by Bilwa S T.
...
(cherry picked from commit d4e36409d4
2021-03-31 21:35:06 +05:30
91d229bf35
MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein
...
(cherry picked from commit ede490d131
2021-03-22 18:44:41 +00:00
ad74038e02
MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
...
(cherry picked from commit 299b8062f1
2021-03-15 20:17:02 +00:00
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1
2021-03-11 14:57:03 +09:00
138021ce23
[MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file.
...
Contributed by Eric Badger.
(cherry picked from commit a17dc4e821
2021-03-02 21:49:09 +00:00
792329fde9
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
...
(cherry picked from commit e04bcb3a06
2021-02-26 19:56:07 +00:00
3339ff5966
MAPREDUCE-7323. Remove job_history_summary.py. ( #2712 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655
2021-02-22 23:16:05 +09:00
37971c71d1
MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log.
...
Contributed by Jim Brennan.
(cherry picked from commit 462561654b
2021-02-09 19:03:20 +00:00
ebdacedc83
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. ( #2624 )
...
Contributed by Jungtaek Lim.
Change-Id: Iaff2f55e5378c22ce8a92ae776f5aba3f0fc304e
2021-01-27 19:08:54 +00:00
5be450393c
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. ( #2579 )
...
Part of the HADOOP-16830 IOStatistics API feature.
If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null.
Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.
Contributed by Steve Loughran.
Change-Id: Iff1485297c2c7e181b54eaf1d2c4f80faeee7cfa
2021-01-14 13:20:38 +00:00
b3f85576e9
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest ( #2499 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad3
2021-01-12 15:45:54 +09:00
e95ee67632
Make upstream aware of 3.2.2 release.
2021-01-09 18:07:10 +08:00
bc0c3ece0f
HADOOP-17390. Skip license check on lz4 code files ( #2478 )
...
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license
Contributed by Zhihua Deng.
2020-12-29 13:32:26 -08:00
87064df1f2
HADOOP-17292. Using lz4-java in Lz4Codec ( #2350 )
...
Contributed by Liang-Chi Hsieh.
2020-12-29 13:17:26 -08:00
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 ) ( #2538 )
...
(cherry picked from commit 1b29c9bfee
2020-12-11 13:58:02 +09:00
8378ab9f92
HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505
2020-12-10 05:50:55 +05:30
efd74cc8ae
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails ( #2136 )
...
(cherry picked from commit 3e7b65c3e5
2020-12-04 10:07:58 +09:00
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287
2020-11-26 11:34:39 +09:00
Steve Loughran
Viraj Jasani
Ashutosh Gupta
slfan1989
Masatake Iwasaki
Kengo Seki
Chao Sun
lzx404243
jenny
Eric Payne
Jim Brennan
Akira Ajisaka
Wei-Chiu Chuang
Eric Badger
lichaojacobs
Surendra Singh Lilhore
Ahmed Hussein
Jungtaek Lim
He Xiaoqiao
dengzh
Liang-Chi Hsieh
Ayush Saxena