e079fa6577
Preparing for 3.3.3 development
2021-11-16 16:02:34 -08:00
77ee5a4266
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8071dbb9c6
2021-10-07 11:00:19 +09:00
9fd0832a99
Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko."
...
This reverts commit c40f0f1eb3
2021-09-23 15:04:26 -07:00
d2c02f5afc
MAPREDUCE-7311. Clear filesystem statistics after tests in TestTaskProgressReporter ( #2500 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
(cherry picked from commit 6187f76f11
2021-09-01 17:15:31 +09:00
4a93ca78f9
MAPREDUCE-7342. Stop RMService in TestClientRedirect.testRedirect() ( #2968 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7b5be74228
2021-08-30 08:41:46 +09:00
3645a13586
HADOOP-14922. Build of Mapreduce Native Task module fails with unknown opcode "bswap". Contributed by Anup Halarnkar.
...
(cherry picked from commit 0d59500e8c
2021-08-25 01:54:36 +00:00
b8a8821735
MAPREDUCE-7258. HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from taskcounterGroup to taskCounterGroup ( #1808 )
...
Co-authored-by: chenjuanni <chenjuanni@inspur.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e31169c864
2021-08-02 15:39:38 +09:00
e395711164
MAPREDUCE-7353: Mapreduce job fails when NM is stopped. Contributed by Bilwa S T (BilwaST)
...
(cherry picked from commit 7581413156
2021-07-07 20:57:32 +00:00
75f8198aa8
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
...
(cherry picked from commit 7c7d02edbd
2021-06-25 20:36:41 +00:00
4825c7c28d
MAPREDUCE-7354. Use empty array constant present in TaskCompletionEvent to avoid creating redundant objects ( #3123 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11461eaa
2021-06-21 16:47:37 +09:00
e14d00469a
MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. ( #3053 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 8a489ce78e
2021-05-26 15:48:51 +09:00
86c28f0639
Revert "HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )"
...
This reverts commit 4ffe5eb1dd
2021-05-24 17:37:18 +08:00
fa4915fdbb
Preparing for 3.3.2 development
2021-05-19 21:52:37 +08:00
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )
...
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb
* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661 ). Contributed by Vinayakumar B.
(cherry picked from commit c4c0683dff
2021-05-17 18:57:46 -07:00
930f384e30
MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka.
...
(cherry picked from commit da93cd962c
2021-04-23 18:42:50 +00:00
068f114066
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x ( #2775 )
...
(cherry picked from commit 663ca14a76
2021-04-09 12:00:38 +09:00
e079aaa820
MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T
...
(cherry picked from commit a1b0697d37
2021-04-02 21:31:45 +05:30
c70f5eb8fa
MAPREDUCE-6826. Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING. Contributed by Bilwa S T.
...
(cherry picked from commit d4e36409d4
2021-03-31 21:35:06 +05:30
91d229bf35
MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein
...
(cherry picked from commit ede490d131
2021-03-22 18:44:41 +00:00
ad74038e02
MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
...
(cherry picked from commit 299b8062f1
2021-03-15 20:17:02 +00:00
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1
2021-03-11 14:57:03 +09:00
138021ce23
[MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file.
...
Contributed by Eric Badger.
(cherry picked from commit a17dc4e821
2021-03-02 21:49:09 +00:00
792329fde9
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
...
(cherry picked from commit e04bcb3a06
2021-02-26 19:56:07 +00:00
3339ff5966
MAPREDUCE-7323. Remove job_history_summary.py. ( #2712 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655
2021-02-22 23:16:05 +09:00
37971c71d1
MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log.
...
Contributed by Jim Brennan.
(cherry picked from commit 462561654b
2021-02-09 19:03:20 +00:00
ebdacedc83
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. ( #2624 )
...
Contributed by Jungtaek Lim.
Change-Id: Iaff2f55e5378c22ce8a92ae776f5aba3f0fc304e
2021-01-27 19:08:54 +00:00
5be450393c
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. ( #2579 )
...
Part of the HADOOP-16830 IOStatistics API feature.
If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null.
Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.
Contributed by Steve Loughran.
Change-Id: Iff1485297c2c7e181b54eaf1d2c4f80faeee7cfa
2021-01-14 13:20:38 +00:00
b3f85576e9
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest ( #2499 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad3
2021-01-12 15:45:54 +09:00
e95ee67632
Make upstream aware of 3.2.2 release.
2021-01-09 18:07:10 +08:00
bc0c3ece0f
HADOOP-17390. Skip license check on lz4 code files ( #2478 )
...
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license
Contributed by Zhihua Deng.
2020-12-29 13:32:26 -08:00
87064df1f2
HADOOP-17292. Using lz4-java in Lz4Codec ( #2350 )
...
Contributed by Liang-Chi Hsieh.
2020-12-29 13:17:26 -08:00
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 ) ( #2538 )
...
(cherry picked from commit 1b29c9bfee
2020-12-11 13:58:02 +09:00
8378ab9f92
HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505
2020-12-10 05:50:55 +05:30
efd74cc8ae
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails ( #2136 )
...
(cherry picked from commit 3e7b65c3e5
2020-12-04 10:07:58 +09:00
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287
2020-11-26 11:34:39 +09:00
8ed565382f
MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan.
2020-11-25 11:36:58 +01:00
abc87aef18
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. ( #2469 )
...
Contributed by Zhihua Deng.
Change-Id: Iee62539d02bd8f8a928171d8258e640487050a05
2020-11-23 16:33:41 +00:00
ced08fd87f
MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann
2020-11-20 13:14:49 +01:00
71c106b412
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. ( #2463 )
...
(cherry picked from commit 2045a9df13
2020-11-18 14:16:45 +09:00
c40f0f1eb3
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko.
...
(cherry picked from commit 7bc305db5d
2020-10-23 04:41:37 +09:00
05a73ded93
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-10-12 11:09:08 -07:00
8f60a90688
HADOOP-17125. Use snappy-java in SnappyCodec ( #2297 )
...
This switches the SnappyCodec to use the java-snappy codec, rather than the native one.
To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.
This comesin as an avro dependency, so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7
Contributed by DB Tsai and Liang-Chi Hsieh
Change-Id: Id52a404a0005480e68917cd17f0a27b7744aea4e
2020-10-06 17:15:17 +01:00
eb6134cd22
MAPREDUCE-7289. Fix wrong comment in LongLong.java ( #2338 )
...
(cherry picked from commit 143bdd4188
2020-09-29 23:07:58 +09:00
e5e91397de
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache ( #2223 )
...
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-09-19 23:26:37 -07:00
ffb96775e3
MAPREDUCE-7288. Fix TestLongLong#testRightShift ( #2183 )
...
(cherry picked from commit dc5470ae86
2020-08-06 09:36:34 +09:00
2efa28cb79
MAPREDUCE-7051. Fix typo in MultipleOutputFormat ( #338 )
...
(cherry picked from commit cf4eb75608
2020-07-30 13:28:35 +09:00
e3b8d4eb05
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-22 18:21:14 +05:30
5969922305
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f
2020-07-15 09:57:36 -05:00
56d72adbdd
MAPREDUCE-7281. Fix NoClassDefFoundError on 'mapred minicluster'. ( #2077 )
...
(cherry picked from commit 8fd0fdf889
2020-06-20 21:39:57 +09:00
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:20:10 +05:30
Chao Sun
Viraj Jasani
lzx404243
Masatake Iwasaki
jenny
Eric Payne
Jim Brennan
Akira Ajisaka
Wei-Chiu Chuang
Eric Badger
lichaojacobs
Surendra Singh Lilhore
Ahmed Hussein
Jungtaek Lim
Steve Loughran
He Xiaoqiao
dengzh
Liang-Chi Hsieh
Ayush Saxena
Peter Bacsko
Swaroopa Kadam
Wanqiang Ji
zz
ywheel
Ahmed Hussein
Vinayakumar B