fa4915fdbb
Preparing for 3.3.2 development
2021-05-19 21:52:37 +08:00
4ffe5eb1dd
HADOOP-17669. Backport HADOOP-17079, HADOOP-17505 to branch-3.3 ( #2959 )
...
* HADOOP-17079. Optimize UGI#getGroups by adding UGI#getGroupsSet.
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
Change-Id: I0f31409923ece24a82dfba4c4610d8a38c52d9fb
* HADOOP-17505. public interface GroupMappingServiceProvider needs default impl for getGroupsSet() (#2661 ). Contributed by Vinayakumar B.
(cherry picked from commit c4c0683dff
2021-05-17 18:57:46 -07:00
930f384e30
MAPREDUCE-7302. Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail. Contributed by Peter Bacsko. Reviewed by Akira Ajisaka.
...
(cherry picked from commit da93cd962c
2021-04-23 18:42:50 +00:00
068f114066
MAPREDUCE-7329: HadoopPipes task may fail when linux kernel version change from 3.x to 4.x ( #2775 )
...
(cherry picked from commit 663ca14a76
2021-04-09 12:00:38 +09:00
e079aaa820
MAPREDUCE-7199. HsJobsBlock reuse JobACLsManager for checkAccess. Contributed by Bilwa S T
...
(cherry picked from commit a1b0697d37
2021-04-02 21:31:45 +05:30
c70f5eb8fa
MAPREDUCE-6826. Job fails with InvalidStateTransitonException: Invalid event: JOB_TASK_COMPLETED at SUCCEEDED/COMMITTING. Contributed by Bilwa S T.
...
(cherry picked from commit d4e36409d4
2021-03-31 21:35:06 +05:30
91d229bf35
MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein
...
(cherry picked from commit ede490d131
2021-03-22 18:44:41 +00:00
ad74038e02
MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
...
(cherry picked from commit 299b8062f1
2021-03-15 20:17:02 +00:00
de2904f123
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed1
2021-03-11 14:57:03 +09:00
138021ce23
[MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file.
...
Contributed by Eric Badger.
(cherry picked from commit a17dc4e821
2021-03-02 21:49:09 +00:00
792329fde9
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
...
(cherry picked from commit e04bcb3a06
2021-02-26 19:56:07 +00:00
3339ff5966
MAPREDUCE-7323. Remove job_history_summary.py. ( #2712 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655
2021-02-22 23:16:05 +09:00
37971c71d1
MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log.
...
Contributed by Jim Brennan.
(cherry picked from commit 462561654b
2021-02-09 19:03:20 +00:00
ebdacedc83
MAPREDUCE-7317. Add latency information in FileOutputCommitter.mergePaths. ( #2624 )
...
Contributed by Jungtaek Lim.
Change-Id: Iaff2f55e5378c22ce8a92ae776f5aba3f0fc304e
2021-01-27 19:08:54 +00:00
5be450393c
MAPREDUCE-7315. LocatedFileStatusFetcher to collect/publish IOStatistics. ( #2579 )
...
Part of the HADOOP-16830 IOStatistics API feature.
If the source FileSystem's listing RemoteIterators
implement IOStatisticsSource, these are collected and served through
the IOStatisticsSource API. If they are not: getIOStatistics() returns
null.
Only the listing statistics are collected; FileSystem.globStatus() doesn't
provide any, so IO use there is not included in the aggregate results.
Contributed by Steve Loughran.
Change-Id: Iff1485297c2c7e181b54eaf1d2c4f80faeee7cfa
2021-01-14 13:20:38 +00:00
b3f85576e9
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest ( #2499 )
...
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad3
2021-01-12 15:45:54 +09:00
e95ee67632
Make upstream aware of 3.2.2 release.
2021-01-09 18:07:10 +08:00
bc0c3ece0f
HADOOP-17390. Skip license check on lz4 code files ( #2478 )
...
The relevant files are `Copyright (C) 2011-present, Yann Collet.` and licensed on the BSD 2-clause license
Contributed by Zhihua Deng.
2020-12-29 13:32:26 -08:00
87064df1f2
HADOOP-17292. Using lz4-java in Lz4Codec ( #2350 )
...
Contributed by Liang-Chi Hsieh.
2020-12-29 13:17:26 -08:00
71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. ( #2155 ) ( #2538 )
...
(cherry picked from commit 1b29c9bfee
2020-12-11 13:58:02 +09:00
8378ab9f92
HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505
2020-12-10 05:50:55 +05:30
efd74cc8ae
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails ( #2136 )
...
(cherry picked from commit 3e7b65c3e5
2020-12-04 10:07:58 +09:00
db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287
2020-11-26 11:34:39 +09:00
8ed565382f
MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan.
2020-11-25 11:36:58 +01:00
abc87aef18
MAPREDUCE-7307. Potential thread leak in LocatedFileStatusFetcher. ( #2469 )
...
Contributed by Zhihua Deng.
Change-Id: Iee62539d02bd8f8a928171d8258e640487050a05
2020-11-23 16:33:41 +00:00
ced08fd87f
MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann
2020-11-20 13:14:49 +01:00
71c106b412
MAPREDUCE-7305. [JDK 11] TestMRJobsWithProfiler fails. ( #2463 )
...
(cherry picked from commit 2045a9df13
2020-11-18 14:16:45 +09:00
c40f0f1eb3
MAPREDUCE-7303. Fix TestJobResourceUploader failures after HADOOP-16878. Contributed by Peter Bacsko.
...
(cherry picked from commit 7bc305db5d
2020-10-23 04:41:37 +09:00
05a73ded93
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing
...
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-10-12 11:09:08 -07:00
8f60a90688
HADOOP-17125. Use snappy-java in SnappyCodec ( #2297 )
...
This switches the SnappyCodec to use the java-snappy codec, rather than the native one.
To use the codec, snappy-java.jar (from org.xerial.snappy) needs to be on the classpath.
This comesin as an avro dependency, so it is already on the hadoop-common classpath,
as well as in hadoop-common/lib.
The version used is now managed in the hadoop-project POM; initially 1.1.7.7
Contributed by DB Tsai and Liang-Chi Hsieh
Change-Id: Id52a404a0005480e68917cd17f0a27b7744aea4e
2020-10-06 17:15:17 +01:00
eb6134cd22
MAPREDUCE-7289. Fix wrong comment in LongLong.java ( #2338 )
...
(cherry picked from commit 143bdd4188
2020-09-29 23:07:58 +09:00
e5e91397de
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache ( #2223 )
...
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-09-19 23:26:37 -07:00
ffb96775e3
MAPREDUCE-7288. Fix TestLongLong#testRightShift ( #2183 )
...
(cherry picked from commit dc5470ae86
2020-08-06 09:36:34 +09:00
2efa28cb79
MAPREDUCE-7051. Fix typo in MultipleOutputFormat ( #338 )
...
(cherry picked from commit cf4eb75608
2020-07-30 13:28:35 +09:00
e3b8d4eb05
HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein.
2020-07-22 18:21:14 +05:30
5969922305
HADOOP-17101. Replace Guava Function with Java8+ Function
...
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f
2020-07-15 09:57:36 -05:00
56d72adbdd
MAPREDUCE-7281. Fix NoClassDefFoundError on 'mapred minicluster'. ( #2077 )
...
(cherry picked from commit 8fd0fdf889
2020-06-20 21:39:57 +09:00
534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes ( #2026 )
2020-06-12 23:20:10 +05:30
890617c7ac
Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)."
...
This reverts commit 741fcf2c63
2020-06-08 20:25:02 +00:00
586d9427e3
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
...
Contributed by Tarun Parimi.
(cherry picked from commit 10db97df1c
2020-05-28 13:36:21 +10:00
dfa7f160a5
Preparing for 3.3.1 development
2020-04-30 13:33:42 +09:00
e45faae1d4
YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori
...
(cherry picked from commit ab36429559
2020-04-28 18:56:29 +02:00
741fcf2c63
MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles).
...
(cherry picked from commit e2322e1117
2020-04-27 19:34:38 +00:00
b397a3a875
MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein)
...
(cherry picked from commit 11d17417ce
2020-04-13 18:51:00 +00:00
ef01e9d40e
MAPREDUCE-7266. historyContext doesn't need to be a class attribute inside JobHistoryServer. Contributed by Siddharth Ahuja
...
(cherry picked from commit e2c87b7917
2020-04-09 12:40:22 +02:00
cb806221e9
MAPREDUCE-7269. TestNetworkedJob fails ( #1940 )
...
(cherry picked from commit ab7495dc7d
2020-04-06 17:58:25 +09:00
215a62e99f
MAPREDUCE-7268. Fix TestMapreduceConfigFields ( #1935 )
...
(cherry picked from commit f8e598f278
2020-04-03 11:00:02 +09:00
ea688631b0
MAPREDUCE-7237. Supports config the shuffle's path cache related parameters ( #1397 )
2020-03-16 11:28:36 +09:00
8d6ff87c18
MAPREDUCE-7263. Remove obsolete validateTargetPath() from FrameworkUploader. Contributed by Marton Hudaky
2020-02-12 15:53:33 +01:00
7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency ( #1635 ). Contributed by Vinayakumar B.
2020-02-07 14:51:24 +05:30
Wei-Chiu Chuang
Eric Badger
lichaojacobs
Surendra Singh Lilhore
Jim Brennan
Akira Ajisaka
Ahmed Hussein
Jungtaek Lim
Steve Loughran
lzx404243
He Xiaoqiao
dengzh
Liang-Chi Hsieh
Ayush Saxena
Peter Bacsko
Swaroopa Kadam
Wanqiang Ji
zz
ywheel
Ahmed Hussein
Masatake Iwasaki
Vinayakumar B
Wilfred Spiegelenburg
Szilard Nemeth
Eric E Payne
Wanqiang Ji