812 Commits

Author SHA1 Message Date
Weiwei Yang
5db733fb09 MAPREDUCE-7059. Downward Compatibility issue: MR job fails because of unknown setErasureCodingPolicy method from 3.x client to HDFS 2.x cluster. Contributed by Jiandan Yang. 2018-03-02 10:43:03 +08:00
Steve Loughran
c960ae0b9a MAPREDUCE-7060. Cherry Pick PathOutputCommitter class/factory to branch-3.0.
Contributed by Steve Loughran.
2018-03-01 18:39:54 +00:00
Jason Lowe
bc2d4ba507 MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko
(cherry picked from commit 87e2570a1419d3616de2de3b553108ad1a8af425)
2018-02-12 14:49:35 -06:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Eric Payne
1d53c9d3a9 MAPREDUCE-7033: Map outputs implicitly rely on permissive umask for shuffle. Contributed by Jason Lowe
(cherry picked from commit 5a725bb886eb0b9182840c4a91466c361ecc11e0)
2018-01-31 16:01:46 -06:00
Jason Lowe
d7acc45ce4 MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko
(cherry picked from commit 6eef3d7f1a1e5e3f27fb3bf7596663640d786181)
2018-01-26 15:37:16 -06:00
Chris Douglas
03f0b8b3f2 MAPREDUCE-7011. TestClientDistributedCacheManager::testDetermineCacheVisibilities assumes all parent dirs set other exec
(cherry picked from commit 782ba3bf9da52699b27405a3f147464975d1df99)
2017-11-21 20:42:59 -08:00
Andrew Wang
1c8d334287 Preparing for 3.0.1 development 2017-11-14 11:17:18 -08:00
Naganarasimha
8e9369d387 MAPREDUCE-6975. Logging task counters. Contributed by Prabhu Joseph.
(cherry picked from commit 61bbdc511e4e98c11a05d7c3bc01f3e971adbe13)
2017-11-06 12:59:52 +05:30
Akira Ajisaka
12f92e636e
MAPREDUCE-6983. Moving logging APIs over to slf4j in hadoop-mapreduce-client-core. Contributed by Jinjiang Ling.
(cherry picked from commit 178751ed8c9d47038acf8616c226f1f52e884feb)
2017-11-02 17:43:55 +09:00
Daniel Templeton
7689d4781a YARN-6927. Add support for individual resource types requests in MapReduce
(Contributed by Gergo Repas via Daniel Templeton)

(cherry picked from commit 9a7e81083801a57d6bb96584988415cbef67460d)
2017-10-31 15:06:18 -07:00
Akira Ajisaka
a887016e1d
MAPREDUCE-6972. Enable try-with-resources for RecordReader. Contributed by Zoltan Haindrich.
(cherry picked from commit 2523e1cce5f6d6c3762c958c7f3138ab486c8497)
2017-10-18 11:19:00 +09:00
Chris Trezzo
24b03eb79f MAPREDUCE-5951. Add support for the YARN Shared Cache.
(cherry picked from commit e46d5bb962b0c942f993afc505b165b1cd96e51b)
2017-10-12 11:15:46 -07:00
Andrew Wang
5a4f37019a Preparing for 3.0.0 GA development 2017-09-28 11:32:01 -07:00
Jason Lowe
bf03363d83 MAPREDUCE-6968. Staging directory erasure coding config property has a typo. Contributed by Jason Lowe
(cherry picked from commit 9df05005ac258441eeceec25ecef7b28f9f50a02)
2017-09-26 14:53:47 -05:00
Robert Kanter
917e875e51 MAPREDUCE-6954. Disable erasure coding for files that are uploaded to the MR staging area (pbacsko via rkanter)
(cherry picked from commit 0adc0471d0c06f66a31060f270dcb50a7b4ffafa)
2017-09-18 10:42:01 -07:00
Robert Kanter
3847b8c861 Revert "YARN-7162. Remove XML excludes file format (rkanter)" - wrong commit message
This reverts commit 293ab432e116b93ada6ee56e518ae07aa853ab41.
2017-09-18 10:41:46 -07:00
Robert Kanter
293ab432e1 YARN-7162. Remove XML excludes file format (rkanter)
(cherry picked from commit 3a8d57a0a2e047b34be82f602a2b6cf5593d2125)
2017-09-15 12:01:41 -07:00
Steve Loughran
b5e9982355
MAPREDUCE-6956 FileOutputCommitter to gain abstract superclass PathOutputCommitter.
Contributed by Steve Loughran

(cherry picked from commit 11390c2d111910b01d9c4d3e39dee49babae272f)
2017-09-15 17:00:16 +01:00
Jason Lowe
12e9558ed1 MAPREDUCE-6957. shuffle hangs after a node manager connection timeout. Contributed by Jooseong Kim
(cherry picked from commit 4d98936eec1b5d196053426c70d455cf8f83f84f)
2017-09-13 17:22:29 -05:00
Ray Chiang
656c6d1acc HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)
(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)
2017-09-12 10:19:58 -07:00
Allen Wittenauer
190410085b
HADOOP-14364. refresh changelog/release notes with newer Apache Yetus build
Signed-off-by: Andrew Wang <wang@apache.org>
2017-08-31 19:06:49 -07:00
Haibo Chen
d04f85f387 MAPREDUCE-6892. Issues with the count of failed/killed tasks in the jhist file. (Peter Bacsko via Haibo Chen) 2017-08-30 10:07:48 -07:00
Varun Saxena
3d00c8f394 Made fixes for whitespace errors and checstyle warnings before merge. 2017-08-30 11:29:54 +05:30
Sangjin Lee
092fead5d9 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 2017-08-30 11:29:51 +05:30
Mingliang Liu
ae7abad3d9 MAPREDUCE-6945. TestMapFileOutputFormat missing @after annotation. Contributed by Ajay Kumar 2017-08-27 15:19:55 -07:00
Arpit Agarwal
8b7cbe3840 HADOOP-14729. Upgrade JUnit 3 test cases to JUnit 4. Contributed by Ajay Kumar. 2017-08-25 09:55:46 -07:00
Andrew Wang
dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Jason Lowe
0acc5e0036 MAPREDUCE-6940. Copy-paste error in the TaskAttemptUnsuccessfulCompletionEvent constructor. Contributed by Oleg Danilov 2017-08-16 16:34:06 -05:00
Haibo Chen
a32e0138fb MAPREDUCE-6870. Add configuration for MR job to finish when all reducers are complete. (Peter Bacsko via Haibo Chen) 2017-08-10 15:17:36 -07:00
Yufei Gu
b8e8241854 MAPREDUCE-6914. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 11:44:34 -07:00
Junping Du
fbb7d6bcbb Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)"
This reverts commit e8a31f2e1c34514fba2f480e8db652f6e2ed65d8.
2017-07-31 14:09:16 -07:00
Junping Du
713349a9af Revert "MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)"
This reverts commit 390a7c12f543b2c94a74f08d6d2a28410472043a.
2017-07-28 14:21:04 -07:00
Junping Du
e15f928295 Revert "MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu."
This reverts commit 433542904aba5ddebf9bd9d299378647351eb13a.
2017-07-28 14:06:59 -07:00
Sunil G
fb3b5d33ff MAPREDUCE-6889. Add Job#close API to shutdown MR client services. Contributed by Rohith Sharma K S. 2017-07-17 13:35:15 +05:30
Junping Du
f484a6ff60 MAPREDUCE-6246. DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2. Contributed by ramtin and Gergely Novák. 2017-07-07 13:26:16 -07:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Xiao Chen
4ebc23ba7b HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively. 2017-06-16 09:49:33 -07:00
Sean Busbey
f5517a8200
HADOOP-13921. Remove log4j classes from JobConf.
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2017-06-01 13:35:14 +09:00
Akira Ajisaka
d4015f8628
MAPREDUCE-6887. Modifier 'static' is redundant for inner enums. Contributed by ZhangBing Lin. 2017-05-30 14:48:58 +09:00
Andrew Wang
16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Sunil G
b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Jason Lowe
1e71fe8c42 MAPREDUCE-6882. Increase MapReduce test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 2017-05-10 09:57:41 -05:00
Chris Douglas
45382bf047 AuditLogger and TestAuditLogger are dead code. Contributed by Vrushali C 2017-05-09 10:31:52 -07:00
Akira Ajisaka
3ed3062fe3
MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. Contributed by Weiwei Yang. 2017-04-27 15:45:33 +09:00
Robert Kanter
3721cfe1fb MAPREDUCE-6871. Allow users to specify racks and nodes for strict locality for AMs (rkanter) 2017-04-21 16:12:01 -07:00
Daniel Templeton
fc0885da29 MAPREDUCE-6846. Fragments specified for libjar paths are not handled correctly
(Contributed by Chris Trezzo via Daniel Templeton)
2017-04-05 17:25:40 -07:00
Chris Douglas
6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Chris Douglas
7297e82cb4 HADOOP-14250. Correct spelling of 'separate' and variants. Contributed by Doris Gu 2017-03-30 10:17:11 -07:00
Zhe Zhang
de7efd2687 MAPREDUCE-6873. MR Job Submission Fails if MR framework application path not on defaultFS. Contributed by Erik Krogen. 2017-03-29 22:10:55 -07:00