2795 Commits

Author SHA1 Message Date
Jim Brennan
c1b25422a5 MAPREDUCE-7325. Intermediate data encryption is broken in LocalJobRunner. Contributed by Ahmed Hussein
(cherry picked from commit ede490d13182f0fd3831afe6db97e01917add5b4)
2021-03-22 19:28:23 +00:00
Jim Brennan
a92300e0a9 MAPREDUCE-7322. revisiting TestMRIntermediateDataEncryption. Contributed by Ahmed Hussein.
(cherry picked from commit 38f86cc8c72201b081a083cd32bc3a322c6d9398)
2021-03-16 19:19:01 +00:00
Eric Badger
14dea40154 [MAPREDUCE-7234] ClientHSSecurityInfo class is in wrong META-INF file.
Contributed by Eric Badger.

(cherry picked from commit a17dc4e8218b77e7fb5b5e253cb2d20014073b49)
2021-03-02 21:50:12 +00:00
Jim Brennan
4d323cd47c MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 21:39:04 +00:00
Akira Ajisaka
4cbab66547
MAPREDUCE-7323. Remove job_history_summary.py. (#2712)
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 8bca105655d91d949f8aa8daae656f34ed950e78)
2021-02-22 23:16:53 +09:00
Eric Badger
9c620c6302 MAPREDUCE-7319. Log list of mappers at trace level in ShuffleHandler audit log.
Contributed by Jim Brennan.

(cherry picked from commit 462561654bb37dc1d453bfe15c3e75c274bbfb02)
2021-02-09 19:23:23 +00:00
lzx404243
286009b869
MAPREDUCE-7310. Clear the fileMap in JHEventHandlerForSigtermTest (#2499)
Co-authored-by: Zhengxi Li <zli89@illinois.edu>
Reviewed-by: Wanqiang Ji <jiwq@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ca7dd5fad331e3942547b88fdc4c0ad59865a71a)
2021-01-12 16:08:15 +09:00
Akira Ajisaka
e17b7efd3b
MAPREDUCE-7284. TestCombineFileInputFormat#testMissingBlocks fails (#2136)
(cherry picked from commit 3e7b65c3e5920ed78bd87b68de1075003d86e7a0)
2020-12-04 10:43:32 +09:00
Peter Bacsko
7d60217754 MAPREDUCE-7309. Improve performance of reading resource request for mapper/reducers from config. Contributed by Peter Bacsko & Wangda Tan. 2020-11-25 11:46:00 +01:00
Peter Bacsko
6062978768 MAPREDUCE-7304. Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom class. Contributed by Zoltan Erdmann 2020-11-20 13:16:48 +01:00
Swaroopa Kadam
0d5534430c
MAPREDUCE-7301: Expose Mini MR Cluster attribute for testing
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-10-12 11:10:23 -07:00
Wanqiang Ji
cbc946285e
MAPREDUCE-7289. Fix wrong comment in LongLong.java (#2338)
(cherry picked from commit 143bdd41885a43c7bc6e11f0c0d6e5820768aa2c)
2020-09-29 23:08:50 +09:00
zz
06ff4d1416
MAPREDUCE-7294. Only application master should upload resource to Yarn Shared Cache (#2223)
Contributed by Zhenzhao Wang <zhenzhaowang@gmail.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-09-20 00:07:07 -07:00
Kihwal Lee
0c35e44184 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan
(cherry picked from commit 4571351cccf6d4977469d3d623cf045b06a5f5f0)
2020-08-13 19:02:03 -05:00
Wanqiang Ji
f3ac1dde5e
MAPREDUCE-7288. Fix TestLongLong#testRightShift (#2183)
(cherry picked from commit dc5470ae86b96d6f84abd7354c3c056d3627871b)
2020-08-06 09:37:28 +09:00
ywheel
2dfc6bf471
MAPREDUCE-7051. Fix typo in MultipleOutputFormat (#338)
(cherry picked from commit cf4eb75608527a8a56ad9d13a3009ccc38b29c8e)
2020-07-30 13:31:18 +09:00
Ayush Saxena
4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Ahmed Hussein
34a6cbb5bb HADOOP-17101. Replace Guava Function with Java8+ Function
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
(cherry picked from commit 43a865dc07437e0f767a902c4cfe6ca0472bf57f)
2020-07-15 11:01:34 -05:00
Eric Badger
0799901d48 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)."
This reverts commit 7e05577c4755802e24d14d80282edfdbe14304ab.
2020-06-08 20:12:00 +00:00
Wilfred Spiegelenburg
0e0d88ca97
MAPREDUCE-7278. Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are false
Contributed by Tarun Parimi.

(cherry picked from commit 10db97df1c8562a9e29b00e60d5bde1773c09188)
2020-05-28 16:38:34 +10:00
Gabor Bota
ec6d2a8402 Preparing for 3.1.5 development
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
2020-04-29 11:18:18 +02:00
Eric E Payne
7e05577c47 MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles).
(cherry picked from commit e2322e1117a2a3435aabecd49de0ad3c8d2f64b9)
2020-04-27 20:12:27 +00:00
Eric E Payne
8bcd7cfc0b MAPREDUCE-7272. TaskAttemptListenerImpl excessive log messages. Contributed by Ahmed Hussein (ahussein)
(cherry picked from commit 11d17417ceba0f1a2944e0c8b4286515c4883889)
2020-04-13 19:22:56 +00:00
Eric E Payne
d4ecb65629 MAPREDUCE-7079: JobHistory#ServiceStop implementation is incorrect. Contributed by Ahmed Hussein (ahussein)
(cherry picked from commit b897f6834ba69d443c3acd1fab52261c00d675a1)
2020-01-29 17:21:40 +00:00
Ahmed Hussein
f3ff61865a MAPREDUCE-7259. testSpeculateSuccessfulWithUpdateEvents fails Intermittently
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 08251538fe2550d9dd86f9daf79994f5b8bdf7fa)
2020-01-28 11:04:25 -06:00
Ahmed Hussein
1f315c28d9 MAPREDUCE-7262. MRApp helpers block for long intervals (500ms)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 3f01c481060585ccd37be9db8aa4d1e33d2e2d6b)
2020-01-27 16:23:06 -06:00
kevinzhao1661
ad34d7032e
MAPREDUCE-7247. Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int (#1684)
(cherry picked from commit b8518e6dabc1cf73fa7c43b3f8904d2e2beb2bf3)
2020-01-16 12:58:30 +09:00
Masatake Iwasaki
ce52a66fcb
MAPREDUCE-7256. Fix javadoc error in SimpleExponentialSmoothing. (#1804)
(cherry picked from commit a2fdd7c2b59463c39a4d4250f908a9e2d3653ead)
2020-01-15 12:12:27 +09:00
Eric E Payne
ec40c1f400 YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla)
(cherry picked from commit 93233a7d6e4d6b8098622a1aa830355cc18d9589)
2020-01-09 17:41:10 +00:00
Ahmed Hussein
6ad2f662eb MAPREDUCE-7252. Handling 0 progress in SimpleExponential task runtime estimator
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit cdd6efd3ab6917e30b4c5c7b261f61838901bb37)
2020-01-08 11:14:40 -06:00
Sergey Pogorelov
d55d02d25a
MAPREDUCE-7255. Fix typo in MapReduce documentaion example (#1793)
(cherry picked from commit b343e1533b1f4ebbbf10b063c35d79d53a741321)
2020-01-06 12:41:03 +09:00
prabhujoseph
a97f7776bd MAPREDUCE-7249. Fix Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure.
Contributed by Wilfred Spiegelenburg.
2019-11-28 14:34:52 +05:30
prabhujoseph
6a4966f336 MAPREDUCE-7240. Fix Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER.
Contributed by Huachao and Peter Bacsko. Reviewed by Wilfred Spiegelenburg.
2019-11-27 16:51:47 +05:30
Ahmed Hussein
cbd501636c MAPREDUCE-7208. Tuning TaskRuntimeEstimator. (Ahmed Hussein via jeagles)
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit ed302f1fed6d124d682486d24dae958946dba9be)
2019-11-05 15:29:19 -06:00
Jonathan Hung
4d274f60bc Make upstream aware of 2.10.0 release
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aac608746946dc2ed5711549a01a2055)
2019-10-30 21:00:55 -07:00
Szilard Nemeth
19755b9b36 MAPREDUCE-6441. Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes (wattsinabox, rchiang, haibochen via snemeth) 2019-10-18 15:25:50 +02:00
Jonathan Hung
0dc85bc182 YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:18:52 -07:00
Tao Yang
3cc21350b5 YARN-9817. Fix failing testcases due to not initialized AsyncDispatcher - ArithmeticException: / by zero. Contributed by Prabhu Joseph. 2019-09-07 07:40:50 +08:00
Zhankun Tang
ef79d98788 Preparing for 3.1.4 development 2019-09-04 16:11:36 +08:00
Szilard Nemeth
a3fb032010 MAPREDUCE-7230. TestHSWebApp.testLogsViewSingle fails. Contributed by Prabhu Joseph
(cherry picked from commit 3f4f097f1f29249c8ad5d59ccefcddf0a66b2e6b)
2019-08-15 17:08:53 +02:00
Szilard Nemeth
639763c561 MAPREDUCE-7225: Fix broken current folder expansion during MR job start. Contributed by Peter Bacsko. 2019-08-01 16:08:14 +02:00
pingsutw
1a2aba85d8
MAPREDUCE-7076. TestNNBench#testNNBenchCreateReadAndDelete failing in our internal build
This closes #1089

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ee87e9a42e4ff1f27a6f1e5b7c7de97f8989d9b2)
2019-07-23 10:36:01 +09:00
Sean Mackrory
fee1e67453 HADOOP-16213. Update guava to 27.0-jre. Contributed by Gabor Bota. 2019-06-13 07:38:43 -06:00
Takanobu Asanuma
8098ddaf40 HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:39:49 +09:00
Akira Ajisaka
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:21 +09:00
Wanqiang Ji
6690955358
MAPREDUCE-7205. Treat container scheduler kill exit code as a task attempt killing event.
This closes #821

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 67f9a7b165edecbec7c8063758202be4d8cff0f5)
2019-05-22 14:41:20 +09:00
Adam Antal
52536ff887 MAPREDUCE-7190. Add SleepJob additional parameter to make parallel runs distinguishable. Contributed by Adam Antal.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 856cbf62d3fd1c885ae8374f08d7c18c83941791)
(cherry picked from commit ddf88596550be13889ff7002aadd0ebf797996a1)
2019-04-01 10:39:18 -07:00
Weiwei Yang
fdd56b5b05 MAPREDUCE-7191. JobHistoryServer should log exception when loading/parsing history file failed. Contributed by Jiandan Yang.
(cherry picked from commit f0605146b34ddbd19fc9e52d4761f9a347d5173e)
2019-03-11 16:40:01 +08:00
Wangda Tan
a1e09b4c0c Make upstream aware of 3.1.2 release
Change-Id: I397bc6ef75498726df4763bd07a8bf8fe1c38365
(cherry picked from commit 308f3168fa221a336ef0dbb1a17358b1ad6f9356)
(cherry picked from commit 649da5af04d57ee910dbe7fb0f4a7d9cb4c40eb0)
2019-02-05 14:09:33 -08:00
Akira Ajisaka
72b3baf207
MAPREDUCE-7177. Disable speculative execution in TestDFSIO. Contributed by Zhaohui Xin.
(cherry picked from commit c991e2ca5a65eae62bad7e5baf81f1eb633b7ec0)
2019-02-04 14:04:26 +09:00