Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,262 Commits 326 Branches 371 Tags
Commit Graph

2360 Commits

Author SHA1 Message Date
Jason Lowe
e3a1647477 MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko 2018-01-26 15:46:01 -06:00
Haibo Chen
5f44bd3eba MAPREDUCE-7041. MR should not try to clean up at first job attempt. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cc10852252c2d69294eabc68bd032cc630a53b18)
(cherry picked from commit e6c66baba97c34514e1d7491d5fe80867930f184)
 2018-01-25 17:23:08 -08:00
Jason Lowe
1ec82e5789 MAPREDUCE-7015. Possible race condition in JHS if the job is not loaded. Contributed by Peter Bacsko 
(cherry picked from commit cff9edd4b514bdcfe22cd49964e3707fb78ab876)
 2018-01-24 14:51:22 -06:00
Haibo Chen
a286acd7be MAPREDUCE-6984. MR AM to clean up temporary files from previous attempt in case of no recovery. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cce71dceef9e82d31fe8ec59648b2a4a50c8869a)
(cherry picked from commit fc669778bec543427ea94abe12aff34ee1533b5c)
 2018-01-19 13:12:26 -08:00
Jason Lowe
c228a7c707 MAPREDUCE-7029. FileOutputCommitter is slow on filesystems lacking recursive delete. Contributed by Karthik Palaniappan 2018-01-17 10:32:06 -06:00
Jason Lowe
f2a1d5f3a1 MAPREDUCE-7028. Concurrent task progress updates causing NPE in Application Master. Contributed by Gergo Repas 
(cherry picked from commit fe35103591ece0209f8345aba5544313e45a073c)
 2018-01-03 11:08:51 -06:00
Jason Lowe
d0fc1cd0c8 MAPREDUCE-5124. AM lacks flow control for task events. Contributed by Peter Bacsko 2017-12-01 14:15:25 -06:00
Arun Suresh
7af9b8ad1e Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh) 2017-11-09 15:10:21 -08:00
Naganarasimha
45e6d36e5c MAPREDUCE-6975. Logging task counters. Contributed by Prabhu Joseph. 
(cherry picked from commit 61bbdc511e4e98c11a05d7c3bc01f3e971adbe13)
 2017-11-06 13:10:41 +05:30
Arun Suresh
71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Jason Lowe
347997c959 HADOOP-14919. BZip2 drops records when reading data in splits. Contributed by Jason Lowe 
(cherry picked from commit 2fae63aa60c43b62bd908a9499562fe528603185)
 2017-10-31 09:32:45 -05:00
Jason Lowe
9093ad6168 YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 
(cherry picked from commit 665bb147aa3fc198e53335931562dd69f9f0e345)
 2017-10-27 17:09:29 -05:00
Junping Du
1f2a2ba2be Add several jdiff xml files for 2.8.2 release. 
(cherry picked from commit a25b5aa0cf5189247fa38e7b0a188d568eba1b6c)
 2017-10-26 23:20:24 -07:00
Varun Saxena
56e0873e22 Fix javac, javadoc and checkstyle issues before branch-2 merge. 2017-10-17 23:36:41 +05:30
Jian He
cbd5e89b24 MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 
(cherry picked from commit c6ce422a3423305ac665f80f039f61294adaa32c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2017-10-17 23:36:40 +05:30
Rohith Sharma K S
5a7c272e0a YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-10-17 23:36:39 +05:30
Sangjin Lee
bb100d35fd YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R. 
(cherry picked from commit 73235ab30361b41293846189f3c5fef321ae7cac)
 2017-10-17 23:36:36 +05:30
Sangjin Lee
c8a8e680bd MAPREDUCE-6818. Remove direct reference to TimelineClientImpl. Contributed by Li Lu. 
(cherry picked from commit 6217b87f4a056cf704cef2e073b386b7803415de)
 2017-10-17 23:36:35 +05:30
Sangjin Lee
b8cfb4fcb3 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 
(cherry picked from commit f734977b27a514ce0561638c0a6a17b1ef093026)
 2017-10-17 23:36:35 +05:30
Varun Saxena
0820df611d MAPREDUCE-6731. TestMRTimelineEventHandling#testMRNewTimelineServiceEventHandling fails randomly for concurrent tests. (Sangjin Lee via Varun Saxena). 
(cherry picked from commit d14e729a55c0ca4053c537290392b74f732ce939)
 2017-10-17 23:36:33 +05:30
Sangjin Lee
e212e562ec YARN-5355. Backported YARN-2928 into our branch-2 feature branch. 2017-10-17 23:36:33 +05:30
Chris Trezzo
3e7c06af41 MAPREDUCE-5951. Add support for the YARN Shared Cache. 
(cherry picked from commit e46d5bb962b0c942f993afc505b165b1cd96e51b)
 2017-10-12 11:30:38 -07:00
Rohith Sharma K S
c2b8d4b9ab MAPREDUCE-6951. Improve exception message when mapreduce.jobhistory.webapp.address is in wrong format. Contributed by Prabhu Joseph. 
(cherry picked from commit 639f98cc9dc9fca034d0d7c26f30d5c588b9d7b8)
 2017-10-11 14:38:00 +05:30
Jason Lowe
b94acb6efb MAPREDUCE-6965. QuasiMonteCarlo should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit 908d8e914ef55aca69cb8e725e62bda29d1073cc)
 2017-09-22 09:37:09 -05:00
Akira Ajisaka
d80e2f2eae
MAPREDUCE-6966. DistSum should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari. 
(cherry picked from commit c71d13796039815350d3c73c51f9fe48531af0fa)
 2017-09-22 15:10:19 +09:00
Akira Ajisaka
9d4de9c556
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergely Novák. 
(cherry picked from commit 53be075241f1ba92bfe47e89c2dfc3f0664e2578)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
 2017-09-22 13:29:17 +09:00
Jason Lowe
cf2615961a MAPREDUCE-6964. BaileyBorweinPlouffe should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b9db0afa1e65d0343ad4e5760c36e042a3e704b0)
 2017-09-21 09:39:46 -05:00
Akira Ajisaka
7d747df52d
Revert "MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák." 
This reverts commit e61baf94965e25b6a7ab2264523142d066b490b2.
 2017-09-21 11:16:48 +09:00
Eric Payne
d2386725ff MAPREDUCE-6960. Shuffle Handler prints disk error stack traces for every read failure. 
(cherry picked from commit 595d478408104bdfe1f08efd79930e18862fafbb)
 2017-09-19 10:46:05 -05:00
Jason Lowe
41c2a73c52 MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe 
(cherry picked from commit 3a20debddeac69596ceb5b36f8413529ea8570e6)
 2017-09-19 09:14:41 -05:00
Daniel Templeton
1944b0b286 MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log (Contributed by Gergő Pásztor via Daniel Templeton) 
(cherry picked from commit a7463b6c88f698950a2f326030261001aa51b35e)
 2017-09-19 09:05:39 -05:00
Akira Ajisaka
e61baf9496
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák. 
(cherry picked from commit 2018538fdba1a95a6556187569e872fce7f9e1c3)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
 2017-09-19 11:25:29 +09:00
Jason Lowe
f525dfc5ec MAPREDUCE-6957. shuffle hangs after a node manager connection timeout. Contributed by Jooseong Kim 
(cherry picked from commit 4d98936eec1b5d196053426c70d455cf8f83f84f)
 2017-09-13 17:22:58 -05:00
Robert Kanter
72823b2738 MAPREDUCE-6953. Skip the testcase testJobWithChangePriority if FairScheduler is used (pbacsko via rkanter) 
(cherry picked from commit a323f73bae65483652de290ef819268850803a17)
(cherry picked from commit f5f343a6654647721120380556fdc2e401a39275)
 2017-09-08 13:18:59 -07:00
Haibo Chen
59f1ca7258 MAPREDUCE-6937. Backport MAPREDUCE-6870 to branch-2 while preserving compatibility. (Peter Bacsko via Haibo Chen) 2017-08-31 09:28:19 -07:00
Konstantin V Shvachko
9a0aac4170 MAPREDUCE-6931. Remove TestDFSIO "Total Throughput" calculation. Contributed by Dennis Huo. 
(cherry picked from commit 3e0e2033cc28d48a7909822416478aff7487bfe6)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
 2017-08-30 14:42:36 -07:00
Jason Lowe
cd08aa72a6 MAPREDUCE-6641. TestTaskAttempt fails in trunk. Contributed by Haibo Chen. 2017-08-29 12:24:42 -05:00
Robert Kanter
9de332d502 MAPREDUCE-6936. Remove unnecessary dependency of hadoop-yarn-server-common from hadoop-mapreduce-client-common (haibochen via rkanter) 
(cherry picked from commit ab051bd42ee1d7c4d3b7cc71e6b2734a0955e767)
 2017-08-16 16:15:48 -07:00
Jason Lowe
107e1722d4 MAPREDUCE-6940. Copy-paste error in the TaskAttemptUnsuccessfulCompletionEvent constructor. Contributed by Oleg Danilov 2017-08-16 16:35:41 -05:00
Ravi Prakash
c556cdaaf4 MAPREDUCE-6923. Optimize MapReduce Shuffle I/O for small partitions. Contributed by Robert Schmidtke. 
(cherry picked from commit ac7d0604bc73c0925eff240ad9837e14719d57b7)
 2017-08-09 15:40:57 -07:00
Jason Lowe
50dfef7275 MAPREDUCE-6927. MR job should only set tracking url if history was successfully written. Contributed by Eric Badger 
(cherry picked from commit 735fce5bec17f4e1799daf922625c475cf588114)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
 2017-08-08 14:55:55 -05:00
Arun Suresh
88d6adf419 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 35dc7829236f92054d5ce6ea78d3a44ca6c8f3d3)
 2017-08-03 21:18:43 -07:00
Yufei Gu
3ef7adc748 MAPREDUCE-6914. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 
(cherry picked from commit b8e8241854904e3a92b636ffaa6f4c651e47cea2)
 2017-08-03 11:47:09 -07:00
Akira Ajisaka
338d84ab71
MAPREDUCE-6921. TestUmbilicalProtocolWithJobToken#testJobTokenRpc fails. Contributed by Sonia Garudi. 
(cherry picked from commit ceacadc51e58bb94ad3f3669488515a61e886d88)
 2017-08-01 15:02:20 +09:00
Jason Lowe
67d31a8e7e Revert "MAPREDUCE-6829. Add peak memory usage counter for each task. (Miklos Szegedi via kasha)" 
This reverts commit a91cc63d374a5770b17cd8465080caf7f91da2a8.
 2017-07-31 16:44:13 -05:00
Junping Du
cae33098e6 Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)" 
This reverts commit e8a31f2e1c34514fba2f480e8db652f6e2ed65d8.

(cherry picked from commit fbb7d6bcbb887ce52ab1e9d5a1fed67a7f8a4be8)
 2017-07-31 14:15:06 -07:00
Junping Du
bf360da63f Revert "MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)" 
This reverts commit f9341c1e2cffc4e3ac37cb7da4e535ca5c6d3913.
 2017-07-31 14:13:26 -07:00
Junping Du
c0f8fd6432 Revert "MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu." 
This reverts commit 7c72c7f5290f0991f65102c2bd413f965a8ca409.
 2017-07-31 14:11:37 -07:00
Varun Saxena
0b342a644a YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit f66fd11e514fb326fd1f37a88b444a5276f0947b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
 2017-07-28 00:33:37 +05:30
Akira Ajisaka
22f237c5d9
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 
(cherry picked from commit cca51e916b7387ea358688e8f8188ead948fbdcc)
 2017-07-25 15:25:36 +09:00