Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,647 Commits 326 Branches 371 Tags
Commit Graph

1695 Commits

Author SHA1 Message Date
Daniel Templeton
1505058440
MAPREDUCE-6896. Document wrong spelling in usage of MapredTestDriver tools. 
(Contributed by LiXin Ge via Daniel Templeton)

(cherry picked from commit 036a24bab3f3ff89e3d52c5487fce0525c0d1d78)
 2018-10-04 10:12:57 +09:00
Jason Lowe
726aee360a MAPREDUCE-7138. ThrottledContainerAllocator in MRAppBenchmark should implement RMHeartbeatHandler. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 8382b860d4ef4f20d000537ded42a88e98bd2190)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java
 2018-09-18 17:19:21 -05:00
Jason Lowe
f5cc9f71e6 MAPREDUCE-7137. MRAppBenchmark.benchmark1() fails with NullPointerException. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 34b2237e420cfbe3a97ddd44968de8bbe1ed30ab)
 2018-09-18 17:15:58 -05:00
Eric E Payne
97a13eb046 MAPREDUCE-3801: org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testExponentialEstimator fails intermittently. Contributed by Jason Lowe 
(cherry picked from commit 2cf8927683d49f4cdc3cba038480128c473c5461)
 2018-09-18 20:54:07 +00:00
Jason Lowe
1e61f389e9 MAPREDUCE-7133. History Server task attempts REST API returns invalid data. Contributed by Oleksandr Shevchenko 
(cherry picked from commit 2886024ac3a8613ecc27f1595b278ce6fc2d03ba)
 2018-09-13 14:51:19 -05:00
Jason Lowe
3eb70d4dcb MAPREDUCE-7131. Job History Server has race condition where it moves files from intermediate to finished but thinks file is in intermediate. Contributed by Anthony Hsu 
(cherry picked from commit eb0b5a844f960017f6f48d746174d0f5826f0e5f)
 2018-09-06 15:27:45 -05:00
Arun Suresh
358fe4c61a MAPREDUCE-7101. Add config parameter to allow JHS to alway scan user dir irrespective of modTime. (Thomas Marquardt via asuresh) 
(cherry picked from commit 5670e89b2ec69ab71e32dcd5acbd3a57ca6abea5)
 2018-06-12 16:25:01 -07:00
Gera Shegalov
2e059ff4e6 MAPREDUCE-7108. TestFileOutputCommitter fails on Windows. (Zuoming Zhang via gera) 
(cherry picked from commit 04b74eddc67888142879ed114d21348e8a4aad78)
 2018-06-12 12:43:20 -07:00
Inigo Goiri
ea2dd6b5a9 Revert "MAPREDUCE-7103. Fix TestHistoryViewerPrinter on windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola." 
This reverts commit cd34c229a7c31f656638dda2c5f8edbcca197c0c.
 2018-06-06 12:46:03 -07:00
Inigo Goiri
cd34c229a7 MAPREDUCE-7103. Fix TestHistoryViewerPrinter on windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 0afc036deb35df7e86ede3dcebc430c8f05ed368)
 2018-06-05 12:27:10 -07:00
Inigo Goiri
1540062040 MAPREDUCE-7103. Fix TestHistoryViewerPrinter on windows due to a mismatch line separator. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 0afc036deb35df7e86ede3dcebc430c8f05ed368)
 2018-06-05 12:26:41 -07:00
Inigo Goiri
d00a58ff29 MAPREDUCE-7102. Fix TestJavaSerialization for Windows due a mismatch line separator. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit a804b7c9d2986556c91c1741c1ae7ac2b9c579f3)
 2018-06-02 20:27:03 -07:00
Bibin A Chundatt
58bcb90e70
MAPREDUCE-7073. Optimize TokenCache#obtainTokensForNamenodesInternal 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1a95a4524a8c6c7be601ce8b92640a6a76164a2c)
 2018-05-02 16:45:24 +09:00
Robert Kanter
b47275fe28 MAPREDUCE-7072. mapred job -history prints duplicate counter in human output (wilfreds via rkanter) 2018-04-27 09:57:31 -07:00
Sunil G
600f4d402f MAPREDUCE-7042. Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled. Contributed by Rohith Sharma K S. 
(cherry picked from commit 83e60cd2db20f655e272958ef43b1b5a084ef3e3)
 2018-04-27 12:00:00 +05:30
Shashikant Banerjee
228869e5f9
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247da77d03c6015de4f18be75924fcb22)
 2018-04-19 08:38:11 +09:00
Erik Krogen
31d061e4e8 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen 
(cherry picked from commit 1a407bc9906306801690bc75ff0f0456f8f265fd)
 2018-04-12 23:39:47 -07:00
Eric E Payne
b0dfb18daa YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 
(cherry picked from commit 6bb128dfb893cf0e4aa2d3ecc65440668a1fc8d7)
 2018-04-12 16:38:50 +00:00
Rohith Sharma K S
754e5189ee MAPREDUCE-7062. Update mapreduce.job.tags description for making use for ATSv2 purpose. Contributed by Charan Hebri. 
(cherry picked from commit 6cc59a09e7330dc893b386d84c8f2aa86c02eace)
 2018-04-11 09:52:17 +05:30
Vrushali C
fe4c7ee94c YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 
(cherry picked from commit 345e7624d58a058a1bad666bd1e5ce4b346a9056)
 2018-04-11 09:51:42 +05:30
Jason Lowe
121daaf474 MAPREDUCE-6930. mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml. Contributed by Sen Zhao 
(cherry picked from commit 32fa3a63e0e7d8bfb3d3b9b3c500ecb3a4874ecf)
 2018-03-09 10:45:47 -06:00
Ray Chiang
eeb731307f HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang) 
(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)
 2018-02-22 18:08:52 +05:30
Jason Lowe
b1d5e73c1f HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka 
(cherry picked from commit 0898ff42e9e5c53f2fce7ccdeb4e1cd7d0f123b3)
 2018-02-16 15:00:10 -06:00
Eric Payne
904a6bf263 MAPREDUCE-7053: Timed out tasks can fail to produce thread dump. Contributed by Jason Lowe. 2018-02-16 08:47:42 -06:00
Jason Lowe
a87bccdc3f MAPREDUCE-7052. TestFixedLengthInputFormat#testFormatCompressedIn is flaky. Contributed by Peter Bacsko 
(cherry picked from commit a53d62ab26e170a0338f93e228718da52e9196e4)
 2018-02-15 15:27:46 -06:00
Kihwal Lee
56b950d777 Revert "HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)" 
This reverts commit 577a267198b1d745eee85d1ad9af64d4506bbd0a.
 2018-02-14 15:20:45 -06:00
Ray Chiang
577a267198 HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang) 
(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)
 2018-02-14 00:07:09 +05:30
Jason Lowe
08feac4c83 MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko 2018-02-12 14:30:42 -06:00
Jason Lowe
e3a1647477 MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko 2018-01-26 15:46:01 -06:00
Haibo Chen
5f44bd3eba MAPREDUCE-7041. MR should not try to clean up at first job attempt. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cc10852252c2d69294eabc68bd032cc630a53b18)
(cherry picked from commit e6c66baba97c34514e1d7491d5fe80867930f184)
 2018-01-25 17:23:08 -08:00
Jason Lowe
1ec82e5789 MAPREDUCE-7015. Possible race condition in JHS if the job is not loaded. Contributed by Peter Bacsko 
(cherry picked from commit cff9edd4b514bdcfe22cd49964e3707fb78ab876)
 2018-01-24 14:51:22 -06:00
Haibo Chen
a286acd7be MAPREDUCE-6984. MR AM to clean up temporary files from previous attempt in case of no recovery. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cce71dceef9e82d31fe8ec59648b2a4a50c8869a)
(cherry picked from commit fc669778bec543427ea94abe12aff34ee1533b5c)
 2018-01-19 13:12:26 -08:00
Jason Lowe
c228a7c707 MAPREDUCE-7029. FileOutputCommitter is slow on filesystems lacking recursive delete. Contributed by Karthik Palaniappan 2018-01-17 10:32:06 -06:00
Jason Lowe
f2a1d5f3a1 MAPREDUCE-7028. Concurrent task progress updates causing NPE in Application Master. Contributed by Gergo Repas 
(cherry picked from commit fe35103591ece0209f8345aba5544313e45a073c)
 2018-01-03 11:08:51 -06:00
Jason Lowe
d0fc1cd0c8 MAPREDUCE-5124. AM lacks flow control for task events. Contributed by Peter Bacsko 2017-12-01 14:15:25 -06:00
Arun Suresh
7af9b8ad1e Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh) 2017-11-09 15:10:21 -08:00
Naganarasimha
45e6d36e5c MAPREDUCE-6975. Logging task counters. Contributed by Prabhu Joseph. 
(cherry picked from commit 61bbdc511e4e98c11a05d7c3bc01f3e971adbe13)
 2017-11-06 13:10:41 +05:30
Arun Suresh
71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Jason Lowe
347997c959 HADOOP-14919. BZip2 drops records when reading data in splits. Contributed by Jason Lowe 
(cherry picked from commit 2fae63aa60c43b62bd908a9499562fe528603185)
 2017-10-31 09:32:45 -05:00
Jason Lowe
9093ad6168 YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 
(cherry picked from commit 665bb147aa3fc198e53335931562dd69f9f0e345)
 2017-10-27 17:09:29 -05:00
Varun Saxena
56e0873e22 Fix javac, javadoc and checkstyle issues before branch-2 merge. 2017-10-17 23:36:41 +05:30
Jian He
cbd5e89b24 MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 
(cherry picked from commit c6ce422a3423305ac665f80f039f61294adaa32c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2017-10-17 23:36:40 +05:30
Rohith Sharma K S
5a7c272e0a YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-10-17 23:36:39 +05:30
Sangjin Lee
bb100d35fd YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R. 
(cherry picked from commit 73235ab30361b41293846189f3c5fef321ae7cac)
 2017-10-17 23:36:36 +05:30
Sangjin Lee
c8a8e680bd MAPREDUCE-6818. Remove direct reference to TimelineClientImpl. Contributed by Li Lu. 
(cherry picked from commit 6217b87f4a056cf704cef2e073b386b7803415de)
 2017-10-17 23:36:35 +05:30
Sangjin Lee
b8cfb4fcb3 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 
(cherry picked from commit f734977b27a514ce0561638c0a6a17b1ef093026)
 2017-10-17 23:36:35 +05:30
Varun Saxena
0820df611d MAPREDUCE-6731. TestMRTimelineEventHandling#testMRNewTimelineServiceEventHandling fails randomly for concurrent tests. (Sangjin Lee via Varun Saxena). 
(cherry picked from commit d14e729a55c0ca4053c537290392b74f732ce939)
 2017-10-17 23:36:33 +05:30
Sangjin Lee
e212e562ec YARN-5355. Backported YARN-2928 into our branch-2 feature branch. 2017-10-17 23:36:33 +05:30
Chris Trezzo
3e7c06af41 MAPREDUCE-5951. Add support for the YARN Shared Cache. 
(cherry picked from commit e46d5bb962b0c942f993afc505b165b1cd96e51b)
 2017-10-12 11:30:38 -07:00
Rohith Sharma K S
c2b8d4b9ab MAPREDUCE-6951. Improve exception message when mapreduce.jobhistory.webapp.address is in wrong format. Contributed by Prabhu Joseph. 
(cherry picked from commit 639f98cc9dc9fca034d0d7c26f30d5c588b9d7b8)
 2017-10-11 14:38:00 +05:30