Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,474 Commits 326 Branches 371 Tags
Commit Graph

2376 Commits

Author SHA1 Message Date
Bibin A Chundatt
58bcb90e70
MAPREDUCE-7073. Optimize TokenCache#obtainTokensForNamenodesInternal 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1a95a4524a8c6c7be601ce8b92640a6a76164a2c)
 2018-05-02 16:45:24 +09:00
Robert Kanter
b47275fe28 MAPREDUCE-7072. mapred job -history prints duplicate counter in human output (wilfreds via rkanter) 2018-04-27 09:57:31 -07:00
Sunil G
600f4d402f MAPREDUCE-7042. Killed MR job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled. Contributed by Rohith Sharma K S. 
(cherry picked from commit 83e60cd2db20f655e272958ef43b1b5a084ef3e3)
 2018-04-27 12:00:00 +05:30
Shashikant Banerjee
228869e5f9
HADOOP-15396. Some java source files are executable 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e4c39f3247da77d03c6015de4f18be75924fcb22)
 2018-04-19 08:38:11 +09:00
Erik Krogen
31d061e4e8 HADOOP-14970. MiniHadoopClusterManager doesn't respect lack of format option. Contributed by Erik Krogen 
(cherry picked from commit 1a407bc9906306801690bc75ff0f0456f8f265fd)
 2018-04-12 23:39:47 -07:00
Eric E Payne
b0dfb18daa YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 
(cherry picked from commit 6bb128dfb893cf0e4aa2d3ecc65440668a1fc8d7)
 2018-04-12 16:38:50 +00:00
Rohith Sharma K S
754e5189ee MAPREDUCE-7062. Update mapreduce.job.tags description for making use for ATSv2 purpose. Contributed by Charan Hebri. 
(cherry picked from commit 6cc59a09e7330dc893b386d84c8f2aa86c02eace)
 2018-04-11 09:52:17 +05:30
Vrushali C
fe4c7ee94c YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 
(cherry picked from commit 345e7624d58a058a1bad666bd1e5ce4b346a9056)
 2018-04-11 09:51:42 +05:30
Jason Lowe
121daaf474 MAPREDUCE-6930. mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores are both present twice in mapred-default.xml. Contributed by Sen Zhao 
(cherry picked from commit 32fa3a63e0e7d8bfb3d3b9b3c500ecb3a4874ecf)
 2018-03-09 10:45:47 -06:00
Ray Chiang
eeb731307f HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang) 
(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)
 2018-02-22 18:08:52 +05:30
Jason Lowe
b1d5e73c1f HADOOP-15206. BZip2 drops and duplicates records when input split size is small. Contributed by Aki Tanaka 
(cherry picked from commit 0898ff42e9e5c53f2fce7ccdeb4e1cd7d0f123b3)
 2018-02-16 15:00:10 -06:00
Eric Payne
904a6bf263 MAPREDUCE-7053: Timed out tasks can fail to produce thread dump. Contributed by Jason Lowe. 2018-02-16 08:47:42 -06:00
Jason Lowe
a87bccdc3f MAPREDUCE-7052. TestFixedLengthInputFormat#testFormatCompressedIn is flaky. Contributed by Peter Bacsko 
(cherry picked from commit a53d62ab26e170a0338f93e228718da52e9196e4)
 2018-02-15 15:27:46 -06:00
Kihwal Lee
56b950d777 Revert "HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang)" 
This reverts commit 577a267198b1d745eee85d1ad9af64d4506bbd0a.
 2018-02-14 15:20:45 -06:00
Ray Chiang
577a267198 HADOOP-14799. Update nimbus-jose-jwt to 4.41.1. (rchiang) 
(cherry picked from commit 556812c179aa094c21acf610439a8d69fe6420ab)
 2018-02-14 00:07:09 +05:30
Jason Lowe
08feac4c83 MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. Contributed by Peter Bacsko 2018-02-12 14:30:42 -06:00
Jason Lowe
e3a1647477 MAPREDUCE-7020. Task timeout in uber mode can crash AM. Contributed by Peter Bacsko 2018-01-26 15:46:01 -06:00
Haibo Chen
5f44bd3eba MAPREDUCE-7041. MR should not try to clean up at first job attempt. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cc10852252c2d69294eabc68bd032cc630a53b18)
(cherry picked from commit e6c66baba97c34514e1d7491d5fe80867930f184)
 2018-01-25 17:23:08 -08:00
Jason Lowe
1ec82e5789 MAPREDUCE-7015. Possible race condition in JHS if the job is not loaded. Contributed by Peter Bacsko 
(cherry picked from commit cff9edd4b514bdcfe22cd49964e3707fb78ab876)
 2018-01-24 14:51:22 -06:00
Haibo Chen
a286acd7be MAPREDUCE-6984. MR AM to clean up temporary files from previous attempt in case of no recovery. (Gergo Repas via Haibo Chen) 
(cherry picked from commit cce71dceef9e82d31fe8ec59648b2a4a50c8869a)
(cherry picked from commit fc669778bec543427ea94abe12aff34ee1533b5c)
 2018-01-19 13:12:26 -08:00
Jason Lowe
c228a7c707 MAPREDUCE-7029. FileOutputCommitter is slow on filesystems lacking recursive delete. Contributed by Karthik Palaniappan 2018-01-17 10:32:06 -06:00
Jason Lowe
f2a1d5f3a1 MAPREDUCE-7028. Concurrent task progress updates causing NPE in Application Master. Contributed by Gergo Repas 
(cherry picked from commit fe35103591ece0209f8345aba5544313e45a073c)
 2018-01-03 11:08:51 -06:00
Jason Lowe
d0fc1cd0c8 MAPREDUCE-5124. AM lacks flow control for task events. Contributed by Peter Bacsko 2017-12-01 14:15:25 -06:00
Arun Suresh
7af9b8ad1e Fixing Job History Server Configuration parsing. (Jason Lowe via asuresh) 2017-11-09 15:10:21 -08:00
Naganarasimha
45e6d36e5c MAPREDUCE-6975. Logging task counters. Contributed by Prabhu Joseph. 
(cherry picked from commit 61bbdc511e4e98c11a05d7c3bc01f3e971adbe13)
 2017-11-06 13:10:41 +05:30
Arun Suresh
71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Jason Lowe
347997c959 HADOOP-14919. BZip2 drops records when reading data in splits. Contributed by Jason Lowe 
(cherry picked from commit 2fae63aa60c43b62bd908a9499562fe528603185)
 2017-10-31 09:32:45 -05:00
Jason Lowe
9093ad6168 YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 
(cherry picked from commit 665bb147aa3fc198e53335931562dd69f9f0e345)
 2017-10-27 17:09:29 -05:00
Junping Du
1f2a2ba2be Add several jdiff xml files for 2.8.2 release. 
(cherry picked from commit a25b5aa0cf5189247fa38e7b0a188d568eba1b6c)
 2017-10-26 23:20:24 -07:00
Varun Saxena
56e0873e22 Fix javac, javadoc and checkstyle issues before branch-2 merge. 2017-10-17 23:36:41 +05:30
Jian He
cbd5e89b24 MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena 
(cherry picked from commit c6ce422a3423305ac665f80f039f61294adaa32c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2017-10-17 23:36:40 +05:30
Rohith Sharma K S
5a7c272e0a YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-10-17 23:36:39 +05:30
Sangjin Lee
bb100d35fd YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R. 
(cherry picked from commit 73235ab30361b41293846189f3c5fef321ae7cac)
 2017-10-17 23:36:36 +05:30
Sangjin Lee
c8a8e680bd MAPREDUCE-6818. Remove direct reference to TimelineClientImpl. Contributed by Li Lu. 
(cherry picked from commit 6217b87f4a056cf704cef2e073b386b7803415de)
 2017-10-17 23:36:35 +05:30
Sangjin Lee
b8cfb4fcb3 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena. 
(cherry picked from commit f734977b27a514ce0561638c0a6a17b1ef093026)
 2017-10-17 23:36:35 +05:30
Varun Saxena
0820df611d MAPREDUCE-6731. TestMRTimelineEventHandling#testMRNewTimelineServiceEventHandling fails randomly for concurrent tests. (Sangjin Lee via Varun Saxena). 
(cherry picked from commit d14e729a55c0ca4053c537290392b74f732ce939)
 2017-10-17 23:36:33 +05:30
Sangjin Lee
e212e562ec YARN-5355. Backported YARN-2928 into our branch-2 feature branch. 2017-10-17 23:36:33 +05:30
Chris Trezzo
3e7c06af41 MAPREDUCE-5951. Add support for the YARN Shared Cache. 
(cherry picked from commit e46d5bb962b0c942f993afc505b165b1cd96e51b)
 2017-10-12 11:30:38 -07:00
Rohith Sharma K S
c2b8d4b9ab MAPREDUCE-6951. Improve exception message when mapreduce.jobhistory.webapp.address is in wrong format. Contributed by Prabhu Joseph. 
(cherry picked from commit 639f98cc9dc9fca034d0d7c26f30d5c588b9d7b8)
 2017-10-11 14:38:00 +05:30
Jason Lowe
b94acb6efb MAPREDUCE-6965. QuasiMonteCarlo should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit 908d8e914ef55aca69cb8e725e62bda29d1073cc)
 2017-09-22 09:37:09 -05:00
Akira Ajisaka
d80e2f2eae
MAPREDUCE-6966. DistSum should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari. 
(cherry picked from commit c71d13796039815350d3c73c51f9fe48531af0fa)
 2017-09-22 15:10:19 +09:00
Akira Ajisaka
9d4de9c556
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergely Novák. 
(cherry picked from commit 53be075241f1ba92bfe47e89c2dfc3f0664e2578)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
 2017-09-22 13:29:17 +09:00
Jason Lowe
cf2615961a MAPREDUCE-6964. BaileyBorweinPlouffe should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b9db0afa1e65d0343ad4e5760c36e042a3e704b0)
 2017-09-21 09:39:46 -05:00
Akira Ajisaka
7d747df52d
Revert "MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák." 
This reverts commit e61baf94965e25b6a7ab2264523142d066b490b2.
 2017-09-21 11:16:48 +09:00
Eric Payne
d2386725ff MAPREDUCE-6960. Shuffle Handler prints disk error stack traces for every read failure. 
(cherry picked from commit 595d478408104bdfe1f08efd79930e18862fafbb)
 2017-09-19 10:46:05 -05:00
Jason Lowe
41c2a73c52 MAPREDUCE-6958. Shuffle audit logger should log size of shuffle transfer. Contributed by Jason Lowe 
(cherry picked from commit 3a20debddeac69596ceb5b36f8413529ea8570e6)
 2017-09-19 09:14:41 -05:00
Daniel Templeton
1944b0b286 MAPREDUCE-6808. Log map attempts as part of shuffle handler audit log (Contributed by Gergő Pásztor via Daniel Templeton) 
(cherry picked from commit a7463b6c88f698950a2f326030261001aa51b35e)
 2017-09-19 09:05:39 -05:00
Akira Ajisaka
e61baf9496
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák. 
(cherry picked from commit 2018538fdba1a95a6556187569e872fce7f9e1c3)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
 2017-09-19 11:25:29 +09:00
Jason Lowe
f525dfc5ec MAPREDUCE-6957. shuffle hangs after a node manager connection timeout. Contributed by Jooseong Kim 
(cherry picked from commit 4d98936eec1b5d196053426c70d455cf8f83f84f)
 2017-09-13 17:22:58 -05:00
Robert Kanter
72823b2738 MAPREDUCE-6953. Skip the testcase testJobWithChangePriority if FairScheduler is used (pbacsko via rkanter) 
(cherry picked from commit a323f73bae65483652de290ef819268850803a17)
(cherry picked from commit f5f343a6654647721120380556fdc2e401a39275)
 2017-09-08 13:18:59 -07:00