Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,134 Commits 326 Branches 371 Tags
Commit Graph

645 Commits

Author SHA1 Message Date
Akira Ajisaka
1cccd1eeb3 HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. Contributed by Kazuho Fujii. 
(cherry picked from commit dbed757cbaf3ddff0c9ae2d905d42828aadedf2a)
 2015-06-05 01:46:33 +09:00
Gera Shegalov
25c1e54d3f MAPREDUCE-6174. Combine common stream code into parent class for InMemoryMapOutput and OnDiskMapOutput. (Eric Payne via gera) 
(cherry picked from commit d90c13e2da8867661bf19a802add70145ab9a462)
 2015-06-03 16:57:54 -07:00
Jason Lowe
5ecc647ae0 MAPREDUCE-6374. Distributed Cache File visibility should check permission of full path. Contributed by Chang Li 
(cherry picked from commit 107da29ff9b6efd02b21c96df60c58ac51590ebd)
 2015-06-03 20:20:47 +00:00
Wangda Tan
7cd0874dfd MAPREDUCE-6304. Specifying node labels when submitting MR jobs. (Naganarasimha G R via wangda) 
(cherry picked from commit 3164e7d83875aa6b7435d1dfe61ac280aa277f1c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2015-05-27 14:43:20 -07:00
Colin Patrick Mccabe
3dec58dd78 HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe) 
(cherry picked from commit 7dba7005b79994106321b0f86bc8f4ea51a3c185)
 2015-05-26 12:40:41 -07:00
Tsuyoshi Ozawa
e694c3339e MAPREDUCE-6364. Add a Kill link to Task Attempts page. Contributed by Ryu Kobayashi. 
(cherry picked from commit 022f49d59e404678d0f322bb1b67208f3c17b7ef)
 2015-05-26 23:59:20 +09:00
Vinayakumar B
19895a41ad HADOOP-11743. maven doesn't clean all the site files (Contributed by ramtin) 
(cherry picked from commit 55ed6558bad5d76ce4ebf958fc101a48032a0cd1)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
 2015-05-22 16:46:50 +05:30
Vinod Kumar Vavilapalli
87862970f1 Fixing MR intermediate spills. Contributed by Arun Suresh. 
(cherry picked from commit 6b710a42e00acca405e085724c89cda016cf7442)
 2015-05-14 16:56:56 -07:00
Akira Ajisaka
345e62c6d5 Revert "MAPREDUCE-5762. Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2. (aajisaka)" 
This reverts commit 853cc0466811c310486fff6042d7b76ab3b78947.
 2015-05-13 14:02:41 +09:00
Vinod Kumar Vavilapalli
d5755ba241 MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch. 
(cherry picked from commit f24452d14e9ba48cdb82e5e6e5c10ce5b1407308)
 2015-05-12 12:13:23 -07:00
Tsuyoshi Ozawa
fb5b0ebb45 MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. 
(cherry picked from commit f4e2b3cc0b1f4e49c306bc09a9dddd0495225bb2)
 2015-05-13 00:28:36 +09:00
Jason Lowe
53e9973afc MAPREDUCE-5465. Tasks are often killed before they exit on their own. Contributed by Ming Ma 2015-05-11 22:40:05 +00:00
Junping Du
a1a7deebf8 YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc. projects. Contributed by Gabor Liptak. 
(cherry picked from commit 7e543c27fa2881aa65967be384a6203bd5b2304f)
 2015-05-11 05:28:25 -07:00
Chris Douglas
4f301f92c1 MAPREDUCE-2094. LineRecordReader should not seek into non-splittable, compressed streams. 2015-05-08 14:30:56 -07:00
Jason Lowe
dfc4a62f7d MAPREDUCE-3383. Duplicate job.getOutputValueGroupingComparator() in ReduceTask. Contributed by Binglin Chang 
(cherry picked from commit c39012f4a0444f9e4b7d67957d5192127d143d90)
 2015-05-08 21:17:05 +00:00
Devaraj K
a957baeb71 MAPREDUCE-5981. Log levels of certain MR logs can be changed to DEBUG. 
Contributed by Varun Saxena.

(cherry picked from commit dc2b2ae31f2eb6dae324c2e14ed7660ce605a89b)
 2015-05-08 21:28:52 +05:30
Devaraj K
ade5165ecf MAPREDUCE-6342. Make POM project names consistent. Contributed by Rohith. 2015-05-08 15:20:35 +05:30
Tsuyoshi Ozawa
1ae8bf5b21 MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi. 
(cherry picked from commit d18f10ad1b3e497fa1aaaeb85ba055f87d9849f7)
 2015-05-08 16:00:41 +09:00
Akira Ajisaka
f7c011cbaa MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. 
(cherry picked from commit d335071398fdb14153c854b740814b4fd385b658)
 2015-05-07 19:41:44 +09:00
Tsuyoshi Ozawa
4e96175b33 MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. Contributed by Akira AJISAKA. 
(cherry picked from commit 551615fa13f65ae996bae9c1bacff189539b6557)
 2015-05-05 10:23:25 +09:00
Jason Lowe
81f128f293 MAPREDUCE-6259. IllegalArgumentException due to missing job submit time. Contributed by zhihai xu 
(cherry picked from commit bf70c5ae2824a9139c1aa9d7c14020018881cec2)
 2015-05-04 20:40:16 +00:00
Jason Lowe
87c2d915f1 MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov 
(cherry picked from commit 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a)
 2015-05-04 19:03:43 +00:00
Tsuyoshi Ozawa
7e03bda3e7 MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. Contributed by Ray Chiang. 
(cherry picked from commit bb6ef2984d8f117711b806c4ebdc757bd182c06e)
 2015-05-04 15:36:53 +09:00
Tsuyoshi Ozawa
1035f550b9 MAPREDUCE-5905. CountersStrings.toEscapedCompactStrings outputs unnecessary null strings. Contributed by Akira AJISAKA. 
(cherry picked from commit 3ba18362f2a4b83635b89aa0adc5ebaf27d9ca83)
 2015-05-04 15:02:40 +09:00
Gera Shegalov
ff6492d603 MAPREDUCE-6345. Documentation fix for when CRLA is enabled for MRAppMaster logs. (Rohit Agarwal via gera) 
(cherry picked from commit f1a152cc0adc071277c80637ea6f5faa0bf06a1a)
 2015-05-01 19:07:14 -07:00
Jason Lowe
ea828578a3 MAPREDUCE-6334. Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler. Contributed by Eric Payne 
(cherry picked from commit bc1bd7e5c4047b374420683d36a8c30eda6d75b6)
 2015-04-28 20:19:52 +00:00
Tsuyoshi Ozawa
fa9def5a2a MAPREDUCE-6341. Fix typo in mapreduce tutorial. Contributed by John Michael Luy. 
(cherry picked from commit 9a3dda3d345b94cf6f9062c00395500596829cf1)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
 2015-04-28 01:58:12 +09:00
Tsuyoshi Ozawa
542d53a268 MAPREDUCE-6330. Fix typo in some Task Attempt API's URL. Contributed by Ryu Kobayashi. 
(cherry picked from commit e33594dd87f83953b59b7416c82992de760eb4cd)
 2015-04-23 03:22:02 +09:00
Gera Shegalov
6f06e3b7ea MAPREDUCE-6297. Task Id of the failed task in diagnostics should link to the task page. (Siqi Li via gera) 
(cherry picked from commit 89ded89e86e5d9a634d92a5d8a7c889744d97f94)
 2015-04-21 14:21:57 -07:00
Robert Kanter
756c254293 MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) 
(cherry picked from commit d50e8f09287deeb51012d08e326a2ed71a6da869)
 2015-04-20 14:14:49 -07:00
Jian He
42cf8e0554 YARN-3021. YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp. Contributed by Yongjun Zhang 
(cherry picked from commit bb6dde68f19be1885a9e7f7949316a03825b6f3e)
 2015-04-16 19:44:32 -07:00
Akira Ajisaka
f603b9ccca MAPREDUCE-6307. Remove property mapreduce.tasktracker.taskmemorymanager.monitoringinterval. Contributed by J.Andreina. 
(cherry picked from commit 83979e61a49daa67a05eed1ba64c24d31a41610a)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2015-04-10 20:23:46 +09:00
Gera Shegalov
3757e731e6 MAPREDUCE-4844. Counters / AbstractCounters have constant references not declared final. (Brahma Reddy Battula via gera) 
(cherry picked from commit a0795c1e81685b812288cf2f4df6f8d23da30ab8)
 2015-04-03 10:35:18 -07:00
Junping Du
cacadea632 MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe. 
(cherry picked from commit eccb7d46efbf07abcc6a01bd5e7d682f6815b824)
 2015-04-02 12:15:12 -07:00
Vinod Kumar Vavilapalli
c5bc48946d YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. Contributed by Junping Du and Karthik Kambatla. 
(cherry picked from commit c358368f511963ad8e35f030b9babee541e1bd01)
 2015-03-30 10:11:12 -07:00
Harsh J
274db918c3 MAPREDUCE-6291. Correct mapred queue usage command. Contributed by Brahma Reddy Battula. 
(cherry picked from commit f5432cb47dfacaa57faed32a29d00bbeff8a5385)
 2015-03-28 11:58:44 +05:30
Akira Ajisaka
853cc04668 MAPREDUCE-5762. Port MAPREDUCE-3223 (Remove MRv1 config from mapred-default.xml) to branch-2. (aajisaka) 2015-03-28 03:35:03 +09:00
Tsuyoshi Ozawa
f0aa2a7466 MAPREDUCE-6294. Remove an extra parameter described in Javadoc of TockenCache. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 05499b1093ea6ba6a39a1354d67b0a46a2982824)
 2015-03-28 00:08:56 +09:00
Devaraj K
943d9ee603 MAPREDUCE-6242. Progress report log is incredibly excessive in application 
master. Contributed by Varun Saxena.
 2015-03-23 22:48:00 +05:30
Harsh J
6e6e0e4a37 MAPREDUCE-5448. Addendum fix to remove deprecation warning by junit.Assert import in TestFileOutputCommitter. 
(cherry picked from commit 4cd54d9a297435150ab61803284eb05603f114e2)
 2015-03-22 10:33:29 +05:30
Harsh J
d43c0da7d7 MAPREDUCE-5190. Unnecessary condition test in RandomSampler. Contributed by Jingguo Yao. 
(cherry picked from commit 1d5c796d654c8959972d15cc6742731a99380bfc)
 2015-03-22 10:04:19 +05:30
Harsh J
a7f1c1b993 MAPREDUCE-5448. MapFileOutputFormat#getReaders bug with invisible files/folders. Contributed by Maysam Yabandeh. 
(cherry picked from commit b46c2bb51ae524e6640756620f70e5925cda7592)
 2015-03-22 09:46:54 +05:30
Harsh J
7c72c7f529 MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu. 
(cherry picked from commit 433542904aba5ddebf9bd9d299378647351eb13a)
 2015-03-22 02:51:17 +05:30
Harsh J
97da36777f MAPREDUCE-6239. Consolidate TestJobConf classes in hadoop-mapreduce-client-jobclient and hadoop-mapreduce-client-core. Contributed by Varun Saxena. 2015-03-22 02:20:11 +05:30
Jason Lowe
cabe676d67 MAPREDUCE-6275. Race condition in FileOutputCommitter v2 for user-specified task output subdirs. Contributed by Gera Shegalov and Siqi Li 
(cherry picked from commit 91baca145a6c16fe13f455d150c05bd73179531b)
 2015-03-19 21:40:51 +00:00
Tsuyoshi Ozawa
bb828565f8 MAPREDUCE-5755. MapTask.MapOutputBuffer#compare/swap should have @Override annotation. (ozawa) 
(cherry picked from commit bb243cea93b6872ef8956311a2290ca0b83ebb22)
 2015-03-17 14:55:33 +09:00
Harsh J
5e403e8b8d MAPREDUCE-6100. replace mapreduce.job.credentials.binary with MRJobConfig.MAPREDUCE_JOB_CREDENTIALS_BINARY for better readability. Contributed by Zhihai Xu. 
(cherry picked from commit f222bde273cc10a38945dc31e85206a0c4f06a12)
 2015-03-17 11:08:32 +05:30
Harsh J
0e9f78dda2 MAPREDUCE-6105. nconsistent configuration in property mapreduce.reduce.shuffle.merge.percent. Contributed by Ray Chiang. 
(cherry picked from commit 26a23b11598b3757ed4973781890e2cd2fc270df)
 2015-03-17 02:27:33 +05:30
Harsh J
2684b0b440 MAPREDUCE-4414. Add main methods to JobConf and YarnConfiguration, for debug purposes. Contributed by Plamen Jeliazkov. 
(cherry picked from commit 571f75b468a4412be2178466cb5d3888b05cd076)
 2015-03-17 01:02:21 +05:30
Tsuyoshi Ozawa
10745fcecd MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers. Contributed by Zhihai Xu 
(cherry picked from commit 9d38520c8e42530a817a7f69c9aa73a9ad40639c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2015-03-14 16:45:23 +09:00