Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,673 Commits 326 Branches 371 Tags
Commit Graph

1584 Commits

Author SHA1 Message Date
Karthik Kambatla
a91cc63d37 MAPREDUCE-6829. Add peak memory usage counter for each task. (Miklos Szegedi via kasha) 
(cherry picked from commit c65f884fc7e08118524f8c88737119d8196b4c1b)
 2017-01-26 11:08:53 -08:00
Jason Lowe
94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 
(cherry picked from commit 69fa81679f59378fd19a2c65db8019393d7c05a2)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
 2017-01-23 17:23:32 +00:00
Jason Lowe
bec7c56ce5 MAPREDUCE-6831. Flaky test TestJobImpl.testKilledDuringKillAbort. Contributed by Peter Bacsko 
(cherry picked from commit 78b487bde175544ebe40e4dafab35569baa1d79e)
 2017-01-17 16:03:20 +00:00
Arun Suresh
4985217de4 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh) 2017-01-08 08:27:10 -08:00
Junping Du
bf87c523ab MAPREDUCE-6711. JobImpl fails to handle preemption events on state COMMITTING. Contributed by Prabhu Joseph. 
(cherry picked from commit 679478d0c643defb3759a2ecd3c704315edee124)
 2017-01-07 22:05:16 -08:00
Arun Suresh
2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2273a74c1f3895163046cca09ff5e983df301d22)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
 2017-01-06 11:15:10 -08:00
Arun Suresh
368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 
(cherry picked from commit 3219b7b4ac7d12aee343f6ab2980b3357fc618b6)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
 2017-01-06 11:15:09 -08:00
Arun Suresh
7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d401a2f49d0f9ca556343f9ba871e979)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Jason Lowe
937f739804 MAPREDUCE-6682. TestMRCJCFileOutputCommitter fails intermittently Contributed by Akira Ajisaka. 
(cherry picked from commit 8f1c374bec2451568f463ab68f7eb6db734ab14e)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMRTimelineEventHandling.java
 2017-01-06 18:46:28 +00:00
Daniel Templeton
d1aa844dc6 MAPREDUCE-6715. Fix Several Unsafe Practices (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 0b8a7c18ddbe73b356b3c9baf4460659ccaee095)
 2017-01-05 17:57:14 -08:00
Akira Ajisaka
3e2d26a97f HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe9389ccdc153d7f4bf6574bf8e509867c1)
 2017-01-04 14:15:00 +09:00
Brahma Reddy Battula
d6e5e4288c HADOOP-13883. Addendum patch to correct the message and alphabetized with the earlier patch. Contributed by Yiqun Lin. 
(cherry picked from commit 6938b677111a5fa6bf654717b310220f98b7cf20)
 2016-12-31 16:17:15 +05:30
Daniel Templeton
28b605dd1a HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton) 
(cherry picked from commit d432173737a532b7106682116013b751ba162ff0)
 2016-12-27 07:19:37 -08:00
Akira Ajisaka
522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539eaf7745ec8690f9281726fb4cdcdbe94)
 2016-12-23 04:06:53 +09:00
Akira Ajisaka
1e1ab9bc1b MAPREDUCE-6820. Fix dead links in Job relevant classes. Contributed by Yiqun Lin. 
(cherry picked from commit 92a8917ca5a25bf6bee75acbb554aae87766a7e7)
 2016-12-10 10:57:06 +09:00
Karthik Kambatla
0aaaeea268 MAPREDUCE-6817. The format of job start time in JHS is different from those of submit and finish time. (Haibo Chen via kasha) 
(cherry picked from commit a793cec2c9a7ad80aaf67be9dec2245e8ecce63c)
 2016-12-07 13:37:12 -08:00
Akira Ajisaka
03854f0bc5 MAPREDUCE-6816. Progress bars in Web UI always at 100%. Contributed by Shen Yinjie. 
(cherry picked from commit da4ecc9f8223ac82bff8483594fb29be2f8d4d34)
 2016-12-07 06:44:03 +09:00
Robert Kanter
1585b4f7ce MAPREDUCE-6571. JobEndNotification info logs are missing in AM container syslog (haibochen via rkanter) 
(cherry picked from commit 6b4a9970f187ab399adf7469e494a530f08993b5)
 2016-12-06 13:11:20 -08:00
Jason Lowe
e8cd9dc200 MAPREDUCE-6815. Fix flaky TestKill.testKillTask(). Contributed by Haibo Chen 
(cherry picked from commit 0cfd7ad21f4457513ed3416e5d77f3123bfe9da0)
 2016-12-02 17:23:10 +00:00
Robert Kanter
99b046f8a9 MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job overview page in JHS (haibochen via rkanter) 
(cherry picked from commit c87b3a448a00df97149a4e93a8c39d9ad0268bdb)
 2016-12-01 17:30:02 -08:00
Junping Du
376a2439db MAPREDUCE-6565. Configuration to use host name in delegation token service is not read from job.xml during MapReduce job execution. Contributed by Li Lu. 
(cherry picked from commit 8f6e14399a3e77e1bdcc5034f7601e9f62163dea)
 2016-11-29 16:50:17 -08:00
Rohith Sharma K S
4b289d567e MAPREDUCE-6793. io.sort.factor code default and mapred-default.xml values inconsistent. Contributed by Gera Shegalov. 
(cherry picked from commit 683e0c71fe09600a24bdd7b707a613fe70ff1f6e)
 2016-11-21 22:11:59 +05:30
Varun Saxena
ca3b40306b MAPREDUCE-6801. Fix flaky TestKill.testKillJob (Haibo Chen via Varun Saxena) 
(cherry picked from commit 7584fbf4cbafd34fac4b362cefe4e06cec16a2af)
 2016-11-19 01:39:47 +05:30
Varun Saxena
bcfa6d5384 MAPREDUCE-6811. TestPipeApplication#testSubmitter fails after HADOOP-13802 (Brahma Reddy Battula via Varun Saxena) 
(cherry picked from commit b8690a9d25cb212d53a92bfa1f76a33b8c74413c)
 2016-11-17 00:55:12 +05:30
Jason Lowe
41f8275c50 MAPREDUCE-6804. Add timeout when starting JobHistoryServer in MiniMRYarnCluster. Contributed by Andras Bokor 2016-11-16 15:26:19 +00:00
Mingliang Liu
4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b6443715b7a741296c1bd520a1840f9a7c)
 2016-11-15 11:36:08 -08:00
Jason Lowe
5ffb26c2a9 MAPREDUCE-6797. Job history server scans can become blocked on a single, slow entry. Contributed by Prabhu Joseph 
(cherry picked from commit 99c2bbd337942e4bc7b246a88dff53f98e530651)
 2016-11-14 20:30:36 +00:00
Jian He
a422740bd7 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:39:46 -08:00
Andrew Wang
4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa9d87fdfc7474ca63998a13a3929874)
 2016-11-08 16:33:58 -08:00
Varun Saxena
a942d968c9 MAPREDUCE-6782. JHS task page search based on each individual column not working (Ajith S via Varun Saxena) 
(cherry picked from commit 026b39ad9d9336b1efdd8e2ffb9a508cb0db6259)
 2016-11-08 15:02:46 +05:30
Robert Kanter
568b5eecd8 MAPREDUCE-6765. MR should not schedule container requests in cases where reducer or mapper containers demand resource larger than the maximum supported (haibochen via rkanter) 
(cherry picked from commit fc2b69eba1c5df59f6175205c27dc7b584df50c0)
 2016-11-01 20:47:58 -07:00
Akira Ajisaka
a3897f77ae MAPREDUCE-6795. Update the document for JobConf#setNumReduceTasks. Contributed by Yiqun Lin. 
(cherry picked from commit 235f39b7c44c5907bd4de063892905577c35cd80)
 2016-11-01 20:52:35 +09:00
Akira Ajisaka
ef13a5ef9d MAPREDUCE-6799. Document mapreduce.jobhistory.webapp.https.address in mapred-default.xml. Contributed by Yiqun Lin. 
(cherry picked from commit f646fe34be704a22cbea322cce2826f8f201bd34)
 2016-10-31 15:18:34 +09:00
Naganarasimha
1455f56b63 MAPREDUCE-2631. Potential resource leaks in BinaryProtocol$TeeOutputStream.java. Contributed by Sunil G. 
(cherry picked from commit 28660f51af161a9fa301523d96a6f8ae4ebd6edd)
 2016-10-28 06:00:29 +05:30
Varun Saxena
f572448809 YARN-4388. Cleanup mapreduce.job.hdfs-servers from yarn-default.xml (Junping Du via Varun Saxena) 2016-10-28 02:24:24 +05:30
Naganarasimha
fe5c1f249a MAPREDUCE-6541. Exclude scheduled reducer memory when calculating available mapper slots from headroom to avoid deadlock. Contributed by Varun Saxena 
(cherry picked from commit 060558c6f221ded0b014189d5b82eee4cc7b576b)
 2016-10-27 18:05:00 +05:30
Varun Saxena
65fa9827fc YARN-5757. RM REST API documentation is not up to date (Miklos Szegedi via Varun Saxena) 2016-10-27 17:24:24 +05:30
Varun Saxena
14ce42a8ed MAPREDUCE-6798. Fix intermittent failure of TestJobHistoryParsing.testJobHistoryMethods (Haibo Chen via Varun Saxena) 2016-10-27 15:41:27 +05:30
Akira Ajisaka
e29d0a2fd8 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 
(cherry picked from commit d3bb69a66776e9f410150c4030ddb15981f58fb9)
 2016-10-26 15:34:40 +09:00
Junping Du
432657780d MAPREDUCE-6792. Allow user's full principal name as owner of MapReduce staging directory in JobSubmissionFiles#JobStagingDir(). Contributed by Santhosh G Nayak. 
(cherry picked from commit 9b7ce948a52f4ef433dc958ec891b669a669d7c2)
 2016-10-25 11:21:33 -07:00
Robert Kanter
c76eb90f0e MAPREDUCE-6728. Give fetchers hint when ShuffleHandler rejects a shuffling connection (haibochen via rkanter) 2016-10-24 17:47:09 -07:00
Robert Kanter
48ca6be76e MAPREDUCE-6776. yarn.app.mapreduce.client.job.max-retries should have a more useful default (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit f3f37e6fb8172f6434e06eb9a137c0c155b3952e)
 2016-10-07 14:47:23 -07:00
Akira Ajisaka
5f1432d98e MAPREDUCE-6789. Fix TestAMWebApp failure. Contributed by Daniel Templeton. 
(cherry picked from commit 272a21747e8a89b6daccc19b71c21de3d17b8d62)
 2016-10-06 15:59:19 +09:00
Karthik Kambatla
876fc2e61a MAPREDUCE-6638. Do not attempt to recover progress from previous job attempts if spill encryption is enabled. (Haibo Chen via kasha) 
(cherry picked from commit de7a0a92ca1983b35ca4beb7ab712fd700a9e6e0)
 2016-10-03 10:30:44 -07:00
Jason Lowe
c52ad9ee86 MAPREDUCE-6771. RMContainerAllocator sends container diagnostics event after corresponding completion event. Contributed by Haibo Chen 2016-09-29 15:50:50 +00:00
Robert Kanter
85724268c2 MAPREDUCE-6718. add progress log to JHS during startup (haibochen via rkanter) 
(cherry picked from commit 0d6778d800ff16366911e3b064f3af6162dee2e4)
 2016-09-28 16:04:48 -07:00
Wangda Tan
269401dc83 HADOOP-13544. JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs. (vinodkv via wangda) 
(cherry picked from commit 875062b5bc789158290bf93dadc71b5328ca4fee)
 2016-09-27 11:27:25 -07:00
Arun Suresh
7484d0b1b9 YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh 2016-09-26 23:55:13 +08:00
Naganarasimha
0e4a5611ff YARN-3692. Allow REST API to set a user generated message when killing an application. Contributed by Rohith Sharma K S 2016-09-24 21:13:01 +05:30
Robert Kanter
254041b71a MAPREDUCE-6632. Master.getMasterAddress() should be updated to use YARN-4629 (templedf via rkanter) 
(cherry picked from commit 4fc632ae19a1d6b0ec09cc7ead789a3cab1c2f1c)
 2016-09-22 16:13:24 -07:00