MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev

hadoop-mapreduce-project/CHANGES.txt

@@ -1,308 +1,5 @@
 Hadoop MapReduce Change Log
 
-Trunk (Unreleased)
-
-  INCOMPATIBLE CHANGES
-
-    MAPREDUCE-5785. Derive heap size or mapreduce.*.memory.mb automatically.
-    (Gera Shegalov and Karthik Kambatla via gera)
-
-    MAPREDUCE-5653. DistCp does not honour config-overrides for
-    mapreduce.[map,reduce].memory.mb (Ratandeep Ratti via aw)
-
-    MAPREDUCE-4424. 'mapred job -list' command should show the job name
-    as well. (Avinash Kujur via aajisaka)
-
-    MAPREDUCE-6336. Enable v2 FileOutputCommitter by default.
-    (Siqi Li via gera)
-
-  NEW FEATURES
-
-    MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
-
-    MAPREDUCE-2669. Add new examples for Mean, Median, and Standard Deviation.
-    (Plamen Jeliazkov via shv)
-
-    MAPREDUCE-4887. Add RehashPartitioner, to smooth distributions
-    with poor implementations of Object#hashCode().  (Radim Kolar via cutting)
-
-    MAPREDUCE-5232. Add a configuration to be able to log classpath and other
-    system properties on mapreduce JVMs startup.  (Sangjin Lee via vinodkv)
-
-    MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
-    RM-restart. (Rohith via jianhe)
-
-    MAPREDUCE-2841. Add a native implementation of MapOutputCollector.
-    (see section below for detailed breakdown)
-
-  IMPROVEMENTS
-
-    MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
-
-    MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
-                    job-history related information.
-
-    MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
-                    (Vinay Thota via amarrk)
-
-    MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
-                    (Vinay Thota via amarrk)
-
-    MAPREDUCE-2836. Provide option to fail jobs when submitted to non-existent
-    fair scheduler pools. (Ahmed Radwan via todd)
-
-    MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
-    native. (tucu)
-
-    MAPREDUCE-3149. Add a test to verify that TokenCache handles file system
-    uri with no authority. (John George via jitendra)
-
-    MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
-    client APIs cross MR1 and MR2 (Ahmed via tucu)
-
-    MAPREDUCE-2944. Improve checking of input for JobClient.displayTasks()
-    (XieXianshan via harsh)
-
-    MAPREDUCE-3956. Remove the use of the deprecated Syncable.sync() method from
-    TeraOutputFormat in the terasort example.  (szetszwo)
-
-    MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
-    (tomwhite)
-
-    HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
-
-    MAPREDUCE-3302. Remove the last dependency call from
-    org.apache.hadoop.record package in MR. (harsh)
-
-    MAPREDUCE-2384. The job submitter should make sure to validate
-    jobs before creation of necessary files. (harsh)
-
-    MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG
-    (madhukara phatak via bobby)
-
-    MAPREDUCE-4686. hadoop-mapreduce-client-core fails compilation in Eclipse 
-    due to missing Avro-generated classes (Chris Nauroth via harsh)
-
-    MAPREDUCE-4735. Make arguments in TestDFSIO case insensitive.
-    (Brandon Li via suresh)
-
-    MAPREDUCE-5197. Add a service for checkpointing task state.
-    (Carlo Curino via cdouglas)
-
-    MAPREDUCE-5189. Add policies and wiring to respond to preemption requests
-    from YARN. (Carlo Curino via cdouglas)
-
-    MAPREDUCE-5196. Add bookkeeping for managing checkpoints of task state.
-    (Carlo Curino via cdouglas)
-
-    MAPREDUCE-5912. Task.calculateOutputSize does not handle Windows files after
-    MAPREDUCE-5196. (Remus Rusanu via cnauroth)
-
-    MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
-    interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
-
-    MAPREDUCE-6013. [post-HADOOP-9902] mapred version is missing (Akira AJISAKA
-    via aw)
-
-    MAPREDUCE-6250. deprecate sbin/mr-jobhistory-daemon.sh (aw)
-
-    MAPREDUCE-6260. Convert site documentation to markdown (Masatake Iwasaki
-    via aw)
-
-    MAPREDUCE-6057. Remove obsolete entries from mapred-default.xml
-    (Ray Chiang via aw)
-
-    MAPREDUCE-2632. Avoid calling the partitioner when the numReduceTasks is 1.
-    (Ravi Teja Ch N V and Sunil G via kasha)
-
-    MAPREDUCE-6407. Migrate MAPREDUCE nativetask build to new CMake framework
-    (Alan Burlison via Colin P. McCabe)
-
-    MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API
-    (Junping Du via jianhe)
-
-  BUG FIXES
-
-    MAPREDUCE-6191. Improve clearing stale state of Java serialization
-                    testcase.  (Sam Liu via Eric Yang)
-
-    MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.  
-                     (Jinghui Wang via Eric Yang)
-
-    MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
-                     (Jason Lowe via bobby)
-
-    MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
-                    (Ravi Prakash and Ravi Gummadi via amarrk)
-
-    MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks.
-                    (Devaraj K and Amar Kamat via amarrk)
-
-    MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
-
-    MAPREDUCE-3346. [Rumen] LoggedTaskAttempt#getHostName() returns null.
-                    (amarrk)
-
-    MAPREDUCE-2950. [Gridmix] TestUserResolve fails in trunk.
-                    (Ravi Gummadi via amarrk)
-
-    MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
-    ResourceUsageMatcher. (amarrk)
-
-    MAPREDUCE-2978. Fixed test-patch to make Jenkins report correct number of
-    findBugs, correct links to findBugs artifacts and no links to the
-    artifacts when there are no warnings. (Tom White via vinodkv).
-
-    MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
-    (Brandon Li via jitendra)
-
-    MAPREDUCE-1740. NPE in getMatchingLevelForNodes when node locations are 
-    variable depth (ahmed via tucu) [IMPORTANT: this is dead code in trunk]
-
-    MAPREDUCE-3990. MRBench allows Long-sized input-lines value
-    but parses CLI argument as an Integer. (harsh)
-
-    MAPREDUCE-3868. Make Raid Compile. (Weiyan Wang via schen)
-
-    MAPREDUCE-4685. DBCount should not use ACCESS. (Viji via harsh)
-
-    MAPREDUCE-3223. Remove MR1 configs from mapred-default.xml (tlipcon via harsh)
-
-    MAPREDUCE-4695. Fix LocalRunner on trunk after MAPREDUCE-3223 broke it
-    (harsh)
-
-    MAPREDUCE-3914. Mismatched free() / delete / delete [] in HadoopPipes
-    (Joe Mudd via aw)
-
-    MAPREDUCE-1125. SerialUtils.cc: deserializeFloat is out of sync with
-    SerialUtils.hh (Simone Leo via aw)
-
-    MAPREDUCE-4574. Fix TotalOrderParitioner to work with
-    non-WritableComparable key types. (harsh)
-
-    MAPREDUCE-5012. Typo in javadoc for IdentityMapper class. (Adam Monsen
-    via suresh)
-
-    MAPREDUCE-4987. TestMRJobs#testDistributedCache fails on Windows due to
-    classpath problems and unexpected behavior of symlinks (Chris Nauroth via
-    bikas)
-
-    MAPREDUCE-5191. TestQueue#testQueue fails with timeout on Windows. (Ivan
-    Mitic via hitesh)
-
-    MAPREDUCE-5717. Task pings are interpreted as task progress (jlowe)
-
-    MAPREDUCE-5867. Fix NPE in KillAMPreemptionPolicy related to 
-    ProportionalCapacityPreemptionPolicy (Sunil G via devaraj)
-
-    MAPREDUCE-5972. Fix typo 'programatically' in job.xml (and a few other
-    places) (Akira AJISAKA via aw)
-
-    MAPREDUCE-6161. mapred hsadmin command missing from trunk (Allen Wittenauer
-    via jlowe)
-
-    MAPREDUCE-4413. MR lib dir contains jdiff (which is gpl) (Nemon Lou via aw)
-
-    MAPREDUCE-6234. TestHighRamJob fails due to the change in MAPREDUCE-5785. 
-    (Masatake Iwasaki via kasha)
-
-    MAPREDUCE-6343. JobConf.parseMaximumHeapSizeMB() fails to parse value 
-    greater than 2GB expressed in bytes. (Hao Xia via kasha)
-
-    MAPREDUCE-6396. TestPipeApplication fails by NullPointerException.
-    (Brahma Reddy Battula via aajisaka)
-
-    MAPREDUCE-6406. Update FileOutputCommitter.FILEOUTPUTCOMMITTER_ALGORITHM_VERSION_DEFAULT
-    to match mapred-default.xml. (Ray Chiang via devaraj)
-
-    MAPREDUCE-6257. Document encrypted spills (Bibin A Chundatt via aw)
-
-    MAPREDUCE-6391. util/Timer.cc completely misunderstands _POSIX_CPUTIME
-    (Alan Burlison via aw)
-
-    MAPREDUCE-6412. Make hadoop-mapreduce-client Native code -Wall-clean
-    (Alan Burlison via aw)
-
-    MAPREDUCE-6416. Not all platforms have d_type in struct dirent
-    (Alan Burlison via aw)
-
-    MAPREDUCE-6435. MapReduce client assumes the world is x86
-    (Alan Burlison via aw)
-
-    MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
-
-    MAPREDUCE-6555. TestMRAppMaster fails on trunk. (Junping Du via ozawa)
-
-  BREAKDOWN OF MAPREDUCE-2841 (NATIVE TASK) SUBTASKS
-
-    MAPREDUCE-5985. native-task: Fix build on macosx. Contributed by
-    Binglin Chang
-
-    MAPREDUCE-5994. Simplify ByteUtils and fix failing test. (todd)
-
-    MAPREDUCE-5996. native-task: Rename system tests into standard directory
-    layout (todd)
-
-    MAPREDUCE-5997. native-task: Use DirectBufferPool from Hadoop Common (todd)
-
-    MAPREDUCE-6000. native-task: Simplify ByteBufferDataReader/Writer (todd)
-
-    MAPREDUCE-5991. native-task should not run unit tests if native profile is
-    not enabled. (Binglin Chang)
-
-    MAPREDUCE-5995. native-task: Revert changes to Text internals (todd)
-
-    MAPREDUCE-6005. native-task: Fix some valgrind errors (Binglin Chang)
-
-    MAPREDUCE-5984. native-task: Reuse lz4 sources in hadoop-common (Binglin
-    Chang)
-
-    MAPREDUCE-5976. native-task: should not fail to build if snappy is missing
-    (Manu Zhang)
-
-    MAPREDUCE-5978. native-task: remove test case for not supported codec
-    Bzip2Codec and DefaultCodec (Manu Zhang)
-
-    MAPREDUCE-6006. native-task: add native tests to maven and fix bug in
-    pom.xml (Binglin Chang via todd)
-
-    MAPREDUCE-6026. native-task: fix logging (Manu Zhang via todd)
-
-    MAPREDUCE-6035. native-task: sources/test-sources jar distribution (Manu
-    Zhang via todd)
-
-    MAPREDUCE-5977. Fix or suppress native-task gcc warnings (Manu Zhang via
-    todd)
-
-    MAPREDUCE-6054. native-task: Speed up tests (todd)
-
-    MAPREDUCE-6058. native-task: KVTest and LargeKVTest should check mr job is
-    sucessful (Binglin Chang)
-
-    MAPREDUCE-6056. native-task: move system test working dir to target dir and
-    cleanup test config xml files (Manu Zhang via bchang)
-
-    MAPREDUCE-6055. native-task: findbugs, interface annotations, and other misc
-    cleanup (todd)
-
-    MAPREDUCE-6067. native-task: fix some counter issues (Binglin Chang)
-
-    MAPREDUCE-6069. native-task: Lint/style fixes and removal of unused code
-    (todd)
-
-    MAPREDUCE-6074. native-task: fix release audit, javadoc, javac warnings
-    (todd)
-
-    MAPREDUCE-6077. native-task: Remove CustomModule examples in nativetask (seanzhong)
-
-    MAPREDUCE-6078. native-task: fix gtest build on macosx (Binglin Chang)
-
-    MAPREDUCE-5801. Uber mode's log message is missing a vcore reason
-    (Steven Wong via aw)
-
-    MAPREDUCE-6525. Fix test failure of TestMiniMRClientCluster.testRestart.
-    (Masatake Iwasaki via aajisaka)
-
 Release 2.9.0 - UNRELEASED
 
   INCOMPATIBLE CHANGES
@@ -424,6 +121,9 @@ Release 2.8.0 - UNRELEASED
     MAPREDUCE-6384. Add the last reporting reducer info for too many fetch
     failure diagnostics (Chang Li via jlowe)
 
+    MAPREDUCE-5762. Port MAPREDUCE-3223 and MAPREDUCE-4695 (Remove MRv1 config
+    from mapred-default.xml) to branch-2. (aajisaka)
+
     MAPREDUCE-6443. Add JvmPauseMonitor to JobHistoryServer. (Robert Kanter
     via junping_du)
 
@@ -445,6 +145,9 @@ Release 2.8.0 - UNRELEASED
     MAPREDUCE-5763. Warn message about httpshuffle in NM logs.
     (Akira AJISAKA via ozawa)
 
+    MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter
+    API. Contributed by Junping Du
+
     MAPREDUCE-6499. Add elapsed time for retired job in JobHistoryServer WebUI.
     (Lin Yiqun via aajisaka)
 
@@ -563,6 +266,9 @@ Release 2.8.0 - UNRELEASED
     MAPREDUCE-6382. Don't escape HTML links in Diagnostics in JHS job overview.
     (Siqi Li via gera)
 
+    MAPREDUCE-5232. Add a configuration to be able to log classpath and other
+    system properties on mapreduce JVMs startup.  (Sangjin Lee via vinodkv)
+
     MAPREDUCE-6388. Remove deprecation warnings from JobHistoryServer classes
     (Ray Chiang via ozawa).
 
@@ -656,6 +362,8 @@ Release 2.8.0 - UNRELEASED
     MAPREDUCE-6533. testDetermineCacheVisibilities of
     TestClientDistributedCacheManager is broken (Chang Li via jlowe)
 
+    MAPREDUCE-6540. TestMRTimelineEventHandling fails (sjlee)
+
     MAPREDUCE-6553. Replace '\u2b05' with '<-' in rendering job configuration.
     (Gabor Liptak via aajisaka)
 
@@ -672,6 +380,7 @@ Release 2.7.3 - UNRELEASED
 
   INCOMPATIBLE CHANGES
 
+
   NEW FEATURES
 
   IMPROVEMENTS
@@ -747,6 +456,8 @@ Release 2.7.2 - UNRELEASED
     MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when
     strategy is dynamic (Kuhu Shukla via kihwal)
 
+    MAPREDUCE-6566. Add retry support to mapreduce CLI tool. (Varun Vasudev via xgong)
+
 Release 2.7.1 - 2015-07-06 
 
   INCOMPATIBLE CHANGES
@@ -941,17 +652,14 @@ Release 2.7.0 - 2015-04-20
     MAPREDUCE-2815. JavaDoc does not generate correctly for
     MultithreadedMapRunner. (Chris Palmer via aajisaka)
 
-    MAPREDUCE-6223. TestJobConf#testNegativeValueForTaskVmem failures. 
-    (Varun Saxena via kasha)
-
     MAPREDUCE-6268. Fix typo in Task Attempt API's URL. (Ryu Kobayashi
     via ozawa)
 
     MAPREDUCE-6136. MRAppMaster doesn't shutdown file systems. (Brahma 
     Reddy Battula via ozawa)
 
-    MAPREDUCE-5657. Fix Javadoc errors caused by incorrect or illegal tags in doc
+    MAPREDUCE-5657. [JDK8] Fix Javadoc errors caused by incorrect or illegal
-    comments. (Akira AJISAKA and Andrew Purtell via ozawa)
+    tags in doc comments. (Akira AJISAKA and Andrew Purtell via ozawa)
 
     MAPREDUCE-4742. Fix typo in nnbench#displayUsage. (Liang Xie via ozawa)
 
@@ -1081,6 +789,8 @@ Release 2.6.0 - 2014-11-18
   INCOMPATIBLE CHANGES
 
   NEW FEATURES
+    MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
+    RM-restart. Contributed by Rohith
 
     MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server.
     (Robert Kanter via zjshen)
@@ -1093,6 +803,9 @@ Release 2.6.0 - 2014-11-18
     MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with
     compatible/incompatible changes (Junping Du via jlowe)
 
+    MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
+    interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
+
     MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and
       Koji Noguchi via aw)
 
@@ -1148,12 +861,6 @@ Release 2.6.0 - 2014-11-18
     MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH
     (jlowe)
 
-    MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
-    (jlowe)
-
-    MAPREDUCE-5878. some standard JDK APIs are not part of system classes
-    defaults (Sangjin Lee via jlowe)
-
     MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm
       (Akira AJISAKA via aw)
 
@@ -1163,6 +870,9 @@ Release 2.6.0 - 2014-11-18
     MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus 
       and getStatus (Akira AJISAKA via aw)
 
+    MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
+    (jlowe)
+
     MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw)
 
     MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors 
@@ -1176,6 +886,9 @@ Release 2.6.0 - 2014-11-18
 
     MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw)
 
+    MAPREDUCE-5878. some standard JDK APIs are not part of system classes
+    defaults (Sangjin Lee via jlowe)
+
     MAPREDUCE-6032. Made MR jobs write job history files on the default FS when
     the current context's FS is different. (Benjamin Zhitomirsky via zjshen)
 
@@ -1499,6 +1212,9 @@ Release 2.4.1 - 2014-06-23
 
   BUG FIXES
 
+    MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.  
+                     (Jinghui Wang via Eric Yang)
+
     MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen)
 
     MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in
@@ -1560,9 +1276,6 @@ Release 2.4.0 - 2014-04-07
     MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via
     jlowe)
 
-    MAPREDUCE-5766. Moved ping messages from TaskAttempts to be at DEBUG level
-    inside the ApplicationMaster log. (Jian He via vinodkv)
-
     MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera
     Shegalov via jlowe)
 
@@ -2677,9 +2390,6 @@ Release 2.0.3-alpha - 2013-02-06
 
     MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite)
 
-    MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO.
-    (Brandon Li via suresh)
-
     MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and 
     UNASSIGNED states. (Mayank Bansal via sseth)
 
@@ -2697,6 +2407,9 @@ Release 2.0.3-alpha - 2013-02-06
     MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
     TestJobCleanup. (Sandy Ryza via tomwhite)
 
+    MAPREDUCE-4895. Fix compilation failure of org.apache.hadoop.mapred.
+    gridmix.TestResourceUsageEmulators (Dennis Y via tgraves)
+
     MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
     gateway. (Sandy Ryza via tomwhite)
 
@@ -2768,23 +2481,11 @@ Release 2.0.2-alpha - 2012-09-07
     MAPREDUCE-3921. MR AM should act on node health status changes. 
     (Bikas Saha via sseth)
 
-    MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
+    MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
-    mapred-default.xml (Rui Kubo via harsh)
-
-    MAPREDUCE-3907. Document entries mapred-default.xml for the
-    jobhistory server. (Eugene Koontz via harsh)
-
-    MAPREDUCE-3906. Fix inconsistency in documentation regarding
-    mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
-
-    MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
-    is not used. (Gabriel Reid via harsh)
 
     MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease
     development of new applications. (Bikas Saha via acmurthy) 
 
-    MAPREDUCE-4447. Remove aop from cruft from the ant build. (eli)
-
     MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler.
     (Todd Lipcon and Siddharth Seth via sseth)
 
@@ -2830,7 +2531,7 @@ Release 2.0.2-alpha - 2012-09-07
     classpath (rkanter via tucu)
 
     MAPREDUCE-4577. HDFS-3672 broke
-    TestCombineFileInputFormat.testMissingBlocks() test. (atm)
+    TestCombineFileInputFormat.testMissingBlocks() test (atm)
 
     MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu)
 
@@ -2946,6 +2647,9 @@ Release 2.0.0-alpha - 05-23-2012
     MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in 
     MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth)
 
+	  MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
+    (tomwhite)
+
     MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files 
     splitting. (harsh)
 
@@ -2978,6 +2682,20 @@ Release 2.0.0-alpha - 05-23-2012
     MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager 
     (tucu)
 
+    HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
+
+    MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
+    mapred-default.xml (Rui Kubo via harsh)
+
+    MAPREDUCE-3907. Document entries mapred-default.xml for the
+    jobhistory server. (Eugene Koontz via harsh)
+
+    MAPREDUCE-3906. Fix inconsistency in documentation regarding
+    mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
+
+    MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
+    is not used. (Gabriel Reid via harsh)
+
   OPTIMIZATIONS
 
   BUG FIXES
@@ -3034,9 +2752,6 @@ Release 2.0.0-alpha - 05-23-2012
 
     MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu)
 
-    MAPREDUCE-4095. TestJobInProgress#testLocality uses a bogus topology.
-    (Colin Patrick McCabe via eli)
-
     MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu)
 
     MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath 
@@ -3851,6 +3566,9 @@ Release 0.23.3
     MAPREDUCE-4641. Exception in commitJob marks job as successful in job
     history (Jason Lowe via bobby)
 
+    MAPREDUCE-4549. Distributed cache conflicts breaks backwards compatability 
+    (Robert Evans via tucu)
+
 Release 0.23.2 - UNRELEASED
 
   INCOMPATIBLE CHANGES
@@ -3910,6 +3628,7 @@ Release 0.23.2 - UNRELEASED
     sseth)
 
   BUG FIXES
+
     MAPREDUCE-3918  proc_historyserver no longer in command line arguments for
     HistoryServer (Jon Eagles via bobby)
 
@@ -3922,9 +3641,6 @@ Release 0.23.2 - UNRELEASED
     MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas 
     Graves via mahadev)
 
-    MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
-    causes jobs to fail (ahmed via tucu).
-
     MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode
     HTTPS parameters. (todd)
 
@@ -4038,20 +3754,39 @@ Release 0.23.2 - UNRELEASED
 
 Release 0.23.1 - 2012-02-17
 
-  INCOMPATIBLE CHANGES
-
   NEW FEATURES
 
+    MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
+
     MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev)
 
     MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas
     Graves via vinodkv)
 
-   MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster
+    MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster.
     (Anupam Seth via mahadev)
 
+    MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
+
   IMPROVEMENTS
 
+    MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
+
+    MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the 
+                    job-history related information.
+
+    MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. 
+                    (Vinay Thota via amarrk)
+
+    MAPREDUCE-3840.  JobEndNotifier doesn't use the proxyToUse during connecting
+    (Ravi Prakash via bobby)
+
+    MAPREDUCE-3736. Variable substitution depth too large for fs.default.name 
+    causes jobs to fail (ahmed via tucu).
+ 
+    MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests. 
+                    (Vinay Thota via amarrk)
+
     MAPREDUCE-3297. Moved log related components into yarn-common so that
     HistoryServer and clients can use them without depending on the
     yarn-server-nodemanager module. (Siddharth Seth via vinodkv)
@@ -4080,8 +3815,11 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor
     has wrong configuration or permissions. (Hitesh Shah via vinodkv)
 
-    MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR resolution.
+    MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR
-    (tucu)
+    resolution. (tucu)
+
+    MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides 
+    client APIs cross MR1 and MR2. (Ahmed via tucu)
 
     MAPREDUCE-3373. Hadoop scripts unconditionally source
     "$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
@@ -4222,6 +3960,22 @@ Release 0.23.1 - 2012-02-17
 
   BUG FIXES
 
+    MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and 
+                    ResourceUsageMatcher. (amarrk)
+
+    MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
+                     (Jason Lowe via bobby)
+
+    MAPREDUCE-3462. Fix Gridmix JUnit testcase failures. 
+                    (Ravi Prakash and Ravi Gummadi via amarrk)
+
+    MAPREDUCE-2950. [Rumen] Fixed TestUserResolve. (Ravi Gummadi via amarrk)
+
+    MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
+
+    MAPREDUCE-3346 [Rumen] LoggedTaskAttempt#getHostName() returns null.
+                   (amarrk)
+
     MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob
     and fixed bugs in it. (Devaraj K via vinodkv)
 
@@ -4246,6 +4000,8 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing
     TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv)
 
+    MAPREDUCE-3368. Fixed test compilation. (Hitesh Shah via vinodkv)
+
     MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to
     which per-container connections to NodeManager were lingering long enough
     to hit the ulimits on number of processes. (vinodkv)
@@ -4265,6 +4021,9 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed
     by the integration tests. (Hitesh Shah via vinodkv)
 
+    MAPREDUCE-3437. Fix examples pom to refer to the correct 0.23 snapshot
+    version. (Jonathan Eagles via todd)
+
     MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev)
 
     MAPREDUCE-3447. mapreduce examples not working (mahadev)
@@ -4285,12 +4044,18 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to
     cut down noise. (acmurthy) 
 
+    MAPREDUCE-3468. Changed ant based infrastructure to use 0.23.1 version.
+    (sseth via acmurthy) 
+
     MAPREDUCE-3433. Finding counters by legacy group name returns empty
     counters. (tomwhite)
 
     MAPREDUCE-3450. NM port info no longer available in JobHistory.
     (Siddharth Seth via mahadev)
 
+    MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. 
+    (jeagles via tucu)
+
     MAPREDUCE-3488. Streaming jobs are failing because the main class
     isnt set in the pom files. (mahadev)
  
@@ -4311,6 +4076,8 @@ Release 0.23.1 - 2012-02-17
 
     MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite)
 
+    MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
+
     MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for 
     $HADOOP_*_HOME (Eric Payne via mahadev)
 
@@ -4320,11 +4087,14 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as 
     ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev)
 
+    MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with 
+    classpath from the build machine. (tucu)
+
     MAPREDUCE-3496. Fixed client to print queue acls in consistent order.
     (Jonathan Eagles via acmurthy) 
 
-    MAPREDUCE-3147. Handle leaf queues with the same name properly. (Ravi Prakash via
+    MAPREDUCE-3147. Handle leaf queues with the same name properly.
-    mahadev)
+    (Ravi Prakash via mahadev)
 
     MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value 
     for queues (Anupam Seth via mahadev)
@@ -4354,6 +4124,12 @@ Release 0.23.1 - 2012-02-17
     in the correct directory to work properly in secure mode. (Hitesh Shah via
     vinodkv)
 
+    MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be 
+    added as ivy dependency. (tucu)
+
+    MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives 
+    dependency. (tucu)
+
     MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via 
     mahadev)
 
@@ -4403,6 +4179,9 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that
     yarn daemons can start. (Arun C Murthy via vinodkv)
 
+    MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. (Amar
+    Kamat and Devaraj K via sseth)
+
     MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations
     thereby solving race conditions in MR AM shutdown. (vinodkv)
 
@@ -4421,9 +4200,6 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce
     ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv)
 
-    MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
-    adding a file/archive to the path. (Dick King via tucu)
-
     MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly
     (sseth)
 
@@ -4438,6 +4214,9 @@ Release 0.23.1 - 2012-02-17
 
     MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth)
 
+    MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when 
+    adding a file/archive to the path. (Dick King via tucu)
+
     MAPREDUCE-3326. Added detailed information about queue's to the
     CapacityScheduler web-ui. (Jason Lowe via acmurthy) 
 
@@ -4448,29 +4227,6 @@ Release 0.23.1 - 2012-02-17
     yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan
     Eagles via acmurthy) 
 
-    MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
-    missing license header. (Hitesh Shah via tucu).
-
-    MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
-
-    MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
-
-    MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
-    by default. (tucu)
-
-    MAPREDUCE-3477. Hadoop site documentation cannot be built anymore. (jeagles via tucu)
-
-    MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
-
-    MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
-    classpath from the build machine. (tucu)
-
-    MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be added as
-    ivy dependency. (tucu)
-
-    MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives dependency.
-    (tucu)
-
     MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev
     via acmurthy)
 
@@ -4507,6 +4263,9 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable
     speculating either maps or reduces. (Eric Payne via vinodkv)
 
+    MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
+    (Brandon Li via jitendra)
+
     MAPREDUCE-3649. Job End notification gives an error on calling back.
     (Ravi Prakash via mahadev)
 
@@ -4531,6 +4290,9 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3689. RM web UI doesn't handle newline in job name.
     (Thomas Graves via mahadev)
 
+    MAPREDUCE-3701. Delete HadoopYarnRPC from 0.23 branch.
+    (mahadev)
+
     MAPREDUCE-3549. write api documentation for web service apis for RM, NM, 
     mapreduce app master, and job history server (Thomas Graves via mahadev)
 
@@ -4691,9 +4453,6 @@ Release 0.23.1 - 2012-02-17
     MAPREDUCE-3697. Support binary compatibility for Counters after
     MAPREDUCE-901. (mahadev via acmurthy) 
 
-    MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
-    mahadev)
-
     MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive
     jobs. (Arpit Gupta via acmurthy) 
 
@@ -4732,9 +4491,6 @@ Release 0.23.1 - 2012-02-17
 
     MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk)
 
-    MAPREDUCE-3840.  JobEndNotifier doesn't use the proxyToUse during connecting
-    (Ravi Prakash via bobby)
-
     MAPREDUCE-3843. Job summary log file found missing on the RM host 
     (Anupam Seth via tgraves)
 
@@ -5137,6 +4893,12 @@ Release 0.23.0 - 2011-11-01
     MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the
     JobHistoryServer. (Siddharth Seth via vinodkv)
 
+    MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off 
+    by default. (tucu)
+
+    MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
+    native. (tucu)
+
     MAPREDUCE-3146. Added a MR specific command line to dump logs for a
     given TaskAttemptID. (Siddharth Seth via vinodkv)
 
@@ -6306,6 +6068,9 @@ Release 0.23.0 - 2011-11-01
     MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root
     of the install properly. (Xie Xianshan via acmurthy) 
 
+    MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. (Hitesh Shah via
+    acmurthy) 
+
     MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames.
     (vinodkv via acmurthy) 
 
@@ -6413,9 +6178,6 @@ Release 0.23.0 - 2011-11-01
 
     MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev)
 
-    MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
-    tests. (vinodkv)
-
     MAPREDUCE-3162. Separated application-init and container-init event types
     in NodeManager's Application state machine. (Todd Lipcon via vinodkv)
 
@@ -6470,8 +6232,6 @@ Release 0.23.0 - 2011-11-01
     MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application
     directories during app-init. (todd via acmurthy)
 
-    MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
-
     MAPREDUCE-2746. Yarn servers can't communicate with each other with 
     hadoop.security.authorization set to true (acmurthy via mahadev)
 
@@ -6487,9 +6247,6 @@ Release 0.23.0 - 2011-11-01
     MAPREDUCE-3269. Fixed log4j properties to correctly set logging options
     for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy) 
 
-    MAPREDUCE-2977. Fix ResourceManager to renew HDFS delegation tokens for
-    applications. (acmurthy)
-
     MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM 
     and prints a lots of logs. (vinodkv via mahadev)
 
@@ -6513,7 +6270,7 @@ Release 0.23.0 - 2011-11-01
 
     MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its
     containers (process-trees) irrespective of whether the container succeeded,
-    or killed. (Hitesh Shah via vinodkv)
+    or killed. Contributed by Hitesh Shah.
 
     MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv)
 
@@ -6530,7 +6287,7 @@ Release 0.23.0 - 2011-11-01
     process goes down and comes up while job is getting executed. 
     (Eric Payne via mahadev)
 
-    MAPREDUCE-3209. Jenkins reports 160 FindBugs warnings (mahadev)
+    MAPREDUCE-3285. Tests on branch-0.23 failing (Siddharth Seth via mahadev)
 
     MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly.
     (Siddharth Seth via acmurthy)
@@ -6555,12 +6312,28 @@ Release 0.23.0 - 2011-11-01
     MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing
     NodeManager to crash. (vinodkv)
 
+    MAPREDUCE-3295. TestAMAuthorization failing on branch 0.23. (vinodkv via mahadev)
+
+    MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
+    missing license header. (Hitesh Shah via tucu).
+
+    MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
+
+    MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
+    tests. (vinodkv)
+
+    MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
+
+    MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
+
     MAPREDUCE-3256. Added authorization checks for the protocol between
     NodeManager and ApplicationMaster. (vinodkv via acmurthy) 
 
     MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a
     task-scheduling deadlock. (Robert Joseph Evans via vinodkv)
 
+    MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
+
     MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing
     TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv)
 
@@ -6588,10 +6361,10 @@ Release 0.23.0 - 2011-11-01
     MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk)
 
     MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying
-    on current history file name format. (Ravi Gummadi)
+    on current history file name format. (Ravi Gummadi via amarrk)
 
     MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file
-    names also. (Ravi Gummadi)
+                    names also. (Ravi Gummadi via amarrk)
 
     MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas).
 
@@ -7217,9 +6990,6 @@ Release 0.22.0 - 2011-11-29
     MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity 
     scheduler if queue names contain special symbols. (Priyo Mustafi via shv)
 
-    MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
-    (Ming Ma via shv)
-
     MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via
     acmurthy) 
 
@@ -7265,6 +7035,9 @@ Release 0.21.1 - Unreleased
     MAPREDUCE-2127. mapreduce trunk builds are filing on hudson. 
     (Bruno Mahé via eli)
 
+    MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
+    (Ming Ma via shv)
+
 Release 0.21.0 - 2010-08-13
 
   INCOMPATIBLE CHANGES

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java

@@ -26,6 +26,7 @@ import java.util.Set;
 import java.util.HashSet;
 import java.util.Arrays;
 
+import com.google.common.annotations.VisibleForTesting;
 import org.apache.commons.lang.StringUtils;
 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
@@ -43,6 +44,7 @@ import org.apache.hadoop.mapreduce.Job;
 import org.apache.hadoop.mapreduce.JobID;
 import org.apache.hadoop.mapreduce.JobPriority;
 import org.apache.hadoop.mapreduce.JobStatus;
+import org.apache.hadoop.mapreduce.MRJobConfig;
 import org.apache.hadoop.mapreduce.TaskAttemptID;
 import org.apache.hadoop.mapreduce.TaskCompletionEvent;
 import org.apache.hadoop.mapreduce.TaskReport;
@@ -268,7 +270,7 @@ public class CLI extends Configured implements Tool {
         System.out.println("Created job " + job.getJobID());
         exitCode = 0;
       } else if (getStatus) {
-        Job job = cluster.getJob(JobID.forName(jobid));
+        Job job = getJob(JobID.forName(jobid));
         if (job == null) {
           System.out.println("Could not find job " + jobid);
         } else {
@@ -283,7 +285,7 @@ public class CLI extends Configured implements Tool {
           exitCode = 0;
         }
       } else if (getCounter) {
-        Job job = cluster.getJob(JobID.forName(jobid));
+        Job job = getJob(JobID.forName(jobid));
         if (job == null) {
           System.out.println("Could not find job " + jobid);
         } else {
@@ -299,7 +301,7 @@ public class CLI extends Configured implements Tool {
           }
         }
       } else if (killJob) {
-        Job job = cluster.getJob(JobID.forName(jobid));
+        Job job = getJob(JobID.forName(jobid));
         if (job == null) {
           System.out.println("Could not find job " + jobid);
         } else {
@@ -323,7 +325,7 @@ public class CLI extends Configured implements Tool {
           }
         }
       } else if (setJobPriority) {
-        Job job = cluster.getJob(JobID.forName(jobid));
+        Job job = getJob(JobID.forName(jobid));
         if (job == null) {
           System.out.println("Could not find job " + jobid);
         } else {
@@ -339,7 +341,7 @@ public class CLI extends Configured implements Tool {
         viewHistory(historyFile, viewAllHistory);
         exitCode = 0;
       } else if (listEvents) {
-        listEvents(cluster.getJob(JobID.forName(jobid)), fromEvent, nEvents);
+        listEvents(getJob(JobID.forName(jobid)), fromEvent, nEvents);
         exitCode = 0;
       } else if (listJobs) {
         listJobs(cluster);
@@ -354,11 +356,11 @@ public class CLI extends Configured implements Tool {
         listBlacklistedTrackers(cluster);
         exitCode = 0;
       } else if (displayTasks) {
-        displayTasks(cluster.getJob(JobID.forName(jobid)), taskType, taskState);
+        displayTasks(getJob(JobID.forName(jobid)), taskType, taskState);
         exitCode = 0;
       } else if(killTask) {
         TaskAttemptID taskID = TaskAttemptID.forName(taskid);
-        Job job = cluster.getJob(taskID.getJobID());
+        Job job = getJob(taskID.getJobID());
         if (job == null) {
           System.out.println("Could not find job " + jobid);
         } else if (job.killTask(taskID, false)) {
@@ -370,7 +372,7 @@ public class CLI extends Configured implements Tool {
         }
       } else if(failTask) {
         TaskAttemptID taskID = TaskAttemptID.forName(taskid);
-        Job job = cluster.getJob(taskID.getJobID());
+        Job job = getJob(taskID.getJobID());
         if (job == null) {
             System.out.println("Could not find job " + jobid);
         } else if(job.killTask(taskID, true)) {
@@ -532,6 +534,29 @@ public class CLI extends Configured implements Tool {
     return (baseUrl + "/tasklog?plaintext=true&attemptid=" + taskId); 
   }
 
+  @VisibleForTesting
+  Job getJob(JobID jobid) throws IOException, InterruptedException {
+
+    int maxRetry = getConf().getInt(MRJobConfig.MR_CLIENT_JOB_MAX_RETRIES,
+        MRJobConfig.DEFAULT_MR_CLIENT_JOB_MAX_RETRIES);
+    long retryInterval = getConf()
+        .getLong(MRJobConfig.MR_CLIENT_JOB_RETRY_INTERVAL,
+            MRJobConfig.DEFAULT_MR_CLIENT_JOB_RETRY_INTERVAL);
+    Job job = cluster.getJob(jobid);
+
+    for (int i = 0; i < maxRetry; ++i) {
+      if (job != null) {
+        return job;
+      }
+      LOG.info("Could not obtain job info after " + String.valueOf(i + 1)
+          + " attempt(s). Sleeping for " + String.valueOf(retryInterval / 1000)
+          + " seconds and retrying.");
+      Thread.sleep(retryInterval);
+      job = cluster.getJob(jobid);
+    }
+    return job;
+  }
+  
 
   /**
    * Dump a list of currently running jobs

hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java

@@ -20,14 +20,19 @@ package org.apache.hadoop.mapreduce.tools;
 import static org.junit.Assert.*;
 
 import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
 import org.apache.hadoop.mapreduce.Cluster;
 import org.apache.hadoop.mapreduce.Job;
 import org.apache.hadoop.mapreduce.JobID;
+import org.apache.hadoop.mapreduce.MRJobConfig;
 import org.apache.hadoop.mapreduce.TaskReport;
 import org.apache.hadoop.mapreduce.TaskType;
 import org.apache.hadoop.mapreduce.JobPriority;
 import org.apache.hadoop.mapreduce.JobStatus;
 import org.apache.hadoop.mapreduce.JobStatus.State;
+import org.apache.hadoop.util.Time;
+import org.junit.Assert;
 import org.junit.Test;
 import static org.mockito.Mockito.mock;
 import static org.mockito.Mockito.when;
@@ -44,7 +49,7 @@ public class TestCLI {
     JobID jobId = JobID.forName(jobIdStr);
     Cluster mockCluster = mock(Cluster.class);
     Job job = mock(Job.class);
-    CLI cli = spy(new CLI());
+    CLI cli = spy(new CLI(new Configuration()));
 
     doReturn(mockCluster).when(cli).createCluster();
     when(job.getTaskReports(TaskType.MAP)).thenReturn(
@@ -112,7 +117,7 @@ public class TestCLI {
   @Test
   public void testJobKIll() throws Exception {
     Cluster mockCluster = mock(Cluster.class);
-    CLI cli = spy(new CLI());
+    CLI cli = spy(new CLI(new Configuration()));
     doReturn(mockCluster).when(cli).createCluster();
     String jobId1 = "job_1234654654_001";
     String jobId2 = "job_1234654654_002";
@@ -149,4 +154,26 @@ public class TestCLI {
     when(mockJob.getStatus()).thenReturn(status);
     return mockJob;
   }
+
+  @Test
+  public void testGetJob() throws Exception {
+    Configuration conf = new Configuration();
+    long sleepTime = 100;
+    conf.setLong(MRJobConfig.MR_CLIENT_JOB_RETRY_INTERVAL, sleepTime);
+    Cluster mockCluster = mock(Cluster.class);
+    JobID jobId1 = JobID.forName("job_1234654654_001");
+    when(mockCluster.getJob(jobId1)).thenReturn(null);
+
+    for (int i = 0; i < 2; ++i) {
+      conf.setInt(MRJobConfig.MR_CLIENT_JOB_MAX_RETRIES, i);
+      CLI cli = spy(new CLI(conf));
+      cli.cluster = mockCluster;
+      doReturn(mockCluster).when(cli).createCluster();
+      long start = Time.monotonicNow();
+      cli.getJob(jobId1);
+      long end = Time.monotonicNow();
+      Assert.assertTrue(end - start > (i * sleepTime));
+      Assert.assertTrue(end - start < ((i + 1) * sleepTime));
+    }
+  }
 }