7687 lines
285 KiB
Plaintext
7687 lines
285 KiB
Plaintext
Hadoop MapReduce Change Log
|
|
|
|
Release 2.6.0 - UNRELEASED
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
MAPREDUCE-5910. Make MR AM resync with RM in case of work-preserving
|
|
RM-restart. Contributed by Rohith
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5971. Move the default options for distcp -p to
|
|
DistCpOptionSwitch. (clamb via wang)
|
|
|
|
MAPREDUCE-5963. ShuffleHandler DB schema should be versioned with
|
|
compatible/incompatible changes (Junping Du via jlowe)
|
|
|
|
MAPREDUCE-6019. MapReduce changes for exposing YARN/MR endpoints on multiple
|
|
interfaces. (Craig Welch, Milan Potocnik, Arpit Agarwal via xgong)
|
|
|
|
MAPREDUCE-883. harchive: Document how to unarchive (Akira AJISAKA and
|
|
Koji Noguchi via aw)
|
|
|
|
MAPREDUCE-4791. Javadoc for KeyValueTextInputFormat should include default
|
|
separator and how to change it (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5906. Inconsistent configuration in property
|
|
"mapreduce.reduce.shuffle.input.buffer.percent" (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5974. Allow specifying multiple MapOutputCollectors with
|
|
fallback. (Todd Lipcon via kasha)
|
|
|
|
MAPREDUCE-5130. Add missing job config options to mapred-default.xml
|
|
(Ray Chiang via Sandy Ryza)
|
|
|
|
MAPREDUCE-5891. Improved shuffle error handling across NM restarts
|
|
(Junping Du via jlowe)
|
|
|
|
MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN
|
|
scheduler resource type is memory plus cpu. (Peng Zhang and Varun Vasudev
|
|
via zjshen)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5866. TestFixedLengthInputFormat fails in windows.
|
|
(Varun Vasudev via cnauroth)
|
|
|
|
MAPREDUCE-5956. Made MR AM not use maxAttempts to determine if the current
|
|
attempt is the last retry. (Wangda Tan via zjshen)
|
|
|
|
MAPREDUCE-5957. AM throws ClassNotFoundException with job classloader
|
|
enabled if custom output format/committer is used (Sangjin Lee via jlowe)
|
|
|
|
MAPREDUCE-5756. CombineFileInputFormat.getSplits() including directories
|
|
in its results (Jason Dere via jlowe)
|
|
|
|
MAPREDUCE-6014. New task status field in task attempts table can lead to
|
|
an empty web page (Mit Desai via jlowe)
|
|
|
|
MAPREDUCE-6021. MR AM should have working directory in LD_LIBRARY_PATH
|
|
(jlowe)
|
|
|
|
MAPREDUCE-5944. Remove MRv1 commands from CommandsManual.apt.vm
|
|
(Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5943. Separate mapred commands from CommandManual.apt.vm
|
|
(Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5363. Fix doc and spelling for TaskCompletionEvent#getTaskStatus
|
|
and getStatus (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-6010. HistoryServerFileSystemStateStore fails to update tokens
|
|
(jlowe)
|
|
|
|
MAPREDUCE-5595. Typo in MergeManagerImpl.java (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5597. Missing alternatives in javadocs for deprecated constructors
|
|
in mapreduce.Job (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5950. incorrect description in distcp2 document (Akira AJISAKA
|
|
via aw)
|
|
|
|
MAPREDUCE-5998. CompositeInputFormat javadoc is broken (Akira AJISAKA via
|
|
aw)
|
|
|
|
MAPREDUCE-5999. Fix dead link in InputFormat javadoc (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5878. some standard JDK APIs are not part of system classes
|
|
defaults (Sangjin Lee via jlowe)
|
|
|
|
MAPREDUCE-6032. Made MR jobs write job history files on the default FS when
|
|
the current context's FS is different. (Benjamin Zhitomirsky via zjshen)
|
|
|
|
MAPREDUCE-6024. Shortened the time when Fetcher is stuck in retrying before
|
|
concluding the failure by configuration. (Yunjiong Zhao via zjshen)
|
|
|
|
MAPREDUCE-6036. TestJobEndNotifier fails intermittently in branch-2 (chang
|
|
li via jlowe)
|
|
|
|
MAPREDUCE-6012. DBInputSplit creates invalid ranges on Oracle.
|
|
(Wei Yan via kasha)
|
|
|
|
MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir
|
|
creation on Windows. (zjshen)
|
|
|
|
MAPREDUCE-5885. build/test/test.mapred.spill causes release audit warnings
|
|
(Chen He via jlowe)
|
|
|
|
BREAKDOWN OF HDFS-6134 AND HADOOP-10150 SUBTASKS AND RELATED JIRAS
|
|
|
|
MAPREDUCE-5890. Support for encrypting Intermediate
|
|
data and spills in local filesystem. (asuresh via tucu)
|
|
|
|
MAPREDUCE-6007. Add support to distcp to preserve raw.* namespace
|
|
extended attributes. (clamb)
|
|
|
|
MAPREDUCE-6041. Fix TestOptionsParser. (clamb)
|
|
--
|
|
|
|
MAPREDUCE-6051. Fix typos in log messages. (Ray Chiang via cdouglas)
|
|
|
|
MAPREDUCE-5931. Validate SleepJob command line parameters (Gera Shegalov
|
|
via jlowe)
|
|
|
|
MAPREDUCE-6063. Correct spill size calculation for spills wrapping the
|
|
circular buffer. (zhihai xu via cdouglas)
|
|
|
|
MAPREDUCE-6071. JobImpl#makeUberDecision doesn't log that Uber mode is
|
|
disabled because of too much CPUs (Tsuyoshi OZAWA via jlowe)
|
|
|
|
MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length
|
|
files (jlowe)
|
|
|
|
MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but
|
|
is not documented (Tsuyoshi OZAWA via jlowe)
|
|
|
|
MAPREDUCE-6090. mapred hsadmin getGroups fails to connect in some cases
|
|
(Robert Kanter via jlowe)
|
|
|
|
MAPREDUCE-6086. mapreduce.job.credentials.binary should allow all URIs.
|
|
(Zhihai Xu via kasha)
|
|
|
|
MAPREDUCE-6091. YARNRunner.getJobStatus() fails with
|
|
ApplicationNotFoundException if the job rolled off the RM view (Sangjin
|
|
Lee via jlowe)
|
|
|
|
MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs (Gera Shegalov
|
|
via jlowe)
|
|
|
|
MAPREDUCE-6104. TestJobHistoryParsing.testPartialJob fails in branch-2
|
|
(Mit Desai via jlowe)
|
|
|
|
MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb
|
|
via aw)
|
|
|
|
MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw)
|
|
|
|
MAPREDUCE-5831. Make MR client ignore unknown counters received from AM.
|
|
(Junping Du via zjshen)
|
|
|
|
MAPREDUCE-6073. Description of mapreduce.job.speculative.slowtaskthreshold
|
|
in mapred-default should be moved into description tags (Tsuyoshi OZAWA
|
|
via aw)
|
|
|
|
MAPREDUCE-5796. Use current version of the archive name in
|
|
DistributedCacheDeploy document (Akira AJISAKA via aw)
|
|
|
|
MAPREDUCE-5945. Update the description of GenericOptionsParser -jt
|
|
option (Akira AJISAKA via aw)
|
|
|
|
Release 2.5.1 - 2014-09-05
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-6033. Updated access check for displaying job information
|
|
(Yu Gao via Eric Yang)
|
|
|
|
Release 2.5.0 - 2014-08-11
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5671. NaN can be created by client and assign to Progress (Chen
|
|
He via jeagles)
|
|
|
|
MAPREDUCE-5665. Add audience annotations to MiniMRYarnCluster and
|
|
MiniMRCluster. (Anubhav Dhoot via kasha)
|
|
|
|
MAPREDUCE-5765. Update hadoop-pipes examples README (Mit Desai via jeagles)
|
|
|
|
MAPREDUCE-5713. InputFormat and JobConf JavaDoc Fixes (Chen He via jeagles)
|
|
|
|
MAPREDUCE-5456. TestFetcher.testCopyFromHostExtraBytes is missing (Jason
|
|
Lowe via jeagles)
|
|
|
|
MAPREDUCE-5804. TestMRJobsWithProfiler#testProfiler timesout (Mit Desai
|
|
via kihwal)
|
|
|
|
MAPREDUCE-5825. Provide diagnostics for reducers killed during ramp down
|
|
(Gera Shegalov via jlowe)
|
|
|
|
MAPREDUCE-5836. Fix typo in RandomTextWriter (Akira AJISAKA via jeagles)
|
|
|
|
MAPREDUCE-5852. Prepare MapReduce codebase for JUnit 4.11. (cnauroth)
|
|
|
|
MAPREDUCE-5639. Port DistCp2 document to trunk (Akira AJISAKA via jeagles)
|
|
|
|
MAPREDUCE-5812. Make job context available to
|
|
OutputCommitter.isRecoverySupported() (Mohammad Kamrul Islam via jlowe)
|
|
|
|
MAPREDUCE-5638. Port Hadoop Archives document to trunk (Akira AJISAKA via
|
|
jeagles)
|
|
|
|
MAPREDUCE-5402. In DynamicInputFormat, change MAX_CHUNKS_TOLERABLE,
|
|
MAX_CHUNKS_IDEAL, MIN_RECORDS_PER_CHUNK and SPLIT_RATIO to be configurable.
|
|
(Tsuyoshi OZAWA via szetszwo)
|
|
|
|
MAPREDUCE-5637. Convert Hadoop Streaming document to APT (Akira AJISAKA via
|
|
jeagles)
|
|
|
|
MAPREDUCE-5636. Convert MapReduce Tutorial document to APT (Akira AJISAKA
|
|
via jeagles)
|
|
|
|
MAPREDUCE-5774. Job overview in History UI should list reducer phases in
|
|
chronological order. (Gera Shegalov via kasha)
|
|
|
|
MAPREDUCE-5652. NM Recovery. ShuffleHandler should handle NM restarts.
|
|
(Jason Lowe via kasha)
|
|
|
|
MAPREDUCE-5861. finishedSubMaps field in LocalContainerLauncher does not
|
|
need to be volatile. (Tsuyoshi OZAWA via junping_du)
|
|
|
|
MAPREDUCE-5809. Enhance distcp to support preserving HDFS ACLs. (cnauroth)
|
|
|
|
MAPREDUCE-5899. Support incremental data copy in DistCp. (jing9)
|
|
|
|
MAPREDUCE-5886. Allow wordcount example job to accept multiple input paths.
|
|
(cnauroth)
|
|
|
|
MAPREDUCE-5834. Increased test-timeouts in TestGridMixClasses to avoid
|
|
occassional failures. (Mit Desai via vinodkv)
|
|
|
|
MAPREDUCE-5896. InputSplits should indicate which locations have the block
|
|
cached in memory. (Sandy Ryza via kasha)
|
|
|
|
MAPREDUCE-5844. Add a configurable delay to reducer-preemption.
|
|
(Maysam Yabandeh via kasha)
|
|
|
|
MAPREDUCE-5790. Made it easier to enable hprof profile options by default.
|
|
(Gera Shegalov via vinodkv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5759. Remove unnecessary conf load in Limits (Sandy Ryza)
|
|
|
|
MAPREDUCE-5014. Extend Distcp to accept a custom CopyListing.
|
|
(Srikanth Sundarrajan via amareshwari)
|
|
|
|
MAPREDUCE-5775. Remove unnecessary job.setNumReduceTasks in SleepJob.createJob
|
|
(jhanver chand sharma via devaraj)
|
|
|
|
MAPREDUCE-4937. MR AM handles an oversized split metainfo file poorly
|
|
(Eric Payne via jlowe)
|
|
|
|
MAPREDUCE-5642. TestMiniMRChildTask fails on Windows.
|
|
(Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5846. Rumen doesn't understand JobQueueChangedEvent (Nathan Roberts via raviprak)
|
|
|
|
MAPREDUCE-5837. MRAppMaster fails when checking on uber mode. (wheat9)
|
|
|
|
MAPREDUCE-5749. TestRMContainerAllocator#testReportedAppProgress Failed
|
|
(jlowe)
|
|
|
|
MAPREDUCE-5884. History server uses short user name when canceling tokens
|
|
(Mohammad Kamrul Islam via jlowe)
|
|
|
|
MAPREDUCE-5888. Failed job leaves hung AM after it unregisters (Jason Lowe
|
|
via jeagles)
|
|
|
|
MAPREDUCE-5814. fat jar with *-default.xml may fail when
|
|
mapreduce.job.classloader=true. (Gera Shegalov via jlowe)
|
|
|
|
MAPREDUCE-5309. 2.0.4 JobHistoryParser can't parse certain failed job
|
|
history files generated by 2.0.3 history server (Rushabh S Shah via jlowe)
|
|
|
|
MAPREDUCE-5862. Line records longer than 2x split size aren't handled
|
|
correctly (bc Wong via jlowe)
|
|
|
|
MAPREDUCE-5895. Close streams properly to avoid leakage in TaskLog.
|
|
(Kousuke Saruta via devaraj)
|
|
|
|
MAPREDUCE-5777. Support utf-8 text with Byte Order Marker.
|
|
(Zhihai Xu via kasha)
|
|
|
|
MAPREDUCE-5898. distcp to support preserving HDFS extended attributes(XAttrs)
|
|
(Yi Liu via umamahesh)
|
|
|
|
MAPREDUCE-5920. Add Xattr option in DistCp docs. (Yi Liu via cnauroth)
|
|
|
|
MAPREDUCE-5924. Changed TaskAttemptImpl to ignore TA_COMMIT_PENDING event
|
|
at COMMIT_PENDING state. (Zhijie Shen via jianhe)
|
|
|
|
MAPREDUCE-5939. StartTime showing up as the epoch time in JHS UI after
|
|
upgrade (Chen He via jlowe)
|
|
|
|
MAPREDUCE-5900. Changed to the interpret container preemption exit code as a
|
|
task attempt killing event. (Mayank Bansal via zjshen)
|
|
|
|
MAPREDUCE-5868. Fixed an issue with TestPipeApplication that was causing the
|
|
nightly builds to fail. (Akira Ajisaka via vinodkv)
|
|
|
|
MAPREDUCE-5517. Fixed MapReduce ApplicationMaster to not validate reduce side
|
|
resource configuration for deciding uber-mode on map-only jobs. (Siqi Li via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-5952. LocalContainerLauncher#renameMapOutputForReduce incorrectly
|
|
assumes a single dir for mapOutIndex. (Gera Shegalov via kasha)
|
|
|
|
MAPREDUCE-6002. Made MR task avoid reporting error to AM when the task process
|
|
is shutting down. (Wangda Tan via zjshen)
|
|
|
|
Release 2.4.1 - 2014-06-23
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5830. Added back the private API HostUtil.getTaskLogUrl(..) for
|
|
binary compatibility with older clients like Hive 0.13. (Akira Ajisaka via
|
|
vinodkv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5714. Removed forceful JVM exit in shutDownJob.
|
|
(Jinghui Wang via Eric Yang)
|
|
|
|
MAPREDUCE-5818. Added "hsadmin" command into mapred.cmd. (Jian He via zjshen)
|
|
|
|
MAPREDUCE-5824. Fixed test-failure of TestPipesNonJavaInputFormat in
|
|
Windows. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5815. Fixed test-failure of TestMRAppMaster by making MRAppMaster
|
|
gracefully handle empty-queue names. (Akira Ajisaka via vinodkv)
|
|
|
|
MAPREDUCE-5826. Fixed HistoryServerFileSystemStore to use right permissions
|
|
on Windows for temporary files and thus also fix the test-issue with
|
|
TestHistoryServerFileSystemStateStoreService. (Varun Vasudev via vinodkv)
|
|
|
|
MAPREDUCE-5828. Fixed a test issue with TestMapReduceJobControl that was
|
|
causing it to fail on Windows. (vinodkv)
|
|
|
|
MAPREDUCE-5827. TestSpeculativeExecutionWithMRApp fails.
|
|
(Zhijie Shen via cnauroth)
|
|
|
|
MAPREDUCE-5833. TestRMContainerAllocator fails ocassionally.
|
|
(Zhijie Shen via cnauroth)
|
|
|
|
MAPREDUCE-5832. Fixed TestJobClient to not fail on JDK7 or on Windows. (Jian
|
|
He and Vinod Kumar Vavilapalli via vinodkv)
|
|
|
|
MAPREDUCE-5841. uber job doesn't terminate on getting mapred job kill
|
|
(Sangjin Lee via jlowe)
|
|
|
|
MAPREDUCE-5843. Fixed TestMRKeyValueTextInputFormat to not leak files and
|
|
thus avoid failing on Windows. (Varun Vasudev via vinodkv)
|
|
|
|
MAPREDUCE-5835. Killing Task might cause the job to go to ERROR state
|
|
(Ming Ma via jlowe)
|
|
|
|
MAPREDUCE-5821. Avoid unintentional reallocation of byte arrays in segments
|
|
during merge. (Todd Lipcon via cdouglas)
|
|
|
|
Release 2.4.0 - 2014-04-07
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-5787. Added the ability to keep alive shuffle connections in the
|
|
MapReduce shuffle-handler. (Rajesh Balamohan via vinodkv)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5464. Add analogs of the SLOTS_MILLIS counters that jive with the
|
|
YARN resource model (Sandy Ryza)
|
|
|
|
MAPREDUCE-5732. Report proper queue when job has been automatically placed
|
|
(Sandy Ryza)
|
|
|
|
MAPREDUCE-5699. Allow setting tags on MR jobs (kasha)
|
|
|
|
MAPREDUCE-5761. Added a simple log message to denote when encrypted shuffle
|
|
is on in the shuffle-handler. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5754. Preserve Job diagnostics in history (Gera Shegalov via
|
|
jlowe)
|
|
|
|
MAPREDUCE-5773. Provide dedicated MRAppMaster syslog length limit (Gera
|
|
Shegalov via jlowe)
|
|
|
|
MAPREDUCE-5553. Allow users to easily access
|
|
completed/pending/successful/failed tasks on MR AM web-ui. (Paul Han via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-4052. Improved MapReduce clients to use NodeManagers' ability to
|
|
handle cross platform application submissions. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-2349. Modified FileInputFormat to be able to issue file and block
|
|
location calls in parallel. (Siddharth Seth via vinodkv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5746. Job diagnostics can implicate wrong task for a failed job.
|
|
(Jason Lowe via kasha)
|
|
|
|
MAPREDUCE-5670. CombineFileRecordReader should report progress when moving
|
|
to the next file (Chen He via jlowe)
|
|
|
|
MAPREDUCE-5757. ConcurrentModificationException in JobControl.toList
|
|
(jlowe)
|
|
|
|
MAPREDUCE-5770. Fixed MapReduce ApplicationMaster to correctly redirect
|
|
to the YARN's web-app proxy with the correct scheme prefix. (Jian He via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-5768. TestMRJobs.testContainerRollingLog fails on trunk (Gera
|
|
Shegalov via jlowe)
|
|
|
|
MAPREDUCE-5780. SliveTest should use the specified path to get the
|
|
particular FileSystem instead of using the default FileSystem. (szetszwo)
|
|
|
|
MAPREDUCE-5028. Fixed a bug in MapTask that was causing mappers to fail
|
|
when a large value of io.sort.mb is set. (Karthik Kambatla via vinodkv)
|
|
|
|
MAPREDUCE-5778. JobSummary does not escape newlines in the job name (Akira
|
|
AJISAKA via jlowe)
|
|
|
|
MAPREDUCE-5789. Average Reduce time is incorrect on Job Overview page
|
|
(Rushabh S Shah via jlowe)
|
|
|
|
MAPREDUCE-5794. SliveMapper always uses default FileSystem. (szetszwo via
|
|
Arpit Agarwal)
|
|
|
|
MAPREDUCE-5751. MR app master fails to start in some cases if
|
|
mapreduce.job.classloader is true (Sangjin Lee via jlowe)
|
|
|
|
MAPREDUCE-5688. TestStagingCleanup fails intermittently with JDK7 (Mit
|
|
Desai via jeagles)
|
|
|
|
MAPREDUCE-5769. Unregistration to RM should not be called if AM is crashed
|
|
before registering with RM (Rohith via jlowe)
|
|
|
|
MAPREDUCE-5570. Map task attempt with fetch failure has incorrect attempt
|
|
finish time (Rushabh S Shah via jlowe)
|
|
|
|
MAPREDUCE-5806. Fixed a bug in MRAppMaster so as to enable users to properly
|
|
override HADOOP_ROOT_LOGGER or HADOOP_CLIENT_OPTS. (Varun Vasudev via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-5791. Shuffle phase is slow in Windows -
|
|
FadviseFileRegion::transferTo does not read disks efficiently.
|
|
(Nikola Vujic via cnauroth)
|
|
|
|
MAPREDUCE-5795. Fixed MRAppMaster to record the correct job-state after it
|
|
recovers from a commit during a previous attempt. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5805. Fixed MapReduce JobHistory encoding of queue-name to escape
|
|
hyphens and thus avoid parsing errors. (Akira AJISAKA via vinodkv)
|
|
|
|
MAPREDUCE-5810. Removed the faulty and failing streaming test
|
|
TestStreamingTaskLog. (Akira Ajisaka via vinodkv)
|
|
|
|
MAPREDUCE-5813. Fix YarnChild to explicitly load job.xml from the
|
|
local-filesystem, rather than rely on the classpath. (Gera Shegalov via
|
|
acmurthy)
|
|
|
|
Release 2.3.1 - UNRELEASED
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
Release 2.3.0 - 2014-02-18
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-5265. History server admin service to refresh user and superuser
|
|
group mappings (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5356. Ability to refresh aggregated log retention period and
|
|
check interval (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5386. Ability to refresh history server job retention and job
|
|
cleaner settings (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5411. Refresh size of loaded job cache on history server (Ashwin
|
|
Shankar via jlowe)
|
|
|
|
MAPREDUCE-5332. Support token-preserving restart of history server (jlowe)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5329. Allow MR applications to use additional AuxServices,
|
|
which are compatible with the default MapReduce shuffle.
|
|
(Avner BenHanoch via sseth)
|
|
|
|
MAPREDUCE-5463. Deprecate SLOTS_MILLIS counters (Tzuyoshi Ozawa via Sandy
|
|
Ryza)
|
|
|
|
MAPREDUCE-5457. Add a KeyOnlyTextOutputReader to enable streaming to write
|
|
out text files without separators (Sandy Ryza)
|
|
|
|
MAPREDUCE-5596. Allow configuring the number of threads used to serve
|
|
shuffle connections (Sandy Ryza via jlowe)
|
|
|
|
MAPREDUCE-434. LocalJobRunner limited to single reducer (Sandy Ryza and
|
|
Aaron Kimball via Sandy Ryza)
|
|
|
|
MAPREDUCE-4421. Run MapReduce framework via the distributed cache (jlowe)
|
|
|
|
MAPREDUCE-1176. FixedLengthInputFormat and FixedLengthRecordReader
|
|
(Mariappan Asokan and BitsOfInfo via Sandy Ryza)
|
|
|
|
MAPREDUCE-5613. DefaultSpeculator holds and checks hashmap that is always
|
|
empty (Gera Shegalov via Sandy Ryza)
|
|
|
|
MAPREDUCE-5431. Missing pom dependency in MR-client (Timothy St. Clair
|
|
via stevel)
|
|
|
|
MAPREDUCE-5624 Move grizzly-test and junit dependencies to test scope
|
|
(Ted Yu via stevel)
|
|
|
|
MAPREDUCE-5481. Enable uber jobs to have multiple reducers (Sandy Ryza)
|
|
|
|
MAPREDUCE-5052. Job History UI and web services confusing job start time and
|
|
job submit time (Chen He via jeagles)
|
|
|
|
MAPREDUCE-5692. Add explicit diagnostics when a task attempt is killed due
|
|
to speculative execution (Gera Shegalov via Sandy Ryza)
|
|
|
|
MAPREDUCE-5550. Task Status message (reporter.setStatus) not shown in UI
|
|
with Hadoop 2.0 (Gera Shegalov via Sandy Ryza)
|
|
|
|
MAPREDUCE-3310. Custom grouping comparator cannot be set for Combiners (tucu)
|
|
|
|
MAPREDUCE-5672. Provide optional RollingFileAppender for container log4j
|
|
(syslog) (Gera Shegalov via jlowe)
|
|
|
|
MAPREDUCE-5725. Make explicit that TestNetworkedJob relies on the Capacity
|
|
Scheduler (Sandy Ryza)
|
|
|
|
MAPREDUCE-5744. Job hangs because
|
|
RMContainerAllocator$AssignedRequests.preemptReduce() violates the
|
|
comparator contract (Gera Shegalov via kasha)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-4680. Job history cleaner should only check timestamps of files in
|
|
old enough directories (Robert Kanter via Sandy Ryza)
|
|
|
|
MAPREDUCE-5484. YarnChild unnecessarily loads job conf twice (Sandy Ryza)
|
|
|
|
MAPREDUCE-5487. In task processes, JobConf is unnecessarily loaded again
|
|
in Limits (Sandy Ryza)
|
|
|
|
MAPREDUCE-5601. ShuffleHandler fadvises file regions as DONTNEED even when
|
|
fetch fails (Sandy Ryza)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5569. FloatSplitter is not generating correct splits (Nathan
|
|
Roberts via jlowe)
|
|
|
|
MAPREDUCE-5546. mapred.cmd on Windows set HADOOP_OPTS incorrectly (Chuan Liu
|
|
via cnauroth)
|
|
|
|
MAPREDUCE-5518. Fixed typo "can't read paritions file". (Albert Chu
|
|
via devaraj)
|
|
|
|
MAPREDUCE-5561. org.apache.hadoop.mapreduce.v2.app.job.impl.TestJobImpl
|
|
testcase failing on trunk (Karthik Kambatla via jlowe)
|
|
|
|
MAPREDUCE-5598. TestUserDefinedCounters.testMapReduceJob is flakey
|
|
(Robert Kanter via jlowe)
|
|
|
|
MAPREDUCE-5604. TestMRAMWithNonNormalizedCapabilities fails on Windows due to
|
|
exceeding max path length. (cnauroth)
|
|
|
|
MAPREDUCE-5451. MR uses LD_LIBRARY_PATH which doesn't mean anything in
|
|
Windows. (Yingda Chen via cnauroth)
|
|
|
|
MAPREDUCE-5409. MRAppMaster throws InvalidStateTransitonException: Invalid
|
|
event: TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl (Gera
|
|
Shegalov via jlowe)
|
|
|
|
MAPREDUCE-5674. Missing start and finish time in mapred.JobStatus.
|
|
(Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5650. Job fails when hprof mapreduce.task.profile.map/reduce.params
|
|
is specified (Gera Shegalov via Sandy Ryza)
|
|
|
|
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
|
|
task-state (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5380. Invalid mapred command should return non-zero exit code
|
|
(Stephen Chu via jlowe)
|
|
|
|
MAPREDUCE-5404. HSAdminServer does not use ephemeral ports in minicluster
|
|
mode (Ted Yu via jlowe)
|
|
|
|
MAPREDUCE-5522. Incorrect oreder expected from JobQueueInfo (Jinghui Wang
|
|
via bobby)
|
|
|
|
MAPREDUCE-5514. Fix TestRMContainerAllocator. (Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5102. fix coverage org.apache.hadoop.mapreduce.lib.db and
|
|
org.apache.hadoop.mapred.lib.db (Aleksey Gorshkov, Andrey Klochkov, and
|
|
Nathan Roberts via jlowe)
|
|
|
|
MAPREDUCE-5585. TestCopyCommitter#testNoCommitAction Fails on JDK7
|
|
(jeagles)
|
|
|
|
MAPREDUCE-5186. mapreduce.job.max.split.locations causes some splits
|
|
created by CombineFileInputFormat to fail (Robert Parker and Jason Lowe
|
|
via jlowe)
|
|
|
|
MAPREDUCE-5610. TestSleepJob fails in jdk7 (Jonathan Eagles via jlowe)
|
|
|
|
MAPREDUCE-5616. MR Client-AppMaster RPC max retries on socket timeout is too
|
|
high. (cnauroth)
|
|
|
|
MAPREDUCE-5625. TestFixedLengthInputFormat fails in jdk7 environment
|
|
(Mariappan Asokan via jeagles)
|
|
|
|
MAPREDUCE-5631. TestJobEndNotifier.testNotifyRetries fails with Should
|
|
have taken more than 5 seconds in jdk7 (Jonathan Eagles via jlowe)
|
|
|
|
MAPREDUCE-5645. TestFixedLengthInputFormat fails with native libs (Mit
|
|
Desai via jeagles)
|
|
|
|
MAPREDUCE-5632. TestRMContainerAllocator#testUpdatedNodes fails (jeagles)
|
|
|
|
MAPREDUCE-5656. bzip2 codec can drop records when reading data in splits
|
|
(jlowe)
|
|
|
|
MAPREDUCE-5623. TestJobCleanup fails because of RejectedExecutionException
|
|
and NPE. (jlowe)
|
|
|
|
MAPREDUCE-5679. TestJobHistoryParsing has race condition (Liyin Liang via
|
|
jlowe)
|
|
|
|
MAPREDUCE-5687. Fixed failure in TestYARNRunner caused by YARN-1446. (Jian He
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-5694. Fixed MR AppMaster to shutdown the LogManager so as to avoid
|
|
losing syslog in some conditions. (Mohammad Kamrul Islam via vinodkv)
|
|
|
|
MAPREDUCE-5685. Fixed a bug with JobContext getCacheFiles API inside the
|
|
WrappedReducer class. (Yi Song via vinodkv)
|
|
|
|
MAPREDUCE-5689. MRAppMaster does not preempt reducers when scheduled maps
|
|
cannot be fulfilled. (lohit via kasha)
|
|
|
|
MAPREDUCE-5724. JobHistoryServer does not start if HDFS is not running.
|
|
(tucu)
|
|
|
|
MAPREDUCE-5729. mapred job -list throws NPE (kasha)
|
|
|
|
MAPREDUCE-5693. Restore MRv1 behavior for log flush (Gera Shegalov via
|
|
jlowe)
|
|
|
|
MAPREDUCE-5723. MR AM container log can be truncated or empty.
|
|
(Mohammad Kamrul Islam via kasha)
|
|
|
|
MAPREDUCE-5743. Fixed the test failure in TestRMContainerAllocator.
|
|
(Ted Yu and Vinod Kumar Vavilapalli via zjshen)
|
|
|
|
Release 2.2.0 - 2013-10-13
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta
|
|
via tgraves)
|
|
|
|
MAPREDUCE-5488. Changed MR client to keep trying to reach the application
|
|
when it sees that on attempt's AM is down. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5515. Fixed MR AM's webapp to depend on a new config
|
|
mapreduce.ssl.enabled to enable https and disabling it by default as MR AM
|
|
needs to set up its own certificates etc and not depend on clusters'.
|
|
(Omkar Vinit Joshi via vinodkv)
|
|
|
|
MAPREDUCE-5505. Clients should be notified job finished only after job
|
|
successfully unregistered (Zhijie Shen via bikas)
|
|
|
|
MAPREDUCE-5503. Fixed a test issue in TestMRJobClient. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5170. Fixed a wrong log message in CombineFileInputFormat class.
|
|
(Sangjin Lee via vinodkv)
|
|
|
|
MAPREDUCE-5525. Increase timeout of TestDFSIO.testAppend and
|
|
TestMRJobsWithHistoryService.testJobHistoryData. (Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert
|
|
Parker via jlowe)
|
|
|
|
MAPREDUCE-5531. Fix compat with hadoop-1 in mapreduce.(TaskID,
|
|
TaskAttemptID) by re-introducing missing constructors. (Robert Kanter via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-5545. org.apache.hadoop.mapred.TestTaskAttemptListenerImpl.testCommitWindow
|
|
times out (Robert Kanter via jlowe)
|
|
|
|
MAPREDUCE-5529. Fix compat with hadoop-1 in mapred.TotalOrderPartitioner
|
|
by re-introducing (get,set)PartitionFile which takes in JobConf. (Robert
|
|
Kanter via acmurthy)
|
|
|
|
MAPREDUCE-5538. Fixed MR AppMaster to send job-notification URL only after
|
|
the job is really done - a bug caused by MAPREDUCE-5505. (Zhijie Shen via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-5551. Fix compat with hadoop-1 in
|
|
SequenceFileAsBinaryOutputFormat.WritableValueBytes by re-introducing
|
|
missing constructors. (Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5544. JobClient#getJob loads job conf twice. (Sandy Ryza)
|
|
|
|
MAPREDUCE-5536. Fixed MR AM and JHS to respect
|
|
mapreduce.jobhistory.webapp.https.address. (Omkar Vinit Joshi via vinodkv)
|
|
|
|
MAPREDUCE-5530. Fix compat with hadoop-1 in
|
|
mapred.lib.CombinFileInputFormat by re-introducing
|
|
isSplittable(FileSystem, Path) api and ensuring semantic compatibility.
|
|
(Robert Kanter via acmurthy)
|
|
|
|
MAPREDUCE-5459. Update documentation on how to run MRv1 examples on YARN.
|
|
(Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5554. hdfs-site.xml included in hadoop-mapreduce-client-jobclient
|
|
tests jar is breaking tests for downstream components (Robert Kanter via
|
|
Sandy Ryza)
|
|
|
|
MAPREDUCE-5489. MR jobs hangs as it does not use the node-blacklisting
|
|
feature in RM requests (Zhijie Shen via bikas)
|
|
|
|
MAPREDUCE-5442. $HADOOP_MAPRED_HOME/$HADOOP_CONF_DIR setting not working on
|
|
Windows. (Yingda Chen via cnauroth)
|
|
|
|
MAPREDUCE-5533. Fixed MR speculation code to track any TaskAttempts that
|
|
aren't heart-beating for a while, so that we can aggressively speculate
|
|
instead of waiting for task-timeout (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5562. Fixed MR App Master to perform pending tasks like staging-dir
|
|
cleanup, sending job-end notification correctly when unregister with RM
|
|
fails. (Zhijie Shen via vinodkv)
|
|
|
|
Release 2.1.1-beta - 2013-09-23
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5478. TeraInputFormat unnecessarily defines its own FileSplit
|
|
subclass (Sandy Ryza)
|
|
|
|
MAPREDUCE-5497. Changed MRAppMaster to sleep only after doing everything else
|
|
but just before ClientService to avoid race conditions during RM restart.
|
|
(Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5379. Include token tracking ids in jobconf. (kkambatl via tucu)
|
|
|
|
MAPREDUCE-5523. Added separate configuration properties for https for JHS
|
|
without which even when https is enabled, it starts on http port itself.
|
|
(Omkar Vinit Joshi via vinodkv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-5446. TestJobHistoryEvents and TestJobHistoryParsing have race
|
|
conditions (jlowe via kihwal)
|
|
|
|
MAPREDUCE-5462. In map-side sort, swap entire meta entries instead of
|
|
indexes for better cache performance. (Sandy Ryza)
|
|
|
|
MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus
|
|
(Hairong Kuang and Jason Lowe via jlowe)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5385. Fixed a bug with JobContext getCacheFiles API. (Omkar Vinit
|
|
Joshi via vinodkv)
|
|
|
|
MAPREDUCE-5428. HistoryFileManager doesn't stop threads when service is
|
|
stopped (Karthik Kambatla via jlowe)
|
|
|
|
MAPREDUCE-5251. Reducer should not implicate map attempt if it has
|
|
insufficient space to fetch map output (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via
|
|
jlowe)
|
|
|
|
MAPREDUCE-5358. MRAppMaster throws invalid transitions for JobImpl
|
|
(Devaraj K via jlowe)
|
|
|
|
MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the
|
|
input path dir (Devaraj K via jlowe)
|
|
|
|
MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe)
|
|
|
|
MAPREDUCE-5367. Local jobs all use same local working directory
|
|
(Sandy Ryza)
|
|
|
|
MAPREDUCE-5425. Junit in TestJobHistoryServer failing in jdk 7 (Robert
|
|
Parker via jlowe)
|
|
|
|
MAPREDUCE-5454. TestDFSIO fails intermittently on JDK7 (Karthik Kambatla
|
|
via Sandy Ryza)
|
|
|
|
MAPREDUCE-5001. LocalJobRunner has race condition resulting in job
|
|
failures (Sandy Ryza via jlowe)
|
|
|
|
MAPREDUCE-5466. Changed MR AM to not promote history files of intermediate
|
|
AMs in case they are exiting because of errors and thus help history-server
|
|
pick up the right history file for the last successful AM. (Jian He via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-5468. Fix MR AM recovery for map-only jobs. (vinodkv via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-5470. LocalJobRunner does not work on Windows. (Sandy Ryza via
|
|
cnauroth)
|
|
|
|
MAPREDUCE-5476. Changed MR AM recovery code to cleanup staging-directory
|
|
only after unregistering from the RM. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5483. revert MAPREDUCE-5357. (rkanter via tucu)
|
|
|
|
MAPREDUCE-5441. Changed MR AM to return RUNNING state if exiting when RM
|
|
commands to reboot, so that client can continue to track the overall job.
|
|
(Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe)
|
|
|
|
MAPREDUCE-5414. TestTaskAttempt fails in JDK7 with NPE (Nemon Lou via
|
|
devaraj)
|
|
|
|
MAPREDUCE-5020. Compile failure with JDK8 (Trevor Robinson via tgraves)
|
|
|
|
MAPREDUCE-5164. mapred job and queue commands omit HADOOP_CLIENT_OPTS
|
|
(Nemon Lou via devaraj)
|
|
|
|
MAPREDUCE-5493. Cleanup in-memory & on-disk segments to prevent leak on
|
|
shuffle completion. (jlowe via acmurthy)
|
|
|
|
Release 2.1.0-beta - 2013-08-22
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-4067. Changed MRClientProtocol api to throw IOException only (Xuan
|
|
Gong via vinodkv)
|
|
|
|
MAPREDUCE-5234. Change mapred.TaskReport and mapreduce.TaskReport for binary
|
|
compatibility with mapred in 1.x but incompatible with 0.23.x. (Mayank Bansal
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run
|
|
1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of
|
|
ProgramDriver. (Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5233. Add methods that are changed or removed from JobControl.Job
|
|
when compared to 1.x. This breaks 0.23.x users of one API in Job. (Mayank
|
|
Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5237. Add methods that were removed from ClusterStatus back into
|
|
2.x so as to be compatible with 1.x. Incompatible as
|
|
ClusterStatus.UNINITIALIZED_MEMORY_VALUE is a long now and so breaks 0.23.x
|
|
but it shouldn't be a big deal in reality. (Zhijie Shen via vinodkv)
|
|
|
|
NEW FEATURES
|
|
|
|
HADOOP-8562. Enhancements to support Hadoop on Windows Server and Windows
|
|
Azure environments. (See breakdown of tasks below for subtasks and
|
|
contributors)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-3008. Improvements to cumulative CPU emulation for short running
|
|
tasks in Gridmix. (amarrk via tgraves)
|
|
|
|
MAPREDUCE-5033. mapred shell script should respect usage flags
|
|
(--help -help -h). (Andrew Wang via atm)
|
|
|
|
MAPREDUCE-4892. Modify CombineFileInputFormat to not skew input slits'
|
|
allocation on small clusters. (Bikas Saha via vinodkv)
|
|
|
|
MAPREDUCE-4990. Construct debug strings conditionally in
|
|
ShuffleHandler.Shuffle#sendMapOutput(). (kkambatl via tucu)
|
|
|
|
MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred
|
|
(Aleksey Gorshkov via bobby)
|
|
|
|
MAPREDUCE-5129. Allow tags to JobHistory for deeper analytics. (billie via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3787. [Gridmix] Optimize job monitoring and STRESS mode for
|
|
faster job submission. (amarrk via tgraves)
|
|
|
|
MAPREDUCE-5079. Changes job recovery to restore state directly from job
|
|
history, instaed of simulating state machine events.
|
|
(Jason Lowe and Robert Parker via sseth)
|
|
|
|
MAPREDUCE-4981. Add WordMean, WordMedian, WordStandardDeviation
|
|
to ExamplesDriver. (Plamen Jeliazkov via shv)
|
|
|
|
MAPREUDUCE-5059. Change average merge time on Job overview page to be the
|
|
time delta between the end of the shuffle and the start of the reduce.
|
|
(Omkar Vinit Joshi via vinodkv)
|
|
|
|
MAPREDUCE-4985. Add compression option to TestDFSIO usage.
|
|
(Plamen Jeliazkov via shv)
|
|
|
|
MAPREDUCE-5152. Make MR App to simply pass through the container from RM
|
|
instead of extracting and populating information itself to start any
|
|
container. (vinodkv)
|
|
|
|
MAPREDUCE-5175. Updated MR App to not set envs that will be set by NMs
|
|
anyways after YARN-561. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5069. add concrete common implementations of
|
|
CombineFileInputFormat (Sangjin Lee via bobby)
|
|
|
|
MAPREDUCE-5145. Changed default max-attempts to be more than one for MR jobs
|
|
inline with YARN. (Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5036. Default shuffle handler port should not be 8080.
|
|
(Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-5159. Change ValueAggregatorJob to add APIs which can support
|
|
binary compatibility with hadoop-1 examples. (Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5157. Bring back old sampler related code so that we can support
|
|
binary compatibility with hadoop-1 sorter example. (Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5222. Bring back some methods and constants in Jobclient for
|
|
binary compatibility with mapred in 1.x. (Karthik Kambatla via vinodkv)
|
|
|
|
MAPREDUCE-5235. Bring back old fields and exceptions in Counters for
|
|
binary compatibility with mapred in 1.x. (Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5246. Specify application-type at the time of job submission after
|
|
YARN-563. (Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5230. Bring back NLineInputFormat.createFileSplit for binary
|
|
compatibility with mapred in 1.x (Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5270. Migrated MR app from using BuilderUtil factory methods to
|
|
individual record factory methods. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5263. Bring back old methods and fields in
|
|
filecache.DistributedCache for binary compatibility with mapred in 1.x.
|
|
(Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5228. Bring back FileInputFormat.Counter and
|
|
FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs.
|
|
(Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5176. Add annotation for tagging tasks as responsive to
|
|
preemption. (Carlo Curino, cdouglas)
|
|
|
|
MAPREDUCE-5275. Bring back a couple of APIs in mapreduce.security.TokenCache
|
|
for binary compatibility with 1.x mapreduce APIs. (Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5231. Bring back a constructor in mapred's
|
|
DBInputFormat.DBRecordReader for binary compatibility with 1.x mapred APIs.
|
|
(Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5273. Bring back a couple of protected variables in mapred and
|
|
mapreduce CombineFileRecordReader for binary compatibility with 1.x APIs.
|
|
(Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5280. Bring back removed constructor and a method in mapreduce
|
|
ClusterMetrics for binary compatibility with 1.x APIs. (Mayank Bansal via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-5289. Updated MR App to use Token directly after YARN-717. (Jian He
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-5229. Put back FileOutputCommitter.TEMP_DIR_NAME in mapreduce for
|
|
binary compatibility with 1.x APIs. (Zhijie Shen via vinodkv)
|
|
|
|
MAPREDUCE-5274. Bring back SecureShuffleUtils.toHex in mapreduce for binary
|
|
compatibility with 1.x APIs. (Mayank Bansal via vinodkv)
|
|
|
|
MAPREDUCE-5300. Fix backward incompatibility for
|
|
o.a.h.mapreduce.filecache.DistributedCache. (Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5283. Over 10 different tests have near identical
|
|
implementations of AppContext (Sandy Ryza via jlowe)
|
|
|
|
MAPREDUCE-5199. Removing ApplicationTokens file as it is no longer needed.
|
|
(Daryn Sharp via vinodkv)
|
|
|
|
MAPREDUCE-5192. Allow for alternate resolutions of TaskCompletionEvents.
|
|
(cdouglas via acmurthy)
|
|
|
|
MAPREDUCE-5184. Document compatibility for MapReduce applications in
|
|
hadoop-2 vis-a-vis hadoop-1. (Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (cdouglas)
|
|
|
|
MAPREDUCE-5326. Added version to shuffle header. (Zhijie Shen via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-5333. Add test that verifies MRAM works correctly when sending
|
|
requests with non-normalized capabilities. (ywskycn via tucu)
|
|
|
|
MAPREDUCE-5398. MR changes for YARN-513 (Jian He via bikas)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-4974. Optimising the LineRecordReader initialize() method
|
|
(Gelesh via bobby)
|
|
|
|
MAPREDUCE-5268. Improve history server startup performance (Karthik
|
|
Kambatla via jlowe)
|
|
|
|
MAPREDUCE-5352. Optimize node local splits generated by
|
|
CombineFileInputFormat. (sseth)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4671. AM does not tell the RM about container requests which are
|
|
no longer needed. (Bikas Saha via sseth)
|
|
|
|
MAPREDUCE-4994. -jt generic command line option does not work. (sandyr via tucu)
|
|
|
|
MAPREDUCE-5000. Fixes getCounters when speculating by fixing the selection
|
|
of the best attempt for a task. (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-4994. Addendum fixing testcases failures. (sandyr via tucu)
|
|
|
|
MAPREDUCE-4846. Some JobQueueInfo methods are public in MR1 but protected
|
|
in MR2. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-5013. mapred.JobStatus compatibility: MR2 missing constructors
|
|
from MR1. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-4951. Container preemption interpreted as task failure.
|
|
(Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-5008. Merger progress miscounts with respect to EOF_MARKER.
|
|
(Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-4693. History server should include counters for failed tasks.
|
|
(Xuan Gong via sseth)
|
|
|
|
MAPREDUCE-4896. mapred queue -info spits out ugly exception when queue does
|
|
not exist. (sandyr via tucu)
|
|
|
|
MAPREDUCE-3685. Fix bugs in MergeManager to ensure compression codec is
|
|
appropriately used and that on-disk segments are correctly sorted on
|
|
file-size. (Anty Rao and Ravi Prakash via acmurthy)
|
|
|
|
MAPREDUCE-4571. TestHsWebServicesJobs fails on jdk7. (tgraves via tucu)
|
|
|
|
MAPREDUCE-4716. TestHsWebServicesJobsQuery.testJobsQueryStateInvalid
|
|
fails with jdk7. (tgraves via tucu)
|
|
|
|
MAPREDUCE-5075. DistCp leaks input file handles since ThrottledInputStream
|
|
does not close the wrapped InputStream. (Chris Nauroth via szetszwo)
|
|
|
|
MAPREDUCE-3872. Fix an event handling races in ContainerLauncherImpl.
|
|
(Robert Kanter via sseth)
|
|
|
|
MAPREDUCE-5062. Fix MR AM to read max-retries from the RM. (Zhijie Shen via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when
|
|
input data directory already exists and -generate opton is
|
|
given.(ravigummadi via tgraves)
|
|
|
|
MAPREDUCE-2722. [Gridmix] Gridmix simulated job's map's hdfsBytesRead
|
|
counter is wrong when compressed input is used.(ravigummadi via tgraves)
|
|
|
|
MAPREDUCE-3953. [Gridmix] Gridmix throws NPE and does not simulate a
|
|
job if the trace contains null taskStatus for a task. (ravigummadi via
|
|
tgraves)
|
|
|
|
MAPREDUCE-4087. [Gridmix] GenerateDistCacheData job of Gridmix can
|
|
become slow in some cases (ravigummadi via tgraves).
|
|
|
|
MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related
|
|
code. (Karthik Kambatla via sseth)
|
|
|
|
MAPREDUCE-4083. [Gridmix] NPE in cpu emulation. (amarrk via tgraves)
|
|
|
|
MAPREDUCE-4100. [Gridmix] Bug fixed in compression emulation feature for
|
|
map only jobs. (amarrk via tgraves)
|
|
|
|
MAPREDUCE-4356. [Rumen] Provide access to the method
|
|
ParsedTask.obtainTaskAttempts(). (ravigummadi via tgraves)
|
|
|
|
MAPREDUCE-4149. [Rumen] Rumen fails to parse certain counter
|
|
strings. (ravigummadi via tgraves)
|
|
|
|
MAPREDUCE-3757. [Rumen] Fixed Rumen Folder to adjust shuffleFinished and
|
|
sortFinished times when needed. (Ravi Gummadi via tgraves)
|
|
|
|
MAPREDUCE-5138. Fix LocalDistributedCacheManager after YARN-112. (Omkar Vinit
|
|
Joshi via vinodkv)
|
|
|
|
MAPREDUCE-5086. MR app master deletes staging dir when sent a reboot
|
|
command from the RM. (Jian He via jlowe)
|
|
|
|
MAPREDUCE-5113. Streaming input/output types are ignored with java
|
|
mapper/reducer. (sandyr via tucu)
|
|
|
|
MAPREDUCE-5098. Fix findbugs warnings in gridmix. (kkambatl via tucu)
|
|
|
|
MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error
|
|
(Thomas Graves via jlowe)
|
|
|
|
MAPREDUCE-5136. TestJobImpl->testJobNoTasks fails with IBM JAVA (Amir
|
|
Sanjar via jlowe)
|
|
|
|
MAPREDUCE-5139. Update MR AM to use the modified startContainer API after
|
|
YARN-486. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5151. Update MR AM to use standard exit codes from the API after
|
|
YARN-444. (Sandy Ryza via vinodkv)
|
|
|
|
MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas)
|
|
|
|
MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server
|
|
configs. (sandyr via tucu)
|
|
|
|
MAPREDUCE-4898. FileOutputFormat.checkOutputSpecs and
|
|
FileOutputFormat.setOutputPath incompatible with MR1. (rkanter via tucu)
|
|
|
|
MAPREDUCE-5078. TestMRAppMaster fails on Windows due to mismatched path
|
|
separators. (Chris Nauroth via sseth)
|
|
|
|
MAPREDUCE-4932. mapreduce.job#getTaskCompletionEvents incompatible with
|
|
Hadoop 1. (rkanter via tucu)
|
|
|
|
MAPREDUCE-5163. Update MR App to not use API utility methods for collections
|
|
after YARN-441. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5066. Added a timeout for the job.end.notification.url. (Ivan
|
|
Mitic via acmurthy)
|
|
|
|
MAPREDUCE-5146. application classloader may be used too early to load
|
|
classes. (Sangjin Lee via tomwhite)
|
|
|
|
MAPREDUCE-4737. Ensure that mapreduce APIs are semantically consistent
|
|
with mapred API w.r.t Mapper.cleanup and Reducer.cleanup; in the sense that
|
|
cleanup is now called even if there is an error. The old mapred API
|
|
already ensures that Mapper.close and Reducer.close are invoked during
|
|
error handling. Note that it is an incompatible change, however end-users
|
|
can override Mapper.run and Reducer.run to get the old (inconsistent)
|
|
behaviour. (acmurthy)
|
|
|
|
MAPREDUCE-5166. Fix ConcurrentModificationException due to insufficient
|
|
synchronization on updates to task Counters. (Sandy Ryza via acmurthy)
|
|
|
|
MAPREDUCE-5181. RMCommunicator should not use AMToken from the env.
|
|
(Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-5178. Update MR App to set progress in ApplicationReport after
|
|
YARN-577. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-5167. Update MR App after YARN-562 to use the new builder API
|
|
for the container. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5179. Fix unit test in TestHSWebServices which fails when
|
|
versionInfo has parantheses like when running on a git checkout. (Hitesh Shah
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-5193. A few MR tests use block sizes which are smaller than the
|
|
default minimum block size. (Andrew Wang via atm)
|
|
|
|
MAPREDUCE-5205. Fixed MR App to load tokens correctly. (vinodkv)
|
|
|
|
MAPREDUCE-5204. Handling YarnRemoteException separately from IOException in
|
|
MR app after YARN-629. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5209. Fix units in a ShuffleScheduler log message.
|
|
(Tsuyoshi OZAWA via cdouglas)
|
|
|
|
MAPREDUCE-5212. Handling YarnRemoteException separately from IOException in
|
|
MR App's use of ClientRMProtocol after YARN-631. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-5226. Handling YarnRemoteException separately from IOException in
|
|
MR App's use of AMRMProtocol after YARN-630. (Xuan Gong via vinodkv)
|
|
|
|
MAPREDUCE-4942. mapreduce.Job has a bunch of methods that throw
|
|
InterruptedException so its incompatible with MR1. (rkanter via tucu)
|
|
|
|
MAPREDUCE-5239. Updated MR App to reflect YarnRemoteException changes after
|
|
YARN-634. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-5208. Modified ShuffleHandler to use SecureIOUtils for reading
|
|
local files. (Omkar Vinit Joshi via vinodkv)
|
|
|
|
MAPREDUCE-5220. Setter methods in TaskCompletionEvent are public in MR1 and
|
|
protected in MR2. (sandyr via tucu)
|
|
|
|
MAPREDUCE-5240. Fix a bug in MRAppMaster because of which OutputCommitter
|
|
could not access credentials set by the user. (vinodkv)
|
|
|
|
MAPREDUCE-5244. Two functions changed their visibility in JobStatus.
|
|
(zjshen via tucu)
|
|
|
|
MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific
|
|
counters page for failed job. (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5257. Fix issues in TestContainerLauncherImpl after YARN-617.
|
|
(Omkar Vinit Joshi via vinodkv)
|
|
|
|
MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after
|
|
YARN-716. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead
|
|
of the entire Container in the startContainer call - YARN-684.
|
|
(Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-5299. Fix backward incompatibility for TaskCompletionEvent by
|
|
adding back setTaskID. (Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5296. Fix backward incompatibility for JobControl by adding the
|
|
omitted addJob. (Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5245. Added back constants to JobConf to fix incompatibilities.
|
|
(Zhijie Shen via acmurthy)
|
|
|
|
MAPREDUCE-5297. Updated MR App since BuilderUtils is no longer public
|
|
after YARN-748. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5301. Updated MR code to work with YARN-635 changes of renaming
|
|
YarnRemoteException to YarnException. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching
|
|
multiple compressed map outputs (Nathan Roberts via jlowe)
|
|
|
|
MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe
|
|
via daryn)
|
|
|
|
MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators
|
|
missmatch. (Ivan Mitic via cnauroth)
|
|
|
|
MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar,
|
|
Devaraj K, and B Anil Kumar via jlowe)
|
|
|
|
MAPREDUCE-5334. Fix failing unit tests - TestContainerLauncher,
|
|
TestContainerLauncherImpl. (Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-5325. MR changes related to YARN-727. ClientRMProtocol.getAllApplications
|
|
should accept ApplicationType as a parameter. (Xuan Gong via hitesh)
|
|
|
|
MAPREDUCE-5291. Change MR App to use updated property names in
|
|
container-log4j.properties. (Zhijie Shen via sseth)
|
|
|
|
MAPREDUCE-5303. Changed MR app after moving ProtoBase to package impl.pb via
|
|
YARN-724. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5312. TestRMNMInfo is failing. (sandyr via tucu)
|
|
|
|
MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents
|
|
methods have incompatible signature changes. (kkambatl via tucu)
|
|
|
|
MAPREDUCE-5298. Moved MapReduce services to YARN-530 stricter lifecycle.
|
|
(Steve Loughran via vinodkv)
|
|
|
|
MAPREDUCE-5319. Set user.name in job.xml. (Xuan Gong via acmurthy)
|
|
|
|
MAPREDUCE-5310. MRAM should not normalize allocation request capabilities.
|
|
(tucu)
|
|
|
|
MAPREDUCE-5213. Re-assess TokenCache methods marked @Private.
|
|
(kkambatl via tucu)
|
|
|
|
MAPREDUCE-5412. Update MR app to use multiple containers API of
|
|
ContainerManager after YARN-926. (Jian He via vinodkv)
|
|
|
|
MAPREDUCE-5421. Fixed TestNonExistentJob failure after YARN-873. (Junping Du
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker
|
|
via jlowe)
|
|
|
|
MAPREDUCE-5399. Unnecessary Configuration instantiation in IFileInputStream
|
|
slows down merge. (Stanislav Barton via Sandy Ryza)
|
|
|
|
BREAKDOWN OF HADOOP-8562 SUBTASKS
|
|
|
|
MAPREDUCE-4739. Some MapReduce tests fail to find winutils.
|
|
(Chris Nauroth via suresh)
|
|
|
|
MAPREDUCE-4780. MapReduce distribution build fails on Windows.
|
|
(Chris Nauroth via suresh)
|
|
|
|
MAPREDUCE-4790. MapReduce build script would be more readable using abspath.
|
|
(Chris Nauroth via suresh)
|
|
|
|
MAPREDUCE-4869. Fix TestMapReduceChildJVM. (Chris Nauroth via acmurthy)
|
|
|
|
MAPREDUCE-4870. Fix TestMRJobsWithHistoryService. (Chris Nauroth via acmurthy)
|
|
|
|
MAPREDUCE-4983. Fixed various platform specific assumptions in various tests,
|
|
so that they can pass on Windows too. (Chris Nauroth via vinodkv)
|
|
|
|
HADOOP-9372. Fix bad timeout annotations on tests.
|
|
(Arpit Agarwal via suresh)
|
|
|
|
MAPREDUCE-4885. Streaming tests have multiple failures on Windows. (Chris
|
|
Nauroth via bikas)
|
|
|
|
MAPREDUCE-5177. Use common utils FileUtil#setReadable/Writable/Executable &
|
|
FileUtil#canRead/Write/Execute. (Ivan Mitic via suresh)
|
|
|
|
MAPREDUCE-5349. TestClusterMapReduceTestCase and TestJobName fail on Windows
|
|
in branch-2. (Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5355. MiniMRYarnCluster with localFs does not work on Windows.
|
|
(Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5359. JobHistory should not use File.separator to match timestamp
|
|
in path. (Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5357. Job staging directory owner checking could fail on Windows.
|
|
(Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5360. TestMRJobClient fails on Windows due to path format.
|
|
(Chuan Liu via cnauroth)
|
|
|
|
MAPREDUCE-5366. TestMRAsyncDiskService fails on Windows. (Chuan Liu via
|
|
cnauroth)
|
|
|
|
MAPREDUCE-5187. Create mapreduce command scripts on Windows. (Chuan Liu via
|
|
cnauroth)
|
|
|
|
MAPREDUCE-4374. Fix child task environment variable config and add support
|
|
for Windows. (Chuan Liu via cnauroth)
|
|
|
|
Release 2.0.6-alpha - 08/22/2013
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
Release 2.0.5-alpha - 06/06/2013
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5240 inside of FileOutputCommitter the initialized Credentials cache
|
|
appears to be empty. (vinodkv)
|
|
|
|
Release 2.0.4-alpha - 2013-04-25
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994.
|
|
(Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-5088. MR Client gets an renewer token exception while Oozie is
|
|
submitting a job (Daryn Sharp via cos)
|
|
|
|
MAPREDUCE-5117. Changed MRClientProtocolPBClientImpl to be closeable and thus
|
|
fix failures in renewal of HistoryServer's delegations tokens. (Siddharth
|
|
Seth via vinodkv)
|
|
|
|
MAPREDUCE-5083. MiniMRCluster should use a random component when creating an
|
|
actual cluster (Siddharth Seth via hitesh)
|
|
|
|
MAPREDUCE-5094. Disabled memory monitoring by default in MiniMRYarnCluster
|
|
to avoid some downstream tests failing. (Siddharth Seth via vinodkv)
|
|
|
|
Release 2.0.3-alpha - 2013-02-06
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-4123. Remove the 'mapred groups' command, which is no longer
|
|
supported. (Devaraj K via sseth)
|
|
|
|
MAPREDUCE-4938. Use token request messages defined in hadoop common.
|
|
(suresh)
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-4520. Added support for MapReduce applications to request for
|
|
CPU cores along-with memory post YARN-2. (acmurthy)
|
|
|
|
MAPREDUCE-4810. Added new admin command options for MR AM. (Jerry Chen via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-4049. Experimental api to allow for alternate shuffle plugins.
|
|
(Avner BenHanoch via acmurthy)
|
|
|
|
MAPREDUCE-4807. Allow MapOutputBuffer to be pluggable. (masokan via tucu)
|
|
|
|
MAPREDUCE-4808. Refactor MapOutput and MergeManager to facilitate reuse
|
|
by Shuffle implementations. (masokan via tucu)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-3678. The Map tasks logs should have the value of input
|
|
split it processed. (harsh)
|
|
|
|
MAPREDUCE-4616. Improve javadoc for MultipleOutputs. (Tony Burton via
|
|
acmurthy)
|
|
|
|
HADOOP-8911. CRLF characters in source and text files.
|
|
(Raja Aluri via suresh)
|
|
|
|
MAPREDUCE-4723. Fix warnings found by findbugs 2. (Sandy Ryza via eli)
|
|
|
|
MAPREDUCE-4703. Add the ability to start the MiniMRClientCluster using
|
|
the configurations used before it is being stopped. (ahmed.radwan via tucu)
|
|
|
|
MAPREDUCE-4845. ClusterStatus.getMaxMemory() and getUsedMemory() exist in
|
|
MR1 but not MR2. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-4899. Implemented a MR specific plugin for tracking finished
|
|
applications that YARN's ResourceManager doesn't keep track off anymore
|
|
(Derek Dagit via vinodkv)
|
|
|
|
MAPREDUCE-4920. Use security token protobuf definition from hadoop common.
|
|
(Suresh Srinivas via vinodkv)
|
|
|
|
MAPREDUCE-4907. TrackerDistributedCacheManager issues too many getFileStatus
|
|
calls. (sandyr via tucu)
|
|
|
|
MAPREDUCE-4949. Enable multiple pi jobs to run in parallel. (sandyr via tucu)
|
|
|
|
MAPREDUCE-4809. Change visibility of classes for pluggable sort changes.
|
|
(masokan via tucu)
|
|
|
|
MAPREDUCE-4838. Add additional fields like Locality, Avataar to the
|
|
JobHistory logs. (Zhijie Shen via sseth)
|
|
|
|
MAPREDUCE-4971. Minor extensibility enhancements to Counters &
|
|
FileOutputFormat. (Arun C Murthy via sseth)
|
|
|
|
MAPREDUCE-4977. Documentation for pluggable shuffle and pluggable sort.
|
|
(tucu)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-4893. Fixed MR ApplicationMaster to do optimal assignment of
|
|
containers to get maximum locality. (Bikas Saha via vinodkv)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4272. SortedRanges.Range#compareTo is not spec compliant.
|
|
(Yu Gao via llu)
|
|
|
|
MAPREDUCE-4607. Race condition in ReduceTask completion can result in Task
|
|
being incorrectly failed. (Bikas Saha via tomwhite)
|
|
|
|
MAPREDUCE-4646. Fixed MR framework to send diagnostic information correctly
|
|
to clients in case of failed jobs also. (Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-4674. Hadoop examples secondarysort has a typo
|
|
"secondarysrot" in the usage. (Robert Justice via eli)
|
|
|
|
MAPREDUCE-4681. Fix unit tests broken by HDFS-3910. (acmurthy)
|
|
|
|
MAPREDUCE-4712. mr-jobhistory-daemon.sh doesn't accept --config
|
|
(Vinod Kumar Vavilapalli via tgraves)
|
|
|
|
MAPREDUCE-4654. TestDistCp is ignored. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-4637. Handle TaskAttempt diagnostic updates while in the NEW and
|
|
UNASSIGNED states. (Mayank Bansal via sseth)
|
|
|
|
MAPREDUCE-1806. CombineFileInputFormat does not work with paths not on default FS. (Gera Shegalov via tucu)
|
|
|
|
MAPREDUCE-4777. In TestIFile, testIFileReaderWithCodec relies on
|
|
testIFileWriterWithCodec. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-4800. Cleanup o.a.h.mapred.MapTaskStatus - remove unused
|
|
code. (kkambatl via tucu)
|
|
|
|
MAPREDUCE-4861. Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal.
|
|
(kkambatl via tucu)
|
|
|
|
MAPREDUCE-4856. TestJobOutputCommitter uses same directory as
|
|
TestJobCleanup. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-4895. Fix compilation failure of org.apache.hadoop.mapred.
|
|
gridmix.TestResourceUsageEmulators (Dennis Y via tgraves)
|
|
|
|
MAPREDUCE-4278. Cannot run two local jobs in parallel from the same
|
|
gateway. (Sandy Ryza via tomwhite)
|
|
|
|
MAPREDUCE-1700. User supplied dependencies may conflict with MapReduce
|
|
system JARs. (tomwhite)
|
|
|
|
MAPREDUCE-4936. JobImpl uber checks for cpu are wrong (Arun C Murthy via
|
|
jlowe)
|
|
|
|
MAPREDUCE-4924. flakey test: org.apache.hadoop.mapred.TestClusterMRNotification.testMR.
|
|
(rkanter via tucu)
|
|
|
|
MAPREDUCE-4923. Add toString method to TaggedInputSplit. (sandyr via tucu)
|
|
|
|
MAPREDUCE-4948. Fix a failing unit test TestYARNRunner.testHistoryServerToken.
|
|
(Junping Du via sseth)
|
|
|
|
MAPREDUCE-4803. Remove duplicate copy of TestIndexCache. (Mariappan Asokan
|
|
via sseth)
|
|
|
|
MAPREDUCE-2264. Job status exceeds 100% in some cases.
|
|
(devaraj.k and sandyr via tucu)
|
|
|
|
MAPREDUCE-4969. TestKeyValueTextInputFormat test fails with Open JDK 7.
|
|
(Arpit Agarwal via suresh)
|
|
|
|
MAPREDUCE-4884. Streaming tests fail to start MiniMRCluster due to missing
|
|
queue configuration. (Chris Nauroth via suresh)
|
|
|
|
MAPREDUCE-4953. HadoopPipes misuses fprintf. (Andy Isaacson via atm)
|
|
|
|
Release 2.0.2-alpha - 2012-09-07
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-987. Exposing MiniDFS and MiniMR clusters as a single process
|
|
command-line. (ahmed via tucu)
|
|
|
|
MAPREDUCE-4417. add support for encrypted shuffle (tucu)
|
|
|
|
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
|
|
|
|
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu)
|
|
|
|
MAPREDUCE-4438. Add a simple, generic client to run 'easy' AMs in YARN.
|
|
(Bikas Saha via acmurthy)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-4157. ResourceManager should not kill apps that are well behaved
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4511. Add IFile readahead (ahmed via tucu)
|
|
|
|
MAPREDUCE-4408. allow jobs to set a JAR that is in the distributed cached
|
|
(rkanter via tucu)
|
|
|
|
MAPREDUCE-4440. Changed SchedulerApp and SchedulerNode to be a minimal
|
|
interface to allow schedulers to maintain their own. (acmurthy)
|
|
|
|
MAPREDUCE-4146. Support limits on task status string length and number of
|
|
block locations in branch-2. (Ahmed Radwan via tomwhite)
|
|
|
|
MAPREDUCE-3871. Allow symlinking in LocalJobRunner DistributedCache.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-3921. MR AM should act on node health status changes.
|
|
(Bikas Saha via sseth)
|
|
|
|
MAPREDUCE-4355. Add RunningJob.getJobStatus() (kkambatl via tucu)
|
|
|
|
MAPREDUCE-4427. Added an 'unmanaged' mode for AMs so as to ease
|
|
development of new applications. (Bikas Saha via acmurthy)
|
|
|
|
MAPREDUCE-3289. Make use of fadvise in the NM's shuffle handler.
|
|
(Todd Lipcon and Siddharth Seth via sseth)
|
|
|
|
MAPREDUCE-4580. Change MapReduce to use the yarn-client module.
|
|
(Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-4579. Split TestTaskAttempt into two so as to pass tests on
|
|
jdk7. (Thomas Graves via vinodkv)
|
|
|
|
MAPREDUCE-4638. MR AM supplies MapReduce jars in classpath rather than
|
|
rely on YARN. (acmurthy)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4422. YARN_APPLICATION_CLASSPATH needs a documented default value in
|
|
YarnConfiguration. (ahmed via tucu)
|
|
|
|
MAPREDUCE-4406. Users should be able to specify the MiniCluster ResourceManager
|
|
and JobHistoryServer ports. (ahmed via tucu)
|
|
|
|
MAPREDUCE-4407. Add hadoop-yarn-server-tests-<version>-tests.jar to hadoop dist
|
|
package. (ahmed via tucu)
|
|
|
|
MAPREDUCE-4465. Update description of yarn.nodemanager.address property.
|
|
(bowang via tucu)
|
|
|
|
MAPREDUCE-4342. Distributed Cache gives inconsistent result if cache files
|
|
get deleted from tasktracker. (mayank_bansal via tucu)
|
|
|
|
MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu)
|
|
|
|
MAPREDUCE-4494. TestFifoScheduler failing with Metrics source QueueMetrics,q0=default
|
|
already exists!. (ahmed.radwan via tucu)
|
|
|
|
MAPREDUCE-4484. Incorrect IS_MINI_YARN_CLUSTER property name in YarnConfiguration.
|
|
(ahmed.radwan via tucu)
|
|
|
|
MAPREDUCE-4562. Support for "FileSystemCounter" legacy counter group name
|
|
for compatibility reasons is creating incorrect counter name.
|
|
(Jarek Jarcec Cecho via tomwhite)
|
|
|
|
MAPREDUCE-4068. Jars in lib subdirectory of the submittable JAR are not added to the
|
|
classpath (rkanter via tucu)
|
|
|
|
MAPREDUCE-4577. HDFS-3672 broke
|
|
TestCombineFileInputFormat.testMissingBlocks() test (atm)
|
|
|
|
MAPREDUCE-4470. Fix TestCombineFileInputFormat.testForEmptyFile (ikatsov via tucu)
|
|
|
|
MAPREDUCE-4608. hadoop-mapreduce-client is missing some dependencies.
|
|
(tucu via tomwhite)
|
|
|
|
MAPREDUCE-4610. Support deprecated mapreduce.job.counters.limit property in
|
|
MR2. (tomwhite)
|
|
|
|
MAPREDUCE-4629. Remove JobHistory.DEBUG_MODE (Karthik Kambatla via bobby)
|
|
|
|
MAPREDUCE-4642. MiniMRClientClusterFactory should not use job.setJar() (rkanter via tucu)
|
|
|
|
MAPREDUCE-4148. MapReduce should not have a compile-time dependency on
|
|
HDFS. (tomwhite)
|
|
|
|
MAPREDUCE-4250. hadoop-config.sh missing variable exports, causes Yarn
|
|
jobs to fail with ClassNotFoundException MRAppMaster. (phunt via tucu)
|
|
|
|
MAPREDUCE-4002. MultiFileWordCount job fails if the input path is not
|
|
from default file system. (Bhallamudi Venkata Siva Kamesh via todd)
|
|
|
|
MAPREDUCE-4274 MapOutputBuffer should use native byte order for kvmeta.
|
|
(todd via bobby)
|
|
|
|
MAPREDUCE-4262. NM gives wrong log message saying "Connected to
|
|
ResourceManager" before trying to connect. (Devaraj K via tgraves)
|
|
|
|
MAPREDUCE-4276. Allow setting yarn.nodemanager.delete.debug-delay-sec
|
|
property to "-1" for easier container debugging. (ahmed via tucu)
|
|
|
|
MAPREDUCE-4224. TestFifoScheduler throws
|
|
org.apache.hadoop.metrics2.MetricsException (Devaraj K via tgraves)
|
|
|
|
MAPREDUCE-3493. Add the default mapreduce.shuffle.port property
|
|
to mapred-default.xml (Madhukara Phatak via harsh)
|
|
|
|
MAPREDUCE-4307. TeraInputFormat calls FileSystem.getDefaultBlockSize()
|
|
without a Path - Failure when using ViewFileSystem. (Ahmed Radwan via eli)
|
|
|
|
MAPREDUCE-4313. TestTokenCache doesn't compile due
|
|
TokenCache.getDelegationToken compilation error (bobby)
|
|
|
|
MAPREDUCE-3873. Fixed NodeManagers' decommissioning at RM to accept IP
|
|
addresses also. (xieguiming via vinodkv)
|
|
|
|
MAPREDUCE-4306. Fix distributed shell to work with users other than the one
|
|
running the daemons. (Ahmed Radwan via sseth)
|
|
|
|
MAPREDUCE-4031. Prevent a Node Manager hang during shutdown.
|
|
(Devaraj K via sseth)
|
|
|
|
MAPREDUCE-4336. Distributed Shell fails when used with the CapacityScheduler
|
|
(ahmed via tucu)
|
|
|
|
MAPREDUCE-4290. Fix Yarn Applicaiton Status to MR JobState conversion.
|
|
(Devaraj K via sseth)
|
|
|
|
MAPREDUCE-2289. Permissions race can make getStagingDir fail on local filesystem
|
|
(ahmed via tucu)
|
|
|
|
MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby)
|
|
|
|
MAPREDUCE-4383. HadoopPipes.cc needs to include unistd.h.
|
|
(Andy Isaacson via eli)
|
|
|
|
MAPREDUCE-2739. Update installation docs (remove YarnClientFactory) (bowang via tucu)
|
|
|
|
MAPREDUCE-3993. Graceful handling of codec errors during decompression
|
|
(kkambatl via tucu)
|
|
|
|
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby)
|
|
|
|
MAPREDUCE-4441. Fix build issue caused by MR-3451 (kkambatl via tucu)
|
|
|
|
HADOOP-8499. Lower min.user.id to 500 for the tests.
|
|
(Colin Patrick McCabe via eli)
|
|
|
|
MAPREDUCE-4395. Possible NPE at ClientDistributedCacheManager
|
|
#determineTimestamps (Bhallamudi via bobby)
|
|
|
|
MAPREDUCE-4380. Empty Userlogs directory is getting created under logs
|
|
directory (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4649. Ensure MapReduce JobHistory Daemon doens't assume
|
|
HADOOP_YARN_HOME and HADOOP_MAPRED_HOME are the same. (vinodkv via
|
|
acmurthy)
|
|
|
|
Release 2.0.0-alpha - 05-23-2012
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-3545. Remove Avro RPC. (suresh)
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-2887. Due to HADOOP-7524, change RPC to allow multiple protocols
|
|
including multuple versions of the same protocol (Sanjay Radia)
|
|
|
|
MAPREDUCE-2934. MR portion of HADOOP-7607 - Simplify the RPC proxy cleanup
|
|
process (atm)
|
|
|
|
HADOOP-7862. MR changes to work with HADOOP 7862: Move the support for
|
|
multiple protocols to lower layer so that Writable, PB and Avro can all
|
|
use it (Sanjay Radia)
|
|
|
|
MAPREDUCE-3909 Javadoc the Service interfaces (stevel)
|
|
|
|
MAPREDUCE-3885. Avoid an unnecessary copy for all requests/responses in
|
|
MRs ProtoOverHadoopRpcEngine. (Devaraj Das via sseth)
|
|
|
|
MAPREDUCE-3935. Annotate Counters.Counter and Counters.Group as @Public.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-3991. Streaming FAQ has some wrong instructions about input files
|
|
splitting. (harsh)
|
|
|
|
MAPREDUCE-3773. Add queue metrics with buckets for job run times. (omalley
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3970. Add ServiceOperations class to aid working with Services
|
|
(stevel)
|
|
|
|
MAPREDUCE-3353. Add a channel between RM and AM to get information on
|
|
nodes. (Bikas Saha via acmurthy)
|
|
|
|
MAPREDUCE-3955. Change MR to use ProtobufRpcEngine from hadoop-common
|
|
instead of ProtoOverHadoopRpcEngine. (Jitendra Nath Pandey via sseth)
|
|
|
|
MAPREDUCE-4103. Fix HA docs for changes to shell command fencer args (todd)
|
|
|
|
MAPREDUCE-4093. Improve RM WebApp start up when proxy address is not set
|
|
(Devaraj K vai bobby)
|
|
|
|
MAPREDUCE-4138. Reduce memory usage of counters due to non-static nested
|
|
classes. (tomwhite)
|
|
|
|
MAPREDUCE-3883. Document yarn.nodemanager.delete.debug-delay-sec
|
|
configuration property (Eugene Koontz via tgraves)
|
|
|
|
MAPREDUCE-4219. make default container-executor.conf.dir be a path
|
|
relative to the container-executor binary. (rvs via tucu)
|
|
|
|
MAPREDUCE-4205. retrofit all JVM shutdown hooks to use ShutdownHookManager
|
|
(tucu)
|
|
|
|
HADOOP-8285 MR changes for Use ProtoBuf for RpcPayLoadHeader (sanjay radia)
|
|
|
|
MAPREDUCE-2220. Fix new API FileOutputFormat-related typos in
|
|
mapred-default.xml (Rui Kubo via harsh)
|
|
|
|
MAPREDUCE-3907. Document entries mapred-default.xml for the
|
|
jobhistory server. (Eugene Koontz via harsh)
|
|
|
|
MAPREDUCE-3906. Fix inconsistency in documentation regarding
|
|
mapreduce.jobhistory.principal. (Eugene Koontz via harsh)
|
|
|
|
MAPREDUCE-4432. Confusing warning message when GenericOptionsParser
|
|
is not used. (Gabriel Reid via harsh)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-3740. Fixed broken mapreduce compilation after the patch for
|
|
HADOOP-7965. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3818. Fixed broken compilation in TestSubmitJob after the patch
|
|
for HDFS-2895. (Suresh Srinivas via vinodkv)
|
|
|
|
MAPREDUCE-2942. TestNMAuditLogger.testNMAuditLoggerWithIP failing (Thomas
|
|
Graves via mahadev)
|
|
|
|
MAPREDUCE-3933. Failures because MALLOC_ARENA_MAX is not set (ahmed via tucu)
|
|
|
|
MAPREDUCE-3728. ShuffleHandler can't access results when configured in a
|
|
secure mode (ahmed via tucu)
|
|
|
|
MAPREDUCE-3952. In MR2, when Total input paths to process == 1,
|
|
CombinefileInputFormat.getSplits() returns 0 split. (zhenxiao via tucu)
|
|
|
|
MAPREDUCE-3578. Starting nodemanager as root gives "Unknown -jvm option"
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-3348. Fixed a bug in MR client to redirect to JobHistoryServer
|
|
correctly when RM forgets the app. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3974. TestSubmitJob in MR1 tests doesn't compile after HDFS-162
|
|
merge. (atm)
|
|
|
|
MAPREDUCE-4007. JobClient getJob(JobID) should return NULL if the job
|
|
does not exist (for backwards compatibility) (tucu)
|
|
|
|
MAPREDUCE-3431 NPE in Resource Manager shutdown. (stevel)
|
|
|
|
MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby)
|
|
|
|
MAPREDUCE-3992. Reduce fetcher doesn't verify HTTP status code of response
|
|
(todd)
|
|
|
|
MAPREDUCE-4066. Use default value when fetching MR_AM_STAGING_DIR
|
|
(xieguiming via harsh)
|
|
|
|
MAPREDUCE-3377. Added a unit test to ensure OutputCommitter.checkOutputSpecs
|
|
is called prior to copying job.xml. (Jane Chen via acmurthy)
|
|
|
|
MAPREDUCE-4081. TestMROutputFormat.java does not compile (Jason Lowe via
|
|
bobby)
|
|
|
|
MAPREDUCE-4082. hadoop-mapreduce-client-app's mrapp-generated-classpath
|
|
file should not be in the module JAR (tucu)
|
|
|
|
MAPREDUCE-3916. various issues with running yarn proxyserver (devaraj via tucu)
|
|
|
|
MAPREDUCE-4091. tools testcases failing because of MAPREDUCE-4082 (tucu)
|
|
|
|
MAPREDUCE-4098. TestMRApps testSetClasspath fails (tucu)
|
|
|
|
MAPREDUCE-4097. tools testcases fail because missing mrapp-generated-classpath
|
|
file in classpath (rvs via tucu)
|
|
|
|
MAPREDUCE-4113. Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
|
|
(Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4112. Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
|
|
(Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4111. Fix tests in org.apache.hadoop.mapred.TestJobName (Devaraj
|
|
K via bobby)
|
|
|
|
MAPREDUCE-4110. Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath &
|
|
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers (Devaraj K via
|
|
bobby)
|
|
|
|
MAPREDUCE-4105. Yarn RackResolver ignores rack configurations.
|
|
(Ahmed Radwan via tomwhite)
|
|
|
|
MAPREDUCE-3869. Fix classpath for DistributedShell application. (Devaraj K
|
|
via sseth)
|
|
|
|
MAPREDUCE-4057. Update RAID for the HA and fsdataset changes. (Devaraj K
|
|
via szetszwo)
|
|
|
|
MAPREDUCE-4076. Stream job fails with ZipException when use yarn jar
|
|
command (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4108. Fix tests in org.apache.hadoop.util.TestRunJar
|
|
(Devaraj K via tgraves)
|
|
|
|
MAPREDUCE-4107. Fix tests in org.apache.hadoop.ipc.TestSocketFactory
|
|
(Devaraj K via tgraves)
|
|
|
|
MAPREDUCE-4147. YARN should not have a compile-time dependency on HDFS.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-4008. ResourceManager throws MetricsException on start up
|
|
saying QueueMetrics MBean already exists (Devaraj K via tgraves)
|
|
|
|
MAPREDUCE-3867. MiniMRYarn/MiniYarn uses fixed ports (tucu)
|
|
|
|
MAPREDUCE-4141. clover integration broken, also mapreduce poms are
|
|
pulling in clover as a dependency. (phunt via tucu)
|
|
|
|
MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml.
|
|
(phunt via tomwhite)
|
|
|
|
MAPREDUCE-4202. TestYarnClientProtocolProvider is broken (Daryn Sharp via
|
|
bobby)
|
|
|
|
MAPREDUCE-3173. MRV2 UI doesn't work properly without internet (Devaraj K
|
|
via bobby)
|
|
|
|
MAPREDUCE-3958. RM: Remove RMNodeState and replace it with NodeState
|
|
(Bikas Saha via bobby)
|
|
|
|
MAPREDUCE-4231. Update RAID to use the new BlockCollection interface.
|
|
(szetszwo)
|
|
|
|
MAPREDUCE-4483. 2.0 build does not work (John George via bobby)
|
|
|
|
MAPREDUCE-4444. nodemanager fails to start when one of the local-dirs is
|
|
bad (Jason Lowe via bobby)
|
|
|
|
Release 0.23.10 - UNRELEASED
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5640. Rename TestLineRecordReader in jobclient module (Jason Lowe
|
|
via jeagles)
|
|
|
|
MAPREDUCE-3191. docs for map output compression incorrectly reference
|
|
SequenceFile (Chen He via jeagles)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-1981. Improve getSplits performance by using listLocatedStatus
|
|
(Hairong Kuang and Jason Lowe via jlowe)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-3193. FileInputFormat doesn't read files recursively in the
|
|
input path dir (Devaraj K via jlowe)
|
|
|
|
MAPREDUCE-5380. Invalid mapred command should return non-zero exit code
|
|
(Stephen Chu via jlowe)
|
|
|
|
MAPREDUCE-5317. Stale files left behind for failed jobs (Ravi Prakash via
|
|
jlowe)
|
|
|
|
MAPREDUCE-5251. Reducer should not implicate map attempt if it has
|
|
insufficient space to fetch map output (Ashwin Shankar via jlowe)
|
|
|
|
MAPREDUCE-5419. TestSlive is getting FileNotFound Exception (Robert Parker
|
|
via jlowe)
|
|
|
|
MAPREDUCE-5440. TestCopyCommitter Fails on JDK7 (Robert Parker via jlowe)
|
|
|
|
MAPREDUCE-5001. LocalJobRunner has race condition resulting in job
|
|
failures (Sandy Ryza via jlowe)
|
|
|
|
MAPREDUCE-5475. MRClientService does not verify ACLs properly (jlowe)
|
|
|
|
MAPREDUCE-5504. mapred queue -info inconsistent with types (Kousuke Saruta
|
|
via tgraves)
|
|
|
|
MAPREDUCE-5513. ConcurrentModificationException in JobControl (Robert
|
|
Parker via jlowe)
|
|
|
|
MAPREDUCE-5586. TestCopyMapper#testCopyFailOnBlockSizeDifference fails when
|
|
run from hadoop-tools/hadoop-distcp directory (jeagles)
|
|
|
|
MAPREDUCE-5587. TestTextOutputFormat fails on JDK7 (jeagles)
|
|
|
|
MAPREDUCE-5373. TestFetchFailure.testFetchFailureMultipleReduces could fail
|
|
intermittently (jeagles)
|
|
|
|
Release 0.23.9 - 2013-07-08
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5084. fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and
|
|
org.apache.hadoop.mapreduce.v2.hs.webapp (Aleksey Gorshkov via jeagles)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-5268. Improve history server startup performance (Karthik
|
|
Kambatla via jlowe)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5308. Shuffling to memory can get out-of-sync when fetching
|
|
multiple compressed map outputs (Nathan Roberts via jlowe)
|
|
|
|
MAPREDUCE-5315. DistCp reports success even on failure. (mithun and jlowe
|
|
via daryn)
|
|
|
|
MAPREDUCE-4019. -list-attempt-ids is not working (Ashwin Shankar,
|
|
Devaraj K, and B Anil Kumar via jlowe)
|
|
|
|
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
|
|
task-state (Ashwin Shankar via jlowe)
|
|
|
|
Release 0.23.8 - 2013-06-05
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes
|
|
are different on source/target (Mithun Radhakrishnan via kihwal)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI
|
|
(Aleksey Gorshkov via tgraves)
|
|
|
|
MAPREDUCE-5147. Maven build should create
|
|
hadoop-mapreduce-client-app-VERSION.jar directly (Robert Parker via tgraves)
|
|
|
|
MAPREDUCE-4927. Historyserver 500 error due to NPE when accessing specific
|
|
counters page for failed job. (Ashwin Shankar via jlowe)
|
|
|
|
Release 0.23.7 - 2013-04-18
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-4905. test org.apache.hadoop.mapred.pipes
|
|
(Aleksey Gorshkov via bobby)
|
|
|
|
MAPREDUCE-4989. JSONify DataTables input data for Attempts page (Ravi
|
|
Prakash via jlowe)
|
|
|
|
MAPREDUCE-5027. Shuffle does not limit number of outstanding connections
|
|
(Robert Parker via jeagles)
|
|
|
|
MAPREDUCE-4972. Coverage fixing for org.apache.hadoop.mapreduce.jobhistory
|
|
(Aleksey Gorshkov via bobby)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-4946. Fix a performance problem for large jobs by reducing the
|
|
number of map completion event type conversions. (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-4822. Unnecessary conversions in History Events. (Chu Tong via
|
|
jlowe)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4458. Warn if java.library.path is used for AM or Task
|
|
(Robert Parker via jeagles)
|
|
|
|
MAPREDUCE-4992. AM hangs in RecoveryService when recovering tasks with
|
|
speculative attempts (Robert Parker via jlowe)
|
|
|
|
MAPREDUCE-5009. Killing the Task Attempt slated for commit does not clear
|
|
the value from the Task commitAttempt member (Robert Parker via jeagles)
|
|
|
|
MAPREDUCE-4871. AM uses mapreduce.jobtracker.split.metainfo.maxsize but
|
|
mapred-default has mapreduce.job.split.metainfo.maxsize (Jason Lowe via
|
|
jeagles)
|
|
|
|
MAPREDUCE-4794. DefaultSpeculator generates error messages on normal
|
|
shutdown (Jason Lowe via jeagles)
|
|
|
|
MAPREDUCE-5043. Fetch failure processing can cause AM event queue to
|
|
backup and eventually OOM (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-5023. History Server Web Services missing Job Counters (Ravi
|
|
Prakash via tgraves)
|
|
|
|
MAPREDUCE-5060. Fetch failures that time out only count against the first
|
|
map task (Robert Joseph Evans via jlowe)
|
|
|
|
MAPREDUCE-5042. Reducer unable to fetch for a map task that was recovered
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-5053. java.lang.InternalError from decompression codec cause
|
|
reducer to fail (Robert Parker via jeagles)
|
|
|
|
MAPREDUCE-4991. coverage for gridmix (Aleksey Gorshkov via tgraves)
|
|
|
|
MAPREDUCE-5007. fix coverage org.apache.hadoop.mapreduce.v2.hs (Aleksey
|
|
Gorshkov via tgraves)
|
|
|
|
MAPREDUCE-5137. AM web UI: clicking on Map Task results in 500 error
|
|
(Thomas Graves via jlowe)
|
|
|
|
Release 0.23.6 - 2013-02-06
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-4811. JobHistoryServer should show when it was started in WebUI
|
|
About page (Ravi Prakash via jlowe)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4802. Takes a long time to load the task list on the AM for
|
|
large jobs (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4764. repair TestBinaryTokenFile (Ivan A. Veselovsky via bobby)
|
|
|
|
MAPREDUCE-4825. JobImpl.finished doesn't expect ERROR as a final job state
|
|
(jlowe via bobby)
|
|
|
|
MAPREDUCE-4817. Hardcoded task ping timeout kills tasks localizing large
|
|
amounts of data (tgraves)
|
|
|
|
MAPREDUCE-4836. Elapsed time for running tasks on AM web UI tasks page is 0
|
|
(Ravi Prakash via jeagles)
|
|
|
|
MAPREDUCE-4842. Shuffle race can hang reducer (Mariappan Asokan via jlowe)
|
|
|
|
MAPREDUCE-4833. Task can get stuck in FAIL_CONTAINER_CLEANUP (Robert
|
|
Parker via jlowe)
|
|
|
|
MAPREDUCE-4793. Problem with adding resources when using both -files and
|
|
-file to hadoop streaming (jlowe)
|
|
|
|
MAPREDUCE-4890. Invalid TaskImpl state transitions when task fails while
|
|
speculating (jlowe)
|
|
|
|
MAPREDUCE-4902. Fix typo "receievd" should be "received" in log output
|
|
(Albert Chu via jlowe)
|
|
|
|
MAPREDUCE-4813. AM timing out during job commit (jlowe via bobby)
|
|
|
|
MAPREDUCE-4279. getClusterStatus() fails with null pointer exception when
|
|
running jobs in local mode (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4832. MR AM can get in a split brain situation (jlowe)
|
|
|
|
MAPREDUCE-4894. Renewal / cancellation of JobHistory tokens (Siddharth
|
|
Seth via tgraves)
|
|
|
|
MAPREDUCE-4819. AM can rerun job after reporting final job status to the
|
|
client (bobby and Bikas Saha via bobby)
|
|
|
|
MAPREDUCE-4913. TestMRAppMaster#testMRAppMasterMissingStaging occasionally
|
|
exits (Jason Lowe via tgraves)
|
|
|
|
MAPREDUCE-4848. TaskAttemptContext cast error during AM recovery (Jerry
|
|
Chen via jlowe)
|
|
|
|
MAPREDUCE-4921. JobClient should acquire HS token with RM principal
|
|
(daryn via bobby)
|
|
|
|
MAPREDUCE-4934. Maven RAT plugin is not checking all source files (tgraves)
|
|
|
|
MAPREDUCE-4678. Running the Pentomino example with defaults throws
|
|
java.lang.NegativeArraySizeException (Chris McConnell via harsh)
|
|
|
|
MAPREDUCE-4925. The pentomino option parser may be buggy.
|
|
(Karthik Kambatla via harsh)
|
|
|
|
Release 0.23.5 - 2012-11-28
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-4596. Split StateMachine state from states seen by MRClientProtocol
|
|
for Job, Task and TaskAttempt. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-4752. Reduce MR AM memory usage through String Interning (Robert
|
|
Evans via tgraves)
|
|
|
|
MAPREDUCE-4266. remove Ant remnants from MR (tgraves via bobby)
|
|
|
|
MAPREDUCE-4666. JVM metrics for history server (jlowe via jeagles)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-4720. Browser thinks History Server main page JS is taking too
|
|
long (Ravi Prakash via bobby)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4554. Job Credentials are not transmitted if security is turned
|
|
off (Benoy Antony via bobby)
|
|
|
|
MAPREDUCE-4705. Fix a bug in job history lookup, which makes older jobs
|
|
inaccessible despite the presence of a valid history file. (Jason Lowe
|
|
via sseth)
|
|
|
|
MAPREDUCE-4521. mapreduce.user.classpath.first incompatibility with 0.20/1.x
|
|
(Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4721. Task startup time in JHS is same as job startup time.
|
|
(Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4479. Fix parameter order in assertEquals() in
|
|
TestCombineInputFileFormat.java (Mariappan Asokan via bobby)
|
|
|
|
MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many
|
|
reducers complete consecutively. (Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-4740. only .jars can be added to the Distributed Cache
|
|
classpath. (Robert Joseph Evans via jlowe)
|
|
|
|
MAPREDUCE-4229. Intern counter names in the JT (Miomir Boljanovic and bobby via daryn)
|
|
|
|
MAPREDUCE-4741. WARN and ERROR messages logged during normal AM shutdown.
|
|
(Vinod Kumar Vavilapalli via jlowe)
|
|
|
|
MAPREDUCE-4730. Fix Reducer's EventFetcher to scale the map-completion
|
|
requests slowly to avoid HADOOP-8942. (Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-4748. Invalid event: T_ATTEMPT_SUCCEEDED at SUCCEEDED. (jlowe)
|
|
|
|
MAPREDUCE-4724. job history web ui applications page should be sorted to
|
|
display last app first (tgraves via bobby)
|
|
|
|
MAPREDUCE-4746. The MR Application Master does not have a config to set
|
|
environment variables (Rob Parker via bobby)
|
|
|
|
MAPREDUCE-4729. job history UI not showing all job attempts. (Vinod
|
|
Kumar Vavilapalli via jlowe)
|
|
|
|
MAPREDUCE-4763 repair test TestUmbilicalProtocolWithJobToken (Ivan A.
|
|
Veselovsky via bobby)
|
|
|
|
MAPREDUCE-4771. KeyFieldBasedPartitioner not partitioning properly when
|
|
configured (jlowe via bobby)
|
|
|
|
MAPREDUCE-4772. Fetch failures can take way too long for a map to be
|
|
restarted (bobby)
|
|
|
|
MAPREDUCE-4782. NLineInputFormat skips first line of last InputSplit
|
|
(Mark Fuhs via bobby)
|
|
|
|
MAPREDUCE-4774. JobImpl does not handle asynchronous task events in FAILED
|
|
state (jlowe via bobby)
|
|
|
|
MAPREDUCE-4751. AM stuck in KILL_WAIT for days (vinodkv via bobby)
|
|
|
|
MAPREDUCE-4787. TestJobMonitorAndPrint is broken (Rob Parker via bobby)
|
|
|
|
MAPREDUCE-4425. Speculation + Fetch failures can lead to a hung job (jlowe
|
|
via bobby)
|
|
|
|
MAPREDUCE-4786. Job End Notification retry interval is 5 milliseconds by
|
|
default (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4517. Too many INFO messages written out during AM to RM heartbeat
|
|
(Jason Lowe via tgraves)
|
|
|
|
MAPREDUCE-4797. LocalContainerAllocator can loop forever trying to contact
|
|
the RM (jlowe via bobby)
|
|
|
|
MAPREDUCE-4801. ShuffleHandler can generate large logs due to prematurely
|
|
closed channels (jlowe via bobby)
|
|
|
|
Release 0.23.4
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-2786. Add compression option for TestDFSIO.
|
|
(Plamen Jeliazkov via shv)
|
|
|
|
MAPREDUCE-4645. Provide a random seed to Slive to make the sequence
|
|
of file names deterministic. (Ravi Prakash via shv)
|
|
|
|
MAPREDUCE-4651. Benchmarking random reads with DFSIO. (shv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4647. We should only unjar jobjar if there is a lib directory
|
|
in it. (Robert Evans via tgraves)
|
|
|
|
MAPREDUCE-4691. Historyserver can report "Unknown job" after RM says job
|
|
has completed (Robert Joseph Evans via jlowe)
|
|
|
|
MAPREDUCE-4689. JobClient.getMapTaskReports on failed job results in NPE
|
|
(jlowe via bobby)
|
|
|
|
Release 0.23.3
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-4072. User set java.library.path seems to overwrite default
|
|
creating problems native lib loading (Anupam Seth via bobby)
|
|
|
|
MAPREDUCE-3812. Lower default allocation sizes, fix allocation
|
|
configurations and document them (Harsh J via bobby)
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-4059. The history server should have a separate pluggable
|
|
storage/query interface. (Robert Evans via tgraves)
|
|
|
|
MAPREDUCE-3942. Randomize master key generation for
|
|
ApplicationTokenSecretManager and roll it every so often. (Vinod Kumar
|
|
Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-4151. RM scheduler web page should filter apps to those that
|
|
are relevant to scheduling (Jason Lowe via tgraves)
|
|
|
|
MAPREDUCE-4134. Remove references of mapred.child.ulimit etc. since they
|
|
are not being used any more (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-3972. Fix locking and exception issues in JobHistory server.
|
|
(Robert Joseph Evans via sseth)
|
|
|
|
MAPREDUCE-4161. create sockets consistently (Daryn Sharp via bobby)
|
|
|
|
MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.
|
|
(bobby via tgraves)
|
|
|
|
MAPREDUCE-4210. Expose listener address for WebApp (Daryn Sharp via bobby)
|
|
|
|
MAPREDUCE-4162. Correctly set token service (Daryn Sharp via bobby)
|
|
|
|
MAPREDUCE-4301. Dedupe some strings in MRAM for memory savings
|
|
(bobby via tgraves)
|
|
|
|
MAPREDUCE-4267. mavenize pipes (tgraves via bobby)
|
|
|
|
MAPREDUCE-4375. Show Configuration Tracability in MR UI (bobby
|
|
via tgraves)
|
|
|
|
MAPREDUCE-4569. Fixed TestHsWebServicesJobsQuery to pass on JDK7 by not
|
|
depending on test order. (Thomas Graves via vinodkv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-3850. Avoid redundant calls for tokens in TokenCache (Daryn
|
|
Sharp via bobby)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-4092. commitJob Exception does not fail job (Jon Eagles via
|
|
bobby)
|
|
|
|
MAPREDUCE-4089. Hung Tasks never time out. (Robert Evans via tgraves)
|
|
|
|
MAPREDUCE-4024. RM webservices can't query on finalStatus (Tom Graves
|
|
via bobby)
|
|
|
|
MAPREDUCE-4060. Multiple SLF4J binding warning (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3983. TestTTResourceReporting can fail, and should just be
|
|
deleted (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4012 Hadoop Job setup error leaves no useful info to users
|
|
(when LinuxTaskController is used). (tgraves)
|
|
|
|
MAPREDUCE-4062. AM Launcher thread can hang forever (tgraves via bobby)
|
|
|
|
MAPREDUCE-3988. mapreduce.job.local.dir doesn't point to a single directory
|
|
on a node. (Eric Payne via bobby)
|
|
|
|
MAPREDUCE-3999. Tracking link gives an error if the AppMaster hasn't
|
|
started yet (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4020. Web services returns incorrect JSON for deep queue tree
|
|
(Anupam Seth via tgraves)
|
|
|
|
MAPREDUCE-3672. Killed maps shouldn't be counted towards
|
|
JobCounter.NUM_FAILED_MAPS. (Anupam Seth via tgraves)
|
|
|
|
MAPREDUCE-3682 Tracker URL says AM tasks run on localhost.
|
|
(Ravi Prakash via tgraves)
|
|
|
|
MAPREDUCE-3082. Archive command take wrong path for input file with current
|
|
directory (John George via bobby)
|
|
|
|
MAPREDUCE-3650. testGetTokensForHftpFS() fails (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail
|
|
(Ravi Prakash via tgraves)
|
|
|
|
MAPREDUCE-4073. CS assigns multiple off-switch containers when using
|
|
multi-level-queues (Siddharth Seth via bobby)
|
|
|
|
MAPREDUCE-4051. Remove the empty hadoop-mapreduce-project/assembly/all.xml
|
|
file (Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4117. mapred job -status throws NullPointerException (Devaraj K
|
|
via bobby)
|
|
|
|
MAPREDUCE-4099. ApplicationMaster may fail to remove staging directory
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4017. Add jobname to jobsummary log (tgraves and Koji Noguchi
|
|
via bobby)
|
|
|
|
MAPREDUCE-4040. History links should use hostname rather than IP address.
|
|
(Bhallamudi Venkata Siva Kamesh via sseth)
|
|
|
|
MAPREDUCE-4099 amendment. ApplicationMaster will remove staging directory
|
|
after the history service is stopped. (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-3932. Fix the TaskAttempt state machine to handle
|
|
CONTIANER_LAUNCHED and CONTIANER_LAUNCH_FAILED events in additional
|
|
states. (Robert Joseph Evans via sseth)
|
|
|
|
MAPREDUCE-4140. mapreduce classes incorrectly importing
|
|
"clover.org.apache.*" classes. (Patrick Hunt via tomwhite)
|
|
|
|
MAPREDUCE-4050. For tasks without assigned containers, changes the node
|
|
text on the UI to N/A instead of a link to null. (Bhallamudi Venkata Siva
|
|
Kamesh via sseth)
|
|
|
|
MAPREDUCE-4128. AM Recovery expects all attempts of a completed task to
|
|
also be completed. (Bikas Saha via bobby)
|
|
|
|
MAPREDUCE-4144. Fix a NPE in the ResourceManager when handling node
|
|
updates. (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-4156. ant build fails compiling JobInProgress (tgraves)
|
|
|
|
MAPREDUCE-4160. some mrv1 ant tests fail with timeout - due to 4156
|
|
(tgraves)
|
|
|
|
MAPREDUCE-4074. Client continuously retries to RM When RM goes down
|
|
before launching Application Master (xieguiming via tgraves)
|
|
|
|
MAPREDUCE-4159. Job is running in Uber mode after setting
|
|
"mapreduce.job.ubertask.maxreduces" to zero (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4165. Committing is misspelled as commiting in task logs
|
|
(John Eagles via bobby)
|
|
|
|
MAPREDUCE-4129. Lots of unneeded counters log messages (Ahmed Radwan via
|
|
bobby)
|
|
|
|
MAPREDUCE-3947. yarn.app.mapreduce.am.resource.mb not documented
|
|
(Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4190. Improve web UI for task attempts userlog link (Tom Graves
|
|
via bobby)
|
|
|
|
MAPREDUCE-4133. MR over viewfs is broken (John George via bobby)
|
|
|
|
MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection
|
|
(Jonathan Eagles via bobby)
|
|
|
|
MAPREDUCE-3613. web service calls header contains 2 content types
|
|
(tgraves)
|
|
|
|
MAPREDUCE-4169. Container Logs appear in unsorted order (Jonathan Eagles
|
|
via bobby)
|
|
|
|
MAPREDUCE-4189. TestContainerManagerSecurity is failing (Devaraj K via
|
|
bobby)
|
|
|
|
MAPREDUCE-4209. junit dependency in hadoop-mapreduce-client is missing
|
|
scope test (Radim Kolar via bobby)
|
|
|
|
MAPREDUCE-4206. Sorting by Last Health-Update on the RM nodes page sorts
|
|
does not work correctly (Jonathon Eagles via tgraves)
|
|
|
|
MAPREDUCE-4212. TestJobClientGetJob sometimes fails
|
|
(Daryn Sharp via tgraves)
|
|
|
|
MAPREDUCE-4211. Error conditions (missing appid, appid not found) are
|
|
masked in the RM app page (Jonathan Eagles via bobby)
|
|
|
|
MAPREDUCE-4163. consistently set the bind address (Daryn Sharp via bobby)
|
|
|
|
MAPREDUCE-4048. NullPointerException exception while accessing the
|
|
Application Master UI (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4220. RM apps page starttime/endtime sorts are incorrect
|
|
(Jonathan Eagles via bobby)
|
|
|
|
MAPREDUCE-4226. ConcurrentModificationException in FileSystemCounterGroup.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-4215. RM app page shows 500 error on appid parse error
|
|
(Jonathon Eagles via tgraves)
|
|
|
|
MAPREDUCE-4237. TestNodeStatusUpdater can fail if localhost has a domain
|
|
associated with it (bobby)
|
|
|
|
MAPREDUCE-4233. NPE can happen in RMNMNodeInfo. (bobby)
|
|
|
|
MAPREDUCE-4238. mavenize data_join. (tgraves)
|
|
|
|
MAPREDUCE-4102. job counters not available in Jobhistory webui for
|
|
killed jobs (Bhallamudi Venkata Siva Kamesh via tgraves)
|
|
|
|
MAPREDUCE-3543. Mavenize Gridmix. (tgraves)
|
|
|
|
MAPREDUCE-4197. Include the hsqldb jar in the hadoop-mapreduce tar
|
|
file (Ravi Prakash via tgraves)
|
|
|
|
MAPREDUCE-4269. documentation: Gridmix has javadoc warnings in
|
|
StressJobFactory (Jonathon Eagles via tgraves).
|
|
|
|
MAPREDUCE-3870. Invalid App Metrics
|
|
(Bhallamudi Venkata Siva Kamesh via tgraves).
|
|
|
|
MAPREDUCE-4152. map task left hanging after AM dies trying to connect to RM
|
|
(Tom Graves via bobby)
|
|
|
|
MAPREDUCE-4297. Usersmap file in gridmix should not fail on empty lines
|
|
(Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-4302. NM goes down if error encountered during log aggregation
|
|
(Daryn Sharp via bobby)
|
|
|
|
MAPREDUCE-3350. Per-app RM page should have the list of application-attempts
|
|
like on the app JHS page (Jonathon Eagles via tgraves)
|
|
|
|
MAPREDUCE-3842. Stop webpages from automatic refreshing (tgraves)
|
|
|
|
MAPREDUCE-3927. Shuffle hang when set map.failures.percent
|
|
(Bhallamudi Venkata Siva Kamesh via tgraves)
|
|
|
|
MAPREDUCE-4311. Capacity scheduler.xml does not accept decimal values for
|
|
capacity and maximum-capacity settings (Karthik Kambatla via tgraves)
|
|
|
|
MAPREDUCE-4341. add types to capacity scheduler properties documentation
|
|
(Karthik Kambatla via tgraves)
|
|
|
|
MAPREDUCE-4270. Move the data_join test classes to the correct path.
|
|
(Thomas Graves via sseth)
|
|
|
|
MAPREDUCE-3889. job client tries to use /tasklog interface, but that
|
|
doesn't exist anymore (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4320. gridmix mainClass wrong in pom.xml (tgraves)
|
|
|
|
MAPREDUCE-4295. RM crashes due to DNS issue (tgraves)
|
|
|
|
MAPREDUCE-4228. mapreduce.job.reduce.slowstart.completedmaps is not working
|
|
properly (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4392. Counters.makeCompactString() changed behavior from 0.20
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4384. Race conditions in IndexCache (Kihwal Lee via tgraves)
|
|
|
|
MAPREDUCE-4387. RM gets fatal error and exits during TestRM
|
|
(Kihwal Lee via tgraves)
|
|
|
|
MAPREDUCE-4379. Node Manager throws java.lang.OutOfMemoryError: Java heap
|
|
space due to org.apache.hadoop.fs.LocalDirAllocator.contexts (Devaraj K
|
|
via bobby)
|
|
|
|
MAPREDUCE-4402. TestFileInputFormat fails intermittently (Jason Lowe via
|
|
bobby)
|
|
|
|
MAPREDUCE-4300. OOM in AM can turn it into a zombie. (Robert Evans via
|
|
tgraves)
|
|
|
|
MAPREDUCE-4252. MR2 job never completes with 1 pending task (Tom White via
|
|
bobby)
|
|
|
|
MAPREDUCE-3940. ContainerTokens should have an expiry interval. (Siddharth
|
|
Seth and Vinod Kumar Vavilapalli via vinodkv)
|
|
|
|
MAPREDUCE-4419. ./mapred queue -info <queuename> -showJobs displays all
|
|
the jobs irrespective of <queuename> (Devaraj K via bobby)
|
|
|
|
MAPREDUCE-4299. Terasort hangs with MR2 FifoScheduler (Tom White via
|
|
bobby)
|
|
|
|
MAPREDUCE-4437. Race in MR ApplicationMaster can cause reducers to never be
|
|
scheduled (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4449. Incorrect MR_HISTORY_STORAGE property name in JHAdminConfig
|
|
(Ahmed Radwan via bobby)
|
|
|
|
MAPREDUCE-4283. Display tail of aggregated logs by default (Jason Lowe via
|
|
bobby)
|
|
|
|
MAPREDUCE-4448. Fix NM crash during app cleanup if aggregation didn't
|
|
init. (Jason Lowe via daryn)
|
|
|
|
MAPREDUCE-3893. allow capacity scheduler configs maximum-applications and
|
|
maximum-am-resource-percent configurable on a per queue basis (tgraves via
|
|
bobby)
|
|
|
|
MAPREDUCE-4467. IndexCache failures due to missing synchronization
|
|
(Kihwal Lee via tgraves)
|
|
|
|
MAPREDUCE-4423. Potential infinite fetching of map output (Robert Evans
|
|
via tgraves)
|
|
|
|
MAPREDUCE-4456. LocalDistributedCacheManager can get an
|
|
ArrayIndexOutOfBounds when creating symlinks (Robert Evans via tgraves)
|
|
|
|
MAPREDUCE-4496. AM logs link is missing user name (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4493. Distibuted Cache Compatability Issues (Robert Evans
|
|
via tgraves)
|
|
|
|
MAPREDUCE-4492. Configuring total queue capacity between 100.5 and 99.5 at
|
|
perticular level is sucessfull (Mayank Bansal via bobby)
|
|
|
|
MAPREDUCE-4457. mr job invalid transition TA_TOO_MANY_FETCH_FAILURE at
|
|
FAILED (Robert Evans via tgraves)
|
|
|
|
MAPREDUCE-4234. SortValidator.java is incompatible with multi-user or
|
|
parallel use (due to a /tmp file with static name) (Robert Evans via
|
|
jeagles)
|
|
|
|
MAPREDUCE-4504. SortValidator writes to wrong directory (Robert Evans
|
|
via tgraves)
|
|
|
|
MAPREDUCE-4503. Should throw InvalidJobConfException if duplicates found in
|
|
cacheArchives or cacheFiles (Robert Evans via jeagles)
|
|
|
|
MAPREDUCE-3782. teragen terasort jobs fail when using webhdfs:// (Jason
|
|
Lowe via bobby)
|
|
|
|
MAPREDUCE-4053. Counters group names deprecation is wrong, iterating over
|
|
group names deprecated names don't show up (Robert Evans via tgraves)
|
|
|
|
MAPREDUCE-3506. Calling getPriority on JobInfo after parsing a history log
|
|
with JobHistoryParser throws a NullPointerException (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4570. ProcfsBasedProcessTree#constructProcessInfo() prints a
|
|
warning if procfsDir/<pid>/stat is not found. (Ahmed Radwan via bobby)
|
|
|
|
MAPREDUCE-4600. TestTokenCache.java from MRV1 no longer compiles (daryn
|
|
via bobby)
|
|
|
|
MAPREDUCE-4612. job summary file permissions not set when its created
|
|
(tgraves via bobby)
|
|
|
|
MAPREDUCE-4614. Simplify debugging a job's tokens (daryn via bobby)
|
|
|
|
MAPREDUCE-4611. MR AM dies badly when Node is decommissioned (Robert
|
|
Evans via tgraves)
|
|
|
|
MAPREDUCE-4604. In mapred-default, mapreduce.map.maxattempts &
|
|
mapreduce.reduce.maxattempts defaults are set to 4 as well as
|
|
mapreduce.job.maxtaskfailures.per.tracker. (Ravi Prakash via jeagles)
|
|
|
|
MAPREDUCE-4633. history server doesn't set permissions on all subdirs
|
|
(tgraves via bobby)
|
|
|
|
MAPREDUCE-4641. Exception in commitJob marks job as successful in job
|
|
history (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4549. Distributed cache conflicts breaks backwards compatability
|
|
(Robert Evans via tucu)
|
|
|
|
Release 0.23.2 - UNRELEASED
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-3849. Change TokenCache's reading of the binary token file
|
|
(Daryn Sharp via bobby)
|
|
|
|
MAPREDUCE-3854. Fixed and reenabled tests related to MR child JVM's
|
|
environmental variables in TestMiniMRChildTask. (Tom White via vinodkv)
|
|
|
|
MAPREDUCE-3877 Add a test to formalise the current state transitions
|
|
of the yarn lifecycle. (stevel)
|
|
|
|
MAPREDUCE-3866. Fixed the bin/yarn script to not print the command line
|
|
unnecessarily. (vinodkv)
|
|
|
|
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the
|
|
cluster without waiting for expiry. (Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-2793. Corrected AppIDs, JobIDs, TaskAttemptIDs to be of correct
|
|
format on the web pages. (Bikas Saha via vinodkv)
|
|
|
|
MAPREDUCE-3614. Fixed MR AM to close history file quickly and send a correct
|
|
final state to the RM when it is killed. (Ravi Prakash via vinodkv)
|
|
|
|
MAPREDUCE-3497. Added docs for YARN CLI. (tgraves via acmurthy)
|
|
|
|
MAPREDUCE-3954. Added new envs to separate heap size for different daemons
|
|
started via bin scripts. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-4025. AM can crash if task attempt reports bogus progress value
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-4034. Unable to view task logs on history server with
|
|
mapreduce.job.acl-view-job=* (Jason Lowe and Siddarth Seth via bobby)
|
|
|
|
MAPREDUCE-4043. Secret keys set in Credentials are not seen by tasks
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3989. Cap space usage of default log4j rolling policy.
|
|
(Patrick Hunt via eli)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-3901. Modified JobHistory records in YARN to lazily load job and
|
|
task reports so as to improve UI response times. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2855. Passing a cached class-loader to ResourceBundle creator to
|
|
minimize counter names lookup time. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
|
|
info for a significant performance improvement. (Robert Joseph Evans via
|
|
sseth)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-3918 proc_historyserver no longer in command line arguments for
|
|
HistoryServer (Jon Eagles via bobby)
|
|
|
|
MAPREDUCE-3862. Nodemanager can appear to hang on shutdown due to lingering
|
|
DeletionService threads (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3680. FifoScheduler web service rest API can print out invalid
|
|
JSON. (B Anil Kumar via tgraves)
|
|
|
|
MAPREDUCE-3852. Test TestLinuxResourceCalculatorPlugin failing. (Thomas
|
|
Graves via mahadev)
|
|
|
|
MAPREDUCE-3864. Fix cluster setup docs for correct SecondaryNameNode
|
|
HTTPS parameters. (todd)
|
|
|
|
MAPREDUCE-3583. Change pid to String and stime to BigInteger in order to
|
|
avoid NumberFormatException caused by overflow. (Zhihong Yu via szetszwo)
|
|
|
|
MAPREDUCE-3634. Fixed all daemons to crash instead of hanging around when
|
|
their EventHandlers get exceptions. (vinodkv)
|
|
|
|
MAPREDUCE-3798. Fixed failing TestJobCleanup.testCusomCleanup() and moved it
|
|
to the maven build. (Ravi Prakash via vinodkv)
|
|
|
|
MAPREDUCE-3884. PWD should be first in the classpath of MR tasks (tucu)
|
|
|
|
MAPREDUCE-3878. Null user on filtered jobhistory job page (Jonathon Eagles
|
|
via tgraves)
|
|
|
|
MAPREDUCE-3738. MM can hang during shutdown if AppLogAggregatorImpl thread
|
|
dies unexpectedly (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-3904 Job history produced with mapreduce.cluster.acls.enabled
|
|
false can not be viewed with mapreduce.cluster.acls.enabled true
|
|
(Jonathon Eagles via tgraves)
|
|
|
|
MAPREDUCE-3910. Fixed a bug in CapacityScheduler LeafQueue which was causing
|
|
app-submission to fail. (John George via vinodkv)
|
|
|
|
MAPREDUCE-3686. Fixed two bugs in Counters because of which web app displays
|
|
zero counter values for framework counters. (Bhallamudi Venkata Siva Kamesh
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-3913. RM application webpage is unresponsive after 2000 jobs
|
|
(Jason Lowe via tgraves)
|
|
|
|
MAPREDUCE-3922. Fixed build to not compile 32bit container-executor binary
|
|
by default on all platforms. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3790. Broken pipe on streaming job can lead to truncated output for
|
|
a successful job (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3816. capacity scheduler web ui bar graphs for used capacity wrong
|
|
(tgraves via bobby)
|
|
|
|
MAPREDUCE-3930. Fixed an NPE while accessing the AM page/webservice for a
|
|
task attempt without an assigned container. (Robert Joseph Evans via
|
|
sseth)
|
|
|
|
MAPREDUCE-3931. Changed PB implementation of LocalResource to take locks
|
|
so that race conditions don't fail tasks by inadvertantly changing the
|
|
timestamps. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3687. If AM dies before it returns new tracking URL, proxy
|
|
redirects to http://N/A/ and doesn't return error code (Ravi Prakash via
|
|
bobby)
|
|
|
|
MAPREDUCE-3920. Revise yarn default port number selection
|
|
(Dave Thompson via tgraves)
|
|
|
|
MAPREDUCE-3903. Add support for mapreduce admin users. (Thomas Graves via
|
|
sseth)
|
|
|
|
MAPREDUCE-3706. Fix circular redirect error in job-attempts page. (bobby
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3896. Add user information to the delegation token issued by the
|
|
history server. (Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-3792. Fix "bin/mapred job -list" to display all jobs instead of
|
|
only the jobs owned by the user. (Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-3929. Fixed output of 'bin/mapred queue -showacl' command to
|
|
clarify ACLs for users. (John George via acmurthy)
|
|
|
|
MAPREDUCE-3960. Fix web-proxy to forward request to AM with configured
|
|
hostname or IP. (tgraves via acmurthy)
|
|
|
|
MAPREDUCE-3897. Fixed computation of maxActiveAppsPerUser for queues by
|
|
using capacity and not max-capacity since we are already scaling it by
|
|
userLimitFactor. (Eric Payne via acmurthy)
|
|
|
|
MAPREDUCE-3009. Fixed node link on JobHistory webapp. (chackaravarthy via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-3964. ResourceManager does not have JVM metrics (Jason Lowe via
|
|
bobby)
|
|
|
|
MAPREDUCE-3034. Ensure NodeManager reboots itself on direction from
|
|
ResourceManager. (Devaraj K & Eric Payne via acmurthy)
|
|
|
|
MAPREDUCE-3976. TestRMContainerAllocator failing (Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3961. Map/ReduceSlotMillis computation incorrect (Siddharth Seth
|
|
via bobby)
|
|
|
|
MAPREDUCE-3977. LogAggregationService leaks log aggregator objects
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3975. Default value not set for Configuration parameter
|
|
mapreduce.job.local.dir (Eric Payne via bobby)
|
|
|
|
MAPREDUCE-3982. Fixed FileOutputCommitter to not err out for an 'empty-job'
|
|
whose tasks don't write any outputs. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager
|
|
Application Info page. (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-4006. History server container log web UI sometimes combines
|
|
stderr/stdout/syslog contents together (Siddharth Seth via tgraves)
|
|
|
|
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby)
|
|
|
|
Release 0.23.1 - 2012-02-17
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
|
|
|
|
MAPREDUCE-3121. NodeManager should handle disk-failures (Ravi Gummadi via mahadev)
|
|
|
|
MAPREDUCE-2863. Support web services for YARN and MR components. (Thomas
|
|
Graves via vinodkv)
|
|
|
|
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR ApplicationMaster.
|
|
(Anupam Seth via mahadev)
|
|
|
|
MAPREDUCE-778. Rumen Anonymizer. (Amar Kamat and Chris Douglas via amarrk)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-3481. [Gridmix] Improve Gridmix STRESS mode. (amarrk)
|
|
|
|
MAPREDUCE-3597. [Rumen] Rumen should provide APIs to access all the
|
|
job-history related information.
|
|
|
|
MAPREDUCE-3375. [Gridmix] Memory Emulation system tests.
|
|
(Vinay Thota via amarrk)
|
|
|
|
MAPREDUCE-3840. JobEndNotifier doesn't use the proxyToUse during connecting
|
|
(Ravi Prakash via bobby)
|
|
|
|
MAPREDUCE-3736. Variable substitution depth too large for fs.default.name
|
|
causes jobs to fail (ahmed via tucu).
|
|
|
|
MAPREDUCE-2733. [Gridmix] Gridmix3 cpu emulation system tests.
|
|
(Vinay Thota via amarrk)
|
|
|
|
MAPREDUCE-3297. Moved log related components into yarn-common so that
|
|
HistoryServer and clients can use them without depending on the
|
|
yarn-server-nodemanager module. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3336. Replaced guice internal.Preconditions api usage with the
|
|
public Preconditions API. (Thomas Graves via vinodkv)
|
|
|
|
MAPREDUCE-3280. Removed the unnecessary job user-name configuration in
|
|
mapred-site.xml. (vinodkv)
|
|
|
|
MAPREDUCE-3370. Fixed MiniMRYarnCluster and related tests to not use
|
|
a hard-coded path for the mr-app jar. (Ahmed Radwan via vinodkv)
|
|
|
|
MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3341. Enhance logging of initalized queue limit values.
|
|
(Anupam Seth via mahadev)
|
|
|
|
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3331. Improvement to single node cluster setup documentation for
|
|
0.23 (Anupam Seth via mahadev)
|
|
|
|
MAPREDUCE-3102. Changed NodeManager to fail fast when LinuxContainerExecutor
|
|
has wrong configuration or permissions. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3415. improve MiniMRYarnCluster & DistributedShell JAR
|
|
resolution. (tucu)
|
|
|
|
MAPREDUCE-3169. Create a new MiniMRCluster equivalent which only provides
|
|
client APIs cross MR1 and MR2. (Ahmed via tucu)
|
|
|
|
MAPREDUCE-3373. Hadoop scripts unconditionally source
|
|
"$bin"/../libexec/hadoop-config.sh. (Bruno Mahé via tomwhite)
|
|
|
|
MAPREDUCE-3372. HADOOP_PREFIX cannot be overridden.
|
|
(Bruno Mahé via tomwhite)
|
|
|
|
MAPREDUCE-3411. Performance Upgrade for jQuery (Jonathan Eagles via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3371. Review and improve the yarn-api javadocs. (Ravi Prakash
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3238. Small cleanup in SchedulerApp. (Todd Lipcon via mahadev)
|
|
|
|
MAPREDUCE-3413. RM web ui applications not sorted in any order by default.
|
|
(Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3045. Fixed UI filters to not filter on hidden title-numeric
|
|
sort fields. (Jonathan Eagles via sseth)
|
|
|
|
MAPREDUCE-3448. TestCombineOutputCollector javac unchecked warning on mocked
|
|
generics (Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3169 amendment. Deprecate MiniMRCluster. (Ahmed Radwan via
|
|
sseth)
|
|
|
|
MAPREDUCE-3369. Migrate MR1 tests to run on MR2 using the new interfaces
|
|
introduced in MAPREDUCE-3169. (Ahmed Radwan via tomwhite)
|
|
|
|
MAPREDUCE-3518. mapred queue -info <queue> -showJobs throws NPE.
|
|
(Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3391. Making a trivial change to correct a log message in
|
|
DistributedShell app's AM. (Subroto Sanyal via vinodkv)
|
|
|
|
MAPREDUCE-3547. Added a bunch of unit tests for the the RM/NM webservices.
|
|
(Thomas Graves via acmurthy)
|
|
|
|
MAPREDUCE-3610. Remove use of the 'dfs.block.size' config for default block
|
|
size fetching. Use FS#getDefaultBlocksize instead. (Sho Shimauchi via harsh)
|
|
|
|
MAPREDUCE-3478. Cannot build against ZooKeeper 3.4.0. (Tom White via mahadev)
|
|
|
|
MAPREDUCE-3528. Fixed TaskHeartBeatHandler to use a new configuration
|
|
for the thread loop interval separate from task-timeout configuration
|
|
property. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3312. Modified MR AM to not send a stop-container request for
|
|
a container that isn't launched at all. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3382. Enhanced MR AM to use a proxy to ping the job-end
|
|
notification URL. (Ravi Prakash via vinodkv)
|
|
|
|
MAPREDUCE-3299. Added AMInfo table to the MR AM job pages to list all the
|
|
job-attempts when AM restarts and recovers. (Jonathan Eagles via vinodkv)
|
|
|
|
MAPREDUCE-3251. Network ACLs can prevent some clients to talk to MR AM.
|
|
Improved the earlier patch to not to JobHistoryServer repeatedly.
|
|
(Anupam Seth via vinodkv)
|
|
|
|
MAPREDUCE-3553. Add support for data returned when exceptions thrown from web
|
|
service apis to be in either xml or in JSON. (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3641. Making CapacityScheduler more conservative so as to
|
|
assign only one off-switch container in a single scheduling
|
|
iteration. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3692. yarn-resourcemanager out and log files can get big. (eli)
|
|
|
|
MAPREDUCE-3710. Improved FileInputFormat to return better locality for the
|
|
last split. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2765. DistCp Rewrite. (Mithun Radhakrishnan via mahadev)
|
|
|
|
MAPREDUCE-3737. The Web Application Proxy's is not documented very well.
|
|
(Robert Evans via mahadev)
|
|
|
|
MAPREDUCE-3699. Increased RPC handlers for all YARN servers to reasonable
|
|
values for working at scale. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3693. Added mapreduce.admin.user.env to mapred-default.xml.
|
|
(Roman Shapshonik via acmurthy)
|
|
|
|
MAPREDUCE-3732. Modified CapacityScheduler to use only users with pending
|
|
requests for computing user-limits. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3679. AM logs and others should not automatically refresh after every 1
|
|
second. (Vinod KV via mahadev)
|
|
|
|
MAPREDUCE-3754. Modified RM UI to filter applications based on state of the
|
|
applications. (vinodkv)
|
|
|
|
MAPREDUCE-3774. Moved yarn-default.xml to hadoop-yarn-common from
|
|
hadoop-server-common. (Mahadev Konar via vinodkv)
|
|
|
|
MAPREDUCE-3771. Un-deprecated the old mapred apis, port of MAPREDUCE-1735.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-3784. Fixed CapacityScheduler so that maxActiveApplications and
|
|
maxActiveApplicationsPerUser per queue are not too low for small
|
|
clusters. (Arun C Murthy via vinodkv)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-3567. Extraneous JobConf objects in AM heap. (Vinod Kumar
|
|
Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-3399. Modifying ContainerLocalizer to send a heartbeat to NM
|
|
immediately after downloading a resource instead of always waiting for a
|
|
second. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3568. Optimized Job's progress calculations in MR AM. (vinodkv)
|
|
|
|
MAPREDUCE-3569. TaskAttemptListener holds a global lock for all
|
|
task-updates. (Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-3511. Removed a multitude of cloned/duplicate counters in the AM
|
|
thereby reducing the AM heap size and preventing full GCs. (vinodkv)
|
|
|
|
MAPREDUCE-3618. Fixed TaskHeartbeatHandler to not hold a global lock for all
|
|
task-updates. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3512. Batching JobHistory flushing to DFS so that we don't flush
|
|
for every event slowing down AM. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3718. Change default AM heartbeat interval to 1 second. (Hitesh
|
|
Shah via sseth)
|
|
|
|
MAPREDUCE-3360. Added information about lost/rebooted/decommissioned nodes
|
|
on the webapps. (Bhallamudi Venkata Siva Kamesh and Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-3756. Made single shuffle limit configurable. (Hitesh Shah via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3811. Made jobclient-to-AM retries configurable. (sseth via
|
|
acmurthy)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-2784. [Gridmix] Bug fixes in ExecutionSummarizer and
|
|
ResourceUsageMatcher. (amarrk)
|
|
|
|
MAPREDUCE-3194. "mapred mradmin" command is broken in mrv2
|
|
(Jason Lowe via bobby)
|
|
|
|
MAPREDUCE-3462. Fix Gridmix JUnit testcase failures.
|
|
(Ravi Prakash and Ravi Gummadi via amarrk)
|
|
|
|
MAPREDUCE-2950. [Rumen] Fixed TestUserResolve. (Ravi Gummadi via amarrk)
|
|
|
|
MAPREDUCE-3412. Fix 'ant docs'. (amarrk)
|
|
|
|
MAPREDUCE-3346 [Rumen] LoggedTaskAttempt#getHostName() returns null.
|
|
(amarrk)
|
|
|
|
MAPREDUCE-3221. Reenabled the previously ignored test in TestSubmitJob
|
|
and fixed bugs in it. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3215. Reenabled and fixed bugs in the failing test
|
|
TestNoJobSetupCleanup. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3219. Reenabled and fixed bugs in the failing test
|
|
TestDelegationToken. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3217. Reenabled and fixed bugs in the failing ant test
|
|
TestAuditLogger. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3291. App fail to launch due to delegation token not
|
|
found in cache (Robert Evans via mahadev)
|
|
|
|
MAPREDUCE-3344. o.a.h.mapreduce.Reducer since 0.21 blindly casts to
|
|
ReduceContext.ValueIterator. (Brock Noland via tomwhite)
|
|
|
|
MAPREDUCE-3342. Fixed JobHistoryServer to also show the job's queue
|
|
name. (Jonathan Eagles via vinodkv)
|
|
|
|
MAPREDUCE-3345. Fixed a race condition in ResourceManager that was causing
|
|
TestContainerManagerSecurity to fail sometimes. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3368. Fixed test compilation. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3333. Fixed bugs in ContainerLauncher of MR AppMaster due to
|
|
which per-container connections to NodeManager were lingering long enough
|
|
to hit the ulimits on number of processes. (vinodkv)
|
|
|
|
MAPREDUCE-3392. Fixed Cluster's getDelegationToken's API to return null
|
|
when there isn't a supported token. (John George via vinodkv)
|
|
|
|
MAPREDUCE-3379. Fixed LocalResourceTracker in NodeManager to remove deleted
|
|
cache entries correctly. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are
|
|
accessible through all UI servers (Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3355. Fixed MR AM's ContainerLauncher to handle node-command
|
|
timeouts correctly. (vinodkv)
|
|
|
|
MAPREDUCE-3407. Fixed pom files to refer to the correct MR app-jar needed
|
|
by the integration tests. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3437. Fix examples pom to refer to the correct 0.23 snapshot
|
|
version. (Jonathan Eagles via todd)
|
|
|
|
MAPREDUCE-3434. Nightly build broken (Hitesh Shah via mahadev)
|
|
|
|
MAPREDUCE-3447. mapreduce examples not working (mahadev)
|
|
|
|
MAPREDUCE-3444. trunk/0.23 builds broken (Hitesh Shah via mahadev)
|
|
|
|
MAPREDUCE-3454. [Gridmix] TestDistCacheEmulation is broken (Hitesh Shah
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3408. yarn-daemon.sh unconditionnaly sets yarn.root.logger
|
|
(Bruno Mahe via mahadev)
|
|
|
|
MAPREDUCE-3329. Fixed CapacityScheduler to ensure maximum-capacity cannot
|
|
be lesser than capacity for any queue. (acmurthy)
|
|
|
|
MAPREDUCE-3464. mapreduce jsp pages missing DOCTYPE. (Dave Vronay via mattf)
|
|
|
|
MAPREDUCE-3265. Removed debug logs during job submission to LOG.debug to
|
|
cut down noise. (acmurthy)
|
|
|
|
MAPREDUCE-3468. Changed ant based infrastructure to use 0.23.1 version.
|
|
(sseth via acmurthy)
|
|
|
|
MAPREDUCE-3433. Finding counters by legacy group name returns empty
|
|
counters. (tomwhite)
|
|
|
|
MAPREDUCE-3450. NM port info no longer available in JobHistory.
|
|
(Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-3477. Hadoop site documentation cannot be built anymore.
|
|
(jeagles via tucu)
|
|
|
|
MAPREDUCE-3488. Streaming jobs are failing because the main class
|
|
isnt set in the pom files. (mahadev)
|
|
|
|
MAPREDUCE-3463. Second AM fails to recover properly when first AM is killed with
|
|
java.lang.IllegalArgumentException causing lost job. (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-3452. fifoscheduler web ui page always shows 0% used for the queue.
|
|
(Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3443. JobClient and Job should function in the context of the
|
|
UGI which created them. (Mahadev Konar via sseth)
|
|
|
|
MAPREDUCE-3460. MR AM can hang if containers are allocated on a node
|
|
blacklisted by the AM. (Hitesh Shah and Robert Joseph Evans via sseth)
|
|
|
|
MAPREDUCE-3453. RM web ui application details page shows RM cluster about
|
|
information. (Jonathan Eagles via sseth)
|
|
|
|
MAPREDUCE-3479. JobClient#getJob cannot find local jobs. (tomwhite)
|
|
|
|
MAPREDUCE-3500. MRJobConfig creates an LD_LIBRARY_PATH using the platform ARCH. (tucu)
|
|
|
|
MAPREDUCE-3456. $HADOOP_PREFIX/bin/yarn should set defaults for
|
|
$HADOOP_*_HOME (Eric Payne via mahadev)
|
|
|
|
MAPREDUCE-3458. Fix findbugs warnings in hadoop-examples. (Devaraj K
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3485. DISKS_FAILED -101 error code should be defined in same location as
|
|
ABORTED_CONTAINER_EXIT_STATUS. (Ravi Gummadi via mahadev)
|
|
|
|
MAPREDUCE-3389. MRApps loads the 'mrapp-generated-classpath' file with
|
|
classpath from the build machine. (tucu)
|
|
|
|
MAPREDUCE-3496. Fixed client to print queue acls in consistent order.
|
|
(Jonathan Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3147. Handle leaf queues with the same name properly.
|
|
(Ravi Prakash via mahadev)
|
|
|
|
MAPREDUCE-3327. RM web ui scheduler link doesn't show correct max value
|
|
for queues (Anupam Seth via mahadev)
|
|
|
|
MAPREDUCE-3513. Capacity Scheduler web UI has a spelling mistake for Memory.
|
|
(chackaravarthy via mahadev)
|
|
|
|
MAPREDUCE-3519. Fixed a deadlock in NodeManager LocalDirectories's handling
|
|
service. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-3527. Fix minor API incompatibilities between 1.0 and 0.23.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-3328. mapred queue -list output inconsistent and missing child
|
|
queues. (Ravi Prakash via mahadev)
|
|
|
|
MAPREDUCE-3510. Capacity Scheduler inherited ACLs not displayed by mapred queue
|
|
-showacls (Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3537. Fix race condition in DefaultContainerExecutor which led
|
|
to container localization occuring in wrong directories. (acmurthy)
|
|
|
|
MAPREDUCE-3542. Support "FileSystemCounter" legacy counter group name for
|
|
compatibility. (tomwhite)
|
|
|
|
MAPREDUCE-3426. Fixed MR AM in uber mode to write map intermediate outputs
|
|
in the correct directory to work properly in secure mode. (Hitesh Shah via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-3544. gridmix build is broken, requires hadoop-archives to be
|
|
added as ivy dependency. (tucu)
|
|
|
|
MAPREDUCE-3557. MR1 test fail to compile because of missing hadoop-archives
|
|
dependency. (tucu)
|
|
|
|
MAPREDUCE-3541. Fix broken TestJobQueueClient test. (Ravi Prakash via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3398. Fixed log aggregation to work correctly in secure mode.
|
|
(Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3530. Fixed an NPE occuring during scheduling in the
|
|
ResourceManager. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3484. Fixed JobEndNotifier to not get interrupted before completing
|
|
all its retries. (Ravi Prakash via vinodkv)
|
|
|
|
MAPREDUCE-3531. Fixed a race in ContainerTokenSecretManager. (Robert Joseph
|
|
Evans via sseth)
|
|
|
|
MAPREDUCE-3560. TestRMNodeTransitions is failing on trunk.
|
|
(Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-3487. Fixed JobHistory web-UI to display links to single task's
|
|
counters' page. (Jason Lowe via vinodkv)
|
|
|
|
MAPREDUCE-3564. Fixed failures in TestStagingCleanup and TestJobEndNotifier
|
|
tests. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3422. Counter display names are not being picked up. (Jonathan
|
|
Eagles via sseth)
|
|
|
|
MAPREDUCE-3366. Mapreduce component should use consistent directory structure
|
|
layout as HDFS/common (Eric Yang via mahadev)
|
|
|
|
MAPREDUCE-3387. Fixed AM's tracking URL to always go through the proxy, even
|
|
before the job started, so that it works properly with oozie throughout
|
|
the job execution. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3579. ConverterUtils shouldn't include a port in a path from a url
|
|
without a port. (atm via harsh)
|
|
|
|
MAPREDUCE-3563. Fixed LocalJobRunner to work correctly with new mapreduce
|
|
apis. (acmurthy)
|
|
|
|
MAPREDUCE-3376. Fixed Task to ensure it passes reporter to combiners using
|
|
old MR api. (Subroto Sanyal via acmurthy)
|
|
|
|
MAPREDUCE-3339. Fixed MR AM to stop considering node blacklisting after the
|
|
number of nodes blacklisted crosses a threshold. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3588. Fixed bin/yarn which was broken by MAPREDUCE-3366 so that
|
|
yarn daemons can start. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3349. Log rack-name in JobHistory for unsuccessful tasks. (Amar
|
|
Kamat and Devaraj K via sseth)
|
|
|
|
MAPREDUCE-3586. Modified CompositeService to avoid duplicate stop operations
|
|
thereby solving race conditions in MR AM shutdown. (vinodkv)
|
|
|
|
MAPREDUCE-3604. Fixed streaming to use new mapreduce.framework.name to
|
|
check for local mode. (acmurthy)
|
|
|
|
MAPREDUCE-3521. Fixed streaming to ensure it doesn't silently ignore
|
|
unknown arguments. (Robert Evans via acmurthy)
|
|
|
|
MAPREDUCE-3522. Ensure queues inherit ACLs from parent if they aren't
|
|
explicitly specified. (Jonathan Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3608. Fixed compile issue with MAPREDUCE-3522. (mahadev via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3490. Fixed MapReduce AM to count failed maps also towards Reduce
|
|
ramp up. (Sharad Agarwal and Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3529. TokenCache does not cache viewfs credentials correctly
|
|
(sseth)
|
|
|
|
MAPREDUCE-3595. Add missing TestCounters#testCounterValue test from branch
|
|
1 to 0.23 (Tom White via sseth)
|
|
|
|
MAPREDUCE-3566. Fixed MR AM to construct CLC only once across all tasks.
|
|
(vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3572. Moved AM event dispatcher to a separate thread for
|
|
performance reasons. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3615. Fix some ant test failures. (Thomas Graves via sseth)
|
|
|
|
MAPREDUCE-1744. DistributedCache creates its own FileSytem instance when
|
|
adding a file/archive to the path. (Dick King via tucu)
|
|
|
|
MAPREDUCE-3326. Added detailed information about queue's to the
|
|
CapacityScheduler web-ui. (Jason Lowe via acmurthy)
|
|
|
|
MAPREDUCE-3548. Added more unit tests for MR AM & JHS web-services.
|
|
(Thomas Graves via acmurthy)
|
|
|
|
MAPREDUCE-3617. Removed wrong default value for
|
|
yarn.resourcemanager.principal and yarn.nodemanager.principal. (Jonathan
|
|
Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar. (mahadev
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3616. Thread pool for launching containers in MR AM not
|
|
expanding as expected. (vinodkv via sseth)
|
|
|
|
MAPREDUCE-3639. Fixed TokenCache to work with absent FileSystem canonical
|
|
service-names. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3380. Token infrastructure for running clients which are not kerberos
|
|
authenticated. (mahadev)
|
|
|
|
MAPREDUCE-3648. TestJobConf failing. (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3651. TestQueueManagerRefresh fails. (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3645. TestJobHistory fails. (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3652. org.apache.hadoop.mapred.TestWebUIAuthorization.testWebUIAuthorization
|
|
fails. (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3625. CapacityScheduler web-ui display of queue's used capacity is broken.
|
|
(Jason Lowe via mahadev)
|
|
|
|
MAPREDUCE-3596. Fix scheduler to handle cleaned up containers, which NMs
|
|
may subsequently report as running. (Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-3656. Fixed a race condition in MR AM which is failing the sort
|
|
benchmark consistently. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3532. Modified NM to report correct http address when an ephemeral
|
|
web port is configured. (Bhallamudi Venkata Siva Kamesh via vinodkv)
|
|
|
|
MAPREDUCE-3404. Corrected MR AM to honor speculative configuration and enable
|
|
speculating either maps or reduces. (Eric Payne via vinodkv)
|
|
|
|
MAPREDUCE-3664. Federation Documentation has incorrect configuration example.
|
|
(Brandon Li via jitendra)
|
|
|
|
MAPREDUCE-3649. Job End notification gives an error on calling back.
|
|
(Ravi Prakash via mahadev)
|
|
|
|
MAPREDUCE-3657. State machine visualize build fails. (Jason Lowe
|
|
via mahadev)
|
|
|
|
MAPREDUCE-2450. Fixed a corner case with interrupted communication threads
|
|
leading to a long timeout in Task. (Rajesh Balamohan via acmurthy)
|
|
|
|
MAPREDUCE-3669. Allow clients to talk to MR HistoryServer using both
|
|
delegation tokens and kerberos. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-3684. LocalDistributedCacheManager does not shut down its thread
|
|
pool (tomwhite)
|
|
|
|
MAPREDUCE-3582. Move successfully passing MR1 tests to MR2 maven tree.
|
|
(ahmed via tucu)
|
|
|
|
MAPREDUCE-3698. Client cannot talk to the history server in secure mode.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3689. RM web UI doesn't handle newline in job name.
|
|
(Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3701. Delete HadoopYarnRPC from 0.23 branch.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3549. write api documentation for web service apis for RM, NM,
|
|
mapreduce app master, and job history server (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3705. ant build fails on 0.23 branch. (Thomas Graves via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3691. webservices add support to compress response.
|
|
(Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3702. internal server error trying access application master
|
|
via proxy with filter enabled (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-3646. Remove redundant URL info from "mapred job" output.
|
|
(Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3681. Fixed computation of queue's usedCapacity. (acmurthy)
|
|
|
|
MAPREDUCE-3505. yarn APPLICATION_CLASSPATH needs to be overridable.
|
|
(ahmed via tucu)
|
|
|
|
MAPREDUCE-3714. Fixed EventFetcher and Fetcher threads to shut-down properly
|
|
so that reducers don't hang in corner cases. (vinodkv)
|
|
|
|
MAPREDUCE-3712. The mapreduce tar does not contain the hadoop-mapreduce-client-
|
|
jobclient-tests.jar. (mahadev)
|
|
|
|
MAPREDUCE-3717. JobClient test jar has missing files to run all the test programs.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3630. Fixes a NullPointer exception while running TeraGen - if a
|
|
map is asked to generate 0 records. (Mahadev Konar via sseth)
|
|
|
|
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent
|
|
maxCapacities. (acmurthy)
|
|
|
|
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by
|
|
CapacityScheduler so that it deducts current-usage per user and not
|
|
per-application. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3721. Fixed a race in shuffle which caused reduces to hang.
|
|
(sseth via acmurthy)
|
|
|
|
MAPREDUCE-3733. Add Apache License Header to hadoop-distcp/pom.xml.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3735. Add distcp jar to the distribution (tar).
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3720. Changed bin/mapred job -list to not print job-specific
|
|
information not available at RM. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3742. "yarn logs" command fails with ClassNotFoundException.
|
|
(Jason Lowe via mahadev)
|
|
|
|
MAPREDUCE-3703. ResourceManager should provide node lists in JMX output.
|
|
(Eric Payne via mahadev)
|
|
|
|
MAPREDUCE-3716. Fixing YARN+MR to allow MR jobs to be able to use
|
|
java.io.File.createTempFile to create temporary files as part of their
|
|
tasks. (Jonathan Eagles via vinodkv)
|
|
|
|
MAPREDUCE-3748. Changed a log in CapacityScheduler.nodeUpdate to debug.
|
|
(ramya via acmurthy)
|
|
|
|
MAPREDUCE-3764. Fixed resource usage metrics for queues and users.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-3749. ConcurrentModificationException in counter groups.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-3762. Fixed default CapacityScheduler configs. (mahadev via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3499. New MiniMR does not setup proxyuser configuration
|
|
correctly, thus tests using doAs do not work. (johnvijoe via tucu)
|
|
|
|
MAPREDUCE-3696. MR job via oozie does not work on hadoop 23.
|
|
(John George via mahadev)
|
|
|
|
MAPREDUCE-3427. Fix streaming unit tests broken after mavenization.
|
|
(Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3640. Allow AMRecovery to work with partial JobHistory files.
|
|
(Arun C Murthy via sseth)
|
|
|
|
MAPREDUCE-3752. Modified application limits to include queue max-capacities
|
|
besides the usual user limits. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3744. Fix the yarn logs command line. Improve error messages for
|
|
mapred job -logs. (Jason Lowe via sseth)
|
|
|
|
MAPREDUCE-3780. Fixed a bug where applications killed before getting
|
|
activated were not getting cleaned up properly. (Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3708. Metrics: Incorrect Apps Submitted Count (Bhallamudi via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3727. jobtoken location property in jobconf refers to wrong
|
|
jobtoken file (tucu)
|
|
|
|
MAPREDUCE-3711. Fixed MR AM recovery so that only single selected task
|
|
output is recovered and thus reduce the unnecessarily bloated recovery
|
|
time. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3760. Changed active nodes list to not contain unhealthy nodes
|
|
on the webUI and metrics. (vinodkv)
|
|
|
|
MAPREDUCE-3417. Fixed job-access-controls to work with MR AM and
|
|
JobHistoryServer web-apps. (Jonathan Eagles via vinodkv)
|
|
|
|
MAPREDUCE-3803. Fix broken build of raid contrib due to HDFS-2864.
|
|
(Ravi Prakash via suresh)
|
|
|
|
MAPREDUCE-3791. can't build site in hadoop-yarn-server-common.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3723. TestAMWebServicesJobs & TestHSWebServicesJobs
|
|
incorrectly asserting tests (Bhallamudi Venkata Siva Kamesh
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3795. "job -status" command line output is malformed.
|
|
(vinodkv via mahadev)
|
|
|
|
MAPREDUCE-3759. ClassCastException thrown in -list-active-trackers when
|
|
there are a few unhealthy nodes (vinodkv via mahadev)
|
|
|
|
MAPREDUCE-3775. Change MiniYarnCluster to escape special chars in testname.
|
|
(Hitesh Shah via mahadev)
|
|
|
|
MAPREDUCE-3765. FifoScheduler does not respect yarn.scheduler.fifo.minimum-
|
|
allocation-mb setting (Hitesh Shah via mahadev)
|
|
|
|
MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish
|
|
time to RM Web-UI. (acmurthy)
|
|
|
|
MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined
|
|
hascode/equals for ResourceRequest & Priority. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3813. Added a cache for resolved racks. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3808. Fixed an NPE in FileOutputCommitter for jobs with maps
|
|
but no reduces. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3804. yarn webapp interface vulnerable to cross scripting attacks
|
|
(Dave Thompson via bobby)
|
|
|
|
MAPREDUCE-3354. Changed scripts so that jobhistory server is started by
|
|
bin/mapred instead of bin/yarn. (Jonathan Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3809. Ensure that there is no needless sleep in Task at the end
|
|
of the task. (sseth via acmurthy)
|
|
|
|
MAPREDUCE-3794. Support mapred.Task.Counter and mapred.JobInProgress.Counter
|
|
enums for compatibility (Tom White via mahadev)
|
|
|
|
MAPREDUCE-3697. Support binary compatibility for Counters after
|
|
MAPREDUCE-901. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-3817. Fixed bin/mapred to allow running of distcp and archive
|
|
jobs. (Arpit Gupta via acmurthy)
|
|
|
|
MAPREDUCE-3709. TestDistributedShell is failing. (Hitesh Shah via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3436. JobHistory webapp address should use the host configured
|
|
in the jobhistory address. (Ahmed Radwan via sseth)
|
|
|
|
MAPREDUCE-3815. Fixed MR AM to always use hostnames and never IPs when
|
|
requesting containers so that scheduler can give off data local containers
|
|
correctly. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3833. Fixed a bug in reinitiaziling of queues. (Jason Lowe via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3826. Fixed a bug in RM web-ui which broke sorting. (Jonathan
|
|
Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3823. Ensure counters are calculated only once after a job
|
|
finishes. (Vinod Kumar Vavilapalli via sseth)
|
|
|
|
MAPREDUCE-3827. Changed Counters to use ConcurrentSkipListMap for
|
|
performance. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3822. Changed FS counter computation to use all occurences of
|
|
the same FS scheme, instead of randomly using one. (Mahadev Konar via
|
|
sseth)
|
|
|
|
MAPREDUCE-3834. Changed MR AM to not add the same rack entry multiple times
|
|
into the container request table when multiple hosts for a split happen to
|
|
be on the same rack. (Siddarth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3828. Ensure that urls in single-node mode are correct. (sseth
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3770. Zombie.getJobConf() results into NPE. (amarrk)
|
|
|
|
MAPREDUCE-3843. Job summary log file found missing on the RM host
|
|
(Anupam Seth via tgraves)
|
|
|
|
MAPREDUCE-3846. Addressed MR AM hanging issues during AM restart and then
|
|
the recovery. (vinodkv)
|
|
|
|
MAPREDUCE-3802. Added test to validate that AM can crash multiple times and
|
|
still can recover successfully after MAPREDUCE-3846. (vinodkv)
|
|
|
|
MAPREDUCE-3858. Task attempt failure during commit results in task never completing.
|
|
(Tom White via mahadev)
|
|
|
|
MAPREDUCE-3856. Instances of RunningJob class givs incorrect job tracking
|
|
urls when mutiple jobs are submitted from same client jvm. (Eric Payne via
|
|
sseth)
|
|
|
|
MAPREDUCE-3880. Changed LCE binary to be 32-bit. (acmurthy)
|
|
|
|
Release 0.23.0 - 2011-11-01
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-2455. Remove deprecated JobTracker.State in favour of
|
|
JobTrackerStatus. (tomwhite)
|
|
|
|
MAPREDUCE-2430. Remove mrunit contrib. (nigel via eli)
|
|
|
|
MAPREDUCE-2606. Remove IsolationRunner. (Alejandro Abdelnur via eli)
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-2682. Add "mapred classpath" command to print classpath
|
|
for MR applications. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-2107. [Gridmix] Total heap usage emulation in Gridmix.
|
|
(Amar Kamat and Ravi Gummadi via amarrk)
|
|
|
|
MAPREDUCE-2106. [Gridmix] Cumulative CPU usage emulation in Gridmix.
|
|
(amarrk)
|
|
|
|
MAPREDUCE-2543. [Gridmix] High-Ram feature emulation in Gridmix. (amarrk)
|
|
|
|
MAPREDUCE-2408. [Gridmix] Compression emulation in Gridmix. (amarrk)
|
|
|
|
MAPREDUCE-2473. Add "mapred groups" command to query the server-side groups
|
|
resolved for a user. (Aaron T. Myers via todd)
|
|
|
|
MAPREDUCE-461. Enable ServicePlugins for the JobTracker.
|
|
(Fredrik Hedberg via tomwhite)
|
|
|
|
MAPREDUCE-2521. Create RPM and Debian packages for MapReduce. Changes
|
|
deployment layout to be consistent across the binary tgz, rpm, and deb.
|
|
(Eric Yang via omalley)
|
|
|
|
MAPREDUCE-2323. Add metrics to the fair scheduler. (todd)
|
|
|
|
MAPREDUCE-2037. Capture intermediate progress, CPU and memory usage for
|
|
tasks. (Dick King via acmurthy)
|
|
|
|
MAPREDUCE-279. MapReduce 2.0. Merging MR-279 branch into trunk. Contributed by
|
|
Arun C Murthy, Christopher Douglas, Devaraj Das, Greg Roelofs, Jeffrey
|
|
Naisbitt, Josh Wills, Jonathan Eagles, Krishna Ramachandran, Luke Lu, Mahadev
|
|
Konar, Robert Evans, Sharad Agarwal, Siddharth Seth, Thomas Graves, and Vinod
|
|
Kumar Vavilapalli.
|
|
|
|
MAPREDUCE-2930. Added the ability to be able to generate graphs from the
|
|
state-machine definitions. (Binglin Chang via vinodkv)
|
|
|
|
MAPREDUCE-2719. Add a simple, DistributedShell, application to illustrate
|
|
alternate frameworks on YARN. (Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3104. Implemented Application-acls. (vinodkv)
|
|
|
|
MAPREDUCE-2708. Designed and implemented MR Application Master recovery to
|
|
make MR AMs resume their progress after restart. (Sharad Agarwal via vinodkv)
|
|
|
|
MAPREDUCE-2858. Added a WebApp Proxy for applications. (Robert Evans via
|
|
acmurthy)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-2187. Reporter sends progress during sort/merge. (Anupam Seth via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2365. Add counters to track bytes (read,written) via
|
|
File(Input,Output)Format. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-2680. Display queue name in job client CLI. (acmurthy)
|
|
|
|
MAPREDUCE-2679. Minor changes to sync trunk with MR-279 branch. (acmurthy)
|
|
|
|
MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a
|
|
ServiceProvider for the actual implementation. (tomwhite via acmurthy)
|
|
|
|
MAPREDUCE-2596. [Gridmix] Summarize Gridmix runs. (amarrk)
|
|
|
|
MAPREDUCE-2563. [Gridmix] Add High-Ram emulation system tests to
|
|
Gridmix. (Vinay Kumar Thota via amarrk)
|
|
|
|
MAPREDUCE-2104. [Rumen] Add Cpu, Memory and Heap usages to
|
|
TraceBuilder's output. (amarrk)
|
|
|
|
MAPREDUCE-2554. [Gridmix] Add distributed cache emulation system tests
|
|
to Gridmix. (Vinay Kumar Thota via amarrk)
|
|
|
|
MAPREDUCE-2543. [Gridmix] High-Ram feature emulation testcase. (amarrk)
|
|
|
|
MAPREDUCE-2469. Task counters should also report the total heap usage of
|
|
the task. (Ravi Gummadi and Amar Ramesh Kamat via amarrk)
|
|
|
|
MAPREDUCE-2544. [Gridmix] Add compression emulation system tests to
|
|
Gridmix. (Vinay Kumar Thota via amarrk)
|
|
|
|
MAPREDUCE-2517. [Gridmix] Add system tests to Gridmix.
|
|
(Vinay Kumar Thota via amarrk)
|
|
|
|
MAPREDUCE-2492. The new MapReduce API should make available task's
|
|
progress to the task. (amarrk)
|
|
|
|
MAPREDUCE-2153. Bring in more job configuration properties in to the trace
|
|
file. (Rajesh Balamohan via amarrk)
|
|
|
|
MAPREDUCE-1461. Feature to instruct rumen-folder utility to skip jobs worth
|
|
of specific duration. (Rajesh Balamohan via amarrk)
|
|
|
|
MAPREDUCE-2172. Added test-patch.properties required by test-patch.sh
|
|
(nigel)
|
|
|
|
MAPREDUCE-2156. Raid-aware FSCK. (Patrick Kling via dhruba)
|
|
|
|
MAPREDUCE-2215. A more elegant FileSystem#listCorruptFileBlocks API
|
|
(RAID changes) (Patrick Kling via hairong)
|
|
|
|
MAPREDUCE-1831. BlockPlacement policy for HDFS-RAID.
|
|
(Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1906. Lower minimum heartbeat interval for TaskTracker
|
|
(Scott Carey and Todd Lipcon via todd)
|
|
|
|
MAPREDUCE-1382. MRAsyncDiscService should tolerate missing local.dir.
|
|
(Zheng Shao and tomwhite via tomwhite)
|
|
|
|
MAPREDUCE-2263. MapReduce side of HADOOP-6904: RPC compatibility.
|
|
(hairong)
|
|
|
|
MAPREDUCE-1706. Log RAID recoveries on HDFS. (schen)
|
|
|
|
MAPREDUCE-2334. Update BlockPlacementPolicyRaid for the new method
|
|
in BlockPlacementPolicy. (szetszwo)
|
|
|
|
MAPREDUCE-2254. Allow setting of end-of-record delimiter for
|
|
TextInputFormat (Ahmed Radwan via todd)
|
|
|
|
MAPREDUCE-1927. Unit test for HADOOP-6835 (concatenated gzip support).
|
|
(Greg Roelofs via tomwhite)
|
|
|
|
MAPREDUCE-2206. The task-cleanup tasks should be optional. (schen)
|
|
|
|
MAPREDUCE-2225. MultipleOutputs should not require the use of 'Writable'.
|
|
(Harsh J Chouraria via tomwhite)
|
|
|
|
MAPREDUCE-1811. Job.monitorAndPrintJob() should print status of the job
|
|
at completion. (Harsh J Chouraria via tomwhite)
|
|
|
|
MAPREDUCE-993. bin/hadoop job -events <jobid> <from-event-#> <#-of-events>
|
|
help message is confusing. (Harsh J Chouraria via tomwhite)
|
|
|
|
MAPREDUCE-2302. Add static factory methods in GaloisField. (schen)
|
|
|
|
MAPREDUCE-2351. mapred.job.tracker.history.completed.location should
|
|
support an arbitrary filesystem URI. (tomwhite)
|
|
|
|
MAPREDUCE-2239. BlockPlacementPolicyRaid should call getBlockLocations
|
|
only when necessary. (schen)
|
|
|
|
MAPREDUCE-2331. Add coverage of task graph servlet to fair scheduler system
|
|
test. (todd)
|
|
|
|
MAPREDUCE-2367. Allow using a file to exclude certain tests from build.
|
|
(todd)
|
|
|
|
MAPREDUCE-2202. Generalize CLITest structure and interfaces to faciliate
|
|
upstream adoption (e.g. for web or system testing). (cos)
|
|
|
|
MAPREDUCE-2420. JobTracker should be able to renew delegation token over
|
|
HTTP (Boris Shkolnik via jitendra)
|
|
|
|
MAPREDUCE-2474. Add docs to the new API Partitioner on how to access the
|
|
Job Configuration. (Harsh J Chouraria via todd)
|
|
|
|
MAPREDUCE-2475. Disable IPV6 for junit tests. (suresh srinivas via mahadev)
|
|
|
|
MAPREDUCE-2422. Removed unused internal methods from DistributedCache.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with
|
|
failed fetch notifications in the JobTracker log.
|
|
(Jeffrey Naisbitt via cdouglas)
|
|
|
|
MAPREDUCE-869. Documentation for config to set map/reduce task environment
|
|
(Alejandro Abdelnur via todd)
|
|
|
|
MAPREDUCE-2410. Add entry to streaming FAQ about how streaming reducers
|
|
receive keys. (Harsh J Chouraria via todd)
|
|
|
|
MAPREDUCE-2499. MR part of HADOOP-7291. (eli)
|
|
|
|
MAPREDUCE-2497. Missing spaces in error messages. (eli)
|
|
|
|
MAPREDUCE-2502. JobSubmitter should use mapreduce.job.maps instead of
|
|
its deprecated equivalent. (eli via todd)
|
|
|
|
MAPREDUCE-2381. JobTracker instrumentation not consistent about error
|
|
handling. (Philip Zeyliger via tomwhite)
|
|
|
|
MAPREDUCE-2449. Allow for command line arguments when performing
|
|
"Run on Hadoop" action in Eclipse plugin. (Jeff Zemerick via todd)
|
|
|
|
MAPREDUCE-2483. Remove duplication of jars between Hadoop subprojects
|
|
from build artifacts. (Eric Yang via omalley)
|
|
|
|
MAPREDUCE-2372. TaskLogAppender mechanism shouldn't be set up in
|
|
log4j.properties (todd)
|
|
|
|
MAPREDUCE-2516. Rename webinterface.private.actions to
|
|
mapreduce.jobtracker.webinterface.trusted (Ari Rabkin via todd)
|
|
|
|
MAPREDUCE-2459. Cache HAR filesystem metadata. (Mac Yang via mahadev)
|
|
|
|
HADOOP-7259. Contrib modules should include the build.properties from
|
|
the enclosing hadoop directory. (omalley)
|
|
|
|
MAPREDUCE-2494. Order distributed cache deletions by LRU. (Robert Joseph
|
|
Evans via cdouglas)
|
|
|
|
MAPREDUCE-2452. Makes the cancellation of delegation tokens happen in a
|
|
separate thread. (ddas)
|
|
|
|
HADOOP-7106. Reorganize project SVN layout to "unsplit" the projects.
|
|
(todd, nigel)
|
|
|
|
MAPREDUCE-2249. Check the reflexive property of Counters objects when
|
|
comparing equality. (Devaraj K via todd)
|
|
|
|
MAPREDUCE-2623. Update ClusterMapReduceTestCase to use
|
|
MiniDFSCluster.Builder (Harsh J Chouraria via eli)
|
|
|
|
MAPREDUCE-2602. Allow setting of end-of-record delimiter for
|
|
TextInputFormat for the old API. (Ahmed Radwan via todd)
|
|
|
|
MAPREDUCE-2705. Permits parallel multiple task launches.
|
|
(Thomas Graves via ddas)
|
|
|
|
MAPREDUCE-2489. Jobsplits with random hostnames can make the queue
|
|
unusable (jeffrey naisbit via mahadev)
|
|
|
|
MAPREDUCE-2854. update INSTALL with config necessary run mapred on yarn.
|
|
(thomas graves via mahadev)
|
|
|
|
MAPREDUCE-2701. app/Job.java needs UGI for the user that launched it.
|
|
(Robert Evans via mahadev)
|
|
|
|
MAPREDUCE-2652. Enabled multiple NMs to be runnable on a single node by
|
|
making shuffle service port to be truely configurable. (Robert Evans via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-2735. Add an applications summary log to ResourceManager.
|
|
(Thomas Graves via acmurthy)
|
|
|
|
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running
|
|
applications per-queue & per-user. (acmurthy)
|
|
Configuration changes:
|
|
add yarn.capacity-scheduler.maximum-am-resource-percent
|
|
|
|
MAPREDUCE-2774. Add startup message to ResourceManager & NodeManager on
|
|
startup. (Venu Gopala Rao via acmurthy)
|
|
|
|
MAPREDUCE-2655. Add audit logs to ResourceManager and NodeManager. (Thomas
|
|
Graves via acmurthy)
|
|
|
|
MAPREDUCE-2864. Normalize configuration variable names for YARN. (Robert
|
|
Evans via acmurthy)
|
|
|
|
MAPREDUCE-2690. Web-page for FifoScheduler. (Eric Payne via acmurthy)
|
|
|
|
MAPREDUCE-2711. Update TestBlockPlacementPolicyRaid for the new namesystem
|
|
and block management APIs. (szetszwo)
|
|
|
|
MAPREDUCE-2933. Change allocate call to return ContainerStatus for
|
|
completed containers rather than Container. (acmurthy)
|
|
|
|
MAPREDUCE-2675. Reformat JobHistory Server main page to be more
|
|
useful. (Robert Joseph Evans via vinodkv).
|
|
|
|
MAPREDUCE-2896. Simplify all apis to in
|
|
org.apache.hadoop.yarn.api.records.* to be get/set only. Added javadocs to
|
|
all public records. (acmurthy)
|
|
|
|
MAPREDUCE-2676. MR-279: JobHistory Job page needs reformatted. (Robert Evans via
|
|
mahadev)
|
|
|
|
MAPREDUCE-2899. Replace major parts of ApplicationSubmissionContext with a
|
|
ContainerLaunchContext (Arun Murthy via mahadev)
|
|
|
|
MAPREDUCE-2966. Added ShutDown hooks for MRV2 processes so that they can
|
|
gracefully exit. (Abhijit Suresh Shingate via vinodkv)
|
|
|
|
MAPREDUCE-2672. MR-279: JobHistory Server needs Analysis this job.
|
|
(Robert Evans via mahadev)
|
|
|
|
MAPREDUCE-2965. Streamlined the methods hashCode(), equals(), compareTo()
|
|
and toString() for all IDs. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2726. Added job-file to the AM and JobHistoryServer web
|
|
interfaces. (Jeffrey Naisbitt via vinodkv)
|
|
|
|
MAPREDUCE-2880. Improve classpath-construction for mapreduce AM and
|
|
containers. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3055. Simplified ApplicationAttemptId passing to
|
|
ApplicationMaster via environment variable. (vinodkv)
|
|
|
|
MAPREDUCE-3092. Removed a special comparator for JobIDs in JobHistory as
|
|
JobIDs are already comparable. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3099. Add docs for setting up a single node MRv2 cluster.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3001. Added task-specific counters to AppMaster and JobHistory
|
|
web-UIs. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3098. Fixed RM and MR AM to report YarnApplicationState and
|
|
application's FinalStatus separately. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-2889. Added documentation for writing new YARN applications.
|
|
(Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3134. Added documentation the CapacityScheduler. (acmurthy)
|
|
|
|
MAPREDUCE-3013. Removed YarnConfiguration.YARN_SECURITY_INFO and its usage
|
|
as it doesn't affect security any more. (vinodkv)
|
|
|
|
MAPREDUCE-2907. Changed log level for various messages in ResourceManager
|
|
from INFO to DEBUG. (Ravi Prakash via vinodkv)
|
|
|
|
MAPREDUCE-2702. Added a new API in OutputCommitter for recovering
|
|
the outputs of tasks from a crashed job so as to support MR Application
|
|
Master recovery. (Sharad Agarwal and Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-2738. Added the missing cluster level statistics on the RM web
|
|
UI. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-2988. Reenabled TestLinuxContainerExecutor reflecting the
|
|
current NodeManager code. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3161. Improved some javadocs and fixed some typos in
|
|
YARN. (Todd Lipcon via vinodkv)
|
|
|
|
MAPREDUCE-3148. Ported MAPREDUCE-2702 to old mapred api for aiding task
|
|
recovery. (acmurthy)
|
|
|
|
MAPREDUCE-3133. Running a set of methods in a Single Test Class.
|
|
(Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3059. QueueMetrics do not have metrics for aggregate
|
|
containers-allocated and aggregate containers-released.
|
|
(Devaraj K via mahadev)
|
|
|
|
MAPREDUCE-3187. Add names for various unnamed threads in MR2.
|
|
(Todd Lipcon and Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-3136. Added documentation for setting up Hadoop clusters in both
|
|
non-secure and secure mode for both HDFS & YARN. (acmurthy)
|
|
|
|
MAPREDUCE-3068. Added a whitelist of environment variables for containers
|
|
from the NodeManager and set MALLOC_ARENA_MAX for all daemons and
|
|
containers. (Chris Riccomini via acmurthy)
|
|
|
|
MAPREDUCE-3144. Augmented JobHistory with the information needed for
|
|
serving aggregated logs. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3163. JobClient spews errors when killing MR2 job.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3239. Use new createSocketAddr API in MRv2 to give better
|
|
error messages on misconfig (Todd Lipcon via mahadev)
|
|
|
|
MAPREDUCE-2747. Cleaned up LinuxContainerExecutor binary sources and changed
|
|
the configuration to use yarn names. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3205. Fix memory specifications to be physical rather than
|
|
virtual, allowing for a ratio between the two to be configurable. (todd
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-2986. Fixed MiniYARNCluster to support multiple NodeManagers.
|
|
(Anupam Seth via vinodkv)
|
|
|
|
MAPREDUCE-2736. Remove unused contrib components dependent on MR1. (eli)
|
|
|
|
MAPREDUCE-2989. Modified JobHistory to link to task and AM logs from the
|
|
JobHistoryServer. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3014. Rename and invert logic of '-cbuild' profile to 'native' and off
|
|
by default. (tucu)
|
|
|
|
MAPREDUCE-3171. normalize nodemanager native code compilation with common/hdfs
|
|
native. (tucu)
|
|
|
|
MAPREDUCE-3146. Added a MR specific command line to dump logs for a
|
|
given TaskAttemptID. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3275. Added documentation for AM WebApp Proxy. (Robert Evans via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3322. Added a better index.html and an brief overview of YARN
|
|
architecture. (acmurthy)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-2026. Make JobTracker.getJobCounters() and
|
|
JobInProgress.getCounters() aquire locks in a shorter time period.
|
|
(Joydeep Sen Sarma via schen)
|
|
|
|
MAPREDUCE-2740. MultipleOutputs in new API creates needless
|
|
TaskAttemptContexts. (todd)
|
|
|
|
MAPREDUCE-901. Efficient framework counters. (llu via acmurthy)
|
|
|
|
MAPREDUCE-2629. Workaround a JVM class loading quirk which prevents
|
|
JIT compilation of inner classes methods in ReduceContextImpl.
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-2603. Disable High-Ram emulation in system tests.
|
|
(Vinay Kumar Thota via amarrk)
|
|
|
|
MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-1978. Rumen TraceBuilder should provide recursive
|
|
input folder scanning.
|
|
|
|
MAPREDUCE-2416. Remove the restriction of specifying group names in
|
|
users-list file for Gridmix in RoundRobinUserResolver mode.
|
|
|
|
MAPREDUCE-2417. Fix Gridmix in RoundRobinUserResolver mode to
|
|
map testing/proxy users to unique users in a trace.
|
|
|
|
MAPREDUCE-2307. Exception thrown in Jobtracker logs, when the Scheduler
|
|
configured is FairScheduler. (Devaraj K via matei)
|
|
|
|
MAPREDUCE-2199. build is broken 0.22 branch creation. (cos)
|
|
|
|
MAPREDUCE-1752. Implement getFileBlockLocations in HarFilesystem.
|
|
(Patrick Kling via dhruba)
|
|
|
|
MAPREDUCE-2155. RaidNode should optionally use the mapreduce jobs to
|
|
fix missing blocks. (Patrick Kling via dhruba)
|
|
|
|
MAPREDUCE-1334. Fix TestIndexUpdater by ignoring _SUCCESS file in HDFS.
|
|
(Kay Kay via yhemanth)
|
|
|
|
MAPREDUCE-2232. Add missing methods to TestMapredGroupMappingServiceRefresh.
|
|
(Todd Lipcon via eli)
|
|
|
|
MAPREDUCE-2271. Fix TestSetupTaskScheduling failure on trunk.
|
|
(Liyin Liang via todd)
|
|
|
|
MAPREDUCE-2290. Fix compilation error in TestTaskCommit. (eli)
|
|
|
|
MAPREDUCE-2294. Fix compilation error in mumak. (todd)
|
|
|
|
MAPREDUCE-2300. Fix TestUmbilicalProtocolWithJobToken on trunk after
|
|
HADOOP-6904. (todd)
|
|
|
|
MAPREDUCE-2296. Fix references to misspelled method name
|
|
getProtocolSigature (todd)
|
|
|
|
MAPREDUCE-2311. Fix TestFairScheduler failure (schen)
|
|
|
|
MAPREDUCE-1996. API: Reducer.reduce() method detail misstatement.
|
|
(Harsh J Chouraria via tomwhite)
|
|
|
|
MAPREDUCE-2203. Wrong javadoc for TaskRunner's appendJobJarClasspaths
|
|
method. (Jingguo Yao via tomwhite)
|
|
|
|
MAPREDUCE-2074. Task should fail when symlink creation fails.
|
|
(Priyo Mustafi via tomwhite)
|
|
|
|
MAPREDUCE-1242. Chain APIs error misleading.
|
|
(Harsh J Chouraria via tomwhite)
|
|
|
|
MAPREDUCE-2379. Adds missing DistributedCache configurations in
|
|
mapred-default.xml (Todd Lipcon via amareshwari)
|
|
|
|
MAPREDUCE-2348. Disable mumak tests on trunk since they currently time out
|
|
(todd)
|
|
|
|
MAPREDUCE-2395. TestBlockFixer timing out on trunk. (Ramkumar Vadali via
|
|
todd)
|
|
|
|
MAPREDUCE-2426. Make TestFairSchedulerSystem fail with more verbose output
|
|
(todd)
|
|
|
|
MAPREDUCE-2448. NoSuchMethodError:
|
|
org.apache.hadoop.hdfs.TestDatanodeBlockScanner.corruptReplica(..) (eli)
|
|
|
|
MAPREDUCE-2460. Fix flaky test TestFairSchedulerSystem. (todd)
|
|
|
|
MAPREDUCE-2451. Log the details from health check script at the
|
|
JobTracker. (Thomas Graves via cdouglas)
|
|
|
|
MAPREDUCE-2467. HDFS-1052 changes break the raid contrib module in
|
|
MapReduce. (suresh srinivas via mahadev)
|
|
|
|
MAPREDUCE-2258. IFile reader closes stream and compressor in wrong order.
|
|
(todd via tomwhite)
|
|
|
|
MAPREDUCE-2518. The t flag is missing in distcp help message. (Wei Yongjun
|
|
via szetszwo)
|
|
|
|
MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log message.
|
|
(Jonathan Eagles via cdouglas)
|
|
|
|
MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
|
|
diagnosis of related issues. (Jonathan Eagles via cdouglas)
|
|
|
|
MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanup
|
|
thread dies. (Robert Joseph Evans via cdouglas)
|
|
|
|
MAPREDUCE-2470. Fix NPE in RunningJobs::getCounters. (Robert Joseph Evans
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-2536. Update FsShell -mv command usage in TestMRCLI. (Daryn
|
|
Sharp via szetszwo)
|
|
|
|
MAPREDUCE-2529. Add support for regex-based shuffle metric counting
|
|
exceptions. (Thomas Graves via cdouglas)
|
|
|
|
MAPREDUCE-2559. ant binary fails due to missing c++ lib dir. (eli)
|
|
|
|
MAPREDUCE-2573. Fix new findbugs warning introduced by MAPREDUCE-2494.
|
|
(Robert Joseph Evans via todd)
|
|
|
|
MAPREDUCE-2581. Spelling errors in log messages. (Tim Sell via eli)
|
|
|
|
MAPREDUCE-2588. Change raid to the new DataTransferProtocol API. (szetszwo)
|
|
|
|
MAPREDUCE-2576. Typo in comment in SimulatorLaunchTaskAction.java.
|
|
(Tim Sell via jghoman)
|
|
|
|
MAPREDUCE-2550. Fix bin/mapred to work properly from within a source
|
|
checkout (Eric Yang via todd)
|
|
|
|
MAPREDUCE-2620. Update RAID for HDFS-2087. (szetszwo)
|
|
|
|
MAPREDUCE-2624. Update RAID for HDFS-2107. (szetszwo)
|
|
|
|
MAPREDUCE-2670. Fixing spelling mistake in FairSchedulerServlet.java. (eli)
|
|
|
|
MAPREDUCE-2710. Update JobSubmitter.printTokens(..) for HDFS-2161.
|
|
(szetszwo)
|
|
|
|
MAPREDUCE-2409. DistributedCache maps files and archives to the same path,
|
|
despite semantic incompatibility. (Siddharth Seth via cdouglas)
|
|
|
|
MAPREDUCE-2575. TestMiniMRDFSCaching fails if test.build.dir is set
|
|
to something other than build/test (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-2622. Remove the last remaining reference to the deprecated
|
|
configuration "io.sort.mb". (Harsh J Chouraria via todd)
|
|
|
|
MAPREDUCE-2732. Remove directly accessing FSNamesystem.LOG from
|
|
TestCopyFiles and TestDistCh. (szetszwo)
|
|
|
|
MAPREDUCE-2463. Job history files are not moved to done folder when job
|
|
history location is hdfs. (Devaraj K via szetszwo)
|
|
|
|
MAPREDUCE-2243. Close streams propely in a finally-block to avoid leakage
|
|
in CompletedJobStatusStore, TaskLog, EventWriter and TotalOrderPartitioner.
|
|
(Devaraj K via szetszwo)
|
|
|
|
MAPREDUCE-2741. Make ant build system work with hadoop-common JAR
|
|
generated by Maven. (Alejandro Abdelnur via tomwhite)
|
|
|
|
MAPREDUCE-2760. mapreduce.jobtracker.split.metainfo.maxsize typoed
|
|
in mapred-default.xml. (todd via eli)
|
|
|
|
MAPREDUCE-2797. Update mapreduce tests and RAID for HDFS-2239. (szetszwo)
|
|
|
|
MAPREDUCE-2805. Update RAID for HDFS-2241. (szetszwo)
|
|
|
|
MAPREDUCE-2837. Ported bug fixes from y-merge to prepare for MAPREDUCE-279
|
|
merge. (acmurthy)
|
|
|
|
MAPREDUCE-2541. Fixed a race condition in IndexCache.removeMap. (Binglin
|
|
Chang via acmurthy)
|
|
|
|
MAPREDUCE-2458. Rename sanitized pom.xml in build directory to work around IDE
|
|
bug (Luke Lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Clear application notification if sent once
|
|
to NodeManager (mahadev)
|
|
|
|
MAPREDUCE-2433. YARNApplicationConstants hard code app master jar version (Luke
|
|
Lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implement restart for resource manager
|
|
phase 1 - Helper classes to store and restore the data structures. (mahadev)
|
|
|
|
MAPREDUCE-2414. Change MRv2 to use generic interfaces. (Siddharth Seth via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implement health-checks for the node -
|
|
server side(ResourceManager) changes. (vinodkv)
|
|
|
|
MAPREDUCE-2405: Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2)
|
|
(Greg Roelofs via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implementing Containers' memory monitoring.
|
|
(vinodkv)
|
|
|
|
MAPREDUCE-2440. Name clashes in TypeConverter (luke via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add fail count to the command line of the
|
|
application master. (mahadev)
|
|
|
|
MAPREDUCE-2424. Polish uber-AppMaster: add uber-AM counters and GUI indicators.
|
|
(Greg Roelofs via mahadev)
|
|
|
|
MAPREDUCE-2405. Implement uber-AppMaster (in-cluster LocalJobRunner for MRv2).
|
|
(Greg Roelofs and Sharad Agarwal via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix YarnRemoteException to give more
|
|
details. (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. WebApp for Job History (Krishna
|
|
Ramachandran via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Install sanitized poms for downstream
|
|
sanity (Luke Lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add HistoryCleanerService to Job History
|
|
server. (Krishna Ramachandran via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred job -list' and
|
|
'bin/mapred job -list-active-trackers'. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implement 'bin/mapred queue [-info
|
|
[-showJobs]] [-list] and enhanced 'bin/mapred job -list' to show queue and
|
|
ApplicationMaster information. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed computation of user-limits at
|
|
runtime. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added functionality to refresh queues at
|
|
runtime via the 'bin/yarn rmadmin' command. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added functionality to stop/start queues.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Introducing web-UI for NodeManager and
|
|
linking it from RM UI. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix race condition in TestJobHistoryEvents
|
|
and TestJobHistoryParsing. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add Containers' logs' view to NM UI and
|
|
link it from AM UI. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add ACLs for queues and command-line
|
|
utilities for viewing them. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Recovery of MR Application Master from
|
|
failures. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added support High-RAM applications in
|
|
CapacityScheduler. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Completing the ZooKeeper Store for
|
|
ResourceManager state. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Reorient container localization to be
|
|
per-container rather than per-application. (cdouglas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix file creation in
|
|
JobHistoryEventHandler. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Disable ContainerMonitoring for non-linux
|
|
systems. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix container launch w/ inconsistent
|
|
credential file naming. (cdouglas)
|
|
|
|
MAPREDUCE-2434. Metrics for ResourceManager. (Luke Lu via acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. RM Restart Phase 2 - Completed the recovery
|
|
of components in the RM (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix to send finish application event only
|
|
when the application is finished (mahadev)
|
|
|
|
MAPREDUCE-2462. Write job conf along with JobHistory, other minor improvements.
|
|
(Siddharth Seth via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implement 'delay scheduling' for better
|
|
locality in CapacityScheduler and improved high-ram applications. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Implement Job Acls in MR Application
|
|
Master. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Moving userlogs out of container work-dir
|
|
into a separate directory structure. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Completing RM Restart. Completed Phase 3 of
|
|
making sure events are logged and restored (mahadev)
|
|
|
|
MAPREDUCE-2468. Add metrics for NM Shuffle. (Luke Lu via cdouglas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Adding user log handling for YARN. Making
|
|
NM put the user-logs on DFS and providing log-dump tools. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing three tight-loops in RM that are
|
|
causing high cpu-usage. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Replacing FileContext usage with FileSystem
|
|
to work around security authentication issues with FileContext against a secure
|
|
DFS. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Client reconnect to restarted AM. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix refreshProxy in ClientServiceDelegate.
|
|
(sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix Null Pointer in TestUberAM. (sharad)
|
|
|
|
MAPREDUCE-2478. Improve history server. (Siddharth Seth via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Improved TestJobHistoryEvents and
|
|
TestJobHistoryParsing. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix NM to use multiple disks for local
|
|
files and the userlogs. (vinodkv)
|
|
|
|
MAPREDUCE-2480: MR App should not depend on hard coded version of shuffle (luke
|
|
lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Propagate error back to client in case of a
|
|
job submission failure (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix assembly to add mapreduce shell scripts
|
|
to the assembly package. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix TestQueueMetrics. (Luke Lu via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Removal of stale application-log dirs from
|
|
NM local disks. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add license header and minor cleanup in
|
|
history server. (Siddharth Seth via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Minor fix for install instructions.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix a race in MR task that was causing MR
|
|
containers to overwrite each other's job.xml. Also fix leaking attempt-dirs in
|
|
app-local-dir. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Adding valid state to ASM on a finish when
|
|
its already completed and also disble UberAM. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed CS user limits. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed reservation's bad interaction with
|
|
delay scheduling in CS. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Support mapreduce old (0.20) APIs. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Support fail-fast for MR jobs. (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for clearing container requests on an
|
|
AM failure and add tostring methods to taskids and taskattemptids for better
|
|
grep support. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Speed up communication between MR AM and RM
|
|
by relying on a new config rather than AM_EXPIRY_INTERVAL which is too large.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity to use
|
|
parent's absolute-capacity rather than the leaf queue's absolute-capacity.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in JobIDPbImpl that's causing
|
|
AM to crash randomly. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of maximum capacity in
|
|
ParentQueue to use its parent's absolute-capacity rather than its own
|
|
absolute-capacity. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Launching bin/yarn and bin/mapred only
|
|
*once* in AM for constructing classpaths to avoid multiple forks and huge vmem
|
|
usage by AM. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler to release unused
|
|
reservations on application completion. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix CapacityScheduler (LeafQueue) to not
|
|
allocate DATA_LOCAL containers when they are not required on the rack.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Makes uber-task disabled by default (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Make logging and memory for AM configurable
|
|
for the user via command line (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug in previous patch (r1103657).
|
|
Now bin/yarn truly shouldn't be launched multiple times in a single AM.
|
|
(vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing a bug to do with setting the staging
|
|
dir. (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed Composite Service to shutdown
|
|
services if an error occurs on starting any one of those (mahadev & chris)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix the tests to use jvm fork mode to avoid
|
|
errors in shutting down services (sidharth seth)
|
|
|
|
MAPREDUCE-2500. PB factories are not thread safe (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-2504. race in JobHistoryEventHandler stop (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix job hang if the AM launch fails.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-2509. Fix NPE in UI for pending attempts. (Luke Lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add junit jar to lib in assembly (mahadev
|
|
and luke)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Distributed cache bug fix to pass Terasort.
|
|
(vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix null pointer exception in kill task
|
|
attempt (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Refactored RMContainerAllocator to release
|
|
unused containers. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Changed Scheduler to return available limit
|
|
to AM in the allocate api. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix nodemanager expiry to not throw OOM.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Use DefaultContainerExecutor for
|
|
integration tests. (cdouglas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in test case (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for regression on the scheduling of
|
|
reduces before maps are done (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix distributed-cache related bugs.
|
|
(vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Ensure unused containers released by AM are
|
|
correctly counted for queue-capacity. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix TestRuntimeEstimators (Siddharth Seth
|
|
via ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix queue refresh to correctly record newly
|
|
added queues in CapacityScheduler. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added metrics for tracking reservations in
|
|
CapacityScheduler. (Luke Lu via acmurthy)
|
|
|
|
MAPREDUCE-2522. Security for JobHistory service. (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-2534. Fix CI breaking hard coded version in jobclient pom. (Luke Lu
|
|
via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add public cache. (cdouglas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Made number of RPC server threads
|
|
configurable. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added acl check for RMAdmin. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Adding job kill for any state that the job
|
|
is in with access control. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add debug statements for AM not launching
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing race condition leader to hung jobs
|
|
in scheduler negotiator (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add debug config for delaying delete of
|
|
local files. (cdouglas)
|
|
|
|
MAPREDUCE-2527. Metrics for MRAppMaster (Luke lu via mahadev)
|
|
|
|
MAPREDUCE-2532. Metrics for NodeManager (Luke Lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed an NPE during handling of unnecessary
|
|
reservations in CS. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for public dist cache to work with non
|
|
default hdfs (mahadev &ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Making streaming -file option work. Also
|
|
minor fixes for successful compilation of contrib tests. (vinodkv)
|
|
|
|
MAPREDUCE-2536. Backporting changes to MR-279.
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Bugfix for using user staging directory for
|
|
history files (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. First fix for making basic speculative
|
|
execution work (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixes for TestFail/Kill (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Set correct version of avro-maven-plugin
|
|
that is available in apache maven repositories. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing and reneabling
|
|
TestContainerTokenSecretManager. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Cleaning up configuration constants in
|
|
mapreduce modules. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing NPE on speculator in MRAppMaster and
|
|
making job-history optional in tests to make test goal succeed. (vinodk and
|
|
sharadag).
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed NPE in CS by checking Application
|
|
state before scheduling and fixing synchronization in CS. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Making pipes work with YARN. Changed pipes
|
|
to get log-locations from an environmental variable. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Ensure 'lost' NodeManagers are dealt
|
|
appropriately, the containers are released correctly. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Adding some more logging for AM expiry logs
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Reduce ramp up and zero maps support.
|
|
(sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Allowing hdfs calls from streaming/pipes
|
|
tasks. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added ability to decommission nodes and
|
|
completed RM administration tools to achieve parity with JobTracker. (acmurthy)
|
|
|
|
MAPREDUCE-2551. Added JobSummaryLog. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-2552. Fixed NPE in CompletedJob in JobHistoryServer. (Siddharth Seth
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix reduce slow start. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed TestFifoScheduler. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix build issue for using yarn.version
|
|
instead of hadoop-mapred.version (mahadev and giri)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixes in the handling of KILL events in the
|
|
SUCCEEDED state for tasks in the application master (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestRMNMRPCResponseId.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix a deadlock in the resourcemanager.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. NodeStatus.getNodeHealthStatus().setBlah
|
|
broken (Siddharth Seth)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix another NPE in TestRMNMRPCResponseId.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in TestNMExpiry (siddharth
|
|
seth)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Making each node aggregate all its
|
|
user-logs to a separate hdfs file. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix calculation of max-capacity for a
|
|
queue, also fixed a bug in registration of NodeManagers. (acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. More cleaning up constants, removing stale
|
|
code, and making conspicuous the envs that apps depend on to be provided by
|
|
YARN. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix container size rounding in AM and
|
|
headroom in RM. (acmurthy and sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Disable Job acls until fixed (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix to report job status if the application
|
|
is KILLED/FAILED. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix a corner case in headroom computation -
|
|
now reservations are taken into account and headroom is computed much later to
|
|
account for allocations/reservations. (acmurthy)
|
|
|
|
MAPREDUCE-2537. The RM writes its log to
|
|
yarn-mapred-resourcemanager-<RM_Host>.out (Robert Evans via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix logging for showing the state of job
|
|
(FAILED/KILLED/SUCCEEDED) when it completes (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Re-enabled TestCapacityScheduler.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Support for min and max container capacity.
|
|
(acmurthy and sharad)
|
|
|
|
MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via acmurthy)
|
|
|
|
MAPREDUCE-2539. Fixed NPE in getMapTaskReports in JobClient. (Robert Evans via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing the wrong config key used in
|
|
JobHistory that prevented configuring move-thread interval. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed inconsistency in QueueACL enums.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix various issues with Web UI's. (Luke Lu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in Task abort for
|
|
old mapreduce apis. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add deletion of distributed cache
|
|
resources. (cdouglas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Disable aggregation of logs onto DFS till
|
|
JobHistoryServer starts serving logs. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Cleanup redundant code in TaskAttemptImpl.
|
|
(sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Work around broken signaling in public
|
|
cache. (cdouglas)
|
|
|
|
MAPREDUCE-2566. YarnConfiguration should reloadConfiguration if instantiated
|
|
with a non YarnConfiguration object. (Siddharth Seth)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fully resolve paths when launching
|
|
containers. (Siddharth Seth)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Re-enabling Uber-AM feature. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed deadlock during expiring NMs.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Solving NPEs during
|
|
ContainerManager#StopContainer. Also removing the unused
|
|
ContainerManager#CleanupContainer api. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Remove retries in dist cache so that NM's
|
|
do not shutdown (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix classpath construction for Task.
|
|
(vinodkv via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Sending Job diagnostics from AM to RM and
|
|
redirect to history-url on job completion. (vinodkv and sharadag)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added clover in pom dependency. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster
|
|
Linux desktop. (Luke lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Including source files in release
|
|
distribution (Luke Lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Intermittent TestMRApp failures on faster
|
|
Linux desktop (part 2) (Luke lu via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Disable Uber AM. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Added few job diagnostic messages. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Add ability to includes src files in
|
|
assembly target for maven (Luke Lu via mahadev)
|
|
|
|
MAPREDUCE-2582. Cleanup JobHistory event generation.(Siddharth Seth via sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix rounding off problem in reduce ramp up.
|
|
(sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix more rounding off problems in reduce
|
|
ramp up. Also fix a bug preventing the application of the cap on reduce ramp-up.
|
|
(Sharad Agarwal via vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix to exclude images dir into the tar
|
|
distribution (Luke Lu via gkesavan)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext
|
|
to FileSystem in YarnRunner to handle distributed cache path resolutions on
|
|
non-default filesystems. (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Display failed/killed attempts of the task
|
|
on MR AM UI separately. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Miscellaneous UI fixes + source code
|
|
formatting for MR JobHistoryEventHandler. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing broken link to logs for container on
|
|
NM web UI. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing the bug which was causing FAILED
|
|
jobs to be displayed as COMPLETED on the RM UI. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Job level node blacklisting. (sharad)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in history event handling
|
|
(Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-2569. Ensure root queue allocated 100% capacity. (Jonathan Eagles via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix ClassCastException in JobHistoryServer
|
|
for certain jobs. (Siddharth Seth via llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Changes for invoking rack resolution in the
|
|
RM and in the AM (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix concurrent modification exception in
|
|
the Capacity Scheduler (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix stopContainer for setsid challenged
|
|
platforms. (llu)
|
|
|
|
MAPREDUCE-2587. Generate yarn version for UI. (Thomas Graves via lluts page to
|
|
the history server UI. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Bug fix to set correct state on containers
|
|
so as to avoid duplicate containers from RM to AM. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Hack until MAPREDUCE-2365 is fixed to make
|
|
PIG work with MRV2. (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Changes a couple of usages of FileContext
|
|
to FileSystem in TaskAttemptImpl to handle distributed cache path resolutions on
|
|
non-default filesystems. (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE when requesting attempts for
|
|
completed jobs. (Siddharth Seth via llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Improve logging for AM when requesting
|
|
containers to show the right ask and release fields (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix race condition between multiple
|
|
localizers on a single node. (cdouglas via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix RM app start/finish time and
|
|
diagnostics. (llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix to schedule reduces irrespective of the
|
|
headroom when all maps are done so as to avoid stall in reduce-scheduling when
|
|
slow-start is disabled. (Sharad Agarwal via vinodkv).
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Disabling locality-wait in
|
|
CapacityScheduler for now to prevent uber-slow scheduling for apps with no
|
|
data-locality constraints (sleep-job like). (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixing scheduling deadlock in AM because of
|
|
incorrect headRoom values from RM. The bug happens when AM releases containers
|
|
and RM decrements current memory usage twice for all those containers. (vinodkv)
|
|
|
|
MAPREDUCE-2611. Fix counters, finish times etc. in job history. (Siddharth Seth
|
|
via llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for ConcurrentModification exception
|
|
while iterating through tokens in a UGI in ContainerLauncherImpl. (ddas)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for NPE in YarnChild that was causing
|
|
lots of tasks to fail. (vinodkv)
|
|
|
|
MAPREDUCE-2615. Make killJob go through AM and fix JobSummaryLog. (Siddharth
|
|
Seth via llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix class cast exception in release
|
|
reserved containers in capacity scheduler (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix diagnostics display for more than 100
|
|
apps in RM. (llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix some invalid transitions in the RM.
|
|
(vinodkv via ddas)
|
|
|
|
MAPREDUCE-2618. Fix NPE in 0 map 0 reduce jobs. (Jeffrey Naisbitt via llu)
|
|
|
|
MAPREDUCE-2625. Add version info to nodemanager info page. (Jonathan Eagles via
|
|
llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. (1) Faster retries from AM to HistoryServer
|
|
(2) Correct diagnostics for containers. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Major ASM cleanup. Streamlining classes,
|
|
interface and events. (vinodkv)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Reinstate resolve path fixes for viewfs.
|
|
(Siddharth Seth via llu)
|
|
|
|
MAPREDUCE-2633. Add a getCounter(Enum) method to the Counters record. (Josh
|
|
Wills via sharad)
|
|
|
|
MAPREDUCE-2645. Updates to MRv2 INSTALL and README documentation. (Josh Wills
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-2628. Add compiled on date to NM and RM info/about page.
|
|
|
|
MAPREDUCE-2400. Remove Cluster's dependency on JobTracker via a ServiceProvider
|
|
for the actual implementation. (tomwhite via acmurthy)
|
|
|
|
MAPREDUCE-2663. Refactoring StateMachineFactory inner classes. (ahmed radwan via
|
|
mahadev)
|
|
|
|
MAPREDUCE-2678. minimum-user-limit-percent no longer honored. (naisbitt via
|
|
mahadev)
|
|
|
|
MAPREDUCE-2630. refreshQueues leads to NPEs when used w/FifoScheduler. (Josh
|
|
Wills via mahadev)
|
|
|
|
MAPREDUCE-2644. NodeManager fails to create containers when NM_LOG_DIR is not
|
|
explicitly set in the Configuration. (Josh Wills via vinodkv)
|
|
|
|
MAPREDUCE-2661. Fix TaskImpl to not access MapTaskImpl. (Ahmed Radwan via
|
|
sharad)
|
|
|
|
HADOOP-6929. Backport changes to MR-279 (mahadev and owen)
|
|
|
|
HADOOP-6929. Making Security Info abstract and not an interface (mahadev)
|
|
|
|
MAPREDUCE-2667. mapred job -kill leaves application in RUNNING state (thomas
|
|
graves via mahadev)
|
|
|
|
MAPREDUCE-2664. Implement JobCounters for Mtions as asynchronous. (vinodkv,
|
|
sharad and acmurthy)
|
|
|
|
MAPREDUCE-2773. server.api.records.NodeHealthStatus renamed but not updated in
|
|
client NodeHealthStatus.java (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-2772. Fix MR-279 build after common mavenization. (Robert Joseph Evans
|
|
via llu)
|
|
|
|
MAPREDUCE-2772. Fix MR-279 build after common mavenization, part 2. (Thomas
|
|
Graves via llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Harmonize slf4j versions. (llu)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix NPE in FifoScheduler. (mahadev)
|
|
|
|
MAPREDUCE-2776. Fix some of the yarn findbug warnings. (Siddharth Seth via
|
|
mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules,
|
|
part 1 (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules
|
|
part 2 (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix findbugs warnings in mr-client modules
|
|
part 3 (mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix the poms to enable 0.23 snapshots for
|
|
hdfs/common from apache nightly builds (gkesavan)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix ivy conf to work with the hadoop common
|
|
trunk maven build changes. (Giridharan Kesavan)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Patch for findbugs warnings in Resource
|
|
Manager (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fix for running ant targets to use the
|
|
right set of common/test jars (gkesavan via mahadev)
|
|
|
|
MAPREDUCE-2782. Unit tests for CapacityScheduler. (acmurthy)
|
|
|
|
MAPREDUCE-2706. Log job submission failures. (Jeffrey Naisbitt via acmurthy)
|
|
|
|
MAPREDUCE-2781. mr279 RM application finishtime not set (Thomas Graves via
|
|
mahadev)
|
|
|
|
MAPREDUCE-279. Fix in MR-279 branch. Fixed CS locality wait factor. (acmurthy)
|
|
|
|
MAPREDUCE-2808. pull MAPREDUCE-2797 into mr279 branch (Thomas Graves via
|
|
mahadev)
|
|
|
|
MAPREDUCE-2639. Bug fixes in speculate.DataStatistics. (Josh Wills via acmurthy)
|
|
|
|
MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new and old
|
|
apis. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals 0.
|
|
(Jeffrey Naisbitt via acmurthy)
|
|
|
|
MAPREDUCE-2839. Fixed TokenCache to get delegation tokens using both new
|
|
and old apis. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-2727. Fix divide-by-zero error in SleepJob for sleepCount equals
|
|
0. (Jeffrey Naisbitt via acmurthy)
|
|
|
|
MAPREDUCE-2860. Fix log4j logging in the maven test cases. (mahadev)
|
|
|
|
MAPREDUCE-2867. Remove Unused TestApplicaitonCleanup in resourcemanager/applicationsmanager.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-2868. ant build broken in hadoop-mapreduce dir (mahadev, giri and arun via mahadev)
|
|
|
|
MAPREDUCE-2649. Handling of finished applications in RM. (Thomas Graves
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-2838. Fix MapReduce builds to use new hadoop-common test jars.
|
|
(gkesavan via acmurthy)
|
|
|
|
MAPREDUCE-2859. Fix eclipse plugin contrib module compilation (gkesavan)
|
|
|
|
MAPREDUCE-2846. Fix missing synchronization in the task log management.
|
|
(omalley)
|
|
|
|
MAPREDUCE-2807. Fix AM restart and client redirection. (sharad)
|
|
|
|
MAPREDUCE-2877. Add missing Apache license header in some files in MR
|
|
and also add the rat plugin to the poms. (mahadev)
|
|
|
|
MAPREDUCE-2796. Set start times for MR applications for clients to see.
|
|
(Devaraj K via acmurthy)
|
|
|
|
MAPREDUCE-2879. Fix version for MR-279 to 0.23.0. (acmurthy)
|
|
|
|
MAPREDUCE-2881. Fix to include log4j 1.2.16 depenency (gkesavan)
|
|
|
|
MAPREDUCE-2885. Fix mapred-config.sh to look for hadoop-config.sh in
|
|
HADOOP_COMMON_HOME/libexec. (acmurthy)
|
|
|
|
MAPREDUCE-2893. Remove duplicate entry of YarnClientProtocolProvider in
|
|
ClientProtocolProvider services file. (Liang-Chi Hsieh via acmurthy)
|
|
|
|
MAPREDUCE-2891. Javadoc for AMRMProtocol and related records. (acmurthy)
|
|
|
|
MAPREDUCE-2898. Javadoc for ContainerManager protocol and related records.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-2904. Fixed bin/yarn to correctly include HDFS jars and
|
|
clean up of stale refs to pre-mavenized Hadoop Common and HDFS.
|
|
(Sharad Agarwal and Arun C. Murthy via acmurthy)
|
|
|
|
MAPREDUCE-2737. Update the progress of jobs on client side. (Siddharth Seth
|
|
and Mahadev Konar via mahadev)
|
|
|
|
MAPREDUCE-2886. Fix Javadoc warnings in MapReduce. (mahadev)
|
|
|
|
MAPREDUCE-2897. Javadoc for ClientRMProtocol protocol and related records.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-2916. Ivy build for MRv1 fails with bad organization for
|
|
common daemon. (mahadev)
|
|
|
|
MAPREDUCE-2917. Fixed corner case in container reservation which led to
|
|
starvation and hung jobs. (acmurthy)
|
|
|
|
MAPREDUCE-2756. Better error handling in JobControl for failed jobs.
|
|
(Robert Evans via acmurthy)
|
|
|
|
MAPREDUCE-2716. MRReliabilityTest job fails because of missing
|
|
job-file. (Jeffrey Naisbitt via vinodkv)
|
|
|
|
MAPREDUCE-2882. TestLineRecordReader depends on ant jars. (todd)
|
|
|
|
MAPREDUCE-2687. Fix NodeManager to use the right version of
|
|
LocalDirAllocator.getLocalPathToWrite. (mahadev & acmurthy)
|
|
|
|
MAPREDUCE-2800. Set final progress for tasks to ensure all task information
|
|
is correctly logged to JobHistory. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-2938. Log application submission failure in CapacityScheduler.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-2948. Hadoop streaming test failure, post MR-2767 (mahadev)
|
|
|
|
MAPREDUCE-2908. Fix all findbugs warnings. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-2947. Fixed race condition in AuxiliaryServices. (vinodkv via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2844. Fixed display of nodes in UI. (Ravi Teja Ch N V via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2677. Fixed 404 for some links from HistoryServer. (Robert Evans
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-2937. Ensure reason for application failure is displayed to the
|
|
user. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-2953. Fix a race condition on submission which caused client to
|
|
incorrectly assume application was gone by making submission synchronous
|
|
for RMAppManager. (Thomas Graves via acmurthy)
|
|
|
|
MAPREDUCE-2963. Fix hang in TestMRJobs. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-2954. Fixed a deadlock in NM caused due to wrong synchronization
|
|
in protocol buffer records. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2975. Fixed YARNRunner to use YarnConfiguration rather than
|
|
Configuration. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-2971. ant build mapreduce fails protected access jc.displayJobList
|
|
(jobs) (Thomas Graves via mahadev)
|
|
|
|
MAPREDUCE-2691. Finishing up the cleanup of distributed cache file resources
|
|
and related tests. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2749. Ensure NM registers with RM after starting all its services
|
|
correctly. (Thomas Graves via acmurthy)
|
|
|
|
MAPREDUCE-2979. Removed the needless ClientProtocolProvider configuration
|
|
from the hadoop-mapreduce-client-core module. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2985. Fixed findbugs warnings in ResourceLocalizationService.
|
|
(Thomas Graves via acmurthy)
|
|
|
|
MAPREDUCE-2874. Fix formatting of ApplicationId in web-ui. (Eric Payne via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2995. Better handling of expired containers in MapReduce
|
|
ApplicationMaster. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-2995. Fixed race condition in ContainerLauncher. (vinodkv via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2949. Fixed NodeManager to shut-down correctly if a service
|
|
startup fails. (Ravi Teja via vinodkv)
|
|
|
|
MAPREDUCE-3005. Fix both FifoScheduler and CapacityScheduler to correctly
|
|
enforce locality constraints. (acmurthy)
|
|
|
|
MAPREDUCE-3007. Fixed Yarn Mapreduce client to be able to connect to
|
|
JobHistoryServer in secure mode. (vinodkv)
|
|
|
|
MAPREDUCE-2987. Fixed display of logged user on RM Web-UI. (Thomas Graves
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3006. Fixed MapReduce AM to exit only after properly writing out
|
|
history file. (vinodkv)
|
|
|
|
MAPREDUCE-2925. Fixed Yarn+MR client code to behave saner with completed
|
|
jobs. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3030. Fixed a bug in NodeId.equals() that was causing RM to
|
|
reject all NMs. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3042. Fixed default ResourceTracker address. (Chris Riccomini
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3038. job history server not starting because conf() missing
|
|
HsController (Jeffrey Naisbitt via mahadev)
|
|
|
|
MAPREDUCE-3004. Fix ReduceTask to not assume 'local' mode in YARN. (Hitesh
|
|
Shah via acmurthy)
|
|
|
|
MAPREDUCE-3017. The Web UI shows FINISHED for killed/successful/failed jobs.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3040. Fixed extra copy of Configuration in
|
|
YarnClientProtocolProvider and ensured MiniMRYarnCluster sets JobHistory
|
|
configuration for tests. (acmurthy)
|
|
|
|
MAPREDUCE-3018. Fixed -file option for streaming. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-3036. Fixed metrics for reserved resources in CS. (Robert Evans
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-2998. Fixed a bug in TaskAttemptImpl which caused it to fork
|
|
bin/mapred too many times. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3023. Fixed clients to display queue state correctly. (Ravi
|
|
Prakash via acmurthy)
|
|
|
|
MAPREDUCE-2970. Fixed NPEs in corner cases with different configurations
|
|
for mapreduce.framework.name. (Venu Gopala Rao via vinodkv)
|
|
|
|
MAPREDUCE-3062. Fixed default RMAdmin address. (Chris Riccomini
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3066. Fixed default ResourceTracker address for the NodeManager.
|
|
(Chris Riccomini via acmurthy)
|
|
|
|
MAPREDUCE-3044. Pipes jobs stuck without making progress. (mahadev)
|
|
|
|
MAPREDUCE-2754. Fixed MR AM stdout, stderr and syslog to redirect to
|
|
correct log-files. (Ravi Teja Ch N V via vinodkv)
|
|
|
|
MAPREDUCE-3073. Fixed build issues in MR1. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-2691. Increase threadpool size for launching containers in
|
|
MapReduce ApplicationMaster. (vinodkv via acmurthy)
|
|
|
|
|
|
MAPREDUCE-2990. Fixed display of NodeHealthStatus. (Subroto Sanyal via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3053. Better diagnostic message for unknown methods in ProtoBuf
|
|
RPCs. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-2952. Fixed ResourceManager/MR-client to consume diagnostics
|
|
for AM failures in a couple of corner cases. (Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3064. 27 unit test failures with Invalid
|
|
"mapreduce.jobtracker.address" configuration value for
|
|
JobTracker: "local" (Venu Gopala Rao via mahadev)
|
|
|
|
MAPREDUCE-3090. Fix MR AM to use ApplicationAttemptId rather than
|
|
(ApplicationId, startCount) consistently. (acmurthy)
|
|
|
|
MAPREDUCE-2646. Fixed AMRMProtocol to return containers based on
|
|
priority. (Sharad Agarwal and Arun C Murthy via vinodkv)
|
|
|
|
MAPREDUCE-3031. Proper handling of killed containers to prevent stuck
|
|
containers/AMs on an external kill signal. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-2984. Better error message for displaying completed containers.
|
|
(Devaraj K via acmurthy)
|
|
|
|
MAPREDUCE-3071. app master configuration web UI link under the Job menu
|
|
opens up application menu. (thomas graves via mahadev)
|
|
|
|
MAPREDUCE-3067. Ensure exit-code is set correctly for containers. (Hitesh
|
|
Shah via acmurthy)
|
|
|
|
MAPREDUCE-2999. Fix YARN webapp framework to properly filter servlet
|
|
paths. (Thomas Graves via vinodkv)
|
|
|
|
MAPREDUCE-3095. fairscheduler ivy including wrong version for hdfs.
|
|
(John George via mahadev)
|
|
|
|
MAPREDUCE-3054. Unable to kill submitted jobs. (mahadev)
|
|
|
|
MAPREDUCE-3021. Change base urls for RM web-ui. (Thomas Graves via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3041. Fixed ClientRMProtocol to provide min/max resource
|
|
capabilities along-with new ApplicationId for application submission.
|
|
(Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-2843. Fixed the node-table to be completely displayed and making
|
|
node entries on RM UI to be sortable. (Abhijit Suresh Shingate via vinodkv)
|
|
|
|
MAPREDUCE-3110. Fixed TestRPC failure. (vinodkv)
|
|
|
|
MAPREDUCE-3078. Ensure MapReduce AM reports progress correctly for
|
|
displaying on the RM Web-UI. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3114. Fixed invalid ApplicationURL on RM WebUI. (Subroto Sanyal
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-2791. Added missing info on 'job -status' output. (Devaraj K via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2996. Add uber-ness information to JobHistory. (Jonathan Eagles
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3050. Add ability to get resource usage information for
|
|
applications and nodes. (Robert Evans via acmurthy)
|
|
|
|
MAPREDUCE-3113. Ensure bin/yarn and bin/yarn-daemon.sh identify the root
|
|
of the install properly. (Xie Xianshan via acmurthy)
|
|
|
|
MAPREDUCE-3137. Fix broken merge of MAPREDUCE-2179. (Hitesh Shah via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-2792. Replace usage of node ip-addresses with hostnames.
|
|
(vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3112. Fixed recursive sourcing of HADOOP_OPTS environment
|
|
variable. (Eric Yang)
|
|
|
|
MAPREDUCE-3056. Changed the default staging directory to not include
|
|
user.name to prevent issues with non-secure mode. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-2913. Fixed TestMRJobs.testFailingMapper to assert the correct
|
|
TaskCompletionEventStatus. (Jonathan Eagles via vinodkv)
|
|
|
|
MAPREDUCE-2794. [MR-279] Incorrect metrics value for AvailableGB per
|
|
queue per user. (John George via mahadev)
|
|
|
|
MAPREDUCE-2783. Fixing RM web-UI to show no tracking-URL when AM
|
|
crashes. (Eric Payne via vinodkv)
|
|
|
|
MAPREDUCE-3141. Fix the broken MRAppMaster to work over YARN in security
|
|
mode.(vinodkv)
|
|
|
|
MAPREDUCE-2751. Modified NodeManager to stop leaving around local files
|
|
after application finishes. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3033. Ensure Master interface pays attention to classic v/s yarn
|
|
frameworks. (Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-2802. Ensure JobHistory filenames have jobId. (Jonathan Eagles
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-2876. Use a different config for ContainerAllocationExpirer.
|
|
(Anupam Seth via acmurthy)
|
|
|
|
MAPREDUCE-3153. Fix TestFileOutputCommitter which was broken by
|
|
MAPREDUCE-2702. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-3123. Fix NM to quote symlink names to escape special
|
|
characters. (Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3154. Fix JobSubmitter to check for output specs before copying
|
|
job submission files to fail fast. (Abhijit Suresh Shingate via acmurthy)
|
|
|
|
MAPREDUCE-3158. Fix test failures in MRv1 due to default framework being
|
|
set to yarn. (Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3167. container-executor is not being packaged with the assembly
|
|
target. (mahadev)
|
|
|
|
MAPREDUCE-3020. Fixed TaskAttemptImpl to log the correct node-address for
|
|
a finished Reduce task. (Chackaravarthy via vinodkv)
|
|
|
|
MAPREDUCE-2668. Fixed AuxServices to send a signal on application-finish
|
|
to all the services. (Thomas Graves via vinodkv)
|
|
|
|
MAPREDUCE-3126. Fixed a corner case in CapacityScheduler where headroom
|
|
wasn't updated on changes to cluster size. (acmurthy)
|
|
|
|
MAPREDUCE-3140. Fixed the invalid JobHistory URL for failed
|
|
applications. (Subroto Sanyal via vinodkv)
|
|
|
|
MAPREDUCE-3125. Modified TaskImpl to consider only non-failed, non-killed
|
|
task-attempts for obtaining task's progress. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-2666. Retrieve shuffle port number from JobHistory on MR AM
|
|
restart. (Jonathan Eagles via acmurthy)
|
|
|
|
MAPREDUCE-2789. Complete schedulingInfo on CLI. (Eric Payne via acmurthy)
|
|
|
|
MAPREDUCE-3170. Fixed job output commit for deep hierarchies. (Hitesh Shah
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3124. Fixed location of native libs i.e. libhadoop.so for
|
|
containers. (John George via acmurthy)
|
|
|
|
MAPREDUCE-3057. Job History Server goes of OutOfMemory with 1200 Jobs
|
|
and Heap Size set to 10 GB. (Eric Payne via mahadev)
|
|
|
|
MAPREDUCE-2840. mr279 TestUberAM.testSleepJob test fails. (jonathan eagles
|
|
via mahadev)
|
|
|
|
MAPREDUCE-3190. Ensure bin/yarn fails early with a clear error message
|
|
when HADOOP_COMMON_HOME or HADOOP_HDFS_HOME are not set. (todd & acmurthy
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3189. Add link decoration back to MR2's CSS. (Todd Lipcon via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3127. Changed default value of yarn.resourcemanager.acl.enable
|
|
to true and added some more documentation. (acmurthy)
|
|
|
|
MAPREDUCE-3032. Fixed TaskAttemptImpl so that JobHistory can have error
|
|
information about failed tasks. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3196. TestLinuxContainerExecutorWithMocks fails on Mac OSX.
|
|
(Arun Murthy via mahadev)
|
|
|
|
MAPREDUCE-3197. TestMRClientService failing on building clean checkout of
|
|
branch 0.23 (mahadev)
|
|
|
|
MAPREDUCE-2762. Cleanup MR staging directory on completion. (mahadev via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3165. Ensure logging options are set correctly for MR AM and
|
|
tasks. (todd via acmurthy)
|
|
|
|
MAPREDUCE-3203. Fix some javac warnings in MRAppMaster. (mahadev)
|
|
|
|
MAPREDUCE-3162. Separated application-init and container-init event types
|
|
in NodeManager's Application state machine. (Todd Lipcon via vinodkv)
|
|
|
|
MAPREDUCE-3176. Fixed ant mapreduce tests that are timing out because
|
|
of wrong framework name. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-3181. Fixed MapReduce runtime to load yarn-default.xml and
|
|
yarn-site.xml. (acmurthy)
|
|
|
|
MAPREDUCE-2788. Normalize resource requests in FifoScheduler
|
|
appropriately. (Ahmed Radwan via acmurthy)
|
|
|
|
MAPREDUCE-2693. Fix NPE in job-blacklisting. (Hitesh Shah via acmurthy)
|
|
|
|
MAPREDUCE-3208. Fix NPE task/container log appenders. (liangzhwa via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3212. Fix usage/help message for bin/yarn. (Bhallamudi Venkata
|
|
Siva Kamesh via acmurthy)
|
|
|
|
MAPREDUCE-3179. Ensure failed tests exit with right error code. (Jonathan
|
|
Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3188. Ensure correct shutdown in services. (todd via acmurthy)
|
|
|
|
MAPREDUCE-3226. Fix shutdown of fetcher threads. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3070. Fix NodeManager to use ephemeral ports by default.
|
|
(Devaraj K via acmurthy)
|
|
|
|
MAPREDUCE-3242. Trunk compilation broken with bad interaction from
|
|
MAPREDUCE-3070 and MAPREDUCE-3239. (mahadev)
|
|
|
|
MAPREDUCE-3058. Fixed MR YarnChild to report failure when task throws an
|
|
error and thus prevent a hanging task and job. (vinodkv)
|
|
|
|
MAPREDUCE-3087. Fixed the mapreduce classpath to correctly include the
|
|
generated-classpath file needed for tests. (Ravi Prakash via vinodkv)
|
|
|
|
MAPREDUCE-3233. Fixed a bug in MR Job so as to be able to restart the
|
|
application on AM crash. (Mahadev Konar via vinodkv)
|
|
|
|
MAPREDUCE-3028. Added job-end notification support. (Ravi Prakash via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3249. Ensure shuffle-port is correctly used duringMR AM recovery.
|
|
(vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3252. Fix map tasks to not rewrite data an extra time when
|
|
map output fits in spill buffer. (todd)
|
|
|
|
MAPREDUCE-3159. Ensure DefaultContainerExecutor doesn't delete application
|
|
directories during app-init. (todd via acmurthy)
|
|
|
|
MAPREDUCE-2746. Yarn servers can't communicate with each other with
|
|
hadoop.security.authorization set to true (acmurthy via mahadev)
|
|
|
|
MAPREDUCE-2821. Added missing fields (resourcePerMap & resourcePerReduce)
|
|
to JobSummary logs. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-3253. Fixed ContextFactory to clone JobContext correctly.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-3263. Fixed the MAPREDUCE-3028 commit which broke MR1. (Hitesh
|
|
Shah via acmurthy)
|
|
|
|
MAPREDUCE-3269. Fixed log4j properties to correctly set logging options
|
|
for JobHistoryServer vis-a-vis JobSummary logs. (mahadev via acmurthy)
|
|
|
|
MAPREDUCE-3250. When AM restarts, client keeps reconnecting to the new AM
|
|
and prints a lots of logs. (vinodkv via mahadev)
|
|
|
|
MAPREDUCE-3254. Fixed streaming to set the job.jar by using the right
|
|
JobConf ctor. (acmurthy)
|
|
|
|
MAPREDUCE-3264. mapreduce.job.user.name needs to be set automatically.
|
|
(acmurthy via mahadev)
|
|
|
|
MAPREDUCE-3175. Add authorization to admin web-pages such as /stacks, /jmx
|
|
etc. (Jonathan Eagles via acmurthy)
|
|
|
|
MAPREDUCE-3257. Added authorization checks for the protocol between
|
|
ResourceManager and ApplicationMaster. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3259. Added java.library.path of NodeManager to
|
|
ContainerLocalizer in LinuxContainerExecutor. (Kihwal Lee via acmurthy)
|
|
|
|
MAPREDUCE-3279. Fixed TestJobHistoryParsing which assumed user name to be
|
|
mapred all the time. (Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-3240. Fixed NodeManager to be able to forcefully cleanup its
|
|
containers (process-trees) irrespective of whether the container succeeded,
|
|
or killed. Contributed by Hitesh Shah.
|
|
|
|
MAPREDUCE-3281. Fixed a bug in TestLinuxContainerExecutorWithMocks. (vinodkv)
|
|
|
|
MAPREDUCE-3228. Fixed MR AM to timeout RPCs to bad NodeManagers. (vinodkv
|
|
via acmurthy)
|
|
|
|
MAPREDUCE-3284. Moved JobQueueClient to hadoop-mapreduce-client-core.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-3282. bin/mapred job -list throws exception. (acmurthy via
|
|
mahadev)
|
|
|
|
MAPREDUCE-3186. User jobs are getting hanged if the Resource manager
|
|
process goes down and comes up while job is getting executed.
|
|
(Eric Payne via mahadev)
|
|
|
|
MAPREDUCE-3285. Tests on branch-0.23 failing (Siddharth Seth via mahadev)
|
|
|
|
MAPREDUCE-3258. Fixed AM & JobHistory web-ui to display counters properly.
|
|
(Siddharth Seth via acmurthy)
|
|
|
|
MAPREDUCE-3290. Fixed a NPE in ClientRMService. (acmurthy)
|
|
|
|
MAPREDUCE-3185. RM Web UI does not sort the columns in some cases.
|
|
(Jonathan Eagles via mahadev)
|
|
|
|
MAPREDUCE-3292. In secure mode job submission fails with Provider
|
|
org.apache.hadoop.mapreduce.security.token.JobTokenIndentifier$Renewer
|
|
not found. (mahadev)
|
|
|
|
MAPREDUCE-3296. Fixed the remaining nine FindBugs warnings. (vinodkv)
|
|
|
|
MAPREDUCE-2775. Fixed ResourceManager and NodeManager to force a
|
|
decommissioned node to shutdown. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-3304. Fixed intermittent test failure due to a race in
|
|
TestRMContainerAllocator#testBlackListedNodes. (Ravi Prakash via acmurthy)
|
|
|
|
MAPREDUCE-3306. Fixed a bug in NodeManager ApplicationImpl that was causing
|
|
NodeManager to crash. (vinodkv)
|
|
|
|
MAPREDUCE-3295. TestAMAuthorization failing on branch 0.23. (vinodkv via mahadev)
|
|
|
|
MAPREDUCE-3183. hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
|
|
missing license header. (Hitesh Shah via tucu).
|
|
|
|
MAPREDUCE-3003. Publish MR JARs to Maven snapshot repository. (tucu)
|
|
|
|
MAPREDUCE-3199. Fixed pom files to include correct log4j configuration for
|
|
tests. (vinodkv)
|
|
|
|
MAPREDUCE-3204. mvn site:site fails on MapReduce. (tucu)
|
|
|
|
MAPREDUCE-3248. Fixed log4j properties. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3256. Added authorization checks for the protocol between
|
|
NodeManager and ApplicationMaster. (vinodkv via acmurthy)
|
|
|
|
MAPREDUCE-3274. Fixed a race condition in MRAppMaster that was causing a
|
|
task-scheduling deadlock. (Robert Joseph Evans via vinodkv)
|
|
|
|
MAPREDUCE-3171 merge from trunk reverted changes from MAPREDUCE-2747 MAPREDUCE-3240.
|
|
|
|
MAPREDUCE-3313. Fixed initialization of ClusterMetrics which was failing
|
|
TestResourceTrackerService sometimes. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-2766. Fixed NM to set secure permissions for files and directories
|
|
in distributed-cache. (Hitesh Shah via vinodkv)
|
|
|
|
MAPREDUCE-2696. Fixed NodeManager to cleanup logs in a thread when logs'
|
|
aggregation is not enabled. (Siddharth Seth via vinodkv)
|
|
|
|
MAPREDUCE-3262. Fixed Container's state-machine in NodeManager to handle
|
|
a couple of events in failure states correctly. (Hitesh Shah and Siddharth
|
|
Seth via vinodkv)
|
|
|
|
MAPREDUCE-3035. Fixed MR JobHistory to ensure rack information is present.
|
|
(chakravarthy via acmurthy)
|
|
|
|
MAPREDUCE-3321. Disabled a few MR tests for 0.23. (Hitesh Shah via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3220. Fixed TestCombineOutputCollector. (Devaraj K via acmurthy)
|
|
|
|
MAPREDUCE-3103. Implement Job ACLs for MRAppMaster.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-3241. [Rumen] Fix Rumen to ignore the AMStartedEvent. (amarrk)
|
|
|
|
MAPREDUCE-3166. [Rumen] Make Rumen use job history api instead of relying
|
|
on current history file name format. (Ravi Gummadi via amarrk)
|
|
|
|
MAPREDUCE-3157. [Rumen] Fix TraceBuilder to handle 0.20 history file
|
|
names also. (Ravi Gummadi via amarrk)
|
|
|
|
MAPREDUCE-3081. Fix vaidya startup script. (gkesavan via suhas).
|
|
|
|
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. (Owen via jitendra)
|
|
|
|
MAPREDUCE-3192. Fix Javadoc warning in JobClient.java and Cluster.java.
|
|
(jitendra)
|
|
|
|
MAPREDUCE-3237. Move LocalJobRunner to hadoop-mapreduce-client-core.
|
|
(tomwhite via acmurthy)
|
|
|
|
MAPREDUCE-3316. Rebooted link is not working properly.
|
|
(Bhallamudi Venkata Siva Kamesh via mahadev)
|
|
|
|
MAPREDUCE-3317. Rumen TraceBuilder is emiting null as hostname.
|
|
(Ravi Gummadi via mahadev)
|
|
|
|
MAPREDUCE-3332. contrib/raid compile breaks due to changes in hdfs/protocol/datatransfer/
|
|
Sender#writeBlock related to checksum handling (Hitesh Shah via mahadev)
|
|
|
|
MAPREDUCE-3337. Added missing license headers. (acmurthy)
|
|
|
|
Release 0.22.1 - Unreleased
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
NEW FEATURES
|
|
|
|
IMPROVEMENTS
|
|
|
|
OPTIMIZATIONS
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-3837. Job tracker is not able to recover jobs after crash.
|
|
(Mayank Bansal via shv)
|
|
|
|
Release 0.22.0 - 2011-11-29
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-1866. Removes deprecated class
|
|
org.apache.hadoop.streaming.UTF8ByteArrayUtils. (amareshwari)
|
|
|
|
MAPREDUCE-1664. Changes the behaviour of the combination of job-acls
|
|
when they function together with queue-acls. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-2994. Fixed a bug in ApplicationID parsing that affects RM
|
|
UI. (Devaraj K via vinodkv)
|
|
|
|
MAPREDUCE-1788. o.a.h.mapreduce.Job shouldn't make a copy of the JobConf.
|
|
(Arun Murthy via mahadev)
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-1804. Stress-test tool for HDFS introduced in HDFS-708.
|
|
(Joshua Harlow via shv)
|
|
|
|
MAPREDUCE-220. Collect cpu and memory statistics per task. (Scott Chen via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-1970. Reed-Solomon code implementation for HDFS RAID.
|
|
(Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-2169. Integrated Reed-Solomon code with RaidNode. (Ramkumar
|
|
Vadali via schen)
|
|
|
|
MAPREDUCE-2936. Contrib Raid compilation broken after HDFS-1620. (vinodkv)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-2141. Add an "extra data" field to Task for use by Mesos. (matei)
|
|
|
|
MAPREDUCE-2140. Regenerate fair scheduler design doc PDF. (matei)
|
|
|
|
MAPREDUCE-1546. Redirect all job pages to corresponding history page
|
|
if job is not in JT memory. (Scott Chen via sharad)
|
|
|
|
MAPREDUCE-1092. Enable assertions for unit tests. (Eli Collins via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1680. Add a metric recording JobTracker heartbeats processed.
|
|
(Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1761. FairScheduler allows separate configuration of node
|
|
and rack locality wait time (Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1539. authorization checks for inter-server protocol
|
|
(based on HADOOP-6600) (Boris Shkolnik via shv)
|
|
|
|
MAPREDUCE-1798. Names the configuration keys for the Kerberos
|
|
principals better. (Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-1773. streaming doesn't support jobclient.output.filter.
|
|
(Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-1785. Add streaming config option for not emitting the key.
|
|
(Eli Collins via sharad)
|
|
|
|
MAPREDUCE-572. If #link is missing from uri format of -cacheArchive
|
|
then streaming does not throw error. (Amareshwari Sriramadasu via
|
|
vinodkv)
|
|
|
|
MAPREDUCE-1545. Add timestamps for first task type launched in job summary.
|
|
(Luke Lu via cdouglas)
|
|
|
|
MAPREDUCE-1543. Add an audit log for authentication events. (Amar Kamat and
|
|
Luke Lu via cdouglas)
|
|
|
|
MAPREDUCE-1762. Add ability to set values of task counters. (Scott Chen via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1533. Reduce overhead of logging and string manipulation during
|
|
heartbeat processing. (Amar Kamat and Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1516. JobTracker issues delegation tokens only if the user's
|
|
authentication is Kerberos. (Jitendra Pandey via ddas)
|
|
|
|
MAPREDUCE-647. Update distcp forrest documentation to reflect the changes
|
|
of HADOOP-5472, MAPREDUCE-642 and HADOOP-5620. (Rodrigo Schmidt via
|
|
szetszwo)
|
|
|
|
MAPREDUCE-1851. Documents configuration parameters in streaming.
|
|
(amareshwari)
|
|
|
|
MAPREDUCE-1868. Add a read and connection timeout to JobClient while
|
|
pulling tasklogs. (Krishna Ramachandran via acmurthy)
|
|
|
|
MAPREDUCE-1778. Ensure failure to setup CompletedJobStatusStore is not
|
|
silently ignored by the JobTracker. (Krishna Ramachandran via acmurthy)
|
|
|
|
MAPREDUCE-1850. Includes job submit host information (name and ip) in
|
|
jobconf and jobdetails display (Krishna Ramachandran via amareshwari)
|
|
|
|
MAPREDUCE-1893. Slive with multiple reducers. (shv)
|
|
|
|
MAPREDUCE-1248. Fixes redudant memory copying in StreamKeyValUtil.
|
|
(Ruibang He via amareshwari)
|
|
|
|
MAPREDUCE-1840. Enhancements to Gridmix benchmark simulating user
|
|
diversity, queue replay, and task duration for JobTracker load testing.
|
|
Also includes compatibility with security enhancements, and scalability
|
|
improvements. (Amar Kamat, Rahul Singh, Hong Tang, and cdouglas)
|
|
|
|
MAPREDUCE-1848. Put number of speculative, data local, rack local
|
|
tasks in JobTracker metrics. (Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1935. Makes the Distcp to work in a secure environment.
|
|
(Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-1945. The MapReduce component for HADOOP-6632.
|
|
(Kan Zhang & Jitendra Pandey via ddas)
|
|
|
|
MAPREDUCE-1936. Modify Gridmix3 to support more tunable parameters for
|
|
stress submission and sleep jobs. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-1733. Makes pipes applications secure. (Jitendra Pandey via ddas)
|
|
|
|
MAPREDUCE-1566. Adds a configuration attribute using which job clients can
|
|
specify a credentials file. The tokens from there will be passed to the job.
|
|
(Jitendra Pandey and Owen O'Malley via ddas)
|
|
|
|
MAPREDUCE-1624. Documents the job credentials and associated details to do
|
|
with delegation tokens (on the client side).
|
|
(Jitendra Pandey and Devaraj Das via ddas)
|
|
|
|
MAPREDUCE-1834. TestSimulatorDeterministicReplay timesout on trunk.
|
|
(Hong Tang via mahadev)
|
|
|
|
MAPREDUCE-1993. Fixes test failure
|
|
TestTrackerDistributedCacheManagerWithLinuxTaskController. (Devaraj Das
|
|
via amareshwari)
|
|
|
|
MAPREDUCE-1523. Making Mumak work with Capacity-Scheduler (Anirban Das
|
|
via mahadev)
|
|
|
|
MAPREDUCE-1920. Enables completed jobstatus store by default. (Tom White
|
|
via amareshwari)
|
|
|
|
MAPREDUCE-1881. Improve TaskTrackerInstrumentation to enable collection of
|
|
advanced metrics. (Matei Zaharia via acmurthy)
|
|
|
|
MAPREDUCE-1548. Hadoop archives preserve times and other properties from
|
|
original files. (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1517. Supports streaming job to run in the background. (Bochun Bai
|
|
via amareshwari)
|
|
|
|
MAPREDUCE-1819. RaidNode is now smarter in submitting Raid jobs. (Ramkumar
|
|
Vadali via schen)
|
|
|
|
MAPREDUCE-2132. A command line option in RaidShell to fix blocks using raid
|
|
|
|
MAPREDUCE-2147. Remove redundant lines in JobInProgress ctor.
|
|
(Harsh J Chouraria via cutting)
|
|
|
|
HADOOP-7007. Update the hudson-test-patch ant target to work with the
|
|
latest test-patch.sh script (gkesavan)
|
|
|
|
MAPREDUCE-1818. RaidNode can specify scheduling parameters. (Ramkumar
|
|
Vadali via schen)
|
|
|
|
MAPREDUCE-2051. Contribute a fair scheduler preemption system test.
|
|
(Todd Lipcon via tomwhite)
|
|
|
|
MAPREDUCE-1892. RaidNode can allow layered policies more efficiently.
|
|
(Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-1592. Generate Eclipse's .classpath file from Ivy config.
|
|
(tomwhite via nigel)
|
|
|
|
MAPREDUCE-2073. TestTrackerDistributedCacheManager should be up-front
|
|
about requirements on build environment. (Todd Lipcon via tomwhite)
|
|
|
|
MAPREDUCE-2093. Herriot JT and TT clients should vend statistics. (cos)
|
|
|
|
MAPREDUCE-2167. Faster directory traversal for raid node. (Ramkumar Vadali
|
|
via schen)
|
|
|
|
MAPREDUCE-1931. Gridmix forrest documentation . (Ranjit Mathew via vinodkv).
|
|
|
|
MAPREDUCE-2184. Port DistRaid.java to new mapreduce API. (Ramkumar Vadali
|
|
via schen)
|
|
|
|
MAPREDUCE-1878. Add MRUnit documentation. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-2180. Add coverage of fair scheduler servlet to system test (todd)
|
|
|
|
MAPREDUCE-2250. Fix logging in raid code. (Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-2260. Remove auto-generated native build files. (rvs via eli)
|
|
|
|
MAPREDUCE-2314. configure files that are generated as part of the released
|
|
tarball need to have executable bit set (rvs via cos)
|
|
|
|
MAPREDUCE-1159. Limit Job name in web UI to be 80 char long. (Harsh J
|
|
Chouraria via szetszwo)
|
|
|
|
MAPREDUCE-2337. Remove dependence of public MapReduce API on classes in
|
|
server package. (tomwhite)
|
|
|
|
MAPREDUCE-2383. Improve documentation of DistributedCache methods (Harsh J
|
|
Chouraria via todd)
|
|
|
|
MAPREDUCE-2222. Ivy resolve force mode should be turned off by default.
|
|
(Luke Lu via tomwhite)
|
|
|
|
MAPREDUCE-2103. task-controller shouldn't require o-r permissions.
|
|
(todd via eli)
|
|
|
|
MAPREDUCE-2505. Explain how to use ACLs in the fair scheduler.
|
|
(matei via eli)
|
|
|
|
MAPREDUCE-3138. Add a utility to help applications bridge changes in
|
|
Context Objects APIs due to MAPREDUCE-954. (omalley via acmurthy)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-1354. Enhancements to JobTracker for better performance and
|
|
scalability. (Arun C. Murthy & Richard King via acmurthy)
|
|
|
|
MAPREDUCE-1829. JobInProgress.findSpeculativeTask should use min() to
|
|
find the candidate instead of sort(). (Scott Chen via vinodkv)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-1845. FairScheduler.tasksToPreempt() can return negative number.
|
|
(Scott Chen via matei)
|
|
|
|
MAPREDUCE-1707. TaskRunner can get NPE in getting ugi from TaskTracker.
|
|
(Vinod Kumar Vavilapalli)
|
|
|
|
MAPREDUCE-1532. Ensures that delegation tokens is obtained as the
|
|
actual user when the proxy-user is used for submitting jobs. Also
|
|
refactors the DelegationTokenToRenew class. (ddas)
|
|
|
|
MAPREDUCE-1558. Fixes MRAdmin to look up the principal of the
|
|
JobTracker and use that in the RefreshUserToGroupsMapping protocol and
|
|
RefreshAuthorizationPolicyProtocol. (Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-1662. Remove unused methods from TaskRunner. (Amareshwari
|
|
Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1617. Use IPv4 stack for unit tests. (Amar Kamat and Luke Lu via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1599. Fixes MRBench so that it reuses tokens across jobs
|
|
correctly. (Jitendra Nath Pandey via ddas)
|
|
|
|
MAPREDUCE-1836. Refresh for proxy superuser config (mr part for HDFS-1096).
|
|
(Boris Shkolnik via shv)
|
|
|
|
MAPREDUCE-1505. Create RPC client on job submission, not in cstr of Job
|
|
instance. (Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1813. NPE in PipeMapred.MRErrorThread. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1225. Fixes DistributedCache to check if the file is fresh or not,
|
|
for the first localization also. (Zhong Wang via amareshwari)
|
|
|
|
MAPREDUCE-1559. Fixes the token renewer to use the JobTracker's
|
|
credentials for talking to the NameNode. (ddas)
|
|
|
|
MAPREDUCE-1492. Delete obsolete har files used on the parity files
|
|
of hdfs raid. (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1857. Removes unused configuration parameters in streaming.
|
|
(amareshwari)
|
|
|
|
MAPREDUCE-1887. MRAsyncDiskService now properly absolutizes volume root
|
|
paths. (Aaron Kimball via zshao)
|
|
|
|
MAPREDUCE-1863. Fix NPE in Rumen when processing null CDF for failed task
|
|
attempts. (Amar Kamat via cdouglas)
|
|
|
|
MAPREDUCE-1864. Removes uninitialized/unused variables in
|
|
org.apache.hadoop.streaming.PipeMapRed. (amareshwari)
|
|
|
|
MAPREDUCE-1888. Fixes Streaming to override output key and value types,
|
|
only if mapper/reducer is a command. (Ravi Gummadi via amareshwari)
|
|
|
|
MAPREDUCE-577. Fixes duplicate records in StreamXmlRecordReader.
|
|
(Ravi Gummadi via amareshwari)
|
|
|
|
MAPREDUCE-1894. Fixed a bug in DistributedRaidFileSystem.readFully()
|
|
that was causing it to loop infinitely. (Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-1838. Reduce the time needed for raiding a bunch of files
|
|
by randomly assigning files to map tasks. (Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-1820. Fix InputSampler to clone sampled keys. (Alex Kozlov via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1528. Incorporates the changes to the credentials API done in
|
|
HADOOP-6845. Also, introduces Credentials in JobConf, and in JobContext.
|
|
(Jitendra Pandey and Arun Murthy via ddas)
|
|
|
|
MAPREDUCE-1865. Rumen should also support jobhistory files generated using
|
|
trunk. (Amar Kamat via amareshwari)
|
|
|
|
MAPREDUCE-1621. Fixes NPE in TextOutputReader.getLastOutput if it has never
|
|
read any output. (amareshwari)
|
|
|
|
MAPREDUCE-1911. Fixes errors in -info message in streaming. (amareshwari)
|
|
|
|
MAPREDUCE-1772. Corrects errors in streaming documentation in forrest.
|
|
(amareshwari)
|
|
|
|
MAPREDUCE-1925. Fix failing TestRumenJobTraces. (Ravi Gummadi via cdouglas)
|
|
|
|
MAPREDUCE-1718. Fixes a bug in the construction of jobconf key for the
|
|
mapping that the tasks use at runtime for looking up delegation tokens.
|
|
(Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-1701. Fixes a problem to do with exception handling in
|
|
delegation-token renewals. (Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-1686. Fixes StreamUtil.goodClassOrNull to find classes without
|
|
package names. (Paul Burkhardt via amareshwari)
|
|
|
|
MAPREDUCE-1288. Fixes TrackerDistributedCacheManager to take into account
|
|
the owner of the localized file in the mapping from cache URIs to
|
|
CacheStatus objects. (ddas)
|
|
|
|
MAPREDUCE-1982. Fixes Rumen's TraceBuilder to extract job name from either
|
|
of configuration properties "mapreduce.job.name" and "mapred.job.name".
|
|
(Ravi Gummadi via amareshwari)
|
|
|
|
MAPREDUCE-1958. The MapReduce part corresponding to the HADOOP-6873.
|
|
(Boris Shkolnik & Owen O'Malley via ddas)
|
|
|
|
MAPREDUCE-1900. TaskTracker and JobTracker closes FileSystems, opened on
|
|
behalf of users that it no longer requires. (Kan Zhang and ddas via ddas)
|
|
|
|
MAPREDUCE-1992. Fixes a problem to do with bringing up the JobTracker in
|
|
unsecure mode. (Kan Zhang via ddas)
|
|
|
|
MAPREDUCE-1999. Fixes ClientProtocol to use the correct
|
|
DelegationTokenSelector. (Jitendra Pandey via ddas)
|
|
|
|
MAPREDUCE-1780. AccessControlList.toString() is used for serialization of
|
|
ACL in JobStatus.java. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1961. Fix ConcurrentModificationException in Gridmix during
|
|
shutdown. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-2000. Fix parsing of JobHistory lines in Rumen when quotes are
|
|
escaped. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-2022. Fixes compilation errors in TestSubmitJob. (amareshwari)
|
|
|
|
MAPREDUCE-1670. RAID policies should not scan their own destination path.
|
|
(Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-1668. RaidNode Hars a directory only if all its parity files
|
|
have been created. (Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-2021. Fixes duplicate hostnames in CombineFileInputFormat's
|
|
split locations. (amareshwari)
|
|
|
|
MAPREDUCE-1375. Fixes flaky test TestFileArgs. (Todd Lipcon via
|
|
amareshwari)
|
|
|
|
MAPREDUCE-2023. TestDFSIO should not stop reading if curSize != bufferSize.
|
|
(Hong Tang via szetszwo)
|
|
|
|
MAPREDUCE-2031. Fixes test failures TestTaskLauncher and
|
|
TestTaskTrackerLocalization. (Ravi Gummadi via amareshwari)
|
|
|
|
MAPREDUCE-2046. Fixes CombineFileInputFormat to allow splits with size
|
|
less than DFS block size. (dhruba borthakur via amareshwari)
|
|
|
|
MAPREDUCE-1975. Fixes unnecessary InterruptedException log in gridmix.
|
|
(Ravi Gummadi via amareshwari)
|
|
|
|
MAPREDUCE-1597. Fixes CombineFileInputFormat to work with non-splittable
|
|
files. (amareshwari)
|
|
|
|
MAPREDUCE-2032. Fixes TestJobCleanup to cleanup test directory in
|
|
tearDown. (Dick King via amareshwari)
|
|
|
|
MAPREDUCE-1979. Fixes "Output directory already exists" error in gridmix
|
|
when gridmix.output.directory is not defined. (Ravi Gummadi via
|
|
amareshwari)
|
|
|
|
MAPREDUCE-1918. Adds documentation to Rumen. (Amar Kamat via amareshwari)
|
|
|
|
MAPREDUCE-2078. Fixes TraceBuilder to generate traces when a globbed job
|
|
history path is given. (Amar Kamat via amareshwari)
|
|
|
|
MAPREDUCE-1989. Fixes error message in gridmix when user resolver is set
|
|
and no user list is given. (Ravi Gummadi via amareshwari)
|
|
|
|
MAPREDUCE-2067. Distinct minicluster services (e.g. NN and JT) overwrite
|
|
each other's service policies. (Aaron T. Myers via tomwhite)
|
|
|
|
MAPREDUCE-2029. DistributedRaidFileSystem removes itself from FileSystem
|
|
cache when it is closed. (Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-1816. HAR files used for RAID parity-bite have configurable
|
|
partfile size. (Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-2082. Fixes Pipes to create the jobtoken file in the right
|
|
place. (Jitendra Pandey via ddas)
|
|
|
|
MAPREDUCE-2095. Fixes Gridmix to run from compressed traces. (Ranjit
|
|
Mathew via amareshwari)
|
|
|
|
MAPREDUCE-1908. DistributedRaidFileSystem now handles ChecksumException
|
|
correctly. (Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-2126. JobQueueJobInProgressListener's javadoc is inconsistent
|
|
with source code. (Jingguo Yao via tomwhite)
|
|
|
|
MAPREDUCE-2143. HarFileSystem is able to handle spaces in pathnames.
|
|
(Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-1867. Remove unused methods in
|
|
org.apache.hadoop.streaming.StreamUtil. (amareshwari via tomwhite)
|
|
|
|
MAPREDUCE-2146. Raid does not affect access time of a source file.
|
|
(Ramkumar Vadali via dhruba)
|
|
|
|
MAPREDUCE-2150. RaidNode periodically fixes corrupt blocks. (Ramkumar Vadali via
|
|
schen)
|
|
|
|
MAPREDUCE-2099. RaidNode recreates outdated parity HARs. (Ramkumar Vadali
|
|
via schen)
|
|
|
|
MAPREDUCE-2173. Fix race condition in TestBlockFixer that was
|
|
causing intermittent failure (Patrick Kling via dhruba)
|
|
|
|
MAPREDUCE-2142. Refactor RaidNode so that the map-reduce component is
|
|
clearly separated out. (Patrick Kling via dhruba)
|
|
|
|
MAPREDUCE-2179. Fix RaidBlockSender compilation failure. (Ramkumar Vadali
|
|
via schen)
|
|
|
|
MAPREDUCE-2034. TestSubmitJob triggers NPE instead of permissions error.
|
|
(Todd Lipcon via tomwhite)
|
|
|
|
MAPREDUCE-2195. New property for local conf directory in
|
|
system-test-mapreduce.xml file. (cos)
|
|
|
|
MAPREDUCE-1783. FairScheduler initializes tasks only when the job can be
|
|
run. (Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-2224. Fix synchronization bugs in JvmManager. (todd)
|
|
|
|
MAPREDUCE-714. JobConf.findContainingJar unescapes unnecessarily on linux (todd)
|
|
|
|
MAPREDUCE-2096. Secure local filesystem IO from symlink vulnerabilities (todd)
|
|
|
|
MAPREDUCE-2234. If Localizer can't create task log directory, it should fail
|
|
on the spot. (todd)
|
|
|
|
MAPREDUCE-2219. JobTracker should not try to remove mapred.system.dir
|
|
during startup. (todd)
|
|
|
|
MAPREDUCE-2207. Task-cleanup task should not be scheduled on the node that
|
|
the task just failed. (Liyin Liang via schen)
|
|
|
|
MAPREDUCE-2084. Remove deprecate annotation for package file. The package
|
|
classes themselves are already deprecated. This removes an Eclipse error.
|
|
(tomwhite via nigel)
|
|
|
|
MAPREDUCE-2248. DistributedRaidFileSystem should unraid only the corrupt
|
|
block (Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-1085. For tasks, "ulimit -v -1" is being run when user doesn't
|
|
specify a ulimit (todd)
|
|
|
|
MAPREDUCE-2282. Fix TestMRServerPorts for the changes in
|
|
TestHDFSServerPorts. (shv via szetszwo)
|
|
|
|
MAPREDUCE-2238. Fix permissions handling to avoid leaving undeletable
|
|
directories in local dirs. (todd)
|
|
|
|
MAPREDUCE-2277. TestCapacitySchedulerWithJobTracker needs to wait for jobs
|
|
to complete before testing status. (todd)
|
|
|
|
MAPREDUCE-2253. Servlets should specify content type (todd)
|
|
|
|
MAPREDUCE-2283. Add timeout for Raid Tests (Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-1754. Replace mapred.persmissions.supergroup with an
|
|
acl : mapreduce.cluster.administrators (Amareshwari Sriramadasu via shv)
|
|
|
|
MAPREDUCE-2256. FairScheduler fairshare preemption from multiple pools may
|
|
preempt all tasks from one pool causing that pool to go below fairshare.
|
|
(Priyo Mustafi via shv)
|
|
|
|
MAPREDUCE-2281. MR part of HADOOP-6642. (Chris Douglas, Po Cheung via shv)
|
|
|
|
MAPREDUCE-2200. TestUmbilicalProtocolWithJobToken is failing without Krb
|
|
evironment: needs to be conditional. (cos)
|
|
|
|
MAPREDUCE-2077. Resolve name clash in the deprecated
|
|
o.a.h.util.MemoryCalculatorPlugin (Luke Lu via shv)
|
|
|
|
MAPREDUCE-2188. The new API MultithreadedMapper doesn't initialize
|
|
RecordReader. (Owen O'Malley via shv)
|
|
|
|
MAPREDUCE-1915. Fix IndexOutOfBoundsException in IndexCache.
|
|
(Priyo Mustafi via shv)
|
|
|
|
MAPREDUCE-1974. Fix multiple preemtions of the same task in FairScheduler.
|
|
(Scott Chen via shv)
|
|
|
|
MAPREDUCE-2304. Fix TestMRCLI to allow hostname with a hyphen (-).
|
|
(Priyo Mustafi via shv)
|
|
|
|
MAPREDUCE-1825. jobqueue_details.jsp and FairSchedulerServelet should not
|
|
call finishedMaps and finishedReduces when job is not initialized.
|
|
(Scott Chen via shv)
|
|
|
|
MAPREDUCE-2285. MiniMRCluster does not start after ant test-patch
|
|
(todd)
|
|
|
|
MAPREDUCE-2315. javadoc is failing in nightly build (todd)
|
|
|
|
MAPREDUCE-2054. Hierarchical queue implementation broke dynamic queue
|
|
addition in Dynamic Scheduler. (Thomas Sandholm via tomwhite)
|
|
|
|
MAPREDUCE-2272. Job ACL file should not be executable
|
|
(Harsh J Chouraria via todd)
|
|
|
|
MAPREDUCE-2241. ClusterWithLinuxTaskController should accept relative path
|
|
on the command line. (todd)
|
|
|
|
MAPREDUCE-2251. Remove unused mapreduce.job.userhistorylocation config.
|
|
(Harsh J Chouraria via todd)
|
|
|
|
MAPREDUCE-2284. TestLocalRunner.testMultiMaps times out (todd)
|
|
|
|
MAPREDUCE-2336. Tool-related packages should be in the Tool javadoc group.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-2394. JUnit output format doesn't propagate into raid contrib
|
|
build. (todd)
|
|
|
|
MAPREDUCE-2392. TaskTracker shutdown in the tests sometimes take 60s.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-2437. SLive uses only part* files to generating the final report.
|
|
(shv)
|
|
|
|
MAPREDUCE-2428. start-mapred.sh script fails if HADOOP_HOME is not set.
|
|
(tomwhite via eli)
|
|
|
|
MAPREDUCE-2445. Fix TestMiniMRWithDFSWithDistinctUsers to be a valid test.
|
|
(todd)
|
|
|
|
MAPREDUCE-2457. Job submission should inject group.name on the JobTracker
|
|
(Alejandro Abdelnur via todd)
|
|
|
|
MAPREDUCE-2472. Extra whitespace in mapred.child.java.opts breaks JVM
|
|
initialization. (Aaron T. Myers via todd)
|
|
|
|
MAPREDUCE-2222. Ivy resolve force mode should be turned off by default.
|
|
(Luke Lu via tomwhite)
|
|
|
|
MAPREDUCE-2486. Incorrect snapshot dependency published in .pom files
|
|
(todd)
|
|
|
|
MAPREDUCE-2327. MapTask doesn't need to put username information in
|
|
SpillRecord. (todd via tomwhite)
|
|
|
|
MAPREDUCE-2515. MapReduce code references some deprecated options
|
|
(Ari Rabkin via todd)
|
|
|
|
MAPREDUCE-2487. ChainReducer uses MAPPER_BY_VALUE instead of
|
|
REDUCER_BY_VALUE. (Devaraj K via todd)
|
|
|
|
MAPREDUCE-2185. Fix infinite loop at creating splits using
|
|
CombineFileInputFormat. (Ramkumar Vadali via schen)
|
|
|
|
MAPREDUCE-2571. CombineFileInputFormat.getSplits throws a
|
|
java.lang.ArrayStoreException. (Bochun Bai via todd)
|
|
|
|
MAPREDUCE-2767. Remove Linux task-controller. (Milind Bhandarkar via shv)
|
|
|
|
MAPREDUCE-2991. queueinfo.jsp fails to show queue status for Capacity
|
|
scheduler if queue names contain special symbols. (Priyo Mustafi via shv)
|
|
|
|
MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. (Robert Evans via
|
|
acmurthy)
|
|
|
|
MAPREDUCE-3139. SlivePartitioner generates negative partitions. (jghoman)
|
|
|
|
Release 0.21.1 - Unreleased
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-2040. Forrest Documentation for Dynamic Priority Scheduler.
|
|
(Thomas Sandholm via tomwhite)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-1897. trunk build broken on compile-mapred-test (cos)
|
|
|
|
MAPREDUCE-1280. Update Eclipse plugin to the new eclipse.jdt API.
|
|
(Alex Kozlov via szetszwo)
|
|
|
|
MAPREDUCE-1984. herriot TestCluster fails because exclusion is not there
|
|
(Balaji Rajagopalan via cos)
|
|
|
|
MAPREDUCE-2090. Clover build doesn't generate per-test coverage. (cos)
|
|
|
|
MAPREDUCE-2134. ant binary-system is broken in mapreduce project. (cos)
|
|
|
|
MAPREDUCE-1905. Fixes Context.setStatus() and progress() apis.
|
|
(amareshwari)
|
|
|
|
MAPREDUCE-1809. Ant build changes for Streaming system tests in contrib
|
|
projects. (Vinay Kumar Thota via amareshwari)
|
|
|
|
MAPREDUCE-2223. TestMRCLI might fail on Ubuntu with default /etc/hosts
|
|
(cos)
|
|
|
|
MAPREDUCE-2228. Remove java5 dependencies from build. (cos)
|
|
|
|
MAPREDUCE-1929. Allow artifacts to be published to the staging Apache Nexus
|
|
Maven Repository. (tomwhite)
|
|
|
|
MAPREDUCE-2317. Fix a NPE in HadoopArchives. (Devaraj K via szetszwo)
|
|
|
|
MAPREDUCE-2127. mapreduce trunk builds are filing on hudson.
|
|
(Bruno Mahé via eli)
|
|
|
|
MAPREDUCE-2779. JobSplitWriter.java can't handle large job.split file.
|
|
(Ming Ma via shv)
|
|
|
|
Release 0.21.0 - 2010-08-13
|
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
MAPREDUCE-516. Fix the starvation problem in the Capacity Scheduler
|
|
when running High RAM Jobs. (Arun Murthy via yhemanth)
|
|
|
|
MAPREDUCE-358. Change org.apache.hadoop.examples. AggregateWordCount
|
|
and org.apache.hadoop.examples.AggregateWordHistogram to use new
|
|
mapreduce api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-245. Change Job and jobcontrol classes to use the List interface
|
|
rather than ArrayList in APIs. (Tom White via cdouglas)
|
|
|
|
MAPREDUCE-766. Enhanced list-blacklisted-trackers to display reasons
|
|
for blacklisting a node. (Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-817. Add a cache for retired jobs with minimal job info and
|
|
provide a way to access history file url. (sharad)
|
|
|
|
MAPREDUCE-711. Moved Distributed Cache from Common to Map/Reduce
|
|
project. (Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-895. Per the contract elucidated in HADOOP-6201, throw
|
|
FileNotFoundException from FileSystem::listStatus rather than returning
|
|
null. (Jakob Homan via cdouglas)
|
|
|
|
MAPREDUCE-479. Provide full task id to map output servlet rather than the
|
|
reduce id, only. (Jiaqi Tan via cdouglas)
|
|
|
|
MAPREDUCE-873. Simplify job recovery. Incomplete jobs are resubmitted on
|
|
jobtracker restart. Removes a public constructor in JobInProgress. (sharad)
|
|
|
|
HADOOP-6230. Moved process tree and memory calculator related classes from
|
|
Common to Map/Reduce. (Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-157. Refactor job history APIs and change the history format to
|
|
JSON. (Jothi Padmanabhan via sharad)
|
|
|
|
MAPREDUCE-849. Rename configuration properties. (Amareshwari Sriramadasu
|
|
via sharad)
|
|
|
|
MAPREDUCE-1287. Only call the partitioner with more than one reducer.
|
|
(cdouglas)
|
|
|
|
MAPREDUCE-1385. Use the new UserGroupInformation from HADOOP-6299.
|
|
(ddas via omalley)
|
|
|
|
MAPREDUCE-1493. Authorization for job-history pages. (vinodkv)
|
|
|
|
MAPREDUCE-1607. Task controller may not set permissions for a
|
|
task cleanup attempt's log directory (Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-1683. Remove JNI calls from ClusterStatus cstr. (Arun Murthy and
|
|
Luke Lu via cdouglas)
|
|
|
|
MAPREDUCE-1855. Makes the refresh methods (for groups and proxy users)
|
|
independent of the client side configuration. (Boris Shkolnik via ddas)
|
|
|
|
NEW FEATURES
|
|
|
|
MAPREDUCE-1774. Large-scale Automated Framework (Sharad Agarwal, Sreekanth
|
|
Ramakrishnan, Konstantin Boudnik, at all via cos)
|
|
|
|
MAPREDUCE-706. Support for FIFO pools in the fair scheduler.
|
|
(Matei Zaharia)
|
|
|
|
MAPREDUCE-546. Provide sample fair scheduler config file in conf/ and use
|
|
it by default if no other config file is specified. (Matei Zaharia)
|
|
|
|
MAPREDUCE-551. Preemption support in the Fair Scheduler. (Matei Zaharia)
|
|
|
|
MAPREDUCE-567. Add a new example MR that always fails. (Philip Zeyliger
|
|
via tomwhite)
|
|
|
|
MAPREDUCE-211. Provides ability to run a health check script on the
|
|
tasktracker nodes and blacklist nodes if they are unhealthy.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-637. Add an example, distbbp, which able to compute the n th bit
|
|
of Pi for some large n. (szetszwo)
|
|
|
|
MAPREDUCE-532. Provide a way to limit the number of used slots
|
|
per queue in the capacity scheduler.
|
|
(Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-467. Provide ability to collect statistics about total tasks
|
|
and succeeded tasks in different time windows. (sharad)
|
|
|
|
MAPREDUCE-740. Log a job-summary at the end of a job, while allowing it
|
|
to be configured to use a custom appender if desired. (acmurthy)
|
|
|
|
MAPREDUCE-814. Provide a way to configure completed job history files
|
|
to be on HDFS. (sharad)
|
|
|
|
MAPREDUCE-800. MRUnit should support the new API. (Aaron Kimball via
|
|
tomwhite)
|
|
|
|
MAPREDUCE-798. MRUnit should be able to test a succession of MapReduce
|
|
passes. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-768. Provide an option to dump jobtracker configuration in JSON
|
|
format to standard output. (V.V.Chaitanya Krishna via yhemanth)
|
|
|
|
MAPREDUCE-824. Add support for a hierarchy of queues in the capacity
|
|
scheduler. (Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-751. Add Rumen, a tool for extracting statistics from job tracker
|
|
logs and generating job traces for simulation and analysis. (Dick King via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-830. Add support for splittable compression to TextInputFormats.
|
|
(Abdul Qadeer via cdouglas)
|
|
|
|
MAPREDUCE-861. Add support for hierarchical queues in the Map/Reduce
|
|
framework. (Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-776. Add Gridmix, a benchmark processing Rumen traces to simulate
|
|
a measured mix of jobs on a cluster. (cdouglas)
|
|
|
|
MAPREDUCE-862. Enhance JobTracker UI to display hierarchical queues.
|
|
(V.V.Chaitanya Krishna via yhemanth)
|
|
|
|
MAPREDUCE-777. Brand new apis to track and query jobs as a
|
|
replacement for JobClient. (Amareshwari Sriramadasu via acmurthy)
|
|
|
|
MAPREDUCE-775. Add native and streaming support for Vertica as an input
|
|
or output format taking advantage of parallel read and write properties of
|
|
the DBMS. (Omer Trajman via ddas)
|
|
|
|
MAPREDUCE-679. XML-based metrics as JSP servlet for JobTracker.
|
|
(Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-980. Modify JobHistory to use Avro for serialization. (cutting)
|
|
|
|
MAPREDUCE-728. Add Mumak, a Hadoop map/reduce simulator. (Arun C Murthy,
|
|
Tamas Sarlos, Anirban Dasgupta, Guanying Wang, and Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-1383. Automates fetching of delegation tokens in File*Formats
|
|
Distributed Cache and Distcp. Also, provides a config
|
|
mapreduce.job.hdfs-servers that the jobs can populate with a comma
|
|
separated list of namenodes. The job client automatically fetches
|
|
delegation tokens from those namenodes. (Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-698. Per-pool task limits for the fair scheduler.
|
|
(Kevin Peterson via matei)
|
|
|
|
MAPREDUCE-1026. Does mutual authentication of the shuffle
|
|
transfers using a shared JobTracker generated key.
|
|
(Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-744. Introduces the notion of a public distributed cache.
|
|
(Devaraj Das)
|
|
|
|
MAPREDUCE-1338. Introduces the notion of token cache using which
|
|
tokens and secrets can be sent by the Job client to the JobTracker.
|
|
(Boris Shkolnik via ddas)
|
|
|
|
HDFS-503. This patch implements an optional layer over HDFS that
|
|
implements offline erasure-coding. It can be used to reduce the
|
|
total storage requirements of HDFS. (dhruba)
|
|
|
|
MAPREDUCE-1432. Adds hooks in the jobtracker and tasktracker
|
|
for loading the tokens in the user's ugi. This is required
|
|
for the copying of files from the hdfs. (ddas)
|
|
|
|
MAPREDUCE-1335. Adds SASL Kerberos/Digest authentication in MapReduce.
|
|
(Kan Zhang via ddas)
|
|
|
|
MAPREDUCE-1464. Makes a compatible change in JobTokenIdentifier to
|
|
account for HADOOP-6510. (Jitendra Nath Pandey via ddas)
|
|
|
|
MAPREDUCE-1433. Add a delegation token for MapReduce. (omalley)
|
|
|
|
MAPREDUCE-1307. Introduces the Job level ACLs feature.
|
|
(Vinod Kumar Vavilapalli via ddas)
|
|
|
|
MAPREDUCE-1430. JobTracker automatically renews delegation tokens for jobs.
|
|
(Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-1455. Introduces job-level authorization for mapreduce servlets.
|
|
(Ravi Gummadi via vinodkv)
|
|
|
|
IMPROVEMENTS
|
|
|
|
MAPREDUCE-463. Makes job setup and cleanup tasks as optional.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-502. Allow jobtracker to be configured with zero completed jobs
|
|
in memory. (Amar Kamat via sharad)
|
|
|
|
MAPREDUCE-416. Moves the history file to a "done" folder whenever a job
|
|
completes. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-646. Increase srcfilelist replication number in dictcp job.
|
|
(Ravi Gummadi via szetszwo)
|
|
|
|
HADOOP-6106. Updated hadoop-core and test jars from hudson trunk
|
|
build #12. (Giridharan Kesavan)
|
|
|
|
MAPREDUCE-642. A option to distcp that allows preserving the full
|
|
source path of a file in the specified destination directory.
|
|
(Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-686. Move TestSpeculativeExecution.Fake* into a separate class
|
|
so that it can be used by other tests. (Jothi Padmanabhan via sharad)
|
|
|
|
MAPREDUCE-625. Modify TestTaskLimits to improve execution time.
|
|
(Jothi Padmanabhan via sharad)
|
|
|
|
MAPREDUCE-465. Deprecate o.a.h.mapred.lib.MultithreadedMapRunner and add
|
|
test for o.a.h.mapreduce.lib.MultithreadedMapper.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-701. Improves the runtime of the TestRackAwareTaskPlacement
|
|
by making it a unit test. (Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-371. Change KeyFieldBasedComparator and KeyFieldBasedPartitioner
|
|
to use new api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-623. Resolve javac warnings in mapreduce. (Jothi Padmanabhan
|
|
via sharad)
|
|
|
|
MAPREDUCE-655. Change KeyValueLineRecordReader and KeyValueTextInputFormat
|
|
to use new mapreduce api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-632. Merge TestCustomOutputCommitter with
|
|
TestCommandLineJobSubmission. (Jothi Padmanabhan via sharad)
|
|
|
|
MAPREDUCE-627. Improves execution time of TestTrackerBlacklistAcrossJobs.
|
|
(Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-630. Improves execution time of TestKillCompletedJob.
|
|
(Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-626. Improves the execution time of TestLostTracker.
|
|
(Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-353. Makes the shuffle read and connection timeouts
|
|
configurable. (Ravi Gummadi via ddas)
|
|
|
|
MAPREDUCE-739. Allow relative paths to be created in archives. (Mahadev
|
|
Konar via cdouglas)
|
|
|
|
MAPREDUCE-772. Merge HADOOP-4010 changes to LineRecordReader into mapreduce
|
|
package. (Abdul Qadeer via cdouglas)
|
|
|
|
MAPREDUCE-785. Separate sub-test of TestReduceFetch to be included in
|
|
MR-670. (Jothi Padmanabhan via cdouglas)
|
|
|
|
MAPREDUCE-784. Modify TestUserDefinedCounters to use LocalJobRunner
|
|
instead of MiniMR. (Jothi Padmanabhan via sharad)
|
|
|
|
HADOOP-6160. Fix releaseaudit target to run on specific directories.
|
|
(gkesavan)
|
|
|
|
MAPREDUCE-782. Use PureJavaCrc32 in SpillRecord. (Todd Lipcon via
|
|
szetszwo)
|
|
|
|
MAPREDUCE-369. Change org.apache.hadoop.mapred.lib.MultipleInputs to
|
|
use new api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-373. Change org.apache.hadoop.mapred.lib.FieldSelectionMapReduce
|
|
to use new api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-628. Improves the execution time of TestJobInProgress.
|
|
(Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-793. Creates a new test that consolidates a few tests to
|
|
include in the commit-test list. (Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-797. Adds combiner support to MRUnit MapReduceDriver.
|
|
(Aaron Kimball via johan)
|
|
|
|
MAPREDUCE-656. Change org.apache.hadoop.mapred.SequenceFile* classes
|
|
to use new mapreduce api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-670. Creates ant target for 10 mins patch test build.
|
|
(Jothi Padmanabhan via gkesavan)
|
|
|
|
MAPREDUCE-375. Change org.apache.hadoop.mapred.lib.NLineInputFormat
|
|
and org.apache.hadoop.mapred.MapFileOutputFormat to use new api.
|
|
(Amareshwari Sriramadasu via ddas)
|
|
|
|
MAPREDUCE-779. Added node health failure counts into
|
|
JobTrackerStatistics. (Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-842. Setup secure permissions for localized job files,
|
|
intermediate outputs and log files on tasktrackers.
|
|
(Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-478. Allow map and reduce jvm parameters, environment variables
|
|
and ulimit to be set separately.
|
|
Configuration changes:
|
|
add mapred.map.child.java.opts
|
|
add mapred.reduce.child.java.opts
|
|
add mapred.map.child.env
|
|
add mapred.reduce.child.ulimit
|
|
add mapred.map.child.env
|
|
add mapred.reduce.child.ulimit
|
|
deprecated mapred.child.java.opts
|
|
deprecated mapred.child.env
|
|
deprecated mapred.child.ulimit
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-767. Remove the dependence on the CLI 2.0 snapshot.
|
|
(Amar Kamat via omalley)
|
|
|
|
MAPREDUCE-712. Minor efficiency tweaks to RandomTextWriter. (cdouglas)
|
|
|
|
MAPREDUCE-870. Remove the job retire thread and the associated
|
|
config parameters. (sharad)
|
|
|
|
MAPREDUCE-874. Rename the PiEstimator example to QuasiMonteCarlo.
|
|
(szetszwo)
|
|
|
|
MAPREDUCE-336. Allow logging level of map/reduce tasks to be configurable.
|
|
Configuration changes:
|
|
add mapred.map.child.log.level
|
|
add mapred.reduce.child.log.level
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-355. Update mapred.join package to use the new API. (Amareshwari
|
|
Sriramadasu via cdouglas)
|
|
|
|
HADOOP-6184. Updated hadoop common and test jars to get the new API
|
|
in Configuration for dumping in JSON format from Hudson trunk build #68.
|
|
(yhemanth)
|
|
|
|
MAPREDUCE-476. Extend DistributedCache to work locally (LocalJobRunner).
|
|
(Philip Zeyliger via tomwhite)
|
|
|
|
MAPREDUCE-825. JobClient completion poll interval of 5s causes slow tests
|
|
in local mode. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-910. Support counters in MRUnit. (Aaron Kimball via cdouglas)
|
|
|
|
MAPREDUCE-788. Update gridmix2 to use the new API (Amareshwari Sriramadasu
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-875. Make DBRecordReader execute queries lazily. (Aaron Kimball
|
|
via enis)
|
|
|
|
MAPREDUCE-318. Modularizes the shuffle code. (Jothi Padmanabhan and
|
|
Arun Murthy via ddas)
|
|
|
|
MAPREDUCE-936. Allow a load difference for fairshare scheduler.
|
|
(Zheng Shao via dhruba)
|
|
|
|
MAPREDUCE-370. Update MultipleOutputs to use the API, merge funcitonality
|
|
of MultipleOutputFormat. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-898. Changes DistributedCache to use the new API.
|
|
(Amareshwari Sriramadasu via ddas)
|
|
|
|
MAPREDUCE-144. Includes dump of the process tree in task diagnostics when
|
|
a task is killed due to exceeding memory limits.
|
|
(Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-945. Modifies MRBench and TestMapRed to use ToolRunner so that
|
|
options such as queue name can be passed via command line.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-963. Deprecate o.a.h.mapred.FileAlreadyExistsException and
|
|
replace it with o.a.h.fs.FileAlreadyExistsException. (Boris Shkolnik
|
|
via szetszwo)
|
|
|
|
MAPREDUCE-960. Remove an unnecessary intermediate copy and obsolete API
|
|
from KeyValueLineRecordReader. (cdouglas)
|
|
|
|
MAPREDUCE-930. Modify Rumen to resolve paths in the canonical way, rather
|
|
than defaulting to the local filesystem. (cdouglas)
|
|
|
|
MAPREDUCE-944. Extend the LoadManager API of the fair-share scheduler
|
|
to support regulating tasks for a job based on resources currently in use
|
|
by that job. (dhruba)
|
|
|
|
MAPREDUCE-973. Move FailJob and SleepJob from examples to test. (cdouglas
|
|
via omalley)
|
|
|
|
MAPREDUCE-966. Modify Rumen to clean up interfaces and simplify integration
|
|
with other tools. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-856. Setup secure permissions for distributed cache files.
|
|
(Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-885. More efficient SQL queries for DBInputFormat. (Aaron Kimball
|
|
via enis)
|
|
|
|
MAPREDUCE-284. Enables ipc.client.tcpnodelay in Tasktracker's Child.
|
|
(Ravi Gummadi via sharad)
|
|
|
|
MAPREDUCE-916. Split the documentation to match the project split.
|
|
(Corinne Chandel via omalley)
|
|
|
|
MAPREDUCE-649. Validate a copy by comparing the source and destination
|
|
checksums in distcp. Also adds an intra-task retry mechanism for errors
|
|
detected during the copy. (Ravi Gummadi via cdouglas)
|
|
|
|
MAPREDUCE-654. Add a -dryrun option to distcp printing a summary of the
|
|
file data to be copied, without actually performing the copy. (Ravi Gummadi
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-664. Display the number of files deleted by distcp when the
|
|
-delete option is specified. (Ravi Gummadi via cdouglas)
|
|
|
|
MAPREDUCE-781. Let the name of distcp jobs be configurable. (Venkatesh S
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-975. Add an API in job client to get the history file url for
|
|
a given job id. (sharad)
|
|
|
|
MAPREDUCE-905. Add Eclipse launch tasks for MapReduce. (Philip Zeyliger
|
|
via tomwhite)
|
|
|
|
MAPREDUCE-277. Makes job history counters available on the job history
|
|
viewers. (Jothi Padmanabhan via ddas)
|
|
|
|
MAPREDUCE-893. Provides an ability to refresh queue configuration
|
|
without restarting the JobTracker.
|
|
(Vinod Kumar Vavilapalli and Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-1011. Add build.properties to svn and git ignore. (omalley)
|
|
|
|
MAPREDUCE-954. Change Map-Reduce context objects to be interfaces.
|
|
(acmurthy)
|
|
|
|
MAPREDUCE-639. Change Terasort example to reflect the 2009 updates.
|
|
(omalley)
|
|
|
|
MAPREDUCE-1063. Document gridmix benchmark. (cdouglas)
|
|
|
|
MAPREDUCE-931. Use built-in interpolation classes for making up task
|
|
runtimes in Rumen. (Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1012. Mark Context interfaces as public evolving. (Tom White via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-971. Document use of distcp when copying to s3, managing timeouts
|
|
in particular. (Aaron Kimball via cdouglas)
|
|
|
|
HDFS-663. DFSIO for append. (shv)
|
|
|
|
HDFS-641. Move all of the components that depend on map/reduce to
|
|
map/reduce. (omalley)
|
|
|
|
HADOOP-5107. Use Maven ant tasks to publish artifacts. (Giridharan Kesavan
|
|
via omalley)
|
|
|
|
MAPREDUCE-1229. Allow customization of job submission policy in Mumak.
|
|
(Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-1317. Reduce the memory footprint of Rumen objects by interning
|
|
host Strings. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-1097. Add support for Vertica 3.5 to its contrib module. (Omer
|
|
Trajman via cdouglas)
|
|
|
|
MAPREDUCE-1627. HadoopArchives should not uses a method in DistCp.
|
|
(szetszwo)
|
|
|
|
MAPREDUCE-1198. Alternatively schedule different types of tasks in
|
|
fair share scheduler. (Scott Chen via matei)
|
|
|
|
MAPREDUCE-707. Provide a jobconf property for explicitly assigning a job to
|
|
a pool in the Fair Scheduler. (Alan Heirich via matei)
|
|
|
|
MAPREDUCE-947. Added commitJob and abortJob apis to OutputCommitter.
|
|
Enhanced FileOutputCommitter to create a _SUCCESS file for successful
|
|
jobs. (Amar Kamat & Jothi Padmanabhan via acmurthy)
|
|
|
|
MAPREDUCE-1103. Added more metrics to Jobtracker. (sharad)
|
|
|
|
MAPREDUCE-1048. Add occupied/reserved slot usage summary on jobtracker UI.
|
|
(Amareshwari Sriramadasu and Hemanth Yamijala via sharad)
|
|
|
|
MAPREDUCE-1090. Modified log statement in TaskMemoryManagerThread to
|
|
include task attempt id. (yhemanth)
|
|
|
|
MAPREDUCE-1189. Reduce ivy console output to ovservable level (cos)
|
|
|
|
MAPREDUCE-1167. ProcfsBasedProcessTree collects rss memory information.
|
|
(Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1231. Added a new DistCp option, -skipcrccheck, so that the CRC
|
|
check during setup can be skipped. (Jothi Padmanabhan via szetszwo)
|
|
|
|
MAPREDUCE-1190. Add package documentation for BBP example.
|
|
(Tsz Wo (Nicholas) Sze via cdouglas)
|
|
|
|
MAPREDUCE-1119. When tasks fail to report status, show tasks's stack dump
|
|
before killing. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1185. Redirect running job url to history url if job is already
|
|
retired. (Amareshwari Sriramadasu and Sharad Agarwal via sharad)
|
|
|
|
MAPREDUCE-1050. Introduce a mock object testing framework. (tomwhite)
|
|
|
|
MAPREDUCE-1084. Implementing aspects development and fault injeciton
|
|
framework for MapReduce. (Sreekanth Ramakrishnan via cos)
|
|
|
|
MAPREDUCE-1209. Move common specific part of the test TestReflectionUtils
|
|
out of mapred into common. (Todd Lipcon via tomwhite)
|
|
|
|
MAPREDUCE-967. TaskTracker does not need to fully unjar job jars.
|
|
(Todd Lipcon via tomwhite)
|
|
|
|
MAPREDUCE-1083. Changes in MapReduce so that group information of users
|
|
can be refreshed in the JobTracker via command line.
|
|
(Boris Shkolnik via ddas)
|
|
|
|
MAPREDUCE-181. Changes the job submission process to be secure.
|
|
(Devaraj Das)
|
|
|
|
MAPREDUCE-1250. Refactors the JobToken to use Common's Token interface.
|
|
(Kan Zhang via ddas)
|
|
|
|
MAPREDUCE-896. Enhance tasktracker to cleanup files that might have
|
|
been created by user tasks with non-writable permissions.
|
|
(Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-372. Change org.apache.hadoop.mapred.lib.ChainMapper/Reducer
|
|
to use new mapreduce api. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-1295. Add a tool in Rumen for folding and manipulating job
|
|
traces. (Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1302. TrackerDistributedCacheManager deletes file
|
|
asynchronously, thus reducing task initialization delays.
|
|
(Zheng Shao via dhruba)
|
|
|
|
MAPREDUCE-1218. TaskTrackers send cpu and memory usage of
|
|
node to JobTracker. (Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-847. Fix Releaseaudit warning count to zero
|
|
(Giridharan Kesavan)
|
|
|
|
MAPREDUCE-1337. Use generics in StreamJob to improve readability of that
|
|
class. (Kay Kay via cdouglas)
|
|
|
|
MAPREDUCE-361. Port terasort example to the new mapreduce API. (Amareshwari
|
|
Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1367. LocalJobRunner should support parallel mapper execution.
|
|
(Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-64. Eliminate io.sort.record.percent from MapTask configuration.
|
|
(cdouglas)
|
|
|
|
MAPREDUCE-1440. Replace the long user name in MapReduce with the local
|
|
name. (omalley)
|
|
|
|
MAPREDUCE-1470. Move delegation tokens from HDFS to Common so that
|
|
MapReduce can use them too. (omalley)
|
|
|
|
MAPREDUCE-1425. Reduce memory usage by archive. (mahadev via szetszwo)
|
|
|
|
MAPREDUCE-1441. Trim whitespace from directory lists pulled from the
|
|
configuration. (Todd Lipcon via cdouglas)
|
|
|
|
MAPREDUCE-1309. Refactor Rumen trace generator to improve code structure
|
|
and add extensible support for log formats. (Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1503. Delegation token renewing and cancelling should provide
|
|
meaningful exceptions when there are failures instead of returning
|
|
false. (omalley)
|
|
|
|
HADOOP-6579. Upgrade commons-codec library to 1.4. (omalley)
|
|
|
|
MAPREDUCE-1423. Improve performance of CombineFileInputFormat when multiple
|
|
pools are configured. (Dhruba Borthakur via zshao)
|
|
|
|
MAPREDUCE-1454. Quote user supplied strings in Tracker servlets. (cdouglas)
|
|
|
|
MAPREDUCE-1408. Add customizable job submission policies to Gridmix. (Rahul
|
|
Singh via cdouglas)
|
|
|
|
MAPREDUCE-1527. Better warning logged when mapred.queue.names is
|
|
overshadowed by mapred-queues.xml. (Hong Tang via acmurthy)
|
|
|
|
MAPREDUCE-1403. Save the size and number of distributed cache artifacts in
|
|
the configuration. (Arun Murthy via cdouglas)
|
|
|
|
MAPREDUCE-1482. Truncate state string and diagnostic information in
|
|
TaskStatus. (Amar Kamat via szetszwo)
|
|
|
|
MAPREDUCE-1593. [Rumen] Improvements to random seed generation (tamas via
|
|
mahadev)
|
|
|
|
MAPREDUCE-1460. Oracle support in DataDrivenDBInputFormat.
|
|
(Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1569. Pass configuration through mocked contexts in MRUnit.
|
|
(Chris White via cdouglas)
|
|
|
|
MAPREDUCE-1590. Move HarFileSystem from Hadoop Common to Mapreduce tools.
|
|
(mahadev)
|
|
|
|
MAPREDUCE-1629. Get rid of fakeBlockLocations() on HarFileSystem, since
|
|
it's not used (mahadev)
|
|
|
|
MAPREDUCE-1489. DataDrivenDBInputFormat should not query the database
|
|
when generating only one split. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1514. Add documentation on replication, permissions, new options,
|
|
limitations and internals of har. (mahadev via szetszwo)
|
|
|
|
MAPREDUCE-1428. Make block size and the size of archive created files
|
|
configurable. (mahadev via szetszwo)
|
|
|
|
MAPREDUCE-1656. JobStory should provide queue info. (hong via mahadev)
|
|
|
|
MAPREDUCE-1466. Record number of files processed in FileInputFormat in the
|
|
Configuration for offline analysis. (Luke Lu and Arun Murthy via cdouglas)
|
|
|
|
MAPREDUCE-1538. TrackerDistributedCacheManager manages the
|
|
number of files. (Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1673. Scripts to start and stop RaidNode.
|
|
(Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1659. RaidNode writes temp files on configured tmp directory and
|
|
add random numbers to their names to avoid conflicts
|
|
(Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1221. Allow admins to control physical memory limits per-task
|
|
and per-node. (Scott Chen via acmurthy)
|
|
|
|
MAPREDUCE-1065. Update mapred tutorial to use the new API. (Aaron Kimball
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-1304. Add a task counter tracking time spent in GC. (Aaron
|
|
Kimball via cdouglas)
|
|
|
|
MAPREDUCE-1570. Add grouping comparators to MRUnit. (Chris White via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1650. Exclude Private elements from generated MapReduce
|
|
Javadoc. (tomwhite)
|
|
|
|
MAPREDUCE-1625. Improve grouping of packages in Javadoc. (tomwhite)
|
|
|
|
MAPREDUCE-1417. Forrest documentation should be updated to reflect
|
|
the changes in MAPREDUCE-744. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1568. TrackerDistributedCacheManager should clean up cache
|
|
in a background thread. (Scott Chen via zshao)
|
|
|
|
MAPREDUCE-1749. Move configuration strings out of JobContext so that it
|
|
can be made public stable. (omalley)
|
|
|
|
MAPREDUCE-1623. Apply audience and stability notations to Hadoop
|
|
Map-Reduce. (tomwhite via acmurthy)
|
|
|
|
MAPREDUCE-1751. Change MapReduce to depend on Hadoop 'common' artifacts
|
|
instead of 'core'. (tomwhite)
|
|
|
|
MAPREDUCE-1535. Replace usage of FileStatus#isDir(). (Eli Collins via
|
|
tomwhite)
|
|
|
|
MAPREDUCE-1832. Allow file sizes less than 1MB in DFSIO benchmark. (shv)
|
|
|
|
MAPREDUCE-1404. Move Cluster-Setup and Single-Node-Setup Docs from
|
|
MapReduce to Common. (tomwhite)
|
|
|
|
MAPREDUCE-1697. Document the behavior of -file option and deprecate it
|
|
in favour of -files option in streaming. (Amareshwari Sriramadasu
|
|
via vinodkv)
|
|
|
|
MAPREDUCE-1033. Resolve location of scripts and configuration files after
|
|
project split. (tomwhite)
|
|
|
|
MAPREDUCE-1018. Document changes to the memory management and scheduling
|
|
model. (Hemanth Yamijala via vinodkv)
|
|
|
|
MAPREDUCE-1896. [Herriot] New property for multi user list. (Vinay Thota
|
|
via cos)
|
|
|
|
MAPREDUCE-1812. New properties for suspend and resume process. (Vinay
|
|
Thota via cos)
|
|
|
|
OPTIMIZATIONS
|
|
|
|
MAPREDUCE-270. Fix the tasktracker to optionally send an out-of-band
|
|
heartbeat on task-completion for better job-latency. (acmurthy)
|
|
Configuration changes:
|
|
add mapreduce.tasktracker.outofband.heartbeat
|
|
|
|
MAPREDUCE-1186. Modified code in distributed cache to set permissions
|
|
only on required set of localized paths.
|
|
(Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1501. FileInputFormat supports multi-level, recursive
|
|
directory listing. (Zheng Shao via dhruba)
|
|
|
|
MAPREDUCE-1556. upgrade to Avro 1.3.0. (cutting via tomwhite)
|
|
|
|
MAPREDUCE-1613. Install/deploy source jars to Maven repo
|
|
(Patrick Angeles via ddas)
|
|
|
|
MAPREDUCE-1610. Forrest documentation should be updated to reflect
|
|
the changes in MAPREDUCE-856. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1853. Adds caching for TaskAttemptContext in MultipleOutputs.
|
|
(Torsten Curdt via amareshwari)
|
|
|
|
BUG FIXES
|
|
|
|
MAPREDUCE-878. Rename fair scheduler design doc to
|
|
fair-scheduler-design-doc.tex and add Apache license header (matei)
|
|
|
|
HADOOP-4687. MapReduce is split from Hadoop Core. It is a subproject under
|
|
Hadoop (Owen O'Malley)
|
|
|
|
HADOOP-6096. Fix Eclipse project and classpath files following project
|
|
split. (tomwhite)
|
|
|
|
MAPREDUCE-419. Reconcile mapred.userlog.limit.kb defaults in configuration
|
|
and code. (Philip Zeyliger via cdouglas)
|
|
|
|
MAPREDUCE-2. Fixes a bug in KeyFieldBasedPartitioner in handling empty
|
|
keys. (Amar Kamat via sharad)
|
|
|
|
MAPREDUCE-130. Delete the jobconf copy from the log directory of the
|
|
JobTracker when the job is retired. (Amar Kamat via sharad)
|
|
|
|
MAPREDUCE-657. Fix hardcoded filesystem problem in CompletedJobStatusStore.
|
|
(Amar Kamat via sharad)
|
|
|
|
MAPREDUCE-179. Update progress in new RecordReaders. (cdouglas)
|
|
|
|
MAPREDUCE-658. Replace NPE in distcp with a meaningful error message when
|
|
the source path does not exist. (Ravi Gummadi via cdouglas)
|
|
|
|
MAPREDUCE-671. Update ignore list to include untracked, generated
|
|
build artifacts and config files. (cdouglas)
|
|
|
|
MAPREDUCE-433. Use more reliable counters in TestReduceFetch. (cdouglas)
|
|
|
|
MAPREDUCE-124. Fix a bug in failure handling of abort task of
|
|
OutputCommiter. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-694. Fix to add jsp-api jars to capacity-scheduler classpath.
|
|
(Giridharan Kesavan)
|
|
|
|
MAPREDUCE-702. Fix eclipse-plugin jar target (Giridharan Kesavan)
|
|
|
|
MAPREDUCE-522. Replace TestQueueCapacities with simpler test case to
|
|
test integration between capacity scheduler and MR framework.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-683. Fixes an initialization problem in the JobHistory.
|
|
The initialization of JobHistoryFilesManager is now done in the
|
|
JobHistory.init call. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-708. Fixes a bug to allow updating the reason for
|
|
blacklisting a node on the JobTracker UI.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-709. Fixes message displayed for a blacklisted node where
|
|
the reason for blacklisting is due to the health check script
|
|
timing out. (Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-676. Existing diagnostic rules fail for MAP ONLY jobs.
|
|
(Suhas Gogate via tomwhite)
|
|
|
|
MAPREDUCE-722. Fixes a bug with tasktracker reservations for
|
|
high memory jobs in capacity scheduler.
|
|
(Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
HADOOP-6090. Updates gridmix script to use new mapreduce api output
|
|
format. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-732. Removed spurious log statements in the node
|
|
blacklisting logic. (Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-734. Fix a ConcurrentModificationException in unreserving
|
|
unused reservations for a job when it completes.
|
|
(Arun Murthy and Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-733. Fix a RuntimeException while unreserving trackers
|
|
that are blacklisted for a job.
|
|
(Arun Murthy and Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-677. Fix timeout in TestNodeRefresh. (Amar Kamat via
|
|
sharad)
|
|
|
|
MAPREDUCE-153. Fix timeout in TestJobInProgressListener. (Amar
|
|
Kamat via sharad)
|
|
|
|
MAPREDUCE-742. Fix output messages and java comments in the Pi related
|
|
examples. (szetszwo)
|
|
|
|
MAPREDUCE-565. Fix partitioner to work with new API. (Owen O'Malley via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-680. Fix so MRUnit can handle reuse of Writable objects.
|
|
(Aaron Kimball via johan)
|
|
|
|
MAPREDUCE-18. Puts some checks for cross checking whether a reduce
|
|
task gets the correct shuffle data. (Ravi Gummadi via ddas)
|
|
|
|
MAPREDUCE-771. Fix scheduling of setup and cleanup tasks to use
|
|
free slots instead of tasks for scheduling. (yhemanth)
|
|
|
|
MAPREDUCE-717. Fixes some corner case issues in speculative
|
|
execution heuristics. (Devaraj Das)
|
|
|
|
MAPREDUCE-716. Make DBInputFormat work with Oracle. (Aaron Kimball
|
|
via tomwhite)
|
|
|
|
MAPREDUCE-735. Fixes a problem in the KeyFieldHelper to do with
|
|
the end index for some inputs (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-682. Removes reservations on tasktrackers which are
|
|
blacklisted. (Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-743. Fixes a problem to do with progress reporting
|
|
in the map phase. (Ravi Gummadi via ddas)
|
|
|
|
MAPREDUCE-765. Eliminate the deprecated warnings introduced by H-5438.
|
|
(He Yongqiang via szetszwo)
|
|
|
|
MAPREDUCE-383. Fix a bug in Pipes combiner due to bytes count not
|
|
getting reset after the spill. (Christian Kunz via sharad)
|
|
|
|
MAPREDUCE-809. Fix job-summary logs to correctly record status of FAILED
|
|
and KILLED jobs. (acmurthy)
|
|
|
|
MAPREDUCE-792. Fix unchecked warnings in DBInputFormat. (Aaron Kimball
|
|
via szetszwo)
|
|
|
|
MAPREDUCE-760. Fix a timing issue in TestNodeRefresh. (Amar Kamat via
|
|
sharad)
|
|
|
|
MAPREDUCE-40. Keep memory management backwards compatible for job
|
|
configuration parameters and limits. (Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-587. Fixes a OOM issue in TestStreamingExitStatus.
|
|
(Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-408. Fixes an assertion problem in TestKillSubProcesses
|
|
(Ravi Gummadi via ddas)
|
|
|
|
MAPREDUCE-659. Fix gridmix2 compilation. (Giridharan Kesavan)
|
|
|
|
MAPREDUCE-796. Fixes a ClassCastException in an exception log in
|
|
MultiThreadedMapRunner. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-808. Fixes a serialization problem in TypedBytes.
|
|
(Klaas Bosteels via ddas)
|
|
|
|
MAPREDUCE-845. Fix a findbugs heap size problem in build.xml and add
|
|
a new property findbugs.heap.size. (Lee Tucker via szetszwo)
|
|
|
|
MAPREDUCE-838. Fixes a problem in the way commit of task outputs
|
|
happens. The bug was that even if commit failed, the task would
|
|
be declared as successful. (Amareshwari Sriramadasu via ddas)
|
|
|
|
MAPREDUCE-813. Updates Streaming and M/R tutorial documents.
|
|
(Corinne Chandel via ddas)
|
|
|
|
MAPREDUCE-805. Fixes some deadlocks in the JobTracker due to the fact
|
|
the JobTracker lock hierarchy wasn't maintained in some JobInProgress
|
|
method calls. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-799. Fixes so all of the MRUnit self-tests run.
|
|
(Aaron Kimball via johan)
|
|
|
|
MAPREDUCE-848. Fixes a problem to do with TestCapacityScheduler
|
|
failing (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-840. DBInputFormat leaves open transaction.
|
|
(Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-859. Adds Avro and its dependencies required by Hadoop
|
|
common. (Ravi Gummadi via sharad)
|
|
|
|
MAPREDUCE-867. Fix ivy conf to look for avro jar from maven repo.
|
|
(Giridharan Kesavan)
|
|
|
|
MAPREDUCE-877. Added avro as a dependency to contrib ivy settings.
|
|
(Tsz Wo (Nicholas) Sze via yhemanth)
|
|
|
|
MAPREDUCE-852. In build.xml, remove the Main-Class, which is incorrectly
|
|
set in tools, and rename the target "tools-jar" to "tools". (szetszwo)
|
|
|
|
MAPREDUCE-773. Sends progress reports for compressed gzip inputs in maps.
|
|
Fixes a native direct buffer leak in LineRecordReader classes.
|
|
(Hong Tang and ddas)
|
|
|
|
MAPREDUCE-832. Reduce number of warning messages printed when
|
|
deprecated memory variables are used. (Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-745. Fixes a testcase problem to do with generation of JobTracker
|
|
IDs. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-834. Enables memory management on tasktrackers when old
|
|
memory management parameters are used in configuration.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-818. Fixes Counters#getGroup API. (Amareshwari Sriramadasu
|
|
via sharad)
|
|
|
|
MAPREDUCE-807. Handles the AccessControlException during the deletion of
|
|
mapred.system.dir in the JobTracker. The JobTracker will bail out if it
|
|
encounters such an exception. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-430. Fix a bug related to task getting stuck in case of
|
|
OOM error. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-871. Fix ownership of Job/Task local files to have correct
|
|
group ownership according to the egid of the tasktracker.
|
|
(Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-911. Fix a bug in TestTaskFail related to speculative
|
|
execution. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-687. Fix an assertion in TestMiniMRMapRedDebugScript.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-924. Fixes the TestPipes testcase to use Tool.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-903. Add Avro jar to eclipse classpath.
|
|
(Philip Zeyliger via tomwhite)
|
|
|
|
MAPREDUCE-943. Removes a testcase in TestNodeRefresh that doesn't make
|
|
sense in the new Job recovery model. (Amar Kamat via ddas)
|
|
|
|
MAPREDUCE-764. TypedBytesInput's readRaw() does not preserve custom type
|
|
codes. (Klaas Bosteels via tomwhite)
|
|
|
|
HADOOP-6243. Fixes a NullPointerException in handling deprecated keys.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-968. NPE in distcp encountered when placing _logs directory on
|
|
S3FileSystem. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-826. harchive doesn't use ToolRunner / harchive returns 0 even
|
|
if the job fails with exception (koji Noguchi via mahadev)
|
|
|
|
MAPREDUCE-839. unit test TestMiniMRChildTask fails on mac os-x (hong tang
|
|
via mahadev)
|
|
|
|
MAPREDUCE-112. Add counters for reduce input, output records to the new API.
|
|
(Jothi Padmanabhan via cdouglas)
|
|
|
|
MAPREDUCE-648. Fix two distcp bugs: (1) it should not launch a job if all
|
|
src paths are directories, and (2) it does not skip copying when updating
|
|
a single file. (Ravi Gummadi via szetszwo)
|
|
|
|
MAPREDUCE-946. Fix a regression in LineRecordReader where the
|
|
maxBytesToConsume parameter is not set correctly. (cdouglas)
|
|
|
|
MAPREDUCE-977. Missing jackson jars from Eclipse template. (tomwhite)
|
|
|
|
MAPREDUCE-988. Fix a packaging issue in the contrib modules. (Hong Tang via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-971. distcp does not always remove distcp.tmp.dir. (Aaron Kimball
|
|
via tomwhite)
|
|
|
|
MAPREDUCE-995. Fix a bug in JobHistory where tasks completing after the job
|
|
is closed cause a NPE. (Jothi Padmanabhan via cdouglas)
|
|
|
|
MAPREDUCE-953. Fix QueueManager to dump queue configuration in JSON format.
|
|
(V.V. Chaitanya Krishna via yhemanth)
|
|
|
|
MAPREDUCE-645. Prevent distcp from running a job when the destination is a
|
|
file, but the source is not. (Ravi Gummadi via cdouglas)
|
|
|
|
MAPREDUCE-1002. Flushed writer in JobQueueClient so queue information is
|
|
printed correctly. (V.V. Chaitanya Krishna via yhemanth)
|
|
|
|
MAPREDUCE-1003. Fix compilation problem in eclipse plugin when
|
|
eclipse.home is set. (Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-941. Vaidya script fails on Solaris. (Chad Metcalf
|
|
via tomwhite)
|
|
|
|
MAPREDUCE-912. Add and standardize Apache license headers. (Chad Metcalf
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-1022. Fix compilation of vertica testcases. (Vinod Kumar
|
|
Vavilapalli via acmurthy)
|
|
|
|
MAPREDUCE-1000. Handle corrupt history files in JobHistory.initDone().
|
|
(Jothi Padmanabhan via sharad)
|
|
|
|
MAPREDUCE-1028. Fixed number of slots occupied by cleanup tasks to one
|
|
irrespective of slot size for the job.
|
|
(Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-964. Fixed start and finish times of TaskStatus to be
|
|
consistent, thereby fixing inconsistencies in metering tasks.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-1076. Deprecate ClusterStatus and add javadoc in ClusterMetrics.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-979. Fixed JobConf APIs related to memory parameters to return
|
|
values of new configuration variables when deprecated variables are
|
|
disabled. (Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-1030. Modified scheduling algorithm to return a map and reduce
|
|
task per heartbeat in the capacity scheduler.
|
|
(Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-1071. Use DataInputStream rather than FSDataInputStream in the
|
|
JobHistory EventReader. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-986. Fix Rumen to work with truncated task lines. (Dick King via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1029. Fix failing TestCopyFiles by restoring the unzipping of
|
|
HDFS webapps from the hdfs jar. (Aaron Kimball and Jothi Padmanabhan via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-769. Make findbugs and javac warnings to zero.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-1104. Initialize RecoveryManager in JobTracker cstr called by
|
|
Mumak. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-1061. Add unit test validating byte specifications for gridmix
|
|
jobs. (cdouglas)
|
|
|
|
MAPREDUCE-1077. Fix Rumen so that truncated tasks do not mark the job as
|
|
successful. (Dick King via cdouglas)
|
|
|
|
MAPREDUCE-1041. Make TaskInProgress::taskStatuses map package-private.
|
|
(Jothi Padmanabhan via cdouglas)
|
|
|
|
MAPREDUCE-1070. Prevent a deadlock in the fair scheduler servlet.
|
|
(Todd Lipcon via cdouglas)
|
|
|
|
MAPREDUCE-1086. Setup Hadoop logging environment for tasks to point to
|
|
task related parameters. (Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-1105. Remove max limit configuration in capacity scheduler in
|
|
favor of max capacity percentage thus allowing the limit to go over
|
|
queue capacity. (Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-1016. Make the job history log format JSON. (cutting)
|
|
|
|
MAPREDUCE-1038. Weave Mumak aspects only if related files have changed.
|
|
(Aaron Kimball via cdouglas)
|
|
|
|
MAPREDUCE-1163. Remove unused, hard-coded paths from libhdfs. (Allen
|
|
Wittenauer via cdouglas)
|
|
|
|
MAPREDUCE-962. Fix a NullPointerException while killing task process
|
|
trees. (Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-1177. Correct setup/cleanup inversion in
|
|
JobTracker::getTaskReports. (Vinod Kumar Vavilapalli via cdouglas)
|
|
|
|
MAPREDUCE-1178. Fix ClassCastException in MultipleInputs by adding
|
|
a DelegatingRecordReader. (Amareshwari Sriramadasu and Jay Booth
|
|
via sharad)
|
|
|
|
MAPREDUCE-1068. Fix streaming job to show proper message if file is
|
|
is not present. (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-1147. Add map output counters to new API. (Amar Kamat via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-915. The debug scripts are run as the job user. (ddas)
|
|
|
|
MAPREDUCE-1007. Fix NPE in CapacityTaskScheduler.getJobs().
|
|
(V.V.Chaitanya Krishna via sharad)
|
|
|
|
MAPREDUCE-28. Refactor TestQueueManager and fix default ACLs.
|
|
(V.V.Chaitanya Krishna and Rahul K Singh via sharad)
|
|
|
|
MAPREDUCE-1182. Fix overflow in reduce causing allocations to exceed the
|
|
configured threshold. (cdouglas)
|
|
|
|
MAPREDUCE-1239. Fix contrib components build dependencies.
|
|
(Giridharan Kesavan and omalley)
|
|
|
|
MAPREDUCE-787. Fix JobSubmitter to honor user given symlink path.
|
|
(Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-1249. Update config default value for socket read timeout to
|
|
match code default. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1161. Remove ineffective synchronization in NotificationTestCase.
|
|
(Owen O'Malley via cdouglas)
|
|
|
|
MAPREDUCE-1244. Fix eclipse-plugin's build dependencies. (gkesavan)
|
|
|
|
MAPREDUCE-1075. Fix JobTracker to not throw an NPE for a non-existent
|
|
queue. (V.V.Chaitanya Krishna via yhemanth)
|
|
|
|
MAPREDUCE-754. Fix NPE in expiry thread when a TT is lost. (Amar Kamat
|
|
via sharad)
|
|
|
|
MAPREDUCE-1074. Document Reducer mark/reset functionality. (Jothi
|
|
Padmanabhan via cdouglas)
|
|
|
|
MAPREDUCE-1267. Fix typo in mapred-default.xml. (Todd Lipcon via cdouglas)
|
|
|
|
MAPREDUCE-952. Remove inadvertently reintroduced Task.Counter enum. (Jothi
|
|
Padmanabhan via cdouglas)
|
|
|
|
MAPREDUCE-1230. Fix handling of null records in VerticaInputFormat. (Omer
|
|
Trajman via cdouglas)
|
|
|
|
MAPREDUCE-1171. Allow shuffle retries and read-error reporting to be
|
|
configurable. (Amareshwari Sriramadasu via acmurthy)
|
|
|
|
MAPREDUCE-879. Fix broken unit test TestTaskTrackerLocalization on MacOS.
|
|
(Sreekanth Ramakrishnan via yhemanth)
|
|
|
|
MAPREDUCE-1124. Fix imprecise byte counts in Gridmix. (cdouglas)
|
|
|
|
MAPREDUCE-1222. Add an option to exclude numeric IP addresses in topologies
|
|
processed by Mumak. (Hong Tang via cdouglas)
|
|
|
|
MAPREDUCE-1284. Fix fts_open() call in task-controller that was failing
|
|
LinuxTaskController unit tests. (Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-1143. Fix running task counters to be updated correctly
|
|
when speculative attempts are running for a TIP.
|
|
(Rahul Kumar Singh via yhemanth)
|
|
|
|
MAPREDUCE-1241. Use a default queue configuration in JobTracker when
|
|
mapred-queues.xml is unavailable. (Todd Lipcon via cdouglas)
|
|
|
|
MAPREDUCE-1301. Fix set up of permission checking script used in
|
|
localization tests. (Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1286. Remove quoting from client opts in TaskRunner. (Yuri
|
|
Pradkin via cdouglas)
|
|
|
|
MAPREDUCE-1059. Use distcp.bytes.per.map when adding sync markers in
|
|
distcp. (Aaron Kimball via cdouglas)
|
|
|
|
MAPREDUCE-1009. Update forrest documentation describing hierarchical
|
|
queues. (Vinod Kumar Vavilapalli via yhemanth)
|
|
|
|
MAPREDUCE-1342. Fixed deadlock in global blacklisting of tasktrackers.
|
|
(Amareshwari Sriramadasu via acmurthy)
|
|
|
|
MAPREDUCE-1316. Fixes a memory leak of TaskInProgress instances in
|
|
the jobtracker. (Amar Kamat via yhemanth)
|
|
|
|
MAPREDUCE-1359. TypedBytes TestIO doesn't mkdir its test dir first.
|
|
(Anatoli Fomenko via cos)
|
|
|
|
MAPREDUCE-1314. Correct errant mapreduce.x.mapreduce.x replacements from
|
|
bulk change. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1365. Restore accidentally renamed test in
|
|
TestTaskTrackerBloacklisting. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1406. Fix spelling of JobContext.MAP_COMBINE_MIN_SPILLS.
|
|
(cdouglas)
|
|
|
|
MAPREDUCE-1369. JUnit tests should never depend on anything in conf
|
|
(Anatoli Fomenko via cos)
|
|
|
|
MAPREDUCE-1412. Fix timer granularity issue causing failures in
|
|
TestTaskTrackerBlacklisting. (cdouglas)
|
|
|
|
MAPREDUCE-1448. Respect --config option in Mumak script. (Hong Tang via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1251. c++ utils doesn't compile. (Eli Collins via tomwhite)
|
|
|
|
MAPREDUCE-1522. FileInputFormat may use the default FileSystem for the
|
|
input path. (Tsz Wo (Nicholas), SZE via cdouglas)
|
|
|
|
MAPREDUCE-1407. Update javadoc in mapreduce.{Mapper,Reducer} to match
|
|
actual usage. (Benoit Sigoure via cdouglas)
|
|
|
|
MAPREDUCE-1258. Fix fair scheduler event log not logging job info.
|
|
(matei)
|
|
|
|
MAPREDUCE-1089. Fix NPE in fair scheduler preemption when tasks are
|
|
scheduled but not running. (Todd Lipcon via matei)
|
|
|
|
MAPREDUCE-1014. Fix the libraries for common and hdfs. (omalley)
|
|
|
|
MAPREDUCE-1111. JT Jetty UI not working if we run mumak.sh
|
|
off packaged distribution directory. (hong tang via mahadev)
|
|
|
|
MAPREDUCE-1133. Eclipse .classpath template has outdated jar files and is
|
|
missing some new ones. (cos)
|
|
|
|
MAPREDUCE-1098. Fixed the distributed-cache to not do i/o while holding a
|
|
global lock. (Amareshwari Sriramadasu via acmurthy)
|
|
|
|
MAPREDUCE-1158. Fix JT running maps and running reduces metrics.
|
|
(sharad)
|
|
|
|
MAPREDUCE-1160. Reduce verbosity of log lines in some Map/Reduce classes
|
|
to avoid filling up jobtracker logs on a busy cluster.
|
|
(Ravi Gummadi and Hong Tang via yhemanth)
|
|
|
|
MAPREDUCE-1153. Fix tasktracker metrics when trackers are decommissioned.
|
|
(sharad)
|
|
|
|
MAPREDUCE-1128. Fix MRUnit to prohibit iterating over values twice. (Aaron
|
|
Kimball via cdouglas)
|
|
|
|
MAPREDUCE-665. Move libhdfs to HDFS subproject. (Eli Collins via dhruba)
|
|
|
|
MAPREDUCE-1196. Fix FileOutputCommitter to use the deprecated cleanupJob
|
|
api correctly. (acmurthy)
|
|
|
|
MAPREDUCE-1244. Fix eclipse-plugin's build dependencies. (gkesavan)
|
|
|
|
MAPREDUCE-1140. Fix DistributedCache to not decrement reference counts for
|
|
unreferenced files in error conditions.
|
|
(Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1245. Fix TestFairScheduler failures by instantiating lightweight
|
|
Jobtracker. (sharad)
|
|
|
|
MAPREDUCE-1260. Update Eclipse configuration to match changes to Ivy
|
|
configuration. (Edwin Chan via cos)
|
|
|
|
MAPREDUCE-1152. Distinguish between failed and killed tasks in
|
|
JobTrackerInstrumentation. (Sharad Agarwal via cdouglas)
|
|
|
|
MAPREDUCE-1285. In DistCp.deleteNonexisting(..), get class from the
|
|
parameter instead of using FileStatus.class. (Peter Romianowski via
|
|
szetszwo)
|
|
|
|
MAPREDUCE-1294. Build fails to pull latest hadoop-core-* artifacts (cos)
|
|
|
|
MAPREDUCE-1213. TaskTrackers restart is faster because it deletes
|
|
distributed cache directory asynchronously. (Zheng Shao via dhruba)
|
|
|
|
MAPREDUCE-1265. The task attempt error log prints the name of the
|
|
tasktracker machine. (Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1201. ProcfsBasedProcessTree collects CPU usage information.
|
|
(Scott Chen via dhruba)
|
|
|
|
MAPREDUCE-1326. fi tests don't use fi-site.xml (cos)
|
|
|
|
MAPREDUCE-1165. Replace non-portable function name with C99 equivalent.
|
|
(Allen Wittenauer via cdouglas)
|
|
|
|
MAPREDUCE-1331. Fixes a typo in a testcase (Devaraj Das)
|
|
|
|
MAPREDUCE-1293. AutoInputFormat doesn't work with non-default FileSystems.
|
|
(Andrew Hitchcock via tomwhite)
|
|
|
|
MAPREDUCE-1131. Using profilers other than hprof can cause JobClient to
|
|
report job failure. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1155. Streaming tests swallow exceptions.
|
|
(Todd Lipcon via tomwhite)
|
|
|
|
MAPREDUCE-1212. Mapreduce contrib project ivy dependencies are not included
|
|
in binary target. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1388. Move the HDFS RAID package from HDFS to MAPREDUCE.
|
|
(Eli Collins via dhruba)
|
|
|
|
MAPREDUCE-1322. Defines default value for staging directory to be user
|
|
based fixing a failing streaming test.
|
|
(Devaraj Das and Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-899. Modified LinuxTaskController to check that task-controller
|
|
has right permissions and ownership before performing any actions.
|
|
(Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1443. DBInputFormat can leak connections.
|
|
(Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1457. Fixes JobTracker to get the FileSystem object within
|
|
getStagingAreaDir within a privileged block. Fixes Child.java to use the
|
|
appropriate UGIs while getting the TaskUmbilicalProtocol proxy and
|
|
while executing the task. (Jakob Homan via ddas)
|
|
|
|
MAPREDUCE-1399. The archive command shows a null error message (nicholas
|
|
via mahadev)
|
|
|
|
MAPREDUCE-1305. Improve efficiency of distcp -delete. (Peter Romianowski
|
|
via cdouglas)
|
|
|
|
MAPREDUCE-1474. Update forrest documentation for Hadoop Archives. (Mahadev
|
|
Konar via cdouglas)
|
|
|
|
MAPREDUCE-1400. Use tr rather than sed to effect literal substitution in
|
|
the build script. (Allen Wittenauer via cdouglas)
|
|
|
|
MAPREDUCE-1358. Avoid false positives in OutputLogFilter. (Todd Lipcon via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1490. Fix a NullPointerException that could occur during
|
|
instantiation and initialization of the DistributedRaidFileSystem.
|
|
(Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1476. Fix the M/R framework to not call commit for special
|
|
tasks like job setup/cleanup and task cleanup.
|
|
(Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1398. Fix TaskLauncher to stop waiting for slots on a TIP that
|
|
is killed / failed.
|
|
(Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1491. The parity files created by the RAID are combined
|
|
using Hadoop Archive Files (HAR). (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1378. URL encode link in jobhistory.jsp to avoid errors caused by
|
|
unescaped characters. (E. Sammer via cdouglas)
|
|
|
|
MAPREDUCE-1519. RaidNode fails to create new parity file
|
|
if an older version already exists. (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1537. Fixes a compilation problem in a testcase after commit
|
|
HDFS-984. (Jitendra Nath Pandey via ddas)
|
|
|
|
MAPREDUCE-1537. The patch makes the job client call the getDelegationToken
|
|
only when security is enabled. (Jitendra Nath Pandey via ddas)
|
|
|
|
MAPREDUCE-1510. RAID should regenerate parity files if they get deleted.
|
|
(Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1421. Fix the LinuxTaskController tests failing on trunk after
|
|
the commit of MAPREDUCE-1385. (Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-1520. Fix TestMiniMRLocalFS failure caused by regression in
|
|
getting user working dir. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1512. RAID uses HarFileSystem directly instead of
|
|
FileSystem.get (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1435. Fix symlink handling in task work directory when
|
|
cleaning up, essentially to avoid following links.
|
|
(Ravi Gummadi via yhemanth)
|
|
|
|
MAPREDUCE-1518. RaidNode does not run the deletion check on the
|
|
directory that stores the parity files. (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1573. TestStreamingAsDifferentUser fails if run as tt_user.
|
|
(Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-927. Cleanup of task-logs should happen in TaskTracker instead
|
|
of the Child. (Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-1578. Decouple HadoopArchives vesrion from HarFileSystem version.
|
|
(Rodrigo Schmidt via szetszwo)
|
|
|
|
MAPREDUCE-1422. Fix cleanup of localized job directory to work if files
|
|
with non-deletable permissions are created within it.
|
|
(Amar Kamat via yhemanth)
|
|
|
|
MAPREDUCE-1306. Randomize the arrival of heartbeat responses in Mumak.
|
|
(Tamas Sarlos via cdouglas)
|
|
|
|
MAPREDUCE-1579. archive: check and possibly replace the space charater
|
|
in source paths. (szetszwo)
|
|
|
|
MAPREDUCE-1536. DataDrivenDBInputFormat does not split date columns correctly.
|
|
(Aaron Kimball via enis)
|
|
|
|
MAPREDUCE-890. After HADOOP-4491, the user who started mapred system is
|
|
not able to run job. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1615. Fix compilation of TestSubmitJob. (cdouglas)
|
|
|
|
MAPREDUCE-1508. Protect against NPE in TestMultipleLevelCaching. (Aaron
|
|
Kimball via cdouglas)
|
|
|
|
MAPREDUCE-1497. Suppress spurious findbugs warning about IndexCache
|
|
synchronization. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1420. Fix TestTTResourceReporting failure. (Scott Chen via
|
|
cdouglas)
|
|
|
|
MAPREDUCE-1480. Correctly initialize child RecordReaders in
|
|
CombineFileInputFormat. (Aaron Kimball via cdouglas)
|
|
|
|
MAPREDUCE-1348. Fix block forensics packaging. (Tom White via cdouglas)
|
|
|
|
MAPREDUCE-1628. HarFileSystem shows incorrect replication numbers and
|
|
permissions. (szetszwo via mahadev)
|
|
|
|
MAPREDUCE-1602. Fix the error message for the case that src does not
|
|
exist. (szetszwo)
|
|
|
|
MAPREDUCE-1585. Create Hadoop Archives version 2 with filenames
|
|
URL-encoded (rodrigo via mahadev)
|
|
|
|
MAPREDUCE-1523. Sometimes rumen trace generator fails to extract the job
|
|
finish time. (dick king via mahadev)
|
|
|
|
MAPREDUCE-1635. ResourceEstimator does not work after MAPREDUCE-842.
|
|
(Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-889. binary communication formats added to Streaming by
|
|
HADOOP-1722 should be documented. (Klaas Bosteels via tomwhite)
|
|
|
|
MAPREDUCE-1031. ant tar target doens't seem to compile tests in contrib
|
|
projects. (Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1692. Removed unused testcase TestStreamedMerge.
|
|
(Sreekanth Ramakrishnan and Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1062. Fix ReliabilityTest to work with retired jobs. (Sreekanth
|
|
Ramakrishnan via cdouglas)
|
|
|
|
MAPREDUCE-1409. IOExceptions thrown from FIleOutputCommitter::abortTask
|
|
should cause the task to fail. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1695. Include capacity scheduler in findbugs and javadoc-dev
|
|
targets and also fix existing warnings. (Hong Tang via yhemanth)
|
|
|
|
MAPREDUCE-1494. Ensure TestJobDirCleanup verifies the correct paths.
|
|
(Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1622. Include missing slf4j dependencies. (cdouglas)
|
|
|
|
MAPREDUCE-1515. Accept java5.home from build.properties, not only from the
|
|
command line when building forrest docs. (Al Thompson via cdouglas)
|
|
|
|
MAPREDUCE-1618. Add missing javadoc to JobStatus::*JobAcls. (Amareshwari
|
|
Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1219. Remove job level metrics from jobtracker metrics to ease
|
|
undue load on jobtracker. (Sreekanth Ramakrishnan via sharad)
|
|
|
|
MAPREDUCE-1604. Add Forrest documentation for Job ACLs.
|
|
(Amareshwari Sriramadasu via yhemanth)
|
|
|
|
MAPREDUCE-1705. Archiving and Purging of HDFS parity files should
|
|
handle globbed policies accurately. (Rodrigo Schmidt via dhruba)
|
|
|
|
MAPREDUCE-1612. job conf file is not accessible from job history web page.
|
|
(Ravi Gummadi and Sreekanth Ramakrishnan via vinodkv)
|
|
|
|
MAPREDUCE-1397. NullPointerException observed during task failures.
|
|
(Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-1728. Oracle timezone strings do not match Java.
|
|
(Aaron Kimball via tomwhite)
|
|
|
|
MAPREDUCE-1609. TaskTracker.localizeJob should not set permissions on
|
|
job log directory recursively. (Amareshwari Sriramadasu via vinodkv)
|
|
|
|
MAPREDUCE-1657. After task logs directory is deleted, tasklog servlet
|
|
displays wrong error message about job ACLs. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1727. TestJobACLs fails after HADOOP-6686. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1611. Refresh nodes and refresh queues doesnt work with service
|
|
authorization enabled. (Amar Kamat via vinodkv)
|
|
|
|
MAPREDUCE-1276. Correct flaws in the shuffle related to connection setup
|
|
and failure attribution. (Amareshwari Sriramadasu via cdouglas)
|
|
|
|
MAPREDUCE-1372. ConcurrentModificationException in JobInProgress.
|
|
(Dick King and Amareshwari Sriramadasu via tomwhite)
|
|
|
|
MAPREDUCE-118. Fix Job.getJobID(). (Amareshwari Sriramadasu via sharad)
|
|
|
|
MAPREDUCE-913. TaskRunner crashes with NPE resulting in held up slots,
|
|
UNINITIALIZED tasks and hung TaskTracker. (Amareshwari Sriramadasu and
|
|
Sreekanth Ramakrishnan via vinodkv)
|
|
|
|
MAPREDUCE-1725. Fix MapReduce API incompatibilities between 0.20 and 0.21.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-1606. TestJobACLs may timeout as there are no slots for launching
|
|
JOB_CLEANUP task. (Ravi Gummadi via vinodkv)
|
|
|
|
MAPREDUCE-1765. Correct streaming documentation for StreamXmlRecordReader.
|
|
(Corinne Chandel via amareshwari)
|
|
|
|
MAPREDUCE-1880. Fix BigDecimal.divide(..) in the pi example. (szetszwo)
|
|
|
|
MAPREDUCE-1885. Revert FileSystem create method that takes CreateFlags
|
|
(MapReduce part of HADOOP-6826). (Ravi Gummadi via tomwhite)
|
|
|
|
MAPREDUCE-1870. Harmonize MapReduce JAR library versions with Common and
|
|
HDFS. (tomwhite)
|
|
|
|
MAPREDUCE-1791. Remote cluster control functionality needs JavaDocs
|
|
improvement (Konstantin Boudnik)
|
|
|
|
MAPREDUCE-1942. 'compile-fault-inject' should never be called directly.
|
|
(Konstantin Boudnik)
|
|
|
|
MAPREDUCE-1876. Fixes TaskAttemptStartedEvent to correctly log event type
|
|
for all task types. (Amar Kamat via amareshwari)
|
|
|
|
MAPREDUCE-1926. MapReduce distribution is missing build-utils.xml.
|
|
(tomwhite)
|
|
|
|
MAPREDUCE-2012. Some contrib tests fail in branch 0.21 and trunk.
|
|
(Amareshwari Sriramadasu via tomwhite)
|
|
|
|
MAPREDUCE-1980. Fixes TaskAttemptUnsuccessfulCompletionEvent and
|
|
TaskAttemptFinishedEvent to correctly log event type for all task types.
|
|
(Amar Kamat via amareshwari)
|
|
|
|
MAPREDUCE-1856. Extract a subset of tests for smoke (DOA) validation (cos)
|
|
|