Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Updating all CHANGES.txt files to move entires from future releases into 2.6.1 

section given the large number of backports to 2.6.1.
This commit is contained in:
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 2015-09-10 13:45:57 -07:00
parent 77666105b4
commit fbbb7ff1ed
4 changed files with 408 additions and 412 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

115
hadoop-common-project/hadoop-common/CHANGES.txt
View File

@ -342,9 +342,6 @@ Trunk (Unreleased)
HADOOP-7256. Resource leak during failure scenario of closing HADOOP-7256. Resource leak during failure scenario of closing
of resources. (Ramkrishna S. Vasudevan via harsh) of resources. (Ramkrishna S. Vasudevan via harsh)
HADOOP-8151. Error handling in snappy decompressor throws invalid
exceptions. (Matt Foley via harsh)
HADOOP-8813. Add InterfaceAudience and InterfaceStability annotations HADOOP-8813. Add InterfaceAudience and InterfaceStability annotations
to RPC Server and Client classes. (Brandon Li via suresh) to RPC Server and Client classes. (Brandon Li via suresh)
@ -742,9 +739,6 @@ Release 2.8.0 - UNRELEASED
command-line arguments passed by the user (Masatake Iwasaki via Colin P. command-line arguments passed by the user (Masatake Iwasaki via Colin P.
McCabe) McCabe)
HADOOP-12280. Skip unit tests based on maven profile rather than
NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe)
HADOOP-12318. Expose underlying LDAP exceptions in SaslPlainServer. (Mike HADOOP-12318. Expose underlying LDAP exceptions in SaslPlainServer. (Mike
Yoder via atm) Yoder via atm)
@ -776,9 +770,6 @@ Release 2.8.0 - UNRELEASED
HADOOP-11785. Reduce the number of listStatus operation in distcp HADOOP-11785. Reduce the number of listStatus operation in distcp
buildListing (Zoran Dimitrijevic via Colin P. McCabe) buildListing (Zoran Dimitrijevic via Colin P. McCabe)
HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up
split calculation (gera)
HADOOP-11970. Replace uses of ThreadLocal<Random> with JDK7 HADOOP-11970. Replace uses of ThreadLocal<Random> with JDK7
ThreadLocalRandom. (Sean Busbey via Colin P. McCabe) ThreadLocalRandom. (Sean Busbey via Colin P. McCabe)
@ -814,9 +805,6 @@ Release 2.8.0 - UNRELEASED
HADOOP-12352. Delay in checkpointing Trash can leave trash for 2 intervals HADOOP-12352. Delay in checkpointing Trash can leave trash for 2 intervals
before deleting (Casey Brotherton via harsh) before deleting (Casey Brotherton via harsh)
HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm (cmccabe)
HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of
jclass to GetStaticObjectField. (Hui Zheng via cnauroth) jclass to GetStaticObjectField. (Hui Zheng via cnauroth)
@ -865,9 +853,6 @@ Release 2.8.0 - UNRELEASED
HADOOP-11866. increase readability and reliability of checkstyle, HADOOP-11866. increase readability and reliability of checkstyle,
shellcheck, and whitespace reports (aw) shellcheck, and whitespace reports (aw)
HADOOP-11491. HarFs incorrectly declared as requiring an authority.
(Brahma Reddy Battula via gera)
HADOOP-11889. Make checkstyle runnable from root project HADOOP-11889. Make checkstyle runnable from root project
(Gera Shegalov via jeagles) (Gera Shegalov via jeagles)
@ -1129,9 +1114,6 @@ Release 2.7.2 - UNRELEASED
HADOOP-12304. Applications using FileContext fail with the default file HADOOP-12304. Applications using FileContext fail with the default file
system configured to be wasb/s3/etc. (cnauroth) system configured to be wasb/s3/etc. (cnauroth)
HADOOP-11932. MetricsSinkAdapter may hang when being stopped.
(Brahma Reddy Battula via jianhe)
HADOOP-12061. Incorrect command in single cluster setup document. HADOOP-12061. Incorrect command in single cluster setup document.
(Kengo Seki via aajisaka) (Kengo Seki via aajisaka)
@ -1162,12 +1144,6 @@ Release 2.7.1 - 2015-07-06
HADOOP-11868. Invalid user logins trigger large backtraces in server log HADOOP-11868. Invalid user logins trigger large backtraces in server log
(Chang Li via jlowe) (Chang Li via jlowe)
HADOOP-11730. Regression: s3n read failure recovery broken.
(Takenori Sato via stevel)
HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm (cmccabe)
HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh)
HADOOP-9658. SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when HADOOP-9658. SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when
@ -1182,9 +1158,6 @@ Release 2.7.1 - 2015-07-06
HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get
created with ACLs. (Gregory Chanan via asuresh) created with ACLs. (Gregory Chanan via asuresh)
HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes
infinite loop. (Larry McCay via cnauroth)
HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages.
(Kazuho Fujii via aajisaka) (Kazuho Fujii via aajisaka)
@ -1448,18 +1421,12 @@ Release 2.7.0 - 2015-04-20
HADOOP-11323. WritableComparator#compare keeps reference to byte array. HADOOP-11323. WritableComparator#compare keeps reference to byte array.
(Wilfred Spiegelenburg via wang) (Wilfred Spiegelenburg via wang)
HADOOP-11238. Update the NameNode's Group Cache in the background when
possible (Chris Li via Colin P. McCabe)
HADOOP-10809. hadoop-azure: page blob support. (Dexter Bradshaw, HADOOP-10809. hadoop-azure: page blob support. (Dexter Bradshaw,
Mostafa Elhemali, Eric Hanson, and Mike Liddell via cnauroth) Mostafa Elhemali, Eric Hanson, and Mike Liddell via cnauroth)
HADOOP-11188. hadoop-azure: automatically expand page blobs when they become HADOOP-11188. hadoop-azure: automatically expand page blobs when they become
full. (Eric Hanson via cnauroth) full. (Eric Hanson via cnauroth)
HADOOP-11506. Configuration variable expansion regex expensive for long
values. (Gera Shegalov via gera)
HADOOP-11620. Add support for load balancing across a group of KMS for HA. HADOOP-11620. Add support for load balancing across a group of KMS for HA.
(Arun Suresh via wang) (Arun Suresh via wang)
@ -1566,9 +1533,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11355. When accessing data in HDFS and the key has been deleted, HADOOP-11355. When accessing data in HDFS and the key has been deleted,
a Null Pointer Exception is shown. (Arun Suresh via wang) a Null Pointer Exception is shown. (Arun Suresh via wang)
HADOOP-11343. Overflow is not properly handled in caclulating final iv for
AES CTR. (Jerry Chen via wang)
HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. HADOOP-11354. ThrottledInputStream doesn't perform effective throttling.
(Ted Yu via jing9) (Ted Yu via jing9)
@ -1584,9 +1548,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client,
non-core directories. (Li Lu via wheat9) non-core directories. (Li Lu via wheat9)
HADOOP-11368. Fix SSLFactory truststore reloader thread leak in
KMSClientProvider. (Arun Suresh via wang)
HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. HADOOP-11372. Fix new findbugs warnings in mapreduce-examples.
(Li Lu via wheat9) (Li Lu via wheat9)
@ -1682,9 +1643,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11462. TestSocketIOWithTimeout needs change for PowerPC platform. HADOOP-11462. TestSocketIOWithTimeout needs change for PowerPC platform.
(Ayappan via cnauroth) (Ayappan via cnauroth)
HADOOP-11350. The size of header buffer of HttpServer is too small when
HTTPS is enabled. (Benoy Antony via wheat9)
HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore
retrieveBlock(). (Ted Yu via stevel) retrieveBlock(). (Ted Yu via stevel)
@ -1709,9 +1667,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11507 Hadoop RPC Authentication problem with different user locale. HADOOP-11507 Hadoop RPC Authentication problem with different user locale.
(Talat UYARER via stevel) (Talat UYARER via stevel)
HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy
user. Contributed by Arun Suresh.
HADOOP-11499. Check of executorThreadsStarted in HADOOP-11499. Check of executorThreadsStarted in
ValueQueue#submitRefillTask() evades lock acquisition (Ted Yu via jlowe) ValueQueue#submitRefillTask() evades lock acquisition (Ted Yu via jlowe)
@ -1780,9 +1735,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of
the object from S3. (Dan Hecht via stevel). the object from S3. (Dan Hecht via stevel).
HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is
full. (Ming Ma via kihwal)
HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when
IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa) IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa)
@ -1795,9 +1747,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-9087. Queue size metric for metric sinks isn't actually maintained HADOOP-9087. Queue size metric for metric sinks isn't actually maintained
(Akira AJISAKA via jlowe) (Akira AJISAKA via jlowe)
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread. (cnauroth)
HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+. HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+.
(rkanter) (rkanter)
@ -1831,9 +1780,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it
to wrapped FS. (gera) to wrapped FS. (gera)
HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
should be non static. (Sean Busbey via yliu)
HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel)
HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa)
@ -1847,9 +1793,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11693. Azure Storage FileSystem rename operations are throttled too HADOOP-11693. Azure Storage FileSystem rename operations are throttled too
aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth) aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth)
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
synchronization. (Sean Busbey via yliu)
HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre
Matsumoto via ozawa) Matsumoto via ozawa)
@ -1925,7 +1868,7 @@ Release 2.6.2 - UNRELEASED
BUG FIXES BUG FIXES
Release 2.6.1 - UNRELEASED Release 2.6.1 - 2015-09-09
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -1936,11 +1879,20 @@ Release 2.6.1 - UNRELEASED
HADOOP-7139. Allow appending to existing SequenceFiles HADOOP-7139. Allow appending to existing SequenceFiles
(kanaka kumar avvaru via vinayakumarb) (kanaka kumar avvaru via vinayakumarb)
HADOOP-12280. Skip unit tests based on maven profile rather than
NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe)
OPTIMIZATIONS OPTIMIZATIONS
HADOOP-11238. Update the NameNode's Group Cache in the background when
possible (Chris Li via Colin P. McCabe)
HADOOP-11506. Configuration variable expansion regex expensive for long
values. (Gera Shegalov via gera)
BUG FIXES BUG FIXES
HADOOP-11466. FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC HADOOP-11466: FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC
architecture because it is slower there (Suman Somasundar via Colin P. architecture because it is slower there (Suman Somasundar via Colin P.
McCabe) McCabe)
@ -1949,6 +1901,51 @@ Release 2.6.1 - UNRELEASED
HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification
pipe is full (zhaoyunjiong via cmccabe) pipe is full (zhaoyunjiong via cmccabe)
HADOOP-11343. Overflow is not properly handled in caclulating final iv for
AES CTR. (Jerry Chen via wang)
HADOOP-11368. Fix SSLFactory truststore reloader thread leak in
KMSClientProvider. (Arun Suresh via wang)
HADOOP-11350. The size of header buffer of HttpServer is too small when
HTTPS is enabled. (Benoy Antony via wheat9)
HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy
user. Contributed by Arun Suresh.
HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is
full. (Ming Ma via kihwal)
HADOOP-11604. Prevent ConcurrentModificationException while closing domain
sockets during shutdown of DomainSocketWatcher thread. (cnauroth)
HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream
should be non static. (Sean Busbey via yliu)
HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt
synchronization. (Sean Busbey via yliu)
HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up
split calculation (gera)
HADOOP-11730. Regression: s3n read failure recovery broken.
(Takenori Sato via stevel)
HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there
is an I/O error during requestShortCircuitShm (cmccabe)
HADOOP-11491. HarFs incorrectly declared as requiring an authority.
(Brahma Reddy Battula via gera)
HADOOP-8151. Error handling in snappy decompressor throws invalid
exceptions. (Matt Foley via harsh)
HADOOP-11932. MetricsSinkAdapter may hang when being stopped.
(Brahma Reddy Battula via jianhe)
HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes
infinite loop. (Larry McCay via cnauroth)
Release 2.6.0 - 2014-11-18 Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

260
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
View File

@ -441,8 +441,6 @@ Release 2.8.0 - UNRELEASED
HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package. HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package.
(szetszwo) (szetszwo)
HDFS-8046. Allow better control of getContentSummary (kihwal)
HDFS-8076. Code cleanup for DFSInputStream: use offset instead of HDFS-8076. Code cleanup for DFSInputStream: use offset instead of
LocatedBlock when possible. (Zhe Zhang via wang) LocatedBlock when possible. (Zhe Zhang via wang)
@ -843,9 +841,6 @@ Release 2.8.0 - UNRELEASED
HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu)
HDFS-8846. Add a unit test for INotify functionality across a layout
version upgrade (Zhe Zhang via Colin P. McCabe)
HDFS-8951. Move the shortcircuit package to hdfs-client. HDFS-8951. Move the shortcircuit package to hdfs-client.
(Mingliang Liu via wheat9) (Mingliang Liu via wheat9)
@ -894,9 +889,6 @@ Release 2.8.0 - UNRELEASED
HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to
hadoop-hdfs-client module. (Mingliang Liu via wheat9) hadoop-hdfs-client module. (Mingliang Liu via wheat9)
HDFS-8384. Allow NN to startup if there are files having a lease but are not
under construction. (jing9)
HDFS-8984. Move replication queues related methods in FSNamesystem to HDFS-8984. Move replication queues related methods in FSNamesystem to
BlockManager. (wheat9) BlockManager. (wheat9)
@ -1041,9 +1033,6 @@ Release 2.8.0 - UNRELEASED
HDFS-8290. WebHDFS calls before namesystem initialization can cause HDFS-8290. WebHDFS calls before namesystem initialization can cause
NullPointerException. (cnauroth) NullPointerException. (cnauroth)
HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart.
(surendra singh lilhore via Xiaoyu Yao)
HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows. HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows.
(Kiran Kumar M R via Xiaoyu Yao) (Kiran Kumar M R via Xiaoyu Yao)
@ -1150,18 +1139,12 @@ Release 2.8.0 - UNRELEASED
HDFS-8268. Port conflict log for data node server is not sufficient HDFS-8268. Port conflict log for data node server is not sufficient
(Mohammad Shahid Khan via vinayakumarb) (Mohammad Shahid Khan via vinayakumarb)
HDFS-8431. hdfs crypto class not found in Windows.
(Anu Engineer via cnauroth)
HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake
Iwasaki via Colin P. McCabe) Iwasaki via Colin P. McCabe)
HDFS-7401. Add block info to DFSInputStream' WARN message when it adds HDFS-7401. Add block info to DFSInputStream' WARN message when it adds
node to deadNodes (Arshad Mohammad via vinayakumarb) node to deadNodes (Arshad Mohammad via vinayakumarb)
HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits.
(Ming Ma via jing9)
HDFS-8490. Typo in trace enabled log in ExceptionHandler of WebHDFS. HDFS-8490. Typo in trace enabled log in ExceptionHandler of WebHDFS.
(Archana T via ozawa) (Archana T via ozawa)
@ -1365,9 +1348,6 @@ Release 2.7.2 - UNRELEASED
HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9) HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9)
HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is
flawed. (Kihwal Lee via yliu)
HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9) HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9)
HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect. HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect.
@ -1412,15 +1392,9 @@ Release 2.7.1 - 2015-07-06
(Surendra Singh Lilhore via szetszwo) (Surendra Singh Lilhore via szetszwo)
OPTIMIZATIONS OPTIMIZATIONS
HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using
hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang
via Colin P. McCabe)
BUG FIXES BUG FIXES
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
finalize upgrade. (jing9)
HDFS-8151. Always use snapshot path as source when invalid snapshot names HDFS-8151. Always use snapshot path as source when invalid snapshot names
are used for diff based distcp. (jing9) are used for diff based distcp. (jing9)
@ -1442,9 +1416,6 @@ Release 2.7.1 - 2015-07-06
HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode.
(surendra singh lilhore via szetszwo) (surendra singh lilhore via szetszwo)
HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on
post-HDFS-7915 DataNode (cmccabe)
HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the
lock. (wheat9) lock. (wheat9)
@ -1457,17 +1428,8 @@ Release 2.7.1 - 2015-07-06
HDFS-8305: HDFS INotify: the destination field of RenameOp should always HDFS-8305: HDFS INotify: the destination field of RenameOp should always
end with the file name (cmccabe) end with the file name (cmccabe)
HDFS-7980. Incremental BlockReport will dramatically slow down namenode
startup. (Walter Su via szetszwo)
HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb) HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb)
HDFS-7894. Rolling upgrade readiness is not updated in jmx until query
command is issued. (Brahma Reddy Battula via kihwal)
HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block
request is in edit log. (Rushabh S Shah via kihwal)
HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor
goes for infinite loop (Rushabh S Shah via kihwal) goes for infinite loop (Rushabh S Shah via kihwal)
@ -1476,9 +1438,6 @@ Release 2.7.1 - 2015-07-06
HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo) HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo)
HDFS-8404. Pending block replication can get stuck using older genstamp
(Nathan Roberts via kihwal)
HDFS-8451. DFSClient probe for encryption testing interprets empty URI HDFS-8451. DFSClient probe for encryption testing interprets empty URI
property for "enabled". (Steve Loughran via xyao) property for "enabled". (Steve Loughran via xyao)
@ -1527,9 +1486,6 @@ Release 2.7.0 - 2015-04-20
NEW FEATURES NEW FEATURES
HDFS-7278. Add a command that allows sysadmins to manually trigger full
block reports from a DN (cmccabe)
HDFS-6663. Admin command to track file and locations from block id. HDFS-6663. Admin command to track file and locations from block id.
(Chen He via kihwal) (Chen He via kihwal)
@ -1623,9 +1579,6 @@ Release 2.7.0 - 2015-04-20
HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could
have a better error message. (Yongjun Zhang via wang) have a better error message. (Yongjun Zhang via wang)
HDFS-7035. Make adding a new data directory to the DataNode an atomic
operation and improve error handling (Lei Xu via Colin P. McCabe)
HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease, HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease,
etc. (cmccabe) etc. (cmccabe)
@ -1711,9 +1664,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7462. Consolidate implementation of mkdirs() into a single class. HDFS-7462. Consolidate implementation of mkdirs() into a single class.
(wheat9) (wheat9)
HDFS-7446. HDFS inotify should have the ability to determine what txid it
has read up to (cmccabe)
HDFS-6735. A minor optimization to avoid pread() be blocked by read() HDFS-6735. A minor optimization to avoid pread() be blocked by read()
inside the same DFSInputStream (Lars Hofhansl via stack) inside the same DFSInputStream (Lars Hofhansl via stack)
@ -1756,9 +1706,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7528. Consolidate symlink-related implementation into a single class. HDFS-7528. Consolidate symlink-related implementation into a single class.
(wheat9) (wheat9)
HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin
P. McCabe)
HDFS-7373. Clean up temporary files after fsimage transfer failures. HDFS-7373. Clean up temporary files after fsimage transfer failures.
(kihwal) (kihwal)
@ -1776,8 +1723,6 @@ Release 2.7.0 - 2015-04-20
HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop
codebase. (Sangjin Lee via Colin P. McCabe) codebase. (Sangjin Lee via Colin P. McCabe)
HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9)
HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin. HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin.
(jing9 via yliu) (jing9 via yliu)
@ -1971,9 +1916,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7410. Support CreateFlags with append() to support hsync() for HDFS-7410. Support CreateFlags with append() to support hsync() for
appending streams (Vinayakumar B via Colin P. McCabe) appending streams (Vinayakumar B via Colin P. McCabe)
HDFS-7742. Favoring decommissioning node for replication can cause a block
to stay underreplicated for long periods (Nathan Roberts via kihwal)
HDFS-8008. Support client-side back off when the datanodes are congested. HDFS-8008. Support client-side back off when the datanodes are congested.
(wheat9) (wheat9)
@ -2121,9 +2063,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7431. log message for InvalidMagicNumberException may be incorrect. HDFS-7431. log message for InvalidMagicNumberException may be incorrect.
(Yi Liu via cnauroth) (Yi Liu via cnauroth)
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe)
HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java
(Colin P.McCabe) (Colin P.McCabe)
@ -2150,21 +2089,9 @@ Release 2.7.0 - 2015-04-20
HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs. HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs.
(Zhanwei Wang via cnauroth) (Zhanwei Wang via cnauroth)
HDFS-7579. Improve log reporting during block report rpc failure.
(Charles Lamb via cnauroth)
HDFS-7596. NameNode should prune dead storages from storageMap.
(Arpit Agarwal via cnauroth)
HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade
shutdown command (Eric Payne via kihwal)
HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader
incorrectly (Jonathan Mace via Colin P. McCabe) incorrectly (Jonathan Mace via Colin P. McCabe)
HDFS-7470. SecondaryNameNode need twice memory when calling
reloadFromImageFile. (zhaoyunjiong via cnauroth)
HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures
with page sizes other than 4096 (Sam Liu via Colin P. McCabe) with page sizes other than 4096 (Sam Liu via Colin P. McCabe)
@ -2182,15 +2109,9 @@ Release 2.7.0 - 2015-04-20
HDFS-7496. Fix FsVolume removal race conditions on the DataNode by HDFS-7496. Fix FsVolume removal race conditions on the DataNode by
reference-counting the volume instances (lei via cmccabe) reference-counting the volume instances (lei via cmccabe)
HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via
Colin P. McCabe)
HDFS-7548. Corrupt block reporting delayed until datablock scanner thread HDFS-7548. Corrupt block reporting delayed until datablock scanner thread
detects it (Rushabh Shah via kihwal) detects it (Rushabh Shah via kihwal)
HDFS-7575. Upgrade should generate a unique storage ID for each
volume. (Arpit Agarwal)
HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace. HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace.
(Ming Ma via cnauroth) (Ming Ma via cnauroth)
@ -2235,9 +2156,6 @@ Release 2.7.0 - 2015-04-20
HDFS-6651. Deletion failure can leak inodes permanently. HDFS-6651. Deletion failure can leak inodes permanently.
(Jing Zhao via wheat9) (Jing Zhao via wheat9)
HDFS-7707. Edit log corruption due to delayed block removal again.
(Yongjun Zhang via kihwal)
HDFS-7734. Class cast exception in NameNode#main. (yliu via wang) HDFS-7734. Class cast exception in NameNode#main. (yliu via wang)
HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some
@ -2264,9 +2182,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider
threads when using FileContext (Arun Suresh via Colin P. McCabe) threads when using FileContext (Arun Suresh via Colin P. McCabe)
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
DataNode to register successfully with only one NameNode.(vinayakumarb)
HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir.
(szetszwo) (szetszwo)
@ -2300,15 +2215,9 @@ Release 2.7.0 - 2015-04-20
HDFS-6662. WebHDFS cannot open a file if its path contains "%". HDFS-6662. WebHDFS cannot open a file if its path contains "%".
(Gerson Carlos via wheat9) (Gerson Carlos via wheat9)
HDFS-7788. Post-2.6 namenode may not start up with an image containing
inodes created with an old release. (Rushabh Shah via kihwal)
HDFS-7814. Fix usage string of storageType parameter for HDFS-7814. Fix usage string of storageType parameter for
"dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth) "dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth)
HDFS-7009. Active NN and standby NN have different live nodes.
(Ming Ma via cnauroth)
HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe) HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe)
HDFS-7805. NameNode recovery prompt should be printed on console (Surendra HDFS-7805. NameNode recovery prompt should be printed on console (Surendra
@ -2320,9 +2229,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7831. Fix the starting index and end condition of the loop in HDFS-7831. Fix the starting index and end condition of the loop in
FileDiffList.findEarlierSnapshotBlocks(). (Konstantin Shvachko via jing9) FileDiffList.findEarlierSnapshotBlocks(). (Konstantin Shvachko via jing9)
HDFS-7763. fix zkfc hung issue due to not catching exception in a corner
case. (Liang Xie via wang)
HDFS-7843. A truncated file is corrupted after rollback from a rolling HDFS-7843. A truncated file is corrupted after rollback from a rolling
upgrade. (szetszwo) upgrade. (szetszwo)
@ -2335,9 +2241,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7785. Improve diagnostics information for HttpPutFailedException. HDFS-7785. Improve diagnostics information for HttpPutFailedException.
(Chengbing Liu via wheat9) (Chengbing Liu via wheat9)
HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception"
message. (jing9)
HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula
via Arpit Agarwal) via Arpit Agarwal)
@ -2355,15 +2258,9 @@ Release 2.7.0 - 2015-04-20
HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal) HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal)
HDFS-7885. Datanode should not trust the generation stamp provided by
client. (Tsz Wo Nicholas Sze via jing9)
HDFS-7818. OffsetParam should return the default value instead of throwing HDFS-7818. OffsetParam should return the default value instead of throwing
NPE when the value is unspecified. (Eric Payne via wheat9) NPE when the value is unspecified. (Eric Payne via wheat9)
HDFS-7830. DataNode does not release the volume lock when adding a volume
fails. (Lei Xu via Colin P. Mccabe)
HDFS-6833. DirectoryScanner should not register a deleting block with HDFS-6833. DirectoryScanner should not register a deleting block with
memory of DataNode. (Shinichi Yamashita via szetszwo) memory of DataNode. (Shinichi Yamashita via szetszwo)
@ -2373,9 +2270,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7903. Cannot recover block after truncate and delete snapshot. HDFS-7903. Cannot recover block after truncate and delete snapshot.
(Plamen Jeliazkov via shv) (Plamen Jeliazkov via shv)
HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and
fail to tell the DFSClient about it because of a network error (cmccabe)
HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv) HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv)
HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao
@ -2409,18 +2303,10 @@ Release 2.7.0 - 2015-04-20
HDFS-7943. Append cannot handle the last block with length greater than HDFS-7943. Append cannot handle the last block with length greater than
the preferred block size. (jing9) the preferred block size. (jing9)
HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade
starts (Zhe Zhang via Colin P. McCabe)
HDFS-7587. Edit log corruption can happen if append fails with a quota
violation. (jing9)
HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal) HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal)
HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal) HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal)
HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu)
HDFS-7957. Truncate should verify quota before making changes. (jing9) HDFS-7957. Truncate should verify quota before making changes. (jing9)
HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now() HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now()
@ -2428,13 +2314,6 @@ Release 2.7.0 - 2015-04-20
HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli) HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli)
HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp
provided by the client is larger than the one stored in the datanode.
(Brahma Reddy Battula via szetszwo)
HDFS-7960. The full block report should prune zombie storages even if
they're not empty. (cmccabe and Eddy Xu via wang)
HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang) HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang)
HDFS-7977. NFS couldn't take percentile intervals (brandonli) HDFS-7977. NFS couldn't take percentile intervals (brandonli)
@ -2467,15 +2346,9 @@ Release 2.7.0 - 2015-04-20
HDFS-8051. FsVolumeList#addVolume should release volume reference if not HDFS-8051. FsVolumeList#addVolume should release volume reference if not
put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe)
HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl
lock for a very long time (sinago via cmccabe)
HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in
platform-specific format. (Xiaoyu Yao via cnauroth) platform-specific format. (Xiaoyu Yao via cnauroth)
HDFS-8072. Reserved RBW space is not released if client terminates while
writing block. (Arpit Agarwal)
HDFS-8063: Fix intermittent test failures in TestTracing (Masatake Iwasaki HDFS-8063: Fix intermittent test failures in TestTracing (Masatake Iwasaki
via Colin P. McCabe) via Colin P. McCabe)
@ -2517,16 +2390,42 @@ Release 2.6.2 - UNRELEASED
BUG FIXES BUG FIXES
Release 2.6.1 - UNRELEASED Release 2.6.1 - 2015-09-09
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
NEW FEATURES NEW FEATURES
HDFS-7278. Add a command that allows sysadmins to manually trigger full
block reports from a DN (cmccabe)
IMPROVEMENTS IMPROVEMENTS
HDFS-7035. Make adding a new data directory to the DataNode an atomic
operation and improve error handling (Lei Xu via Colin P. McCabe)
HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin
P. McCabe)
HDFS-7579. Improve log reporting during block report rpc failure.
(Charles Lamb via cnauroth)
HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9)
HDFS-7596. NameNode should prune dead storages from storageMap.
(Arpit Agarwal via cnauroth)
HDFS-8046. Allow better control of getContentSummary (kihwal)
HDFS-8384. Allow NN to startup if there are files having a lease but are not
under construction. (jing9)
OPTIMIZATIONS OPTIMIZATIONS
HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using
hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang
via Colin P. McCabe)
BUG FIXES BUG FIXES
HDFS-7425. NameNode block deletion logging uses incorrect appender. HDFS-7425. NameNode block deletion logging uses incorrect appender.
@ -2550,8 +2449,7 @@ Release 2.6.1 - UNRELEASED
HDFS-7733. NFS: readdir/readdirplus return null directory HDFS-7733. NFS: readdir/readdirplus return null directory
attribute on failure. (Arpit Agarwal) attribute on failure. (Arpit Agarwal)
HDFS-8486. DN startup may cause severe data loss (Daryn Sharp via Colin P. HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe)
McCabe)
HDFS-7213. processIncrementalBlockReport performance degradation. HDFS-7213. processIncrementalBlockReport performance degradation.
(Eric Payne via kihwal) (Eric Payne via kihwal)
@ -2565,9 +2463,111 @@ Release 2.6.1 - UNRELEASED
HDFS-7225. Remove stale block invalidation work when DN re-registers with HDFS-7225. Remove stale block invalidation work when DN re-registers with
different UUID. (Zhe Zhang and Andrew Wang) different UUID. (Zhe Zhang and Andrew Wang)
HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade
shutdown command (Eric Payne via kihwal)
HDFS-7575. Upgrade should generate a unique storage ID for each
volume. (Arpit Agarwal)
HDFS-7707. Edit log corruption due to delayed block removal again.
(Yongjun Zhang via kihwal)
HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause
DataNode to register successfully with only one NameNode.(vinayakumarb)
HDFS-7788. Post-2.6 namenode may not start up with an image containing
inodes created with an old release. (Rushabh Shah via kihwal)
HDFS-7009. Active NN and standby NN have different live nodes.
(Ming Ma via cnauroth)
HDFS-7763. fix zkfc hung issue due to not catching exception in a corner
case. (Liang Xie via wang)
HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception"
message. (jing9)
HDFS-7885. Datanode should not trust the generation stamp provided by
client. (Tsz Wo Nicholas Sze via jing9)
HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via
Colin P. McCabe)
HDFS-7830. DataNode does not release the volume lock when adding a volume
fails. (Lei Xu via Colin P. Mccabe)
HDFS-7587. Edit log corruption can happen if append fails with a quota
violation. (jing9)
HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade
starts (Zhe Zhang via Colin P. McCabe)
HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu)
HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp
provided by the client is larger than the one stored in the datanode.
(Brahma Reddy Battula via szetszwo)
HDFS-7960. The full block report should prune zombie storages even if
they're not empty. (cmccabe and Eddy Xu via wang)
HDFS-7742. Favoring decommissioning node for replication can cause a block
to stay underreplicated for long periods (Nathan Roberts via kihwal)
HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl
lock for a very long time (sinago via cmccabe)
HDFS-8072. Reserved RBW space is not released if client terminates while
writing block. (Arpit Agarwal)
HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to
finalize upgrade. (jing9)
HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and
fail to tell the DFSClient about it because of a network error (cmccabe)
HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on
post-HDFS-7915 DataNode (cmccabe)
HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart.
(surendra singh lilhore via Xiaoyu Yao)
HDFS-7894. Rolling upgrade readiness is not updated in jmx until query
command is issued. (Brahma Reddy Battula via kihwal)
HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block
request is in edit log. (Rushabh S Shah via kihwal)
HDFS-8404. Pending block replication can get stuck using older genstamp
(Nathan Roberts via kihwal)
HDFS-8431. hdfs crypto class not found in Windows.
(Anu Engineer via cnauroth)
HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits.
(Ming Ma via jing9)
HDFS-8270. create() always retried with hardcoded timeout when file already HDFS-8270. create() always retried with hardcoded timeout when file already
exists with open lease (J.Andreina via vinayakumarb) exists with open lease (J.Andreina via vinayakumarb)
HDFS-7980. Incremental BlockReport will dramatically slow down namenode
startup. (Walter Su via szetszwo)
HDFS-7446. HDFS inotify should have the ability to determine what txid it
has read up to (cmccabe)
HDFS-8846. Add a unit test for INotify functionality across a layout
version upgrade (Zhe Zhang via Colin P. McCabe)
HDFS-7470. SecondaryNameNode need twice memory when calling
reloadFromImageFile. (zhaoyunjiong via cnauroth)
HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is
flawed. (Kihwal Lee via yliu)
HDFS-7552. Change FsVolumeList toString() to fix
TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe)
Release 2.6.0 - 2014-11-18 Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

56
hadoop-mapreduce-project/CHANGES.txt
View File

@ -448,9 +448,6 @@ Release 2.8.0 - UNRELEASED
MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce. MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce.
lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. (Ray Chiang via ozawa) lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. (Ray Chiang via ozawa)
MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge
(Gera Shegalov via jlowe)
MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8.
(Akira AJISAKA via ozawa) (Akira AJISAKA via ozawa)
@ -605,14 +602,6 @@ Release 2.7.1 - 2015-07-06
BUG FIXES BUG FIXES
MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka)
MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options
which is a regression from MR1 (zxu via rkanter)
MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM
tokens when they roll-over. (Jason Lowe via vinodkv)
MAPREDUCE-6252. JobHistoryServer should not fail when encountering a MAPREDUCE-6252. JobHistoryServer should not fail when encountering a
missing directory. (Craig Welch via devaraj) missing directory. (Craig Welch via devaraj)
@ -629,10 +618,6 @@ Release 2.7.1 - 2015-07-06
that they don't fail on history-server backed by DFSes with not so strong that they don't fail on history-server backed by DFSes with not so strong
guarantees. (Craig Welch via vinodkv) guarantees. (Craig Welch via vinodkv)
MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between
copySucceeded() in one thread and copyFailed() in another thread on the
same host. (Junping Du via ozawa)
MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at
the end. (Arun Suresh via kasha) the end. (Arun Suresh via kasha)
@ -714,9 +699,6 @@ Release 2.7.0 - 2015-04-20
MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public
API in DistCp. (Jing Zhao via vinodkv) API in DistCp. (Jing Zhao via vinodkv)
MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own
class. (Chris Trezzo via kasha)
MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the
application and forcefully kill. (Eric Payne via junping_du) application and forcefully kill. (Eric Payne via junping_du)
@ -766,9 +748,6 @@ Release 2.7.0 - 2015-04-20
MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output
directory. (gera) directory. (gera)
MAPREDUCE-6166. Reducers do not validate checksum of map outputs when
fetching directly to disk. (Eric Payne via gera)
MAPREDUCE-6045. need close the DataInputStream after open it in MAPREDUCE-6045. need close the DataInputStream after open it in
TestMapReduce.java (zxu via rkanter) TestMapReduce.java (zxu via rkanter)
@ -778,9 +757,6 @@ Release 2.7.0 - 2015-04-20
MAPREDUCE-3283. mapred classpath CLI does not display the complete classpath MAPREDUCE-3283. mapred classpath CLI does not display the complete classpath
(Varun Saxena via cnauroth) (Varun Saxena via cnauroth)
MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken
service name properly. (Jason Lowe via jianhe)
MAPREDUCE-6231. Grep example job is not working on a fully-distributed MAPREDUCE-6231. Grep example job is not working on a fully-distributed
cluster. (aajisaka) cluster. (aajisaka)
@ -831,9 +807,6 @@ Release 2.7.0 - 2015-04-20
MAPREDUCE-6285. ClientServiceDelegate should not retry upon MAPREDUCE-6285. ClientServiceDelegate should not retry upon
AuthenticationException. (Jonathan Eagles via ozawa) AuthenticationException. (Jonathan Eagles via ozawa)
MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
to a reducer. (Jason Lowe via junping_du)
Release 2.6.2 - UNRELEASED Release 2.6.2 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -846,7 +819,7 @@ Release 2.6.2 - UNRELEASED
BUG FIXES BUG FIXES
Release 2.6.1 - UNRELEASED Release 2.6.1 - 2015-09-09
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -861,6 +834,33 @@ Release 2.6.1 - UNRELEASED
MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of
reusing conections. (Kannan Rajah via ozawa) reusing conections. (Kannan Rajah via ozawa)
MAPREDUCE-6166. Reducers do not validate checksum of map outputs when
fetching directly to disk. (Eric Payne via gera)
MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken
service name properly. (Jason Lowe via jianhe)
MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal
to a reducer. (Jason Lowe via junping_du)
MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka)
MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own
class. (Chris Trezzo via kasha)
MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options
which is a regression from MR1 (zxu via rkanter)
MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM
tokens when they roll-over. (Jason Lowe via vinodkv)
MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge
(Gera Shegalov via jlowe)
MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between
copySucceeded() in one thread and copyFailed() in another thread on the
same host. (Junping Du via ozawa)
Release 2.6.0 - 2014-11-18 Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

389
hadoop-yarn-project/CHANGES.txt
View File

@ -338,9 +338,6 @@ Release 2.8.0 - UNRELEASED
YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI. YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI.
(Naganarasimha G R via wangda) (Naganarasimha G R via wangda)
YARN-3700. Made generic history service load a number of latest applications
according to the parameter or the configuration. (Xuan Gong via zjshen)
YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils. YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils.
(Masatake Iwasaki via devaraj) (Masatake Iwasaki via devaraj)
@ -498,9 +495,6 @@ Release 2.8.0 - UNRELEASED
YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender.
(Varun Vasudev via wangda) (Varun Vasudev via wangda)
YARN-2890. MiniYarnCluster should turn on timeline service if
configured to do so. (Mit Desai via hitesh)
YARN-3266. RMContext#inactiveNodes should have NodeId as map key. YARN-3266. RMContext#inactiveNodes should have NodeId as map key.
(Chengbing Liu via jianhe) (Chengbing Liu via jianhe)
@ -797,9 +791,6 @@ Release 2.8.0 - UNRELEASED
YARN-4005. Completed container whose app is finished is possibly not YARN-4005. Completed container whose app is finished is possibly not
removed from NMStateStore. (Jun Gong via jianhe) removed from NMStateStore. (Jun Gong via jianhe)
YARN-4047. ClientRMService getApplications has high scheduler lock contention.
(Jason Lowe via jianhe)
YARN-3987. Send AM container completed msg to NM once AM finishes. YARN-3987. Send AM container completed msg to NM once AM finishes.
(sandflee via jianhe) (sandflee via jianhe)
@ -844,9 +835,6 @@ Release 2.7.2 - UNRELEASED
YARN-3967. Fetch the application report from the AHS if the RM does not know about it. YARN-3967. Fetch the application report from the AHS if the RM does not know about it.
(Mit Desai via xgong) (Mit Desai via xgong)
YARN-3978. Configurably turn off the saving of container info in Generic AHS
(Eric Payne via jeagles)
YARN-4092. Fixed UI redirection to print useful messages when both RMs are YARN-4092. Fixed UI redirection to print useful messages when both RMs are
in standby mode. (Xuan Gong via jianhe) in standby mode. (Xuan Gong via jianhe)
@ -880,11 +868,6 @@ Release 2.7.2 - UNRELEASED
YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container
log files from full disks. (zhihai xu via jlowe) log files from full disks. (zhihai xu via jlowe)
YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when
Node is connected/disconnected (Bibin A Chundatt via jlowe)
YARN-3999. RM hangs on draing events. (Jian He via xgong)
YARN-3857: Memory leak in ResourceManager with SIMPLE mode. YARN-3857: Memory leak in ResourceManager with SIMPLE mode.
(mujunchao via zxu) (mujunchao via zxu)
@ -933,9 +916,6 @@ Release 2.7.1 - 2015-07-06
BUG FIXES BUG FIXES
YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when
calling getQueue (Jason Lowe via wangda)
YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf
without making a copy. (Jason Lowe via jianhe) without making a copy. (Jason Lowe via jianhe)
@ -959,9 +939,6 @@ Release 2.7.1 - 2015-07-06
YARN-3516. killing ContainerLocalizer action doesn't take effect when YARN-3516. killing ContainerLocalizer action doesn't take effect when
private localizer receives FETCH_FAILURE status.(zhihai xu via xgong) private localizer receives FETCH_FAILURE status.(zhihai xu via xgong)
YARN-3464. Race condition in LocalizerRunner kills localizer before
localizing all resources. (Zhihai Xu via kasha)
YARN-3485. FairScheduler headroom calculation doesn't consider YARN-3485. FairScheduler headroom calculation doesn't consider
maxResources for Fifo and FairShare policies. (kasha) maxResources for Fifo and FairShare policies. (kasha)
@ -986,9 +963,6 @@ Release 2.7.1 - 2015-07-06
YARN-3434. Interaction between reservations and userlimit can result in YARN-3434. Interaction between reservations and userlimit can result in
significant ULF violation (tgraves) significant ULF violation (tgraves)
YARN-3493. RM fails to come up with error "Failed to load/recover state"
when mem settings are changed. (Jian He via wangda)
YARN-3626. On Windows localized resources are not moved to the front YARN-3626. On Windows localized resources are not moved to the front
of the classpath when they should be. (Craig Welch via xgong) of the classpath when they should be. (Craig Welch via xgong)
@ -998,20 +972,11 @@ Release 2.7.1 - 2015-07-06
YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore
invoked (Brahma Reddy Battula via jlowe) invoked (Brahma Reddy Battula via jlowe)
YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when
exceptions happen in stopping NM's sub-services. (Junping Du via jlowe)
YARN-3526. ApplicationMaster tracking URL is incorrectly redirected
on a QJM cluster. (Weiwei Yang via xgong)
YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong) YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong)
YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager.
(Vinod Kumar Vavilapalli via ozawa) (Vinod Kumar Vavilapalli via ozawa)
YARN-2918. RM should not fail on startup if queue's configured labels do
not exist in cluster-node-labels. (Wangda Tan via jianhe)
YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. YARN-3681. yarn cmd says "could not find main class 'queue'" in windows.
(Craig Welch and Varun Saxena via xgong) (Craig Welch and Varun Saxena via xgong)
@ -1036,18 +1001,6 @@ Release 2.7.1 - 2015-07-06
YARN-3686. CapacityScheduler should trim default_node_label_expression. YARN-3686. CapacityScheduler should trim default_node_label_expression.
(Sunil G via wangda) (Sunil G via wangda)
YARN-2900. Application (Attempt and Container) Not Found in AHS results
in Internal Server Error (500). (Zhijie Shen and Mit Desai via xgong)
YARN-3725. App submission via REST API is broken in secure mode due to
Timeline DT service address is empty. (Zhijie Shen via wangda)
YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM
recovery is enabled (Rohith Sharmaks via jlowe)
YARN-3733. Fix DominantRC#compare() does not work as expected if
cluster resource is empty. (Rohith Sharmaks via wangda)
YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent
to another. (Wangda Tan via jianhe) to another. (Wangda Tan via jianhe)
@ -1060,13 +1013,6 @@ Release 2.7.1 - 2015-07-06
YARN-3809. Failed to launch new attempts because YARN-3809. Failed to launch new attempts because
ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe) ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe)
YARN-3832. Resource Localization fails on a cluster due to existing cache
directories (Brahma Reddy Battula via jlowe)
YARN-3850. NM fails to read files from full disks which can lead to
container logs being lost and other issues (Varun Saxena via jlowe)
Release 2.7.0 - 2015-04-20 Release 2.7.0 - 2015-04-20
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -1116,9 +1062,6 @@ Release 2.7.0 - 2015-04-20
YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) YARN-2994. Document work-preserving RM restart. (Jian He via ozawa)
YARN-3249. Add a 'kill application' button to Resource Manager's Web UI.
(Ryu Kobayashi via ozawa)
YARN-2786. Created a yarn cluster CLI and seeded with one command for listing YARN-2786. Created a yarn cluster CLI and seeded with one command for listing
node-labels collection. (Wangda Tan via vinodkv) node-labels collection. (Wangda Tan via vinodkv)
@ -1189,8 +1132,6 @@ Release 2.7.0 - 2015-04-20
YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics
for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du) for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du)
YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe)
YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe) YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe)
YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments
@ -1244,12 +1185,6 @@ Release 2.7.0 - 2015-04-20
YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable
node labels feature. (Wangda Tan via ozawa) node labels feature. (Wangda Tan via ozawa)
YARN-3024. LocalizerRunner should give DIE action when all resources are
localized. (Chengbing Liu via xgong)
YARN-3092. Created a common ResourceUsage class to track labeled resource
usages in Capacity Scheduler. (Wangda Tan via jianhe)
YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. YARN-3086. Make NodeManager memory configurable in MiniYARNCluster.
(Robert Metzger via ozawa) (Robert Metzger via ozawa)
@ -1275,9 +1210,6 @@ Release 2.7.0 - 2015-04-20
YARN-3022. Expose Container resource information from NodeManager for YARN-3022. Expose Container resource information from NodeManager for
monitoring (adhoot via ranter) monitoring (adhoot via ranter)
YARN-3098. Created common QueueCapacities class in Capacity Scheduler to
track capacities-by-labels of queues. (Wangda Tan via jianhe)
YARN-3075. NodeLabelsManager implementation to retrieve label to node YARN-3075. NodeLabelsManager implementation to retrieve label to node
mapping (Varun Saxena via wangda) mapping (Varun Saxena via wangda)
@ -1318,9 +1250,6 @@ Release 2.7.0 - 2015-04-20
YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager
restart. (Jason Lowe via junping_du) restart. (Jason Lowe via junping_du)
YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track
capacities-by-label. (Wangda Tan via jianhe)
YARN-3158. Correct log messages in ResourceTrackerService. YARN-3158. Correct log messages in ResourceTrackerService.
(Varun Saxena via xgong) (Varun Saxena via xgong)
@ -1345,8 +1274,6 @@ Release 2.7.0 - 2015-04-20
YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90.
(Zhihai Xu via junping_du) (Zhihai Xu via junping_du)
YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
YARN-3237. AppLogAggregatorImpl fails to log error cause. YARN-3237. AppLogAggregatorImpl fails to log error cause.
(Rushabh S Shah via xgong) (Rushabh S Shah via xgong)
@ -1380,9 +1307,6 @@ Release 2.7.0 - 2015-04-20
YARN-3122. Metrics for container's actual CPU usage. YARN-3122. Metrics for container's actual CPU usage.
(Anubhav Dhoot via kasha) (Anubhav Dhoot via kasha)
YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and
Xuan Gong via jianhe)
YARN-2190. Added CPU and memory limit options to the default container YARN-2190. Added CPU and memory limit options to the default container
executor for Windows containers. (Chuan Liu via jianhe) executor for Windows containers. (Chuan Liu via jianhe)
@ -1417,56 +1341,26 @@ Release 2.7.0 - 2015-04-20
YARN-2713. "RM Home" link in NM should point to one of the RMs in an YARN-2713. "RM Home" link in NM should point to one of the RMs in an
HA setup. (kasha) HA setup. (kasha)
YARN-2766. Made ApplicationHistoryManager return a sorted list of apps,
attempts and containers. (Robert Kanter via zjshen)
YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state
on app recovery. (Rohith Sharmaks via jianhe)
YARN-2857. ConcurrentModificationException in ContainerLogAppender YARN-2857. ConcurrentModificationException in ContainerLogAppender
(Mohammad Kamrul Islam via jlowe) (Mohammad Kamrul Islam via jlowe)
YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu
via jlowe)
YARN-2432. RMStateStore should process the pending events before close. YARN-2432. RMStateStore should process the pending events before close.
(Varun Saxena via jianhe) (Varun Saxena via jianhe)
YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. YARN-1703. Fixed ResourceManager web-proxy to close connections correctly.
(Rohith Sharma via vinodkv) (Rohith Sharma via vinodkv)
YARN-2414. RM web UI: app page will crash if app is failed before any
attempt has been created (Wangda Tan via jlowe)
YARN-2870. Updated the command to run the timeline server in the document. YARN-2870. Updated the command to run the timeline server in the document.
(Masatake Iwasaki via zjshen) (Masatake Iwasaki via zjshen)
YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via
jianhe) jianhe)
YARN-2865. Fixed RM to always create a new RMContext when transtions from
StandBy to Active. (Rohith Sharmaks via jianhe)
YARN-2315. FairScheduler: Set current capacity in addition to capacity. YARN-2315. FairScheduler: Set current capacity in addition to capacity.
(Zhihai Xu via kasha) (Zhihai Xu via kasha)
YARN-1984. LeveldbTimelineStore does not handle db exceptions properly
(Varun Saxena via jlowe)
YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen) YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen)
YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users.
(Jason Lowe via jianhe)
YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated
log file is corrupted (Varun Saxena via jlowe)
YARN-2894. Fixed a bug regarding application view acl when RM fails over.
(Rohith Sharmaks via jianhe)
YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute
any further apps. (Naganarasimha G R via kasha)
YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in
YarnConfiguration. (rchiang via rkanter) YarnConfiguration. (rchiang via rkanter)
@ -1479,18 +1373,12 @@ Release 2.7.0 - 2015-04-20
YARN-2931. PublicLocalizer may fail until directory is initialized by YARN-2931. PublicLocalizer may fail until directory is initialized by
LocalizeRunner. (Anubhav Dhoot via kasha) LocalizeRunner. (Anubhav Dhoot via kasha)
YARN-2910. FSLeafQueue can throw ConcurrentModificationException.
(Wilfred Spiegelenburg via kasha)
YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager
intermittent failure. (Wangda Tan via jianhe) intermittent failure. (Wangda Tan via jianhe)
YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case.
(Wangda Tan via jianhe) (Wangda Tan via jianhe)
YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher
(Rohith Sharmaks via jianhe)
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
SchedulerApplicationAttempt ctor. (devaraj) SchedulerApplicationAttempt ctor. (devaraj)
@ -1508,15 +1396,9 @@ Release 2.7.0 - 2015-04-20
YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance.
(Chris Trezzo via kasha) (Chris Trezzo via kasha)
YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie)
(Jian He via jlowe)
YARN-2675. containersKilled metrics is not updated when the container is killed YARN-2675. containersKilled metrics is not updated when the container is killed
during localization. (Zhihai Xu via kasha) during localization. (Zhihai Xu via kasha)
YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks
via jianhe)
YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha)
YARN-2977. Fixed intermittent TestNMClient failure. YARN-2977. Fixed intermittent TestNMClient failure.
@ -1524,12 +1406,6 @@ Release 2.7.0 - 2015-04-20
YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du)
YARN-2920. Changed CapacityScheduler to kill containers on nodes where
node labels are changed. (Wangda Tan via jianhe)
YARN-2340. Fixed NPE when queue is stopped during RM restart.
(Rohith Sharmaks via jianhe)
YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu
via junping_du) via junping_du)
@ -1541,9 +1417,6 @@ Release 2.7.0 - 2015-04-20
YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa)
YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla
via jianhe)
YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and
hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen) hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen)
@ -1553,15 +1426,9 @@ Release 2.7.0 - 2015-04-20
YARN-2991. Fixed DrainDispatcher to reuse the draining code path in YARN-2991. Fixed DrainDispatcher to reuse the draining code path in
AsyncDispatcher. (Rohith Sharmaks via zjshen) AsyncDispatcher. (Rohith Sharmaks via zjshen)
YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue.
(Rohith Sharmaks via ozawa)
YARN-2958. Made RMStateStore not update the last sequence number when updating the YARN-2958. Made RMStateStore not update the last sequence number when updating the
delegation token. (Varun Saxena via zjshen) delegation token. (Varun Saxena via zjshen)
YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via
jianhe)
YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat
via jianhe) via jianhe)
@ -1571,18 +1438,12 @@ Release 2.7.0 - 2015-04-20
YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on
getProto method. (Varun Saxena via jianhe) getProto method. (Varun Saxena via jianhe)
YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed
container statuses on heartbeat. (Chengbing Liu via jianhe)
YARN-3014. Replaces labels on a host should update all NM's labels on that YARN-3014. Replaces labels on a host should update all NM's labels on that
host. (Wangda Tan via jianhe) host. (Wangda Tan via jianhe)
YARN-3027. Scheduler should use totalAvailable resource from node instead of YARN-3027. Scheduler should use totalAvailable resource from node instead of
availableResource for maxAllocation. (adhoot via rkanter) availableResource for maxAllocation. (adhoot via rkanter)
YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler
when activating applications. (Craig Welch via jianhe)
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs.
(Zhijie Shen via jianhe) (Zhijie Shen via jianhe)
@ -1613,21 +1474,12 @@ Release 2.7.0 - 2015-04-20
YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native
executor returns an error (Eric Payne via jlowe) executor returns an error (Eric Payne via jlowe)
YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService
might lead NM to crash. (Varun Saxena via jianhe)
YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe
via jianhe)
YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource.
(Zhihai Xu via wangda) (Zhihai Xu via wangda)
YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not
work everywhere. (Varun Saxena via ozawa) work everywhere. (Varun Saxena via ozawa)
YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage
to track used-resources-by-label.(Wangda Tan via jianhe)
YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe) YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe)
YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal) YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal)
@ -1653,18 +1505,12 @@ Release 2.7.0 - 2015-04-20
YARN-1537. Fix race condition in YARN-1537. Fix race condition in
TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy) TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy)
YARN-2694. Ensure only single node label specified in ResourceRequest.
(Wangda Tan via jianhe)
YARN-3089. LinuxContainerExecutor does not handle file arguments to YARN-3089. LinuxContainerExecutor does not handle file arguments to
deleteAsUser (Eric Payne via jlowe) deleteAsUser (Eric Payne via jlowe)
YARN-3143. RM Apps REST API can return NPE or entries missing id and other YARN-3143. RM Apps REST API can return NPE or entries missing id and other
fields (jlowe) fields (jlowe)
YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong
via jianhe)
YARN-2971. RM uses conf instead of token service address to renew timeline YARN-2971. RM uses conf instead of token service address to renew timeline
delegation tokens (jeagles) delegation tokens (jeagles)
@ -1674,10 +1520,6 @@ Release 2.7.0 - 2015-04-20
YARN-2809. Implement workaround for linux kernel panic when removing YARN-2809. Implement workaround for linux kernel panic when removing
cgroup (Nathan Roberts via jlowe) cgroup (Nathan Roberts via jlowe)
YARN-2246. Made the proxy tracking URL always be
http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj
K via zjshen)
YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl.
(Chengbing Liu via junping_du) (Chengbing Liu via junping_du)
@ -1708,9 +1550,6 @@ Release 2.7.0 - 2015-04-20
YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk.
(Xuan Gong via junping_du) (Xuan Gong via junping_du)
YARN-3207. Secondary filter matches entites which do not have the key being
filtered for. (Zhijie Shen via xgong)
YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping
when node becomes deactivated. (Wangda Tan via jianhe) when node becomes deactivated. (Wangda Tan via jianhe)
@ -1723,15 +1562,9 @@ Release 2.7.0 - 2015-04-20
YARN-3194. RM should handle NMContainerStatuses sent by NM while YARN-3194. RM should handle NMContainerStatuses sent by NM while
registering if NM is Reconnected node (Rohith via jlowe) registering if NM is Reconnected node (Rohith via jlowe)
YARN-3238. Connection timeouts to nodemanagers are retried at
multiple levels (Jason Lowe via xgong)
YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. YARN-3247. TestQueueMappings should use CapacityScheduler explicitly.
(Zhihai Xu via ozawa) (Zhihai Xu via ozawa)
YARN-3239. WebAppProxy does not support a final tracking url which has
query fragments and params (Jian He via jlowe)
YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against
all Schedulers even when using ParameterizedSchedulerTestBase. all Schedulers even when using ParameterizedSchedulerTestBase.
(Anubhav Dhoot via devaraj) (Anubhav Dhoot via devaraj)
@ -1742,27 +1575,12 @@ Release 2.7.0 - 2015-04-20
YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's
available resource-limit from the parent queue. (Wangda Tan via vinodkv) available resource-limit from the parent queue. (Wangda Tan via vinodkv)
YARN-3222. Fixed RMNode to send scheduler events in sequential order when a
node reconnects. (Rohith Sharma K S via jianhe)
YARN-3131. YarnClientImpl should check FAILED and KILLED state in YARN-3131. YarnClientImpl should check FAILED and KILLED state in
submitApplication (Chang Li via jlowe) submitApplication (Chang Li via jlowe)
YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending
jobs. (Siqi Li via kasha)
YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving
events for old client. (Zhihai Xu via kasha)
YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired
(Zhijie Shen via xgong)
YARN-3275. CapacityScheduler: Preemption happening on non-preemptable YARN-3275. CapacityScheduler: Preemption happening on non-preemptable
queues (Eric Payne via jlowe) queues (Eric Payne via jlowe)
YARN-3287. Made TimelineClient put methods do as the correct login context.
(Daryn Sharp and Jonathan Eagles via zjshen)
YARN-3300. Outstanding_resource_requests table should not be shown in AHS. YARN-3300. Outstanding_resource_requests table should not be shown in AHS.
(Xuan Gong via jianhe) (Xuan Gong via jianhe)
@ -1777,9 +1595,6 @@ Release 2.7.0 - 2015-04-20
YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating
running logs of application when rolling is enabled. (Xuan Gong via vinodkv) running logs of application when rolling is enabled. (Xuan Gong via vinodkv)
YARN-3267. Timelineserver applies the ACL rules after applying the limit on
the number of records (Chang Li via jeagles)
YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work
in ATS / RM web ui. (Naganarasimha G R via xgong) in ATS / RM web ui. (Naganarasimha G R via xgong)
@ -1792,18 +1607,12 @@ Release 2.7.0 - 2015-04-20
YARN-3379. Fixed missing data in localityTable and ResourceRequests table YARN-3379. Fixed missing data in localityTable and ResourceRequests table
in RM WebUI. (Xuan Gong via jianhe) in RM WebUI. (Xuan Gong via jianhe)
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die.
(Brahma Reddy Battula via wangda)
YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-3384. TestLogAggregationService.verifyContainerLogs fails after
YARN-2777. (Naganarasimha G R via ozawa) YARN-2777. (Naganarasimha G R via ozawa)
YARN-3336. FileSystem memory leak in DelegationTokenRenewer. YARN-3336. FileSystem memory leak in DelegationTokenRenewer.
(Zhihai Xu via cnauroth) (Zhihai Xu via cnauroth)
YARN-3393. Getting application(s) goes wrong when app finishes before
starting the attempt. (Zhijie Shen via xgong)
YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG.
(Varun Saxena via xgong) (Varun Saxena via xgong)
@ -1817,9 +1626,6 @@ Release 2.7.0 - 2015-04-20
YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers
and node-label column (Jason Lowe via wangda) and node-label column (Jason Lowe via wangda)
YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token
renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv)
Release 2.6.2 - UNRELEASED Release 2.6.2 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
@ -1832,18 +1638,211 @@ Release 2.6.2 - UNRELEASED
BUG FIXES BUG FIXES
Release 2.6.1 - UNRELEASED Release 2.6.1 - 2015-09-09
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES
NEW FEATURES NEW FEATURES
YARN-3249. Add a 'kill application' button to Resource Manager's Web UI.
(Ryu Kobayashi via ozawa)
IMPROVEMENTS IMPROVEMENTS
YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and
Xuan Gong via jianhe)
YARN-3092. Created a common ResourceUsage class to track labeled resource
usages in Capacity Scheduler. (Wangda Tan via jianhe)
YARN-3098. Created common QueueCapacities class in Capacity Scheduler to
track capacities-by-labels of queues. (Wangda Tan via jianhe)
YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe)
YARN-3978. Configurably turn off the saving of container info in Generic AHS
(Eric Payne via jeagles)
OPTIMIZATIONS OPTIMIZATIONS
BUG FIXES BUG FIXES
YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state
on app recovery. (Rohith Sharmaks via jianhe)
YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu
via jlowe)
YARN-2414. RM web UI: app page will crash if app is failed before any
attempt has been created (Wangda Tan via jlowe)
YARN-2865. Fixed RM to always create a new RMContext when transtions from
StandBy to Active. (Rohith Sharmaks via jianhe)
YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users.
(Jason Lowe via jianhe)
YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated
log file is corrupted (Varun Saxena via jlowe)
YARN-2890. MiniYARNCluster should start the timeline server based on the
configuration. (Mit Desai via zjshen)
YARN-2894. Fixed a bug regarding application view acl when RM fails over.
(Rohith Sharmaks via jianhe)
YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute
any further apps. (Naganarasimha G R via kasha)
YARN-2910. FSLeafQueue can throw ConcurrentModificationException.
(Wilfred Spiegelenburg via kasha)
YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher
(Rohith Sharmaks via jianhe)
YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie)
(Jian He via jlowe)
YARN-1984. LeveldbTimelineStore does not handle db exceptions properly
(Varun Saxena via jlowe)
YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks
via jianhe)
YARN-2340. Fixed NPE when queue is stopped during RM restart.
(Rohith Sharmaks via jianhe)
YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla
via jianhe)
YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue.
(Rohith Sharmaks via ozawa)
YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed
container statuses on heartbeat. (Chengbing Liu via jianhe)
YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService
might lead NM to crash. (Varun Saxena via jianhe)
YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe
via jianhe)
YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong
via jianhe)
YARN-2246. Made the proxy tracking URL always be
http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. (Devaraj
K via zjshen)
YARN-3207. Secondary filter matches entites which do not have the key being
filtered for. (Zhijie Shen via xgong)
YARN-3238. Connection timeouts to nodemanagers are retried at
multiple levels (Jason Lowe via xgong)
YARN-3239. WebAppProxy does not support a final tracking url which has
query fragments and params (Jian He via jlowe)
YARN-3222. Fixed RMNode to send scheduler events in sequential order when a
node reconnects. (Rohith Sharma K S via jianhe)
YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending
jobs. (Siqi Li via kasha)
YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving
events for old client. (Zhihai Xu via kasha)
YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired
(Zhijie Shen via xgong)
YARN-3287. Made TimelineClient put methods do as the correct login context.
(Daryn Sharp and Jonathan Eagles via zjshen)
YARN-3267. Timelineserver applies the ACL rules after applying the limit on
the number of records (Chang Li via jeagles)
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die.
(Brahma Reddy Battula via wangda)
YARN-3393. Getting application(s) goes wrong when app finishes before
starting the attempt. (Zhijie Shen via xgong)
YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token
renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv)
YARN-3493. RM fails to come up with error "Failed to load/recover state"
when mem settings are changed. (Jian He via wangda)
YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when
calling getQueue (Jason Lowe via wangda)
YARN-3024. LocalizerRunner should give DIE action when all resources are
localized. (Chengbing Liu via xgong)
YARN-3464. Race condition in LocalizerRunner kills localizer before
localizing all resources. (Zhihai Xu via kasha)
YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when
exceptions happen in stopping NM's sub-services. (Junping Du via jlowe)
YARN-3526. ApplicationMaster tracking URL is incorrectly redirected
on a QJM cluster. (Weiwei Yang via xgong)
YARN-2766. Made ApplicationHistoryManager return a sorted list of apps,
attempts and containers. (Robert Kanter via zjshen)
YARN-3700. Made generic history service load a number of latest applications
according to the parameter or the configuration. (Xuan Gong via zjshen)
YARN-2900. Application (Attempt and Container) Not Found in AHS results
in InternalServer Error (500). (Zhijie Shen and Mit Desai via xgong)
YARN-3725. App submission via REST API is broken in secure mode due to
Timeline DT service address is empty. (Zhijie Shen via wangda)
YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM
recovery is enabled (Rohith Sharmaks via jlowe)
YARN-3832. Resource Localization fails on a cluster due to existing cache
directories (Brahma Reddy Battula via jlowe)
YARN-3850. NM fails to read files from full disks which can lead to
container logs being lost and other issues (Varun Saxena via jlowe)
YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when
Node is connected/disconnected (Bibin A Chundatt via jlowe)
YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler
when activating applications. (Craig Welch via jianhe)
YARN-3733. Fix DominantRC#compare() does not work as expected if
cluster resource is empty. (Rohith Sharmaks via wangda)
YARN-2920. Changed CapacityScheduler to kill containers on nodes where
node labels are changed. (Wangda Tan via jianhe)
YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via
jianhe)
YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage
to track used-resources-by-label.(Wangda Tan via jianhe)
YARN-2694. Ensure only single node label specified in ResourceRequest.
(Wangda Tan via jianhe)
YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track
capacities-by-label. (Wangda Tan via jianhe)
YARN-2918. RM should not fail on startup if queue's configured labels do
not exist in cluster-node-labels. (Wangda Tan via jianhe)
YARN-3999. RM hangs on draing events. (Jian He via xgong)
YARN-4047. ClientRMService getApplications has high scheduler lock contention.
(Jason Lowe via jianhe)
Release 2.6.0 - 2014-11-18 Release 2.6.0 - 2014-11-18
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES