From fbbb7ff1ed111884d0375c177239073153139848 Mon Sep 17 00:00:00 2001 From: "Vinod Kumar Vavilapalli (I am also known as @tshooter.)" Date: Thu, 10 Sep 2015 13:45:57 -0700 Subject: [PATCH] Updating all CHANGES.txt files to move entires from future releases into 2.6.1 section given the large number of backports to 2.6.1. --- .../hadoop-common/CHANGES.txt | 115 +++--- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 260 ++++++------ hadoop-mapreduce-project/CHANGES.txt | 56 +-- hadoop-yarn-project/CHANGES.txt | 389 +++++++++--------- 4 files changed, 408 insertions(+), 412 deletions(-) diff --git a/hadoop-common-project/hadoop-common/CHANGES.txt b/hadoop-common-project/hadoop-common/CHANGES.txt index 95e58afa4a0..c04bfd06939 100644 --- a/hadoop-common-project/hadoop-common/CHANGES.txt +++ b/hadoop-common-project/hadoop-common/CHANGES.txt @@ -342,9 +342,6 @@ Trunk (Unreleased) HADOOP-7256. Resource leak during failure scenario of closing of resources. (Ramkrishna S. Vasudevan via harsh) - HADOOP-8151. Error handling in snappy decompressor throws invalid - exceptions. (Matt Foley via harsh) - HADOOP-8813. Add InterfaceAudience and InterfaceStability annotations to RPC Server and Client classes. (Brandon Li via suresh) @@ -742,9 +739,6 @@ Release 2.8.0 - UNRELEASED command-line arguments passed by the user (Masatake Iwasaki via Colin P. McCabe) - HADOOP-12280. Skip unit tests based on maven profile rather than - NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) - HADOOP-12318. Expose underlying LDAP exceptions in SaslPlainServer. (Mike Yoder via atm) @@ -776,9 +770,6 @@ Release 2.8.0 - UNRELEASED HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe) - HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up - split calculation (gera) - HADOOP-11970. Replace uses of ThreadLocal with JDK7 ThreadLocalRandom. (Sean Busbey via Colin P. McCabe) @@ -814,9 +805,6 @@ Release 2.8.0 - UNRELEASED HADOOP-12352. Delay in checkpointing Trash can leave trash for 2 intervals before deleting (Casey Brotherton via harsh) - HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there - is an I/O error during requestShortCircuitShm (cmccabe) - HADOOP-10027. *Compressor_deflateBytesDirect passes instance instead of jclass to GetStaticObjectField. (Hui Zheng via cnauroth) @@ -865,9 +853,6 @@ Release 2.8.0 - UNRELEASED HADOOP-11866. increase readability and reliability of checkstyle, shellcheck, and whitespace reports (aw) - HADOOP-11491. HarFs incorrectly declared as requiring an authority. - (Brahma Reddy Battula via gera) - HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) @@ -1129,9 +1114,6 @@ Release 2.7.2 - UNRELEASED HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. (cnauroth) - HADOOP-11932. MetricsSinkAdapter may hang when being stopped. - (Brahma Reddy Battula via jianhe) - HADOOP-12061. Incorrect command in single cluster setup document. (Kengo Seki via aajisaka) @@ -1162,12 +1144,6 @@ Release 2.7.1 - 2015-07-06 HADOOP-11868. Invalid user logins trigger large backtraces in server log (Chang Li via jlowe) - HADOOP-11730. Regression: s3n read failure recovery broken. - (Takenori Sato via stevel) - - HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there - is an I/O error during requestShortCircuitShm (cmccabe) - HADOOP-11891. OsSecureRandom should lazily fill its reservoir (asuresh) HADOOP-9658. SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when @@ -1182,9 +1158,6 @@ Release 2.7.1 - 2015-07-06 HADOOP-11973. Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs. (Gregory Chanan via asuresh) - HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes - infinite loop. (Larry McCay via cnauroth) - HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. (Kazuho Fujii via aajisaka) @@ -1448,18 +1421,12 @@ Release 2.7.0 - 2015-04-20 HADOOP-11323. WritableComparator#compare keeps reference to byte array. (Wilfred Spiegelenburg via wang) - HADOOP-11238. Update the NameNode's Group Cache in the background when - possible (Chris Li via Colin P. McCabe) - HADOOP-10809. hadoop-azure: page blob support. (Dexter Bradshaw, Mostafa Elhemali, Eric Hanson, and Mike Liddell via cnauroth) HADOOP-11188. hadoop-azure: automatically expand page blobs when they become full. (Eric Hanson via cnauroth) - HADOOP-11506. Configuration variable expansion regex expensive for long - values. (Gera Shegalov via gera) - HADOOP-11620. Add support for load balancing across a group of KMS for HA. (Arun Suresh via wang) @@ -1566,9 +1533,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11355. When accessing data in HDFS and the key has been deleted, a Null Pointer Exception is shown. (Arun Suresh via wang) - HADOOP-11343. Overflow is not properly handled in caclulating final iv for - AES CTR. (Jerry Chen via wang) - HADOOP-11354. ThrottledInputStream doesn't perform effective throttling. (Ted Yu via jing9) @@ -1584,9 +1548,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, non-core directories. (Li Lu via wheat9) - HADOOP-11368. Fix SSLFactory truststore reloader thread leak in - KMSClientProvider. (Arun Suresh via wang) - HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. (Li Lu via wheat9) @@ -1682,9 +1643,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11462. TestSocketIOWithTimeout needs change for PowerPC platform. (Ayappan via cnauroth) - HADOOP-11350. The size of header buffer of HttpServer is too small when - HTTPS is enabled. (Benoy Antony via wheat9) - HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore retrieveBlock(). (Ted Yu via stevel) @@ -1709,9 +1667,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11507 Hadoop RPC Authentication problem with different user locale. (Talat UYARER via stevel) - HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy - user. Contributed by Arun Suresh. - HADOOP-11499. Check of executorThreadsStarted in ValueQueue#submitRefillTask() evades lock acquisition (Ted Yu via jlowe) @@ -1780,9 +1735,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel). - HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is - full. (Ming Ma via kihwal) - HADOOP-11599. Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured. (zhihai xu via ozawa) @@ -1795,9 +1747,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-9087. Queue size metric for metric sinks isn't actually maintained (Akira AJISAKA via jlowe) - HADOOP-11604. Prevent ConcurrentModificationException while closing domain - sockets during shutdown of DomainSocketWatcher thread. (cnauroth) - HADOOP-11612. Workaround for Curator's ChildReaper requiring Guava 15+. (rkanter) @@ -1831,9 +1780,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11605. FilterFileSystem#create with ChecksumOpt should propagate it to wrapped FS. (gera) - HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream - should be non static. (Sean Busbey via yliu) - HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel) HADOOP-11602. Fix toUpperCase/toLowerCase to use Locale.ENGLISH. (ozawa) @@ -1847,9 +1793,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11693. Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving. (Duo Xu via cnauroth) - HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt - synchronization. (Sean Busbey via yliu) - HADOOP-11558. Fix dead links to doc of hadoop-tools. (Jean-Pierre Matsumoto via ozawa) @@ -1925,7 +1868,7 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES @@ -1936,11 +1879,20 @@ Release 2.6.1 - UNRELEASED HADOOP-7139. Allow appending to existing SequenceFiles (kanaka kumar avvaru via vinayakumarb) + HADOOP-12280. Skip unit tests based on maven profile rather than + NativeCodeLoader.isNativeCodeLoaded (Masatake Iwasaki via Colin P. McCabe) + OPTIMIZATIONS + HADOOP-11238. Update the NameNode's Group Cache in the background when + possible (Chris Li via Colin P. McCabe) + + HADOOP-11506. Configuration variable expansion regex expensive for long + values. (Gera Shegalov via gera) + BUG FIXES - HADOOP-11466. FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC + HADOOP-11466: FastByteComparisons: do not use UNSAFE_COMPARER on the SPARC architecture because it is slower there (Suman Somasundar via Colin P. McCabe) @@ -1949,6 +1901,51 @@ Release 2.6.1 - UNRELEASED HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full (zhaoyunjiong via cmccabe) + HADOOP-11343. Overflow is not properly handled in caclulating final iv for + AES CTR. (Jerry Chen via wang) + + HADOOP-11368. Fix SSLFactory truststore reloader thread leak in + KMSClientProvider. (Arun Suresh via wang) + + HADOOP-11350. The size of header buffer of HttpServer is too small when + HTTPS is enabled. (Benoy Antony via wheat9) + + HADOOP-11482. Use correct UGI when KMSClientProvider is called by a proxy + user. Contributed by Arun Suresh. + + HADOOP-11295. RPC Server Reader thread can't shutdown if RPCCallQueue is + full. (Ming Ma via kihwal) + + HADOOP-11604. Prevent ConcurrentModificationException while closing domain + sockets during shutdown of DomainSocketWatcher thread. (cnauroth) + + HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream + should be non static. (Sean Busbey via yliu) + + HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt + synchronization. (Sean Busbey via yliu) + + HADOOP-11812. Implement listLocatedStatus for ViewFileSystem to speed up + split calculation (gera) + + HADOOP-11730. Regression: s3n read failure recovery broken. + (Takenori Sato via stevel) + + HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there + is an I/O error during requestShortCircuitShm (cmccabe) + + HADOOP-11491. HarFs incorrectly declared as requiring an authority. + (Brahma Reddy Battula via gera) + + HADOOP-8151. Error handling in snappy decompressor throws invalid + exceptions. (Matt Foley via harsh) + + HADOOP-11932. MetricsSinkAdapter may hang when being stopped. + (Brahma Reddy Battula via jianhe) + + HADOOP-11934. Use of JavaKeyStoreProvider in LdapGroupsMapping causes + infinite loop. (Larry McCay via cnauroth) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index e241460f102..0c2645db7d5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -441,8 +441,6 @@ Release 2.8.0 - UNRELEASED HDFS-8085. Move CorruptFileBlockIterator to a new hdfs.client.impl package. (szetszwo) - HDFS-8046. Allow better control of getContentSummary (kihwal) - HDFS-8076. Code cleanup for DFSInputStream: use offset instead of LocatedBlock when possible. (Zhe Zhang via wang) @@ -843,9 +841,6 @@ Release 2.8.0 - UNRELEASED HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) - HDFS-8846. Add a unit test for INotify functionality across a layout - version upgrade (Zhe Zhang via Colin P. McCabe) - HDFS-8951. Move the shortcircuit package to hdfs-client. (Mingliang Liu via wheat9) @@ -894,9 +889,6 @@ Release 2.8.0 - UNRELEASED HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to hadoop-hdfs-client module. (Mingliang Liu via wheat9) - HDFS-8384. Allow NN to startup if there are files having a lease but are not - under construction. (jing9) - HDFS-8984. Move replication queues related methods in FSNamesystem to BlockManager. (wheat9) @@ -1041,9 +1033,6 @@ Release 2.8.0 - UNRELEASED HDFS-8290. WebHDFS calls before namesystem initialization can cause NullPointerException. (cnauroth) - HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. - (surendra singh lilhore via Xiaoyu Yao) - HDFS-8310. Fix TestCLI.testAll "help: help for find" on Windows. (Kiran Kumar M R via Xiaoyu Yao) @@ -1150,18 +1139,12 @@ Release 2.8.0 - UNRELEASED HDFS-8268. Port conflict log for data node server is not sufficient (Mohammad Shahid Khan via vinayakumarb) - HDFS-8431. hdfs crypto class not found in Windows. - (Anu Engineer via cnauroth) - HDFS-8407. hdfsListDirectory must set errno to 0 on success (Masatake Iwasaki via Colin P. McCabe) HDFS-7401. Add block info to DFSInputStream' WARN message when it adds node to deadNodes (Arshad Mohammad via vinayakumarb) - HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. - (Ming Ma via jing9) - HDFS-8490. Typo in trace enabled log in ExceptionHandler of WebHDFS. (Archana T via ozawa) @@ -1365,9 +1348,6 @@ Release 2.7.2 - UNRELEASED HDFS-8867. Enable optimized block reports. (Daryn Sharp via jing9) - HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is - flawed. (Kihwal Lee via yliu) - HDFS-8891. HDFS concat should keep srcs order. (Yong Zhang via jing9) HDFS-8995. Flaw in registration bookeeping can make DN die on reconnect. @@ -1412,15 +1392,9 @@ Release 2.7.1 - 2015-07-06 (Surendra Singh Lilhore via szetszwo) OPTIMIZATIONS - HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using - hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang - via Colin P. McCabe) BUG FIXES - HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to - finalize upgrade. (jing9) - HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. (jing9) @@ -1442,9 +1416,6 @@ Release 2.7.1 - 2015-07-06 HDFS-8147. StorageGroup in Dispatcher should override equals nad hashCode. (surendra singh lilhore via szetszwo) - HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on - post-HDFS-7915 DataNode (cmccabe) - HDFS-8273. FSNamesystem#Delete() should not call logSync() when holding the lock. (wheat9) @@ -1457,17 +1428,8 @@ Release 2.7.1 - 2015-07-06 HDFS-8305: HDFS INotify: the destination field of RenameOp should always end with the file name (cmccabe) - HDFS-7980. Incremental BlockReport will dramatically slow down namenode - startup. (Walter Su via szetszwo) - HDFS-8226. Non-HA rollback compatibility broken (J.Andreina via vinayakumarb) - HDFS-7894. Rolling upgrade readiness is not updated in jmx until query - command is issued. (Brahma Reddy Battula via kihwal) - - HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block - request is in edit log. (Rushabh S Shah via kihwal) - HDFS-7916. 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop (Rushabh S Shah via kihwal) @@ -1476,9 +1438,6 @@ Release 2.7.1 - 2015-07-06 HDFS-8405. Fix a typo in NamenodeFsck. (Takanobu Asanuma via szetszwo) - HDFS-8404. Pending block replication can get stuck using older genstamp - (Nathan Roberts via kihwal) - HDFS-8451. DFSClient probe for encryption testing interprets empty URI property for "enabled". (Steve Loughran via xyao) @@ -1527,9 +1486,6 @@ Release 2.7.0 - 2015-04-20 NEW FEATURES - HDFS-7278. Add a command that allows sysadmins to manually trigger full - block reports from a DN (cmccabe) - HDFS-6663. Admin command to track file and locations from block id. (Chen He via kihwal) @@ -1623,9 +1579,6 @@ Release 2.7.0 - 2015-04-20 HDFS-3342. SocketTimeoutException in BlockSender.sendChunks could have a better error message. (Yongjun Zhang via wang) - HDFS-7035. Make adding a new data directory to the DataNode an atomic - operation and improve error handling (Lei Xu via Colin P. McCabe) - HDFS-6917. Add an hdfs debug command to validate blocks, call recoverlease, etc. (cmccabe) @@ -1711,9 +1664,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7462. Consolidate implementation of mkdirs() into a single class. (wheat9) - HDFS-7446. HDFS inotify should have the ability to determine what txid it - has read up to (cmccabe) - HDFS-6735. A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream (Lars Hofhansl via stack) @@ -1756,9 +1706,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7528. Consolidate symlink-related implementation into a single class. (wheat9) - HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin - P. McCabe) - HDFS-7373. Clean up temporary files after fsimage transfer failures. (kihwal) @@ -1776,8 +1723,6 @@ Release 2.7.0 - 2015-04-20 HADOOP-11470. Remove some uses of obsolete guava APIs from the hadoop codebase. (Sangjin Lee via Colin P. McCabe) - HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) - HDFS-7323. Move the get/setStoragePolicy commands out from dfsadmin. (jing9 via yliu) @@ -1971,9 +1916,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7410. Support CreateFlags with append() to support hsync() for appending streams (Vinayakumar B via Colin P. McCabe) - HDFS-7742. Favoring decommissioning node for replication can cause a block - to stay underreplicated for long periods (Nathan Roberts via kihwal) - HDFS-8008. Support client-side back off when the datanodes are congested. (wheat9) @@ -2121,9 +2063,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7431. log message for InvalidMagicNumberException may be incorrect. (Yi Liu via cnauroth) - HDFS-7552. Change FsVolumeList toString() to fix - TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) - HDFS-7557. Fix spacing for a few keys in DFSConfigKeys.java (Colin P.McCabe) @@ -2150,21 +2089,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7589. Break the dependency between libnative_mini_dfs and libhdfs. (Zhanwei Wang via cnauroth) - HDFS-7579. Improve log reporting during block report rpc failure. - (Charles Lamb via cnauroth) - - HDFS-7596. NameNode should prune dead storages from storageMap. - (Arpit Agarwal via cnauroth) - - HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade - shutdown command (Eric Payne via kihwal) - HDFS-5445. PacketReceiver populates the packetLen field in PacketHeader incorrectly (Jonathan Mace via Colin P. McCabe) - HDFS-7470. SecondaryNameNode need twice memory when calling - reloadFromImageFile. (zhaoyunjiong via cnauroth) - HDFS-7585. Get TestEnhancedByteBufferAccess working on CPU architectures with page sizes other than 4096 (Sam Liu via Colin P. McCabe) @@ -2182,15 +2109,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7496. Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances (lei via cmccabe) - HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via - Colin P. McCabe) - HDFS-7548. Corrupt block reporting delayed until datablock scanner thread detects it (Rushabh Shah via kihwal) - HDFS-7575. Upgrade should generate a unique storage ID for each - volume. (Arpit Agarwal) - HDFS-3519. Checkpoint upload may interfere with a concurrent saveNamespace. (Ming Ma via cnauroth) @@ -2235,9 +2156,6 @@ Release 2.7.0 - 2015-04-20 HDFS-6651. Deletion failure can leak inodes permanently. (Jing Zhao via wheat9) - HDFS-7707. Edit log corruption due to delayed block removal again. - (Yongjun Zhang via kihwal) - HDFS-7734. Class cast exception in NameNode#main. (yliu via wang) HDFS-7719. BlockPoolSliceStorage#removeVolumes fails to remove some @@ -2264,9 +2182,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7718. Store KeyProvider in ClientContext to avoid leaking key provider threads when using FileContext (Arun Suresh via Colin P. McCabe) - HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause - DataNode to register successfully with only one NameNode.(vinayakumarb) - HDFS-7769. TestHDFSCLI should not create files in hdfs project root dir. (szetszwo) @@ -2300,15 +2215,9 @@ Release 2.7.0 - 2015-04-20 HDFS-6662. WebHDFS cannot open a file if its path contains "%". (Gerson Carlos via wheat9) - HDFS-7788. Post-2.6 namenode may not start up with an image containing - inodes created with an old release. (Rushabh Shah via kihwal) - HDFS-7814. Fix usage string of storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota". (Xiaoyu Yao via cnauroth) - HDFS-7009. Active NN and standby NN have different live nodes. - (Ming Ma via cnauroth) - HDFS-7807. libhdfs htable.c: fix htable resizing, add unit test (cmccabe) HDFS-7805. NameNode recovery prompt should be printed on console (Surendra @@ -2320,9 +2229,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks(). (Konstantin Shvachko via jing9) - HDFS-7763. fix zkfc hung issue due to not catching exception in a corner - case. (Liang Xie via wang) - HDFS-7843. A truncated file is corrupted after rollback from a rolling upgrade. (szetszwo) @@ -2335,9 +2241,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7785. Improve diagnostics information for HttpPutFailedException. (Chengbing Liu via wheat9) - HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" - message. (jing9) - HDFS-7757. Misleading error messages in FSImage.java. (Brahma Reddy Battula via Arpit Agarwal) @@ -2355,15 +2258,9 @@ Release 2.7.0 - 2015-04-20 HDFS-7434. DatanodeID hashCode should not be mutable. (daryn via kihwal) - HDFS-7885. Datanode should not trust the generation stamp provided by - client. (Tsz Wo Nicholas Sze via jing9) - HDFS-7818. OffsetParam should return the default value instead of throwing NPE when the value is unspecified. (Eric Payne via wheat9) - HDFS-7830. DataNode does not release the volume lock when adding a volume - fails. (Lei Xu via Colin P. Mccabe) - HDFS-6833. DirectoryScanner should not register a deleting block with memory of DataNode. (Shinichi Yamashita via szetszwo) @@ -2373,9 +2270,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7903. Cannot recover block after truncate and delete snapshot. (Plamen Jeliazkov via shv) - HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and - fail to tell the DFSClient about it because of a network error (cmccabe) - HDFS-7886. Fix TestFileTruncate falures. (Plamen Jeliazkov and shv) HDFS-7946. TestDataNodeVolumeFailureReporting NPE on Windows. (Xiaoyu Yao @@ -2409,18 +2303,10 @@ Release 2.7.0 - 2015-04-20 HDFS-7943. Append cannot handle the last block with length greater than the preferred block size. (jing9) - HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade - starts (Zhe Zhang via Colin P. McCabe) - - HDFS-7587. Edit log corruption can happen if append fails with a quota - violation. (jing9) - HDFS-7816. Unable to open webhdfs paths with "+". (wheat9 via kihwal) HDFS-7932. Speed up the shutdown of datanode during rolling upgrade.(kihwal) - HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) - HDFS-7957. Truncate should verify quota before making changes. (jing9) HDFS-6841. Use Time.monotonicNow() wherever applicable instead of Time.now() @@ -2428,13 +2314,6 @@ Release 2.7.0 - 2015-04-20 HDFS-7942. NFS: support regexp grouping in nfs.exports.allowed.hosts (brandonli) - HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp - provided by the client is larger than the one stored in the datanode. - (Brahma Reddy Battula via szetszwo) - - HDFS-7960. The full block report should prune zombie storages even if - they're not empty. (cmccabe and Eddy Xu via wang) - HDFS-7961. Trigger full block report after hot swapping disk. (Eddy Xu via wang) HDFS-7977. NFS couldn't take percentile intervals (brandonli) @@ -2467,15 +2346,9 @@ Release 2.7.0 - 2015-04-20 HDFS-8051. FsVolumeList#addVolume should release volume reference if not put it into BlockScanner. (Lei (Eddy) Xu via Colin P. McCabe) - HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl - lock for a very long time (sinago via cmccabe) - HDFS-8038. PBImageDelimitedTextWriter#getEntry output HDFS path in platform-specific format. (Xiaoyu Yao via cnauroth) - HDFS-8072. Reserved RBW space is not released if client terminates while - writing block. (Arpit Agarwal) - HDFS-8063: Fix intermittent test failures in TestTracing (Masatake Iwasaki via Colin P. McCabe) @@ -2517,16 +2390,42 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES NEW FEATURES + HDFS-7278. Add a command that allows sysadmins to manually trigger full + block reports from a DN (cmccabe) + IMPROVEMENTS + HDFS-7035. Make adding a new data directory to the DataNode an atomic + operation and improve error handling (Lei Xu via Colin P. McCabe) + + HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin + P. McCabe) + + HDFS-7579. Improve log reporting during block report rpc failure. + (Charles Lamb via cnauroth) + + HDFS-7182. JMX metrics aren't accessible when NN is busy. (Ming Ma via jing9) + + HDFS-7596. NameNode should prune dead storages from storageMap. + (Arpit Agarwal via cnauroth) + + HDFS-8046. Allow better control of getContentSummary (kihwal) + + HDFS-8384. Allow NN to startup if there are files having a lease but are not + under construction. (jing9) + OPTIMIZATIONS + HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using + hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang + via Colin P. McCabe) + BUG FIXES HDFS-7425. NameNode block deletion logging uses incorrect appender. @@ -2550,8 +2449,7 @@ Release 2.6.1 - UNRELEASED HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. (Arpit Agarwal) - HDFS-8486. DN startup may cause severe data loss (Daryn Sharp via Colin P. - McCabe) + HDFS-8486. DN startup may cause severe data loss. (daryn via cmccabe) HDFS-7213. processIncrementalBlockReport performance degradation. (Eric Payne via kihwal) @@ -2565,9 +2463,111 @@ Release 2.6.1 - UNRELEASED HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang) + HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade + shutdown command (Eric Payne via kihwal) + + HDFS-7575. Upgrade should generate a unique storage ID for each + volume. (Arpit Agarwal) + + HDFS-7707. Edit log corruption due to delayed block removal again. + (Yongjun Zhang via kihwal) + + HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause + DataNode to register successfully with only one NameNode.(vinayakumarb) + + HDFS-7788. Post-2.6 namenode may not start up with an image containing + inodes created with an old release. (Rushabh Shah via kihwal) + + HDFS-7009. Active NN and standby NN have different live nodes. + (Ming Ma via cnauroth) + + HDFS-7763. fix zkfc hung issue due to not catching exception in a corner + case. (Liang Xie via wang) + + HDFS-7871. NameNodeEditLogRoller can keep printing "Swallowing exception" + message. (jing9) + + HDFS-7885. Datanode should not trust the generation stamp provided by + client. (Tsz Wo Nicholas Sze via jing9) + + HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via + Colin P. McCabe) + + HDFS-7830. DataNode does not release the volume lock when adding a volume + fails. (Lei Xu via Colin P. Mccabe) + + HDFS-7587. Edit log corruption can happen if append fails with a quota + violation. (jing9) + + HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade + starts (Zhe Zhang via Colin P. McCabe) + + HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) + + HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp + provided by the client is larger than the one stored in the datanode. + (Brahma Reddy Battula via szetszwo) + + HDFS-7960. The full block report should prune zombie storages even if + they're not empty. (cmccabe and Eddy Xu via wang) + + HDFS-7742. Favoring decommissioning node for replication can cause a block + to stay underreplicated for long periods (Nathan Roberts via kihwal) + + HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl + lock for a very long time (sinago via cmccabe) + + HDFS-8072. Reserved RBW space is not released if client terminates while + writing block. (Arpit Agarwal) + + HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to + finalize upgrade. (jing9) + + HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and + fail to tell the DFSClient about it because of a network error (cmccabe) + + HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on + post-HDFS-7915 DataNode (cmccabe) + + HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. + (surendra singh lilhore via Xiaoyu Yao) + + HDFS-7894. Rolling upgrade readiness is not updated in jmx until query + command is issued. (Brahma Reddy Battula via kihwal) + + HDFS-8254. Standby namenode doesn't process DELETED_BLOCK if the add block + request is in edit log. (Rushabh S Shah via kihwal) + + HDFS-8404. Pending block replication can get stuck using older genstamp + (Nathan Roberts via kihwal) + + HDFS-8431. hdfs crypto class not found in Windows. + (Anu Engineer via cnauroth) + + HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. + (Ming Ma via jing9) + HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (J.Andreina via vinayakumarb) + HDFS-7980. Incremental BlockReport will dramatically slow down namenode + startup. (Walter Su via szetszwo) + + HDFS-7446. HDFS inotify should have the ability to determine what txid it + has read up to (cmccabe) + + HDFS-8846. Add a unit test for INotify functionality across a layout + version upgrade (Zhe Zhang via Colin P. McCabe) + + HDFS-7470. SecondaryNameNode need twice memory when calling + reloadFromImageFile. (zhaoyunjiong via cnauroth) + + HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is + flawed. (Kihwal Lee via yliu) + + HDFS-7552. Change FsVolumeList toString() to fix + TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES diff --git a/hadoop-mapreduce-project/CHANGES.txt b/hadoop-mapreduce-project/CHANGES.txt index 02c1f1f28c5..8c647385ffa 100644 --- a/hadoop-mapreduce-project/CHANGES.txt +++ b/hadoop-mapreduce-project/CHANGES.txt @@ -448,9 +448,6 @@ Release 2.8.0 - UNRELEASED MAPREDUCE-6349. Fix typo in property org.apache.hadoop.mapreduce. lib.chain.Chain.REDUCER_INPUT_VALUE_CLASS. (Ray Chiang via ozawa) - MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge - (Gera Shegalov via jlowe) - MAPREDUCE-6165. [JDK8] TestCombineFileInputFormat failed on JDK8. (Akira AJISAKA via ozawa) @@ -605,14 +602,6 @@ Release 2.7.1 - 2015-07-06 BUG FIXES - MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) - - MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options - which is a regression from MR1 (zxu via rkanter) - - MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM - tokens when they roll-over. (Jason Lowe via vinodkv) - MAPREDUCE-6252. JobHistoryServer should not fail when encountering a missing directory. (Craig Welch via devaraj) @@ -629,10 +618,6 @@ Release 2.7.1 - 2015-07-06 that they don't fail on history-server backed by DFSes with not so strong guarantees. (Craig Welch via vinodkv) - MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between - copySucceeded() in one thread and copyFailed() in another thread on the - same host. (Junping Du via ozawa) - MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at the end. (Arun Suresh via kasha) @@ -714,9 +699,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-6248. Exposed the internal MapReduce job's information as a public API in DistCp. (Jing Zhao via vinodkv) - MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own - class. (Chris Trezzo via kasha) - MAPREDUCE-6263. Configurable timeout between YARNRunner terminate the application and forcefully kill. (Eric Payne via junping_du) @@ -766,9 +748,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output directory. (gera) - MAPREDUCE-6166. Reducers do not validate checksum of map outputs when - fetching directly to disk. (Eric Payne via gera) - MAPREDUCE-6045. need close the DataInputStream after open it in TestMapReduce.java (zxu via rkanter) @@ -778,9 +757,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-3283. mapred classpath CLI does not display the complete classpath (Varun Saxena via cnauroth) - MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken - service name properly. (Jason Lowe via jianhe) - MAPREDUCE-6231. Grep example job is not working on a fully-distributed cluster. (aajisaka) @@ -831,9 +807,6 @@ Release 2.7.0 - 2015-04-20 MAPREDUCE-6285. ClientServiceDelegate should not retry upon AuthenticationException. (Jonathan Eagles via ozawa) - MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal - to a reducer. (Jason Lowe via junping_du) - Release 2.6.2 - UNRELEASED INCOMPATIBLE CHANGES @@ -846,7 +819,7 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES @@ -861,6 +834,33 @@ Release 2.6.1 - UNRELEASED MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of reusing conections. (Kannan Rajah via ozawa) + MAPREDUCE-6166. Reducers do not validate checksum of map outputs when + fetching directly to disk. (Eric Payne via gera) + + MAPREDUCE-6230. Fixed RMContainerAllocator to update the new AMRMToken + service name properly. (Jason Lowe via jianhe) + + MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal + to a reducer. (Jason Lowe via junping_du) + + MAPREDUCE-6300. Task list sort by task id broken. (Siqi Li via aajisaka) + + MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own + class. (Chris Trezzo via kasha) + + MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options + which is a regression from MR1 (zxu via rkanter) + + MAPREDUCE-6324. Fixed MapReduce uber jobs to not fail the udpate of AM-RM + tokens when they roll-over. (Jason Lowe via vinodkv) + + MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge + (Gera Shegalov via jlowe) + + MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between + copySucceeded() in one thread and copyFailed() in another thread on the + same host. (Junping Du via ozawa) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 4815be3e9d0..5a706a344a5 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -338,9 +338,6 @@ Release 2.8.0 - UNRELEASED YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI. (Naganarasimha G R via wangda) - YARN-3700. Made generic history service load a number of latest applications - according to the parameter or the configuration. (Xuan Gong via zjshen) - YARN-3722. Merge multiple TestWebAppUtils into o.a.h.yarn.webapp.util.TestWebAppUtils. (Masatake Iwasaki via devaraj) @@ -498,9 +495,6 @@ Release 2.8.0 - UNRELEASED YARN-3459. Fix failiure of TestLog4jWarningErrorMetricsAppender. (Varun Vasudev via wangda) - YARN-2890. MiniYarnCluster should turn on timeline service if - configured to do so. (Mit Desai via hitesh) - YARN-3266. RMContext#inactiveNodes should have NodeId as map key. (Chengbing Liu via jianhe) @@ -797,9 +791,6 @@ Release 2.8.0 - UNRELEASED YARN-4005. Completed container whose app is finished is possibly not removed from NMStateStore. (Jun Gong via jianhe) - YARN-4047. ClientRMService getApplications has high scheduler lock contention. - (Jason Lowe via jianhe) - YARN-3987. Send AM container completed msg to NM once AM finishes. (sandflee via jianhe) @@ -844,9 +835,6 @@ Release 2.7.2 - UNRELEASED YARN-3967. Fetch the application report from the AHS if the RM does not know about it. (Mit Desai via xgong) - YARN-3978. Configurably turn off the saving of container info in Generic AHS - (Eric Payne via jeagles) - YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. (Xuan Gong via jianhe) @@ -880,11 +868,6 @@ Release 2.7.2 - UNRELEASED YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. (zhihai xu via jlowe) - YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when - Node is connected/disconnected (Bibin A Chundatt via jlowe) - - YARN-3999. RM hangs on draing events. (Jian He via xgong) - YARN-3857: Memory leak in ResourceManager with SIMPLE mode. (mujunchao via zxu) @@ -933,9 +916,6 @@ Release 2.7.1 - 2015-07-06 BUG FIXES - YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when - calling getQueue (Jason Lowe via wangda) - YARN-3497. ContainerManagementProtocolProxy modifies IPC timeout conf without making a copy. (Jason Lowe via jianhe) @@ -959,9 +939,6 @@ Release 2.7.1 - 2015-07-06 YARN-3516. killing ContainerLocalizer action doesn't take effect when private localizer receives FETCH_FAILURE status.(zhihai xu via xgong) - YARN-3464. Race condition in LocalizerRunner kills localizer before - localizing all resources. (Zhihai Xu via kasha) - YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha) @@ -986,9 +963,6 @@ Release 2.7.1 - 2015-07-06 YARN-3434. Interaction between reservations and userlimit can result in significant ULF violation (tgraves) - YARN-3493. RM fails to come up with error "Failed to load/recover state" - when mem settings are changed. (Jian He via wangda) - YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. (Craig Welch via xgong) @@ -998,20 +972,11 @@ Release 2.7.1 - 2015-07-06 YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked (Brahma Reddy Battula via jlowe) - YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when - exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) - - YARN-3526. ApplicationMaster tracking URL is incorrectly redirected - on a QJM cluster. (Weiwei Yang via xgong) - YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. (Weiwei Yang via xgong) YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. (Vinod Kumar Vavilapalli via ozawa) - YARN-2918. RM should not fail on startup if queue's configured labels do - not exist in cluster-node-labels. (Wangda Tan via jianhe) - YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. (Craig Welch and Varun Saxena via xgong) @@ -1036,18 +1001,6 @@ Release 2.7.1 - 2015-07-06 YARN-3686. CapacityScheduler should trim default_node_label_expression. (Sunil G via wangda) - YARN-2900. Application (Attempt and Container) Not Found in AHS results - in Internal Server Error (500). (Zhijie Shen and Mit Desai via xgong) - - YARN-3725. App submission via REST API is broken in secure mode due to - Timeline DT service address is empty. (Zhijie Shen via wangda) - - YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM - recovery is enabled (Rohith Sharmaks via jlowe) - - YARN-3733. Fix DominantRC#compare() does not work as expected if - cluster resource is empty. (Rohith Sharmaks via wangda) - YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent to another. (Wangda Tan via jianhe) @@ -1060,13 +1013,6 @@ Release 2.7.1 - 2015-07-06 YARN-3809. Failed to launch new attempts because ApplicationMasterLauncher's threads all hang (Jun Gong via jlowe) - YARN-3832. Resource Localization fails on a cluster due to existing cache - directories (Brahma Reddy Battula via jlowe) - - YARN-3850. NM fails to read files from full disks which can lead to - container logs being lost and other issues (Varun Saxena via jlowe) - - Release 2.7.0 - 2015-04-20 INCOMPATIBLE CHANGES @@ -1116,9 +1062,6 @@ Release 2.7.0 - 2015-04-20 YARN-2994. Document work-preserving RM restart. (Jian He via ozawa) - YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. - (Ryu Kobayashi via ozawa) - YARN-2786. Created a yarn cluster CLI and seeded with one command for listing node-labels collection. (Wangda Tan via vinodkv) @@ -1189,8 +1132,6 @@ Release 2.7.0 - 2015-04-20 YARN-1156. Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values. (Tsuyoshi OZAWA via junping_du) - YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) - YARN-2056. Disable preemption at Queue level (Eric Payne via jlowe) YARN-2762. Fixed RMAdminCLI to trim and check node-label related arguments @@ -1244,12 +1185,6 @@ Release 2.7.0 - 2015-04-20 YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. (Wangda Tan via ozawa) - YARN-3024. LocalizerRunner should give DIE action when all resources are - localized. (Chengbing Liu via xgong) - - YARN-3092. Created a common ResourceUsage class to track labeled resource - usages in Capacity Scheduler. (Wangda Tan via jianhe) - YARN-3086. Make NodeManager memory configurable in MiniYARNCluster. (Robert Metzger via ozawa) @@ -1275,9 +1210,6 @@ Release 2.7.0 - 2015-04-20 YARN-3022. Expose Container resource information from NodeManager for monitoring (adhoot via ranter) - YARN-3098. Created common QueueCapacities class in Capacity Scheduler to - track capacities-by-labels of queues. (Wangda Tan via jianhe) - YARN-3075. NodeLabelsManager implementation to retrieve label to node mapping (Varun Saxena via wangda) @@ -1318,9 +1250,6 @@ Release 2.7.0 - 2015-04-20 YARN-2079. Recover NonAggregatingLogHandler state upon nodemanager restart. (Jason Lowe via junping_du) - YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track - capacities-by-label. (Wangda Tan via jianhe) - YARN-3158. Correct log messages in ResourceTrackerService. (Varun Saxena via xgong) @@ -1345,8 +1274,6 @@ Release 2.7.0 - 2015-04-20 YARN-2799. Cleanup TestLogAggregationService based on the change in YARN-90. (Zhihai Xu via junping_du) - YARN-3230. Clarify application states on the web UI. (Jian He via wangda) - YARN-3237. AppLogAggregatorImpl fails to log error cause. (Rushabh S Shah via xgong) @@ -1380,9 +1307,6 @@ Release 2.7.0 - 2015-04-20 YARN-3122. Metrics for container's actual CPU usage. (Anubhav Dhoot via kasha) - YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and - Xuan Gong via jianhe) - YARN-2190. Added CPU and memory limit options to the default container executor for Windows containers. (Chuan Liu via jianhe) @@ -1417,56 +1341,26 @@ Release 2.7.0 - 2015-04-20 YARN-2713. "RM Home" link in NM should point to one of the RMs in an HA setup. (kasha) - YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, - attempts and containers. (Robert Kanter via zjshen) - - YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state - on app recovery. (Rohith Sharmaks via jianhe) - YARN-2857. ConcurrentModificationException in ContainerLogAppender (Mohammad Kamrul Islam via jlowe) - YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu - via jlowe) - YARN-2432. RMStateStore should process the pending events before close. (Varun Saxena via jianhe) YARN-1703. Fixed ResourceManager web-proxy to close connections correctly. (Rohith Sharma via vinodkv) - YARN-2414. RM web UI: app page will crash if app is failed before any - attempt has been created (Wangda Tan via jlowe) - YARN-2870. Updated the command to run the timeline server in the document. (Masatake Iwasaki via zjshen) YARN-2878. Fix DockerContainerExecutor.apt.vm formatting. (Abin Shahab via jianhe) - YARN-2865. Fixed RM to always create a new RMContext when transtions from - StandBy to Active. (Rohith Sharmaks via jianhe) - YARN-2315. FairScheduler: Set current capacity in addition to capacity. (Zhihai Xu via kasha) - YARN-1984. LeveldbTimelineStore does not handle db exceptions properly - (Varun Saxena via jlowe) - YARN-2697. Remove useless RMAuthenticationHandler. (Haosong Huang via zjshen) - YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. - (Jason Lowe via jianhe) - - YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated - log file is corrupted (Varun Saxena via jlowe) - - YARN-2894. Fixed a bug regarding application view acl when RM fails over. - (Rohith Sharmaks via jianhe) - - YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute - any further apps. (Naganarasimha G R via kasha) - YARN-2461. Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YarnConfiguration. (rchiang via rkanter) @@ -1479,18 +1373,12 @@ Release 2.7.0 - 2015-04-20 YARN-2931. PublicLocalizer may fail until directory is initialized by LocalizeRunner. (Anubhav Dhoot via kasha) - YARN-2910. FSLeafQueue can throw ConcurrentModificationException. - (Wilfred Spiegelenburg via kasha) - YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. (Wangda Tan via jianhe) YARN-2924. Fixed RMAdminCLI to not convert node labels to lower case. (Wangda Tan via jianhe) - YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher - (Rohith Sharmaks via jianhe) - YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor. (devaraj) @@ -1508,15 +1396,9 @@ Release 2.7.0 - 2015-04-20 YARN-2944. InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance. (Chris Trezzo via kasha) - YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) - (Jian He via jlowe) - YARN-2675. containersKilled metrics is not updated when the container is killed during localization. (Zhihai Xu via kasha) - YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks - via jianhe) - YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) YARN-2977. Fixed intermittent TestNMClient failure. @@ -1524,12 +1406,6 @@ Release 2.7.0 - 2015-04-20 YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du) - YARN-2920. Changed CapacityScheduler to kill containers on nodes where - node labels are changed. (Wangda Tan via jianhe) - - YARN-2340. Fixed NPE when queue is stopped during RM restart. - (Rohith Sharmaks via jianhe) - YARN-2940. Fix new findbugs warnings in rest of the hadoop-yarn components. (Li Lu via junping_du) @@ -1541,9 +1417,6 @@ Release 2.7.0 - 2015-04-20 YARN-2988. Graph#save() may leak file descriptors. (Ted Yu via ozawa) - YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla - via jianhe) - YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. (Varun Saxena via zjshen) @@ -1553,15 +1426,9 @@ Release 2.7.0 - 2015-04-20 YARN-2991. Fixed DrainDispatcher to reuse the draining code path in AsyncDispatcher. (Rohith Sharmaks via zjshen) - YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. - (Rohith Sharmaks via ozawa) - YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. (Varun Saxena via zjshen) - YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via - jianhe) - YARN-2230. Fixed few configs description in yarn-default.xml. (Vijay Bhat via jianhe) @@ -1571,18 +1438,12 @@ Release 2.7.0 - 2015-04-20 YARN-2936. Changed YARNDelegationTokenIdentifier to set proto fields on getProto method. (Varun Saxena via jianhe) - YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed - container statuses on heartbeat. (Chengbing Liu via jianhe) - YARN-3014. Replaces labels on a host should update all NM's labels on that host. (Wangda Tan via jianhe) YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter) - YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler - when activating applications. (Craig Welch via jianhe) - YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. (Zhijie Shen via jianhe) @@ -1613,21 +1474,12 @@ Release 2.7.0 - 2015-04-20 YARN-3088. LinuxContainerExecutor.deleteAsUser can throw NPE if native executor returns an error (Eric Payne via jlowe) - YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService - might lead NM to crash. (Varun Saxena via jianhe) - - YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe - via jianhe) - YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda) YARN-3029. FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere. (Varun Saxena via ozawa) - YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage - to track used-resources-by-label.(Wangda Tan via jianhe) - YARN-3077. Fixed RM to create zk root path recursively. (Chun Chen via jianhe) YARN-3113. Release audit warning for Sorting icons.psd. (stevel via kihwal) @@ -1653,18 +1505,12 @@ Release 2.7.0 - 2015-04-20 YARN-1537. Fix race condition in TestLocalResourcesTrackerImpl.testLocalResourceCache. (xgong via acmurthy) - YARN-2694. Ensure only single node label specified in ResourceRequest. - (Wangda Tan via jianhe) - YARN-3089. LinuxContainerExecutor does not handle file arguments to deleteAsUser (Eric Payne via jlowe) YARN-3143. RM Apps REST API can return NPE or entries missing id and other fields (jlowe) - YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong - via jianhe) - YARN-2971. RM uses conf instead of token service address to renew timeline delegation tokens (jeagles) @@ -1674,10 +1520,6 @@ Release 2.7.0 - 2015-04-20 YARN-2809. Implement workaround for linux kernel panic when removing cgroup (Nathan Roberts via jlowe) - YARN-2246. Made the proxy tracking URL always be - http(s)://proxy addr:port/proxy/ to avoid duplicate sections. (Devaraj - K via zjshen) - YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Chengbing Liu via junping_du) @@ -1708,9 +1550,6 @@ Release 2.7.0 - 2015-04-20 YARN-2749. Fix some testcases from TestLogAggregationService fails in trunk. (Xuan Gong via junping_du) - YARN-3207. Secondary filter matches entites which do not have the key being - filtered for. (Zhijie Shen via xgong) - YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. (Wangda Tan via jianhe) @@ -1723,15 +1562,9 @@ Release 2.7.0 - 2015-04-20 YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node (Rohith via jlowe) - YARN-3238. Connection timeouts to nodemanagers are retried at - multiple levels (Jason Lowe via xgong) - YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. (Zhihai Xu via ozawa) - YARN-3239. WebAppProxy does not support a final tracking url which has - query fragments and params (Jian He via jlowe) - YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against all Schedulers even when using ParameterizedSchedulerTestBase. (Anubhav Dhoot via devaraj) @@ -1742,27 +1575,12 @@ Release 2.7.0 - 2015-04-20 YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. (Wangda Tan via vinodkv) - YARN-3222. Fixed RMNode to send scheduler events in sequential order when a - node reconnects. (Rohith Sharma K S via jianhe) - YARN-3131. YarnClientImpl should check FAILED and KILLED state in submitApplication (Chang Li via jlowe) - YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending - jobs. (Siqi Li via kasha) - - YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving - events for old client. (Zhihai Xu via kasha) - - YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired - (Zhijie Shen via xgong) - YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues (Eric Payne via jlowe) - YARN-3287. Made TimelineClient put methods do as the correct login context. - (Daryn Sharp and Jonathan Eagles via zjshen) - YARN-3300. Outstanding_resource_requests table should not be shown in AHS. (Xuan Gong via jianhe) @@ -1777,9 +1595,6 @@ Release 2.7.0 - 2015-04-20 YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. (Xuan Gong via vinodkv) - YARN-3267. Timelineserver applies the ACL rules after applying the limit on - the number of records (Chang Li via jeagles) - YARN-3171. Sort by Application id, AppAttempt and ContainerID doesn't work in ATS / RM web ui. (Naganarasimha G R via xgong) @@ -1792,18 +1607,12 @@ Release 2.7.0 - 2015-04-20 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. (Xuan Gong via jianhe) - YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. - (Brahma Reddy Battula via wangda) - YARN-3384. TestLogAggregationService.verifyContainerLogs fails after YARN-2777. (Naganarasimha G R via ozawa) YARN-3336. FileSystem memory leak in DelegationTokenRenewer. (Zhihai Xu via cnauroth) - YARN-3393. Getting application(s) goes wrong when app finishes before - starting the attempt. (Zhijie Shen via xgong) - YARN-2213. Change proxy-user cookie log in AmIpFilter to DEBUG. (Varun Saxena via xgong) @@ -1817,9 +1626,6 @@ Release 2.7.0 - 2015-04-20 YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column (Jason Lowe via wangda) - YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token - renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) - Release 2.6.2 - UNRELEASED INCOMPATIBLE CHANGES @@ -1832,18 +1638,211 @@ Release 2.6.2 - UNRELEASED BUG FIXES -Release 2.6.1 - UNRELEASED +Release 2.6.1 - 2015-09-09 INCOMPATIBLE CHANGES NEW FEATURES + YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. + (Ryu Kobayashi via ozawa) + IMPROVEMENTS + YARN-3230. Clarify application states on the web UI. (Jian He via wangda) + + YARN-1809. Synchronize RM and TimeLineServer Web-UIs. (Zhijie Shen and + Xuan Gong via jianhe) + + YARN-3092. Created a common ResourceUsage class to track labeled resource + usages in Capacity Scheduler. (Wangda Tan via jianhe) + + YARN-3098. Created common QueueCapacities class in Capacity Scheduler to + track capacities-by-labels of queues. (Wangda Tan via jianhe) + + YARN-2301. Improved yarn container command. (Naganarasimha G R via jianhe) + + YARN-3978. Configurably turn off the saving of container info in Generic AHS + (Eric Payne via jeagles) + OPTIMIZATIONS BUG FIXES + YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state + on app recovery. (Rohith Sharmaks via jianhe) + + YARN-2816. NM fail to start with NPE during container recovery (Zhihai Xu + via jlowe) + + YARN-2414. RM web UI: app page will crash if app is failed before any + attempt has been created (Wangda Tan via jlowe) + + YARN-2865. Fixed RM to always create a new RMContext when transtions from + StandBy to Active. (Rohith Sharmaks via jianhe) + + YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. + (Jason Lowe via jianhe) + + YARN-2905. AggregatedLogsBlock page can infinitely loop if the aggregated + log file is corrupted (Varun Saxena via jlowe) + + YARN-2890. MiniYARNCluster should start the timeline server based on the + configuration. (Mit Desai via zjshen) + + YARN-2894. Fixed a bug regarding application view acl when RM fails over. + (Rohith Sharmaks via jianhe) + + YARN-2874. Dead lock in "DelegationTokenRenewer" which blocks RM to execute + any further apps. (Naganarasimha G R via kasha) + + YARN-2910. FSLeafQueue can throw ConcurrentModificationException. + (Wilfred Spiegelenburg via kasha) + + YARN-2917. Fixed potential deadlock when system.exit is called in AsyncDispatcher + (Rohith Sharmaks via jianhe) + + YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie) + (Jian He via jlowe) + + YARN-1984. LeveldbTimelineStore does not handle db exceptions properly + (Varun Saxena via jlowe) + + YARN-2952. Fixed incorrect version check in StateStore. (Rohith Sharmaks + via jianhe) + + YARN-2340. Fixed NPE when queue is stopped during RM restart. + (Rohith Sharmaks via jianhe) + + YARN-2992. ZKRMStateStore crashes due to session expiry. (Karthik Kambatla + via jianhe) + + YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. + (Rohith Sharmaks via ozawa) + + YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed + container statuses on heartbeat. (Chengbing Liu via jianhe) + + YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService + might lead NM to crash. (Varun Saxena via jianhe) + + YARN-3103. AMRMClientImpl does not update AMRM token properly. (Jason Lowe + via jianhe) + + YARN-3094. Reset timer for liveness monitors after RM recovery. (Jun Gong + via jianhe) + + YARN-2246. Made the proxy tracking URL always be + http(s)://proxy addr:port/proxy/ to avoid duplicate sections. (Devaraj + K via zjshen) + + YARN-3207. Secondary filter matches entites which do not have the key being + filtered for. (Zhijie Shen via xgong) + + YARN-3238. Connection timeouts to nodemanagers are retried at + multiple levels (Jason Lowe via xgong) + + YARN-3239. WebAppProxy does not support a final tracking url which has + query fragments and params (Jian He via jlowe) + + YARN-3222. Fixed RMNode to send scheduler events in sequential order when a + node reconnects. (Rohith Sharma K S via jianhe) + + YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending + jobs. (Siqi Li via kasha) + + YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving + events for old client. (Zhihai Xu via kasha) + + YARN-3227. Timeline renew delegation token fails when RM user's TGT is expired + (Zhijie Shen via xgong) + + YARN-3287. Made TimelineClient put methods do as the correct login context. + (Daryn Sharp and Jonathan Eagles via zjshen) + + YARN-3267. Timelineserver applies the ACL rules after applying the limit on + the number of records (Chang Li via jeagles) + + YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. + (Brahma Reddy Battula via wangda) + + YARN-3393. Getting application(s) goes wrong when app finishes before + starting the attempt. (Zhijie Shen via xgong) + + YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token + renewal of applications part of a bigger workflow. (Daryn Sharp via vinodkv) + + YARN-3493. RM fails to come up with error "Failed to load/recover state" + when mem settings are changed. (Jian He via wangda) + + YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when + calling getQueue (Jason Lowe via wangda) + + YARN-3024. LocalizerRunner should give DIE action when all resources are + localized. (Chengbing Liu via xgong) + + YARN-3464. Race condition in LocalizerRunner kills localizer before + localizing all resources. (Zhihai Xu via kasha) + + YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when + exceptions happen in stopping NM's sub-services. (Junping Du via jlowe) + + YARN-3526. ApplicationMaster tracking URL is incorrectly redirected + on a QJM cluster. (Weiwei Yang via xgong) + + YARN-2766. Made ApplicationHistoryManager return a sorted list of apps, + attempts and containers. (Robert Kanter via zjshen) + + YARN-3700. Made generic history service load a number of latest applications + according to the parameter or the configuration. (Xuan Gong via zjshen) + + YARN-2900. Application (Attempt and Container) Not Found in AHS results + in InternalServer Error (500). (Zhijie Shen and Mit Desai via xgong) + + YARN-3725. App submission via REST API is broken in secure mode due to + Timeline DT service address is empty. (Zhijie Shen via wangda) + + YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM + recovery is enabled (Rohith Sharmaks via jlowe) + + YARN-3832. Resource Localization fails on a cluster due to existing cache + directories (Brahma Reddy Battula via jlowe) + + YARN-3850. NM fails to read files from full disks which can lead to + container logs being lost and other issues (Varun Saxena via jlowe) + + YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when + Node is connected/disconnected (Bibin A Chundatt via jlowe) + + YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler + when activating applications. (Craig Welch via jianhe) + + YARN-3733. Fix DominantRC#compare() does not work as expected if + cluster resource is empty. (Rohith Sharmaks via wangda) + + YARN-2920. Changed CapacityScheduler to kill containers on nodes where + node labels are changed. (Wangda Tan via jianhe) + + YARN-2978. Fixed potential NPE while getting queue info. (Varun Saxena via + jianhe) + + YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage + to track used-resources-by-label.(Wangda Tan via jianhe) + + YARN-2694. Ensure only single node label specified in ResourceRequest. + (Wangda Tan via jianhe) + + YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track + capacities-by-label. (Wangda Tan via jianhe) + + YARN-2918. RM should not fail on startup if queue's configured labels do + not exist in cluster-node-labels. (Wangda Tan via jianhe) + + YARN-3999. RM hangs on draing events. (Jian He via xgong) + + YARN-4047. ClientRMService getApplications has high scheduler lock contention. + (Jason Lowe via jianhe) + Release 2.6.0 - 2014-11-18 INCOMPATIBLE CHANGES