<ahref="http://maven.apache.org/"title="Built by Maven"class="poweredBy">
<imgalt="Built by Maven"src="../../images/logos/maven-feather.png"/>
</a>
</div>
</div>
<divid="bodyColumn">
<divid="contentBox">
<!---
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
-->
<h1>Apache Hadoop 2.7.0 Release Notes</h1>
<p>These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-6252">HDFS-6252</a> | <i>Minor</i> | <b>Phase out the old web UI in HDFS</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-1362">HDFS-1362</a> | <i>Major</i> | <b>Provide volume management functionality for DataNode</b></li>
</ul>
<p>Based on the reconfiguration framework provided by HADOOP-7001, enable reconfigure the dfs.datanode.data.dir and add new volumes into service.</p><hr/>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11311">HADOOP-11311</a> | <i>Major</i> | <b>Restrict uppercase key names from being created with JCEKS</b></li>
</ul>
<p>Keys with uppercase names can no longer be created when using the JavaKeyStoreProvider to resolve ambiguity about case-sensitivity in the KeyStore spec.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-7210">HDFS-7210</a> | <i>Major</i> | <b>Avoid two separate RPC’s namenode.append() and namenode.getFileInfo() for an append call from DFSClient</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/MAPREDUCE-5420">MAPREDUCE-5420</a> | <i>Major</i> | <b>Remove mapreduce.task.tmp.dir from mapred-default.xml</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-9629">HADOOP-9629</a> | <i>Major</i> | <b>Support Windows Azure Storage - Blob as a file system in Hadoop</b></li>
</ul>
<p>Hadoop now supports integration with Azure Storage as an alternative Hadoop Compatible File System.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11385">HADOOP-11385</a> | <i>Critical</i> | <b>Prevent cross site scripting attack on JMXJSONServlet</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11446">HADOOP-11446</a> | <i>Major</i> | <b>S3AOutputStream should use shared thread pool to avoid OutOfMemoryError</b></li>
</ul>
<p>The following parameters are introduced in this JIRA: fs.s3a.threads.max: the maximum number of threads to allow in the pool used by TransferManager fs.s3a.threads.core: the number of threads to keep in the pool used by TransferManager fs.s3a.threads.keepalivetime: when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating fs.s3a.max.total.tasks: the maximum number of tasks that the LinkedBlockingQueue can hold</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11464">HADOOP-11464</a> | <i>Major</i> | <b>Reinstate support for launching Hadoop processes on Windows using Cygwin.</b></li>
</ul>
<p>We have reinstated support for launching Hadoop processes on Windows by using Cygwin to run the shell scripts. All processes still must have access to the native components: hadoop.dll and winutils.exe.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-3689">HDFS-3689</a> | <i>Major</i> | <b>Add support for variable length block</b></li>
</ul>
<olstyle="list-style-type: decimal">
<li>HDFS now can choose to append data to a new block instead of end of the last partial block. Users can pass {{CreateFlag.APPEND}} and {{CreateFlag.NEW_BLOCK}} to the {{append}} API to indicate this requirement.</li>
<li>HDFS now allows users to pass {{SyncFlag.END_BLOCK}} to the {{hsync}} API to finish the current block and write remaining data to a new block.</li>
</ol><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11498">HADOOP-11498</a> | <i>Major</i> | <b>Bump the version of HTrace to 3.1.0-incubating</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-10181">HADOOP-10181</a> | <i>Minor</i> | <b>GangliaContext does not work with multicast ganglia setup</b></li>
</ul>
<p>Hadoop metrics sent to Ganglia over multicast now support optional configuration of socket TTL. The default TTL is 1, which preserves the behavior of prior Hadoop versions. Clusters that span multiple subnets/VLANs will likely want to increase this.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-6651">HDFS-6651</a> | <i>Critical</i> | <b>Deletion failure can leak inodes permanently</b></li>
</ul>
<p><b>WARNING: No release note provided for this change.</b></p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11492">HADOOP-11492</a> | <i>Major</i> | <b>Bump up curator version to 2.7.1</b></li>
</ul><!-- markdown -->
Apache Curator version change: Apache Hadoop has updated the version of Apache Curator used from 2.6.0 to 2.7.1. This change should be binary and source compatible for the majority of downstream users. Notable exceptions:
<ul>
<li>Binary incompatible change: org.apache.curator.utils.PathUtils.validatePath(String) changed return types. Downstream users of this method will need to recompile.</li>
<li>Source incompatible change: org.apache.curator.framework.recipes.shared.SharedCountReader added a method to its interface definition. Downstream users with custom implementations of this interface can continue without binary compatibility problems but will need to modify their source code to recompile.</li>
<li>Source incompatible change: org.apache.curator.framework.recipes.shared.SharedValueReader added a method to its interface definition. Downstream users with custom implementations of this interface can continue without binary compatibility problems but will need to modify their source code to recompile.</li>
</ul>
<p>Downstream users are reminded that while the Hadoop community will attempt to avoid egregious incompatible dependency changes, there is currently no policy around when Hadoop’s exposed dependencies will change across versions (ref <aclass="externalLink"href="http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#Java_Classpath">http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#Java_Classpath</a> ).</p><hr/>
<p>Introduced a new configuration dfs.pipeline.ecn. When the configuration is turned on, DataNodes will signal in the writing pipelines when they are overloaded. The client can back off based on this congestion signal to avoid overloading the system.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-6133">HDFS-6133</a> | <i>Major</i> | <b>Add a feature for replica pinning so that a pinned replica will not be moved by Balancer/Mover.</b></li>
</ul>
<p>Add a feature for replica pinning so that when a replica is pinned in a datanode, it will not be moved by Balancer/Mover. The replica pinning feature can be enabled/disabled by “dfs.datanode.block-pinning.enabled”, where the default is false.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-7584">HDFS-7584</a> | <i>Major</i> | <b>Enable Quota Support for Storage Types</b></li>
</ul>
<olstyle="list-style-type: decimal">
<li>Introduced quota by storage type as a hard limit on the amount of space usage allowed for different storage types (SSD, DISK, ARCHIVE) under the target directory.</li>
<li>Added {{SetQuotaByStorageType}} API and {{-storagetype}} option for {{hdfs dfsadmin -setSpaceQuota/-clrSpaceQuota}} commands to allow set/clear quota by storage type under the target directory.</li>
</ol><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-7806">HDFS-7806</a> | <i>Minor</i> | <b>Refactor: move StorageType from hadoop-hdfs to hadoop-common</b></li>
</ul>
<p>This fix moves the public class StorageType from the package org.apache.hadoop.hdfs to org.apache.hadoop.fs.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-3217">YARN-3217</a> | <i>Major</i> | <b>Remove httpclient dependency from hadoop-yarn-server-web-proxy</b></li>
</ul>
<p>Removed commons-httpclient dependency from hadoop-yarn-server-web-proxy module.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-9922">HADOOP-9922</a> | <i>Major</i> | <b>hadoop windows native build will fail in 32 bit machine</b></li>
</ul>
<p>The Hadoop Common native components now support 32-bit build targets on Windows.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-7774">HDFS-7774</a> | <i>Critical</i> | <b>Unresolved symbols error while compiling HDFS on Windows 7/32 bit</b></li>
</ul>
<p>LibHDFS now supports 32-bit build targets on Windows.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/MAPREDUCE-5583">MAPREDUCE-5583</a> | <i>Major</i> | <b>Ability to limit running map and reduce tasks</b></li>
</ul><!-- markdown -->
This introduces two new MR2 job configs, mentioned below, which allow users to control the maximum simultaneously-running tasks of the submitted job, across the cluster:
<ul>
<li>mapreduce.job.running.map.limit (default: 0, for no limit)</li>
<li>mapreduce.job.running.reduce.limit (default: 0, for no limit)</li>
</ul>
<p>This is controllable at a per-job level.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-1522">HDFS-1522</a> | <i>Major</i> | <b>Merge Block.BLOCK_FILE_PREFIX and DataStorage.BLOCK_FILE_PREFIX into one constant</b></li>
</ul>
<p>This merges Block.BLOCK_FILE_PREFIX and DataStorage.BLOCK_FILE_PREFIX into one constant. Hard-coded literals of “blk_” in various files are also updated to use the same constant.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HDFS-7411">HDFS-7411</a> | <i>Major</i> | <b>Refactor and improve decommissioning logic into DecommissionManager</b></li>
</ul>
<p>This change introduces a new configuration key used to throttle decommissioning work, “dfs.namenode.decommission.blocks.per.interval”. This new key overrides and deprecates the previous related configuration key “dfs.namenode.decommission.nodes.per.interval”. The new key is intended to result in more predictable pause times while scanning decommissioning nodes.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/YARN-3154">YARN-3154</a> | <i>Blocker</i> | <b>Should not upload partial logs for MR jobs or other "short-running’ applications</b></li>
</ul>
<p>Applications which made use of the LogAggregationContext in their application will need to revisit this code in order to make sure that their logs continue to get rolled out.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-9329">HADOOP-9329</a> | <i>Trivial</i> | <b>document native build dependencies in BUILDING.txt</b></li>
</ul>
<p>Added a section to BUILDING.txt on how to install required / optional packages on a clean install of Ubuntu 14.04 LTS Desktop.</p>
<p>Went through the CMakeLists.txt files in the repo and added the following optional library dependencies - Snappy, Bzip2, Linux FUSE and Jansson.</p>
<p>Updated the required packages / version numbers from the trunk branch version of BUILDING.txt.</p><hr/>
<ul>
<li><aclass="externalLink"href="https://issues.apache.org/jira/browse/HADOOP-11801">HADOOP-11801</a> | <i>Minor</i> | <b>Update BUILDING.txt for Ubuntu</b></li>
