951 lines
63 KiB
HTML
951 lines
63 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<!--
|
|
| Generated by Apache Maven Doxia at 2023-03-08
|
|
| Rendered using Apache Maven Stylus Skin 1.5
|
|
-->
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<title>Apache Hadoop 3.4.0-SNAPSHOT – Apache Hadoop 2.9.0 Release Notes</title>
|
|
<style type="text/css" media="all">
|
|
@import url("../../css/maven-base.css");
|
|
@import url("../../css/maven-theme.css");
|
|
@import url("../../css/site.css");
|
|
</style>
|
|
<link rel="stylesheet" href="../../css/print.css" type="text/css" media="print" />
|
|
<meta name="Date-Revision-yyyymmdd" content="20230308" />
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
</head>
|
|
<body class="composite">
|
|
<div id="banner">
|
|
<a href="http://hadoop.apache.org/" id="bannerLeft">
|
|
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
|
|
</a>
|
|
<a href="http://www.apache.org/" id="bannerRight">
|
|
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
|
|
</a>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="breadcrumbs">
|
|
|
|
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
|
|
|
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
|
|
|
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
|
|
|
|
| Last Published: 2023-03-08
|
|
| Version: 3.4.0-SNAPSHOT
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="leftColumn">
|
|
<div id="navcolumn">
|
|
|
|
<h5>General</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Common</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-kms/index.html">Hadoop KMS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/AsyncProfilerServlet.html">Async Profiler</a>
|
|
</li>
|
|
</ul>
|
|
<h5>HDFS</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN Service</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Hadoop Compatible File Systems</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-azure/index.html">Azure Blob Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-huaweicloud/cloud-storage/index.html">Huaweicloud OBS</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Auth</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-auth/index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-auth/Examples.html">Examples</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-auth/Configuration.html">Configuration</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-auth/BuildingIt.html">Building</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Tools</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-distcp/DistCp.html">DistCp</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-federation-balance/HDFSFederationBalance.html">HDFS Federation Balance</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-gridmix/GridMix.html">GridMix</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-rumen/Rumen.html">Rumen</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Reference</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../api/index.html">Java API docs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Configuration</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-kms/kms-default.html">kms-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
|
|
</li>
|
|
</ul>
|
|
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
|
|
<img alt="Built by Maven" src="../../images/logos/maven-feather.png"/>
|
|
</a>
|
|
|
|
</div>
|
|
</div>
|
|
<div id="bodyColumn">
|
|
<div id="contentBox">
|
|
<!---
|
|
# Licensed to the Apache Software Foundation (ASF) under one
|
|
# or more contributor license agreements. See the NOTICE file
|
|
# distributed with this work for additional information
|
|
# regarding copyright ownership. The ASF licenses this file
|
|
# to you under the Apache License, Version 2.0 (the
|
|
# "License"); you may not use this file except in compliance
|
|
# with the License. You may obtain a copy of the License at
|
|
#
|
|
# http://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
-->
|
|
<h1>Apache Hadoop 2.9.0 Release Notes</h1>
|
|
<p>These release notes cover new developer and user-facing incompatibilities, important issues, features, and major improvements.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-9624">HDFS-9624</a> | <i>Major</i> | <b>DataNode start slowly due to the initial DU command operations</b></li>
|
|
</ul>
|
|
<p>Make it configurable how long the cached du file is valid. Useful for rolling upgrade.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-9525">HDFS-9525</a> | <i>Blocker</i> | <b>hadoop utilities need to support provided delegation tokens</b></li>
|
|
</ul>
|
|
<p>If hadoop.token.files property is defined and configured to one or more comma-delimited delegation token files, Hadoop will use those token files to connect to the services as named in the token.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-4762">YARN-4762</a> | <i>Blocker</i> | <b>NMs failing on DelegatingLinuxContainerRuntime init with LCE on</b></li>
|
|
</ul>
|
|
<p>Fixed CgroupHandler’s creation and usage to avoid NodeManagers crashing when LinuxContainerExecutor is enabled.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-1477">HDFS-1477</a> | <i>Major</i> | <b>Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart</b></li>
|
|
</ul>
|
|
<p>Steps to reconfigure: 1. change value of the parameter in corresponding xml configuration file 2. to reconfigure, run hdfs dfsadmin -reconfig namenode <nn_addr>:<ipc_port> start 3. to check status of the most recent reconfigure operation, run hdfs dfsadmin -reconfig namenode <nn_addr>:<ipc_port> status 4. to query a list reconfigurable properties on NN, run hdfs dfsadmin -reconfig namenode <nn_addr>:<ipc_port> properties</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-4732">YARN-4732</a> | <i>Trivial</i> | <b>*ProcessTree classes have too many whitespace issues</b></li>
|
|
</ul><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-9349">HDFS-9349</a> | <i>Major</i> | <b>Support reconfiguring fs.protected.directories without NN restart</b></li>
|
|
</ul>
|
|
<p>Steps to reconfigure: 1. change value of the parameter in corresponding xml configuration file 2. to reconfigure, run hdfs dfsadmin -reconfig namenode <nn_addr>:<ipc_port> start 3. to check status of the most recent reconfigure operation, run hdfs dfsadmin -reconfig namenode <nn_addr>:<ipc_port> status 4. to query a list reconfigurable properties on NN, run hdfs dfsadmin -reconfig namenode <nn_addr>:<ipc_port> properties</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-4784">YARN-4784</a> | <i>Major</i> | <b>Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO</b></li>
|
|
</ul>
|
|
<p>Clusters cannot use FIFO policy as the defaultQueueSchedulingPolicy. Clusters with a single level of queues will have to explicitly set the policy to FIFO if that is desired.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-10694">HADOOP-10694</a> | <i>Major</i> | <b>Remove synchronized input streams from Writable deserialization</b></li>
|
|
</ul>
|
|
<p>Remove invisible synchronization primitives from DataInputBuffer</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12782">HADOOP-12782</a> | <i>Major</i> | <b>Faster LDAP group name resolution with ActiveDirectory</b></li>
|
|
</ul>
|
|
<p>If the user object returned by LDAP server has the user’s group object DN (supported by Active Directory), Hadoop can reduce LDAP group mapping latency by setting hadoop.security.group.mapping.ldap.search.attr.memberof to memberOf.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10328">HDFS-10328</a> | <i>Minor</i> | <b>Add per-cache-pool default replication num configuration</b></li>
|
|
</ul>
|
|
<p>Add per-cache-pool default replication num configuration</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2928">YARN-2928</a> | <i>Critical</i> | <b>YARN Timeline Service v.2: alpha 1</b></li>
|
|
</ul>
|
|
<p>We are introducing an early preview (alpha 1) of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.</p>
|
|
<p>YARN Timeline Service v.2 alpha 1 is provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. It should be used only in a test capacity. Most importantly, security is not enabled. Do not set up or use Timeline Service v.2 until security is implemented if security is a critical requirement.</p>
|
|
<p>More details are available in the <a href="./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">YARN Timeline Service v.2</a> documentation.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13354">HADOOP-13354</a> | <i>Major</i> | <b>Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients</b></li>
|
|
</ul>
|
|
<p>The WASB FileSystem now uses version 4.2.0 of the Azure Storage SDK.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13403">HADOOP-13403</a> | <i>Major</i> | <b>AzureNativeFileSystem rename/delete performance improvements</b></li>
|
|
</ul>
|
|
<p>WASB has added an optional capability to execute certain FileSystem operations in parallel on multiple threads for improved performance. Please refer to the Azure Blob Storage documentation page for more information on how to enable and control the feature.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12747">HADOOP-12747</a> | <i>Major</i> | <b>support wildcard in libjars argument</b></li>
|
|
</ul>
|
|
<p>It is now possible to specify multiple jar files for the libjars argument using a wildcard. For example, you can specify “-libjars ‘libs/*’” as a shorthand for all jars in the libs directory.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5137">YARN-5137</a> | <i>Major</i> | <b>Make DiskChecker pluggable in NodeManager</b></li>
|
|
</ul>
|
|
<p>Added new plugin property yarn.nodemanager.disk-validator to allow the NodeManager to use an alternate class for checking whether a disk is good or not.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-8312">HDFS-8312</a> | <i>Critical</i> | <b>Trash does not descent into child directories to check for permissions</b></li>
|
|
</ul>
|
|
<p>Permissions are now checked when moving a file to Trash.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13588">HADOOP-13588</a> | <i>Major</i> | <b>ConfServlet should respect Accept request header</b></li>
|
|
</ul>
|
|
<p>Conf HTTP service should set response’s content type according to the Accept header in the request.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6776">MAPREDUCE-6776</a> | <i>Major</i> | <b>yarn.app.mapreduce.client.job.max-retries should have a more useful default</b></li>
|
|
</ul>
|
|
<p>The default value of yarn.app.mapreduce.client.job.max-retries has been changed from 0 to 3. This will help protect clients from failures that are transient. True failures may take slightly longer now due to the retries.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13522">HADOOP-13522</a> | <i>Major</i> | <b>Add %A and %a formats for fs -stat command to print permissions</b></li>
|
|
</ul>
|
|
<p>Added permissions to the fs stat command. They are now available as symbolic (%A) and octal (%a) formats, which are in line with Linux.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5388">YARN-5388</a> | <i>Critical</i> | <b>Deprecate and remove DockerContainerExecutor</b></li>
|
|
</ul>
|
|
<p>DockerContainerExecutor is deprecated starting 2.9.0 and removed from 3.0.0. Please use LinuxContainerExecutor with the DockerRuntime to run Docker containers on YARN clusters.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10756">HDFS-10756</a> | <i>Major</i> | <b>Expose getTrashRoot to HTTPFS and WebHDFS</b></li>
|
|
</ul>
|
|
<p>“getTrashRoot” returns a trash root for a path. Currently in DFS if the path “/foo” is a normal path, it returns “/user/$USER/.Trash” for “/foo” and if “/foo” is an encrypted zone, it returns “/foo/.Trash/$USER” for the child file/dir of “/foo”. This patch is about to override the old “getTrashRoot” of httpfs and webhdfs, so that the behavior of returning trash root in httpfs and webhdfs are consistent with DFS.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-12705">HADOOP-12705</a> | <i>Major</i> | <b>Upgrade Jackson 2.2.3 to 2.7.8</b></li>
|
|
</ul>
|
|
<p>We are sorry for causing pain for everyone for whom this Jackson update causes problems, but it was proving impossible to stay on the older version: too much code had moved past it, and by staying back we were limiting what Hadoop could do, and giving everyone who wanted an up to date version of Jackson a different set of problems. We’ve selected Jackson 2.7.8 as it fixed fix a security issue in XML parsing, yet proved compatible at the API level with the Hadoop codebase –and hopefully everything downstream.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13050">HADOOP-13050</a> | <i>Blocker</i> | <b>Upgrade to AWS SDK 1.11.45</b></li>
|
|
</ul>
|
|
<p>The dependency on the AWS SDK has been bumped to 1.11.45.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13953">HADOOP-13953</a> | <i>Major</i> | <b>Make FTPFileSystem’s data connection mode and transfer mode configurable</b></li>
|
|
</ul>
|
|
<p>Added two configuration key fs.ftp.data.connection.mode and fs.ftp.transfer.mode, and configure FTP data connection mode and transfer mode accordingly.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14003">HADOOP-14003</a> | <i>Major</i> | <b>Make additional KMS tomcat settings configurable</b></li>
|
|
</ul><!-- markdown -->
|
|
|
|
<p>The KMS can now be configured with the additional environment variables <code>KMS_PROTOCOL</code>, <code>KMS_ACCEPT_COUNT</code>, and <code>KMS_ACCEPTOR_THREAD_COUNT</code>. See <code>kms-env.sh</code> for more information about these variables.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-6404">MAPREDUCE-6404</a> | <i>Major</i> | <b>Allow AM to specify a port range for starting its webapp</b></li>
|
|
</ul>
|
|
<p>Add a new configuration - “yarn.app.mapreduce.am.webapp.port-range” to specify port-range for webapp launched by AM.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13119">HADOOP-13119</a> | <i>Major</i> | <b>Add ability to secure log servlet using proxy users</b></li>
|
|
</ul>
|
|
<p><b>WARNING: No release note provided for this change.</b></p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13075">HADOOP-13075</a> | <i>Major</i> | <b>Add support for SSE-KMS and SSE-C in s3a filesystem</b></li>
|
|
</ul>
|
|
<p>The new encryption options SSE-KMS and especially SSE-C must be considered experimental at present. If you are using SSE-C, problems may arise if the bucket mixes encrypted and unencrypted files. For SSE-KMS, there may be extra throttling of IO, especially with the fadvise=random option. You may wish to request an increase in your KMS IOPs limits.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6177">YARN-6177</a> | <i>Major</i> | <b>Yarn client should exit with an informative error message if an incompatible Jersey library is used at client</b></li>
|
|
</ul>
|
|
<p>Let yarn client exit with an informative error message if an incompatible Jersey library is used from client side.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13817">HADOOP-13817</a> | <i>Minor</i> | <b>Add a finite shell command timeout to ShellBasedUnixGroupsMapping</b></li>
|
|
</ul>
|
|
<p>A new introduced configuration key “hadoop.security.groups.shell.command.timeout” allows applying a finite wait timeout over the ‘id’ commands launched by the ShellBasedUnixGroupsMapping plugin. Values specified can be in any valid time duration units: <a class="externalLink" href="https://hadoop.apache.org/docs/current/api/org/apache/hadoop/conf/Configuration.html#getTimeDuration-java.lang.String-long-java.util.concurrent.TimeUnit-">https://hadoop.apache.org/docs/current/api/org/apache/hadoop/conf/Configuration.html#getTimeDuration-java.lang.String-long-java.util.concurrent.TimeUnit-</a></p>
|
|
<p>Value defaults to 0, indicating infinite wait (preserving existing behaviour).</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-6801">HADOOP-6801</a> | <i>Minor</i> | <b>io.sort.mb and io.sort.factor were renamed and moved to mapreduce but are still in CommonConfigurationKeysPublic.java and used in SequenceFile.java</b></li>
|
|
</ul>
|
|
<p>Two new configuration keys, seq.io.sort.mb and seq.io.sort.factor have been introduced for the SequenceFile’s Sorter feature to replace older, deprecated property keys of io.sort.mb and io.sort.factor.</p>
|
|
<p>This only affects direct users of the org.apache.hadoop.io.SequenceFile.Sorter Java class. For controlling MR2’s internal sorting instead, use the existing config keys of mapreduce.task.io.sort.mb and mapreduce.task.io.sort.factor.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11499">HDFS-11499</a> | <i>Major</i> | <b>Decommissioning stuck because of failing recovery</b></li>
|
|
</ul>
|
|
<p>Allow a block to complete if the number of replicas on live nodes, decommissioning nodes and nodes in maintenance mode satisfies minimum replication factor. The fix prevents block recovery failure if replica of last block is being decommissioned. Vice versa, the decommissioning will be stuck, waiting for the last block to be completed. In addition, file close() operation will not fail due to last block being decommissioned.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14213">HADOOP-14213</a> | <i>Major</i> | <b>Move Configuration runtime check for hadoop-site.xml to initialization</b></li>
|
|
</ul>
|
|
<p>Move the check for hadoop-site.xml to static initialization of the Configuration class.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14038">HADOOP-14038</a> | <i>Minor</i> | <b>Rename ADLS credential properties</b></li>
|
|
</ul><!-- markdown -->
|
|
|
|
<ul>
|
|
|
|
<li>Properties {{dfs.adls.<i>}} are renamed {{fs.adl.</i>}}</li>
|
|
<li>Property {{adl.dfs.enable.client.latency.tracker}} is renamed {{adl.enable.client.latency.tracker}}</li>
|
|
<li>Old properties are still supported</li>
|
|
</ul><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14174">HADOOP-14174</a> | <i>Major</i> | <b>Set default ADLS access token provider type to ClientCredential</b></li>
|
|
</ul>
|
|
<p>Switch the default ADLS access token provider type from Custom to ClientCredential.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14301">HADOOP-14301</a> | <i>Major</i> | <b>Deprecate SharedInstanceProfileCredentialsProvider in branch-2.</b></li>
|
|
</ul>
|
|
<p>SharedInstanceProfileCredentialsProvider has been deprecated. Users should use InstanceProfileCredentialsProvider provided by AWS SDK instead, which itself enforces a singleton instance to reduce calls to AWS EC2 Instance Metadata Service. SharedInstanceProfileCredentialsProvider will be removed permanently in a future release.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-11794">HADOOP-11794</a> | <i>Major</i> | <b>Enable distcp to copy blocks in parallel</b></li>
|
|
</ul>
|
|
<p>If a positive value is passed to command line switch -blocksperchunk, files with more blocks than this value will be split into chunks of `<blocksperchunk>` blocks to be transferred in parallel, and reassembled on the destination. By default, `<blocksperchunk>` is 0 and the files will be transmitted in their entirety without splitting. This switch is only applicable when both the source file system supports getBlockLocations and target supports concat.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11402">HDFS-11402</a> | <i>Major</i> | <b>HDFS Snapshots should capture point-in-time copies of OPEN files</b></li>
|
|
</ul>
|
|
<p>When the config param “dfs.namenode.snapshot.capture.openfiles” is enabled, HDFS snapshots taken will additionally capture point-in-time copies of the open files that have valid leases. Even when the current version open files grow or shrink in size, the snapshot will always retain the immutable versions of these open files, just as in for all other closed files. Note: The file length captured for open files in the snapshot was the one recorded in NameNode at the time of snapshot and it may be shorter than what the client has written till then. In order to capture the latest length, the client can call hflush/hsync with the flag SyncFlag.UPDATE_LENGTH on the open files handles.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2962">YARN-2962</a> | <i>Critical</i> | <b>ZKRMStateStore: Limit the number of znodes under a znode</b></li>
|
|
</ul>
|
|
<p><b>WARNING: No release note provided for this change.</b></p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-9016">HDFS-9016</a> | <i>Major</i> | <b>Display upgrade domain information in fsck</b></li>
|
|
</ul>
|
|
<p>New fsck option “-upgradedomains” has been added to display upgrade domains of any block.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14419">HADOOP-14419</a> | <i>Minor</i> | <b>Remove findbugs report from docs profile</b></li>
|
|
</ul>
|
|
<p>Findbugs report is no longer part of the documentation.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11661">HDFS-11661</a> | <i>Blocker</i> | <b>GetContentSummary uses excessive amounts of memory</b></li>
|
|
</ul>
|
|
<p>Reverted HDFS-10797 to fix a scalability regression brought by the commit.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14407">HADOOP-14407</a> | <i>Major</i> | <b>DistCp - Introduce a configurable copy buffer size</b></li>
|
|
</ul>
|
|
<p>The copy buffer size can be configured via the new parameter <copybuffersize>. By default the <copybuffersize> is set to 8KB.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6127">YARN-6127</a> | <i>Major</i> | <b>Add support for work preserving NM restart when AMRMProxy is enabled</b></li>
|
|
</ul>
|
|
<p>This breaks rolling upgrades because it changes the major version of the NM state store schema. Therefore when a new NM comes up on an old state store it crashes.</p>
|
|
<p>The state store versions for this change have been updated in YARN-6798.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14536">HADOOP-14536</a> | <i>Major</i> | <b>Update azure-storage sdk to version 5.3.0</b></li>
|
|
</ul>
|
|
<p>The WASB FileSystem now uses version 5.3.0 of the Azure Storage SDK.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14546">HADOOP-14546</a> | <i>Major</i> | <b>Azure: Concurrent I/O does not work when secure.mode is enabled</b></li>
|
|
</ul>
|
|
<p>Fix to <a class="externalLink" href="wasb://">wasb://</a> (Azure) file system that allows the concurrent I/O feature to be used with the secure mode feature.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14535">HADOOP-14535</a> | <i>Major</i> | <b>wasb: implement high-performance random access and seek of block blobs</b></li>
|
|
</ul>
|
|
<p>Random access and seek improvements for the <a class="externalLink" href="wasb://">wasb://</a> (Azure) file system.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6798">YARN-6798</a> | <i>Major</i> | <b>Fix NM startup failure with old state store due to version mismatch</b></li>
|
|
</ul><!-- markdown -->
|
|
|
|
<p>This fixes the LevelDB state store for the NodeManager. As of this patch, the state store versions now correspond to the following table.</p>
|
|
<ul>
|
|
|
|
<li>Previous Patch: YARN-5049</li>
|
|
<li>LevelDB Key: queued</li>
|
|
<li>Hadoop Versions: 2.9.0, 3.0.0-alpha1</li>
|
|
<li>Corresponding LevelDB Version: 1.2</li>
|
|
<li>Previous Patch: YARN-6127</li>
|
|
<li>LevelDB Key: AMRMProxy/NextMasterKey</li>
|
|
<li>Hadoop Versions: 2.9.0, 3.0.0-alpha4</li>
|
|
<li>Corresponding LevelDB Version: 1.1</li>
|
|
</ul><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14539">HADOOP-14539</a> | <i>Major</i> | <b>Move commons logging APIs over to slf4j in hadoop-common</b></li>
|
|
</ul>
|
|
<p>In Hadoop common, fatal log level is changed to error because slf4j API does not support fatal log level.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14518">HADOOP-14518</a> | <i>Minor</i> | <b>Customize User-Agent header sent in HTTP/HTTPS requests by WASB.</b></li>
|
|
</ul>
|
|
<p>WASB now includes the current Apache Hadoop version in the User-Agent string passed to Azure Blob service. Users also may include optional additional information to identify their application. See the documentation of configuration property fs.wasb.user.agent.id for further details.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14722">HADOOP-14722</a> | <i>Major</i> | <b>Azure: BlockBlobInputStream position incorrect after seek</b></li>
|
|
</ul>
|
|
<p>Bug fix to Azure Filesystem related to HADOOP-14535.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14680">HADOOP-14680</a> | <i>Minor</i> | <b>Azure: IndexOutOfBoundsException in BlockBlobInputStream</b></li>
|
|
</ul>
|
|
<p>Bug fix to Azure Filesystem related to HADOOP-14535</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10326">HDFS-10326</a> | <i>Major</i> | <b>Disable setting tcp socket send/receive buffers for write pipelines</b></li>
|
|
</ul>
|
|
<p>The size of the TCP socket buffers are no longer hardcoded by default. Instead the OS now will automatically tune the size for the buffer.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14260">HADOOP-14260</a> | <i>Major</i> | <b>Configuration.dumpConfiguration should redact sensitive information</b></li>
|
|
</ul><!-- markdown -->
|
|
Configuration.dumpConfiguration no longer prints out the clear text values for the sensitive keys listed in `hadoop.security.sensitive-config-keys`. Callers can override the default list of sensitive keys either to redact more keys or print the clear text values for a few extra keys for debugging purpose.
|
|
<hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14660">HADOOP-14660</a> | <i>Major</i> | <b>wasb: improve throughput by 34% when account limit exceeded</b></li>
|
|
</ul>
|
|
<p>Up to 34% throughput improvement for the <a class="externalLink" href="wasb://">wasb://</a> (Azure) file system when fs.azure.selfthrottling.enable is false fs.azure.autothrottling.enable is true.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14769">HADOOP-14769</a> | <i>Major</i> | <b>WASB: delete recursive should not fail if a file is deleted</b></li>
|
|
</ul>
|
|
<p>Recursive directory delete improvement for the wasb filesystem.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5355">YARN-5355</a> | <i>Critical</i> | <b>YARN Timeline Service v.2: alpha 2</b></li>
|
|
</ul>
|
|
<p>We are releasing the alpha2 version of a major revision of YARN Timeline Service: v.2. YARN Timeline Service v.2 addresses two major challenges: improving scalability and reliability of Timeline Service, and enhancing usability by introducing flows and aggregation.</p>
|
|
<p>YARN Timeline Service v.2 alpha1 was introduced in 3.0.0-alpha1 via YARN-2928.</p>
|
|
<p>YARN Timeline Service v.2 alpha2 is now being provided so that users and developers can test it and provide feedback and suggestions for making it a ready replacement for Timeline Service v.1.x. Security is provided via Kerberos Authentication and delegation tokens. There is also a simple read level authorization provided via whitelists.</p>
|
|
<p>Some of the notable improvements since alpha-1 are: - Security via Kerberos Authentication and delegation tokens - Read side simple authorization via whitelist - Client configurable entity sort ordering - New REST APIs for apps, app attempts, containers, fetching metrics by timerange, pagination, sub-app entities - Support for storing sub-application entities (entities that exist outside the scope of an application) - Configurable TTLs (time-to-live) for tables, configurable table prefixes, configurable hbase cluster - Flow level aggregations done as dynamic (table level) coprocessors - Uses latest stable HBase release 1.2.6</p>
|
|
<p>More details are available in the <a href="./hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">YARN Timeline Service v.2</a> documentation.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-13345">HADOOP-13345</a> | <i>Major</i> | <b>S3Guard: Improved Consistency for S3A</b></li>
|
|
</ul>
|
|
<p>S3Guard (pronounced see-guard) is a new feature for the S3A connector to Amazon S3, which uses DynamoDB for a high performance and consistent metadata repository. Essentially: S3Guard caches directory information, so your S3A clients get faster lookups and resilience to inconsistency between S3 list operations and the status of objects. When files are created, with S3Guard, they’ll always be found.</p>
|
|
<p>S3Guard does not address update consistency: if a file is updated, while the directory information will be updated, calling open() on the path may still return the old data. Similarly, deleted objects may also potentially be opened.</p>
|
|
<p>Please consult the S3Guard documentation in the Amazon S3 section of our documentation.</p>
|
|
<p>Note: part of this update includes moving to a new version of the AWS SDK 1.11, one which includes the Dynamo DB client and its a shaded version of Jackson 2. The large aws-sdk-bundle JAR is needed to use the S3A client with or without S3Guard enabled. The good news: because Jackson is shaded, there will be no conflict between any Jackson version used in your application and that which the AWS SDK needs.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14520">HADOOP-14520</a> | <i>Major</i> | <b>WASB: Block compaction for Azure Block Blobs</b></li>
|
|
</ul>
|
|
<p>Block Compaction for Azure Block Blobs. When the number of blocks in a block blob is above 32000, the process of compaction replaces a sequence of small blocks with with one big block.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-11799">HDFS-11799</a> | <i>Major</i> | <b>Introduce a config to allow setting up write pipeline with fewer nodes than replication factor</b></li>
|
|
</ul>
|
|
<p>Added new configuration “dfs.client.block.write.replace-datanode-on-failure.min-replication”.</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>The minimum number of replications that are needed to not to fail
|
|
the write pipeline if new datanodes can not be found to replace
|
|
failed datanodes (could be due to network failure) in the write pipeline.
|
|
If the number of the remaining datanodes in the write pipeline is greater
|
|
than or equal to this property value, continue writing to the remaining nodes.
|
|
Otherwise throw exception.
|
|
|
|
If this is set to 0, an exception will be thrown, when a replacement
|
|
can not be found.
|
|
</pre></div></div>
|
|
<hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2915">YARN-2915</a> | <i>Major</i> | <b>Enable YARN RM scale out via federation using multiple RM’s</b></li>
|
|
</ul>
|
|
<p>A federation-based approach to transparently scale a single YARN cluster to tens of thousands of nodes, by federating multiple YARN standalone clusters (sub-clusters). The applications running in this federated environment will see a single massive YARN cluster and will be able to schedule tasks on any node of the federated cluster. Under the hood, the federation system will negotiate with sub-clusters ResourceManagers and provide resources to the application. The goal is to allow an individual job to “span” sub-clusters seamlessly.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-1492">YARN-1492</a> | <i>Major</i> | <b>truly shared cache for jars (jobjar/libjar)</b></li>
|
|
</ul>
|
|
<p>The YARN Shared Cache provides the facility to upload and manage shared application resources to HDFS in a safe and scalable manner. YARN applications can leverage resources uploaded by other applications or previous runs of the same application without having to re-­upload and localize identical files multiple times. This will save network resources and reduce YARN application startup time.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HDFS-10467">HDFS-10467</a> | <i>Major</i> | <b>Router-based HDFS federation</b></li>
|
|
</ul>
|
|
<p>HDFS Router-based Federation adds a RPC routing layer that provides a federated view of multiple HDFS namespaces. This is similar to the existing ViewFS and HDFS federation functionality, except the mount table is managed on the server-side by the routing layer rather than on the client. This simplifies access to a federated cluster for existing HDFS clients.</p>
|
|
<p>See HDFS-10467 and the HDFS Router-based Federation documentation for more details.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5734">YARN-5734</a> | <i>Major</i> | <b>OrgQueue for easy CapacityScheduler queue configuration management</b></li>
|
|
</ul><!-- markdown -->
|
|
|
|
<p>The OrgQueue extension to the capacity scheduler provides a programmatic way to change configurations by providing a REST API that users can call to modify queue configurations. This enables automation of queue configuration management by administrators in the queue’s <code>administer_queue</code> ACL.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/MAPREDUCE-5951">MAPREDUCE-5951</a> | <i>Major</i> | <b>Add support for the YARN Shared Cache</b></li>
|
|
</ul>
|
|
<p>MapReduce support for the YARN shared cache allows MapReduce jobs to take advantage of additional resource caching. This saves network bandwidth between the job submission client as well as within the YARN cluster itself. This will reduce job submission time and overall job runtime.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-6623">YARN-6623</a> | <i>Blocker</i> | <b>Add support to turn off launching privileged containers in the container-executor</b></li>
|
|
</ul>
|
|
<p>A change in configuration for launching Docker containers under YARN. Docker container capabilities, mounts, networks and allowing privileged container have to specified in the container-executor.cfg. By default, all of the above are turned off. This change will break existing setups launching Docker containers under YARN. Please refer to the Docker containers under YARN documentation for more information.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/HADOOP-14840">HADOOP-14840</a> | <i>Major</i> | <b>Tool to estimate resource requirements of an application pipeline based on prior executions</b></li>
|
|
</ul>
|
|
<p>The first version of Resource Estimator service, a tool that captures the historical resource usage of an app and predicts its future resource requirement.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-2877">YARN-2877</a> | <i>Major</i> | <b>Extend YARN to support distributed scheduling</b></li>
|
|
</ul>
|
|
<p>With this JIRA we are introducing distributed scheduling in YARN. In particular, we make the following contributions: - Introduce the notion of container types. GUARANTEED containers follow the semantics of the existing YARN containers. OPPORTUNISTIC ones can be seen as lower priority containers, and can be preempted in order to make space for GUARANTEED containers to run. - Queuing of tasks at the NMs. This enables us to send more containers in an NM than its available resources. At the moment we are allowing queuing of OPPORTUNISTIC containers. Once resources become available at the NM, such containers can immediately start their execution. - Introduce the AMRMProxy. This is a service running at each node, intercepting the requests between the AM and the RM. It is instrumental for both distributed scheduling and YARN Federation (YARN-2915). - Enable distributed scheduling. To minimize their allocation latency, OPPORTUNISTIC containers are dispatched immediately to NMs in a distributed fashion by using the AMRMProxy of the node where the corresponding AM resides, without needing to go through the ResourceManager.</p>
|
|
<p>All the functionality introduced in this JIRA is disabled by default, so it will not affect the behavior of existing applications. We have introduced parameters in YarnConfiguration to enable NM queuing (yarn.nodemanager.container-queuing-enabled), distributed scheduling (yarn.distributed-scheduling.enabled) and the AMRMProxy service (yarn.nodemanager.amrmproxy.enable). AMs currently need to specify the type of container to be requested for each task. We are in the process of adding in the MapReduce AM the ability to randomly request OPPORTUNISTIC containers for a specified percentage of a job’s tasks, so that users can experiment with the new features.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5220">YARN-5220</a> | <i>Major</i> | <b>Scheduling of OPPORTUNISTIC containers through YARN RM</b></li>
|
|
</ul>
|
|
<p>This extends the centralized YARN RM in to enable the scheduling of OPPORTUNISTIC containers in a centralized fashion. This way, users can use OPPORTUNISTIC containers to improve the cluster’s utilization, without needing to enable distributed scheduling.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5085">YARN-5085</a> | <i>Major</i> | <b>Add support for change of container ExecutionType</b></li>
|
|
</ul>
|
|
<p>This allows the Application Master to ask the Scheduler to change the ExecutionType of a running/allocated container.</p><hr />
|
|
<ul>
|
|
|
|
<li><a class="externalLink" href="https://issues.apache.org/jira/browse/YARN-5049">YARN-5049</a> | <i>Major</i> | <b>Extend NMStateStore to save queued container information</b></li>
|
|
</ul>
|
|
<p>This breaks rolling upgrades because it changes the major version of the NM state store schema. Therefore when a new NM comes up on an old state store it crashes.</p>
|
|
<p>The state store versions for this change have been updated in YARN-6798.</p>
|
|
</div>
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
<div id="footer">
|
|
<div class="xright">
|
|
© 2008-2023
|
|
Apache Software Foundation
|
|
|
|
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
|
|
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|