783 lines
41 KiB
HTML
783 lines
41 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<!--
|
|
| Generated by Apache Maven Doxia at 2023-02-24
|
|
| Rendered using Apache Maven Stylus Skin 1.5
|
|
-->
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<title>Apache Hadoop 3.4.0-SNAPSHOT – HDFS Rolling Upgrade</title>
|
|
<style type="text/css" media="all">
|
|
@import url("./css/maven-base.css");
|
|
@import url("./css/maven-theme.css");
|
|
@import url("./css/site.css");
|
|
</style>
|
|
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
|
|
<meta name="Date-Revision-yyyymmdd" content="20230224" />
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
</head>
|
|
<body class="composite">
|
|
<div id="banner">
|
|
<a href="http://hadoop.apache.org/" id="bannerLeft">
|
|
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
|
|
</a>
|
|
<a href="http://www.apache.org/" id="bannerRight">
|
|
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
|
|
</a>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="breadcrumbs">
|
|
|
|
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
|
|
|
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
|
|
|
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
|
|
|
|
| Last Published: 2023-02-24
|
|
| Version: 3.4.0-SNAPSHOT
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="leftColumn">
|
|
<div id="navcolumn">
|
|
|
|
<h5>General</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Common</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-kms/index.html">Hadoop KMS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/AsyncProfilerServlet.html">Async Profiler</a>
|
|
</li>
|
|
</ul>
|
|
<h5>HDFS</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN Service</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Hadoop Compatible File Systems</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-azure/index.html">Azure Blob Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-huaweicloud/cloud-storage/index.html">Huaweicloud OBS</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Auth</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/Examples.html">Examples</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/Configuration.html">Configuration</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/BuildingIt.html">Building</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Tools</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-distcp/DistCp.html">DistCp</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-federation-balance/HDFSFederationBalance.html">HDFS Federation Balance</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-gridmix/GridMix.html">GridMix</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-rumen/Rumen.html">Rumen</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Reference</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../api/index.html">Java API docs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Configuration</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-kms/kms-default.html">kms-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
|
|
</li>
|
|
</ul>
|
|
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
|
|
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
|
|
</a>
|
|
|
|
</div>
|
|
</div>
|
|
<div id="bodyColumn">
|
|
<div id="contentBox">
|
|
<!--
|
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
contributor license agreements. See the NOTICE file distributed with
|
|
this work for additional information regarding copyright ownership.
|
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
(the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License.
|
|
-->
|
|
<h1>HDFS Rolling Upgrade</h1>
|
|
<ul>
|
|
<li><a href="#Introduction">Introduction</a></li>
|
|
<li><a href="#Upgrade">Upgrade</a>
|
|
<ul>
|
|
<li><a href="#Upgrade_without_Downtime">Upgrade without Downtime</a>
|
|
<ul>
|
|
<li><a href="#Upgrading_Non-Federated_Clusters">Upgrading Non-Federated Clusters</a></li>
|
|
<li><a href="#Upgrading_Federated_Clusters">Upgrading Federated Clusters</a></li></ul></li>
|
|
<li><a href="#Upgrade_with_Downtime">Upgrade with Downtime</a>
|
|
<ul>
|
|
<li><a href="#Upgrading_Non-HA_Clusters">Upgrading Non-HA Clusters</a></li></ul></li></ul></li>
|
|
<li><a href="#Downgrade_and_Rollback">Downgrade and Rollback</a></li>
|
|
<li><a href="#Downgrade">Downgrade</a></li>
|
|
<li><a href="#Rollback">Rollback</a></li>
|
|
<li><a href="#Commands_and_Startup_Options_for_Rolling_Upgrade">Commands and Startup Options for Rolling Upgrade</a>
|
|
<ul>
|
|
<li><a href="#DFSAdmin_Commands">DFSAdmin Commands</a>
|
|
<ul>
|
|
<li><a href="#dfsadmin_-rollingUpgrade">dfsadmin -rollingUpgrade</a></li>
|
|
<li><a href="#dfsadmin_-getDatanodeInfo">dfsadmin -getDatanodeInfo</a></li>
|
|
<li><a href="#dfsadmin_-shutdownDatanode">dfsadmin -shutdownDatanode</a></li></ul></li>
|
|
<li><a href="#NameNode_Startup_Options">NameNode Startup Options</a>
|
|
<ul>
|
|
<li><a href="#namenode_-rollingUpgrade">namenode -rollingUpgrade</a></li></ul></li></ul></li></ul>
|
|
<section>
|
|
<h2><a name="Introduction"></a>Introduction</h2>
|
|
<p><i>HDFS rolling upgrade</i> allows upgrading individual HDFS daemons. For examples, the datanodes can be upgraded independent of the namenodes. A namenode can be upgraded independent of the other namenodes. The namenodes can be upgraded independent of datanodes and journal nodes.</p></section><section>
|
|
<h2><a name="Upgrade"></a>Upgrade</h2>
|
|
<p>In Hadoop v2, HDFS supports highly-available (HA) namenode services and wire compatibility. These two capabilities make it feasible to upgrade HDFS without incurring HDFS downtime. In order to upgrade a HDFS cluster without downtime, the cluster must be setup with HA.</p>
|
|
<p>If there is any new feature which is enabled in new software release, may not work with old software release after upgrade. In such cases upgrade should be done by following steps.</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Disable new feature.</li>
|
|
<li>Upgrade the cluster.</li>
|
|
<li>Enable the new feature.</li>
|
|
</ol>
|
|
<p>Note that rolling upgrade is supported only from Hadoop-2.4.0 onwards.</p><section>
|
|
<h3><a name="Upgrade_without_Downtime"></a>Upgrade without Downtime</h3>
|
|
<p>In an HA cluster, there are two or more <i>NameNodes (NNs)</i>, many <i>DataNodes (DNs)</i>, a few <i>JournalNodes (JNs)</i> and a few <i>ZooKeeperNodes (ZKNs)</i>. <i>JNs</i> is relatively stable and does not require upgrade when upgrading HDFS in most of the cases. In the rolling upgrade procedure described here, only <i>NNs</i> and <i>DNs</i> are considered but <i>JNs</i> and <i>ZKNs</i> are not. Upgrading <i>JNs</i> and <i>ZKNs</i> may incur cluster downtime.</p><section>
|
|
<h4><a name="Upgrading_Non-Federated_Clusters"></a>Upgrading Non-Federated Clusters</h4>
|
|
<p>Suppose there are two namenodes <i>NN1</i> and <i>NN2</i>, where <i>NN1</i> and <i>NN2</i> are respectively in active and standby states. The following are the steps for upgrading an HA cluster:</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Prepare Rolling Upgrade
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Run “<a href="#dfsadmin_-rollingUpgrade"><code>hdfs dfsadmin -rollingUpgrade prepare</code></a>” to create a fsimage for rollback.</li>
|
|
<li>Run “<a href="#dfsadmin_-rollingUpgrade"><code>hdfs dfsadmin -rollingUpgrade query</code></a>” to check the status of the rollback image. Wait and re-run the command until the “<code>Proceed with rolling upgrade</code>” message is shown.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Upgrade Active and Standby <i>NNs</i>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Shutdown and upgrade <i>NN2</i>.</li>
|
|
<li>Start <i>NN2</i> as standby with the “<a href="#namenode_-rollingUpgrade"><code>-rollingUpgrade started</code></a>” option.</li>
|
|
<li>Failover from <i>NN1</i> to <i>NN2</i> so that <i>NN2</i> becomes active and <i>NN1</i> becomes standby.</li>
|
|
<li>Shutdown and upgrade <i>NN1</i>.</li>
|
|
<li>Start <i>NN1</i> as standby with the “<a href="#namenode_-rollingUpgrade"><code>-rollingUpgrade started</code></a>” option.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Upgrade <i>DNs</i>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Choose a small subset of datanodes (e.g. all datanodes under a particular rack).
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Run “<a href="#dfsadmin_-shutdownDatanode"><code>hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade</code></a>” to shutdown one of the chosen datanodes.</li>
|
|
<li>Run “<a href="#dfsadmin_-getDatanodeInfo"><code>hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT></code></a>” to check and wait for the datanode to shutdown.</li>
|
|
<li>Upgrade and restart the datanode.</li>
|
|
<li>Perform the above steps for all the chosen datanodes in the subset in parallel.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Repeat the above steps until all datanodes in the cluster are upgraded.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Finalize Rolling Upgrade
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Run “<a href="#dfsadmin_-rollingUpgrade"><code>hdfs dfsadmin -rollingUpgrade finalize</code></a>” to finalize the rolling upgrade.</li>
|
|
</ol>
|
|
</li>
|
|
</ol></section><section>
|
|
<h4><a name="Upgrading_Federated_Clusters"></a>Upgrading Federated Clusters</h4>
|
|
<p>In a federated cluster, there are multiple namespaces and a pair of active and standby <i>NNs</i> for each namespace. The procedure for upgrading a federated cluster is similar to upgrading a non-federated cluster except that Step 1 and Step 4 are performed on each namespace and Step 2 is performed on each pair of active and standby <i>NNs</i>, i.e.</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Prepare Rolling Upgrade for Each Namespace</li>
|
|
<li>Upgrade Active and Standby <i>NN</i> pairs for Each Namespace</li>
|
|
<li>Upgrade <i>DNs</i></li>
|
|
<li>Finalize Rolling Upgrade for Each Namespace</li>
|
|
</ol></section></section><section>
|
|
<h3><a name="Upgrade_with_Downtime"></a>Upgrade with Downtime</h3>
|
|
<p>For non-HA clusters, it is impossible to upgrade HDFS without downtime since it requires restarting the namenodes. However, datanodes can still be upgraded in a rolling manner.</p><section>
|
|
<h4><a name="Upgrading_Non-HA_Clusters"></a>Upgrading Non-HA Clusters</h4>
|
|
<p>In a non-HA cluster, there are a <i>NameNode (NN)</i>, a <i>SecondaryNameNode (SNN)</i> and many <i>DataNodes (DNs)</i>. The procedure for upgrading a non-HA cluster is similar to upgrading an HA cluster except that Step 2 “Upgrade Active and Standby <i>NNs</i>” is changed to below:</p>
|
|
<ul>
|
|
|
|
<li>Upgrade <i>NN</i> and <i>SNN</i>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Shutdown <i>SNN</i></li>
|
|
<li>Shutdown and upgrade <i>NN</i>.</li>
|
|
<li>Start <i>NN</i> with the “<a href="#namenode_-rollingUpgrade"><code>-rollingUpgrade started</code></a>” option.</li>
|
|
<li>Upgrade and restart <i>SNN</i></li>
|
|
</ol>
|
|
</li>
|
|
</ul></section></section></section><section>
|
|
<h2><a name="Downgrade_and_Rollback"></a>Downgrade and Rollback</h2>
|
|
<p>When the upgraded release is undesirable or, in some unlikely case, the upgrade fails (due to bugs in the newer release), administrators may choose to downgrade HDFS back to the pre-upgrade release, or rollback HDFS to the pre-upgrade release and the pre-upgrade state.</p>
|
|
<p>Note that downgrade can be done in a rolling fashion but rollback cannot. Rollback requires cluster downtime.</p>
|
|
<p>Note also that downgrade and rollback are possible only after a rolling upgrade is started and before the upgrade is terminated. An upgrade can be terminated by either finalize, downgrade or rollback. Therefore, it may not be possible to perform rollback after finalize or downgrade, or to perform downgrade after finalize.</p></section><section>
|
|
<h2><a name="Downgrade"></a>Downgrade</h2>
|
|
<p><i>Downgrade</i> restores the software back to the pre-upgrade release and preserves the user data. Suppose time <i>T</i> is the rolling upgrade start time and the upgrade is terminated by downgrade. Then, the files created before or after <i>T</i> remain available in HDFS. The files deleted before or after <i>T</i> remain deleted in HDFS.</p>
|
|
<p>A newer release is downgradable to the pre-upgrade release only if both the namenode layout version and the datanode layout version are not changed between these two releases.</p>
|
|
<p>In an HA cluster, when a rolling upgrade from an old software release to a new software release is in progress, it is possible to downgrade, in a rolling fashion, the upgraded machines back to the old software release. Same as before, suppose <i>NN1</i> and <i>NN2</i> are respectively in active and standby states. Below are the steps for rolling downgrade without downtime:</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Downgrade <i>DNs</i>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Choose a small subset of datanodes (e.g. all datanodes under a particular rack).
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Run “<a href="#dfsadmin_-shutdownDatanode"><code>hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade</code></a>” to shutdown one of the chosen datanodes.</li>
|
|
<li>Run “<a href="#dfsadmin_-getDatanodeInfo"><code>hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT></code></a>” to check and wait for the datanode to shutdown.</li>
|
|
<li>Downgrade and restart the datanode.</li>
|
|
<li>Perform the above steps for all the chosen datanodes in the subset in parallel.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Repeat the above steps until all upgraded datanodes in the cluster are downgraded.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Downgrade Active and Standby <i>NNs</i>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Shutdown and downgrade <i>NN2</i>.</li>
|
|
<li>Start <i>NN2</i> as standby normally.</li>
|
|
<li>Failover from <i>NN1</i> to <i>NN2</i> so that <i>NN2</i> becomes active and <i>NN1</i> becomes standby.</li>
|
|
<li>Shutdown and downgrade <i>NN1</i>.</li>
|
|
<li>Start <i>NN1</i> as standby normally.</li>
|
|
</ol>
|
|
</li>
|
|
<li>Finalize Rolling Downgrade
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Run “<a href="#dfsadmin_-rollingUpgrade"><code>hdfs dfsadmin -rollingUpgrade finalize</code></a>” to finalize the rolling downgrade.</li>
|
|
</ol>
|
|
</li>
|
|
</ol>
|
|
<p>Note that the datanodes must be downgraded before downgrading the namenodes since protocols may be changed in a backward compatible manner but not forward compatible, i.e. old datanodes can talk to the new namenodes but not vice versa.</p></section><section>
|
|
<h2><a name="Rollback"></a>Rollback</h2>
|
|
<p><i>Rollback</i> restores the software back to the pre-upgrade release but also reverts the user data back to the pre-upgrade state. Suppose time <i>T</i> is the rolling upgrade start time and the upgrade is terminated by rollback. The files created before <i>T</i> remain available in HDFS but the files created after <i>T</i> become unavailable. The files deleted before <i>T</i> remain deleted in HDFS but the files deleted after <i>T</i> are restored.</p>
|
|
<p>Rollback from a newer release to the pre-upgrade release is always supported. However, it cannot be done in a rolling fashion. It requires cluster downtime. Suppose <i>NN1</i> and <i>NN2</i> are respectively in active and standby states. Below are the steps for rollback:</p>
|
|
<ul>
|
|
|
|
<li>Rollback HDFS
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>Shutdown all <i>NNs</i> and <i>DNs</i>.</li>
|
|
<li>Restore the pre-upgrade release in all machines.</li>
|
|
<li>Start <i>NN1</i> as Active with the “<a href="#namenode_-rollingUpgrade"><code>-rollingUpgrade rollback</code></a>” option.</li>
|
|
<li>Run `-bootstrapStandby’ on NN2 and start it normally as standby.</li>
|
|
<li>Start <i>DNs</i> with the “<code>-rollback</code>” option.</li>
|
|
</ol>
|
|
</li>
|
|
</ul></section><section>
|
|
<h2><a name="Commands_and_Startup_Options_for_Rolling_Upgrade"></a>Commands and Startup Options for Rolling Upgrade</h2><section>
|
|
<h3><a name="DFSAdmin_Commands"></a>DFSAdmin Commands</h3><section>
|
|
<h4><a name="dfsadmin_-rollingUpgrade"></a><code>dfsadmin -rollingUpgrade</code></h4>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>hdfs dfsadmin -rollingUpgrade <query|prepare|finalize>
|
|
</pre></div></div>
|
|
|
|
<p>Execute a rolling upgrade action.</p>
|
|
<ul>
|
|
|
|
<li>Options:
|
|
<table border="0" class="bodyTable">
|
|
<thead></thead><tbody>
|
|
|
|
<tr class="a">
|
|
<td> <code>query</code> </td>
|
|
<td> Query the current rolling upgrade status. </td></tr>
|
|
<tr class="b">
|
|
<td> <code>prepare</code> </td>
|
|
<td> Prepare a new rolling upgrade. </td></tr>
|
|
<tr class="a">
|
|
<td> <code>finalize</code> </td>
|
|
<td> Finalize the current rolling upgrade. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
</li>
|
|
</ul></section><section>
|
|
<h4><a name="dfsadmin_-getDatanodeInfo"></a><code>dfsadmin -getDatanodeInfo</code></h4>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT>
|
|
</pre></div></div>
|
|
|
|
<p>Get the information about the given datanode. This command can be used for checking if a datanode is alive like the Unix <code>ping</code> command.</p></section><section>
|
|
<h4><a name="dfsadmin_-shutdownDatanode"></a><code>dfsadmin -shutdownDatanode</code></h4>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> [upgrade]
|
|
</pre></div></div>
|
|
|
|
<p>Submit a shutdown request for the given datanode. If the optional <code>upgrade</code> argument is specified, clients accessing the datanode will be advised to wait for it to restart and the fast start-up mode will be enabled. When the restart does not happen in time, clients will timeout and ignore the datanode. In such case, the fast start-up mode will also be disabled.</p>
|
|
<p>Note that the command does not wait for the datanode shutdown to complete. The “<a href="#dfsadmin_-getDatanodeInfo"><code>dfsadmin -getDatanodeInfo</code></a>” command can be used for checking if the datanode shutdown is completed.</p></section></section><section>
|
|
<h3><a name="NameNode_Startup_Options"></a>NameNode Startup Options</h3><section>
|
|
<h4><a name="namenode_-rollingUpgrade"></a><code>namenode -rollingUpgrade</code></h4>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>hdfs namenode -rollingUpgrade <rollback|started>
|
|
</pre></div></div>
|
|
|
|
<p>When a rolling upgrade is in progress, the <code>-rollingUpgrade</code> namenode startup option is used to specify various rolling upgrade options.</p>
|
|
<ul>
|
|
|
|
<li>Options:
|
|
<table border="0" class="bodyTable">
|
|
<thead></thead><tbody>
|
|
|
|
<tr class="a">
|
|
<td> <code>rollback</code> </td>
|
|
<td> Restores the namenode back to the pre-upgrade release but also reverts the user data back to the pre-upgrade state. </td></tr>
|
|
<tr class="b">
|
|
<td> <code>started</code> </td>
|
|
<td> Specifies a rolling upgrade already started so that the namenode should allow image directories with different layout versions during startup. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
</li>
|
|
</ul>
|
|
<p><b>WARN: downgrade options is obsolete.</b> It is not necessary to start namenode with downgrade options explicitly.</p></section></section></section>
|
|
</div>
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
<div id="footer">
|
|
<div class="xright">
|
|
© 2008-2023
|
|
Apache Software Foundation
|
|
|
|
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
|
|
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|