From fe1f4c64d130235f846988476d82c7bdd7ed0ade Mon Sep 17 00:00:00 2001 From: Jing Zhao Date: Thu, 30 Oct 2014 10:47:05 -0700 Subject: [PATCH] HDFS-7230. Add rolling downgrade documentation. Contributed by Tsz Wo Nicholas Sze. Conflicts: hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt --- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 2 + .../src/site/xdoc/HdfsRollingUpgrade.xml | 77 ++++++++++++++++--- 2 files changed, 67 insertions(+), 12 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 8cd9f695163..ef95121e7f4 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -415,6 +415,8 @@ Release 2.6.0 - UNRELEASED HDFS-6988. Improve HDFS-6581 eviction configuration (Xiaoyu Yao via Colin P. McCabe) + HDFS-7230. Add rolling downgrade documentation. (szetszwo via jing9) + OPTIMIZATIONS HDFS-6690. Deduplicate xattr names in memory. (wang) diff --git a/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml b/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml index 61d7d067f91..1053695b4f5 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml +++ b/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml @@ -152,17 +152,21 @@ or, in some unlikely case, the upgrade fails (due to bugs in the newer release), administrators may choose to downgrade HDFS back to the pre-upgrade release, or rollback HDFS to the pre-upgrade release and the pre-upgrade state. - Both downgrade and rollback require cluster downtime and are not done in a rolling fashion.

- Note that downgrade and rollback are possible only after a rolling upgrade is started and + Note that downgrade can be done in a rolling fashion but rollback cannot. + Rollback requires cluster downtime. +

+

+ Note also that downgrade and rollback are possible only after a rolling upgrade is started and before the upgrade is terminated. An upgrade can be terminated by either finalize, downgrade or rollback. Therefore, it may not be possible to perform rollback after finalize or downgrade, or to perform downgrade after finalize.

+ - +

Downgrade restores the software back to the pre-upgrade release and preserves the user data. @@ -174,21 +178,70 @@ A newer release is downgradable to the pre-upgrade release only if both the namenode layout version and the datenode layout version are not changed between these two releases. - Below are the steps for downgrade:

-
    -
  • Downgrade HDFS
      + + +

      + In a HA cluster, + when a rolling upgrade from an old software release to a new software release is in progress, + it is possible to downgrade, in a rolling fashion, the upgraded machines back to the old software release. + Same as before, suppose NN1 and NN2 are respectively in active and standby states. + Below are the steps for rolling downgrade: +

      +
        +
      1. Downgrade DNs
          +
        1. Choose a small subset of datanodes (e.g. all datanodes under a particular rack).
          2. +
            +
          1. Run "hdfs dfsadmin -shutdownDatanode <DATANODE_HOST:IPC_PORT> upgrade" + to shutdown one of the chosen datanodes.
            2. +
          2. Run "hdfs dfsadmin -getDatanodeInfo <DATANODE_HOST:IPC_PORT>" + to check and wait for the datanode to shutdown.
            3. +
          3. Downgrade and restart the datanode.
            4. +
          4. Perform the above steps for all the chosen datanodes in the subset in parallel.
            5. +
          +
        2. Repeat the above steps until all upgraded datanodes in the cluster are downgraded.
          3. +
        2. +
      2. Downgrade Active and Standby NNs
          +
        1. Shutdown and downgrade NN2.
          2. +
        2. Start NN2 as standby normally. (Note that it is incorrect to use the + "-rollingUpgrade downgrade" + option here.) +
          3. +
        3. Failover from NN1 to NN2 + so that NN2 becomes active and NN1 becomes standby.
          4. +
        4. Shutdown and upgrade NN1.
          5. +
        5. Start NN1 as standby normally. (Note that it is incorrect to use the + "-rollingUpgrade downgrade" + option here.) +
          6. +
        3. +
      3. Finalize Rolling Downgrade
        4. +
      +

      + Note that the datanodes must be downgraded before downgrading the namenodes + since protocols may be changed in a backward compatible manner but not forward compatible, + i.e. old datanodes can talk to the new namenodes but not vice versa. +

      +       + +

      + Administrator may choose to first shutdown the cluster and then downgrade it. + The following are the steps: +

      +
      1. Shutdown all NNs and DNs.
      2. Restore the pre-upgrade release in all machines.
      3. Start NNs with the "-rollingUpgrade downgrade" option.
      4. Start DNs normally.
        5. -
      -
- + +
- +

Rollback restores the software back to the pre-upgrade release but also reverts the user data back to the pre-upgrade state. @@ -198,6 +251,7 @@

Rollback from a newer release to the pre-upgrade release is always supported. + However, it cannot be done in a rolling fashion. It requires cluster downtime. Below are the steps for rollback:

    @@ -210,7 +264,6 @@
-
@@ -248,7 +301,7 @@

Note that the command does not wait for the datanode shutdown to complete. The "dfsadmin -getDatanodeInfo" - command can be used for checking if the datanode shutdown is complete. + command can be used for checking if the datanode shutdown is completed.