HDFS-9884. Use doxia macro to generate in-page TOC of HDFS site documentation. (iwasakims)
This commit is contained in:
Masatake Iwasaki
parent
d20dc8691e
commit
528bff9c42
|
|
@ -15,19 +15,7 @@
|
||||||
Archival Storage, SSD & Memory
|
Archival Storage, SSD & Memory
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
* [Archival Storage, SSD & Memory](#Archival_Storage_SSD__Memory)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [Storage Types and Storage Policies](#Storage_Types_and_Storage_Policies)
|
|
||||||
* [Storage Types: ARCHIVE, DISK, SSD and RAM\_DISK](#Storage_Types:_ARCHIVE_DISK_SSD_and_RAM_DISK)
|
|
||||||
* [Storage Policies: Hot, Warm, Cold, All\_SSD, One\_SSD and Lazy\_Persist](#Storage_Policies:_Hot_Warm_Cold_All_SSD_One_SSD_and_Lazy_Persist)
|
|
||||||
* [Storage Policy Resolution](#Storage_Policy_Resolution)
|
|
||||||
* [Configuration](#Configuration)
|
|
||||||
* [Mover - A New Data Migration Tool](#Mover_-_A_New_Data_Migration_Tool)
|
|
||||||
* [Storage Policy Commands](#Storage_Policy_Commands)
|
|
||||||
* [List Storage Policies](#List_Storage_Policies)
|
|
||||||
* [Set Storage Policy](#Set_Storage_Policy)
|
|
||||||
* [Unset Storage Policy](#Unset_Storage_Policy)
|
|
||||||
* [Get Storage Policy](#Get_Storage_Policy)
|
|
||||||
|
|
||||||
Introduction
|
Introduction
|
||||||
------------
|
------------
|
||||||
|
|
|
||||||
|
|
@ -15,30 +15,7 @@
|
||||||
Centralized Cache Management in HDFS
|
Centralized Cache Management in HDFS
|
||||||
====================================
|
====================================
|
||||||
|
|
||||||
* [Overview](#Overview)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Use Cases](#Use_Cases)
|
|
||||||
* [Architecture](#Architecture)
|
|
||||||
* [Concepts](#Concepts)
|
|
||||||
* [Cache directive](#Cache_directive)
|
|
||||||
* [Cache pool](#Cache_pool)
|
|
||||||
* [cacheadmin command-line interface](#cacheadmin_command-line_interface)
|
|
||||||
* [Cache directive commands](#Cache_directive_commands)
|
|
||||||
* [addDirective](#addDirective)
|
|
||||||
* [removeDirective](#removeDirective)
|
|
||||||
* [removeDirectives](#removeDirectives)
|
|
||||||
* [listDirectives](#listDirectives)
|
|
||||||
* [Cache pool commands](#Cache_pool_commands)
|
|
||||||
* [addPool](#addPool)
|
|
||||||
* [modifyPool](#modifyPool)
|
|
||||||
* [removePool](#removePool)
|
|
||||||
* [listPools](#listPools)
|
|
||||||
* [help](#help)
|
|
||||||
* [Configuration](#Configuration)
|
|
||||||
* [Native Libraries](#Native_Libraries)
|
|
||||||
* [Configuration Properties](#Configuration_Properties)
|
|
||||||
* [Required](#Required)
|
|
||||||
* [Optional](#Optional)
|
|
||||||
* [OS Limits](#OS_Limits)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,13 +15,7 @@
|
||||||
Extended Attributes in HDFS
|
Extended Attributes in HDFS
|
||||||
===========================
|
===========================
|
||||||
|
|
||||||
* [Overview](#Overview)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [HDFS extended attributes](#HDFS_extended_attributes)
|
|
||||||
* [Namespaces and Permissions](#Namespaces_and_Permissions)
|
|
||||||
* [Interacting with extended attributes](#Interacting_with_extended_attributes)
|
|
||||||
* [getfattr](#getfattr)
|
|
||||||
* [setfattr](#setfattr)
|
|
||||||
* [Configuration options](#Configuration_options)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,20 +15,7 @@
|
||||||
HDFS Federation
|
HDFS Federation
|
||||||
===============
|
===============
|
||||||
|
|
||||||
* [HDFS Federation](#HDFS_Federation)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Background](#Background)
|
|
||||||
* [Multiple Namenodes/Namespaces](#Multiple_NamenodesNamespaces)
|
|
||||||
* [Key Benefits](#Key_Benefits)
|
|
||||||
* [Federation Configuration](#Federation_Configuration)
|
|
||||||
* [Configuration:](#Configuration:)
|
|
||||||
* [Formatting Namenodes](#Formatting_Namenodes)
|
|
||||||
* [Upgrading from an older release and configuring federation](#Upgrading_from_an_older_release_and_configuring_federation)
|
|
||||||
* [Adding a new Namenode to an existing HDFS cluster](#Adding_a_new_Namenode_to_an_existing_HDFS_cluster)
|
|
||||||
* [Managing the cluster](#Managing_the_cluster)
|
|
||||||
* [Starting and stopping cluster](#Starting_and_stopping_cluster)
|
|
||||||
* [Balancer](#Balancer)
|
|
||||||
* [Decommissioning](#Decommissioning)
|
|
||||||
* [Cluster Web Console](#Cluster_Web_Console)
|
|
||||||
|
|
||||||
This guide provides an overview of the HDFS Federation feature and how to configure and manage the federated cluster.
|
This guide provides an overview of the HDFS Federation feature and how to configure and manage the federated cluster.
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,40 +15,7 @@
|
||||||
HDFS Commands Guide
|
HDFS Commands Guide
|
||||||
===================
|
===================
|
||||||
|
|
||||||
* [Overview](#Overview)
|
<!-- MACRO{toc|fromDepth=0|toDepth=2} -->
|
||||||
* [User Commands](#User_Commands)
|
|
||||||
* [classpath](#classpath)
|
|
||||||
* [dfs](#dfs)
|
|
||||||
* [fetchdt](#fetchdt)
|
|
||||||
* [fsck](#fsck)
|
|
||||||
* [getconf](#getconf)
|
|
||||||
* [groups](#groups)
|
|
||||||
* [lsSnapshottableDir](#lsSnapshottableDir)
|
|
||||||
* [jmxget](#jmxget)
|
|
||||||
* [oev](#oev)
|
|
||||||
* [oiv](#oiv)
|
|
||||||
* [oiv\_legacy](#oiv_legacy)
|
|
||||||
* [snapshotDiff](#snapshotDiff)
|
|
||||||
* [version](#version)
|
|
||||||
* [Administration Commands](#Administration_Commands)
|
|
||||||
* [balancer](#balancer)
|
|
||||||
* [cacheadmin](#cacheadmin)
|
|
||||||
* [crypto](#crypto)
|
|
||||||
* [datanode](#datanode)
|
|
||||||
* [dfsadmin](#dfsadmin)
|
|
||||||
* [haadmin](#haadmin)
|
|
||||||
* [journalnode](#journalnode)
|
|
||||||
* [mover](#mover)
|
|
||||||
* [namenode](#namenode)
|
|
||||||
* [nfs3](#nfs3)
|
|
||||||
* [portmap](#portmap)
|
|
||||||
* [secondarynamenode](#secondarynamenode)
|
|
||||||
* [storagepolicies](#storagepolicies)
|
|
||||||
* [zkfc](#zkfc)
|
|
||||||
* [Debug Commands](#Debug_Commands)
|
|
||||||
* [verifyMeta](#verifyMeta)
|
|
||||||
* [computeMeta](#computeMeta)
|
|
||||||
* [recoverLease](#recoverLease)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,30 +15,7 @@
|
||||||
HDFS High Availability
|
HDFS High Availability
|
||||||
======================
|
======================
|
||||||
|
|
||||||
* [HDFS High Availability](#HDFS_High_Availability)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Purpose](#Purpose)
|
|
||||||
* [Note: Using the Quorum Journal Manager or Conventional Shared Storage](#Note:_Using_the_Quorum_Journal_Manager_or_Conventional_Shared_Storage)
|
|
||||||
* [Background](#Background)
|
|
||||||
* [Architecture](#Architecture)
|
|
||||||
* [Hardware resources](#Hardware_resources)
|
|
||||||
* [Deployment](#Deployment)
|
|
||||||
* [Configuration overview](#Configuration_overview)
|
|
||||||
* [Configuration details](#Configuration_details)
|
|
||||||
* [Deployment details](#Deployment_details)
|
|
||||||
* [Administrative commands](#Administrative_commands)
|
|
||||||
* [Automatic Failover](#Automatic_Failover)
|
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [Components](#Components)
|
|
||||||
* [Deploying ZooKeeper](#Deploying_ZooKeeper)
|
|
||||||
* [Before you begin](#Before_you_begin)
|
|
||||||
* [Configuring automatic failover](#Configuring_automatic_failover)
|
|
||||||
* [Initializing HA state in ZooKeeper](#Initializing_HA_state_in_ZooKeeper)
|
|
||||||
* [Starting the cluster with start-dfs.sh](#Starting_the_cluster_with_start-dfs.sh)
|
|
||||||
* [Starting the cluster manually](#Starting_the_cluster_manually)
|
|
||||||
* [Securing access to ZooKeeper](#Securing_access_to_ZooKeeper)
|
|
||||||
* [Verifying automatic failover](#Verifying_automatic_failover)
|
|
||||||
* [Automatic Failover FAQ](#Automatic_Failover_FAQ)
|
|
||||||
* [BookKeeper as a Shared storage (EXPERIMENTAL)](#BookKeeper_as_a_Shared_storage_EXPERIMENTAL)
|
|
||||||
|
|
||||||
Purpose
|
Purpose
|
||||||
-------
|
-------
|
||||||
|
|
|
||||||
|
|
@ -15,30 +15,7 @@
|
||||||
HDFS High Availability Using the Quorum Journal Manager
|
HDFS High Availability Using the Quorum Journal Manager
|
||||||
=======================================================
|
=======================================================
|
||||||
|
|
||||||
* [HDFS High Availability Using the Quorum Journal Manager](#HDFS_High_Availability_Using_the_Quorum_Journal_Manager)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Purpose](#Purpose)
|
|
||||||
* [Note: Using the Quorum Journal Manager or Conventional Shared Storage](#Note:_Using_the_Quorum_Journal_Manager_or_Conventional_Shared_Storage)
|
|
||||||
* [Background](#Background)
|
|
||||||
* [Architecture](#Architecture)
|
|
||||||
* [Hardware resources](#Hardware_resources)
|
|
||||||
* [Deployment](#Deployment)
|
|
||||||
* [Configuration overview](#Configuration_overview)
|
|
||||||
* [Configuration details](#Configuration_details)
|
|
||||||
* [Deployment details](#Deployment_details)
|
|
||||||
* [Administrative commands](#Administrative_commands)
|
|
||||||
* [Automatic Failover](#Automatic_Failover)
|
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [Components](#Components)
|
|
||||||
* [Deploying ZooKeeper](#Deploying_ZooKeeper)
|
|
||||||
* [Before you begin](#Before_you_begin)
|
|
||||||
* [Configuring automatic failover](#Configuring_automatic_failover)
|
|
||||||
* [Initializing HA state in ZooKeeper](#Initializing_HA_state_in_ZooKeeper)
|
|
||||||
* [Starting the cluster with start-dfs.sh](#Starting_the_cluster_with_start-dfs.sh)
|
|
||||||
* [Starting the cluster manually](#Starting_the_cluster_manually)
|
|
||||||
* [Securing access to ZooKeeper](#Securing_access_to_ZooKeeper)
|
|
||||||
* [Verifying automatic failover](#Verifying_automatic_failover)
|
|
||||||
* [Automatic Failover FAQ](#Automatic_Failover_FAQ)
|
|
||||||
* [HDFS Upgrade/Finalization/Rollback with HA Enabled](#HDFS_UpgradeFinalizationRollback_with_HA_Enabled)
|
|
||||||
|
|
||||||
Purpose
|
Purpose
|
||||||
-------
|
-------
|
||||||
|
|
|
||||||
|
|
@ -15,41 +15,7 @@
|
||||||
HDFS Architecture
|
HDFS Architecture
|
||||||
=================
|
=================
|
||||||
|
|
||||||
* [HDFS Architecture](#HDFS_Architecture)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [Assumptions and Goals](#Assumptions_and_Goals)
|
|
||||||
* [Hardware Failure](#Hardware_Failure)
|
|
||||||
* [Streaming Data Access](#Streaming_Data_Access)
|
|
||||||
* [Large Data Sets](#Large_Data_Sets)
|
|
||||||
* [Simple Coherency Model](#Simple_Coherency_Model)
|
|
||||||
* ["Moving Computation is Cheaper than Moving Data"](#aMoving_Computation_is_Cheaper_than_Moving_Data)
|
|
||||||
* [Portability Across Heterogeneous Hardware and Software Platforms](#Portability_Across_Heterogeneous_Hardware_and_Software_Platforms)
|
|
||||||
* [NameNode and DataNodes](#NameNode_and_DataNodes)
|
|
||||||
* [The File System Namespace](#The_File_System_Namespace)
|
|
||||||
* [Data Replication](#Data_Replication)
|
|
||||||
* [Replica Placement: The First Baby Steps](#Replica_Placement:_The_First_Baby_Steps)
|
|
||||||
* [Replica Selection](#Replica_Selection)
|
|
||||||
* [Safemode](#Safemode)
|
|
||||||
* [The Persistence of File System Metadata](#The_Persistence_of_File_System_Metadata)
|
|
||||||
* [The Communication Protocols](#The_Communication_Protocols)
|
|
||||||
* [Robustness](#Robustness)
|
|
||||||
* [Data Disk Failure, Heartbeats and Re-Replication](#Data_Disk_Failure_Heartbeats_and_Re-Replication)
|
|
||||||
* [Cluster Rebalancing](#Cluster_Rebalancing)
|
|
||||||
* [Data Integrity](#Data_Integrity)
|
|
||||||
* [Metadata Disk Failure](#Metadata_Disk_Failure)
|
|
||||||
* [Snapshots](#Snapshots)
|
|
||||||
* [Data Organization](#Data_Organization)
|
|
||||||
* [Data Blocks](#Data_Blocks)
|
|
||||||
* [Staging](#Staging)
|
|
||||||
* [Replication Pipelining](#Replication_Pipelining)
|
|
||||||
* [Accessibility](#Accessibility)
|
|
||||||
* [FS Shell](#FS_Shell)
|
|
||||||
* [DFSAdmin](#DFSAdmin)
|
|
||||||
* [Browser Interface](#Browser_Interface)
|
|
||||||
* [Space Reclamation](#Space_Reclamation)
|
|
||||||
* [File Deletes and Undeletes](#File_Deletes_and_Undeletes)
|
|
||||||
* [Decrease Replication Factor](#Decrease_Replication_Factor)
|
|
||||||
* [References](#References)
|
|
||||||
|
|
||||||
Introduction
|
Introduction
|
||||||
------------
|
------------
|
||||||
|
|
|
||||||
|
|
@ -15,10 +15,7 @@
|
||||||
Offline Edits Viewer Guide
|
Offline Edits Viewer Guide
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
* [Offline Edits Viewer Guide](#Offline_Edits_Viewer_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Usage](#Usage)
|
|
||||||
* [Case study: Hadoop cluster recovery](#Case_study:_Hadoop_cluster_recovery)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,16 +15,7 @@
|
||||||
Offline Image Viewer Guide
|
Offline Image Viewer Guide
|
||||||
==========================
|
==========================
|
||||||
|
|
||||||
* [Offline Image Viewer Guide](#Offline_Image_Viewer_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Usage](#Usage)
|
|
||||||
* [Web Processor](#Web_Processor)
|
|
||||||
* [XML Processor](#XML_Processor)
|
|
||||||
* [Options](#Options)
|
|
||||||
* [Analyzing Results](#Analyzing_Results)
|
|
||||||
* [oiv\_legacy Command](#oiv_legacy_Command)
|
|
||||||
* [Usage](#Usage)
|
|
||||||
* [Options](#Options)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -17,14 +17,7 @@ HDFS Support for Multihomed Networks
|
||||||
|
|
||||||
This document is targetted to cluster administrators deploying `HDFS` in multihomed networks. Similar support for `YARN`/`MapReduce` is work in progress and will be documented when available.
|
This document is targetted to cluster administrators deploying `HDFS` in multihomed networks. Similar support for `YARN`/`MapReduce` is work in progress and will be documented when available.
|
||||||
|
|
||||||
* [HDFS Support for Multihomed Networks](#HDFS_Support_for_Multihomed_Networks)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Multihoming Background](#Multihoming_Background)
|
|
||||||
* [Fixing Hadoop Issues In Multihomed Environments](#Fixing_Hadoop_Issues_In_Multihomed_Environments)
|
|
||||||
* [Ensuring HDFS Daemons Bind All Interfaces](#Ensuring_HDFS_Daemons_Bind_All_Interfaces)
|
|
||||||
* [Clients use Hostnames when connecting to DataNodes](#Clients_use_Hostnames_when_connecting_to_DataNodes)
|
|
||||||
* [DataNodes use HostNames when connecting to other DataNodes](#DataNodes_use_HostNames_when_connecting_to_other_DataNodes)
|
|
||||||
* [Multihoming and Hadoop Security](#Multihoming_and_Hadoop_Security)
|
|
||||||
* [Hostname Lookup](#Hostname_Lookup)
|
|
||||||
|
|
||||||
Multihoming Background
|
Multihoming Background
|
||||||
----------------------
|
----------------------
|
||||||
|
|
|
||||||
|
|
@ -15,14 +15,7 @@
|
||||||
HDFS NFS Gateway
|
HDFS NFS Gateway
|
||||||
================
|
================
|
||||||
|
|
||||||
* [HDFS NFS Gateway](#HDFS_NFS_Gateway)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Configuration](#Configuration)
|
|
||||||
* [Start and stop NFS gateway service](#Start_and_stop_NFS_gateway_service)
|
|
||||||
* [Verify validity of NFS related services](#Verify_validity_of_NFS_related_services)
|
|
||||||
* [Mount the export "/"](#Mount_the_export_)
|
|
||||||
* [Allow mounts from unprivileged clients](#Allow_mounts_from_unprivileged_clients)
|
|
||||||
* [User authentication and mapping](#User_authentication_and_mapping)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,20 +15,7 @@
|
||||||
HDFS Permissions Guide
|
HDFS Permissions Guide
|
||||||
======================
|
======================
|
||||||
|
|
||||||
* [HDFS Permissions Guide](#HDFS_Permissions_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [User Identity](#User_Identity)
|
|
||||||
* [Group Mapping](#Group_Mapping)
|
|
||||||
* [Permission Checks](#Permission_Checks)
|
|
||||||
* [Understanding the Implementation](#Understanding_the_Implementation)
|
|
||||||
* [Changes to the File System API](#Changes_to_the_File_System_API)
|
|
||||||
* [Changes to the Application Shell](#Changes_to_the_Application_Shell)
|
|
||||||
* [The Super-User](#The_Super-User)
|
|
||||||
* [The Web Server](#The_Web_Server)
|
|
||||||
* [ACLs (Access Control Lists)](#ACLs_Access_Control_Lists)
|
|
||||||
* [ACLs File System API](#ACLs_File_System_API)
|
|
||||||
* [ACLs Shell Commands](#ACLs_Shell_Commands)
|
|
||||||
* [Configuration Parameters](#Configuration_Parameters)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,13 +15,7 @@
|
||||||
HDFS Quotas Guide
|
HDFS Quotas Guide
|
||||||
=================
|
=================
|
||||||
|
|
||||||
* [HDFS Quotas Guide](#HDFS_Quotas_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Name Quotas](#Name_Quotas)
|
|
||||||
* [Space Quotas](#Space_Quotas)
|
|
||||||
* [Storage Type Quotas](#Storage_Type_Quotas)
|
|
||||||
* [Administrative Commands](#Administrative_Commands)
|
|
||||||
* [Reporting Command](#Reporting_Command)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
@ -128,4 +122,4 @@ An an extension to the count command of the HDFS shell reports quota values and
|
||||||
storage type quota set and the available quota remaining for each directory.
|
storage type quota set and the available quota remaining for each directory.
|
||||||
If specific storage types are given after -t option, only quota and remaining
|
If specific storage types are given after -t option, only quota and remaining
|
||||||
quota of the types specified will be displayed. Otherwise, quota and
|
quota of the types specified will be displayed. Otherwise, quota and
|
||||||
remaining quota of all storage types that support quota will be displayed.
|
remaining quota of all storage types that support quota will be displayed.
|
||||||
|
|
|
||||||
|
|
@ -18,25 +18,7 @@
|
||||||
HDFS Rolling Upgrade
|
HDFS Rolling Upgrade
|
||||||
====================
|
====================
|
||||||
|
|
||||||
* [Introduction](#Introduction)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Upgrade](#Upgrade)
|
|
||||||
* [Upgrade without Downtime](#Upgrade_without_Downtime)
|
|
||||||
* [Upgrading Non-Federated Clusters](#Upgrading_Non-Federated_Clusters)
|
|
||||||
* [Upgrading Federated Clusters](#Upgrading_Federated_Clusters)
|
|
||||||
* [Upgrade with Downtime](#Upgrade_with_Downtime)
|
|
||||||
* [Upgrading Non-HA Clusters](#Upgrading_Non-HA_Clusters)
|
|
||||||
* [Downgrade and Rollback](#Downgrade_and_Rollback)
|
|
||||||
* [Downgrade](#Downgrade)
|
|
||||||
* [Downgrade without Downtime](#Downgrade_without_Downtime)
|
|
||||||
* [Downgrade with Downtime](#Downgrade_with_Downtime)
|
|
||||||
* [Rollback](#Rollback)
|
|
||||||
* [Commands and Startup Options for Rolling Upgrade](#Commands_and_Startup_Options_for_Rolling_Upgrade)
|
|
||||||
* [DFSAdmin Commands](#DFSAdmin_Commands)
|
|
||||||
* [dfsadmin -rollingUpgrade](#dfsadmin_-rollingUpgrade)
|
|
||||||
* [dfsadmin -getDatanodeInfo](#dfsadmin_-getDatanodeInfo)
|
|
||||||
* [dfsadmin -shutdownDatanode](#dfsadmin_-shutdownDatanode)
|
|
||||||
* [NameNode Startup Options](#NameNode_Startup_Options)
|
|
||||||
* [namenode -rollingUpgrade](#namenode_-rollingUpgrade)
|
|
||||||
|
|
||||||
|
|
||||||
Introduction
|
Introduction
|
||||||
|
|
|
||||||
|
|
@ -18,21 +18,7 @@
|
||||||
HDFS Snapshots
|
HDFS Snapshots
|
||||||
==============
|
==============
|
||||||
|
|
||||||
* [HDFS Snapshots](#HDFS_Snapshots)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Snapshottable Directories](#Snapshottable_Directories)
|
|
||||||
* [Snapshot Paths](#Snapshot_Paths)
|
|
||||||
* [Upgrading to a version of HDFS with snapshots](#Upgrading_to_a_version_of_HDFS_with_snapshots)
|
|
||||||
* [Snapshot Operations](#Snapshot_Operations)
|
|
||||||
* [Administrator Operations](#Administrator_Operations)
|
|
||||||
* [Allow Snapshots](#Allow_Snapshots)
|
|
||||||
* [Disallow Snapshots](#Disallow_Snapshots)
|
|
||||||
* [User Operations](#User_Operations)
|
|
||||||
* [Create Snapshots](#Create_Snapshots)
|
|
||||||
* [Delete Snapshots](#Delete_Snapshots)
|
|
||||||
* [Rename Snapshots](#Rename_Snapshots)
|
|
||||||
* [Get Snapshottable Directory Listing](#Get_Snapshottable_Directory_Listing)
|
|
||||||
* [Get Snapshots Difference Report](#Get_Snapshots_Difference_Report)
|
|
||||||
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
|
|
|
||||||
|
|
@ -15,28 +15,7 @@
|
||||||
HDFS Users Guide
|
HDFS Users Guide
|
||||||
================
|
================
|
||||||
|
|
||||||
* [HDFS Users Guide](#HDFS_Users_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Purpose](#Purpose)
|
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Prerequisites](#Prerequisites)
|
|
||||||
* [Web Interface](#Web_Interface)
|
|
||||||
* [Shell Commands](#Shell_Commands)
|
|
||||||
* [DFSAdmin Command](#DFSAdmin_Command)
|
|
||||||
* [Secondary NameNode](#Secondary_NameNode)
|
|
||||||
* [Checkpoint Node](#Checkpoint_Node)
|
|
||||||
* [Backup Node](#Backup_Node)
|
|
||||||
* [Import Checkpoint](#Import_Checkpoint)
|
|
||||||
* [Balancer](#Balancer)
|
|
||||||
* [Rack Awareness](#Rack_Awareness)
|
|
||||||
* [Safemode](#Safemode)
|
|
||||||
* [fsck](#fsck)
|
|
||||||
* [fetchdt](#fetchdt)
|
|
||||||
* [Recovery Mode](#Recovery_Mode)
|
|
||||||
* [Upgrade and Rollback](#Upgrade_and_Rollback)
|
|
||||||
* [DataNode Hot Swap Drive](#DataNode_Hot_Swap_Drive)
|
|
||||||
* [File Permissions and Security](#File_Permissions_and_Security)
|
|
||||||
* [Scalability](#Scalability)
|
|
||||||
* [Related Documentation](#Related_Documentation)
|
|
||||||
|
|
||||||
Purpose
|
Purpose
|
||||||
-------
|
-------
|
||||||
|
|
|
||||||
|
|
@ -15,11 +15,7 @@
|
||||||
HFTP Guide
|
HFTP Guide
|
||||||
==========
|
==========
|
||||||
|
|
||||||
* [HFTP Guide](#HFTP_Guilde)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Deprecated](#Deprecated)
|
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [Implementation](#Implementation)
|
|
||||||
* [Configuration Options](#Configuration_Options)
|
|
||||||
|
|
||||||
|
|
||||||
Deprecated
|
Deprecated
|
||||||
|
|
|
||||||
|
|
@ -15,13 +15,7 @@
|
||||||
C API libhdfs
|
C API libhdfs
|
||||||
=============
|
=============
|
||||||
|
|
||||||
* [C API libhdfs](#C_API_libhdfs)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [The APIs](#The_APIs)
|
|
||||||
* [A Sample Program](#A_Sample_Program)
|
|
||||||
* [How To Link With The Library](#How_To_Link_With_The_Library)
|
|
||||||
* [Common Problems](#Common_Problems)
|
|
||||||
* [Thread Safe](#Thread_Safe)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,19 +15,7 @@
|
||||||
Memory Storage Support in HDFS
|
Memory Storage Support in HDFS
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
* [Introduction](#Introduction)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Administrator Configuration](#Administrator_Configuration)
|
|
||||||
* [Limit RAM used for replicas in Memory](#Limit_RAM_used_for_replicas_in_Memory)
|
|
||||||
* [Setup RAM Disks on Data Nodes](#Setup_RAM_Disks_on_Data_Nodes)
|
|
||||||
* [Choosing `tmpfs` \(vs `ramfs`\)](#Choosing_`tmpfs`_\(vs_`ramfs`\))
|
|
||||||
* [Mount RAM Disks](#Mount_RAM_Disks)
|
|
||||||
* [Tag `tmpfs` volume with the RAM\_DISK Storage Type](#Tag_`tmpfs`_volume_with_the_RAM\_DISK_Storage_Type)
|
|
||||||
* [Ensure Storage Policies are enabled](#Ensure_Storage_Policies_are_enabled)
|
|
||||||
* [Application Usage](#Application_Usage)
|
|
||||||
* [Use the LAZY\_PERSIST Storage Policy](#Use_the_LAZY\_PERSIST_Storage_Policy)
|
|
||||||
* [Invoke `hdfs storagepolicies` command for directories](#Invoke_hdfs_storagepolicies_command_for_directories)
|
|
||||||
* [Call `setStoragePolicy` method for directories](#Call_`setStoragePolicy`_method_for_directories)
|
|
||||||
* [Pass `LAZY_PERSIST` `CreateFlag` for new files](#Pass_`LAZY_PERSIST`_`CreateFlag`_for_new_files)
|
|
||||||
|
|
||||||
Introduction
|
Introduction
|
||||||
------------
|
------------
|
||||||
|
|
|
||||||
|
|
@ -15,12 +15,7 @@
|
||||||
Synthetic Load Generator Guide
|
Synthetic Load Generator Guide
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
* [Synthetic Load Generator Guide](#Synthetic_Load_Generator_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Overview](#Overview)
|
|
||||||
* [Synopsis](#Synopsis)
|
|
||||||
* [Test Space Population](#Test_Space_Population)
|
|
||||||
* [Structure Generator](#Structure_Generator)
|
|
||||||
* [Data Generator](#Data_Generator)
|
|
||||||
|
|
||||||
Overview
|
Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,12 +15,7 @@
|
||||||
HDFS Short-Circuit Local Reads
|
HDFS Short-Circuit Local Reads
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
* [HDFS Short-Circuit Local Reads](#HDFS_Short-Circuit_Local_Reads)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Short-Circuit Local Reads](#Short-Circuit_Local_Reads)
|
|
||||||
* [Background](#Background)
|
|
||||||
* [Setup](#Setup)
|
|
||||||
* [Example Configuration](#Example_Configuration)
|
|
||||||
* [Legacy HDFS Short-Circuit Local Reads](#Legacy_HDFS_Short-Circuit_Local_Reads)
|
|
||||||
|
|
||||||
Short-Circuit Local Reads
|
Short-Circuit Local Reads
|
||||||
-------------------------
|
-------------------------
|
||||||
|
|
|
||||||
|
|
@ -15,32 +15,7 @@
|
||||||
Transparent Encryption in HDFS
|
Transparent Encryption in HDFS
|
||||||
==============================
|
==============================
|
||||||
|
|
||||||
* [Overview](#Overview)
|
<!-- MACRO{toc|fromDepth=0|toDepth=2} -->
|
||||||
* [Background](#Background)
|
|
||||||
* [Use Cases](#Use_Cases)
|
|
||||||
* [Architecture](#Architecture)
|
|
||||||
* [Overview](#Architecture_overview)
|
|
||||||
* [Accessing data within an encryption zone](#Accessing_data_within_an_encryption_zone)
|
|
||||||
* [Key Management Server, KeyProvider, EDEKs](#Key_Management_Server_KeyProvider_EDEKs)
|
|
||||||
* [Configuration](#Configuration)
|
|
||||||
* [Configuring the cluster KeyProvider](#Configuring_the_cluster_KeyProvider)
|
|
||||||
* [Selecting an encryption algorithm and codec](#Selecting_an_encryption_algorithm_and_codec)
|
|
||||||
* [Namenode configuration](#Namenode_configuration)
|
|
||||||
* [crypto command-line interface](#crypto_command-line_interface)
|
|
||||||
* [createZone](#createZone)
|
|
||||||
* [listZones](#listZones)
|
|
||||||
* [provisionTrash](#provisionTrash)
|
|
||||||
* [getFileEncryptionInfo](#getFileEncryptionInfo)
|
|
||||||
* [Example usage](#Example_usage)
|
|
||||||
* [Distcp considerations](#Distcp_considerations)
|
|
||||||
* [Running as the superuser](#Running_as_the_superuser)
|
|
||||||
* [Copying into encrypted locations](#Copying_into_encrypted_locations)
|
|
||||||
* [Rename and Trash considerations](#Rename_and_Trash_considerations)
|
|
||||||
* [Attack vectors](#Attack_vectors)
|
|
||||||
* [Hardware access exploits](#Hardware_access_exploits)
|
|
||||||
* [Root access exploits](#Root_access_exploits)
|
|
||||||
* [HDFS admin exploits](#HDFS_admin_exploits)
|
|
||||||
* [Rogue user exploits](#Rogue_user_exploits)
|
|
||||||
|
|
||||||
<a name="Overview"></a>Overview
|
<a name="Overview"></a>Overview
|
||||||
--------
|
--------
|
||||||
|
|
|
||||||
|
|
@ -15,20 +15,7 @@
|
||||||
ViewFs Guide
|
ViewFs Guide
|
||||||
============
|
============
|
||||||
|
|
||||||
* [ViewFs Guide](#ViewFs_Guide)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [The Old World (Prior to Federation)](#The_Old_World_Prior_to_Federation)
|
|
||||||
* [Single Namenode Clusters](#Single_Namenode_Clusters)
|
|
||||||
* [Pathnames Usage Patterns](#Pathnames_Usage_Patterns)
|
|
||||||
* [Pathname Usage Best Practices](#Pathname_Usage_Best_Practices)
|
|
||||||
* [New World – Federation and ViewFs](#New_World__Federation_and_ViewFs)
|
|
||||||
* [How The Clusters Look](#How_The_Clusters_Look)
|
|
||||||
* [A Global Namespace Per Cluster Using ViewFs](#A_Global_Namespace_Per_Cluster_Using_ViewFs)
|
|
||||||
* [Pathname Usage Patterns](#Pathname_Usage_Patterns)
|
|
||||||
* [Pathname Usage Best Practices](#Pathname_Usage_Best_Practices)
|
|
||||||
* [Renaming Pathnames Across Namespaces](#Renaming_Pathnames_Across_Namespaces)
|
|
||||||
* [FAQ](#FAQ)
|
|
||||||
* [Appendix: A Mount Table Configuration Example](#Appendix:_A_Mount_Table_Configuration_Example)
|
|
||||||
|
|
||||||
Introduction
|
Introduction
|
||||||
------------
|
------------
|
||||||
|
|
|
||||||
|
|
@ -15,128 +15,7 @@
|
||||||
WebHDFS REST API
|
WebHDFS REST API
|
||||||
================
|
================
|
||||||
|
|
||||||
* [WebHDFS REST API](#WebHDFS_REST_API)
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
* [Document Conventions](#Document_Conventions)
|
|
||||||
* [Introduction](#Introduction)
|
|
||||||
* [Operations](#Operations)
|
|
||||||
* [FileSystem URIs vs HTTP URLs](#FileSystem_URIs_vs_HTTP_URLs)
|
|
||||||
* [HDFS Configuration Options](#HDFS_Configuration_Options)
|
|
||||||
* [Authentication](#Authentication)
|
|
||||||
* [Proxy Users](#Proxy_Users)
|
|
||||||
* [Cross-Site Request Forgery Prevention](#Cross-Site_Request_Forgery_Prevention)
|
|
||||||
* [WebHDFS Retry Policy](#WebHDFS_Retry_Policy)
|
|
||||||
* [File and Directory Operations](#File_and_Directory_Operations)
|
|
||||||
* [Create and Write to a File](#Create_and_Write_to_a_File)
|
|
||||||
* [Append to a File](#Append_to_a_File)
|
|
||||||
* [Concat File(s)](#Concat_Files)
|
|
||||||
* [Open and Read a File](#Open_and_Read_a_File)
|
|
||||||
* [Make a Directory](#Make_a_Directory)
|
|
||||||
* [Create a Symbolic Link](#Create_a_Symbolic_Link)
|
|
||||||
* [Rename a File/Directory](#Rename_a_FileDirectory)
|
|
||||||
* [Delete a File/Directory](#Delete_a_FileDirectory)
|
|
||||||
* [Truncate a File](#Truncate_a_File)
|
|
||||||
* [Status of a File/Directory](#Status_of_a_FileDirectory)
|
|
||||||
* [List a Directory](#List_a_Directory)
|
|
||||||
* [Iteratively List a Directory](#Iteratively_List_a_Directory)
|
|
||||||
* [Get File Block Locations](#Get_File_Block_Locations)
|
|
||||||
* [Other File System Operations](#Other_File_System_Operations)
|
|
||||||
* [Get Content Summary of a Directory](#Get_Content_Summary_of_a_Directory)
|
|
||||||
* [Get File Checksum](#Get_File_Checksum)
|
|
||||||
* [Get Home Directory](#Get_Home_Directory)
|
|
||||||
* [Get Trash Root](#Get_Trash_Root)
|
|
||||||
* [Set Permission](#Set_Permission)
|
|
||||||
* [Set Owner](#Set_Owner)
|
|
||||||
* [Set Replication Factor](#Set_Replication_Factor)
|
|
||||||
* [Set Access or Modification Time](#Set_Access_or_Modification_Time)
|
|
||||||
* [Modify ACL Entries](#Modify_ACL_Entries)
|
|
||||||
* [Remove ACL Entries](#Remove_ACL_Entries)
|
|
||||||
* [Remove Default ACL](#Remove_Default_ACL)
|
|
||||||
* [Remove ACL](#Remove_ACL)
|
|
||||||
* [Set ACL](#Set_ACL)
|
|
||||||
* [Get ACL Status](#Get_ACL_Status)
|
|
||||||
* [Check access](#Check_access)
|
|
||||||
* [Get all Storage Policies](#Get_all_Storage_Policies)
|
|
||||||
* [Set Storage Policy](#Set_Storage_Policy)
|
|
||||||
* [Unset Storage Policy](#Unset_Storage_Policy)
|
|
||||||
* [Get Storage Policy](#Get_Storage_Policy)
|
|
||||||
* [Extended Attributes(XAttrs) Operations](#Extended_AttributesXAttrs_Operations)
|
|
||||||
* [Set XAttr](#Set_XAttr)
|
|
||||||
* [Remove XAttr](#Remove_XAttr)
|
|
||||||
* [Get an XAttr](#Get_an_XAttr)
|
|
||||||
* [Get multiple XAttrs](#Get_multiple_XAttrs)
|
|
||||||
* [Get all XAttrs](#Get_all_XAttrs)
|
|
||||||
* [List all XAttrs](#List_all_XAttrs)
|
|
||||||
* [Snapshot Operations](#Snapshot_Operations)
|
|
||||||
* [Create Snapshot](#Create_Snapshot)
|
|
||||||
* [Delete Snapshot](#Delete_Snapshot)
|
|
||||||
* [Rename Snapshot](#Rename_Snapshot)
|
|
||||||
* [Delegation Token Operations](#Delegation_Token_Operations)
|
|
||||||
* [Get Delegation Token](#Get_Delegation_Token)
|
|
||||||
* [Renew Delegation Token](#Renew_Delegation_Token)
|
|
||||||
* [Cancel Delegation Token](#Cancel_Delegation_Token)
|
|
||||||
* [Error Responses](#Error_Responses)
|
|
||||||
* [HTTP Response Codes](#HTTP_Response_Codes)
|
|
||||||
* [Illegal Argument Exception](#Illegal_Argument_Exception)
|
|
||||||
* [Security Exception](#Security_Exception)
|
|
||||||
* [Access Control Exception](#Access_Control_Exception)
|
|
||||||
* [File Not Found Exception](#File_Not_Found_Exception)
|
|
||||||
* [JSON Schemas](#JSON_Schemas)
|
|
||||||
* [ACL Status JSON Schema](#ACL_Status_JSON_Schema)
|
|
||||||
* [XAttrs JSON Schema](#XAttrs_JSON_Schema)
|
|
||||||
* [XAttrNames JSON Schema](#XAttrNames_JSON_Schema)
|
|
||||||
* [Boolean JSON Schema](#Boolean_JSON_Schema)
|
|
||||||
* [ContentSummary JSON Schema](#ContentSummary_JSON_Schema)
|
|
||||||
* [FileChecksum JSON Schema](#FileChecksum_JSON_Schema)
|
|
||||||
* [FileStatus JSON Schema](#FileStatus_JSON_Schema)
|
|
||||||
* [FileStatus Properties](#FileStatus_Properties)
|
|
||||||
* [FileStatuses JSON Schema](#FileStatuses_JSON_Schema)
|
|
||||||
* [Long JSON Schema](#Long_JSON_Schema)
|
|
||||||
* [Path JSON Schema](#Path_JSON_Schema)
|
|
||||||
* [RemoteException JSON Schema](#RemoteException_JSON_Schema)
|
|
||||||
* [Token JSON Schema](#Token_JSON_Schema)
|
|
||||||
* [Token Properties](#Token_Properties)
|
|
||||||
* [BlockStoragePolicy JSON Schema](#BlockStoragePolicy_JSON_Schema)
|
|
||||||
* [BlockStoragePolicy Properties](#BlockStoragePolicy_Properties)
|
|
||||||
* [BlockStoragePolicies JSON Schema](#BlockStoragePolicies_JSON_Schema)
|
|
||||||
* [BlockLocation JSON Schema](#BlockLocation_JSON_Schema)
|
|
||||||
* [BlockLocation Properties](#BlockLocation_Properties)
|
|
||||||
* [BlockLocations JSON Schema](#BlockLocations_JSON_Schema)
|
|
||||||
* [HTTP Query Parameter Dictionary](#HTTP_Query_Parameter_Dictionary)
|
|
||||||
* [ACL Spec](#ACL_Spec)
|
|
||||||
* [XAttr Name](#XAttr_Name)
|
|
||||||
* [XAttr Value](#XAttr_Value)
|
|
||||||
* [XAttr set flag](#XAttr_set_flag)
|
|
||||||
* [XAttr value encoding](#XAttr_value_encoding)
|
|
||||||
* [Access Time](#Access_Time)
|
|
||||||
* [Block Size](#Block_Size)
|
|
||||||
* [Buffer Size](#Buffer_Size)
|
|
||||||
* [Create Flag](#Create_Flag)
|
|
||||||
* [Create Parent](#Create_Parent)
|
|
||||||
* [Delegation](#Delegation)
|
|
||||||
* [Destination](#Destination)
|
|
||||||
* [Do As](#Do_As)
|
|
||||||
* [Fs Action](#Fs_Action)
|
|
||||||
* [Group](#Group)
|
|
||||||
* [Length](#Length)
|
|
||||||
* [Modification Time](#Modification_Time)
|
|
||||||
* [Offset](#Offset)
|
|
||||||
* [Old Snapshot Name](#Old_Snapshot_Name)
|
|
||||||
* [Op](#Op)
|
|
||||||
* [Overwrite](#Overwrite)
|
|
||||||
* [Owner](#Owner)
|
|
||||||
* [Permission](#Permission)
|
|
||||||
* [Recursive](#Recursive)
|
|
||||||
* [Renewer](#Renewer)
|
|
||||||
* [Replication](#Replication)
|
|
||||||
* [Snapshot Name](#Snapshot_Name)
|
|
||||||
* [Sources](#Sources)
|
|
||||||
* [Token](#Token)
|
|
||||||
* [Token Kind](#Token_Kind)
|
|
||||||
* [Token Service](#Token_Service)
|
|
||||||
* [Username](#Username)
|
|
||||||
* [NoRedirect](#NoRedirect)
|
|
||||||
* [Storage Policy](#Storage_Policy)
|
|
||||||
* [Start After](#Start_After)
|
|
||||||
|
|
||||||
Document Conventions
|
Document Conventions
|
||||||
--------------------
|
--------------------
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue