Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HDFS-13434. RBF: Fix dead links in RBF document. Contributed by Chetna Chaudhari. 

(cherry picked from commit f469628bba)
This commit is contained in:
Inigo Goiri 2018-04-27 15:13:47 -07:00
parent c9f7209662
commit 71e52c936d
1 changed files with 7 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
hadoop-hdfs-project/hadoop-hdfs-rbf/src/site/markdown/HDFSRouterFederation.md
View File

@ -21,7 +21,7 @@ Introduction
------------ ------------
NameNodes have scalability limits because of the metadata overhead comprised of inodes (files and directories) and file blocks, the number of Datanode heartbeats, and the number of HDFS RPC client requests. NameNodes have scalability limits because of the metadata overhead comprised of inodes (files and directories) and file blocks, the number of Datanode heartbeats, and the number of HDFS RPC client requests.
The common solution is to split the filesystem into smaller subclusters [HDFS Federation](./Federation.html) and provide a federated view [ViewFs](./ViewFs.html). The common solution is to split the filesystem into smaller subclusters [HDFS Federation](../hadoop-hdfs/Federation.html) and provide a federated view [ViewFs](../hadoop-hdfs/ViewFs.html).
The problem is how to maintain the split of the subclusters (e.g., namespace partition), which forces users to connect to multiple subclusters and manage the allocation of folders/files to them. The problem is how to maintain the split of the subclusters (e.g., namespace partition), which forces users to connect to multiple subclusters and manage the allocation of folders/files to them.
@ -37,8 +37,8 @@ This layer must be scalable, highly available, and fault tolerant.
This federation layer comprises multiple components. This federation layer comprises multiple components.
The _Router_ component that has the same interface as a NameNode, and forwards the client requests to the correct subcluster, based on ground-truth information from a State Store. The _Router_ component that has the same interface as a NameNode, and forwards the client requests to the correct subcluster, based on ground-truth information from a State Store.
The _State Store_ combines a remote _Mount Table_ (in the flavor of [ViewFs](./ViewFs.html), but shared between clients) and utilization (load/capacity) information about the subclusters. The _State Store_ combines a remote _Mount Table_ (in the flavor of [ViewFs](../hadoop-hdfs/ViewFs.html), but shared between clients) and utilization (load/capacity) information about the subclusters.
This approach has the same architecture as [YARN federation](../hadoop-yarn/Federation.html). This approach has the same architecture as [YARN federation](../../hadoop-yarn/hadoop-yarn-site/Federation.html).
![Router-based Federation Sequence Diagram | width=800](./images/routerfederation.png) ![Router-based Federation Sequence Diagram | width=800](./images/routerfederation.png)
@ -158,7 +158,7 @@ The Routers discard the entries older than a certain threshold (e.g., ten Router
* **Mount Table**: * **Mount Table**:
This table hosts the mapping between folders and subclusters. This table hosts the mapping between folders and subclusters.
It is similar to the mount table in [ViewFs](.ViewFs.html) where it specifies the federated folder, the destination subcluster and the path in that folder. It is similar to the mount table in [ViewFs](../hadoop-hdfs/ViewFs.html) where it specifies the federated folder, the destination subcluster and the path in that folder.
### Security ### Security
@ -170,7 +170,7 @@ Deployment
By default, the Router is ready to take requests and monitor the NameNode in the local machine. By default, the Router is ready to take requests and monitor the NameNode in the local machine.
It needs to know the State Store endpoint by setting `dfs.federation.router.store.driver.class`. It needs to know the State Store endpoint by setting `dfs.federation.router.store.driver.class`.
The rest of the options are documented in [hdfs-default.xml](./hdfs-default.xml). The rest of the options are documented in [hdfs-default.xml](../hadoop-hdfs/hdfs-default.xml).
Once the Router is configured, it can be started: Once the Router is configured, it can be started:
@ -182,7 +182,7 @@ And to stop it:
### Mount table management ### Mount table management
The mount table entries are pretty much the same as in [ViewFs](./ViewFs.html). The mount table entries are pretty much the same as in [ViewFs](../hadoop-hdfs/ViewFs.html).
A good practice for simplifying the management is to name the federated namespace with the same names as the destination namespaces. A good practice for simplifying the management is to name the federated namespace with the same names as the destination namespaces.
For example, if we to mount `/data/app1` in the federated namespace, it is recommended to have that same name as in the destination namespace. For example, if we to mount `/data/app1` in the federated namespace, it is recommended to have that same name as in the destination namespace.
@ -270,7 +270,7 @@ Router configuration
-------------------- --------------------
One can add the configurations for Router-based federation to **hdfs-site.xml**. One can add the configurations for Router-based federation to **hdfs-site.xml**.
The main options are documented in [hdfs-default.xml](./hdfs-default.xml). The main options are documented in [hdfs-default.xml](../hadoop-hdfs/hdfs-default.xml).
The configuration values are described in this section. The configuration values are described in this section.
### RPC server ### RPC server