HDFS-15330. Document the ViewFSOverloadScheme details in ViewFS guide. Contributed by Uma Maheswara Rao G.
This commit is contained in:
Uma Maheswara Rao G
GitHub
parent
2326123705
commit
76fa0222f0
|
|
@ -693,4 +693,42 @@ Usage: `hdfs debug recoverLease -path <path> [-retries <num-retries>]`
|
||||||
| [`-path` *path*] | HDFS path for which to recover the lease. |
|
| [`-path` *path*] | HDFS path for which to recover the lease. |
|
||||||
| [`-retries` *num-retries*] | Number of times the client will retry calling recoverLease. The default number of retries is 1. |
|
| [`-retries` *num-retries*] | Number of times the client will retry calling recoverLease. The default number of retries is 1. |
|
||||||
|
|
||||||
Recover the lease on the specified path. The path must reside on an HDFS filesystem. The default number of retries is 1.
|
Recover the lease on the specified path. The path must reside on an HDFS file system. The default number of retries is 1.
|
||||||
|
|
||||||
|
dfsadmin with ViewFsOverloadScheme
|
||||||
|
----------------------------------
|
||||||
|
|
||||||
|
Usage: `hdfs dfsadmin -fs <child fs mount link URI> <dfsadmin command options>`
|
||||||
|
|
||||||
|
| COMMAND\_OPTION | Description |
|
||||||
|
|:---- |:---- |
|
||||||
|
| `-fs` *child fs mount link URI* | Its a logical mount link path to child file system in ViewFS world. This uri typically formed as src mount link prefixed with fs.defaultFS. Please note, this is not an actual child file system uri, instead its a logical mount link uri pointing to actual child file system|
|
||||||
|
|
||||||
|
Example command usage:
|
||||||
|
`hdfs dfsadmin -fs hdfs://nn1 -safemode enter`
|
||||||
|
|
||||||
|
In ViewFsOverloadScheme, we may have multiple child file systems as mount point mappings as shown in [ViewFsOverloadScheme Guide](./ViewFsOverloadScheme.html). Here -fs option is an optional generic parameter supported by dfsadmin. When users want to execute commands on one of the child file system, they need to pass that file system mount mapping link uri to -fs option. Let's take an example mount link configuration and dfsadmin command below.
|
||||||
|
|
||||||
|
Mount link:
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.defaultFS</name>
|
||||||
|
<value>hdfs://MyCluster1</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.MyCluster1./user</name>
|
||||||
|
<value>hdfs://MyCluster2/user</value>
|
||||||
|
<!-- mount table name : MyCluster1
|
||||||
|
mount link mapping: hdfs://MyCluster1/user --> hdfs://MyCluster2/user
|
||||||
|
mount link path: /user
|
||||||
|
mount link uri: hdfs://MyCluster1/user
|
||||||
|
mount target uri for /user: hdfs://MyCluster2/user -->
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
|
||||||
|
If user wants to talk to `hdfs://MyCluster2/`, then they can pass -fs option (`-fs hdfs://MyCluster1/user`)
|
||||||
|
Since /user was mapped to a cluster `hdfs://MyCluster2/user`, dfsadmin resolve the passed (`-fs hdfs://MyCluster1/user`) to target fs (`hdfs://MyCluster2/user`).
|
||||||
|
This way users can get the access to all hdfs child file systems in ViewFsOverloadScheme.
|
||||||
|
If there is no `-fs` option provided, then it will try to connect to the configured fs.defaultFS cluster if a cluster running with the fs.defaultFS uri.
|
||||||
|
|
|
||||||
|
|
@ -361,6 +361,12 @@ resume its work, it's a good idea to provision some sort of cron job to purge su
|
||||||
|
|
||||||
Delegation tokens for the cluster to which you are submitting the job (including all mounted volumes for that cluster’s mount table), and for input and output paths to your map-reduce job (including all volumes mounted via mount tables for the specified input and output paths) are all handled automatically. In addition, there is a way to add additional delegation tokens to the base cluster configuration for special circumstances.
|
Delegation tokens for the cluster to which you are submitting the job (including all mounted volumes for that cluster’s mount table), and for input and output paths to your map-reduce job (including all volumes mounted via mount tables for the specified input and output paths) are all handled automatically. In addition, there is a way to add additional delegation tokens to the base cluster configuration for special circumstances.
|
||||||
|
|
||||||
|
Don't want to change scheme or difficult to copy mount-table configurations to all clients?
|
||||||
|
-------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
|
Please refer to the [View File System Overload Scheme Guide](./ViewFsOverloadScheme.html)
|
||||||
|
|
||||||
|
|
||||||
Appendix: A Mount Table Configuration Example
|
Appendix: A Mount Table Configuration Example
|
||||||
---------------------------------------------
|
---------------------------------------------
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,163 @@
|
||||||
|
<!---
|
||||||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
||||||
|
you may not use this file except in compliance with the License.
|
||||||
|
You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License. See accompanying LICENSE file.
|
||||||
|
-->
|
||||||
|
|
||||||
|
View File System Overload Scheme Guide
|
||||||
|
======================================
|
||||||
|
|
||||||
|
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||||
|
|
||||||
|
Introduction
|
||||||
|
------------
|
||||||
|
|
||||||
|
The View File System Overload Scheme introduced to solve two key challenges with the View File System(ViewFS). The first problem is, to use ViewFS, users need to update fs.defaultFS with viewfs scheme (`viewfs://`). The second problem is that users need to copy the mount-table configurations to all the client nodes.
|
||||||
|
The ViewFileSystemOverloadScheme is addressing these challenges.
|
||||||
|
|
||||||
|
View File System Overload Scheme
|
||||||
|
--------------------------------
|
||||||
|
|
||||||
|
### Details
|
||||||
|
|
||||||
|
The View File System Overload Scheme is an extension to the View File System. This will allow users to continue to use their existing fs.defaultFS configured scheme or any new scheme name instead of using scheme `viewfs`. Mount link configurations key, value formats are same as in [ViewFS Guide](./ViewFs.html). If a user wants to continue use the same fs.defaultFS and wants to have more mount points, then mount link configurations should have the current fs.defaultFS authority name as mount table name. Example if fs.defaultFS is `hdfs://mycluster`, then the mount link configuration key name should be like in the following format `fs.viewfs.mounttable.*mycluster*.<mountLinkPath>`. We will discuss more example configurations in following sections.
|
||||||
|
|
||||||
|
Another important improvement with the ViewFileSystemOverloadScheme is, administrators need not copy the `mount-table.xml` configuration file to 1000s of client nodes. Instead they can keep the mount-table configuration file in a Hadoop compatible file system. So, keeping the configuration file in a central place makes administrators life easier as they can update mount-table in single place.
|
||||||
|
|
||||||
|
### Enabling View File System Overload Scheme
|
||||||
|
|
||||||
|
To use this class, the following configurations needed to be added in core-site.xml file.
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.<scheme>.impl</name>
|
||||||
|
<value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
Here `<scheme>` should be same as the uri-scheme configured in fs.defautFS. For example if fs.defaultFS was configured with `hdfs://mycluster`, then the above configuration would be like below:
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.hdfs.impl</name>
|
||||||
|
<value>org.apache.hadoop.fs.viewfs.ViewFileSystemOverloadScheme</value>
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example Configurations
|
||||||
|
|
||||||
|
**Example 1:**
|
||||||
|
|
||||||
|
If users want some of their existing cluster (`hdfs://mycluster`) data to mount with hdfs(`hdfs://mycluster`) and other object store clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucket1/`), the following example configurations can show how to add mount links.
|
||||||
|
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.Cluster./user</name>
|
||||||
|
<value>hdfs://mycluster/user</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.Cluster./data</name>
|
||||||
|
<value>o3fs://bucket1.volume1/data</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.Cluster./backup</name>
|
||||||
|
<value>s3a://bucket1/backup/</value>
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
|
||||||
|
Let's consider the following operations to understand where these operations will be delegated based on mount links.
|
||||||
|
|
||||||
|
*Op1:* Create a file with the the path `hdfs://mycluster/user/fileA`, then physically this file will be created at `hdfs://mycluster/user/fileA`. This delegation happened based on the first configuration parameter in above configurations. Here `/user` mapped to `hdfs://mycluster/user/`.
|
||||||
|
|
||||||
|
*Op2:* Create a file the the path `hdfs://mycluster/data/datafile`, then this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This delegation happened based on second configurations parameter in above configurations. Here `/data` was mapped with `o3fs://bucket1.volume1.omhost/data/`.
|
||||||
|
|
||||||
|
*Op3:* Create a file with the the path `hdfs://Cluster/backup/data.zip`, then physically this file will be created at `s3a://bucket1/backup/data.zip`. This delegation happened based on the third configuration parameter in above configurations. Here `/backup` was mapped to `s3a://bucket1/backup/`.
|
||||||
|
|
||||||
|
|
||||||
|
**Example 2:**
|
||||||
|
|
||||||
|
If users want some of their existing cluster (`s3a://bucketA/`) data to mount with other hdfs cluster(`hdfs://Cluster`) and object store clusters(`o3fs://bucket1.volume1.omhost/`, `s3a://bucketA/`), the following example configurations can show how to add mount links.
|
||||||
|
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.bucketA./user</name>
|
||||||
|
<value>hdfs://Cluster/user</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.bucketA./data</name>
|
||||||
|
<value>o3fs://bucket1.volume1.omhost/data</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.bucketA./salesDB</name>
|
||||||
|
<value>s3a://bucketA/salesDB/</value>
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
Let's consider the following operations to understand to where these operations will be delegated based on mount links.
|
||||||
|
|
||||||
|
*Op1:* Create a file with the the path `s3a://bucketA/user/fileA`, then this file will be created physically at `hdfs://Cluster/user/fileA`. This delegation happened based on the first configuration parameter in above configurations. Here `/user` mapped to `hdfs://Cluster/user`.
|
||||||
|
|
||||||
|
*Op2:* Create a file the the path `s3a://bucketA/data/datafile`, then this file will be created at `o3fs://bucket1.volume1.omhost/data/datafile`. This delegation happened based on second configurations parameter in above configurations. Here `/data` was mapped with `o3fs://bucket1.volume1.omhost/data/`.
|
||||||
|
|
||||||
|
*Op3:* Create a file with the the path `s3a://bucketA/salesDB/dbfile`, then physically this file will be created at `s3a://bucketA/salesDB/dbfile`. This delegation happened based on the third configuration parameter in above configurations. Here `/salesDB` was mapped to `s3a://bucket1/salesDB`.
|
||||||
|
|
||||||
|
Note: In above examples we used create operation only, but the same mechanism applies to any other file system APIs here.
|
||||||
|
|
||||||
|
The following picture shows how the different schemes can be used in ViewFileSystemOverloadScheme compared to the ViewFileSystem.
|
||||||
|
|
||||||
|
<img src="./images/ViewFSOverloadScheme.png" width="1050" height="550"/>
|
||||||
|
|
||||||
|
### Central Mount Table Configurations
|
||||||
|
|
||||||
|
To enable central mount table configuration, we need to configure `fs.viewfs.mounttable.path` in `core-site.xml` with the value as the Hadoop compatible file system directory/file path, where the `mount-table-<versionNumber>.xml` file copied. Here versionNumber is an integer number and need to increase the version number and upload new file in same directory.
|
||||||
|
|
||||||
|
The ViewFileSystemOverloadScheme always loads the highest version number `mount-table-<versionNumber>.xml`. Please don't replace the file with same name. Always increment the version number to take new file picked by newly initializing clients. Why we don't recommend to replace the files is that, some client might have already opened the connections to old mount-table files already and in middle of loading configuration files, and replacing files can make them fail.
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.path</name>
|
||||||
|
<value>hdfs://Cluster/config/mount-table-dir</value>
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
If you are sure, you will never do updates to mount-table file, you can also configure file path directly like below. If you configure file path, it will not check any highest version number loading. Whatever file configured it will be loaded. However file name format should be same.
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<property>
|
||||||
|
<name>fs.viewfs.mounttable.path</name>
|
||||||
|
<value>hdfs://Cluster/config/mount-table-dir/mount-table-<versionNumber>.xml</value>
|
||||||
|
</property>
|
||||||
|
```
|
||||||
|
Note: we recommend not to configure mount-links in `core-site.xml` if you configure above valid path. Otherwise both mount links will be mixed and can lead to a confused behavior.
|
||||||
|
|
||||||
|
If you copy the `mount-table-<versionNumber>.xml`, you may consider having big replication factor depending on your cluster size. So, that file will be available locally to majority of clients as applications(MR/YARN/HBASE..etc) use locality on HDFS when reading `mount-table-<versionNumber>.xml`.
|
||||||
|
|
||||||
|
DFSAdmin commands with View File System Overload Scheme
|
||||||
|
-------------------------------------------------------
|
||||||
|
|
||||||
|
Please refer to the [HDFSCommands Guide](./HDFSCommands.html#dfsadmin_with_ViewFsOverloadScheme)
|
||||||
|
|
||||||
|
Appendix: A Mount Table Configuration with XInclude
|
||||||
|
---------------------------------------------------
|
||||||
|
|
||||||
|
If users have a HTTP server in trusted network and don't need authentication mechanism to it, you can also place your mount-table.xml file in that server and configure
|
||||||
|
XInclude xml tag with `mount-table.xml` file.
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<configuration xmlns:xi="http://www.w3.org/2001/XInclude">
|
||||||
|
<xi:include href="http://myserver/mountTable/mountTable.xml" />
|
||||||
|
</configuration>
|
||||||
|
```
|
||||||
|
|
||||||
|
The Apache Hadoop configuration has the capability to read the http urls from XInclude and load into configurations. If you choose this option, please don't configure mount-table configuration items in `core-site.xml` or at `fs.viewfs.mounttable.path`. Please note, Hadoop configuration XInclude does not use SPNego authentication when opening url. So, this will not work if http server where you placed `mount-table.xml` needs authentication.
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 186 KiB |
|
|
@ -85,6 +85,7 @@
|
||||||
<item name="Observer NameNode" href="hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html"/>
|
<item name="Observer NameNode" href="hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html"/>
|
||||||
<item name="Federation" href="hadoop-project-dist/hadoop-hdfs/Federation.html"/>
|
<item name="Federation" href="hadoop-project-dist/hadoop-hdfs/Federation.html"/>
|
||||||
<item name="ViewFs" href="hadoop-project-dist/hadoop-hdfs/ViewFs.html"/>
|
<item name="ViewFs" href="hadoop-project-dist/hadoop-hdfs/ViewFs.html"/>
|
||||||
|
<item name="ViewFsOverloadScheme" href="hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html"/>
|
||||||
<item name="Snapshots" href="hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html"/>
|
<item name="Snapshots" href="hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html"/>
|
||||||
<item name="Edits Viewer" href="hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html"/>
|
<item name="Edits Viewer" href="hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html"/>
|
||||||
<item name="Image Viewer" href="hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html"/>
|
<item name="Image Viewer" href="hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html"/>
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue