<li><ahref="#Appendix:_A_Mount_Table_Configuration_with_XInclude">Appendix: A Mount Table Configuration with XInclude</a></li></ul>
<section>
<h2><aname="Introduction"></a>Introduction</h2>
<p>The View File System Overload Scheme introduced to solve two key challenges with the View File System(ViewFS). The first problem is, to use ViewFS, users need to update fs.defaultFS with viewfs scheme (<code>viewfs://</code>). The second problem is that users need to copy the mount-table configurations to all the client nodes. The ViewFileSystemOverloadScheme is addressing these challenges.</p></section><section>
<h2><aname="View_File_System_Overload_Scheme"></a>View File System Overload Scheme</h2><section>
<h3><aname="Details"></a>Details</h3>
<p>The View File System Overload Scheme is an extension to the View File System. This will allow users to continue to use their existing fs.defaultFS configured scheme or any new scheme name instead of using scheme <code>viewfs</code>. Mount link configurations key, value formats are same as in <ahref="./ViewFs.html">ViewFS Guide</a>. If a user wants to continue use the same fs.defaultFS and wants to have more mount points, then mount link configurations should have the ViewFileSystemOverloadScheme initialized uri’s hostname as the mount table name. Example if fs.defaultFS is <code>hdfs://mycluster</code>, then the mount link configuration key name should be like in the following format <code>fs.viewfs.mounttable.*mycluster*.link.<mountLinkPath></code>. Even if the initialized fs uri has hostname:port, it will simply ignore the port number and only consider the hostname as the mount table name. We will discuss more example configurations in following sections. If there are no mount links configured with the initializing uri’s hostname as the mount table name, then it will automatically consider the current uri as fallback(<code>fs.viewfs.mounttable.*mycluster*.linkFallback</code>) target fs uri. If the initialized uri contains path part, it will consider only scheme and authority part, but not the path part. Example, if the initialized uri contains <code>hdfs://mycluster/data</code>, it will consider only <code>hdfs://mycluster</code> as fallback target fs uri. The path part <code>data</code> will be ignored.</p>
<p>Another important improvement with the ViewFileSystemOverloadScheme is, administrators need not copy the <code>mount-table.xml</code> configuration file to 1000s of client nodes. Instead, they can keep the mount-table configuration file in a Hadoop compatible file system. So, keeping the configuration file in a central place makes administrators life easier as they can update mount-table in single place.</p></section><section>
<h3><aname="Enabling_View_File_System_Overload_Scheme"></a>Enabling View File System Overload Scheme</h3>
<p>To use this class, the following configurations needed to be added in core-site.xml file.</p>
<p>Here <code><scheme></code> should be same as the uri-scheme configured in fs.defautFS. For example if fs.defaultFS was configured with <code>hdfs://mycluster</code>, then the above configuration would be like below:</p>
<p>If users want some of their existing cluster (<code>hdfs://cluster</code>) data to mount with hdfs(<code>hdfs://cluster</code>) and other object store clusters(<code>o3fs://bucket1.volume1.omhost/</code>, <code>s3a://bucket1/</code>), the following example configurations can show how to add mount links.</p>
<p>Let’s consider the following operations to understand where these operations will be delegated based on mount links.</p>
<p><i>Op1:</i> Create a file with the path <code>hdfs://cluster/user/fileA</code>, then physically this file will be created at <code>hdfs://cluster/user/fileA</code>. This delegation happened based on the first configuration parameter in above configurations. Here <code>/user</code> mapped to <code>hdfs://cluster/user/</code>.</p>
<p><i>Op2:</i> Create a file the path <code>hdfs://cluster/data/datafile</code>, then this file will be created at <code>o3fs://bucket1.volume1.omhost/data/datafile</code>. This delegation happened based on second configurations parameter in above configurations. Here <code>/data</code> was mapped with <code>o3fs://bucket1.volume1.omhost/data/</code>.</p>
<p><i>Op3:</i> Create a file with the path <code>hdfs://cluster/backup/data.zip</code>, then physically this file will be created at <code>s3a://bucket1/backup/data.zip</code>. This delegation happened based on the third configuration parameter in above configurations. Here <code>/backup</code> was mapped to <code>s3a://bucket1/backup/</code>.</p>
<p><b>Example 2:</b></p>
<p>If users want some of their existing cluster (<code>s3a://bucketA/</code>) data to mount with other hdfs cluster(<code>hdfs://cluster</code>) and object store clusters(<code>o3fs://bucket1.volume1.omhost/</code>, <code>s3a://bucketA/</code>), the following example configurations can show how to add mount links.</p>
<p>Let’s consider the following operations to understand to where these operations will be delegated based on mount links.</p>
<p><i>Op1:</i> Create a file with the path <code>s3a://bucketA/user/fileA</code>, then this file will be created physically at <code>hdfs://cluster/user/fileA</code>. This delegation happened based on the first configuration parameter in above configurations. Here <code>/user</code> mapped to <code>hdfs://cluster/user</code>.</p>
<p><i>Op2:</i> Create a file the path <code>s3a://bucketA/data/datafile</code>, then this file will be created at <code>o3fs://bucket1.volume1.omhost/data/datafile</code>. This delegation happened based on second configurations parameter in above configurations. Here <code>/data</code> was mapped with <code>o3fs://bucket1.volume1.omhost/data/</code>.</p>
<p><i>Op3:</i> Create a file with the path <code>s3a://bucketA/salesDB/dbfile</code>, then physically this file will be created at <code>s3a://bucketA/salesDB/dbfile</code>. This delegation happened based on the third configuration parameter in above configurations. Here <code>/salesDB</code> was mapped to <code>s3a://bucket1/salesDB</code>.</p>
<p>Note: In above examples we used create operation only, but the same mechanism applies to any other file system APIs here.</p>
<p>The following picture shows how the different schemes can be used in ViewFileSystemOverloadScheme compared to the ViewFileSystem.</p>
<p>Note: In ViewFsOverloadScheme, by default the mount links will not be represented as symlinks. The permission bits and isDirectory value will be propagated from the target directory/file.</p></section><section>
<h3><aname="Central_Mount_Table_Configurations"></a>Central Mount Table Configurations</h3>
<p>To enable central mount table configuration, we need to configure <code>fs.viewfs.mounttable.path</code> in <code>core-site.xml</code> with the value as the Hadoop compatible file system directory/file path, where the <code>mount-table.<versionNumber>.xml</code> file copied. Here versionNumber is an integer number and need to increase the version number and upload new file in same directory.</p>
<p>The ViewFileSystemOverloadScheme always loads the highest version number <code>mount-table.<versionNumber>.xml</code>. Please don’t replace the file with same name. Always increment the version number to take new file picked by newly initializing clients. Why we don’t recommend to replace the files is that, some client might have already opened the connections to old mount-table files already and in middle of loading configuration files, and replacing files can make them fail.</p>
<p>If you are sure, you will never do updates to mount-table file, you can also configure file path directly like below. If you configure file path, it will not check any highest version number loading. Whatever file configured it will be loaded. However file name format should be same.</p>
<p>Note: we recommend not to configure mount-links in <code>core-site.xml</code> if you configure above valid path. Otherwise both mount links will be mixed and can lead to a confused behavior.</p>
<p>If you copy the <code>mount-table.<versionNumber>.xml</code>, you may consider having big replication factor depending on your cluster size. So, that file will be available locally to majority of clients as applications(MR/YARN/HBASE..etc) use locality on HDFS when reading <code>mount-table.<versionNumber>.xml</code>.</p></section></section><section>
<h2><aname="DFSAdmin_commands_with_View_File_System_Overload_Scheme"></a>DFSAdmin commands with View File System Overload Scheme</h2>
<p>Please refer to the <ahref="./HDFSCommands.html#dfsadmin_with_ViewFsOverloadScheme">HDFSCommands Guide</a></p></section><section>
<h2><aname="Accessing_paths_without_authority"></a>Accessing paths without authority</h2>
<p>Accessing paths like <code>hdfs:///foo/bar</code>, <code>hdfs:/foo/bar</code> or <code>viewfs:/foo/bar</code>, where the authority (cluster name or hostname) of the path is not specified, is very common. This is especially true when the same code is expected to run on multiple clusters with different names or HDFS Namenodes.</p>
<p>When <code>ViewFileSystemOverloadScheme</code> is used (as described above), and if (a) the scheme of the path being accessed is different from the scheme of the path specified as <code>fs.defaultFS</code> and (b) if the path doesn’t have an authority specified, accessing the path can result in an error like <code>Empty Mount table in config for viewfs://default/</code>. For example, when the following configuration is used but a path like <code>viewfs:/foo/bar</code> or <code>viewfs:///foo/bar</code> is accessed, such an error arises.</p>
<p>To avoid the above problem, the configuration <code>fs.viewfs.mounttable.default.name.key</code> has to be set to the name of the cluster, i.e, the following should be added to <code>core-site.xml</code></p>
<p>The string in this configuration <code>cluster</code> should match the name of the authority in the value of <code>fs.defaultFS</code>. Further, the configuration should have a mount table configured correctly as in the above examples, i.e., the configurations <code>fs.viewfs.mounttable.*cluster*.link.<mountLinkPath></code> should be set (note the same string <code>cluster</code> is used in these configurations).</p></section></section><section>
<h2><aname="Appendix:_A_Mount_Table_Configuration_with_XInclude"></a>Appendix: A Mount Table Configuration with XInclude</h2>
<p>If users have a HTTP server in trusted network and don’t need authentication mechanism to it, you can also place your mount-table.xml file in that server and configure XInclude xml tag with <code>mount-table.xml</code> file.</p>
<p>The Apache Hadoop configuration has the capability to read the http urls from XInclude and load into configurations. If you choose this option, please don’t configure mount-table configuration items in <code>core-site.xml</code> or at <code>fs.viewfs.mounttable.path</code>. Please note, Hadoop configuration XInclude does not use SPNego authentication when opening url. So, this will not work if http server where you placed <code>mount-table.xml</code> needs authentication.</p></section>
