HDFS-12973. RBF: Document global quota supporting in federation. Contributed by Yiqun Lin.

This commit is contained in:
Yiqun Lin 2018-01-19 14:18:19 +08:00
parent bc93ac229e
commit 9e4f52d323
3 changed files with 35 additions and 1 deletions

View File

@ -84,7 +84,8 @@ public class RouterAdmin extends Configured implements Tool {
+ "[-readonly] -owner <owner> -group <group> -mode <mode>]\n"
+ "\t[-rm <source>]\n"
+ "\t[-ls <path>]\n"
+ "\t[-setQuota <path> -ns <nsQuota> -ss <ssQuota>]\n"
+ "\t[-setQuota <path> -nsQuota <nsQuota> -ssQuota "
+ "<quota in bytes or quota size string>]\n"
+ "\t[-clrQuota <path>\n";
System.out.println(usage);
}

View File

@ -428,12 +428,16 @@ Usage:
[-add <source> <nameservice> <destination> [-readonly] -owner <owner> -group <group> -mode <mode>]
[-rm <source>]
[-ls <path>]
[-setQuota <path> -nsQuota <nsQuota> -ssQuota <quota in bytes or quota size string>]
[-clrQuota <path>]
| COMMAND\_OPTION | Description |
|:---- |:---- |
| `-add` *source* *nameservice* *destination* | Add a mount table entry or update if it exists. |
| `-rm` *source* | Remove mount point of specified path. |
| `-ls` *path* | List mount points under specified path. |
| `-setQuota` *path* `-nsQuota` *nsQuota* `-ssQuota` *ssQuota* | Set quota for specified path. See [HDFS Quotas Guide](./HdfsQuotaAdminGuide.html) for the quota detail. |
| `-clrQuota` *path* | Clear quota of given mount point. See [HDFS Quotas Guide](./HdfsQuotaAdminGuide.html) for the quota detail. |
The commands for managing Router-based federation. See [Mount table management](./HDFSRouterFederation.html#Mount_table_management) for more info.

View File

@ -127,6 +127,11 @@ Examples users may encounter include the following.
* Copy file/folder in two different nameservices.
* Write into a file/folder being rebalanced.
### Quota management
Federation supports and controls global quota at mount table level.
For performance reasons, the Router caches the quota usage and updates it periodically. These quota usage values
will be used for quota-verification during each WRITE RPC call invoked in RouterRPCSever. See [HDFS Quotas Guide](./HdfsQuotaAdminGuide.html)
for the quota detail.
### State Store
The (logically centralized, but physically distributed) State Store maintains:
@ -199,6 +204,21 @@ Mount table permission can be set by following command:
The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
Router-based federation supports global quota at mount table level. Mount table entries may spread multiple subclusters and the global quota will be
accounted across these subclusters.
The federation admin tool supports setting quotas for specified mount table entries:
[hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -setQuota /path -nsQuota 100 -ssQuota 1024
The above command means that we allow the path to have a maximum of 100 file/directories and use at most 1024 bytes storage space. The parameter for *ssQuota*
supports multiple size-unit suffix (e.g. 1k is 1KB, 5m is 5MB). If no suffix is specified then bytes is assumed.
Ls command will show below information for each mount table entry:
Source Destinations Owner Group Mode Quota/Usage
/path ns0->/path root supergroup rwxr-xr-x [NsQuota: 50/0, SsQuota: 100 B/0 B]
Client configuration
--------------------
@ -322,6 +342,15 @@ Monitor the namenodes in the subclusters for forwarding the client requests.
| dfs.federation.router.monitor.namenode | | The identifier of the namenodes to monitor and heartbeat. |
| dfs.federation.router.monitor.localnamenode.enable | `true` | If `true`, the Router should monitor the namenode in the local machine. |
### Quota
Global quota supported in federation.
| Property | Default | Description|
|:---- |:---- |:---- |
| dfs.federation.router.quota.enable | `false` | If `true`, the quota system enabled in the Router. |
| dfs.federation.router.quota-cache.update.interval | 60s | How often the Router updates quota cache. This setting supports multiple time unit suffixes. If no suffix is specified then milliseconds is assumed. |
Metrics
-------