HDFS-12973. RBF: Document global quota supporting in federation. Contributed by Yiqun Lin.
This commit is contained in:
parent
bc93ac229e
commit
9e4f52d323
|
@ -84,7 +84,8 @@ public class RouterAdmin extends Configured implements Tool {
|
||||||
+ "[-readonly] -owner <owner> -group <group> -mode <mode>]\n"
|
+ "[-readonly] -owner <owner> -group <group> -mode <mode>]\n"
|
||||||
+ "\t[-rm <source>]\n"
|
+ "\t[-rm <source>]\n"
|
||||||
+ "\t[-ls <path>]\n"
|
+ "\t[-ls <path>]\n"
|
||||||
+ "\t[-setQuota <path> -ns <nsQuota> -ss <ssQuota>]\n"
|
+ "\t[-setQuota <path> -nsQuota <nsQuota> -ssQuota "
|
||||||
|
+ "<quota in bytes or quota size string>]\n"
|
||||||
+ "\t[-clrQuota <path>\n";
|
+ "\t[-clrQuota <path>\n";
|
||||||
System.out.println(usage);
|
System.out.println(usage);
|
||||||
}
|
}
|
||||||
|
|
|
@ -428,12 +428,16 @@ Usage:
|
||||||
[-add <source> <nameservice> <destination> [-readonly] -owner <owner> -group <group> -mode <mode>]
|
[-add <source> <nameservice> <destination> [-readonly] -owner <owner> -group <group> -mode <mode>]
|
||||||
[-rm <source>]
|
[-rm <source>]
|
||||||
[-ls <path>]
|
[-ls <path>]
|
||||||
|
[-setQuota <path> -nsQuota <nsQuota> -ssQuota <quota in bytes or quota size string>]
|
||||||
|
[-clrQuota <path>]
|
||||||
|
|
||||||
| COMMAND\_OPTION | Description |
|
| COMMAND\_OPTION | Description |
|
||||||
|:---- |:---- |
|
|:---- |:---- |
|
||||||
| `-add` *source* *nameservice* *destination* | Add a mount table entry or update if it exists. |
|
| `-add` *source* *nameservice* *destination* | Add a mount table entry or update if it exists. |
|
||||||
| `-rm` *source* | Remove mount point of specified path. |
|
| `-rm` *source* | Remove mount point of specified path. |
|
||||||
| `-ls` *path* | List mount points under specified path. |
|
| `-ls` *path* | List mount points under specified path. |
|
||||||
|
| `-setQuota` *path* `-nsQuota` *nsQuota* `-ssQuota` *ssQuota* | Set quota for specified path. See [HDFS Quotas Guide](./HdfsQuotaAdminGuide.html) for the quota detail. |
|
||||||
|
| `-clrQuota` *path* | Clear quota of given mount point. See [HDFS Quotas Guide](./HdfsQuotaAdminGuide.html) for the quota detail. |
|
||||||
|
|
||||||
The commands for managing Router-based federation. See [Mount table management](./HDFSRouterFederation.html#Mount_table_management) for more info.
|
The commands for managing Router-based federation. See [Mount table management](./HDFSRouterFederation.html#Mount_table_management) for more info.
|
||||||
|
|
||||||
|
|
|
@ -127,6 +127,11 @@ Examples users may encounter include the following.
|
||||||
* Copy file/folder in two different nameservices.
|
* Copy file/folder in two different nameservices.
|
||||||
* Write into a file/folder being rebalanced.
|
* Write into a file/folder being rebalanced.
|
||||||
|
|
||||||
|
### Quota management
|
||||||
|
Federation supports and controls global quota at mount table level.
|
||||||
|
For performance reasons, the Router caches the quota usage and updates it periodically. These quota usage values
|
||||||
|
will be used for quota-verification during each WRITE RPC call invoked in RouterRPCSever. See [HDFS Quotas Guide](./HdfsQuotaAdminGuide.html)
|
||||||
|
for the quota detail.
|
||||||
|
|
||||||
### State Store
|
### State Store
|
||||||
The (logically centralized, but physically distributed) State Store maintains:
|
The (logically centralized, but physically distributed) State Store maintains:
|
||||||
|
@ -199,6 +204,21 @@ Mount table permission can be set by following command:
|
||||||
|
|
||||||
The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
|
The option mode is UNIX-style permissions for the mount table. Permissions are specified in octal, e.g. 0755. By default, this is set to 0755.
|
||||||
|
|
||||||
|
Router-based federation supports global quota at mount table level. Mount table entries may spread multiple subclusters and the global quota will be
|
||||||
|
accounted across these subclusters.
|
||||||
|
|
||||||
|
The federation admin tool supports setting quotas for specified mount table entries:
|
||||||
|
|
||||||
|
[hdfs]$ $HADOOP_HOME/bin/hdfs dfsrouteradmin -setQuota /path -nsQuota 100 -ssQuota 1024
|
||||||
|
|
||||||
|
The above command means that we allow the path to have a maximum of 100 file/directories and use at most 1024 bytes storage space. The parameter for *ssQuota*
|
||||||
|
supports multiple size-unit suffix (e.g. 1k is 1KB, 5m is 5MB). If no suffix is specified then bytes is assumed.
|
||||||
|
|
||||||
|
Ls command will show below information for each mount table entry:
|
||||||
|
|
||||||
|
Source Destinations Owner Group Mode Quota/Usage
|
||||||
|
/path ns0->/path root supergroup rwxr-xr-x [NsQuota: 50/0, SsQuota: 100 B/0 B]
|
||||||
|
|
||||||
Client configuration
|
Client configuration
|
||||||
--------------------
|
--------------------
|
||||||
|
|
||||||
|
@ -322,6 +342,15 @@ Monitor the namenodes in the subclusters for forwarding the client requests.
|
||||||
| dfs.federation.router.monitor.namenode | | The identifier of the namenodes to monitor and heartbeat. |
|
| dfs.federation.router.monitor.namenode | | The identifier of the namenodes to monitor and heartbeat. |
|
||||||
| dfs.federation.router.monitor.localnamenode.enable | `true` | If `true`, the Router should monitor the namenode in the local machine. |
|
| dfs.federation.router.monitor.localnamenode.enable | `true` | If `true`, the Router should monitor the namenode in the local machine. |
|
||||||
|
|
||||||
|
### Quota
|
||||||
|
|
||||||
|
Global quota supported in federation.
|
||||||
|
|
||||||
|
| Property | Default | Description|
|
||||||
|
|:---- |:---- |:---- |
|
||||||
|
| dfs.federation.router.quota.enable | `false` | If `true`, the quota system enabled in the Router. |
|
||||||
|
| dfs.federation.router.quota-cache.update.interval | 60s | How often the Router updates quota cache. This setting supports multiple time unit suffixes. If no suffix is specified then milliseconds is assumed. |
|
||||||
|
|
||||||
Metrics
|
Metrics
|
||||||
-------
|
-------
|
||||||
|
|
||||||
|
|
Loading…
Reference in New Issue