Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HDFS-15047. Document the new decommission monitor (HDFS-14854). (#1755)

This commit is contained in:
Masatake Iwasaki 2019-12-13 00:35:41 +09:00 committed by GitHub
parent 0e28cd8f63
commit bdd00f10b4
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 29 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

29
hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HdfsDataNodeAdminGuide.md
View File

@ -148,6 +148,35 @@ dfs.namenode.decommission.blocks.per.interval
dfs.namenode.decommission.max.concurrent.tracked.nodes dfs.namenode.decommission.max.concurrent.tracked.nodes
``` ```
Backing-off Decommission Monitor (experimental)
------------
The original decommissioning algorithm has issues when DataNodes having lots of
blocks are decommissioned such as
* Write lock in the NameNode could be held for a long time for queueing re-replication.
* Re-replication work progresses node by node if there are multiple decommissioning DataNodes.
[HDFS-14854](https://issues.apache.org/jira/browse/HDFS-14854) introduced
new decommission monitor in order to mitigate those issues.
This feature is currently marked as experimental and disabled by default.
You can enable this by setting the value of
`dfs.namenode.decommission.monitor.class` to
`org.apache.hadoop.hdfs.server.blockmanagement.DatanodeAdminBackoffMonitor`
in hdfs-site.xml.
The relevant configuration properties are listed in the table below.
Please refer to [hdfs-default.xml](./hdfs-default.xml)
for descriptions and default values.
| Property |
|:-------- |
| `dfs.namenode.decommission.monitor.class` |
| `dfs.namenode.decommission.backoff.monitor.pending.limit` |
| `dfs.namenode.decommission.backoff.monitor.pending.blocks.per.lock` |
Metrics Metrics
----------- -----------