From dbdfd8e8d12a6841ce00896a7bf59a4ed4f3b9a6 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Fri, 18 Dec 2015 12:40:00 -0800 Subject: [PATCH] HBASE-13129 Add troubleshooting hints around WAL retention from replication Signed-off-by: Jonathan Hsieh Signed-off-by: Sean Busbey --- src/main/asciidoc/_chapters/ops_mgt.adoc | 7 ++++++- src/main/asciidoc/_chapters/troubleshooting.adoc | 11 +++++++---- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/src/main/asciidoc/_chapters/ops_mgt.adoc b/src/main/asciidoc/_chapters/ops_mgt.adoc index 53aee3303fa..6a104933e7b 100644 --- a/src/main/asciidoc/_chapters/ops_mgt.adoc +++ b/src/main/asciidoc/_chapters/ops_mgt.adoc @@ -1361,7 +1361,10 @@ list_peers:: list all replication relationships known by this cluster enable_peer :: Enable a previously-disabled replication relationship disable_peer :: - Disable a replication relationship. HBase will no longer send edits to that peer cluster, but it still keeps track of all the new WALs that it will need to replicate if and when it is re-enabled. + Disable a replication relationship. HBase will no longer send edits to that + peer cluster, but it still keeps track of all the new WALs that it will need + to replicate if and when it is re-enabled. WALs are retained when enabling or disabling + replication as long as peers exist. remove_peer :: Disable and remove a replication relationship. HBase will no longer send edits to that peer cluster or keep track of WALs. enable_table_replication :: @@ -1501,6 +1504,8 @@ The default behavior is augmented so that if a log is past its TTL, the cleaning If the log is not found in any queues, the log will be deleted. The next time the cleaning process needs to look for a log, it starts by using its cached list. +NOTE: WALs are saved when replication is enabled or disabled as long as peers exist. + [[rs.failover.details]] ==== Region Server Failover diff --git a/src/main/asciidoc/_chapters/troubleshooting.adoc b/src/main/asciidoc/_chapters/troubleshooting.adoc index 8b2011d4add..fc9aadb1148 100644 --- a/src/main/asciidoc/_chapters/troubleshooting.adoc +++ b/src/main/asciidoc/_chapters/troubleshooting.adoc @@ -859,11 +859,14 @@ Snapshots:: HBase Shell commands for managing them. For more information, see <>. WAL:: - Write-ahead logs (WALs) are stored in subdirectories of `/hbase/.logs/`, depending - on their status. Already-processed WALs are stored in `/hbase/.logs/oldWALs/` and - corrupt WALs are stored in `/hbase/.logs/.corrupt/` for examination. - If the size of any subdirectory of `/hbase/.logs/` is growing, examine the HBase + Write-ahead logs (WALs) are stored in subdirectories of the HBase root directory, + typically `/hbase/`, depending on their status. Already-processed WALs are stored + in `/hbase/oldWALs/` and corrupt WALs are stored in `/hbase/.corrupt/` for examination. + If the size of one of these subdirectories is growing, examine the HBase server logs to find the root cause for why WALs are not being processed correctly. ++ +If you use replication and `/hbase/oldWALs/` is using more space than you expect, +remember that WALs are saved when replication is disabled, as long as there are peers. *Do not* manage WALs manually via HDFS.