From b29c731a57e2f37c4dd4401833ee2f29d0689ae3 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Mon, 10 Aug 2015 09:04:48 +1000 Subject: [PATCH] HBASE-14071 Document troubleshooting unexpected filesystem usage by snapshots and WALs --- .../asciidoc/_chapters/troubleshooting.adoc | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/src/main/asciidoc/_chapters/troubleshooting.adoc b/src/main/asciidoc/_chapters/troubleshooting.adoc index a2c9247530e..8ae61b4140d 100644 --- a/src/main/asciidoc/_chapters/troubleshooting.adoc +++ b/src/main/asciidoc/_chapters/troubleshooting.adoc @@ -842,6 +842,31 @@ Two common use-cases for querying HDFS for HBase objects is research the degree If there are a large number of StoreFiles for each ColumnFamily it could indicate the need for a major compaction. Additionally, after a major compaction if the resulting StoreFile is "small" it could indicate the need for a reduction of ColumnFamilies for the table. +=== Unexpected Filesystem Growth + +If you see an unexpected spike in filesystem usage by HBase, two possible culprits +are snapshots and WALs. + +Snapshots:: + When you create a snapshot, HBase retains everything it needs to recreate the table's + state at that time of tne snapshot. This includes deleted cells or expired versions. + For this reason, your snapshot usage pattern should be well-planned, and you should + prune snapshots that you no longer need. Snapshots are stored in `/hbase/.snapshots`, + and archives needed to restore snapshots are stored in + `/hbase/.archive/<_tablename_>/<_region_>/<_column_family_>/`. + + *Do not* manage snapshots or archives manually via HDFS. HBase provides APIs and + HBase Shell commands for managing them. For more information, see <>. + +WAL:: + Write-ahead logs (WALs) are stored in subdirectories of `/hbase/.logs/`, depending + on their status. Already-processed WALs are stored in `/hbase/.logs/oldWALs/` and + corrupt WALs are stored in `/hbase/.logs/.corrupt/` for examination. + If the size of any subdirectory of `/hbase/.logs/` is growing, examine the HBase + server logs to find the root cause for why WALs are not being processed correctly. + +*Do not* manage WALs manually via HDFS. + [[trouble.network]] == Network