From f261595ae0e7765fab19e30d8c9e365e8c3f377d Mon Sep 17 00:00:00 2001 From: Doug Meil Date: Mon, 26 Sep 2011 23:48:24 +0000 Subject: [PATCH] HBASE-4493 book.xml git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1176135 13f79535-47bb-0310-9956-ffa450edef68 --- src/docbkx/book.xml | 154 ++++++++++++++++++++++---------------------- 1 file changed, 77 insertions(+), 77 deletions(-) diff --git a/src/docbkx/book.xml b/src/docbkx/book.xml index baf4674b981..8530f14e0e8 100644 --- a/src/docbkx/book.xml +++ b/src/docbkx/book.xml @@ -1316,8 +1316,85 @@ HTable table2 = new HTable(conf2, "myTable"); +
+ Block Cache + The Block Cache contains three levels of block priority to allow for scan-resistance and in-memory ColumnFamilies. A block is added with an in-memory + flag if the containing ColumnFamily is defined in-memory, otherwise a block becomes a single access priority. Once a block is accessed again, it changes to multiple access. + This is used to prevent scans from thrashing the cache, adding a least-frequently-used element to the eviction algorithm. Blocks from in-memory ColumnFamilies + are the last to be evicted. + + + For more information, see the LruBlockCache source +
+
+ Write Ahead Log (WAL) + +
+ Purpose + + Each RegionServer adds updates (Puts, Deletes) to its write-ahead log (WAL) + first, and then to the for the affected . + This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure + before each MemStore is flushed and new StoreFiles are written. HLog + is the HBase WAL implementation, and there is one HLog instance per RegionServer. + The WAL is in HDFS in /hbase/.logs/ with subdirectories per region. + + For more general information about the concept of write ahead logs, see the Wikipedia + Write-Ahead Log article. + +
+
+ WAL Flushing + TODO (describe). + +
+ +
+ WAL Splitting + +
How edits are recovered from a crashed RegionServer + When a RegionServer crashes, it will lose its ephemeral lease in + ZooKeeper...TODO +
+
+ <varname>hbase.hlog.split.skip.errors</varname> + + When set to true, the default, any error + encountered splitting will be logged, the problematic WAL will be + moved into the .corrupt directory under the hbase + rootdir, and processing will continue. If set to + false, the exception will be propagated and the + split logged as failed. + See HBASE-2958 + When hbase.hlog.split.skip.errors is set to false, we fail the + split but thats it. We need to do more than just fail split + if this flag is set. + +
+ +
+ How EOFExceptions are treated when splitting a crashed + RegionServers' WALs + + If we get an EOF while splitting logs, we proceed with the split + even when hbase.hlog.split.skip.errors == + false. An EOF while reading the last log in the + set of files to split is near-guaranteed since the RegionServer likely + crashed mid-write of a record. But we'll continue even if we got an + EOF reading other than the last file in the set. + For background, see HBASE-2643 + Figure how to deal with eof splitting logs + +
+
+
+ + +
Regions This section is all about Regions. @@ -1499,83 +1576,6 @@ HTable table2 = new HTable(conf2, "myTable");
-
- Block Cache - The Block Cache contains three levels of block priority to allow for scan-resistance and in-memory ColumnFamilies. A block is added with an in-memory - flag if the containing ColumnFamily is defined in-memory, otherwise a block becomes a single access priority. Once a block is accessed again, it changes to multiple access. - This is used to prevent scans from thrashing the cache, adding a least-frequently-used element to the eviction algorithm. Blocks from in-memory ColumnFamilies - are the last to be evicted. - - - For more information, see the LruBlockCache source - -
- - -
- Write Ahead Log (WAL) - -
- Purpose - - Each RegionServer adds updates (Puts, Deletes) to its write-ahead log (WAL) - first, and then to the for the affected . - This ensures that HBase has durable writes. Without WAL, there is the possibility of data loss in the case of a RegionServer failure - before each MemStore is flushed and new StoreFiles are written. HLog - is the HBase WAL implementation, and there is one HLog instance per RegionServer. - The WAL is in HDFS in /hbase/.logs/ with subdirectories per region. - - For more general information about the concept of write ahead logs, see the Wikipedia - Write-Ahead Log article. - -
-
- WAL Flushing - TODO (describe). - -
- -
- WAL Splitting - -
How edits are recovered from a crashed RegionServer - When a RegionServer crashes, it will lose its ephemeral lease in - ZooKeeper...TODO -
-
- <varname>hbase.hlog.split.skip.errors</varname> - - When set to true, the default, any error - encountered splitting will be logged, the problematic WAL will be - moved into the .corrupt directory under the hbase - rootdir, and processing will continue. If set to - false, the exception will be propagated and the - split logged as failed. - See HBASE-2958 - When hbase.hlog.split.skip.errors is set to false, we fail the - split but thats it. We need to do more than just fail split - if this flag is set. - -
- -
- How EOFExceptions are treated when splitting a crashed - RegionServers' WALs - - If we get an EOF while splitting logs, we proceed with the split - even when hbase.hlog.split.skip.errors == - false. An EOF while reading the last log in the - set of files to split is near-guaranteed since the RegionServer likely - crashed mid-write of a record. But we'll continue even if we got an - EOF reading other than the last file in the set. - For background, see HBASE-2643 - Figure how to deal with eof splitting logs - -
-
-