Add javadoc on how BBIOEngine works (by Anoop Sam John)
This commit is contained in:
parent
7073f69993
commit
891569bbd9
|
@ -31,7 +31,38 @@ import org.apache.hadoop.hbase.util.ByteBufferArray;
|
|||
|
||||
/**
|
||||
* IO engine that stores data in memory using an array of ByteBuffers
|
||||
* {@link ByteBufferArray}
|
||||
* {@link ByteBufferArray}.
|
||||
*
|
||||
*<h2>How it Works</h2>
|
||||
* First, see {@link ByteBufferArray} and how it gives a view across multiple ByteBuffers managed
|
||||
* by it internally. This class does the physical BB create and the write and read to the
|
||||
* underlying BBs. So we will create N BBs based on the total BC capacity specified on create
|
||||
* of the ByteBufferArray. So say we have 10 GB of off heap BucketCache, we will create 2560 such
|
||||
* BBs inside our ByteBufferArray.
|
||||
*
|
||||
* <p>Now the way BucketCache works is that the entire 10 GB is split into diff sized buckets: by
|
||||
* default from 5 KB to 513 KB. Within each bucket of a particular size, there are
|
||||
* usually more than one bucket 'block'. The way it is calculate in bucketcache is that the total
|
||||
* bucketcache size is divided by 4 (hard-coded currently) * max size option. So using defaults,
|
||||
* buckets will be is 4 * 513kb (the biggest default value) = 2052kb. A bucket of 2052kb at offset
|
||||
* zero will serve out bucket 'blocks' of 5kb, the next bucket will do the next size up and so on
|
||||
* up to the maximum (default) of 513kb).
|
||||
*
|
||||
* <p>When we write blocks to the bucketcache, we will see which bucket size group it best fits.
|
||||
* So a 4 KB block size goes to the 5 KB size group. Each of the block writes, writes within its
|
||||
* appropriate bucket. Though the bucket is '4kb' in size, it will occupy one of the
|
||||
* 5 KB bucket 'blocks' (even if actual size of the bucket is less). Bucket 'blocks' will not span
|
||||
* buckets.
|
||||
*
|
||||
* <p>But you can see the physical memory under the bucket 'blocks' can be split across the
|
||||
* underlying backing BBs from ByteBufferArray. All is split into 4 MB sized BBs.
|
||||
*
|
||||
* <p>Each Bucket knows its offset in the entire space of BC and when block is written the offset
|
||||
* arrives at ByteBufferArray and it figures which BB to write to. It may so happen that the entire
|
||||
* block to be written does not fit a particular backing ByteBufferArray so the remainder goes to
|
||||
* another BB. See {@link ByteBufferArray#putMultiple(long, int, byte[])}.
|
||||
|
||||
So said all these, when we read a block it may be possible that the bytes of that blocks is physically placed in 2 adjucent BBs. In such case also, we avoid any copy need by having the MBB...
|
||||
*/
|
||||
@InterfaceAudience.Private
|
||||
public class ByteBufferIOEngine implements IOEngine {
|
||||
|
|
Loading…
Reference in New Issue