Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HBASE-4577 Region server reports storefileSizeMB bigger than storefileUncompressedSizeMB 

git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1201106 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Michael Stack 2011-11-11 23:40:29 +00:00
parent 902cd7e192
commit 3e37baa9f3
3 changed files with 12 additions and 52 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

54
CHANGES.txt
View File

@ -1,55 +1,7 @@
HBase Change Log
Release 0.93.0 - Unreleased
NEW FEATURE
HBASE-4460 Support running an embedded ThriftServer within a RegionServer (jgray)
HBASE-4536 Allow CF to retain deleted rows (Lars H)
HBASE-4629 Enable automated patch testing for hbase (Giridharan Kesavan)
HBASE-4528 The put operation can release the rowlock before sync-ing the
Hlog (dhruba via jgray)
IMPROVEMENT
HBASE-4132 Extend the WALActionsListener API to accomodate log archival
(dhruba borthakur)
HBASE-4461 Expose getRowOrBefore via Thrift (jgray)
HBASE-4433 avoid extra next (potentially a seek) if done with column/row
(kannan via jgray)
HBASE-4131 Make the Replication Service pluggable via a standard
interface definition (dhruba via jgray)
HBASE-4477 Ability for an application to store metadata into the
transaction log (dhruba via jgray)
HBASE-4145 Provide metrics for hbase client (Ming Ma)
HBASE-4465 Lazy-seek optimization for StoreFile scanners (mikhail/liyin)
HBASE-4422 Move block cache parameters and references into single
CacheConf class (jgray)
HBASE-4102 atomicAppend: A put that appends to the latest version of a cell (Lars H)
HBASE-4469 Avoid top row seek by looking up bloomfilter (liyin via jgray)
HBASE-4418 Show all the hbase configuration in the web ui
HBASE-4489 Better key splitting in RegionSplitter
HBASE-4532 Avoid top row seek by dedicated bloom filter for delete family bloom filter
HBASE-4626 Filters unnecessarily copy byte arrays (Lars H)
HBASE-4691 Remove more unnecessary byte[] copies from KeyValues (Lars H)
HBASE-4669 Add an option of using round-robin assignment for enabling table
(Jieshan Bean)
HBASE-4696 HRegionThriftServer' might have to indefinitely do redirtects (jgray)
HBASE-1744 Thrift server to match the new java api (Tim Sell)
HBASE-4746 Use a random ZK client port in unit tests so we can run them in parallel
(Mikhail Bautin)
HBASE-4737 Categorize the tests into small/medium/large; allow small tests to be run
in parallel within a single JVM (N Keywal)
BUG FIXES
HBASE-4488 Store could miss rows during flush (Lars H via jgray)
HBASE-4673 NPE in HFileReaderV2.close during major compaction when
hfile.block.cache.size is set to 0 (Lars H)
HBASE-4751 Make TestAdmin#testEnableTableRoundRobinAssignment friendly to
concurrent tests (Jieshan Bean)
TESTS
HBASE-4534 A new unit test for lazy seek and StoreScanner in general
(mikhail via jgray)
HBASE-4545 TestHLog doesn't clean up after itself
HBASE-4690 Intermittent TestRegionServerCoprocessorExceptionWithAbort failure
HBASE-4480 Testing script to simplify local testing (Scott Kuehn)
*DO NOT ADD ISSUES HERE ON COMMIT ANY MORE. WE'LL GENERATE THE LIST
FROM JIRA INSTEAD WHEN WE MAKE A RELEASE*
Release 0.92.0 - Unreleased
INCOMPATIBLE CHANGES
@ -470,6 +422,8 @@ Release 0.92.0 - Unreleased
HBASE-4723 Loads of NotAllMetaRegionsOnlineException traces when starting
the master
HBASE-4511 There is data loss when master failovers
HBASE-4577 Region server reports storefileSizeMB bigger than
storefileUncompressedSizeMB (gaojinchao)
TESTS
HBASE-4450 test for number of blocks read: to serve as baseline for expected

2
src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java
View File

@ -266,7 +266,7 @@ public abstract class AbstractHFileWriter implements HFile.Writer {
*/
protected void finishClose(FixedFileTrailer trailer) throws IOException {
trailer.setMetaIndexCount(metaNames.size());
trailer.setTotalUncompressedBytes(totalUncompressedBytes);
trailer.setTotalUncompressedBytes(totalUncompressedBytes+ trailer.getTrailerSize());
trailer.setEntryCount(entryCount);
trailer.setCompressionCodec(compressAlgo);

8
src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java
View File

@ -238,6 +238,7 @@ public class HFileWriterV2 extends AbstractHFileWriter {
fsBlockWriter.writeHeaderAndData(outputStream);
ibw.blockWritten(offset, fsBlockWriter.getOnDiskSizeWithHeader(),
fsBlockWriter.getUncompressedSizeWithoutHeader());
totalUncompressedBytes += fsBlockWriter.getUncompressedSizeWithHeader();
if (cacheThisBlock) {
// Cache this block on write.
@ -390,6 +391,7 @@ public class HFileWriterV2 extends AbstractHFileWriter {
metaData.get(i).write(dos);
fsBlockWriter.writeHeaderAndData(outputStream);
totalUncompressedBytes += fsBlockWriter.getUncompressedSizeWithHeader();
// Add the new meta block to the meta index.
metaBlockIndexWriter.addEntry(metaNames.get(i), offset,
@ -413,15 +415,19 @@ public class HFileWriterV2 extends AbstractHFileWriter {
metaBlockIndexWriter.writeSingleLevelIndex(fsBlockWriter.startWriting(
BlockType.ROOT_INDEX, false), "meta");
fsBlockWriter.writeHeaderAndData(outputStream);
totalUncompressedBytes += fsBlockWriter.getUncompressedSizeWithHeader();
// File info
writeFileInfo(trailer, fsBlockWriter.startWriting(BlockType.FILE_INFO,
false));
fsBlockWriter.writeHeaderAndData(outputStream);
totalUncompressedBytes += fsBlockWriter.getUncompressedSizeWithHeader();
// Load-on-open data supplied by higher levels, e.g. Bloom filters.
for (BlockWritable w : additionalLoadOnOpenData)
for (BlockWritable w : additionalLoadOnOpenData){
fsBlockWriter.writeBlock(w, outputStream);
totalUncompressedBytes += fsBlockWriter.getUncompressedSizeWithHeader();
}
// Now finish off the trailer.
trailer.setNumDataIndexLevels(dataBlockIndexWriter.getNumLevels());