M MultiRegionTable
Make deterministic by creating the regions directly and not rely on
the asychronous nature of cache flushes, compactions and splits. The
regions are small, but the point of this class is to generate a
table with multiple regions so we can test map / reduce, region
onlining / offlining, etc.
Removed PUNCTUATION from row keys. Not sure why it was there in the
first place, other than perhaps to verify that a row key can have
punctuation in it provided it is not the first character. This will
become moot when row keys change from Text to byte[] anyways.
Incorporate repeated code
{code}
region.close();
region.getLog().closeAndDelete();
{code}
into private method closeRegionAndDeleteLog
M TestSplit
extends HBaseClusterTestCase instead of MultiRegionTable. It didn't
use the output of MultiRegionTable, so all that work was just wasted
by this test.
M TestTableIndex, TestTableMapReduce
The only two tests that currently use MultiRegionTable. Minor
modifications needed because MultiRegionTable now handles starting
and stopping of the mini-DFS cluster. With the new MultiRegionTable
class, if these tests fail now it will be because something they are
testing has regressed and not because MultiRegionTable failed.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@652587 13f79535-47bb-0310-9956-ffa450edef68
Add a ChangedReadersObserver interface. HStore notifies registered
observers when list of HStoreFile Readers changes -- at flush time
and at compaction time. Scanners are only current observers.
Also fix a deadlock flushing by changing lock types and moving flush
request out from under lock.
M src/test/org/apache/hadoop/hbase/MultiRegionTable.java
(getCacheFlushListern): Renamed as getFlushRequester
M src/java/org/apache/hadoop/hbase/regionserver/HStore.java
Added synchronizations on this.storefiles.
(activeScanners, newScannerLock): Removed. We no longer try to
block out scanners when compacting (Turns out same functionality
for blocking scanners is up in HRegion used around closing/split).
(changedReadersObservers): Added.
(updateReaders): New method that manages the insertion of new
reader on flush. Also calls new notifyChangedReadersObservers.
(notifyChagnedReadersObservers, addChangedReadersObserver,
deleteChangedReadersObservers): Added.
(completeCompaction): Previous deleting old store files, we'd
remove from this.storefiles and delete all in one step; now we
do the remove first, notify all observers of readers, and then
do the delete so observers have a chance to clean up any old
references to files about to be deleted. Removed all the lockout
of new scanner creation and wait on old scanners to come int.
(updateActiveScanners): Removed.
(getStorefiles): Accessor. Added.
M src/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
Added implementation of new ChangedReadersObserver interface.
Added a lock that we hold when 'nexting' and when changing the
set of readers out from under the scanner.
Changed the constructor moving bulk into new openReaders method
that we reuse when list of Readers changes.
(next): Moved head of this method into new getNextViableRow (used
to be called chosenRow and chosenTimestamp). New method returns
simple datastructure of row and timestamp (ViableRow).
(close): On close, remove ourselves as ChangedReadersObserver (we
added ourselves in the constructor).
(updateReaders): Changed the set of Readers out from under the
Scanner.
A src/java/org/apache/hadoop/hbase/regionserver/ChangedReadersObserver.java
Added.
M src/java/org/apache/hadoop/hbase/regionserver/Flusher.java
Changed name of the interface we implement from CacheFlushListener to
FlushRequester.
D src/java/org/apache/hadoop/hbase/regionserver/CacheFlushListener.java
Renamed as FlushRequester.
M src/java/org/apache/hadoop/hbase/regionserver/HStoreScanner.java
Remove update of activeScanners.
A src/java/org/apache/hadoop/hbase/regionserver/FlushRequester.java
Added. Rename of CacheFlushListener.
M src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Renamed method getCacheFlushListener as getFlushRequester.
M src/java/org/apache/hadoop/hbase/regionserver/HAbstractScanner.java
Formatting.
M src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
CacheFlushListener was renamed as FlushListener.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@651017 13f79535-47bb-0310-9956-ffa450edef68
knows about workings of memcache
HStore knows about workings of memcache
This patch moves the running of mecache snapshots out of the control
of HRegion and hides the memcache details in HStore. This patch also
does a bunch of work on the merge tool fixing a bug in the metautils along
the way. The merge tool test was failing. We weren't setting into the
HLog the maximum sequence id after we'd opened a region -- as HRS does --
so were losing edits. On the way, refactored the merge tool test to get
rid of duplicated code. Finally, cleans up logging in HStore to aid
debugging; e.g. we always refer to the sequence id as the 'sequence id'
in log messages rather than as seqId or 'sequence record' so can sort
log as see state of sequence id transitions.
Version 2 changes the order in which things are run in memcache. 532
made it so flushing did snapshot and then cleared the snapshot. Now,
we snapshot before we flush a store, then inside in the store flush,
we call getSnapshot and then clearSnapshot.
M src/test/org/apache/hadoop/hbase/regionserver/TestHMemcache.java
How snapshotting changed. Change test in accordance.
M src/test/org/apache/hadoop/hbase/regionserver/TestHLog.java
Use accessor to get file number.
M src/test/org/apache/hadoop/hbase/util/TestMergeTool.java
Refactored to remove duplicated code so could tell what was going on.
(mergeAndVerify, verifyMerge): Addd.
M src/java/org/apache/hadoop/hbase/regionserver/Memcache.java
(snapshot): Changed so it no longer returns snapshot.
M src/java/org/apache/hadoop/hbase/regionserver/HStore.java
Changed log messages removing the useless and adding info to others.
(this.maxSeqId): We used to add 1 to this in here in HStore. Let
HRegion do it. Its the one that does the machinations w/ sequenceids
anyways. Make flushes return the amount flushed. Use this updating
the regions memcacheSize accounting.
(snapshot): Added method for the region to call.
M src/java/org/apache/hadoop/hbase/regionserver/HLog.java
Made data members private.
(getFileNum): Added accessor for tests.
M src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Added info to logs..
(snapshotMemcaches): removed.
(internalFlushcache): No longer takes startime. Internally now does
some of what used happen in snapshotMemcaches including sending of
message to stores to snapshot.
(getEntrySize): Added method for calculating size of an update. Used
by HRegion and flushing so both come up w/ same answer.
M src/java/org/apache/hadoop/hbase/util/Merge.java
Add logging of whats happening during merges and fail earlier than we
used if stuff is not right.
Renamed local variables from region1 to r1, etc., so didn't clash
with data members of same name.
M src/java/org/apache/hadoop/hbase/util/MetaUtils.java
Added a TODO
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@650298 13f79535-47bb-0310-9956-ffa450edef68
Found an issue in the 532 commit. The MemcacheScanner had a flipped isWildcardScanner
test; fixing it returned wrong columns because using okCols rather than the literals
passed in.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@648912 13f79535-47bb-0310-9956-ffa450edef68
-Removed HScannerInterface and HInternalScannerInterface
-Created new interfaces Scanner for clients and InternalScanner for internal consumers
-Internal and client scanners no longer share common interface
-Client scanner's next() method and iterables are in RowResults
-Updated tests and internal consumers to use Scanner in place of HScannerInterface
-HTable obtainScanner(*) are now renamed getScanner(*)
-Tests have ScannerIncommon to turn Scanners into InternalScanners for some tests
-Fixed a bug in HMaster that was eating TableExistsExceptions (unrelated)
-Updated TableInputFormat to provide RowResults instead of MapWritables
-Updated TableOutputFormat to take BatchUpdates instead of MapWritables
-Updated TableMap, TableReduce, and friends to correctly hook up to new input/output formats
HBASE-567 Reused BatchUpdate instances accumulate BatchOperations
- Fix to BatchUpdate that allows correct reuse of BatchUpdate instances (readFields didn't clear BatchOperation map)
- Update TestSerialization to prove above is fixed
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@646104 13f79535-47bb-0310-9956-ffa450edef68
Michael Stack
Jim Kellerman
Bryan Duxbury