M branches/0.1/src/java/org/apache/hadoop/hbase/util/MetaUtils.java
M trunk/src/java/org/apache/hadoop/hbase/util/MetaUtils.java
(changeOnlineStatus): Added.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@643486 13f79535-47bb-0310-9956-ffa450edef68
server reports that it is processing the open request
This is patch reviewed with Jim but with the number of edits between
reports made into a configurable.
Have the HRegionServer pass down a Progressable implementation down into
Region and then down int Store where edits are replayed. Call progress
after every couple of thousand edits.
M src/java/org/apache/hadoop/hbase/HStore.java
Take a Progessable in the constructor. Call it when applying edits.
M src/java/org/apache/hadoop/hbase/HMaster.java
Update commment around MSG_REPORT_PROCESS_OPEN so its expected
that we can get more than one of these messages during a region open.
M src/java/org/apache/hadoop/hbase/HRegion.java
New constructor that takes a Progressable. Pass it to Stores on construction.
M src/java/org/apache/hadoop/hbase/HRegionServer.java
On open of a region, pass in a Progressable that adds a
MSG_REPORT_PROCESS_OPEN every time its called.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@643223 13f79535-47bb-0310-9956-ffa450edef68
on eachiteration, edits are aggregated up into the millions
M src/java/org/apache/hadoop/hbase/HLog.java
(splitLog): If an exception processing a split, catch it.
In finally, close and delete the split. Don't try retrying.
While in some circumstance, we might recover, its also
likely that we just get same exception again. If so, and
multiple files, we'll just accumulate edits until the
kingdom comes.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@643142 13f79535-47bb-0310-9956-ffa450edef68
M src/java/org/apache/hadoop/hbase/HStore.java
(Constructor) If an exception out of reconstructionLog method, log it and
keep going. Presumption is that its result of a lack of HADOOP--1700.
(reconstructionLog): Check for empty log file.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@643110 13f79535-47bb-0310-9956-ffa450edef68
-Changed MiniHBaseCluster to not start up a MiniDFS
-Changed HBaseClusterTestCase to do the work of starting up a MiniDFS.
-Added pre and post setup method to HBaseClusterTestCase so you can control what happen before MiniHBaseCluster is booted up
-Converted AbstractMergeTestCase to be a HBaseClusterTestCase
-Converted any test that used a raw MIniDFS or MiniHBaseCluster to use HBaseClusterTestCase instead
-Split TestTimestamp into two tests - one for clientside (now in o.a.h.h.client) and one for serverside (o.a.h.h.regionserver)
-Merged in Stack's changes to make bin/hbase have hadoop jars first on the classpath
-Updated PerformanceEvaluation (in --miniCluster mode) to start up a DFS first
-Fixed a bug in BaseScanner that would have allowed NPEs to be generated
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@640526 13f79535-47bb-0310-9956-ffa450edef68
-Changed HStore and Memcache methods for computing closest row at or before
-Added more test cases for verifying this functionality
-Simplified the getClosestRowBefore interface so that it does not take timestamps
-Noted that getClosestRowBefore is assumed to work correctly ONLY on tables where updates are always with ascending timestamps (method is still not a part of HTable interface, so not available to clients)
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@638598 13f79535-47bb-0310-9956-ffa450edef68
M conf/hbase-default.xml
Add hbase.hbasemaster.maxregionopen property.
M src/java/org/apache/hadoop/hbase/HStore.java
Change way we log. Do way less. Just emit sums of edits applied
and skipped rather than individual edits.
M src/java/org/apache/hadoop/hbase/HRegionServer.java
Make sleeper instance a local rather than data member.
(reportForDuty): Take a sleeper instance.
(run): Removed redundant wrap of a 'for' by a 'while'.
(constructor): If IOE, do not offline the region. Seen to be
an overreaction.
M src/java/org/apache/hadoop/hbase/HLog.java
Don't output map of all files being cleaned everytime a new
entry is added; instead just log new entry. Remove emission
of every 10k edits.
M src/java/org/apache/hadoop/hbase/HMaster.java
Up default for maxregionopen. Was seeing that playing edits
could take a long time (mostly because we used log every
edit) but no harm in this being longer. On REPORT_CLOSE,
emit region info, not just region so can see the properties
(W/o, made it hard to figure who was responsible for offlining).
Add logging of attempt # in shutdown processing.
Add logging of state flags passed to the close region. Helps
debugging. Also in close offline ONLY if we are NOT reassigning
the region (jimk find).
M src/java/org/apache/hadoop/hbase/util/Sleeper.java
Add logging of extraordinary sleeps or calculated periods
(suspicion is that we're sleeping way longer on loaded machies
and the regionserver appears hung).
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@636849 13f79535-47bb-0310-9956-ffa450edef68
-Moved out classes Memcache, StoreFileScanner, MapFileCompactionReader, and HStoreScanner, and interface CompactionReader
-Updated TestHMemcache to use the correct type declaration.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@636815 13f79535-47bb-0310-9956-ffa450edef68
Summary of changes:
HMaster:
- When a row has an empty HRegionInfo (info:regioninfo), log it with the row name and and the other keys still in the row.
- Log the number of rows with empty HRegionInfo
- Delete the rows
- Make RowMap inner class static, change methods to have package scope to avoid synthetic accessors.
- Provide row name to getHRegionInfo so it can issue better log messages
- add method deleteEmptyMetaRows to remove rows with empty HRegionInfo
HRegion
- change removeRegionFromMETA to use deleteAll rather than using a BatchUpdate containing deletes for each cell.
TestEmptyMetaInfo
- new test case
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@636589 13f79535-47bb-0310-9956-ffa450edef68
HLog
- don't overwrite oldlogfile in splitLog if it already exists. Rename it and copy it into the new oldlogfile. Then delete it once it has been copied.
- use FileUtil.fullyDelete to delete region server log directory.
HRegion
- delete oldlogfile once it has been successfully processed
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@636396 13f79535-47bb-0310-9956-ffa450edef68
Other miscellaneous changes included:
IdentityTableReduce
- Added SuppressWarnings("unused") for reporter argument
- Removed unnecessary cast.
AbstractMergeTestBase
- Removed unnecessary compaction
StaticTestEnvironment
- Change logging level for client connections which are too noisy in most cases
TestBloomFilters
- Removed unnecessary config settings
- Modified to use BatchUpdate instead of deprecated startUpdate, etc.
TestScannerAPI
- Modified to use BatchUpdate instead of deprecated startUpdate, etc.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@630394 13f79535-47bb-0310-9956-ffa450edef68
(Recommit. We'd backed it out when it was HADOOP-1398)
This commit includes updating version from 0.1.0-dev to 0.2.0-dev
and an update of the hadoop lib to move it post 0.16.0 branch.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@619526 13f79535-47bb-0310-9956-ffa450edef68
HADOOP-2587 Splits blocked by compactions cause region to be offline for duration of compaction.
Fix bug in TestCompaction in which two mini dfs clusters were being started for the same test.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@611681 13f79535-47bb-0310-9956-ffa450edef68
HADOOP-2392, HADOOP-2324:
Chore
- initialChore() now returns boolean
HMaster
- rather than retry in root and meta scanners, return if a scan fails. It will get retried on the next scan. This has two effects: 1) scanners exit more quickly during shutdown and 2) they don't keep retrying to connect to a dead server, allowing them to recover from a server going down more quickly.
- initialScan in root and meta scanners return boolean and do not progress to maintenanceScan until the initial scan completes successfully.
HRegionServer
- speed up region server exit by reordering join's so that we join with threads in the order that we told them to stop
TestTableMapReduce
- remove overrides of heartbeat and thread wake intervals
HADOOP-2396:
HMaster
- move check for null HRegionInfo before first attempt to dereference it.
HADOOP-2397:
- HMaster$BaseScanner.checkAssigned: don't try to split dead server's log if initial startup has completed.
HADOOP-2353:
HMsg
- change toString() to only output the region name rather than calling HRegionInfo.toString()
StaticTestEnvironment
- make logging a bit less verbose
TestHLog
- was writing to local file system and failing on Windows
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@603428 13f79535-47bb-0310-9956-ffa450edef68
There are a lot of changes in this patch. The memcache has been changed from a per/region object to a per/column object, and HLocking has been removed since we do not have to maintain any locks across RPC calls.
This necessitated major changes to HRegion and HStore
Additionally there were many changes required to the unit tests since they tend to exploit some private interfaces that weren't designed to be public. Some of those interfaces changed so the test cases did as well.
This patch is the result of extensive analysis of the multiple threads in HBase that contend for shared resources: updates, reads, scanners, cache flushing, compaction and region splitting.
Many of the tests are timing sensitive, and since we tend to make "dormant" intervals as short as possible to speed up the Hudson build, we may go through several iterations of getting them right before Hudson is happy. This is especially true since two test cases failed on my dual cpu windows machine while running the tests under Ant, but ran fine under Eclipse.
However, now that the tests are passing locally, I believe the changes are doing the right thing, but may require some parameter tweaks.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@596835 13f79535-47bb-0310-9956-ffa450edef68
- Fix another race condition in processing dead servers,
- Fix error online meta regions: was using region name and not startKey as key for map.put.
- Change TestRegionServerExit to always kill the region server for the META region. This makes the test more deterministic and getting META reassigned was problematic.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@591880 13f79535-47bb-0310-9956-ffa450edef68
HLog.splitLog was generating incorrect file names, HRegion was generating file names that could be far too long especially for local file systems, HMaster had a race condition in which an old HLog would get split by two threads simultaneously.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@586680 13f79535-47bb-0310-9956-ffa450edef68
Set hbase.root in test/hbase-site.xml; when running a test, the default does not work consistantly.
When a HBase mini cluster is started on top of an existing mini dfs cluster, it should not shut down the mini dfs cluster when the mini HBase cluster is shut down.
TestDFSAbort catches exceptions, prints the stack trace and re-throws the exception, so you can see when the exception happened in the log.
Catch runtime exceptions that were escaping from FSUtils.isFileSystemAvailable, enabling more reliable detection of dfs failure. HRegionServer also now checks to see if it is still accepting client requests.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@580745 13f79535-47bb-0310-9956-ffa450edef68
of deletes. Added a deleteAll to remove all cells equal to or older than
passed timestamp. Fixed compaction so deleted cells do not make it out
into compacted output. Ensure also that versions > column max are dropped
compacting.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@574287 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
(addContents): Added overrides that allow specifying a timestamp.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestTimestamp.java
Made it so test inherits from HBaseTestCase instead of from HBaseClusterTestCase
so could add in tests that do not use cluster.
(testTimestampScanning): Added test for hadoop-1834 bug.
(testTimestamp): Refactored to remove duplicated code.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
(getNext): Make it respect the timestamp set on construction.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Removed eclipse yellow flag warnings around empty parens and
auto-boxing longs.
(getNext): Make it respect the timestamp set on construction.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@572980 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ConsoleTable.java
Make a PrintStream that outputs utf8. Have all printing use it.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/SelectCommand.java
Fix few places where we make Strings w/o stipulating UTF-8 encoding.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@571317 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestTableMapReduce.java
(localTestSingleRegionTable, localTestMultiRegionTable, verify): Added.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
Javadoc for addContents and Loader interface and implementations.
Methods have been made static so accessible w/o subclassing.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/MultiRegionTable.java
Guts of TestSplit has been moved here so other tests can have
access to a multiregion table.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Bulk moved to MultiRegionTable utility class. Use this new class
instead.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTable.java
Added '@deprecated' javadoc.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Was throwing RuntimeException when a msgQueue.put was interrupted
but this is a likely event on shutdown. Log a message instead.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/mapred/TableInputFormat.java
Actually fix for HADOOP-1785... reverse test of row comparison.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@570983 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Fix outputing fail message on each compaction though there was none.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
(rename): Refactor so return only happens on end..
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
(assignRegions): Make synchronized. In presence of concurrent visits
by regionservers, both visiting threads could grab same set of regions
for assignment.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@569589 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpContents.java
Add mention of new 'fs' operator.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/FsCommand.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HBaseShell.jj
Added support of new 'fs' operator.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserTokenManager.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserConstants.java
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/Parser.java
Generated files. Changes come of mods to HBaseShell.jj
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@569484 13f79535-47bb-0310-9956-ffa450edef68
result
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
Minor fix of a log message.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
(COMPACTION_DIR, WORKING_COMPACTION): Removed.
(compactdir): Renamed compactionDir.
Removed from constructor our checking if a compaction was left undone.
Instead, just ignore it. When compaction reruns whatever as left on
filesystem will just be cleaned up and we'll rerun the compaction
(Likelihood of a crash mid-compaction in exactly the area where
the compaction was recoverable are low -- more robust just redoing
the compaction from scratch).
(compactHelper): We were deleting HBaseRoot/compaction.tmp dir
after a compaction completed. Usually fine but on a cluster of
more than one machine, if two compactions were near-concurrent, one
machine could remove the compaction working directory while another
was mid-way through its compaction. Result was odd failures
during compaction of result file, during the move of the resulting
compacting file or subsequently trying to open reader on the
resulting compaction file (See HADOOP-1765).
a region fsck tool).
(getFilesToCompact): Added.
(processReadyCompaction): Added. Reorganized compaction so that the
window during which loss-of-data is possible is narrowed and even
then, we log a message with how a restore might be performed manually
(TODO: Add a repair tool).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
(rename): More checking around rename that it was successful.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLog.java
An empty-log gives HLog trouble. Added handling.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Cleanup of debug level logging.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Minor javadoc and changed a log from info to debug.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@569446 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Removed some empty lines so I can squeeze more code into a screenful.
(assignedRegions): Factored out some code into own methods so
this method is made a bit shorter. Added early returns near
top -- if nothing to assign, etc. -- so less nesting.
Added fix: Instead of iterating over unassignedRegions after
all the loadings have been calculated, instead iterate over
the locally calculated map, regionsToAssign (Otherwise, we
were running over the same territory each time through the
loop and were thus giving out same region multiple times).
(regionsPerServer, assignRegionsToOneServer,
getRegionsToAssign): Added.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@568404 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Use new HColumnDescriptor accessors rather than make direct accesses
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HColumnDescriptor.java
(COMPRESSION_NONE, COMPRESSION_RECORD, COMPRESSION_BLOCK): Removed.
Use enum ordinals instead. Removed mapping between these defines and
enum equivalents. Made data members private and added accessors.
(DEFAULT_IN_MEMORY, DEFAULT_COMPRESSION_TYPE,
DEFAULT_BLOOM_FILTER_DESCRIPTOR, DEFAULT_MAX_VALUE_LENGTH): Added.
M hbase/src/test/org/apache/hadoop/hbase/TestToString.java
Fix because enum strings are upper-case (was expecting lowercase).
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@568275 13f79535-47bb-0310-9956-ffa450edef68
Added handling for legal null value scanning META table and added
logging of unexpected exceptions that arise scanning.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Refactored to do a staged removal of daughter references.
(compact, recalibrate): Added.
(getSplitParent): Refactored as getSplitParentInfo.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
Added formatting of the find table result string so shorter
(when 30-odd regions fills page with its output).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTable.java
Formatting to clean eclipse warnings.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
The split column in a parent meta table entry can be null (Happens
if a daughter split no longer has references -- it removes its
entry from parent). Add handling and clean up around split
management code. Added logging of unexpected exceptions
scanning a region.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Added fix for NPE when client asks for scanner but passes
non-existent columns.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
(getHRegionInfo, getHRegionInfoOrNull): Added.:
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@567308 13f79535-47bb-0310-9956-ffa450edef68
HADOOP-1725 Client find of table regions should not include offlined, split parents
Changes:
New class MapWritable replaces KeyedData and KeyedDataArrayWritable
HBaseAdmin, HConnectionManager, HMaster, HRegionInterface,
HRegionServer, HTable, TestScanner2:
- getRow returns MapWritable instead of array of KeyedData
- next returns MapWritable instead of array of KeyedData
GroupingTableMap, IdentityTableMap, IdentityTableReduce,
TableInputFormat, TableMap, TableOutputCollector, TableOutputFormat,
TestTableMapReduce:
- use MapWritable instead of KeyedData and KeyedDataArrayWritable
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566878 13f79535-47bb-0310-9956-ffa450edef68
Disentangles flushes and compactions; flushes can proceed while a
compaction is happening. Also, don't compact unless we hit
compaction threshold: i.e. don't automatically compact on HRegion
startup so regions can come online the faster.
M src/contrib/hbase/conf/hbase-default.xml
(hbase.hregion.compactionThreashold): Moved to be a hstore property
as part of encapsulating compaction decision inside hstore.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
Refactored. Moved here generalized content loading code that can
be shared by tests. Add to setup and teardown the setup and removal
of local test dir (if it exists).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestCompare.java
Added test of HStoreKey compare (It works other than one would at
first expect).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Bulk of content loading code has been moved up into the parent class.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
(tableExists): Restore to a check of if the asked-for table is in list of
tables. As it was, a check for tableExists would just wait on all timeouts
and retries to expire and then report table does not exist.. Fixed up
debug message listing regions of a table. Added protection against meta
table not having a COL_REGINFO (Seen in cluster testing -- probably a bug
in row removal).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
Loading store files, even if it was noticed that there was no corresponding
map file, was still counting file as valid. Also fix merger -- was
constructing MapFile.Reader directly rather than asking HStoreFile for
the reader (HStoreFile knows how to do MapFile references)
(rename): Added check that move succeeded and logging. In cluster-testing,
the hdfs move of compacted file into place has failed on occasion (Need
more info).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Encapsulate ruling on whether a compaction should take place inside HStore.
Added reading of the compactionThreshold her. Compaction threshold is
currently just number of store files. Later may include other factors such
as count of reference files. Cleaned up debug messages around
reconstruction log. Removed compaction if size > 1 from constructor. Let
compaction happen after we've been deployed (Compactions that happen while
we are online can continue to take updates. Compaction in the constructor
puts off our being able to take in updates).
(close): Changed so it now returns set of store files. This used to be done
by calls to flush. Since flush and compaction have been disentangled, a
compaction can come in after flush and the list of files could be off.
Having it done by close, can be sure list of files is complete.
(flushCache): No longer returns set of store files. Added 'merging compaction'
where we pick an arbitrary store file from disk and merge into it the content
of memcache (Needs work).
(getAllMapFiles): Renamed getAllStoreFiles.
(needsCompaction): Added.
(compactHelper): Added passing of maximum sequence number if already
calculated. If compacting one file only, we used skip without rewriting
the info file. Fixed.
Refactored. Moved guts to new compact(outFile, listOfStores) method.
(compact, CompactionReader): Added overrides and interface to support
'merging compaction' that takes files and memcache. In compaction,
if we failed the move of the compacted file, all data had already been
deleted. Changing, so deletion happens after confirmed move of
compacted file.
(getFull): Fixed bug where NPE when read of maps came back null.
Revealed by our NOT compacting stores on startup. Meant could be two
backing stores one of which had no data regards queried key.
(getNMaps): Renamed countOfStoreFiles.
(toString): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreKey.java
Added comment on 'odd'-looking comparison.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Javadoc edit.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogEdit.java
Only return first 128 bytes of value when toStringing (On cluster,
was returning complete web pages in log).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Removed confusing debug message (made sense once -- but not now).
Test rootRegionLocation for null before using it (can be null).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Added comment that delete behavior needs study.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Fixed merge so it doesn't do the incremental based off files
returned by flush. Instead all is done in the one go after
region closes (using files returned by close).
Moved duplicated code to new filesByFamily method.
(WriteState): Removed writesOngoing in favor of compacting and
flushing flags.
(flushCache): No longer returns list of files.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
Fix javadoc.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566459 13f79535-47bb-0310-9956-ffa450edef68
Splits are now near-instantaneous. On split, daughter splits create
'references' to store files up in the parent region using new 'HalfMapFile'
class to proxy accesses against the top-half or bottom-half of
backing MapFile. Parent region is deleted after all references in daughter
regions have been let go.
Below includes other cleanups and at least one bug fix for fails adding
>32k records and improvements to make it more likely TestRegionServerAbort
will complete..
A src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHStoreFile.java
Added. Tests new Reference HStoreFiles. Test new HalfMapFileReader inner
class of HStoreFile. Test that we do the right thing when HStoreFiles
are smaller than a MapFile index range (i.e. there is not 'MidKey').
Test we do right thing when key is outside of a HalfMapFile.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestGet.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestScanner.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestTimestamp.java
getHRegionDir moved from HStoreFile to HRegion.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestBatchUpdate.java
Let out exception rather than catch and call 'fail'.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/MiniHBaseCluster.java
Refactored so can start and stop a minihbasecluster w/o having to
subclass this TestCase. Refactored methods in this class to use the
newly added methods listed below.
(MasterThread, RegionServerThread, startMaster, startRegionServers
shutdown): Added.
Added logging of abort, close and wait. Also on abort/close
was doing a remove that made it so subsequent wait had nothing to
wait on.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Added tests that assert all works properly at region level on
multiple levels of splits and then do same on a cluster.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHRegion.java
Removed catch and 'fail()'.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
Javadoc to explain how split now works. Have constructors flow
into each other rather than replicate setup per instance. Moved
in here operations such as delete, rename, and length of store files
(No need of clients to remember to delete map and info files).
(REF_NAME_PARSER, Reference, HalfMapFile, isReference,
writeReferenceFiles, writeSplitInfo, readSplitInfo,
createOrFail, getReader, getWriter, toString): Added.
(getMapDir, getMapFilePath, getInfoDir, getInfoFilePath): Added
a bunch of overrides for reference handling.
(loadHStoreFiles): Amended to load references off disk.
(splitStoreFiles): Redone to instead write references into
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Rename maps as readers and mapFiles as storefiles.
Moved BloomFilterReader and Writer into HStoreFile. Removed
getMapFileReader and getMapFileWriter (They are in HStoreFile now).
(getReaders): Added.
(HStoreSize): Added. Data Structure to hold aggregated size
of all HStoreFiles in HStore, the largest, its midkey, and
if the HStore is splitable (May not be if references).
Previous we only did largest file; less accurate.
(getLargestFileSize): Renamed size and redone to aggregate
sizes, etc.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Have constructors waterfall down through each other rather than
repeat initializations.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMerge.java
Use new HStoreSize structure.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Added delayed remove of HRegion (Now done in HMaster as part of
meta scan). Change LOG.error and LOG.warn so they throw stack trace
instead of just the Exception.toString as message.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConstants.java
(COLUMN_FAMILY_STR): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLog.java
Added why to log of splitting.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogEdit.java
Short is not big enough to hold edits tha could contain a sizable
web page.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTable.java
(getTableName): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Added constructor to BaseScanner that takes name of table we're
scanning (ROOT or META usually). Added to scanOneRegion handling
of split regions. Collect splits to check while scanning and
then outside of the scanning, so we can modify the META table
is needed, do the checks of daughter regions and update on
change of state. Made LOG.warn and LOG.error print stack trace.
(isSplitParent, cleanupSplits, hasReferences): Added.
Added toString to each of the PendingOperation implementations.
In the ShutdownPendingOperation scan of meta data, removed
check of startcode (if the server name is that of the dead
server, it needs reassigning even if start code is good).
Also, if server name is null -- possible if we are missing
edits off end of log -- then the region should be reassigned
just in case its from the dead server. Also, if reassigning,
clear from pendingRegions. Server may have died after sending
region is up but before the server confirms receipt in the
meta scan. Added mare detail to each log. In OpenPendingOperation
we were trying to clear pendingRegion in wrong place -- it was
never executed (regions were always pending).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionInfo.java
Add split boolean. Output offline and split status in toString.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Comments.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Moved getRegionDir here from HStoreFile.
(COL_SPLITA, COL_SPLITB): Added.
(closeAndSplit): Refactored to use new fast split method.
StringUtils.formatTimeDiff(System.currentTimeMillis(), startTime));
(splitStoreFile): Moved into HStoreFile.
(getSplitRegionDir, getSplitsDir, toString): Added.
(needsSplit): Refactored to exploit new HStoreSize structure.
Also manages notion of 'unsplitable' region.
(largestHStore): Refactored.
(removeSplitFromMETA, writeSplitToMETA, getSplit, hasReference): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Keying.java
(intToBytes, getBytes): Added.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
Utility reading and writing Writables.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@564012 13f79535-47bb-0310-9956-ffa450edef68
HTable
* added public method close
* added protected method checkClosed
* make getConnection public
HConnectionManager
* a call to getTableServers or reloadTableServers will cause information for closed
tables to be reloaded
TestHTable
* new test case
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@562294 13f79535-47bb-0310-9956-ffa450edef68
(except TestHClient and HBaseShell) have been converted to use the new client
side objects (HTable/HBaseAdmin/HConnection) instead of HClient.
HBaseAdmin
- Expose connection methods getMaster, isMasterRunning and listTables
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@562041 13f79535-47bb-0310-9956-ffa450edef68
Michael Stack
Bryan Duxbury
Jim Kellerman