Patch brought over from 0.90_master_rewrite branch.
Replication test is broke as are some of the rest tests.
Others should be passing.
Some of the changes made in this fat patch:
+ In HLogKey, we now use encoded region name instead of full region name.
+ On split, daughters are opened on the parent's regionserver; let the new balancer
sort them out later when it cuts in.
+ Added move region from one server to another as well as enable/disable balancer.
+ All .META. and -ROOT- edits go via new *Editor and *Reader classes -- no more
do we have 5 different ways of reading and editing .META.
+ Rather than 3 different listeners to hlog each w/ own way of listening, instead
we only have WALObserver now.
+ New Server Interface that has whats common to HMaster and RegionServer. Also
new Services Interface. This should make test writing cleaner making it so
less need of full cluster context testing anything -- e.g. the new
Interfaces are good w/ Mockito.
+ New balacner that runs on a period and takes into consideration all load
across cluster.
+ Table online/offline is now a flag in ZK; the offline flag on a region is
just used splitting from here on out.
+ Moved fixup of failed add of daughter edits to .META. into shutdown server
recover code (It used to be in basescanner).
+ The heartbeat now sends master the regionserver load and is used sending
shutdown message from master to regionserver ONLY; all other messages are
via zk (HMsg is pretty bare now).
+ No more Worker in RS and ToDoQueue in master. Both in master and regionserver
we use handlers instead run out of Executors.
+ Client can not send split, flush, compact direct to RS; no longer does
it go via master.
+ Server shutdown runs differently now. All are watching a flag in zk.
When RS notices its gone, it closes all user-space regions. If thats all
it was carrying, then it goes down. Otherwise, waits on master to send
the shutdown msg via heartbeat.
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@991397 13f79535-47bb-0310-9956-ffa450edef68
on absolute path (via Nitay Joffe)
HBASE-1187 After disabling/enabling a table, the regions seems to
be assigned to only 1-2 region servers
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@742137 13f79535-47bb-0310-9956-ffa450edef68
- Passes all unit tests.
- Runs Performance evaluation random write in 8min, 43sec on 4 node cluster. I believe this is a new speed record.
- Eliminates time-based log rolling and cache-flushing (because of append support)
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@707247 13f79535-47bb-0310-9956-ffa450edef68
M conf/hbase-default.xml
Added a new config hbase.client.scanner.caching, fixed triplification
M src/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Implements new next(long, int) method for batching
M src/java/org/apache/hadoop/hbase/ipc/HRegionInterface.java
Defines that new next(long, int) method
M src/java/org/apache/hadoop/hbase/client/HTable.java
Adds new property that is passed to scanners. ClientScanner now handles caching of rows
M src/java/org/apache/hadoop/hbase/client/MetaScanner.java
Small changes related to ScannerCallable
M src/java/org/apache/hadoop/hbase/client/HConnectionManager.java
Same sort of small changes
M src/java/org/apache/hadoop/hbase/client/ScannerCallable.java
Now modified because of new methods in HRS, will be able to handle caching
M src/java/org/apache/hadoop/hbase/client/transactional/TransactionalTable.java
Passes caching to its super class
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@698198 13f79535-47bb-0310-9956-ffa450edef68
This commit finishes up the DDL
M src/test/org/apache/hadoop/hbase/TestBloomFilters.java
Changed name of default constants.
M src/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Changed name of default constants.
(getNameAsString): Added.
M src/java/org/apache/hadoop/hbase/master/ModifyColumn.java
toString column family byte array name.
M bin/HBase.rb
(alter, hcd): Added.
M bin/hirb.rb
More help doc.
(alter, admin): Added.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@667684 13f79535-47bb-0310-9956-ffa450edef68
M conf/hbase-default.xml
Add hbase.hbasemaster.maxregionopen property.
M src/java/org/apache/hadoop/hbase/HStore.java
Change way we log. Do way less. Just emit sums of edits applied
and skipped rather than individual edits.
M src/java/org/apache/hadoop/hbase/HRegionServer.java
Make sleeper instance a local rather than data member.
(reportForDuty): Take a sleeper instance.
(run): Removed redundant wrap of a 'for' by a 'while'.
(constructor): If IOE, do not offline the region. Seen to be
an overreaction.
M src/java/org/apache/hadoop/hbase/HLog.java
Don't output map of all files being cleaned everytime a new
entry is added; instead just log new entry. Remove emission
of every 10k edits.
M src/java/org/apache/hadoop/hbase/HMaster.java
Up default for maxregionopen. Was seeing that playing edits
could take a long time (mostly because we used log every
edit) but no harm in this being longer. On REPORT_CLOSE,
emit region info, not just region so can see the properties
(W/o, made it hard to figure who was responsible for offlining).
Add logging of attempt # in shutdown processing.
Add logging of state flags passed to the close region. Helps
debugging. Also in close offline ONLY if we are NOT reassigning
the region (jimk find).
M src/java/org/apache/hadoop/hbase/util/Sleeper.java
Add logging of extraordinary sleeps or calculated periods
(suspicion is that we're sleeping way longer on loaded machies
and the regionserver appears hung).
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@636849 13f79535-47bb-0310-9956-ffa450edef68
(Recommit. We'd backed it out when it was HADOOP-1398)
This commit includes updating version from 0.1.0-dev to 0.2.0-dev
and an update of the hadoop lib to move it post 0.16.0 branch.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@619526 13f79535-47bb-0310-9956-ffa450edef68
Removed mention of all of the HADOOP_* environment variables. Made
HBASE_* equivs. hbase-env.sh is not near dup of hadoop-env.sh.
Removed options on scripts that used take both hbase and hadoop
conf and home. Now we just work w/ hbase. Fixed logging (logs
dir now under hbase).
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@618765 13f79535-47bb-0310-9956-ffa450edef68
Make it so scripts basically work again. Make up a log4j
basic properites file (currently broke).
M trunk/conf/hbase-env.sh
Put a JAVA_HOME in here. Need to set it here now instead of up in hadoop-env
A trunk/conf/log4j.properties
Basic log4j for hbase.
M trunk/bin/hbase
Made pointers up into hadoop instead point to hbase substitutions.
M trunk/bin/start-hbase.sh
Remove wait on dfs for moment. Need to figure how to do it now
we're detached (Besides there is talk that we'd acually have the
servers come up even if they can't talk to dfs).
git-svn-id: https://svn.apache.org/repos/asf/hadoop/hbase/trunk@618560 13f79535-47bb-0310-9956-ffa450edef68
Disentangles flushes and compactions; flushes can proceed while a
compaction is happening. Also, don't compact unless we hit
compaction threshold: i.e. don't automatically compact on HRegion
startup so regions can come online the faster.
M src/contrib/hbase/conf/hbase-default.xml
(hbase.hregion.compactionThreashold): Moved to be a hstore property
as part of encapsulating compaction decision inside hstore.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
Refactored. Moved here generalized content loading code that can
be shared by tests. Add to setup and teardown the setup and removal
of local test dir (if it exists).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestCompare.java
Added test of HStoreKey compare (It works other than one would at
first expect).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Bulk of content loading code has been moved up into the parent class.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
(tableExists): Restore to a check of if the asked-for table is in list of
tables. As it was, a check for tableExists would just wait on all timeouts
and retries to expire and then report table does not exist.. Fixed up
debug message listing regions of a table. Added protection against meta
table not having a COL_REGINFO (Seen in cluster testing -- probably a bug
in row removal).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
Loading store files, even if it was noticed that there was no corresponding
map file, was still counting file as valid. Also fix merger -- was
constructing MapFile.Reader directly rather than asking HStoreFile for
the reader (HStoreFile knows how to do MapFile references)
(rename): Added check that move succeeded and logging. In cluster-testing,
the hdfs move of compacted file into place has failed on occasion (Need
more info).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Encapsulate ruling on whether a compaction should take place inside HStore.
Added reading of the compactionThreshold her. Compaction threshold is
currently just number of store files. Later may include other factors such
as count of reference files. Cleaned up debug messages around
reconstruction log. Removed compaction if size > 1 from constructor. Let
compaction happen after we've been deployed (Compactions that happen while
we are online can continue to take updates. Compaction in the constructor
puts off our being able to take in updates).
(close): Changed so it now returns set of store files. This used to be done
by calls to flush. Since flush and compaction have been disentangled, a
compaction can come in after flush and the list of files could be off.
Having it done by close, can be sure list of files is complete.
(flushCache): No longer returns set of store files. Added 'merging compaction'
where we pick an arbitrary store file from disk and merge into it the content
of memcache (Needs work).
(getAllMapFiles): Renamed getAllStoreFiles.
(needsCompaction): Added.
(compactHelper): Added passing of maximum sequence number if already
calculated. If compacting one file only, we used skip without rewriting
the info file. Fixed.
Refactored. Moved guts to new compact(outFile, listOfStores) method.
(compact, CompactionReader): Added overrides and interface to support
'merging compaction' that takes files and memcache. In compaction,
if we failed the move of the compacted file, all data had already been
deleted. Changing, so deletion happens after confirmed move of
compacted file.
(getFull): Fixed bug where NPE when read of maps came back null.
Revealed by our NOT compacting stores on startup. Meant could be two
backing stores one of which had no data regards queried key.
(getNMaps): Renamed countOfStoreFiles.
(toString): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreKey.java
Added comment on 'odd'-looking comparison.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Javadoc edit.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogEdit.java
Only return first 128 bytes of value when toStringing (On cluster,
was returning complete web pages in log).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Removed confusing debug message (made sense once -- but not now).
Test rootRegionLocation for null before using it (can be null).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Added comment that delete behavior needs study.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Fixed merge so it doesn't do the incremental based off files
returned by flush. Instead all is done in the one go after
region closes (using files returned by close).
Moved duplicated code to new filesByFamily method.
(WriteState): Removed writesOngoing in favor of compacting and
flushing flags.
(flushCache): No longer returns list of files.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
Fix javadoc.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566459 13f79535-47bb-0310-9956-ffa450edef68
M src/contrib/hbase/NOTICE.txt
Add notice of udanax contributions.
Msrc/contrib/hbase/conf/hbase-default.xml
(hbaseshell.jline.bell.enabled): Added.
M src/contrib/hbase/CHANGES.txt
(hadoop-1375) Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/package.html
Add note on how to start up hbase shell
M src/contrib/hbase/bin/hbase
Add 'shell'. Remove 'client' (shell does what it used do and more).
Removed all reader and logreader until better developed. Starting
up a reader or logreader on a running hbase system could do damage).
M src/contrib/hbase/build.xml
Add a javacc target to generate content of shell/generated subpackage.
A src/contrib/hbase/src/test/org/apache/hadoop/hbase/shell/TestHBaseShell.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/Shell.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/DeleteCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/CreateCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/DropCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/InsertCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/CommandFactory.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpContents.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ExitCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ConsoleTable.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/DescCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/SelectCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/Command.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ShowCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/BasicCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpManager.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/ReturnMsg.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HelpCommand.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/HBaseShell.jj
Added.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/Token.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/TokenMgrError.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/SimpleCharStream.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserTokenManager.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParseException.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/ParserConstants.java
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/shell/generated/Parser.java
Added javacc generated files.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@555415 13f79535-47bb-0310-9956-ffa450edef68
On shutdown, region servers and masters were just cancelling leases
without letting 'lease expired' code run -- code to clean up
outstanding locks in region server. Outstanding read locks were
getting in the way of region server getting necessary write locks
needed for the shutdown process. Also, cleaned up messaging around
shutdown so its clean -- no timeout messages as region servers try
to talk to a master that has already shutdown -- even when region
servers take their time going down.
M src/contrib/hbase/conf/hbase-default.xml
Make region server timeout 30 seconds instead of 3 minutes.
Clients retry anyways. Make so its likely region servers report
in their shutdown message before their lease expires on master.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/Leases.java
(closeAfterLeasesExpire): Added.
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Added comments.
(stop): Converted from public to default access (master shuts
down regionservers).
(run): Use leases.closeAfterLeasesExpire instead of leases.close.
Changed log of main thread exit from DEBUG to INFO.
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
(letRegionsServersShutdown): Add better explaination of shutdown
process to method doc. Changed timeout waits from
hbase.regionserver.msginterval to threadWakeFrequency.
(regionServerReport): If closing, we used to immediately respond
to region server with a MSG_REGIONSERVER_STOP. This meant that
we avoided handling of the region servers MSG_REPORT_EXITING sent
on shutdown so region servers had no chance to cancel their lease
in the master. Reordered. Moved sending of MSG_REGIONSERVER_STOP
to after handling of MSG_REPORT_EXITING. Also, in handling of
MSG_REGIONSERER_STOP removed cancelling of leases. Let leases
expire normally (or get cancelled when the region server comes in
with MSG_RPORT_EXITING).
* src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMsg.java
(MSG_REGIONSERVER_STOP_IN_ARRAY): Added.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@552376 13f79535-47bb-0310-9956-ffa450edef68
so that region servers are assigned the regions have a log to apply edits from. Enhance fail over
capabilities.
For all the files modified, clean up javadoc, class method and field visibility.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@546192 13f79535-47bb-0310-9956-ffa450edef68
Michael Stack
Andrew Kyle Purtell
Jean-Daniel Cryans
Jonathan Gray
Nitay Joffe
Jim Kellerman
Bryan Duxbury
Thomas White
Doug Cutting