Refactor of CatalogJanitor so it generates a
Report on the state of hbase:meta when it runs. Also
refactor so CJ runs even if RIT (previous it would
punt on running if RIT) so it can generate a 'Report'
on the interval regardless. If RIT, it just doesn't
go on to do the merge/split GC as it used to.
If report finds an issue, dump as a WARN message
to the master log.
Follow-on is to make the Report actionable/available
for the Master to pull when it goes to draw the hbck
UI page (could also consider shipping the Report as
part of ClusterMetrics?)
Adds new, fatter Visitor to CJ, one that generates
Report on each run keeping around more findings as
it runs.
Moved some methods around so class reads better;
previous methods were randomly ordered in the class.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Make a few handy methods public.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
Add utility as defaults on the Inteface; i.e. is this the first region
in table, is it last, does a passed region come next, or does passed
region overlap this region (added tests for this new stuff).
M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
Bugfix... handle case where buffer passed is null.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Lots of change, reorg., but mostly adding consistency checking
to the visitor used scanning hbase:meta on a period and the
generation of a Report on what the scan has found traversing
hbase:meta. Added a main so could try the CatalogJanitor against
a running cluster.
A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorCluster.java
Fat ugly test for CatalogJanitor consistency checking.
M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
Add tests for new functionality in RI.
M hbase-shell/src/main/ruby/hbase/table.rb
Bug fix for case where meta has a null regioninfo; scan was aborting.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Add shell support for the following:
- List decommissioned/draining region servers
- Decommission a list of region servers, optionally offload corresponding regions
- Recommission a region server, optionally load a list of passed regions
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Add switching on/off of compactions.
Switching off compactions will also interrupt any currently ongoing compactions.
Adds a "compaction_switch" to hbase shell. Switching off compactions will
interrupt any currently ongoing compactions. State set from shell will be
lost on restart. To persist the changes across region servers modify
hbase.regionserver.compaction.enabled in hbase-site.xml and restart.
Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
Signed-off-by: Michael Stack <stack@apache.org>
* modify the jar checking script to take args; make hadoop stuff optional
* separate out checking the artifacts that have hadoop vs those that don't.
* * Unfortunately means we need two modules for checking things
* * put in a safety check that the support script for checking jar contents is maintained in both modules
* * have to carve out an exception for o.a.hadoop.metrics2. :(
* fix duplicated class warning
* clean up dependencies in hbase-server and some modules that depend on it.
* allow Hadoop to have its own htrace where it needs it
* add a precommit check to make sure we're not using old htrace imports
Conflicts:
hbase-backup/pom.xml
hbase-checkstyle/src/main/resources/hbase/checkstyle-suppressions.xml
Signed-off-by: Mike Drob <mdrob@apache.org>
* makes commands always pass any results back to hirb
* print warning if hirb is given the --return-values flag
* add some docs on how to avoid the console clutter that HBASE-15965 sought to address
* add an upgrade section note about this change.
* cleanup where the get_splits command does its printing so there's a building block that doesn't print
* some rubocop suggested tweaks and opt-out for classlength check on table and shell classes.
Signed-off-by: Mike Drob <mdrob@apache.org>
Conflicts:
src/main/asciidoc/_chapters/upgrading.adoc
* backport leaves off refguide change about upgrading.
Changes for HBASE-20027 seem to cause UI not showing up on default port in standalone mode. For concurrent
unit test execution, individual tests can set hbase.localcluster.assign.random.ports to true or modify
test/resources/hbase-site.xml.
* rely on git plumbing commands when checking if we've built the site for a particular commit already
* switch to forcing '-e' for bash
* add command line switches for: path to hbase, working directory, and publishing
* only export JAVA/MAVEN HOME if they aren't already set.
* add some docs about assumptions
* Update javadoc plugin to consistently be version 3.0.0
* avoid duplicative site invocations on reactor modules
* update use of cp command so it works both on linux and mac
* manually skip enforcer plugin during build
* still doing install of all jars due to MJAVADOC-490, but then skip rebuilding during aggregate reports.
* avoid the pager on git-diff by teeing to a log file, which also helps later reviewing in the case of big changesets.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
Conflicts:
hbase-backup/pom.xml
hbase-spark-it/pom.xml
Uses a new option to the shell to specify that return values are
unwanted instead of overloading the interactive option. Enable
return_values when the shell is non-interactive.
Signed-off-by: Apekshit Sharma <appy@apache.org>
Changes:
- replaced commons-logging to slf4j everywhere
- log.XXX(Throwable) calls were replaced with log.XXX(t.toString(), t)
- log.XXX(Object) calls were replaced with log.XXX(Objects.toString(obj))
- log.fatal() calls were replaced with log.error(HBaseMarkers.FATAL, ...)
- programmatic log4j configuration was removed from the unit test
This commit does not affect the current logging configurations, because log4j
is still on the classpath. slf4j-log4j12 binds log4j to slf4j.
Signed-off-by: Michael Stack <stack@apache.org>
Move the hadoop-hdfs guava exclude in modules up to the top pom.
Looks like an exclude in a module is not additive but rather exclusive
blanking out the top level set of exclusions.
Tested by looking in lib dir of the built tarball.
It seems like the original reason this execution filter was added is no
longer an issue for 2.0. Actually, these entries actually preclude
Eclipse from correctly using the Java8 source/target version that we
have specified (which creates numerous compilation errors in Eclipse)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
- Moved DrainingServerTracker and RegionServerTracker to hbase-server:o.a.h.h.master.
- Moved SplitOrMergeTracker to oahh.master (because it depends on a PB)
- Moving hbase-client:oahh.zookeeper.* to hbase-zookeeper module. After HBASE-19200, hbase-client doesn't need them anymore (except 3 classes).
- Renamed some classes to use a consistent naming for classes - ZK instead of mix of ZK, Zk , ZooKeeper. Couldn't rename following public classes: MiniZooKeeperCluster, ZooKeeperConnectionException. Left RecoverableZooKeeper for lack of better name. (suggestions?)
- Sadly, can't move tests out because they depend on HBaseTestingUtility (which defeats part of the purpose - trimming down hbase-server tests. We need to promote more use of mocks in our tests)
Updated HTrace version to 4.2
Created TraceUtil class to wrap htrace methods. Uses try with resources.
Signed-off-by: Balazs Meszaros <balazs.meszaros@cloudera.com>
Signed-off-by: Michael Stack <stack@apache.org>
* Change imports from org.codehaus to com.fasterxml
* Exclude transitive jackson1 from hadoop and others
* Minor test cleanup to add assert messages, fix some parameter order
* Add anti-pattern check for using jackson 1 imports
* Add explicit non-null serialization directive to ScannerModel
- Table moving to RSG was buggy, because it left the table unassigned.
Now it is fixed we immediately assign to an appropriate RS
(MoveRegionProcedure).
- Table was locked while moving, but unassign operation hung, because
locked table queues are not scheduled while locked. Fixed.
- ProcedureSyncWait was buggy, because it searched the procId in
executor, but executor does not store the return values of internal
operations (they are stored, but immediately removed by the cleaner).
- list_rsgroups in the shell show also the assigned tables and servers.
Signed-off-by: Michael Stack <stack@apache.org>
Includes partial backport of hbase-build-configuration module
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
Main changes:
- ProcedureInfo and LockInfo were removed, we use JSON instead of them
- Procedure and LockedResource are their server side equivalent
- Procedure protobuf state_data became obsolate, it is only kept for
reading previously written WAL
- Procedure protobuf contains a state_message field, which stores the internal
state messages (Any type instead of bytes)
- Procedure.serializeStateData and deserializeStateData were changed slightly
- Procedures internal states are available on client side
- Procedures are displayed on web UI and in shell in the following jruby format:
{ ID => '1', PARENT_ID = '-1', PARAMETERS => [ ..extra state information.. ] }
Signed-off-by: Michael Stack <stack@apache.org>
Do a pass with dependency:analyze; remove unused and
explicity list the dependencies we exploit.
Remove the parent dependencies set which had junit, mockito,
log4j, and findbugs annotations (had to put junit back
temporarily in subsequent version of this patch TODO). Listing in
parent set meant these libs were dependencies for all modules
which in practice was not the case. Edited all modules so
those that need any from this parent set now do explicit listing.
Ran the dependency:analyze over the project. Acted on most
suggested removals and requests for explicit listing. Some
grey areas remain around transitives that come in with
hadoop -needs better excludes, another project- and that
the dependency:analyze tool is not always accurate in its
reporting.
Contains the following commits:
HBASE-17748 Include HBase snapshots in space quotas
Introduces a new Chore in the Master which computes the size
of the snapshots included in a cluster. The size of these
snapshots are included in the table's which the snapshot was created
from HDFS usage.
Includes some test stabilization, trying to make the tests more
deterministic by ensuring we observe stable values as we know
that those values are mutable. This should help avoid problems
where size reports are delayed and we see an incomplete value.
HBASE-17752 Shell command to list snapshot sizes WRT quotas
HBASE-17840 Update hbase book to space quotas on snapshots
Pull in guava 22.0 by using the shaded version up in new hbase-thirdparty project.
In poms, exclude guava everywhere except on hadoop-common. Do this so
we minimize transitive includes. hadoop-common is needed because hadoop
Configuration uses guava doing preconditions.
Everywhere we used guava, instead use shaded so fix a load of imports.
Stopwatch API changed as did hashing and toStringHelper which is now
in MoreObjects class. Otherwise, minimal changes to come up on 22.0
meiyi
Viraj Jasani
Guanghao Zhang
stack
Jan Hentschel
Viraj Jasani
Duo Zhang
Andrew Purtell
Murtaza Hassan
Sean Busbey
meiyi
xuqinya
Toshihiro Suzuki
yaojingyi
Wellington Chevreuil
Nihal Jain
wanghuan
Sakthi
Guangxu Cheng
meiyi
Reid Chan
jingyuntian
subrat.mishra
Sakthi
Mingliang Liu
Mohit Goel
Ankit Singhal
Enis Soztutar
Thoralf Gutierrez
Gabor Bota
openinx
Peter Somogyi
Csaba Skrabak
Balazs Meszaros
Umesh Agashe
Chia-Ping Tsai
Reid Chan
Duo Zhang
Josh Elser
Artem Ervits
Sergey Soldatov
Janos Gub
Mike Drob
tedyu
libisthanks
Peter Somogyi
Apekshit Sharma
Tamas Penzes
Zach York
Yi Liang
Chun-Hao Tang
Chenxi Tong
Phil Yang
Yi Liang