Commit Graph

111 Commits

Author SHA1 Message Date
bsglz dbbba7932c HBASE-23334 The table-lock node of zk is not needed since HBASE-16786 (#873)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2019-11-25 19:49:31 +08:00
Mingliang Liu 8bfdfe1b85 HBASE-23289 Update links to Hadoop wiki in book and code
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2019-11-18 08:58:07 -08:00
stack 2ebdcbc3b0 Revert " HBASE-23055 Alter hbase:meta (#667)"
Revert because new feedback and requested survey of master usage
figuring table state.

This reverts commit 52176184df.
2019-10-01 09:24:10 -07:00
Michael Stack 52176184df
HBASE-23055 Alter hbase:meta (#667)
Make it so hbase:meta can be altered. TableState for hbase:meta
    is kept in Master. State is in-memory transient so if  Master
    fails, hbase:meta is ENABLED again. hbase:meta schema will be
    bootstrapped from the filesystem. Changes to filesystem schema
    are atomic so we should be ok if Master fails mid-edit (TBD)
    Undoes a bunch of guards that prevented our being able to edit
    hbase:meta. At minimmum, need to add in a bunch of WARNING.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     For table state, go to master rather than go to meta direct. Going
     to meta won't work for hbase;meta state. Puts load on Master.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
     Change isTableDisabled/Enabled implementation to ask the Master instead.
     This will give the Master's TableStateManager's opinion rather than
     client figuring it for themselves reading meta table direct.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
     TODO: Cleanup in here. Go to master for state, not to meta.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
     Logging cleanup.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta.
     Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

 Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
2019-09-30 09:29:51 -07:00
Viraj Jasani 1dcc8ee50c HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity (#617) 2019-09-13 10:34:01 +05:30
Jan Hentschel 5106f2826e
HBASE-22744 Removed deprecated status and load classes in client module
Signed-off-by: stack <stack@apache.org>
2019-08-27 19:46:08 +02:00
Jan Hentschel 773414ff53
HBASE-22791 Fixed order of asserts in TestZKMulti
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-08-05 11:36:32 +02:00
Jan Hentschel abb60faa6c
HBASE-22787 Cleaned up tests in hbase-zookeeper
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-08-04 20:49:11 +02:00
Jan Hentschel 24b970ec51
Revert "HBASE-22744 Removed deprecated status and load classes in client module"
This reverts commit 4a61c8b10a.
2019-07-31 12:29:19 +02:00
Jan Hentschel 4a61c8b10a
HBASE-22744 Removed deprecated status and load classes in client module
Signed-off-by: stack <stack@apache.org>
2019-07-30 12:40:42 +02:00
Viraj Jasani 202d2adba1 HBASE-22638 ZooKeeper Utility enhancements
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 18:50:30 +02:00
syedmurtazahassan 9f343c2efa HBASE-22571 Javadoc Warnings related to @return tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 13:34:48 +02:00
syedmurtazahassan fabf2b8282 HBASE-22572 Javadoc Warnings: @link reference not found (#306)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2019-07-01 21:15:00 -07:00
Viraj Jasani 214553d17a [HBASE-22591] : RecoverableZooKeeper improvement for getData, getChil… (#310)
HBASE-22591 RecoverableZooKeeper improvement for getData, getChildren, exists and removal of unused reference

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-06-17 15:22:46 +01:00
Jan Hentschel 99ea828d1c HBASE-22373 Moved maven-eclipse-plugin to eclipse profile 2019-06-10 00:51:18 +02:00
Andrew Purtell a30b186568
HBASE-22449 https everywhere in Maven metadata (#247) 2019-05-21 12:34:25 -07:00
Wellington Chevreuil fab2e15ae4
HBASE-21784 Dump replication queue should show list of wal files ordered chronologically
Change-Id: I18c372406290e2b1e2b5503e2c87adcb9bf6fe91

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-05-14 14:04:54 -07:00
zhangduo df27820958 HBASE-21658 Should get the meta replica number from zk instead of config at client side 2019-05-11 09:22:00 +08:00
Sean Busbey bf140acd20 HBASE-22083 move eclipse settings into a profile.
Signed-off-by: stack <stack@apache.org>
2019-04-25 14:17:18 -05:00
Josh Elser 59406c44e3 HBASE-22057 Cap the size of the nodes we delete in one ZK.multi call
If we try to delete too many znodes at once, we'll smack into the
jute.maxbuffer size. Try to prevent that from happening. The dominating
factor of the ZK client request should be the znode side on a delete.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-03-27 13:49:51 -04:00
stack 089a639724 HBASE-22052 pom cleaning; filter out jersey-core in hadoop2 to match hadoop3 and remove redunant version specifications
This is a reapply of a reverted commit. This commit includes
HBASE-22059 amendment and subsequent ammendments to HBASE-22052.
See HBASE-22052 for full story.

jersey-core is problematic. It was transitively included from hadoop
and polluting our CLASSPATH with an implementation of a 1.x version
of the javax.ws.rs.core.Response Interface from jsr311-api when we
want the javax.ws.rs-api 2.x version.

    M hbase-endpoint/pom.xml
    M hbase-http/pom.xml
    M hbase-mapreduce/pom.xml
    M hbase-rest/pom.xml
    M hbase-server/pom.xml
    M hbase-zookeeper/pom.xml
     Remove redundant version specification (and the odd property define
     done already up in parent pom).
    M hbase-it/pom.xml
    M hbase-rest/pom.xml
     Exclude jersey-core explicitly.

    M hbase-procedure/pom.xml
     Remove redundant version and classifier.

    M pom.xml
     Add jersey-core exclusions to all dependencies that pull it in
     except hadoop-minicluster. mr tests fail w/o the jersey-core
     so let it in for minicluster and then in modules, exclude it
     where it causes damage as in hbase-it.
2019-03-25 09:30:09 -04:00
Josh Elser ae0198084c HBASE-21872 Use a call that defaults to UTF-8 charset for string to byte encoding
Fixed commit message

Signed-off-by: Sean Busbey <busbey@apache.org>
2019-02-15 12:11:40 -05:00
Josh Elser d0e4912f67 Revert "HBASE-21782 Use a call that defaults to UTF-8 charset for string to byte encoding"
This reverts commit 3d4e1d57e5. Fixing incorrect Jira id.
2019-02-15 12:11:29 -05:00
Josh Elser 3d4e1d57e5 HBASE-21782 Use a call that defaults to UTF-8 charset for string to byte encoding
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-02-14 12:02:49 -05:00
zhangduo 8bf966c8e9 HBASE-21541 Move MetaTableLocator.verifyRegionLocation to hbase-rsgroup module 2018-12-04 22:33:36 +08:00
maoling 5e84997f2f HBASE-13468 hbase.zookeeper.quorum supports ipv6 address
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-11-10 06:09:29 -08:00
Mike Drob bc7628a8c3
HBASE-21073 Redo concept of maintenance mode
Instead of being an ephemeral state set by hbck, maintenance mode is now
an explicit toggle set by either configuration property or environment
variable. In maintenance mode, master will host system tables and not
assign any user-space tables to RSs. This gives operators the ability to
affect repairs to meta table with fewer moving parts.
2018-10-18 16:27:53 -05:00
Andrew Purtell 1010992ee8
HBASE-21203 TestZKMainServer#testCommandLineWorks won't pass with default 4lw whitelist
Recent versions of ZooKeeper whitelist the so-called 4-letter word admin
commands, and 'stat' is not in the default whitelist. Set system property
zookeeper.4lw.commands.whitelist=* in MiniZooKeeperCluster#setupTestEnv
as we do not need to whitelist 4-letter commands for unit tests.
2018-09-21 15:36:54 -07:00
subrat.mishra 49ae8549cf HBASE-21040 Replace call to printStackTrace() with proper logger call
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-08-15 08:29:54 -07:00
Sahil Aggarwal 952bb96c8a
HBASE-19164: Remove UUID.randomUUID in tests.
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-27 10:34:16 -05:00
Guanghao Zhang ec66434380 HBASE-20561 The way we stop a ReplicationSource may cause the RS down 2018-06-13 17:58:59 +08:00
maoling 1b98a96caa HBASE-19761:Fix Checkstyle errors in hbase-zookeeper
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2018-06-02 10:08:15 +02:00
Yu Li 061a31fad1 HBASE-20159 Support using separate ZK quorums for client 2018-03-29 02:37:26 +08:00
huzheng 7a1d00c7a0 HBASE-20138 Find a way to deal with the conflicts when updating replication position 2018-03-26 21:07:22 +08:00
Chia-Ping Tsai a6eeb26cc0 HBASE-20212 Make all Public classes have InterfaceAudience category
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-22 18:10:23 +08:00
Sean Busbey 2a65066b35 HBASE-20070 refactor website generation
* rely on git plumbing commands when checking if we've built the site for a particular commit already
* switch to forcing '-e' for bash
* add command line switches for: path to hbase, working directory, and publishing
* only export JAVA/MAVEN HOME if they aren't already set.
* add some docs about assumptions
* Update javadoc plugin to consistently be version 3.0.0
* avoid duplicative site invocations on reactor modules
* update use of cp command so it works both on linux and mac
* manually skip enforcer plugin during build
* still doing install of all jars due to MJAVADOC-490, but then skip rebuilding during aggregate reports.
* avoid the pager on git-diff by teeing to a log file, which also helps later reviewing in the case of big changesets.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
2018-03-02 09:25:10 -06:00
Michael Stack 0732ef5ebf Revert "HBASE-19598 Fix TestAssignmentManagerMetrics flaky test"
Pushed prematurely. Revert.

This reverts commit 016cf0c64b.
2018-02-28 23:39:54 -08:00
Michael Stack 5b64de45ed HBASE-20110 Findbugs in zk and mr caused nightly #409 branch-2 to fail 2018-02-28 23:39:21 -08:00
Michael Stack 016cf0c64b HBASE-19598 Fix TestAssignmentManagerMetrics flaky test
Master never left waitForMasterActive because it never checked state of
the clusterUp flag. The test here was aborting regionserver and then
just exiting. The minihbasecluster shutdown sets the cluster down flag
but we were never looking at it so Master thread was staying up.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
The tableOnMaster check in waitForMasterActive looks wrong. It was
making it so a 'normal' Master was getting stuck in here. This is not
the place to worry about tablesOnMaster. That is for the balancer to be
concerned with. There is a problem with Master hosting
system-tables-only. After further study, Master can carry regions like a
regionserver but making it so it carries system tables only is tricky
given meta assign happens ahead of all others which means that the
Master needs to have checked-in as a regionserver super early... It
needs work. Punted for now.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
Mostly renaming so lists and maps of region infos have same name as they
have elsewhere in code base and cleaning up confusion that may arise
when we talk of servers-for-system-tables....It is talking about
something else in the code changes here that is other than the normal
understanding. It is about filtering regionservers by their version
numbers so we favor regions with higher version numbers. Needs to go
back up into the balancer.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/BaseLoadBalancer.java
It was possible for the Master to be given regions if no regionservers
available (as per the failing unit test in this case).

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Minor reordering moving the waitForMasterActive later in the initialize
and wrapping each test in a check if we are to keep looping (which
checks cluster status flag).

M hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.java
This was an old test from the days when Master carried system tables.
Updated test and fixed metrics. Metrics count the hbase:meta along with
the userspace region so upped expected numbers (previously the
hbase:meta was hosted on the master so metrics were not incremented).

M hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestRegionsOnMasterOptions.java
I took a look at this test again but nope, needs a load of work still to
make it pass.

M hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
Stop being so whiney.
2018-02-28 22:25:24 -08:00
Michael Stack 0593dda663 HBASE-19951 Cleanup the explicit timeout value for test method 2018-02-10 09:24:31 -08:00
Michael Stack 06dec20582
HBASE-19919 Tidying up logging 2018-02-03 08:42:02 -08:00
zhangduo 7c318cead9 HBASE-19887 Do not overwrite the surefire junit listener property in the pom of sub modules 2018-01-31 20:49:18 +08:00
Chia-Ping Tsai fcbcfaa120 HBASE-19877 hbase-common and hbase-zookeeper don't add the log4j.properties to the resource path for testing 2018-01-29 16:52:19 +08:00
zhangduo 221eb95768 HBASE-19870 Fix the NPE in ReadOnlyZKClient#run
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2018-01-29 16:28:59 +08:00
zhangduo 918599ef12 HBASE-19873 Add a CategoryBasedTimeout ClassRule for all UTs 2018-01-29 08:43:56 +08:00
zhangduo e546034f09 HBASE-19795 Move the tests which only need zookeeper in TestZooKeeper to hbase-zookeeper module 2018-01-15 10:57:51 +08:00
zhangduo d24fddf3ed HBASE-19772 Do not close connection to zk when there are still pending request in ReadOnlyZKClient 2018-01-14 17:08:23 +08:00
Josh Elser 057e80c163 HBASE-19769 Remove ZK metrics because of classloader issues
When we run MapReduce jobs via `yarn jar`, the special classloader
which is set up by YARN creates a situation where our invocation of
package-private Hadoop classes throws an IllegalAccessError. It's
easiest to just remove these and rethink how to avoid further
Hadoop metrics2 issues.

Signed-off-by: Michael Stack <stack@apache.org>
2018-01-12 12:50:18 -05:00
Michael Stack a7f9668c33 HBASE-19753 Miscellany of fixes for hbase-zookeeper tests to make them more robust
First, we add test resources to CLASSPATH when tests run. W/o it, there
was no logging of hbase-zookeeper test output (not sure why I have to
add this here and not over in hbase-server; research turns up nothing
so far).

M hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKMainServer.java
 Improve fail log message.

M hbase-zookeeper/src/test/java/org/apache/hadoop/hbase/zookeeper/TestReadOnlyZKClient.java
M hbase-zookeeper/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKNodeTracker.java
 Wait until ZK is connected before progressing. On my slow zk, it could
 be a while post construction before zk connected. Using an unconnected
 zk caused test to fail.

M hbase-zookeeper/src/test/java/org/apache/hadoop/hbase/zookeeper/TestZKMainServer.java
 Change session timeout to default 30s from 1s which was way too short.

M hbase-zookeeper/src/test/resources/log4j.properties
 Set zk logs to DEBUG level in this module at least.

Adds a ZooKeeperHelper class that has utility to help interacting w/ ZK.
2018-01-11 11:23:37 -08:00
Michael Stack adf70ac329 HBASE-19744 Fix flakey TestZKLeaderManager 2018-01-09 16:31:09 -08:00
huzheng f89920a60f HBASE-19622 Reimplement ReplicationPeers with the new replication storage interface 2018-01-09 13:11:01 +08:00
Mike Drob c3b4f788b1 HBASE-19552 find-and-replace thirdparty offset 2017-12-28 11:52:32 -06:00
Balazs Meszaros f572c4b80e HBASE-10092 Move up on to log4j2
Changes:
- replaced commons-logging to slf4j everywhere
- log.XXX(Throwable) calls were replaced with log.XXX(t.toString(), t)
- log.XXX(Object) calls were replaced with log.XXX(Objects.toString(obj))
- log.fatal() calls were replaced with log.error(HBaseMarkers.FATAL, ...)
- programmatic log4j configuration was removed from the unit test

This commit does not affect the current logging configurations, because log4j
is still on the classpath. slf4j-log4j12 binds log4j to slf4j.

Signed-off-by: Michael Stack <stack@apache.org>
2017-12-20 22:21:33 -08:00
Jan Hentschel 26b69dc45b HBASE-19474 Brought down the number of Checkstyle errors in hbase-zookeeper 2017-12-17 14:30:09 +01:00
Peter Somogyi 59529a78f0
HBASE-19498 Fix findbugs and error-prone warnings in hbase-client (branch-2)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Apekshit Sharma <appy@apache.org>
2017-12-15 15:55:51 -08:00
Josh Elser 4a1c3b4210 HBASE-19267 Remove compiler-plugin mapping executions as it breaks Java8 detection
It seems like the original reason this execution filter was added is no
longer an issue for 2.0. Actually, these entries actually preclude
Eclipse from correctly using the Java8 source/target version that we
have specified (which creates numerous compilation errors in Eclipse)

Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-12-14 15:11:19 -05:00
Jan Hentschel 5b174b65c0 HBASE-19360 Removed unused imports from hbase-zookeeper module 2017-12-08 23:12:09 +01:00
zhangduo 75cdbb5700 HBASE-19410 Move zookeeper related UTs to hbase-zookeeper and mark them as ZKTests 2017-12-06 16:38:34 +08:00
zhangduo 7a5b078306 HBASE-19399 Purge curator dependency from hbase-client 2017-12-03 08:30:30 +08:00
Apekshit Sharma 81b95afbee HBASE-19367 Refactoring in RegionStates, and RSProcedureDispatcher
- Adding javadoc comments
- Bug: ServerStateNode#regions is HashSet but there's no synchronization to prevent concurrent addRegion/removeRegion. Let's use concurrent set instead.
- Use getRegionsInTransitionCount() directly to avoid instead of getRegionsInTransition().size() because the latter copies everything into a new array - what a waste for just the size.
- There's mixed use of getRegionNode and getRegionStateNode for same return type - RegionStateNode. Changing everything to getRegionStateNode. Similarly rename other *RegionNode() fns to *RegionStateNode().
- RegionStateNode#transitionState() return value is useless since it always returns it's first param.
- Other minor improvements
2017-11-29 22:40:11 -08:00
Apekshit Sharma 330b0d05b9 HBASE-19114 Split out o.a.h.h.zookeeper from hbase-server and hbase-client
- Moved DrainingServerTracker and RegionServerTracker to hbase-server:o.a.h.h.master.
- Moved SplitOrMergeTracker to oahh.master (because it depends on a PB)
- Moving hbase-client:oahh.zookeeper.*  to hbase-zookeeper module.  After HBASE-19200, hbase-client doesn't need them anymore (except 3 classes).
- Renamed some classes to use a consistent naming for classes - ZK instead of mix of ZK, Zk , ZooKeeper. Couldn't rename following public classes: MiniZooKeeperCluster, ZooKeeperConnectionException. Left RecoverableZooKeeper for lack of better name. (suggestions?)
- Sadly, can't move tests out because they depend on HBaseTestingUtility (which defeats part of the purpose - trimming down hbase-server tests. We need to promote more use of mocks in our tests)
2017-11-17 13:20:18 -08:00