Commit Graph

75 Commits

Author SHA1 Message Date
stack 2d2e1d965d HBASE-24220 Allow that zk NOTEMPTY multi exception is retryable by running in-series
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
 Cleanup checkstyle warnings. Don't depend on hbase-client
 ScannerCallable.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Cut down on cluster resource usage.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController3.java
 Debug

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
 Allow that NONEMPTY is retryable by running in series.
2020-04-20 15:51:11 -07:00
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
Jianfei Jiang ba34a2ca30
HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 (#1453)
* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

Co-authored-by: 姜建飞 10222269 <jiang.jianfei@zte.com.cn>

Signed-off-by: Mate Szalay-Beko <szalay.beko.mate@gmail.com>
Signed-off-by: Norbert Kalmar <nkalmar@cloudera.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangdo@apache.org>
2020-04-13 13:46:37 -07:00
Michael Stack 3e1cf00c71 HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:26:06 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
stack 6777e2c2d1 HBASE-23899 [Flakey Test] Stabilizations and Debug
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from  medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
  Fixes occasional mismatched filesystems where the difference is file:// vs file:///
  or we pick up hdfs schema when it a local fs test. Had to do this
  vetting of how we do make qualified on a Path in a few places, not
  just here as a few tests failed with this same issue. Code in here is
  used by a lot of tests that each in turn suffered this mismatch.

  Refactor for clarity

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
  Unused import.

hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
  This test fails if tmp dir is not where it expects because tries to
  make rootdir there. Give it a rootdir under test data dir.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
  This change is probably useless. I think the issue is actually
  a problem addressed later where our test for zk server being
  up gets stuck and never times out.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
 Move off deprecated APIs.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Log when we fail balance check for DEBUG Currently just says 'false'

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
 NPEs on way out if setup failed.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
 Add logging when assert fails to help w/ DEBUG

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
 Don't bother removing stuff on teardown. All gets thrown away anyways.
 Saw a few hangs in here in the teardown where hdfs was down before
 expected messing up shutdown.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Add timeout on socket; was seeing check for zk server getting stuck
 and never timing out (test time out in startup)

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
 Write to test data dir instead.
 Be careful about how we make qualified paths.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
 Remove snowflake configs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
 Add a hacky pause. Tried adding barriers but didn't work. Needs deep
 dive.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Remove code copied from zk and use zk methods directly instead.
 A general problem is that zk cluster doesn't come up occasionally but
 no clue why. Add thread dumping and state check.
2020-02-28 13:39:38 -08:00
Bharath Vissapragada 8126566b27 HBASE-23281: Track meta region locations in masters (#830)
* HBASE-23281: Track meta region changes on masters

This patch adds a simple cache that tracks the meta region replica
locations. It keeps an eye on the region movements so that the
cached locations are not stale.

This information is used for servicing client RPCs for connections
that use master based registry (HBASE-18095). The RPC end points
will be added in a separate patch.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Michael Stack 661abeb730 HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 10:28:19 -08:00
Bharath Vissapragada d9453203ae HBASE-23764: Switch to IP address for ZK ensemble
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 14:57:51 -08:00
Michael Stack ba3463d9de HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 15:02:40 -08:00
stack c8496bc694 Revert "HBASE-23055 Alter hbase:meta"
This reverts commit 9abdb7b5ae.
2020-01-11 09:17:17 -08:00
stack 9abdb7b5ae HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
    was hardcoded ENABLED. Make it dynamic. State is now kept in
    current active Master. It is transient so falls back to default
    if Master crashes. Add to registry a getMetaTableState
    which reads mirrored state from zookeeper (NOT from Master and
    defaults ENABLED if no implementation or error fetching state).
    hbase:meta schema will be bootstrapped from the filesystem.
    Changes to filesystem schema are atomic so we should be ok if
    Master fails mid-edit (TBD). Undoes a bunch of guards that
    prevented our being able to edit hbase:meta.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     Add fetching of hbase:meta table state from registry. Adds cache of
     tablestates w/ a ttl of 1 second (adjustable).

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
      Add querying registry for hbase:meta table state.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
      Add querying of mirrored table state for hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta. Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-09 16:25:16 -08:00
Jan Hentschel 703ee60c90
HBASE-23621 Reduced the number of Checkstyle violations in tests of hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-28 23:03:05 +05:30
Jan Hentschel e9b998012f
HBASE-23619 Used built-in formatting for logger in hbase-zookeeper
Signed-off-by: stack <stack@apache.org>
2019-12-27 10:39:46 +01:00
bsglz dbbba7932c HBASE-23334 The table-lock node of zk is not needed since HBASE-16786 (#873)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2019-11-25 19:49:31 +08:00
Mingliang Liu 8bfdfe1b85 HBASE-23289 Update links to Hadoop wiki in book and code
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2019-11-18 08:58:07 -08:00
stack 2ebdcbc3b0 Revert " HBASE-23055 Alter hbase:meta (#667)"
Revert because new feedback and requested survey of master usage
figuring table state.

This reverts commit 52176184df.
2019-10-01 09:24:10 -07:00
Michael Stack 52176184df
HBASE-23055 Alter hbase:meta (#667)
Make it so hbase:meta can be altered. TableState for hbase:meta
    is kept in Master. State is in-memory transient so if  Master
    fails, hbase:meta is ENABLED again. hbase:meta schema will be
    bootstrapped from the filesystem. Changes to filesystem schema
    are atomic so we should be ok if Master fails mid-edit (TBD)
    Undoes a bunch of guards that prevented our being able to edit
    hbase:meta. At minimmum, need to add in a bunch of WARNING.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     For table state, go to master rather than go to meta direct. Going
     to meta won't work for hbase;meta state. Puts load on Master.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
     Change isTableDisabled/Enabled implementation to ask the Master instead.
     This will give the Master's TableStateManager's opinion rather than
     client figuring it for themselves reading meta table direct.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
     TODO: Cleanup in here. Go to master for state, not to meta.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
     Logging cleanup.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta.
     Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

 Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
2019-09-30 09:29:51 -07:00
Viraj Jasani 1dcc8ee50c HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity (#617) 2019-09-13 10:34:01 +05:30
Jan Hentschel 5106f2826e
HBASE-22744 Removed deprecated status and load classes in client module
Signed-off-by: stack <stack@apache.org>
2019-08-27 19:46:08 +02:00
Jan Hentschel 773414ff53
HBASE-22791 Fixed order of asserts in TestZKMulti
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-08-05 11:36:32 +02:00
Jan Hentschel abb60faa6c
HBASE-22787 Cleaned up tests in hbase-zookeeper
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-08-04 20:49:11 +02:00
Jan Hentschel 24b970ec51
Revert "HBASE-22744 Removed deprecated status and load classes in client module"
This reverts commit 4a61c8b10a.
2019-07-31 12:29:19 +02:00
Jan Hentschel 4a61c8b10a
HBASE-22744 Removed deprecated status and load classes in client module
Signed-off-by: stack <stack@apache.org>
2019-07-30 12:40:42 +02:00
Viraj Jasani 202d2adba1 HBASE-22638 ZooKeeper Utility enhancements
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 18:50:30 +02:00
syedmurtazahassan 9f343c2efa HBASE-22571 Javadoc Warnings related to @return tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 13:34:48 +02:00
syedmurtazahassan fabf2b8282 HBASE-22572 Javadoc Warnings: @link reference not found (#306)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2019-07-01 21:15:00 -07:00
Viraj Jasani 214553d17a [HBASE-22591] : RecoverableZooKeeper improvement for getData, getChil… (#310)
HBASE-22591 RecoverableZooKeeper improvement for getData, getChildren, exists and removal of unused reference

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-06-17 15:22:46 +01:00
Jan Hentschel 99ea828d1c HBASE-22373 Moved maven-eclipse-plugin to eclipse profile 2019-06-10 00:51:18 +02:00
Andrew Purtell a30b186568
HBASE-22449 https everywhere in Maven metadata (#247) 2019-05-21 12:34:25 -07:00
Wellington Chevreuil fab2e15ae4
HBASE-21784 Dump replication queue should show list of wal files ordered chronologically
Change-Id: I18c372406290e2b1e2b5503e2c87adcb9bf6fe91

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-05-14 14:04:54 -07:00
zhangduo df27820958 HBASE-21658 Should get the meta replica number from zk instead of config at client side 2019-05-11 09:22:00 +08:00
Sean Busbey bf140acd20 HBASE-22083 move eclipse settings into a profile.
Signed-off-by: stack <stack@apache.org>
2019-04-25 14:17:18 -05:00
Josh Elser 59406c44e3 HBASE-22057 Cap the size of the nodes we delete in one ZK.multi call
If we try to delete too many znodes at once, we'll smack into the
jute.maxbuffer size. Try to prevent that from happening. The dominating
factor of the ZK client request should be the znode side on a delete.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-03-27 13:49:51 -04:00
stack 089a639724 HBASE-22052 pom cleaning; filter out jersey-core in hadoop2 to match hadoop3 and remove redunant version specifications
This is a reapply of a reverted commit. This commit includes
HBASE-22059 amendment and subsequent ammendments to HBASE-22052.
See HBASE-22052 for full story.

jersey-core is problematic. It was transitively included from hadoop
and polluting our CLASSPATH with an implementation of a 1.x version
of the javax.ws.rs.core.Response Interface from jsr311-api when we
want the javax.ws.rs-api 2.x version.

    M hbase-endpoint/pom.xml
    M hbase-http/pom.xml
    M hbase-mapreduce/pom.xml
    M hbase-rest/pom.xml
    M hbase-server/pom.xml
    M hbase-zookeeper/pom.xml
     Remove redundant version specification (and the odd property define
     done already up in parent pom).
    M hbase-it/pom.xml
    M hbase-rest/pom.xml
     Exclude jersey-core explicitly.

    M hbase-procedure/pom.xml
     Remove redundant version and classifier.

    M pom.xml
     Add jersey-core exclusions to all dependencies that pull it in
     except hadoop-minicluster. mr tests fail w/o the jersey-core
     so let it in for minicluster and then in modules, exclude it
     where it causes damage as in hbase-it.
2019-03-25 09:30:09 -04:00
Josh Elser ae0198084c HBASE-21872 Use a call that defaults to UTF-8 charset for string to byte encoding
Fixed commit message

Signed-off-by: Sean Busbey <busbey@apache.org>
2019-02-15 12:11:40 -05:00
Josh Elser d0e4912f67 Revert "HBASE-21782 Use a call that defaults to UTF-8 charset for string to byte encoding"
This reverts commit 3d4e1d57e5. Fixing incorrect Jira id.
2019-02-15 12:11:29 -05:00
Josh Elser 3d4e1d57e5 HBASE-21782 Use a call that defaults to UTF-8 charset for string to byte encoding
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-02-14 12:02:49 -05:00
zhangduo 8bf966c8e9 HBASE-21541 Move MetaTableLocator.verifyRegionLocation to hbase-rsgroup module 2018-12-04 22:33:36 +08:00
maoling 5e84997f2f HBASE-13468 hbase.zookeeper.quorum supports ipv6 address
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-11-10 06:09:29 -08:00
Mike Drob bc7628a8c3
HBASE-21073 Redo concept of maintenance mode
Instead of being an ephemeral state set by hbck, maintenance mode is now
an explicit toggle set by either configuration property or environment
variable. In maintenance mode, master will host system tables and not
assign any user-space tables to RSs. This gives operators the ability to
affect repairs to meta table with fewer moving parts.
2018-10-18 16:27:53 -05:00
Andrew Purtell 1010992ee8
HBASE-21203 TestZKMainServer#testCommandLineWorks won't pass with default 4lw whitelist
Recent versions of ZooKeeper whitelist the so-called 4-letter word admin
commands, and 'stat' is not in the default whitelist. Set system property
zookeeper.4lw.commands.whitelist=* in MiniZooKeeperCluster#setupTestEnv
as we do not need to whitelist 4-letter commands for unit tests.
2018-09-21 15:36:54 -07:00
subrat.mishra 49ae8549cf HBASE-21040 Replace call to printStackTrace() with proper logger call
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-08-15 08:29:54 -07:00
Sahil Aggarwal 952bb96c8a
HBASE-19164: Remove UUID.randomUUID in tests.
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-27 10:34:16 -05:00
Guanghao Zhang ec66434380 HBASE-20561 The way we stop a ReplicationSource may cause the RS down 2018-06-13 17:58:59 +08:00
maoling 1b98a96caa HBASE-19761:Fix Checkstyle errors in hbase-zookeeper
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2018-06-02 10:08:15 +02:00
Yu Li 061a31fad1 HBASE-20159 Support using separate ZK quorums for client 2018-03-29 02:37:26 +08:00
huzheng 7a1d00c7a0 HBASE-20138 Find a way to deal with the conflicts when updating replication position 2018-03-26 21:07:22 +08:00
Chia-Ping Tsai a6eeb26cc0 HBASE-20212 Make all Public classes have InterfaceAudience category
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-22 18:10:23 +08:00
Sean Busbey 2a65066b35 HBASE-20070 refactor website generation
* rely on git plumbing commands when checking if we've built the site for a particular commit already
* switch to forcing '-e' for bash
* add command line switches for: path to hbase, working directory, and publishing
* only export JAVA/MAVEN HOME if they aren't already set.
* add some docs about assumptions
* Update javadoc plugin to consistently be version 3.0.0
* avoid duplicative site invocations on reactor modules
* update use of cp command so it works both on linux and mac
* manually skip enforcer plugin during build
* still doing install of all jars due to MJAVADOC-490, but then skip rebuilding during aggregate reports.
* avoid the pager on git-diff by teeing to a log file, which also helps later reviewing in the case of big changesets.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
2018-03-02 09:25:10 -06:00