Commit Graph

8137 Commits

Author SHA1 Message Date
niuyulin d4ab645531
HBASE-23949 refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced (#1279)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 21:54:57 +08:00
Pankaj ca533a77cb
HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:39:56 +08:00
Toshihiro Suzuki 9b4e75a7f6
HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 11:55:52 +09:00
Toshihiro Suzuki b53fa3c51f
HBASE-24031 TestHRegion.testCheckAndMutate_WithFilters is flaky (#1316)
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2020-03-22 05:04:16 +09:00
Michael Stack 5592d0c191 HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown (#1311)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Change parameter name and add javadoc to make it more clear what the
 param actually is.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 Move postOpenDeployTasks so if it fails to talk to the Master -- which
 can happen on cluster shutdown -- then we will do cleanup of state;
 without this the RS can get stuck and won't go down.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 Add handleException so CRH looks more like UnassignRegionHandler and
 AssignRegionHandler around exception handling. Add a bit of doc on
 why CRH.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Right shift most of the body of process so can add in a finally
 that cleans up rs.getRegionsInTransitionInRS is on exception
 (otherwise outstanding entries can stop a RS going down on cluster
 shutdown)

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-20 15:26:58 -07:00
Viraj Jasani 44b3ac6457
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (ADDENDUM) 2020-03-20 13:27:06 +05:30
Viraj Jasani 26b31e351c
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (#1286)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-19 15:35:49 +05:30
Michael Stack 3e1cf00c71 HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:26:06 -07:00
stack 9804f73c66 HBASE-23971 protoc warns: "no protobuf syntax specified" (#1278)
Remove overcommit in TestUsersOperationsWithSecureHadoop
(Noticed by ndimiduk)
2020-03-17 15:28:26 -07:00
Wei-Chiu Chuang 60de4c2e31
HBASE-22103. HDFS-13209 in Hadoop 3.3.0 breaks asyncwal. (#1284)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-17 14:36:56 +08:00
Jan Hentschel 1b163d98b9
HBASE-23847 Removed deprecated setStartRow from Scan (#1220)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-13 08:43:17 +08:00
stack 02b6bbe7ce HBASE-23979 Disable TestSlowLogRecorder until parent addressed 2020-03-12 17:24:21 -07:00
WenFeiYi abf0a22259
HBASE-23971 protoc warns: "no protobuf syntax specified" (#1278)
Signed-off-by: stack <stack@apache.org>
2020-03-12 08:32:22 -07:00
stack 26f3702735 HBASE-23974 [Flakey Tests] Allow that server may not yet be cleared from DeadServers in TestHBCKSCP 2020-03-11 23:04:16 -07:00
stack d42e9768d0 HBASE-23956 Use less resources running tests (#1266)
Addendum: TestNamespaceCommands was using up the limit on
nioeventloopgroup.... so create them per run of client.
2020-03-11 22:48:22 -07:00
stack 20658c5139 HBASE-23956 Use less resources running tests (#1266)
Addendum: Two more tests that had issues because parent cut
down the number of default RegionServers launched.
2020-03-11 21:03:10 -07:00
stack 4b64558ebc HBASE-23956 Use less resources running tests (#1266)
Addendum: This test was depending on hard-coded number for
log cleaners; the parent patch made them less in test context.
2020-03-11 20:34:29 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Minwoo Kang 6f1e8abc30
HBASE-23561 acquire lock before update regionsMap (#1263)
Addendum: tie add and delete of regionMap and encodedRegionsMap

Signed-off-by: stack <stack@apache.org>
2020-03-11 16:57:21 -07:00
Michael Stack aaf9a148a0 HBASE-23956 Use less resources running tests (#1266)
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
 Rename the threads we create in here so they are NOT named same was
 threads created by Hadoop RPC.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
 Allow configuring eventloopgroup thread count (so can override for
 tests)

hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
 Enable socket resuse.

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
 Enable socket resuse and config for how many threads to use.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
 Thread name edit; drop the redundant 'Thread' suffix.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
 Make closeable and shutdown executor when called.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 Call close on HFileReplicator

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
 HDFS creates lots of threads. Use less of it so less threads overall.

hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
 Constrain resources when running in test context.

hbase-server/src/test/resources/log4j.properties
 Enable debug on netty to see netty configs in our log

pom.xml
 Add system properties when we launch JVMs to constrain thread counts in
 tests

 Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-11 10:37:50 -07:00
stack d019fd2eed HBASE-23963 Split TestFromClientSide; it takes too long to complete timing out
Split TestGetFromClientSide. Means have to also split the superclass
TestFromClientSideWithCoprocessor.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-10 22:13:24 -07:00
niuyulin e345a95461
HBASE-23953 SimpleBalancer bug when second pass to fill up to min (#1262)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-11 10:30:02 +08:00
niuyulin 6e6b241e2b
HBASE-23954 SplitParent region should not be balanced (#1264)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-10 11:14:43 -07:00
Michael Stack e1e8f396ca HBASE-23851 Log networks and bind addresses when multicast publisher/listener enabled (#1173)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-10 11:00:00 -07:00
Nick Dimiduk 5750876191
HBASE-23932 Minor improvements to Region Normalizer (#1258)
- consolidate checks made by master on behalf of balancer and
   normalizer: deciding if the master is in a healthy state for
   running any actions at all (skipRegionManagementAction). Normalizer
   now does as balancer did previously.
 - both balancer and normalizer make one final check on above
   conditions between calculating an action plan and executing the
   plan. should make the process more responsive to shutdown
   requests.
 - change normalizer to only consider acting on a region when it is in
   the OPEN state. previously we would normalizer attempt to merge a
   region that was already in a MERGING_NEW,MERGING,MERGED state.
 - fix some typos in variable names.

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-10 10:08:37 -07:00
nyl3532016 819965a35a
HBASE-23944 The method setClusterLoad of SimpleLoadBalancer is incorrect when balance by table (#1243)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-07 16:18:41 +08:00
Guanghao Zhang 9b0d214b7b
HBASE-23895 STUCK Region-In-Transition when failed to insert procedure to procedure store (#1221)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-07 15:50:09 +08:00
Guanghao Zhang df62dde394
HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table descriptor directly (#1223)
Signed-off-by: Pankaj <pankajkumar@apache.org>
2020-03-06 19:06:59 +08:00
Wellington Ramos Chevreuil 9c67abef18
HBASE-23939 Remove unused variables from HBaseSaslRpcServer (#1239)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-06 10:05:04 +00:00
Abhinaba Sarkar b6eefcaeb7
HBASE-23788 ROW_INDEX_V1 encoder should consider the secondary index size with the encoded data size tracking (#1219)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-03-04 23:14:43 +05:30
Wei-Chiu Chuang 6d9802fc2e
HBASE-23861. Reconcile Hadoop version. (#1179)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-04 08:01:40 -08:00
stack f411e395d3 HBASE-22978 : Online slow response log (#1228)
Addendum to fix errorprone ERROR
2020-03-03 09:19:21 -08:00
Guanghao Zhang 5dcbe687ae
HBASE-23912 Resolve the TODO of FSTableDescriptor's construct method (#1225)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-03 22:13:16 +08:00
Viraj Jasani c0301e3fdf
HBASE-23868 : Replace usages of HColumnDescriptor(byte [] familyName)… (#1222)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-02 17:20:03 +05:30
Duo Zhang 7f2d823164 HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the code base (#1152)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 7386369fec HBASE-23276 Add admin methods to get tables within a group (#1118)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 5de0a5ef7e HBASE-23235 Re-enable TestRSGroupsKillRS.testLowerMetaGroupVersion (#1117)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang c8d892cdef HBASE-23253 Rewrite rsgroup related UTs with the new methods introduced in HBASE-22932 (#813)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 72cbb129a4 HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Guangxu Cheng e8e9eecef1 HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 0f4a87c24d HBASE-23081 Add an option to enable/disable rs group feature (#691)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b10b39ad03 HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 0e29d62a2d HBASE-22987 Calculate the region servers in default group in foreground (#599)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Guanghao Zhang 2bf2781df9 HBASE-22729 Start RSGroupInfoManager as default (#555)
Amending-Author: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b3399ef365 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 2e0039c2a3 HBASE-22820 Do not need to persist default rs group now (#482)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 44ddb30914 HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 3709f335a8 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang f0a13bb74d HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2020-03-02 15:43:40 +08:00
Duo Zhang 662149e193 HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
stack 6777e2c2d1 HBASE-23899 [Flakey Test] Stabilizations and Debug
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from  medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
  Fixes occasional mismatched filesystems where the difference is file:// vs file:///
  or we pick up hdfs schema when it a local fs test. Had to do this
  vetting of how we do make qualified on a Path in a few places, not
  just here as a few tests failed with this same issue. Code in here is
  used by a lot of tests that each in turn suffered this mismatch.

  Refactor for clarity

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
  Unused import.

hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
  This test fails if tmp dir is not where it expects because tries to
  make rootdir there. Give it a rootdir under test data dir.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
  This change is probably useless. I think the issue is actually
  a problem addressed later where our test for zk server being
  up gets stuck and never times out.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
 Move off deprecated APIs.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Log when we fail balance check for DEBUG Currently just says 'false'

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
 NPEs on way out if setup failed.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
 Add logging when assert fails to help w/ DEBUG

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
 Don't bother removing stuff on teardown. All gets thrown away anyways.
 Saw a few hangs in here in the teardown where hdfs was down before
 expected messing up shutdown.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Add timeout on socket; was seeing check for zk server getting stuck
 and never timing out (test time out in startup)

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
 Write to test data dir instead.
 Be careful about how we make qualified paths.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
 Remove snowflake configs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
 Add a hacky pause. Tried adding barriers but didn't work. Needs deep
 dive.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Remove code copied from zk and use zk methods directly instead.
 A general problem is that zk cluster doesn't come up occasionally but
 no clue why. Add thread dumping and state check.
2020-02-28 13:39:38 -08:00
Wellington Ramos Chevreuil 00ef6c624a
HBASE-23892 SecureTestCluster should allow its subclasses to pass their Class reference on HBaseKerberosUtils.setSSLConfiguration (#1207)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-28 18:35:15 +00:00
Viraj Jasani b24ea32b3b
HBASE-22978 : Online slow response log
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-28 19:47:59 +05:30
Pankaj 3c3aae9a2f
HBASE-23740 Invalid StoreFile WARN log message printed for recovered.… (#1198)
* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory

* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory
2020-02-26 17:52:21 +05:30
Toshihiro Suzuki ecbed33092
HBASE-23146 Support CheckAndMutate with multiple conditions (#1114)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-02-26 08:09:04 +09:00
Jan Hentschel 9f223c2236
HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-25 21:10:09 +01:00
Guanghao Zhang 418234356f HBASE-23864 No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (addendum) 2020-02-24 22:11:41 +08:00
Minwoo Kang 3ab0135c0e
HBASE-23561 Look up of Region in Master by encoded region name is O(n) (#1193)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-21 16:40:45 +00:00
Bharath Vissapragada c34dbc3c29 HBASE-23804: Fix default master addr hostname in master registry (#1137)
Master rpc server end point doesn't bind to localhost's
IP address by default. Instead, it looks up the hostname and
binds to the endpoint to which it resolves. MasterRegistry should
do the same when building the default server end point to talk to.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 02bd0eca53 HBASE-23752: Fix remaining test failures from nightly runs (#1102)
TestFromClientSideWithCoprocessor: Initialization bug causing parameterized
runs to fail.
TestCustomSaslAuthenticationProvider: Test config had to be fixed because
it was written pre-master registry implementation.
TestSnapshotScannerHDFSAclController: Cluster restart did not reset the
cached connection state.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 5c88672d54 HBASE-23731: De-flake TestFromClientSide (#1091)
There were a couple of issues.

- There was a leak of a file descriptor for hbck lock file. This
was contributing to all the "ConnectionRefused" stack traces since
it was trying to renew lease for an already expired mini dfs cluster.
This issue was there for a while, just that we noticed it now.

- After upgrade to JUnit 4.13, it looks like the behavior for test
timeouts has changed. Earlier the timeout seems to have applied for
each parameterized run, but now it looks like it is applied across
all the runs.

This patch fixes both the issues.

Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 229b8aaaf3 HBASE-23647: Make MasterRegistry the default impl. (#1039)
Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada d8b3f55518 HBASE-23330: Fix delegation token fetch with MasterRegistry (#1084)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 1aa6a4efb9 HBASE-23305: Master based registry implementation (#954)
Implements a master based registry for clients.

 - Supports hedged RPCs (fan out configured via configs).
 - Parameterized existing client tests to run with multiple registry combinations.
 - Added unit-test coverage for the new registry implementation.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 0a1c3b2055 HBASE-23604: Clarify AsyncRegistry usage in the code. (#957)
* HBASE-23604: Cleanup AsyncRegistry interface

- Cleans up the method names to make more sense and adds a little
more javadocs for context. In future patches we can revisit
the name of the actual class to make it more self explanatory.

- Does AsyncRegistry -> ConnectionRegistry rename.
"async" ness of the registry is kind of implicit based on
the interface contents and need not be reflected in the name.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 023078ffa4 HBASE-23304: RPCs needed for client meta information lookup (#904)
* HBASE-23304: RPCs needed for client meta information lookup

This patch implements the RPCs needed for the meta information
lookup during connection init. New tests added to cover the RPC
code paths. HBASE-23305 builds on this to implement the client
side logic.

Fixed a bunch of checkstyle nits around the places the patch
touches.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 8126566b27 HBASE-23281: Track meta region locations in masters (#830)
* HBASE-23281: Track meta region changes on masters

This patch adds a simple cache that tracks the meta region replica
locations. It keeps an eye on the region movements so that the
cached locations are not stale.

This information is used for servicing client RPCs for connections
that use master based registry (HBASE-18095). The RPC end points
will be added in a separate patch.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 3d298b087a HBASE-23275: Track active master's address in ActiveMasterManager (#812)
* HBASE-23275: Track active master's address in ActiveMasterManager

Currently we just track whether an active master exists.
It helps to also track the address of the active master in
all the masters to help serve the client RPC requests to
know which master is active.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
stack 3168cf210c HBASE-23877 [Flakey Test] TestStochasticLoadBalancerRegionReplicaHighReplication "Two or more region replicas are hosted on the same host after balance" 2020-02-19 16:09:11 -08:00
stack 435b79335e HBASE-23872 [Flakey Test] TestGenerateDelegationToken; Master not initialized after 200000ms 2020-02-19 15:48:33 -08:00
Vladimir Rodionov b8194b4902 HBASE-22749 Distributed MOB compactions
- MOB compaction is now handled in-line with per-region compaction on region
  servers
- regions with mob data store per-hfile metadata about which mob hfiles are
  referenced
- admin requested major compaction will also rewrite MOB files; periodic RS
  initiated major compaction will not
- periodically a chore in the master will initiate a major compaction that
  will rewrite MOB values to ensure it happens. controlled by
  'hbase.mob.compaction.chore.period'. default is weekly
- control how many RS the chore requests major compaction on in parallel
  with 'hbase.mob.major.compaction.region.batch.size'. default is as
  parallel as possible.
- periodic chore in master will scan backing hfiles from regions to get the
  set of referenced mob hfiles and archive those that are no longer
  referenced. control period with 'hbase.master.mob.cleaner.period'
- Optionally, RS that are compacting mob files can limit write
  amplification by not rewriting values from mob hfiles over a certain size
  limit. opt-in by setting 'hbase.mob.compaction.type' to 'optimized'.
  control threshold by 'hbase.mob.compactions.max.file.size'.
  default is 1GiB
- Should smoothly integrate with existing MOB users via rolling upgrade.
  will delay old MOB file cleanup until per-region compaction has managed
  to compact each region at least once so that used mob hfile metadata can
  be gathered.
2020-02-19 16:06:38 -06:00
Guanghao Zhang 53afc1dc10
HBASE-23864 No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (#1182)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-02-19 18:20:12 +08:00
stack 156ababa4e HBASE-23867 [Flakey Test] TestStochasticLoadBalancerRegionReplicaSameHosts#testRegionReplicationOnMidClusterSameHosts 2020-02-18 22:40:34 -08:00
stack be4b6d6824 HBASE-23866 More test classifications
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncMetaRegionLocator.java
 Add wait on no region in transition in setup before proceeding.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncResultScannerCursor.java
 Make test large. Takes longer than 50seconds.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableRSCrashPublish.java
 Make large because it fails when networking is weird... when VPN or
 something is up. See HBASE-23850. When large, doesn't get run
 when you do mvn test.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableScanAll.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide2.java
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestMultiRowRangeFilter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancerRegionReplica.java
 Make it large. Takes too long.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSeparateClientZKCluster.java
 Use data dir instead of /tmp.

M hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestInfoServersACL.java
 Wait till mbean is up before proceeding w/ test.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java
 Test for null CLUSTER in tearDown.
2020-02-18 21:11:21 -08:00
GeorryHuang c4fb1cd234
HBASE-23859 Modify "Block locality" of RegionServer Web UI to human readable percentage (#1176)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-18 10:06:47 +08:00
GeorryHuang e11dbf36a5
HBASE-23855: Change bytes size to human readable size for Server Metrics of RegionServer Web UI (#1175)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-17 09:14:53 +08:00
Jan Hentschel ae781e569c
HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-15 21:19:51 +01:00
Nick Dimiduk 533302adde
HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… (#1141)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-02-13 14:02:15 -08:00
stack b7ef225609 HBASE-23838 Adding debug logging to a few ExportSnapshot tests
* hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
 Add logging of temp output dir and if tmp dir is being used at all.

* hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Don't create new Configurations. Use current. Set it into the launched
 ExportSnapshot too. Log as we make progress through stages so easier to
 find location of exception source.

* hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
 Add makeQualified though looks to be redundant.

* hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestInfoServersACL.java
 Might fix a local fail where hbase doesn't seem to be up when we query
 jmx. Wait on meta being online.

M hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionStateStore.java
 Run all on the one cluster instance. See if helps w/ odd failure when
 run locally.
2020-02-13 13:50:43 -08:00
maoling 12f98bc50e
HBASE-20623: Introduce the helper method "getCellBuilder()" to Mutation (#1121)
Signed-off-by: stack <stack@apache.org>
2020-02-10 10:03:35 -08:00
stack a6774ca7f7 HBASE-23812 [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7> 2020-02-10 09:45:43 -08:00
meiyi 29a36568b1
HBASE-23824 TestSnapshotScannerHDFSAclController is flakey (#1157)
Signed-off-by: stack <stack@apache.org>
2020-02-10 09:40:05 -08:00
stack 93c04e4f60 HBASE-23815 [Flakey Test] AbstractTestAsyncTableRegionReplicasRead family of tests fails with no breadcrumbs on why 2020-02-10 09:04:36 -08:00
stack a4aa183f45 HBASE-23814 Add null checks and logging to misc set of tests
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Zach York <zyork@apache.org>

* TestFullLogReconstruction log the server we've chosen to expire and then note where we starting counting rows
* TestAsyncTableScanException use a define for row counts
* TestRawAsyncTableLimitedScanWithFilter check connection was made before closing it in tearDown
* TestLogsCleaner use single mod time. Make it for sure less than now in case test runs all in the same millisecond (would cause test fail)
* TestReplicationBase test table is non-null before closing in tearDown
2020-02-10 08:51:56 -08:00
GeorryHuang 35989c1968
HBASE-23822:Fix typo in procedures.jsp (#1155)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-10 09:48:01 +08:00
binlijin 24823ecfc9
HBASE-23682 Fix NPE when disable DeadServerMetricRegionChore (#1026)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-08 15:20:23 -08:00
stack 8654597be0 HBASE-23805 [Flakey Test] TestRaceBetweenSCPAndDTP 2020-02-07 08:35:43 -08:00
Jan Hentschel 93c5e7691f
HBASE-23775 Removed deprecated method createLocalHTU(Configuration) from HBaseTestingUtility
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-07 09:51:04 +01:00
Nick Dimiduk 5b4545de5e
HBASE-23802 Remove unnecessary Configuration instantiation in LossyAccounting (#1127)
Signed-off-by: stack <stack@apache.org>
2020-02-05 14:41:50 -08:00
stack 3a1a39d40d HBASE-23789 [Flakey Tests] ERROR [Time-limited test] balancer.HeterogeneousRegionCountCostFunction(199): cannot read rules file located at ' /tmp/hbase-balancer.rules '; ADDENDUM Missed adding these files. 2020-02-04 16:40:53 -08:00
Mark Robert Miller 299b6bebc5
HBASE-23783: Address tests writing and reading SSL/Security files in a common location. (#1116)
This is causing me issues with parallel test runs.

Also allow setting the surefire reports and temp directories via command line.

Signed-off-by: stack <stack@apache.org>
2020-02-04 14:38:56 -08:00
Duo Zhang bb14bdad62 HBASE-23782 Addendum fix error prone warning 2020-02-04 10:14:27 +08:00
stack d797ed04e5 HBASE-23786 [Flakey Test] TestMasterNotCarryTable.testMasterMemStoreLAB 2020-02-03 17:34:22 -08:00
Jan Hentschel b591a3d205
HBASE-23772 Removed deprecated getTimeStampOfLastShippedOp from MetricsSource
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-04 00:25:58 +01:00
Michael Stack 661abeb730 HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 10:28:19 -08:00
Duo Zhang ab0039974d
HBASE-23782 We still reference the hard coded meta descriptor in some places when listing table descriptors (#1115)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 14:15:57 +08:00
Jan Hentschel 12f1c7c1ce
HBASE-23777 Removed deprecated createTableDescriptor(String, int, int, int, KeepDeletedCells) from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 00:24:19 +01:00
Jan Hentschel 5a3ad6f08f
HBASE-23776 Removed deprecated method createLocalHTU() from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
2020-02-02 00:28:25 +01:00
Bharath Vissapragada d9453203ae HBASE-23764: Switch to IP address for ZK ensemble
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 14:57:51 -08:00
stack ba9f026d69 HBASE-23770 [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart 2020-01-31 08:39:46 -08:00
stack 1a6bdc7c50 HBASE-23771 [Flakey Tests] Test TestSplitTransactionOnCluster Again 2020-01-31 08:35:58 -08:00
abhinaba.sarkar 77229c79e3 HBASE-23350 Make compaction files cacheonWrite configurable based on threshold
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-31 11:35:11 +05:30
Josh Elser 8b00f9f0b1 HBASE-17115 Define UI admins via an ACL
The Hadoop AccessControlList allows us to specify admins of the webUI
via a list of users and/or groups. Admins of the WebUI can mutate the
system, potentially seeing sensitive data or modifying the system.

hbase.security.authentication.spnego.admin.users is a comma-separated
list of users who are admins.
hbase.security.authentication.spnego.admin.groups is a comma-separated
list of groups whose membership are admins. Either of these
configuration properties may also contain an asterisk (*) which denotes
"any entity" (e.g user, group).

Previously, when a user was denied from some endpoint that was
designated for admins, they received an HTTP/401. In this case, it is
more correct to return HTTP/403 as they were correctly authenticated,
but they were disallowed from fetching the given resource. This commit
incorporates this change.

hbase.security.authentication.ui.config.protected also exists for users
who have sensitive information stored in the Hadoop service
configuration and want to limit access to this endpoint. By default,
the Hadoop configuration endpoint is not protected and any
authenticated user can access it.

The test is based off of work by Nihal Jain in HBASE-20472.

Co-authored-by: Nihal Jain <nihaljain.cs@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-29 16:36:55 -05:00
Viraj Jasani 66d198d35f
HBASE-23749 : TestHFileWriterV3 for all DataBlock Encoders
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-29 23:54:18 +05:30
Jan Hentschel 60b80b6298
HBASE-23736 Removed deprecated getTimeStampOfLastAppliedOp from MetricsSink
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2020-01-29 17:02:09 +01:00
stack 98cff8a26a HBASE-23753 Update of errorprone generated failures
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>

hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
 Complains about mismatch in types when Compare. Implement Compare in
 base Interface.

hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
 Complains pbs never return null.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java
 Needed redo because errorprone complains can't mock Service from guava.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Unrelated...adding one-liner debug statements chasing other test
 failures.
2020-01-28 18:54:35 -08:00
stack 92a4a1d7b7 HBASE-23746 [Flakey Tests] Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush and hsync 2020-01-28 09:45:08 -08:00
Jan Hentschel e836f5684f
HBASE-23738 Removed deprecated createLocalHRegion from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-28 12:40:00 +01:00
stack 375e01dac4 HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time
Second part of this issue -- changes to the test.
2020-01-26 10:41:42 -08:00
stack d82c3a5ebd HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time; AMENDMENT
This is actual fix; previous added debug to test.
2020-01-26 10:31:59 -08:00
stack 1690414263 HBASE-23735 [Flakey Tests] TestClusterRestartFailover & TestClusterRestartFailoverSplitWithoutZk 2020-01-24 18:21:33 -08:00
Michael Stack 753cc99814 HBASE-23726 Forward-port HBASE-21345 to branch-2.2, 2.3 & master as well.
HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'.
Just remove the check state from the getClusterStatus call.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthi@apache.org>
(cherry picked from commit dd8496a546)
2020-01-24 12:00:20 -08:00
stack 3c1bccb0f8 HBASE-23733 [Flakey Tests] TestSplitTransactionOnCluster 2020-01-24 11:25:02 -08:00
Geoffrey Jacoby 2d6bb81dff HBASE-23710 - Priority configuration for system coprocessors (#1077)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 16:23:28 -08:00
Michael Stack 373857854a
HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… (#1012)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 09:34:54 -08:00
Wellington Ramos Chevreuil d6ac8b34eb
HBASE-23715 MasterFileSystem should not create MasterProcWALs dir on … (#1078)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-01-23 09:25:31 +00:00
Michael Stack 6cdc4b1f05
HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use.  Backfill.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
 Adds a new compareRows with default implementation that takes a ByteBuffer.
 Needed by the index in a block encoder implementation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
 Adds implementation for meta of new compareRows method. Adds utility
 method for figuring comparator based off tablename.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
 Comparator is in context. Remove redundant handling.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
 Comparator is in context. Remove redundant handling. Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
 Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
 Cache context so can use it to get comparator to use later.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Cache cellcomparator to use. Javdoc on diff between HFileContext and
 HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
 Add CellComparator

M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
 Remove comparator caching. Get from context instead.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
 Skip a reflection if we can.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
 Javadoc. Removed unused filed.

Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 20:00:21 -08:00
Geoffrey Jacoby bb56dfa974 HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL (#1079)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-22 00:07:15 -08:00
Michael Stack ba3463d9de HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 15:02:40 -08:00
Peter Somogyi 00e64d83e8
HBASE-23347 Allow custom authentication methods for RPCs; addendum (#1060)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-21 18:47:13 +01:00
Duo Zhang 167892ce64
HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
2020-01-18 20:30:47 +08:00
Peter Somogyi 65bcf55892
HBASE-23653 Expose content of meta table in web ui; addendum (#1061)
Fix error prone problem

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-18 09:33:21 +01:00
Viraj Jasani 3b64ea50ec
HBASE-23569 : Validate that all default chores of HRegionServer are scheduled (ADDENDUM)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-01-16 22:46:28 -08:00
Nick Dimiduk 00fc46756a
HBASE-23653 Expose content of meta table in web ui (#1020)
Adds a display of the content of 'hbase:meta' to the Master's
table.jsp, when that table is selected. Supports basic pagination,
filtering, &c.

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-16 08:46:39 -08:00
Josh Elser 04d789f1a7 HBASE-23347 Allow custom authentication methods for RPCs
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.

Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.

Closes #884

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-01-16 10:27:28 -05:00
Duo Zhang 0321f56765
HBASE-23652 Move the unsupported procedure type check before migrating to RegionProcedureStore (#1018)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-16 22:58:16 +08:00
Wellington Ramos Chevreuil 19d3bed1d4 HBASE-23694 After RegionProcedureStore completes migration of WALProcedureStore, still running WALProcedureStore.syncThread keeps trying to delete now inexistent log files. (#1048)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-16 14:12:37 +08:00
dingbaosheng 278d9fd153 HBASE-23674 Too many rit page Numbers show confusion 2020-01-15 13:54:29 -08:00
WenFeiYi a44f3b50e4 HBASE-23677 fix 500/NPE of region.jsp (#1033)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-01-15 18:06:57 +00:00
Wellington Chevreuil ceaeece2c5 Revert "fix 500/NPE of region.jsp (#1033)"
Missed the jira number on that commit message. Will re-apply it with the jira number.

This reverts commit d60ce17c17.
2020-01-15 18:05:23 +00:00
Bharath Vissapragada fd05aabf02 HBASE-23665: Split unit tests from TestTableName into a separate test-only class. (#1032)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-15 09:16:28 -08:00
Wellington Ramos Chevreuil cb78b103a7
HBASE-23683 Make HBaseInterClusterReplicationEndpoint more extensible (#1027)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-01-15 11:48:29 +00:00
WenFeiYi d60ce17c17 fix 500/NPE of region.jsp (#1033)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-01-15 11:32:27 +00:00
Viraj Jasani a075d6a011
HBASE-23569 : Validate that all default chores of HMaster are scheduled
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-01-14 23:03:16 -08:00
Michael Stack 8b7b097905 HBASE-23687 DEBUG logging cleanup (#1040)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-14 22:07:55 -08:00
Nick Dimiduk 499ff32f00
HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing (#917)
The current process for an operator, after fixing holes in meta, is to
manually disable and enable the whole table. Let's try to avoid
bringing the whole table offline if we can. Have the master attempt to
queue up assignment procedures for any new regions it creates.

Signed-off-by: stack <stack@apache.org>
2020-01-13 16:34:02 -08:00
Josh Elser 1d2c3efc69 HBASE-23679 FileSystem objects leak when cleaned up in cleanupBulkLoad
The cleanupBulkLoad method is only called for the first Region in the
table which was being bulk loaded into. This means that potentially N-1
other RegionServers (where N is the number of RegionServers) will leak
one FileSystem object into the FileSystem cache which will never be
cleaned up. We need to do this clean-up as a part of secureBulkLoadHFiles
otherwise we cannot guarantee that heap usage won't grow unbounded.

Closes #1029

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-13 18:30:30 -05:00
Duo Zhang 4ad12e03b8
HBASE-23681 Add UT for procedure store region flusher (#1024)
Signed-off-by: stack <stack@apache.org>
2020-01-12 13:22:48 +08:00
stack c8496bc694 Revert "HBASE-23055 Alter hbase:meta"
This reverts commit 9abdb7b5ae.
2020-01-11 09:17:17 -08:00
Viraj Jasani 2e0edacf72
HBASE-23662 : Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-10 20:42:21 -08:00
stack 4ed466966c Revert "HBASE-23668 Master log start filling with "Flush journal status" messages"
Minor addendum fixing log message.
2020-01-10 15:59:02 -08:00
stack 8ca614857d HBASE-23668 Master log start filling with "Flush journal status" messages"
This reverts commit fb9fa04da7.
i.e. reapplication of patch that was preamaturely applied.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-10 10:28:04 -08:00
Guanghao Zhang 56842d04d7
HBASE-23659 BaseLoadBalancer#wouldLowerAvailability should consider region replicas (#1001)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-10 15:02:53 +08:00
stack 9abdb7b5ae HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
    was hardcoded ENABLED. Make it dynamic. State is now kept in
    current active Master. It is transient so falls back to default
    if Master crashes. Add to registry a getMetaTableState
    which reads mirrored state from zookeeper (NOT from Master and
    defaults ENABLED if no implementation or error fetching state).
    hbase:meta schema will be bootstrapped from the filesystem.
    Changes to filesystem schema are atomic so we should be ok if
    Master fails mid-edit (TBD). Undoes a bunch of guards that
    prevented our being able to edit hbase:meta.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     Add fetching of hbase:meta table state from registry. Adds cache of
     tablestates w/ a ttl of 1 second (adjustable).

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
      Add querying registry for hbase:meta table state.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
      Add querying of mirrored table state for hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta. Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-09 16:25:16 -08:00
belugabehr f3bdb02280 HBASE-23378: Clean Up FSUtil setClusterId (#910)
Signed-off-by: stack <stack@apache.org>
2020-01-09 12:47:12 -08:00
Aman Poonia 12b79a3907 HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-09 11:49:48 -08:00
Guanghao Zhang fbf660c96b
HBASE-23658 Fix flaky TestSnapshotFromMaster (#998)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-09 14:38:52 +08:00
binlijin 9a2a799c96
HBASE-23660 hbase:meta's table.jsp ref to wrong rs address (#999)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-01-09 10:25:54 +08:00
binlijin 66d1258aa3
HBASE-23651 Region balance throttling can be disabled (#991)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-01-08 19:01:22 +08:00
binlijin 33f45d4413
HBASE-23636 Disable table may hang when regionserver stop or abort. (#982)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: virajjasani <34790606+virajjasani@users.noreply.github.com>
2020-01-06 15:19:10 +08:00