Commit Graph

17534 Commits

Author SHA1 Message Date
Nick Dimiduk 040a0a7ac5 HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-01 12:28:10 -07:00
stack e197aa303a HBASE-24100 [Flakey Tests] Add test to check we work properly when port clash setting up thriftserver 2020-04-01 12:25:02 -07:00
Nick Dimiduk e85dfdc511 HBASE-24092 Fix links to build reports generated by nightly job
Signed-off-by: stack <stack@apache.org>
2020-04-01 12:16:53 -07:00
Michael Stack 0595200fb5 HBASE-24097 [Flakey Tests] TestSnapshotScannerHDFSAclController#testRestoreSnapshot (#1405)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationTrackerZKImpl.java
 Add debug for when assert fails (it fails on occasion locally)

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestHDFSAclHelper.java
 Move this inner class out standalone since it used now by two tests.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Moved out testRestoreSnapshot and made methods in here static so could
 be used by a new adacent test. Also made tablenames unique to methods
 thinking that was root of original issue (wasn't but no harm in doing
 this change) Moved out the inner class TestHDFSAclHelper.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController2.java
 New class that sets up same context as
 TestSnapshotScannerHDFSAclController but just to run single
 testRestoreSnapshot method.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java
 Some debug.

Signed-off-by: Yi Mei
2020-04-01 08:38:05 -07:00
Viraj Jasani 3e85407d43
HBASE-23937 : Documentation for get_largelog_responses command (ADDENDUM) 2020-04-01 20:13:11 +05:30
Viraj Jasani 0dcbf80583
HBASE-23937 : Support Online LargeLogs similar to SlowLogs APIs (#1346)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-01 18:22:22 +05:30
Jan Hentschel fb17ecdd07
HBASE-23842 Removed deprecated Scan(byte[], byte[]) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-01 13:11:21 +02:00
Viraj Jasani 834f89da7b
HBASE-23678 : Builder API for version management - setVersionsWithTim… (#1381)
Signed-off-by: Xu Cang <xucang@apache.org>
2020-04-01 16:16:40 +05:30
binlijin 52c101389e
HBASE-24096 Fix TestRSGroupsBalance (#1402)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-01 14:16:57 +08:00
Bharath Vissapragada 6f213e9d5a HBASE-24075: Fix a race between master shutdown and metrics (re)init
JMXCacheBuster resets the metrics state at various points in time. These
events can potentially race with a master shutdown. When the master is
tearing down, metrics initialization can touch a lot of unsafe state,
for example invalidated FS objects. To avoid this, this patch makes
the getMetrics() a no-op when the master is either stopped or in the
process of shutting down. Additionally, getClusterId() when the server
is shutting down is made a no-op.

Simulating a test for this is a bit tricky but with the patch I don't
locally see the long stacktraces from the jira.

Signed-off-by: Michael Stack <stack@apache.org>
2020-03-31 16:36:31 -07:00
Nick Dimiduk e5a8435b48 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:22:46 -07:00
Jan Hentschel fded2b9ddc
HBASE-23846 Removed deprecated setMaxVersions(int) from Scan
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-31 10:18:27 +02:00
stack 3d4124c92a HBASE-24079 [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add waits on quota table to come online; etc.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java
 Refactor to avoid NPE timing issue referencing lock during Construction.

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
 Comment

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
 Refactor. Catch NPE during startup and return it instead as failed initialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
 Catch IndexOutOfBounds exception and convert to non-split request.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
 Make less furious. Make it less flakie.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java
 Debug. Catch exception to log, then rethrow.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Guess that waiting longer on compaction to succeed may help make this
 less flakey.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Be explicit about timestamping to avoid concurrent edit landing
 server-side and messing up test expectation.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMasterRegistry.java
 Add wait on meta before proceeding w/ test.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
 Be explicit that edits are distinct.

hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCacheRefCnt.java
 Add @Ignore on RAM test... Fails sporadically.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
 Add wait for all RegionServers going down before proceeding; was
 messing up RS accounting.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Make balancer test sloppier; less restrictive; would fail on occasion
 by being just outside test limits.

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table coming up; helps make this less flakie.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Be explicity about timestamps; see if helps w/ flakie failure.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch and ignore if issue in shutdown; don't care if after test.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
 Comment.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 Add retry to see if helps w/ odd failure; grant hasn't propagated?

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
 Explicit w/ timestamps so no accidental overlap of puts.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
 Hack to deal w/ BindException on startup.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThrift2ServerCmdLine.java
 Use loopback.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
 Disable flakie test.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-03-30 17:08:17 -07:00
stack 030e833dc9 HBASE-24052 Add debug+fix to TestMasterShutdown
Add check for stopped server at a few more points in Master startup.
Defend against NPE in RSProcedureDispatcher; log and retun instead.
2020-03-30 16:07:35 -07:00
huaxiangsun d318ca1741
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1387)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-30 12:11:26 -07:00
huaxiangsun f365cb5264
Revert "HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1378)" (#1386)
This reverts commit d319ce8c6f.
2020-03-30 10:06:13 -07:00
Nick Dimiduk d77dbef5c6
HBASE-24007 Get `-PrunLargeTests` passing on JDK11 (#1351)
Minor tweaks required to get passing runs of `-PrunLargeTests`.
* Minimum Hadoop version is 3.2.0 due to
  [HADOOP-12760](https://issues.apache.org/jira/browse/HADOOP-12760).
* JDK11 looks like it consumes more memory than JDK8, so failures due
  to OOME see more common here. Bumping heap allocated to surefire
  forks allows better pass rate.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-30 09:31:10 -07:00
huaxiangsun d319ce8c6f
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1378)
Co-authored-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-03-30 09:29:18 -07:00
Duo Zhang 5d4e020c3d
HBASE-23798 Remove hbase-prototcol module (#1360)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-30 09:54:59 +08:00
Jan Hentschel aaae46c976
HBASE-23845 Removed deprecated setMaxVersions from Scan (#1208)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-29 17:46:49 +08:00
Peter Somogyi 3f2489467f
HBASE-24057 Add modules to mapreduce job classpaths (#1359)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-27 11:14:37 +01:00
WenFeiYi eb1fec1c53
HBASE-24040 WALFactory.Providers.multiwal causes StackOverflowError (#1338)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-27 10:04:52 +08:00
Josh Elser 78eecd3a7d HBASE-23381 Ensure Netty client receives at least one response before considering SASL negotiation complete
The PLAIN mechanism test added in the Shade authentication example has
different semantics than GSSAPI mechanism -- the client reports that the
handshake is done after the original challenge is computed. The javadoc
on SaslClient, however, tells us that we need to wait for a response
from the server before proceeding.

The client, best as I can see, does not receive any data from HBase;
however the application semantics (e.g. throw an exception on auth'n
error) do not work as we intend as a result of this bug.

Extra trace logging was also added to debug this, should a similar error
ever happen again with some other mechanism.

Closes #1260

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-03-26 18:34:47 -04:00
huaxiangsun 50c8f49c31
HBASE-23853 [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits (#1363)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
  Signed-off-by: Viraj Jasani <vjasani@apache.org>
  Signed-off-by: <stack@apache.org>
2020-03-26 14:06:34 -07:00
Nick Dimiduk d05fccfc24 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh ADDENDUM
* Correctly handle the hadoop2 profile.
* Create the output directory if it doesn't exist
2020-03-26 08:43:50 -07:00
Semen Komissarov d7ff742ce8
HBASE-23975 make hbase-rest use shaded protobuf (#1325)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-26 21:35:36 +08:00
Peter Somogyi 9042bc0058
HBASE-22555 Re-enable TestMasterOperationsForRegionReplicas (#1345)
Signed-off-by: stack <stack@apache.org>
2020-03-26 11:01:19 +01:00
Duo Zhang f3ee9b8aa3
HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 17:05:31 +08:00
Duo Zhang 76e3db6720
HBASE-23981 PBType which is marked as IA.Public references protobuf Message (#1323)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 10:38:41 +08:00
Nick Dimiduk 5e63509781 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh
Signed-off-by: stack <stack@apache.org>
2020-03-25 13:46:11 -07:00
Guanghao Zhang eed730e492
HBASE-24037 Add ut for root dir and wal root dir are different (#1336)
Signed-off-by: stack <stack@apache.org>
2020-03-25 10:53:14 +08:00
Jan Hentschel aa966e354a
HBASE-23983 Fixed Spotbugs complaint in RegionStates related to ignored return value
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-25 00:02:54 +01:00
Wei-Chiu Chuang 84977eeebb
HBASE-8868. add metric to report client shortcircuit reads. (#1334)
Signed-off-by: stack <stack@apache.net>
2020-03-24 15:30:08 -07:00
stack 2ca0a105bc HBASE-24043 [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
 Edit of log about archiving that shows in middle of a table create;
 try to make it less disorientating.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Loosen assert. Compaction may have produced a single file only. Allow
 for this.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
 Make this test less furious given it is inline w/ a bunch of unit
 tests.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Add debug

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table to show up before moving forward; otherwise,
 attempt at quota setting fails.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Remove asserts that expected regions to still have a presence in fs
 after merge when a catalogjanitor may have cleaned up parent dirs.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch exception on way out and log it rather than let it fail test.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Wait on acl table before proceeding.
2020-03-24 14:52:24 -07:00
Esteban Gutierrez 9a212ee487
HBASE-24041 [regression] Increase RESTServer buffer size back to 64k (#1339)
* HBASE-24041 [regression] Increase RESTServer buffer size back to 64k

* HBASE-24041 [regression] Increase RESTServer buffer size back to 64k
2020-03-24 15:12:02 -05:00
Nick Dimiduk aa53493ae8 HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:31:40 -07:00
Reid Chan c3a704ea22
HBASE-24032 [RSGroup] Assign created tables to respective rsgroup automatically instead of manual operations (#1319)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-24 13:33:00 +08:00
Reid Chan bdcfd6a4a8
[RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)
Signed-off-by: stack <stack@apache.org>
2020-03-24 10:34:12 +08:00
stack afc1746757 HBASE-24034 [Flakey Tests] A couple of fixes and cleanups
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
 Remove spurious assert. Just before this it waits an arbitrary 10
 seconds. Compactions could have completed inside this time. The spirit
 of the test remains.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
 Get log cleaner to go down promptly; its sticking around. See if this
 helps with TestMasterShutdown

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
 We get a rare NPE trying to sync. Make local copy of SyncFuture and see
 if that helps.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Compaction  may have completed when not expected; allow for it.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
 Add wait before testing. Compaction may not have completed. Let
 compaction complete before progressing and then test for empty cache.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 Wait till online before we try and do compaction (else request is
 ignored)

hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
 Disable test that fails randomly w/ mockito complaint on some mac os
 x's.

TestMasterShutdown... fix NPE in RSRpcDispatcher... catch it and covert
to false and have master check for successful startup.
2020-03-23 16:26:32 -07:00
Nick Dimiduk af4fdcc556 HBASE-23980 Use enforcer plugin to print JVM info in maven output
Does what it says on the tin. Bound to `initialize` phase so that it
runs early in lifecycle. Uses `<inherited>false</inherited>` so that
the plugin will run only for the base pom's reactor stage and not for
any children.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-23 14:27:55 -07:00
stack 2f26d8a623 HBASE-24035 [Flakey Tests] Disable TestClusterScopeQuotaThrottle#testUserNamespaceClusterScopeQuota 2020-03-23 13:44:34 -07:00
Viraj Jasani 7081ccd53c
HBASE-23936 : Thrift support for Online SlowLog APIs (#1317)
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-03-23 22:59:04 +05:30
Guanghao Zhang 8130858199
HBASE-23741 Data loss when WAL split to HFile enabled (#1254)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-23 14:38:47 +08:00
Duo Zhang eface74407
HBASE-23799 Make our core coprocessors use shaded protobuf (#1280)
Signed-off-by: stack <stack@apache.org>
2020-03-23 08:47:02 +08:00
Huaxiang Sun 542244056c
HBASE-23957 [flakey test] client.TestMultiParallel fails to read hbase-site.xml (#1310)
Signed-off-by: Nick Dimiduk ndimiduk@apache.org
Signed-off-by: stack <stack@apache.org>
2020-03-22 17:43:04 -07:00
Guanghao Zhang 2a60a61a73
HBASE-24033 Add ut for loading the corrupt recovered hfiles (#1322)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-22 22:56:30 +08:00
niuyulin d4ab645531
HBASE-23949 refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced (#1279)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 21:54:57 +08:00
Pankaj ca533a77cb
HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:39:56 +08:00
Toshihiro Suzuki 9b4e75a7f6
HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 11:55:52 +09:00
Toshihiro Suzuki b53fa3c51f
HBASE-24031 TestHRegion.testCheckAndMutate_WithFilters is flaky (#1316)
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2020-03-22 05:04:16 +09:00