17362 Commits

Author SHA1 Message Date
GeorryHuang
e11dbf36a5
HBASE-23855: Change bytes size to human readable size for Server Metrics of RegionServer Web UI (#1175)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-17 09:14:53 +08:00
Jan Hentschel
68f4fd1ee2
HBASE-23844 Removed deprecated Scan(byte[], Filter) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-16 21:51:43 +01:00
Jan Hentschel
ae781e569c
HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-15 21:19:51 +01:00
stack
0e96638898 HBASE-23838 Adding debug logging to a few ExportSnapshot tests
ADDENDUM to fix failing TestExportSnapshot
2020-02-14 16:09:51 -08:00
Sakthi
5f61df4792
HBASE-22827 Expose multi-region merge in shell and Admin API (#1138)
mergeRegionsAsync admin API with two regions as parameters deprecated since 2.3.0 and removed from 4.0
merge_region shell command now supports multiple regions merge since 2.3.0 & 3.0.0

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-13 14:40:46 -08:00
Nick Dimiduk
533302adde
HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… (#1141)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-02-13 14:02:15 -08:00
stack
b7ef225609 HBASE-23838 Adding debug logging to a few ExportSnapshot tests
* hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
 Add logging of temp output dir and if tmp dir is being used at all.

* hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Don't create new Configurations. Use current. Set it into the launched
 ExportSnapshot too. Log as we make progress through stages so easier to
 find location of exception source.

* hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
 Add makeQualified though looks to be redundant.

* hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestInfoServersACL.java
 Might fix a local fail where hbase doesn't seem to be up when we query
 jmx. Wait on meta being online.

M hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionStateStore.java
 Run all on the one cluster instance. See if helps w/ odd failure when
 run locally.
2020-02-13 13:50:43 -08:00
Michael Stack
488c2b2483 HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS! ADDENDUM: Break test in two to see if it makes it more reliable. (#1162)
ADDENDUM that breaks test in two to see if it makes it more reliable.
2020-02-12 09:38:36 -08:00
Pankaj Kumar
4c55c06ff4 Adding Pankaj to the developer list 2020-02-12 19:55:22 +05:30
maoling
12f98bc50e
HBASE-20623: Introduce the helper method "getCellBuilder()" to Mutation (#1121)
Signed-off-by: stack <stack@apache.org>
2020-02-10 10:03:35 -08:00
stack
a6774ca7f7 HBASE-23812 [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7> 2020-02-10 09:45:43 -08:00
meiyi
29a36568b1
HBASE-23824 TestSnapshotScannerHDFSAclController is flakey (#1157)
Signed-off-by: stack <stack@apache.org>
2020-02-10 09:40:05 -08:00
stack
cdc98e906c HBASE-23816 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotV2WithRefsExportFileSystemState(TestExportSnapshotNoCluster.java:91) Wrong FS! 2020-02-10 09:06:50 -08:00
stack
93c04e4f60 HBASE-23815 [Flakey Test] AbstractTestAsyncTableRegionReplicasRead family of tests fails with no breadcrumbs on why 2020-02-10 09:04:36 -08:00
Wellington Ramos Chevreuil
069df4d791
HBASE-23803 [DOC] Fix the maths on the section explaining call queue tuning options (#1128)
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
2020-02-10 16:55:52 +00:00
stack
a4aa183f45 HBASE-23814 Add null checks and logging to misc set of tests
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Zach York <zyork@apache.org>

* TestFullLogReconstruction log the server we've chosen to expire and then note where we starting counting rows
* TestAsyncTableScanException use a define for row counts
* TestRawAsyncTableLimitedScanWithFilter check connection was made before closing it in tearDown
* TestLogsCleaner use single mod time. Make it for sure less than now in case test runs all in the same millisecond (would cause test fail)
* TestReplicationBase test table is non-null before closing in tearDown
2020-02-10 08:51:56 -08:00
GeorryHuang
35989c1968
HBASE-23822:Fix typo in procedures.jsp (#1155)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-10 09:48:01 +08:00
binlijin
24823ecfc9
HBASE-23682 Fix NPE when disable DeadServerMetricRegionChore (#1026)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-08 15:20:23 -08:00
Michael Stack
348ee5966b HBASE-23813 Get surefire version from parent pom rather than define it in hbase-archetypes (#1145)
Signed-off-by: Nick Dimiduk ndimiduk@apache.org
Signed-off-by: Jan Hentschel janh@apache.org
Signed-off-by: Zach York zyork@apache.org
2020-02-08 09:17:22 -08:00
Duo Zhang
4479cb2313 HBASE-23809 Addendum remove rs group code in AbstractTestShell 2020-02-08 22:46:08 +08:00
Duo Zhang
546828685e
HBASE-23809 The RSGroup shell test is missing (#1142)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-08 17:10:00 +08:00
stack
d554689cc4 HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
FINAL ADDENDUM. Removes changes to dev-support/hbase-personality
 leaving it as it was. All else about HBASE-23779 change remains
 changing surefire fork counts to be dependent on cpu count.
2020-02-07 15:53:15 -08:00
stack
ef380e0a2e HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
ADDENDUM: Refactor that comes of discussion up on https://github.com/apache/yetus/pull/86
 because what I committed originally, and amended in a subsequent
 ADDENDUM is not taking effect.
2020-02-07 09:40:51 -08:00
stack
8654597be0 HBASE-23805 [Flakey Test] TestRaceBetweenSCPAndDTP 2020-02-07 08:35:43 -08:00
Jan Hentschel
93c5e7691f
HBASE-23775 Removed deprecated method createLocalHTU(Configuration) from HBaseTestingUtility
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-07 09:51:04 +01:00
Bharath Vissapragada
1295d058ad
Add Bharath Vissapragada to developers list. (#1139) 2020-02-06 13:35:57 -08:00
stack
867b1e9cbc HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
ADDENDUM: Try this way of setting MAVEN_ARGS
2020-02-05 21:47:34 -08:00
Nick Dimiduk
5b4545de5e
HBASE-23802 Remove unnecessary Configuration instantiation in LossyAccounting (#1127)
Signed-off-by: stack <stack@apache.org>
2020-02-05 14:41:50 -08:00
Michael Stack
b49ec58073 HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
Set the fork count for first and second parts to be 0.5C. Add a bit of
doc too on this as well as some qualification on our test categories.
Also adds -T0.5C to MAVEN_ARGS in the hbase personality.
2020-02-05 11:09:44 -08:00
stack
3a1a39d40d HBASE-23789 [Flakey Tests] ERROR [Time-limited test] balancer.HeterogeneousRegionCountCostFunction(199): cannot read rules file located at ' /tmp/hbase-balancer.rules '; ADDENDUM Missed adding these files. 2020-02-04 16:40:53 -08:00
stack
9cc7a711e6 HBASE-23789 [Flakey Tests] ERROR [Time-limited test] balancer.HeterogeneousRegionCountCostFunction(199): cannot read rules file located at ' /tmp/hbase-balancer.rules ' 2020-02-04 16:40:50 -08:00
Nick Dimiduk
6ba1df3b39
HBASE-23792 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState (#1124)
1. Survive flakey rerunning by converting the static BeforeClass stuff
   into instance-level Before.
2. Break the test method into two, one for running over each of the
   snapshot manifest versions.

Signed-off-by: stack <stack@apache.org>
2020-02-04 16:04:14 -08:00
Mark Robert Miller
299b6bebc5
HBASE-23783: Address tests writing and reading SSL/Security files in a common location. (#1116)
This is causing me issues with parallel test runs.

Also allow setting the surefire reports and temp directories via command line.

Signed-off-by: stack <stack@apache.org>
2020-02-04 14:38:56 -08:00
Nick Dimiduk
1cacf27d5c
HBASE-23793 Increase maven heap allocation to 4G in Yetus personality (#1122)
I saw this over on
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2447/console. Looks
like we need to bump the memory allocation for maven. I wonder if this
is the underlying cause of HBASE-22470.

```
6:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47                                Finished build.
16:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47
16:38:47
Post stage
[Pipeline] stash
16:38:48  Warning: overwriting stash 'hadoop2-result'
16:38:48  Stashed 1 file(s)
[Pipeline] junit
16:38:48  Recording test results
16:38:54  Remote call on H2 failed
Error when executing always post condition:
java.io.IOException: Remote call on H2 failed
	at hudson.remoting.Channel.call(Channel.java:963)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
	at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append(XMLStringBuffer.java:208)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(XMLEntityScanner.java:1515)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(XMLDocumentFragmentScannerImpl.java:1654)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3014)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
	at org.dom4j.io.SAXReader.read(SAXReader.java:465)
	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:348)
	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:206)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:178)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:143)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	... 4 more

[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
16:38:54  Failed in branch yetus jdk8 hadoop2 checks
```

Signed-off-by: stack <stack@apache.org>
2020-02-04 09:14:01 -08:00
Duo Zhang
bb14bdad62 HBASE-23782 Addendum fix error prone warning 2020-02-04 10:14:27 +08:00
stack
d797ed04e5 HBASE-23786 [Flakey Test] TestMasterNotCarryTable.testMasterMemStoreLAB 2020-02-03 17:34:22 -08:00
Jan Hentschel
b591a3d205
HBASE-23772 Removed deprecated getTimeStampOfLastShippedOp from MetricsSource
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-04 00:25:58 +01:00
Michael Stack
661abeb730 HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 10:28:19 -08:00
Duo Zhang
ab0039974d
HBASE-23782 We still reference the hard coded meta descriptor in some places when listing table descriptors (#1115)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 14:15:57 +08:00
Jan Hentschel
12f1c7c1ce
HBASE-23777 Removed deprecated createTableDescriptor(String, int, int, int, KeepDeletedCells) from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 00:24:19 +01:00
Jan Hentschel
5a3ad6f08f
HBASE-23776 Removed deprecated method createLocalHTU() from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
2020-02-02 00:28:25 +01:00
stack
9b6d25066d HBASE-23764: Switch to IP address for ZK ensemble
ADDENDUM removing comment just to trigger full test suite in pom.xml
2020-02-01 14:59:01 -08:00
Bharath Vissapragada
d9453203ae HBASE-23764: Switch to IP address for ZK ensemble
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 14:57:51 -08:00
Josh Elser
70699365ad HBASE-23774 Add the user-zh mailing list 2020-01-31 19:33:59 -05:00
stack
ba9f026d69 HBASE-23770 [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart 2020-01-31 08:39:46 -08:00
stack
1a6bdc7c50 HBASE-23771 [Flakey Tests] Test TestSplitTransactionOnCluster Again 2020-01-31 08:35:58 -08:00
abhinaba.sarkar
77229c79e3 HBASE-23350 Make compaction files cacheonWrite configurable based on threshold
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-31 11:35:11 +05:30
Nick Dimiduk
85b0c8e6bc
HBASE-23763 Add 'new on release line report' for git/jira audit tool (#1104)
Codify building the summary of what's new on a release line
branch (i.e., `branch-2`), but not yet released on earlier release
branches of that line.

Builds a cvs report that looks like https://home.apache.org/~ndimiduk/new_for_branch-2.csv
2020-01-30 09:31:01 -08:00
Josh Elser
8b00f9f0b1 HBASE-17115 Define UI admins via an ACL
The Hadoop AccessControlList allows us to specify admins of the webUI
via a list of users and/or groups. Admins of the WebUI can mutate the
system, potentially seeing sensitive data or modifying the system.

hbase.security.authentication.spnego.admin.users is a comma-separated
list of users who are admins.
hbase.security.authentication.spnego.admin.groups is a comma-separated
list of groups whose membership are admins. Either of these
configuration properties may also contain an asterisk (*) which denotes
"any entity" (e.g user, group).

Previously, when a user was denied from some endpoint that was
designated for admins, they received an HTTP/401. In this case, it is
more correct to return HTTP/403 as they were correctly authenticated,
but they were disallowed from fetching the given resource. This commit
incorporates this change.

hbase.security.authentication.ui.config.protected also exists for users
who have sensitive information stored in the Hadoop service
configuration and want to limit access to this endpoint. By default,
the Hadoop configuration endpoint is not protected and any
authenticated user can access it.

The test is based off of work by Nihal Jain in HBASE-20472.

Co-authored-by: Nihal Jain <nihaljain.cs@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-29 16:36:55 -05:00
Viraj Jasani
66d198d35f
HBASE-23749 : TestHFileWriterV3 for all DataBlock Encoders
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-29 23:54:18 +05:30