Commit Graph

17535 Commits

Author SHA1 Message Date
Viraj Jasani b6b8c6973d
HBASE-24140 : Move CandidateGenerator and their implementors out of StochasticLoadBalancer (#1458)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-12 19:13:47 +05:30
Wei-Chiu Chuang a9210ef5fe
HBASE-23998 Addendum Add sign-offs
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-12 11:20:41 +02:00
Wei-Chiu Chuang b3461665e6
HBASE-23998. Update license for jetty-client. (#1297)
(cherry picked from commit 5be67a5e60182f5f06c7a84dbae6fb365f28682b)
2020-04-12 10:45:24 +02:00
Michael Stack 14342b6527
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-10 15:57:47 -07:00
leosunli 12a0c1d5fe
Remove unnecessary super() in MultiVersionConcurrencyControl#MultiVersionConcurrencyControl() (#1468)
Signed-off-by: sunlisheng <lisheng.sun08@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-10 15:43:10 -07:00
Nick Dimiduk b3488fcdec HBASE-24160 create-release fails to process x.y.0 version info correctly
Current code gives me the following; notice default values generated
for `RELEASE_VERSION` and `api_diff_tag`.

```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.-1]: 2.3.0
NEXT_VERSION [2.3.0-SNAPSHOT]: 2.3.1-SNAPSHOT
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0)]:
```

With this patch I get

```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.0]:
NEXT_VERSION [2.3.1-SNAPSHOT]:
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0]:
```

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-10 12:02:24 -07:00
Sean Busbey 09d20fb4d7
HBASE-24163 MOB compactor implementations should use format specifiers when calling String.format (#1479)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-04-10 11:00:53 -05:00
huaxiangsun ac4e94e868
HBASE-24161 [flakey test] locking.TestEntityLocks.testEntityLockTimeout (#1477)
Signed-off-by: stack <stack@apache.org>
2020-04-09 21:13:06 -07:00
Pankaj c92dd28a88
HBASE-24074 ConcurrentModificationException occured in ReplicationSourceManager while refreshing the peer (#1471)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-09 16:40:15 -07:00
huaxiangsun 8aa3304427
HBASE-24159 [flakey test] regionserver.TestRegionMergeTransactionOnCluster (#1472)
Delete tables after each test cases to avoid polluting states.

Signed-off-by: stack <stack@apache.org>
2020-04-09 16:27:19 -07:00
Nick Dimiduk b4af0a0943 HBASE-24136 Add release branch report to git-jira-release-audit tool
* enhancements to git_jira_release_audit.py
  * add aforementioned release branch report
  * include default values in help doc output
  * swap default db to a file on disk instead of in memory
  * set logger to match file name
  * add separate sql query log at DEBUG level
* more detailed usage info in README.md, including example audit query
* update entries in fallback_actions.csv

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-09 09:28:36 -07:00
Reid Chan 1fb693b998
HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be (#1439)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-04-09 16:28:22 +08:00
Viraj Jasani 2907260204
HBASE-24131 : TestExportSnapshot.testExportFileSystemState() changes with correct arg (ADDENDUM) 2020-04-09 10:32:55 +05:30
Sean Busbey eb7df0498c
HBASE-23723 Ensure MOB compaction works in optimized mode after snapshot clone (#1446)
* Reorganize MOB compaction tests for more reuse.
* Add tests for mob compaction after snapshot clone operations
* note the original table used to write a given mob hfile and use that to find it later.

Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2020-04-08 14:16:43 -05:00
Sean Busbey 2d78a286b6
HBASE-24138 log more details about balancer decisions for StochasticLoadBalancer (#1455)
- at DEBUG log messages about RegionCountSkewCostFunction region/server totals
- at DEBUG log messages about the decision to balance or not with total costs
- at TRACE log messages about region count on each server RegionCountSkewCostFunction sees
- at TRACE log message with the individual cost functions used in the decision to balance or not

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-08 14:03:33 -05:00
Andrew Purtell ac6060c3f9
HBASE-24099 Use a fair ReentrantReadWriteLock for the region close lock (#1407)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-08 11:36:48 -07:00
Wei-Chiu Chuang 72727ff9be
HBASE-23833. The relocated hadoop-thirdparty protobuf breaks HBase asyncwal (#1301)
* Use Reflection to access shaded Hadoop protobuf classes.

(cherry picked from commit a321e536989083ca3620bf2c53f12c07740bf5b0)

* Update to improve the code:

1. Added license.
2. Added more comments.
3. Wrap byte array instead of copy to make a ByteString.
4. Moved all reflection instantiation to static class loading time.

* Use LiteralByteString to wrap byte array instead of copying it.


Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-08 11:31:51 -07:00
Michael Stack 082ebdd53c HBASE-24131 [Flakey Tests] TestExportSnapshot takes too long; up against 13min max (#1452)
Split TestExportSnapshot and TestVerifyReplication to two smaller tests
rather than one big one that can take 13minutes+ when contention.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-04-08 11:04:00 -07:00
huaxiangsun ed830222da
HBASE-24120 Flakey Test: TestReplicationAdminWithClusters timeout (#1441)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-08 09:28:51 -07:00
Duo Zhang c383bd1bb3
HBASE-24062 Add 2.1.10 to download page (#1454)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-08 19:56:31 +08:00
huaxiangsun d6451a8e2b
HBASE-24114 [Flakey Tests] TestSnapshotScannerHDFSAclController (#1433)
Signed-off-by: stack <stack@apache.org>
2020-04-07 20:36:11 -07:00
Michael Stack 80d4a09ddc HBASE-24128 [Flakey Tests] Add retry on thrift cmdline if client fails plus misc debug (#1442)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterPortAssignment.java
 Saw case where Master failed startup but it came out as an IOE so we
 did not trip the retry logic.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
 Add some debug and up timeouts. This test fails frequently for me
 locally.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/locking/TestEntityLocks.java
 Up the wait from 2x 200ms to 10x in case a pause on hardware or GC.
 This test fails locally and up on jenkins.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClearRegionBlockCache.java
 Debug. Have assert say what bad count was.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellFlatMapMemStore.java
 Fails on occasion. Found count is off by a few. Tricky to debug. HBASE-24129 to reenable.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Debug. Add wait and check before moving to assert.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Check for null before shutting; can be null if failed start.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
 Add retry if client messes up connection. Fails for me locally.
2020-04-07 09:29:23 -07:00
Minwoo Kang e71c44203e
HBASE-24130 rat plugin complains about having an unlicensed file (#1445)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-07 09:16:10 -07:00
Duo Zhang f60f0bdbf0
HBASE-24055 Make AsyncFSWAL can run on EC cluster (#1437)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-07 23:41:35 +08:00
binlijin 62718cdb28
HBASE-24077 When encounter RowTooBigException, log the row info. (#1379)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-07 10:28:35 +08:00
Nick Dimiduk 15b94cc306 HBASE-24081 Provide documentation for running Yetus with HBase
Signed-off-by: stack <stack@apache.org>
2020-04-06 16:35:57 -07:00
Andrew Purtell 43d7c2cda6
HBASE-24115 Relocate test-only REST "client" from src/ to test/ and mark Private (#1434)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-04-06 16:18:40 -07:00
Nick Dimiduk 7de861bb83 HBASE-24106 Update getting started documentation after HBASE-24086
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-04-06 13:54:08 -07:00
stack 0e30e00c56 HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash
Reapply but as an @Ignore for the flakey test.
2020-04-06 12:38:02 -07:00
stack 0dc2712c90 Revert "HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash"
Reverting in favor of adding an @Ignore on this test until
root cause of flakyness HBASE-24117 is addressed.

This reverts commit 9985c06647.
2020-04-06 12:35:52 -07:00
stack 1a627f0501 HBASE-24122 Change machine ulimit-l to ulimit-a so dumps full ulimit rather than just 'max locked memory' 2020-04-06 10:14:21 -07:00
Duo Zhang 6452396187
HBASE-24119 Polish the protobuf usage in hbase-examples (#1438)
Move the proto files only used by hbase-examples to hbase-examples, to show users how to make use of shaded hbase protos when implementing coprocessor.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-06 22:14:55 +08:00
Peter Somogyi 1784938af7
HBASE-24111 Enable CompactionTool executions on non-HDFS filesystems (#1427)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-04-06 11:54:19 +02:00
Semen Komissarov 740e937210
HBASE-24056 Improve the layout of our proto files in hbase-protocol-shaded module (#1377)
proto files layout:
protobuf/client - client to server messages, client rpc service and protos, used in hbase-client exclusively;
protobuf/rest - hbase-rest messages;
protobuf/rpc - rpc and post-rpc tracing messages;
protobuf/server/coprocessor - coprocessor rpc services;
protobuf/server/coprocessor/example - coprocessors rpc services examples from hbase-examples;
protobuf/server/io - filesystem and hbase-server/io protos;
protobuf/server/maser - master rpc services and messages;
protobuf/server/region - region rpc services and messages (except client rpc service, which is in Client.proto);
protobuf/server/rsgroup - rsgroup protos;
protobuf/server/zookeeper - protos for zookeeper and ones used exclusively in hbase-zookeeper module;
protobuf/server - protos used across other server protos;
protobuf/test - protos used in tests;
protobuf/ - protos used across other protos, exclusive for hbase-mapreduce and hbase-backup, other protos.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-06 10:22:08 +08:00
huaxiangsun da67491a56
HBASE-24105 [Flakey Test] regionserver.TestRegionReplicas (#1425)
Co-authored-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-05 13:20:23 -07:00
Michael Stack 24ad05486f HBASE-24113 Upgrade the maven we use from 3.5.4 to 3.6.3 in nightlies (#1430)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-05 12:54:43 -07:00
stack 9985c06647 HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash 2020-04-04 20:13:18 -07:00
Nick Dimiduk ea69b8711f HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-03 16:57:09 -07:00
Nick Dimiduk 10cbb3fb73 HBASE-24086 Disable output stream capability enforcement when running on LocalFileSystem
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-03 09:17:03 -07:00
Viraj Jasani 2a95fffb3b
HBASE-24102 : Remove decommissioned RS from target servers while unlo… (#1417)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-04-03 18:28:49 +05:30
stack 4565f74af2 HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread
Set the forkcount down from 0.5C to 0.25C.
2020-04-02 21:07:25 -07:00
stack edca9c3a0e HBASE-24107 [Flakey Test] TestThriftServerCmdLine.testRunThriftServer NPEs if InfoServer port clash 2020-04-02 20:31:03 -07:00
huaxiangsun 37aa6690b5
HBASE-24080 [flakey test] TestRegionReplicaFailover.testSecondaryRegionKill fails. (#1421)
Signed-off-by: stack <stack@apache.org>
2020-04-02 14:55:55 -07:00
meiyi b2c9a06dfd
HBASE-24103 [Flakey Tests] TestSnapshotScannerHDFSAclController (#1416)
Signed-off-by: stack <stack@apache.org>
2020-04-02 09:55:47 -07:00
Nick Dimiduk 60959df706 HBASE-24084 Fix missing jdk8 dependencies in hbase-assembly/hadoop-two-compat
After HBASE-23986, running on JDK8/Hadoop2 from the binary tarball
results in

```
Exception in thread "main" java.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3062)
Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 2 more
```

The missing class is from `com.fasterxml.woodstox:woodstox-core`. This
and `org.codehaus.woodstox:stax2-api` are explicit dependencies of
`org.apache.hadoop:hadoop-common:jar:2.10.0`. Thus, include them in
the default jars provided under `lib`.

Signed-off-by: stack <stack@apache.org>
2020-04-02 09:51:08 -07:00
huaxiangsun f00753bacd
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1414)
Addendum:
  For major compaction test, set hbase.hstore.compaction.min to a big number to
  avoid kicking in minor compactions, which will pollute compaction state and
  sometimes, cause major compaction cannot happen.

Co-authored-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-02 08:16:27 -07:00
niuyulin 0b3918fe56
HBASE-24021 Fail fast when bulkLoadHFiles method catch some IOException (#1343)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-04-02 23:15:14 +08:00
Jan Hentschel 4ddf55d20a
HBASE-23843 Removed deprecated Scan(byte[]) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-02 11:44:34 +02:00
stack 4ea819b8f2 HBASE-24100 [Flakey Tests] Add test to check we work properly when port clash setting up thriftserver
Addendum; add missing @ClassRule
2020-04-01 17:50:25 -07:00
stack 554678ea4c Revert "HBASE-24051 Allows indirect inheritance to CanUnbuffer (#1406)"
This reverts commit ae1f928b13.
2020-04-01 15:50:25 -07:00