Commit Graph

601 Commits

Author SHA1 Message Date
Duo Zhang 997d132601
HBASE-27702 Remove 'hbase.regionserver.hlog.writer.impl' config (#5096)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2023-03-20 10:36:14 +08:00
Horváth Dóra 72d5a46899
HBASE-27685 Enable code coverage reporting to SonarQube in HBase (#5076) 2023-03-10 20:14:43 +01:00
Duo Zhang e48c4485db
HBASE-27632 Refactor WAL.Reader implementation so we can better support WAL splitting and replication (#5055)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2023-03-10 21:54:20 +08:00
Rajeshbabu Chintaguntla bc8b13e468
HBASE-27669 chaos-daemon.sh should make use hbase script start/stop chaosagent and chaos monkey runner (#5078)
Co-authored-by: Rajeshbabu Chintaguntla <rchintag@visa.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2023-03-09 10:46:25 +08:00
Nick Dimiduk da261344cc
HBASE-27568 ChaosMonkey add support for JournalNodes (#4963)
Signed-off-by: Reid Chan <reidchan@apache.org>
2023-01-17 09:55:18 +08:00
Nick Dimiduk 7ed2cb99f9
HBASE-27567 Introduce ChaosMonkey Action to print HDFS Cluster status
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2023-01-16 14:40:50 +01:00
Nick Dimiduk 2a7c69d30e HBASE-27563 ChaosMonkey sometimes generates invalid boundaries for random item selection
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2023-01-12 17:54:30 +01:00
Duo Zhang 2fc879e863
HBASE-27434 Use ${revision} as placeholder for maven version to make it easier to control the version from command line (#4836)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-10-24 11:50:48 +08:00
Duo Zhang ced1d642ae
HBASE-27409 Fix the javadoc for WARCRecord (#4814)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-10-06 18:18:06 +08:00
Duo Zhang 63cdd026f0
HBASE-27401 Clean up current broken 'n's in our javadoc (#4812)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-10-06 18:17:34 +08:00
Andrew Purtell 1004876bad HBASE-27252 Clean up error-prone findings in hbase-it
Close #4662

Co-authored-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2022-08-20 23:31:28 +08:00
Duo Zhang 3309108ca7
HBASE-27278 Improve TestTlsIPC to reuse existing IPC test code (#4682)
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org>
2022-08-12 12:17:29 +08:00
Duo Zhang 8b091c4061
HBASE-27222 Purge FutureReturnValueIgnored warnings from error prone (#4634)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-07-26 23:42:37 +08:00
Duo Zhang acf144717b HBASE-27220 Apply the spotless format change in HBASE-27208 to our code base
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-07-19 10:00:43 +08:00
Andrew Purtell ef1641d277
HBASE-27088 IntegrationLoadTestCommonCrawl async load improvements (#4488)
* HBASE-27088 IntegrationLoadTestCommonCrawl async load improvements

- Use an async client and work stealing executor for parallelism during loads.
- Remove the verification read retries, these are not that effective during
  replication lag anyway.
- Increase max task attempts because S3 might throttle.
- Implement a side task that exercises Increments by extracting urls from
  content and updating a cf that tracks referrer counts. These are not
  validated at this time. It could be possible to log the increments, sum
  them with a reducer, and then verify the total, but this is left as a
  future exercise.

Signed-off-by: Viraj Jasani <vjasani@apache.org>

* Sum RPC time for writes (loader) and reads (verifier) and mutation bytes submitted. Expose as job counters.

* Fix an issue with completion chaining

* Pause loading if too many operations are in flight
2022-07-13 09:01:21 -07:00
Narasimha Sharma 22618dadfe
HBASE-18045 Added default SSH connection timeout in Integration tests (#4450) 2022-07-07 10:17:04 -07:00
Duo Zhang 9bcb7493e0
HBASE-27165 Set version as 3.0.0-alpha-4-SNAPSHOT in master (#4582)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-06-29 11:04:21 +08:00
Duo Zhang 02990894ab
HBASE-27121 Set version as 3.0.0-alpha-3 in master in prep for first RC of 3.0.0-alpha-3 (#4535)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-06-15 10:51:31 +08:00
Duo Zhang e555ac4a99
HBASE-27023 Fix license issues after running spotless:apply (#4458)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-06-02 20:18:00 +08:00
huaxiangsun 597098ebba
HBASE-26984 Chaos Monkey thread dies in ITBLL Chaos GracefulRollingRestartRsAction (#4383)
There are two cases here:
  1. Chaos Monkey thread died and there is no chaos after that.
  2. Sometimes, regions are being moved back too quick that region server has not finished its initliazation yet.
     wait sometime to make sure that region server finishes its initialization.

Signed-off-by:  Wellington Chevreuil <wellington.chevreuil@gmail.com>
2022-05-05 09:37:41 -05:00
Duo Zhang 9c8c9e7fbf HBASE-26899 Run spotless:apply
Closes #4312
2022-05-01 22:15:09 +08:00
Duo Zhang 3ae0d9012c
HBASE-26922 Fix LineLength warnings as much as possible if it can not be fixed by spotless (#4324)
Signed-off-by: Yulin Niu <niuyulin@apache.org
2022-04-09 21:38:41 +08:00
Nick Dimiduk f78f232b28
HBASE-26834 Adapt ConnectionRule for both sync and async connections
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-21 12:41:22 +01:00
Duo Zhang 53a7ca794a Revert "HBASE-26813 Remove javax.ws.rs-api dependency (#4191)"
MiniYARNCluster needs it

This reverts commit b1c161ea8e.
2022-03-19 19:45:00 +08:00
Nick Dimiduk b1c161ea8e
HBASE-26813 Remove javax.ws.rs-api dependency (#4191)
This is no longer needed since we've transitioned to the shaded Jersey shipped in
hbase-thirdparty. Also drop supplemental models entry.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-16 16:46:15 +01:00
Andrew Purtell 10471944bd
HBASE-26582 Prune use of Random and SecureRandom objects (#4118)
Avoid the pattern where a Random object is allocated, used once or twice, and
then left for GC. This pattern triggers warnings from some static analysis tools
because this pattern leads to poor effective randomness. In a few cases we were
legitimately suffering from this issue; in others a change is still good to
reduce noise in analysis results.

Use ThreadLocalRandom where there is no requirement to set the seed to gain
good reuse.

Where useful relax use of SecureRandom to simply Random or ThreadLocalRandom,
which are unlikely to block if the system entropy pool is low, if we don't need
crypographically strong randomness for the use case. The exception to this is
normalization of use of Bytes#random to fill byte arrays with randomness.
Because Bytes#random may be used to generate key material it must be backed by
SecureRandom.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-08 13:49:02 -08:00
BukrosSzabolcs 4829806220
HBASE-26707: Reduce number of renames during bulkload (#4066)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-02-17 19:34:48 +00:00
Nick Dimiduk 625d610bcc
HBASE-26614 Refactor code related to "dump"ing ZK nodes (#3969)
The code starting at `ZKUtil.dump(ZKWatcher)` is a small mess – it has cyclic dependencies woven
through itself, `ZKWatcher` and `RecoverableZooKeeper`. It also initializes a static variable in
`ZKUtil` through the factory for `RecoverableZooKeeper` instances. Let's decouple and clean it
up.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2022-01-24 11:33:18 -08:00
Duo Zhang c14a76c4fd
HBASE-26523 Upgrade hbase-thirdparty dependency to 4.0.1 (#3987)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2021-12-31 12:08:01 +08:00
Duo Zhang 3f59f21be0
HBASE-26621 Set version as 3.0.0-alpha-3-SNAPSHOT in master (#3978)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-12-24 14:20:32 +08:00
Duo Zhang e598f2c663 Revert "HBASE-26523 Upgrade hbase-thirdparty dependency to 4.0.0 (#3910)"
Need a new 4.0.1 release

This reverts commit 139f08587a.
2021-12-17 12:25:27 +08:00
Duo Zhang 139f08587a
HBASE-26523 Upgrade hbase-thirdparty dependency to 4.0.0 (#3910)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2021-12-17 10:22:48 +08:00
Wellington Ramos Chevreuil a36d41af73
HBASE-26556 IT and Chaos Monkey improvements (#3932)
Signed-off-by: Josh Elser <elserj@apache.org>
Reviewed-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
2021-12-14 21:22:28 +00:00
Duo Zhang 8bca21b47d
HBASE-26558 Set version as 3.0.0-alpha-2 in master in prep for first RC of 3.0.0-alpha-2 (#3935)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
2021-12-11 20:52:35 +08:00
Andrew Purtell 9e73ea878d
HBASE-26349 Improve recent change to IntegrationTestLoadCommonCrawl (#3744)
Use a hybrid logical clock for timestamping entries.

Using BufferedMutator without HLC was not good because we assign client timestamps,
and the store loop is fast enough that on rare occasion two temporally adjacent URLs
in the set of WARCs are equivalent and the timestamp does not advance, leading later
to a rare false positive CORRUPT finding.

While making changes, support direct S3N paths as input paths on the command line.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-19 13:45:55 -07:00
Andrew Purtell a384c239b9
HBASE-26335 Minor improvements to IntegrationTestLoadCommonCrawl (#3731)
HBASE-26335 Minor improvements to IntegrationTestLoadCommonCrawl

- Use BufferedMutator instead of Table.
- Improve row key generator.
- Improve retries and log levels.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-08 10:00:51 -07:00
Duo Zhang 5f0950558f
HBASE-26096 Cleanup the deprecated methods in HBTU related classes and format code (#3503)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-07-29 10:18:38 +08:00
Duo Zhang 16721239e7
HBASE-26100 Set version as 3.0.0-alpha-2-SNAPSHOT in master (#3508)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-07-20 23:04:08 +08:00
Duo Zhang d30cc27097
HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate (#3478)
Signed-off-by: Michael Stack <stack@apache.org>
2021-07-19 09:29:08 +08:00
Duo Zhang 5118321ec9
HBASE-26059 Set version as 3.0.0-alpha-1 in master in prep for first RC of 3.0.0-alpha-1 (#3453)
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2021-07-02 07:50:41 +08:00
Andrew Purtell 335305e0cf
HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime (#3302)
We introduced EnvironmentEdgeManager as a way to inject alternate clocks
for unit tests. In order for this to be effective, all callers that would
otherwise use System.currentTimeMillis() must call
EnvironmentEdgeManager.currentTime() instead, except the implementers of
EnvironmentEdge.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-06-01 09:57:48 -07:00
Michael Stack 630c73fda4 HBASE-25867 Extra doc around ITBLL (#3242)
* HBASE-25867 Extra doc around ITBLL

Minor edits to a few log messages.
Explain how the '-c' option works when passed to ChaosMonkeyRunner.
Some added notes on ITBLL.
Fix whacky 'R' and 'Not r' thing in Master (shows when you run ITBLL).
In HRS, report hostname and port when it checks in (was debugging issue
where Master and HRS had different notions of its hostname).
Spare a dirty FNFException on startup if base dir not yet in place.

* Address Review by Sean

Signed-off-by: Sean Busbey <busbey@apache.org>
2021-05-11 19:26:57 +01:00
Andrew Purtell 6ad5b9e569
HBASE-25824 IntegrationTestLoadCommonCrawl (#3208)
* HBASE-25824 IntegrationTestLoadCommonCrawl

This integration test loads successful resource retrieval records from
the Common Crawl (https://commoncrawl.org/) public dataset into an HBase
table and writes records that can be used to later verify the presence
and integrity of those records.

Run like:

  ./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \
    -Dfs.s3n.awsAccessKeyId=<AWS access key> \
    -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
    /path/to/test-CC-MAIN-2021-10-warc.paths.gz \
    /path/to/tmp/warc-loader-output

Access to the Common Crawl dataset in S3 is made available to anyone by
Amazon AWS, but Hadoop's S3N filesystem still requires valid access
credentials to initialize.

The input path can either specify a directory or a file. The file may
optionally be compressed with gzip. If a directory, the loader expects
the directory to contain one or more WARC files from the Common Crawl
dataset. If a file, the loader expects a list of Hadoop S3N URIs which
point to S3 locations for one or more WARC files from the Common Crawl
dataset, one URI per line. Lines should be terminated with the UNIX line
terminator.

Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz
is a list of all WARC files comprising the Q1 2021 crawl archive. There
are 64,000 WARC files in this data set, each containing ~1GB of gzipped
data. The WARC files contain several record types, such as metadata,
request, and response, but we only load the response record types. If
the HBase table schema does not specify compression (by default) there
is roughly a 10x expansion. Loading the full crawl archive results in a
table approximately 640 TB in size.

The hadoop-aws jar will be needed at runtime to instantiate the S3N
filesystem. Use the -files ToolRunner argument to add it.

You can also split the Loader and Verify stages:

Load with:

  ./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \
    -files /path/to/hadoop-aws.jar \
    -Dfs.s3n.awsAccessKeyId=<AWS access key> \
    -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
    /path/to/test-CC-MAIN-2021-10-warc.paths.gz \
    /path/to/tmp/warc-loader-output

Verify with:

  ./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \
    /path/to/tmp/warc-loader-output

Signed-off-by: Michael Stack <stack@apache.org>
2021-05-03 17:59:00 -07:00
Duo Zhang f6ff519dd0 HBASE-25591 Upgrade opentelemetry to 0.17.1 (#2971)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang 302d9ea8b8 HBASE-25373 Remove HTrace completely in code base and try to make use of OpenTelemetry
Signed-off-by: stack <stack@apache.org>
2021-04-25 09:23:23 +08:00
niuyulin e8ac1fbe97
HBASE-25777 Fix wrong initialization value in StressAssignmentManagerMonkeyFactory (#3164)
Signed-off-by: meiyi <myimeiyi@gmail.com>
2021-04-19 17:46:57 +08:00
Duo Zhang ba3610d097
HBASE-19577 Use log4j2 instead of log4j for logging (#1708)
Signed-off-by: stack <stack@apache.org>
2021-03-20 09:21:25 +08:00
Pankaj 48d9d196dc
HBASE-25502 IntegrationTestMTTR fails with TableNotFoundException (#2879) 2021-01-13 11:01:26 -08:00
Mate Szalay-Beko 481662ab39
HBASE-25318 Config option for IntegrationTestImportTsv where to generate HFiles to bulkload (#2777)
IntegrationTestImportTsv is generating HFiles under the working directory of the
current hdfs user executing the tool, before bulkloading it into HBase.

Assuming you encrypt the HBase root directory within HDFS (using HDFS
Transparent Encryption), you can bulkload HFiles only if they sit in the same
encryption zone in HDFS as the HBase root directory itself.

When IntegrationTestImportTsv is executed against a real distributed cluster
and the working directory of the current user (e.g. /user/hbase) is not in the
same encryption zone as the HBase root directory (e.g. /hbase/data) then you
will get an exception:

```
ERROR org.apache.hadoop.hbase.regionserver.HRegion: There was a partial failure
due to IO when attempting to load d :
hdfs://mycluster/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/
IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc

org.apache.hadoop.ipc.RemoteException(java.io.IOException):
/user/hbase/test-data/22d8460d-04cc-e032-88ca-2cc20a7dd01c/
IntegrationTestImportTsv/hfiles/d/74655e3f8da142cb94bc31b64f0475cc
can't be moved into an encryption zone.
```

In this commit I make it configurable where the IntegrationTestImportTsv
generates the HFiles.

Co-authored-by: Mate Szalay-Beko <symat@apache.com>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-01-05 09:24:24 +01:00
Lokesh Khurana f8bd22827a
HBASE-24620 : Add a ClusterManager which submits command to ZooKeeper and its Agent which picks and execute those Commands (#2299)
Signed-off-by: Aman Poonia <apoonia@salesforce.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-21 15:33:36 +05:30