Commit Graph

17843 Commits

Author SHA1 Message Date
Bryan Beaudreault fa033e4ac6 HubSpot Backport: HBASE-26783 ScannerCallable doubly clears meta cache on retries (#4147)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-26 12:59:50 -04:00
Xiaolin Ha 35ec27d079 HubSpot Backport: HBASE-25709 Close region may stuck when region is compacting and skipped most cells read (#3117)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-26 12:59:50 -04:00
Andrew Purtell 954004af49 HubSpot Backport: HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write (#4184)
Co-authored-by: Bryan Beaudreault <bbeaudreault@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-26 12:59:50 -04:00
Bryan Beaudreault 44c62c58ba fix 2022-03-26 12:59:50 -04:00
Xiaolin Ha e81f6ced7b HubSpot Backport: HBASE-26552 Introduce retry to logroller to avoid abort (#4170)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-26 12:33:31 -04:00
Bryan Beaudreault 35f86757c9 HubSpot Backport: HBASE-26876 Use toStringBinary for rowkey in RegionServerCallable error string (#4269)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-26 12:26:01 -04:00
Bryan Beaudreault 9e54eeedd0 HubSpot Backport: HBASE-26875 RpcRetryingCallerImpl translateException ignores return value of recursive call (#4270)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-25 12:25:16 -04:00
Bryan Beaudreault 4f67229b2b HubSpot Backport: HBASE-26878 TableInputFormatBase should cache RegionSizeCalculator (#4271)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-25 12:25:12 -04:00
Michael Stack 30a299eb55 HubSpot Backport: HBASE-21065 Try ROW_INDEX_V1 encoding on meta table
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.
2022-03-24 07:18:52 -04:00
Baiqiang Zhao a0a9da8bcf HubSpot Backport: HBASE-25687 Backport "HBASE-25681 Add a switch for server/table query… (#3074)
Signed-off-by: stack <stack@apache.org>
2022-03-17 14:41:44 -04:00
Wei-Chiu Chuang c01cc7cb6a HubSpot Backport: HBASE-21946 Use ByteBuffer pread instead of byte[] pread in HFileBlock when applicable (#3434)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 4a3c7d73b0)
2022-03-17 14:16:41 -04:00
Bryan Beaudreault 0c21465878 Revert "Upstream Callers (#15)"
See HBasePlanning/issues/806 for details

This reverts commit 8cc6bfb5b2.
2022-03-15 12:11:10 -04:00
Bryan Beaudreault 374ed22789 HubSpot Backport: HBASE-26814 Default StoreHotnessProtector to off, with logs to guide when to turn it on 2022-03-09 17:48:17 -05:00
Bryan Beaudreault 8a27821582 HubSpot Backport: HBASE-26784 Use HIGH_QOS for ResultScanner.close requests (#4163)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-08 08:52:13 -05:00
Charles Connell 866ae91111
Merge pull request #6 from HubSpot/blazar-provides
Give hbase-assembly a name in Blazar
2022-03-03 11:47:59 -05:00
Charles Connell 5cb5b3ed3e Give hbase-assembly a name in Blazar 2022-03-03 11:07:39 -05:00
Bri Augenreich 4a38b00588
HubSpot Backport: HBASE-26308 Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger 2022-03-03 10:21:07 -05:00
Bryan Beaudreault 814065d0d9 HubSpot Backport: HBASE-26731 Add metrics for active and expired scanners 2022-03-01 16:38:09 -05:00
Bri Augenreich c7e6f80fc5
Merge pull request #4 from HubSpot/HBASE-26745-hubspot-2
HBASE-26745 Include Balancer Cost metrics in jmx endpoint
2022-02-28 20:10:26 -05:00
Briana Augenreich a9f7907f9d HubSpot Backport: HBASE-26745 Include Balancer Cost metrics in jmx endpoint 2022-02-28 13:48:22 -05:00
Bryan Beaudreault 60e936a463 HubSpot Backport HBASE-26727 Fix CallDroppedException reporting 2022-02-13 19:22:32 -05:00
Bryan Beaudreault 66b2e38300 HubSpot Addendum HBASE-26703: Add executor name to QueueBalancer constructor args 2022-02-04 09:55:43 -05:00
Bryan Beaudreault cf6efffc77 HubSpot Backport: HBASE-26727 Fix CallDroppedException reporting 2022-02-03 10:51:17 -05:00
chenglei 27855ed288 HBASE-26679 Wait on the future returned by FanOutOneBlockAsyncDFSOutput.flush would stuck (#4077)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-01-28 07:57:32 -05:00
Richard Marscher 2450ffcc90 HubSpot Backport: HBASE-26623 Report CallDroppedException in exception metrics (#3980)
`CallDroppedException` can be thrown with `CallRunner.drop()` by queue implementations that decide to drop calls to groom the RPC call backlog. The LifoCoDel queue does this I believe and with Pluggable queue it's possible for 3rd party queue implementations to be using `drop()` for similar reasons. It would be nice for the server to be tracking these exceptions in metrics since otherwise you might have to do some extra lifting on the client side.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
2022-01-27 13:15:14 -05:00
Bryan Beaudreault 68bc63e9cb HubSpot Backport: HBASE-26713 Default to LATEST_TIMESTAMP if no timestamp sent along on Increment/Append 2022-01-27 10:33:53 -05:00
Bryan Beaudreault 0be4643029 HubSpot Backport: HBASE-26703 Allow configuration of IPC queue balancer 2022-01-25 15:36:43 -05:00
Bryan Beaudreault 4ce7c040e9 HubSpot Addendum HBASE-25685: Default to HTML, like in hbase-2.5+ 2022-01-04 08:49:02 -05:00
Bryan Beaudreault 8debdaace4
Add back custom version injection, but set it to the raw HBASE_VERSION (#3)
This reverts commit f556810b52 (#2).
2021-12-20 09:52:57 -05:00
Bryan Beaudreault f556810b52
Remove custom version injection because it does not work well due to multi-module build (#2) 2021-12-17 08:55:35 -05:00
Bryan Beaudreault 1cee26a2a3
Merge pull request #1 from HubSpot/HBASE-26576 2021-12-15 12:59:19 -05:00
Richard Marscher b0007c9975 HubSpot Backport HBASE-26576 Allow pluggable queue to be used with the fast path executor or normal balanced executor 2021-12-15 11:56:04 -05:00
Bryan Beaudreault 8e6589ad9a HubSpot Backport HBASE-26546: hbase-shaded-client missing required thirdparty classes under hadoop 3.3.1 2021-12-07 23:50:49 -05:00
Bryan Beaudreault 102c2bb129 HubSpot Backport HBASE-26304: Reflect out of band locality improvements in metrics and balancer 2021-11-15 14:28:30 -05:00
GeorryHuang b988fd3bb2 HubSpot Backport: HBASE-26251 StochasticLoadBalancer metrics should update even if balancer doesn't run (#3706)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
2021-10-22 14:56:34 -04:00
Duo Zhang 36adc2e6ac HubSpot Backport: HBASE-25032 Do not assign regions to region server which has not called regionServerReport yet (#3268)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2021-10-22 14:56:34 -04:00
richardantal a396547140 HubSpot Backport: HBASE-26289 Hbase scan setMaxResultsPerColumnFamily not giving right results (#3688)
Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-09-29 07:54:44 -04:00
clarax 49453b388e HubSpot Backport: HBASE-26297 Balancer run is improperly triggered by accuracy error of double comparison (#3698)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-29 07:54:28 -04:00
Bryan Beaudreault 1507f893a7 HubSpot Backport: HBASE-26147 Add a dry run mode to the balancer, where moves are calculated but not actually executed
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>

(Re-application of HBASE-26147 with the correct author metadata)
2021-09-24 15:51:16 -04:00
Bryan Beaudreault 8cc6bfb5b2 Upstream Callers (#15)
Adds an upstream caller to the RequestHeader of hbase requests, and makes that available in RpcCallContext
One can inject an upstream caller by setting the conf `hbase.upstream.caller.impl` to a fully qualified class name of a class with a public constructor that implements `org.apache.hadoop.hbase.ipc.UpstreamCaller`
Upstream callers will be available via `RpcCallContext#getUpstreamCaller` or `RpcCallContext#getRequestUserName`. The latter combines the real username and upstream caller to create a fully qualified name like `<upstreamCaller>.via.<user>`
2021-09-24 15:51:16 -04:00
Bryan Beaudreault c7e4262990 Fixes to get HubSpot build to work
- Our branch version munging script set-maven-versions can't parse the pom.xml due to unicode, remove it.
- Our build adds build.json files to the jars. Allowlist those files in jar content verification.

We could potentially try to upstream generic versions changes of these at some point
2021-09-24 15:51:16 -04:00
Bryan Beaudreault fd5b526398 Add maven and RPM building in blazar
This uses our standard buildpack, but each module must get a custom .blazar.yaml to pass in required
properties to the build. One can distribute that .blazar.yaml using build-scripts/copy_blazar_yaml.sh.
In order to minimize the duplication, most of the customizations are consolidated in
build-scripts/prepare_environment.sh.

The RPM build is hooked into the standard build hbase-assembly, as a post-build step. The RPM build
scripts in hbase-assembly/rpm-build handle turning the assembled tar.gz into multiple RPMs. We required
some minor changes to the descriptor in hbase-assembly to account for the fact that our build processes
each module independently instead of as a single reactor build. As a result, moduleSet does not work, so
moduleSet usages were converted to dependencySet.
2021-09-24 15:51:16 -04:00
Andrew Purtell 7374d396c2 Preparing hbase release 2.4.6RC0; tagging and updates to CHANGES.md and RELEASENOTES.md
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-09-03 09:41:52 -07:00
Rushabh Shah f9a9b6eaee HBASE-26106 AbstractFSWALProvider#getArchivedLogPath doesn't look for wal file in all oldWALs directory. (#3636)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>

Conflicts:
	hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALRecordReader.java
2021-09-02 12:08:49 -07:00
Andrew Purtell d1aa68bd11 HBASE-25588 Excessive logging of "hbase.zookeeper.useMulti is deprecated. Default to true always." (#3640)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Duo Zhang <zhangduo@apache.org>
2021-09-02 10:18:47 -07:00
XinSun 067f609ec6 HBASE-25773 TestSnapshotScannerHDFSAclController.setupBeforeClass is flaky (#3651)
Signed-off-by: Duo Zhang <zhangduo@apache.org>

(cherry picked from commit 345d7256c8)
2021-09-02 15:33:24 +08:00
bitterfox 875f3afc2a HBASE-26205 Use specified cluster conf for UserProvider in TableMRUtil#initCredentialsForCluster (#3592)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Rushabh Shah <shahrs87@gmail.com>
2021-09-01 20:48:26 +08:00
chenglei cf4c764d16 HBASE-26210 HBase Write should be doomed to hang when cell size exceeds InmemoryFlushSize for CompactingMemStore (#3604)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-01 18:17:26 +08:00
chenglei 8085e4c5de HBASE-26244 Avoid trim the error stack trace when running UT with maven (#3654)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-01 15:21:30 +08:00
Andrew Purtell 122efbcd82 HBASE-25642 Fix or stop warning about already cached block (#3638)
Our logs have as a fairly common occurrence: 2021-03-05 22:24:31,034 WARN
[StoreFileOpener-foo-1] hfile.BlockCacheUtil: Caching an already cached
block: blah.bub. This is harmless and can happen in rare cases (see HBASE-8547)

Because it is harmless, log at DEBUG level, not WARN.

Signed-off-by: Baiqiang Zhao <zhaobaiqiang@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Rushabh Shah <rushabh.shah@salesforce.com>
2021-08-31 09:26:14 -07:00