* HBASE-23723 Ensure MOB compaction works in optimized mode after snapshot clone (#1446)
* Reorganize MOB compaction tests for more reuse.
* Add tests for mob compaction after snapshot clone operations
* note the original table used to write a given mob hfile and use that to find it later.
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
* spotless:apply to fix HBaseTestingUtility
* Fix error-prone errors
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
Co-authored-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
Co-authored-by: Andrew Purtell <apurtell@apache.org>
Co-authored-by: Sean Busbey <busbey@apache.org>
Fix test case failures in org.apache.hadoop.hbase.http.log.TestLogLevel under Openjdk 17 because of a missing export of java.security.jgss/sun.security.krb5.
Removed option --illegal-access=permit ignored since Openjdk 17.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit e10c15d030)
Revert "HBASE-24579: Failed SASL authentication does not result in an exception on client side (#1921)"
This reverts commit bd79c4065c.
When Kerberos authentication succeeds, on the server side, after
receiving the final SASL token from the client, we simply wait for
the client to continue by sending the connection header.
After HBASE-24579, on the client side, an additional readStatus()
was added, which mistakenly assumes that after negotiation has
completed a status code will be sent. However when authentication
has succeeded the server will not send one. As a result the client
will hang and only throw an exception when the configured read
timeout is reached, which is 20 seconds by default.
We cannot unilaterally send the expected additional status code
from the server side because older clients will not expect it. The
first call will fail because the client finds unexpected bytes in
the stream ahead of the call response. Fabricating a call response
also does not seem a viable strategy for backwards compatibility.
The HBASE-24579 change needs to be reconsidered given the
difficult backwards compatibility challenges here.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
The configuration settings "hbase.netty.worker.count" and
"hbase.netty.eventloop.rpcserver.thread.count" appear to duplicate each
other.
Also, formalizes another setting found in NettyEventLoopGroupConfig,
"hbase.netty.nativetransport".
Also, native epoll is not limited to amd64. aarch64 supports it too.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Conflicts:
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
On branch-2.5, the affected code is in hbase-server.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Conflicts:
hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/RegionLocationFinder.java
Add test coverage for SimpleRpcServer.
Improve the way we test both SimpleRpcServer and NettyRpcServer. Use
LoadTestKVGenerator to generate random values with varying sizes between
1000 bytes and 1M bytes, and also to verify them when reading the values
back.
Add secure test coverage for both SimpleRpcServer and NettyRpcServer.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* HBASE-27088 IntegrationLoadTestCommonCrawl async load improvements
- Use an async client and work stealing executor for parallelism during loads.
- Remove the verification read retries, these are not that effective during
replication lag anyway.
- Increase max task attempts because S3 might throttle.
- Implement a side task that exercises Increments by extracting urls from
content and updating a cf that tracks referrer counts. These are not
validated at this time. It could be possible to log the increments, sum
them with a reducer, and then verify the total, but this is left as a
future exercise.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* Sum RPC time for writes (loader) and reads (verifier) and mutation bytes submitted. Expose as job counters.
* Fix an issue with completion chaining
* Pause loading if too many operations are in flight
* HBASE-22749 Distributed MOB compactions
- MOB compaction is now handled in-line with per-region compaction on region
servers
- regions with mob data store per-hfile metadata about which mob hfiles are
referenced
- admin requested major compaction will also rewrite MOB files; periodic RS
initiated major compaction will not
- periodically a chore in the master will initiate a major compaction that
will rewrite MOB values to ensure it happens. controlled by
'hbase.mob.compaction.chore.period'. default is weekly
- control how many RS the chore requests major compaction on in parallel
with 'hbase.mob.major.compaction.region.batch.size'. default is as
parallel as possible.
- periodic chore in master will scan backing hfiles from regions to get the
set of referenced mob hfiles and archive those that are no longer
referenced. control period with 'hbase.master.mob.cleaner.period'
- Optionally, RS that are compacting mob files can limit write
amplification by not rewriting values from mob hfiles over a certain size
limit. opt-in by setting 'hbase.mob.compaction.type' to 'optimized'.
control threshold by 'hbase.mob.compactions.max.file.size'.
default is 1GiB
- Should smoothly integrate with existing MOB users via rolling upgrade.
will delay old MOB file cleanup until per-region compaction has managed
to compact each region at least once so that used mob hfile metadata can
be gathered.
* HBASE-22749 Distributed MOB compactions
fix RestrictedApi
Co-authored-by: Vladimir Rodionov <vrodionov@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Duo Zhang
huaxiangsun
liangxs
SiCheng-Zheng
dependabot[bot]
Peter Somogyi
Andrew Purtell
Masatake Iwasaki
Bryan Beaudreault
chaijunjie0101
LiangJun He
ryakhovskiy
Nick Dimiduk
chenglei
Ramón García Fernández
Xiaolin Ha
Minwoo Kang
Nihal Jain
BukrosSzabolcs