Commit Graph

19005 Commits

Author SHA1 Message Date
NishthaShah 9e47358ecb
HBASE-26369 Fix checkstyle issues for KeyValue and ByteBufferUtils (#3765)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-22 19:50:06 +05:30
Duo Zhang ffc154ff83
HBASE-26382 Use gen_redirect_html for linking flaky test logs (#3780)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2021-10-22 21:43:58 +08:00
Sourabh Dhanotia 0d8e5e5219
HBASE-26368 Fix checkstyle issues for HRegionServer (#3764)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-22 12:50:45 +05:30
Andrew Purtell e0813e5402 Revert "HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3748)"
This reverts commit bfa4584125.

This is not ready yet. There are some code paths remaining where store
configuration (CompoundConfiguration) is not passed into the block decoding
context. Found with additional integration tests.
2021-10-21 18:42:05 -07:00
clarax 0d4982404c
HBASE-26308 Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger (#3710)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2021-10-21 11:19:21 -07:00
Duo Zhang 69656e0df8
HBASE-26362 Upload mvn site artifacts for nightly build to nightlies (#3778)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-21 23:07:30 +08:00
binlijin 4838246ba3
HBASE-26385 Clear CellScanner when replay (#3773)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2021-10-21 21:30:19 +08:00
Nick Dimiduk 169f2cff30
HBASE-26380 Add 2.3.7 to downloads page (#3770)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-21 21:11:20 +08:00
Andrew Purtell 64eb237120
HBASE-26383 HBCK incorrectly reports inconsistencies for recently split regions following a master failover (#3775)
A regression was introduced by HBASE-25847 which changed regionInfo#isParentSplit
to regionState#isSplit. The region state after restart is CLOSED instead of SPLIT.
We need to check both regionState and regionInfo for split status.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-20 15:48:36 -07:00
Viraj Jasani 856d08e863
HBASE-26371 Prioritize meta region move over other region moves in region_mover (#3767)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-10-20 23:05:12 +05:30
Andrew Purtell 9e73ea878d
HBASE-26349 Improve recent change to IntegrationTestLoadCommonCrawl (#3744)
Use a hybrid logical clock for timestamping entries.

Using BufferedMutator without HLC was not good because we assign client timestamps,
and the store loop is fast enough that on rare occasion two temporally adjacent URLs
in the set of WARCs are equivalent and the timestamp does not advance, leading later
to a rare false positive CORRUPT finding.

While making changes, support direct S3N paths as input paths on the command line.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-19 13:45:55 -07:00
Andrew Purtell bfa4584125
HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3748)
ZStandard supports initialization of compressors and decompressors with a
precomputed dictionary, which can dramatically improve and speed up compression
of tables with small values. For more details, please see

  The Case For Small Data Compression
  https://github.com/facebook/zstd#the-case-for-small-data-compression

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-19 13:37:24 -07:00
Andrew Purtell 8a6fed7b19
HBASE-26316 Per-table or per-CF compression codec setting overrides (#3730)
We get and retain Compressor instances in HFileBlockDefaultEncodingContext,
and could in theory call Compressor#reinit when setting up the context,
to update compression parameters like level and buffer size, but we do
not plumb through the CompoundConfiguration from the Store into the
encoding context. As a consequence we can only update codec parameters
globally in system site conf files.

Fine grained configurability is important for algorithms like ZStandard
(ZSTD), which offers more than 20 compression levels, where at level 1
it is almost as fast as LZ4, and where at higher levels it utilizes
computationally expensive techniques to rival LZMA at compression ratio
but trades off significantly for reduced compresson throughput. The ZSTD
level that should be set for a given column family or table will vary by
use case.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-19 12:04:25 -07:00
Andrew Purtell 26ab9d0e0b HBASE-26367 Update downloads.xml for release 2.4.7
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-10-18 17:42:57 -07:00
z-york 23e7aa5ccc
HBASE-26320 Implement a separate thread pool for the LogCleaner (#3712)
This avoids starvation when the archive directory is large and takes a long time
to iterate through.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
2021-10-18 16:21:01 -07:00
Nick Dimiduk cadac1889a HBASE-26361 Enable OpenTelemetry to be used from developer sandbox
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-18 15:20:56 -07:00
Wellington Ramos Chevreuil 4454c2f909
HBASE-26364 TestThriftServer is failing 100% in our flaky test job (#3759)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-18 13:59:42 +01:00
Viraj Jasani 1020fe1c38
HBASE-26345 Design doc for HBase NamedQueue framework (ADDENDUM) 2021-10-18 15:55:38 +05:30
Duo Zhang 736f3e77c8
HBASE-26360 Use gen_redirect_html for linking test logs (#3755)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-17 23:16:33 +08:00
Huang Zhuoyue 7bb2b9e8e4 Add new committer huangzhuoyue to pom.xml 2021-10-15 16:38:34 +08:00
Josh Elser ad7d698622 HBASE-26350 Add a DEBUG when we fail the SASL handshake
Closes #3743

Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-10-14 15:19:30 -04:00
Duo Zhang 10f16a0e4f
HBASE-26341 Upload dashboard html for flaky find job to nightlies (#3742)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-14 23:04:54 +08:00
chenglei b898661ad0
HBASE-26344 Fix Bug for MultiByteBuff.put(int, byte) (#3741)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-14 22:53:09 +08:00
Duo Zhang 6e7e69a0a7
HBASE-26348 Implement a special procedure to migrate rs group information when upgrading (#3746)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Reviewed-by: GeorryHuang <215175212@qq.com>
2021-10-14 22:51:37 +08:00
KulwantSingh011 891e90f08f
HBASE-26190 High rate logging of Bucket Allocator Allocation fails (#3648)
* HBASE-26190 High rate logging of BucketAllocatorException: Allocation too big
Reduce the frequency of allocation failed traces by printing them
preiodically (once per minute). Record the allocation failures in the
Bucket Cache Stats and let the stat thread dump cumulative allocation
failures alongside other traces it dumps.
Also, this change adds trace for the Table name, Column Family and
HFileName for the most recent allocation failure in last 1 minute.

Signed-off-by: Anoop <anoopsamjohn@apache.org>
2021-10-14 13:13:17 +05:30
Nick Dimiduk 4d27c4726d HBASE-26359 Loosen Dockerfile pinned package versions for `create-release/mac-sshd-gpg-agent/Dockerfile`
Signed-off-by: Sean Busbey <busbey@apache.org>
2021-10-13 12:15:35 -07:00
Viraj Jasani ede4d2715d
HBASE-26345 Design doc for HBase NamedQueue framework 2021-10-12 20:43:17 +05:30
Horváth Dóra ce44e16939
HBASE-26284 Add HBase Thrift API to get all table names along with wh… (#3693)
Signed-off-by: Norbert Kalmar <nkalmar@cloudera.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-10-11 13:04:03 +01:00
Sean Busbey 82ccd33186
HBASE-26186 jenkins script for caching artifacts should verify cached file before relying on it. (#3590)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-09 13:19:43 +08:00
xicm 7832518d8a
HBASE-26312 Shell scan fails with timestamp (#3734)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-09 10:28:25 +08:00
Duo Zhang c410baebe3
HBASE-26339 SshPublisher will skip uploading artifacts if the build is failure (#3736)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2021-10-09 09:43:43 +08:00
Andrew Purtell a384c239b9
HBASE-26335 Minor improvements to IntegrationTestLoadCommonCrawl (#3731)
HBASE-26335 Minor improvements to IntegrationTestLoadCommonCrawl

- Use BufferedMutator instead of Table.
- Improve row key generator.
- Improve retries and log levels.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-08 10:00:51 -07:00
Andrew Purtell 39a20c528e
HBASE-26324 Reuse compressors and decompressors in WAL CompressionContext (#3728)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-07 09:45:17 -07:00
Duo Zhang 7ae71e880d
HBASE-26318 Publish test logs for flaky jobs to nightlies (#3717)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-10-07 21:58:48 +08:00
Duo Zhang aad893530b
HBASE-26317 Publish the test logs for pre commit jenkins job to nightlies (#3715)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-10-07 21:58:28 +08:00
Duo Zhang 11b94d3384
HBASE-26313 Publish the test logs for our nightly jobs to nightlies.apache.org (#3713)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-10-07 21:57:21 +08:00
Andrew Purtell 42fe5e5baa
HBASE-26259 Fallback support to pure Java compression (#3691)
This change introduces provided compression codecs to HBase as
new Maven modules. Each module provides compression codec support
that formerly required Hadoop native codecs, which in turn relies
on native code integration, which may or may not be available on
a given hardware platform or in an operational environment. We
now provide codecs in the HBase distribution for users whom for
whatever reason cannot or do not wish to deploy the Hadoop native
codecs.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-06 13:17:18 -07:00
Peter Somogyi b94188ff94
HBASE-26329 Upgrade commons-io to 2.11.0 (#3722)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-06 10:19:17 +02:00
imbajin f65b769e8c
HBASE-24601: Change default Hfile storage policy from HOT to NONE for HDFS (#1940)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-02 16:35:14 +08:00
Yutong Xiao d3c86f4f90
HBASE-26305 Move NavigableSet add operation to writer thread in BucketCache (#3705)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: chenglei <chenglei@apache.org>
2021-10-01 20:57:51 +08:00
chenglei 121bdea230
HBASE-26295 BucketCache could not free BucketEntry which restored fro… (#3699)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-29 21:24:12 +08:00
Duo Zhang 1152a61b5b
HBASE-26220 Use P2P communicate between region servers to sync the list for bootstrap node (#3697)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-09-29 21:12:58 +08:00
richardantal f000b77532
HBASE-26289 Hbase scan setMaxResultsPerColumnFamily not giving right results (#3688)
Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-09-29 10:03:53 +02:00
Duo Zhang 113cc2a591
HBASE-26293 Use reservoir sampling when selecting bootstrap nodes (#3702)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2021-09-28 23:27:47 +08:00
bitterfox 748a6827c4
HBASE-26238 Short message by Result#compareResults for VerifyReplication to avoid OOME (#3647)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-28 21:48:32 +08:00
GeorryHuang 63306942b1
HBASE-26251 StochasticLoadBalancer metrics should update even if balancer doesn't run (#3678)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
2021-09-28 21:20:48 +08:00
clarax cbebf85b3c
HBASE-26297 Balancer run is improperly triggered by accuracy error of double comparison (#3698)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-26 15:10:44 +08:00
clarax 2b26dfbaf4
HBASE-26178 Improve data structure and algorithm for BalanceClusterState to improve computation speed for large cluster (#3682)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-25 20:02:28 +08:00
Tak Lon (Stephen) Wu d956828aac HBASE-26274 Create an option to reintroduce BlockCache to mapreduce job (#3684)
Introduce `hfile.onheap.block.cache.fixed.size`
and default to disable. when using ClientSideRegionScanner
it will be enabled with a fixed size for caching
INDEX/LEAF_INDEX block when a client, e.g.
snapshot scanner, scans the entire HFile
and does not need to seek/reseek to index
block multiple times.

Signed-off-by: Josh Elser <elserj@apache.org>
2021-09-22 09:26:57 -07:00
Tak Lon (Stephen) Wu ecd35f77ae
Revert "HBASE-26274 Create an option to reintroduce BlockCache to mapreduce job (#3684)" (#3695)
This reverts commit 6556a5ee91.
2021-09-22 09:19:05 -07:00