Commit Graph

18000 Commits

Author SHA1 Message Date
Xiaolin Ha 2fbb375cd5 HBASE-25322 Redundant Reference file in bottom region of split (#3814)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-11-08 14:48:55 +08:00
haxiaolin ebd89b58eb Revert "HBASE-25322 Redundant Reference file in bottom region of split (#3814)"
This reverts commit a6edf44175.
2021-11-08 11:40:43 +08:00
Xiaolin Ha a6edf44175 HBASE-25322 Redundant Reference file in bottom region of split (#3814)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-11-05 18:17:47 +08:00
Nick Dimiduk 2a2f618c9b HBASE-26363 OpenTelemetry configuration support for per-process service names
Exposes process-specific tracing arguments for most of the commands that we can launch from
`bin/hbase`. In addition to serving as an on/off flag, `HBASE_TRACE_OPTS` acts as the site-wide
configuration setting. Additional variables are provided for each applicable command, giving
operators fine-grain control over which processes participate in the tracing system, and to what
degree.

Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2021-11-04 11:39:36 -07:00
clarax e06208f3d6
HBASE-26311 Balancer gets stuck in cohosted replica distribution (#3804)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2021-11-03 11:58:38 -07:00
clarax 9eb3706021
HBASE-26309 Balancer tends to move regions to the server at the end of list (#3811)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-11-02 21:58:50 +08:00
XinSun 063e0e2e92 HBASE-26406 Can not add peer replicating to non-HBase (#3806)
Signed-off-by: Rushabh Shah <shahrs87@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>

(cherry picked from commit b9b7fec57f)
2021-11-02 15:03:33 +08:00
chenglei e89f62e611
HBASE-26384 Segment already flushed to hfile may still be remained in CompactingMemStore (#3779)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-11-01 23:05:57 +08:00
Rushabh Shah 19433ba677
HBASE-26404 Update javadoc for CellUtil#createCell with tags methods. (#3801)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-11-01 15:58:31 +08:00
Andrew Purtell 60f01153f7 Amend HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3787)
Resolve a new spotbugs warning in DictionaryCache

Signed-off-by: Andrew Purtell <apache.org>
2021-10-29 09:24:54 -07:00
Duo Zhang 9ce0066cde HBASE-26319 Make flaky find job track more builds (#3791)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2021-10-29 17:00:23 +08:00
Istvan Toth 7353c281fc HBASE-26398 CellCounter fails for large tables filling up local disk (#3798)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-10-28 09:48:28 +01:00
Andrew Purtell f2c58fcf68 HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3787)
ZStandard supports initialization of compressors and decompressors with a
precomputed dictionary, which can dramatically improve and speed up compression
of tables with small values. For more details, please see

  The Case For Small Data Compression
  https://github.com/facebook/zstd#the-case-for-small-data-compression

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Conflicts:
	hbase-compression/hbase-compression-zstd/src/main/java/org/apache/hadoop/hbase/io/compress/zstd/ZstdCodec.java
2021-10-27 07:57:16 -07:00
Yutong Xiao 82fe021b5f HBASE-26392 Update ClassSize.BYTE_BUFFER for JDK17 (#3784)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-27 22:04:39 +08:00
chenglei b4177577ac HBASE-26394 Cache in RSRpcServices.executeProcedures does not take effect (#3794)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-26 15:44:17 +08:00
clarax 32eb23c8f3
HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering balancer (#3788)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2021-10-25 15:44:11 -07:00
Duo Zhang 362117b52c HBASE-26390 Upload src tarball to nightlies for nightly jobs (#3793)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2021-10-25 23:34:15 +08:00
Anoop Sam John 444dec9289 HBASE-26190 High rate logging of BucketAllocatorException: Allocation too big - Addendum (#3792) 2021-10-24 09:22:26 +05:30
clarax f299d56934
HBASE-26308 Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger (#3783) 2021-10-22 09:46:58 -07:00
NishthaShah 73e5e03c04
HBASE-26369 Fix checkstyle issues for KeyValue and ByteBufferUtils (#3785) (#3765)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-22 19:51:20 +05:30
Duo Zhang 1253121db2 HBASE-26382 Use gen_redirect_html for linking flaky test logs (#3780)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2021-10-22 21:53:55 +08:00
Sourabh Dhanotia eea2ea8576
HBASE-26368 Fix checkstyle issues for HRegionServer for branch-2 (#3781) (#3764)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-22 12:51:29 +05:30
Andrew Purtell 644f820c6c Revert "HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3748)"
This reverts commit 8ac0b5ed7f.

This is not ready yet. There are some code paths remaining where store
configuration (CompoundConfiguration) is not passed into the block decoding
context. Found with additional integration tests.
2021-10-21 18:41:59 -07:00
Duo Zhang ecddf04126 HBASE-26362 Upload mvn site artifacts for nightly build to nightlies (#3778)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-21 23:11:30 +08:00
binlijin 0eaadd92d8 HBASE-26385 Clear CellScanner when replay (#3773)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2021-10-21 21:56:39 +08:00
Andrew Purtell dce45ec16a HBASE-26383 HBCK incorrectly reports inconsistencies for recently split regions following a master failover (#3775)
A regression was introduced by HBASE-25847 which changed regionInfo#isParentSplit
to regionState#isSplit. The region state after restart is CLOSED instead of SPLIT.
We need to check both regionState and regionInfo for split status.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-20 15:55:07 -07:00
Viraj Jasani 5c4a053f2a
HBASE-26371 Prioritize meta region move over other region moves in region_mover (#3771) (#3767)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-10-20 23:06:25 +05:30
Andrew Purtell 42ff3ac22e HBASE-26349 Improve recent change to IntegrationTestLoadCommonCrawl (#3744)
Use a hybrid logical clock for timestamping entries.

Using BufferedMutator without HLC was not good because we assign client timestamps,
and the store loop is fast enough that on rare occasion two temporally adjacent URLs
in the set of WARCs are equivalent and the timestamp does not advance, leading later
to a rare false positive CORRUPT finding.

While making changes, support direct S3N paths as input paths on the command line.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-19 14:20:08 -07:00
Andrew Purtell 8ac0b5ed7f HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3748)
ZStandard supports initialization of compressors and decompressors with a
precomputed dictionary, which can dramatically improve and speed up compression
of tables with small values. For more details, please see

  The Case For Small Data Compression
  https://github.com/facebook/zstd#the-case-for-small-data-compression

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-19 14:12:28 -07:00
Andrew Purtell b6bb18022f HBASE-26316 Per-table or per-CF compression codec setting overrides (#3730)
We get and retain Compressor instances in HFileBlockDefaultEncodingContext,
and could in theory call Compressor#reinit when setting up the context,
to update compression parameters like level and buffer size, but we do
not plumb through the CompoundConfiguration from the Store into the
encoding context. As a consequence we can only update codec parameters
globally in system site conf files.

Fine grained configurability is important for algorithms like ZStandard
(ZSTD), which offers more than 20 compression levels, where at level 1
it is almost as fast as LZ4, and where at higher levels it utilizes
computationally expensive techniques to rival LZMA at compression ratio
but trades off significantly for reduced compresson throughput. The ZSTD
level that should be set for a given column family or table will vary by
use case.

Signed-off-by: Viraj Jasani <vjasani@apache.org>

Conflicts:
	hbase-compression/hbase-compression-zstd/src/main/java/org/apache/hadoop/hbase/io/compress/zstd/ZstdDecompressor.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/compress/HFileTestBase.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestSeekToBlockWithEncoders.java
2021-10-19 13:59:04 -07:00
KulwantSingh011 7e574c9f1d
HBASE-26190 High rate logging of BucketAllocatorException: Allocation too big (#3752)
Reduce the frequency of allocation failed traces by printing them
preiodically (once per minute). Record the allocation failures in the
Bucket Cache Stats and let the stat thread dump cumulative allocation
failures alongside other traces it dumps.

Also, this change adds trace for the Table name, Column Family and
HFileName for the most recent allocation failure in last 1 minute.

Signed-off-by: Anoop <anoopsamjohn@apache.org>
2021-10-19 13:21:11 +05:30
Nick Dimiduk a69617d602 HBASE-26361 Enable OpenTelemetry to be used from developer sandbox
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-18 15:51:38 -07:00
Wellington Ramos Chevreuil be9877efdc HBASE-26364 TestThriftServer is failing 100% in our flaky test job (#3759)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 4454c2f909)
2021-10-18 14:46:31 +01:00
Horváth Dóra 6d0777a551
HBASE-26284 Add HBase Thrift API to get all table names along with wh… (#3693) (#3745)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-10-18 10:06:32 +01:00
Duo Zhang 1866992246 HBASE-26360 Use gen_redirect_html for linking test logs (#3755)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-17 23:24:31 +08:00
Josh Elser 45b09aa4d7 HBASE-26350 Add a DEBUG when we fail the SASL handshake
Closes #3743

Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-10-14 15:27:32 -04:00
Duo Zhang 9920d29b6b HBASE-26341 Upload dashboard html for flaky find job to nightlies (#3742)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-14 23:07:54 +08:00
chenglei 7ae7d01faa HBASE-26344 Fix Bug for MultiByteBuff.put(int, byte) (#3741)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-14 22:55:44 +08:00
Tak Lon (Stephen) Wu 3ce9052188
HBASE-24833: Bootstrap should not delete the META table directory if … (#2237)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
2021-10-12 10:56:00 -07:00
Duo Zhang 10584d70d2
HBASE-26306 Backport "HBASE-26220 Use P2P communicate between region servers to sync the list for bootstrap node" to branch-2 (#3727)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-10 22:56:12 +08:00
Sean Busbey 1d8570cde1 HBASE-26186 jenkins script for caching artifacts should verify cached file before relying on it. (#3590)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-09 13:25:08 +08:00
xicm ef514cc35e HBASE-26312 Shell scan fails with timestamp (#3734)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-09 10:33:59 +08:00
Duo Zhang a5340bd1b1 HBASE-26339 SshPublisher will skip uploading artifacts if the build is failure (#3736)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2021-10-09 09:56:43 +08:00
Andrew Purtell c3c7d36578 HBASE-26335 Minor improvements to IntegrationTestLoadCommonCrawl (#3731)
- Use BufferedMutator instead of Table.
- Improve row key generator.
- Improve retries and log levels.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-08 10:01:58 -07:00
Andrew Purtell 5af02201d2 HBASE-26324 Reuse compressors and decompressors in WAL CompressionContext (#3728)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-07 09:46:38 -07:00
Duo Zhang e4b56fee72 HBASE-26317 Publish the test logs for pre commit jenkins job to nightlies (#3715)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-10-07 23:45:49 +08:00
Duo Zhang 8358f34456 HBASE-26313 Publish the test logs for our nightly jobs to nightlies.apache.org (#3713)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-10-07 23:38:06 +08:00
Duo Zhang 00ffa6c252 HBASE-26318 Publish test logs for flaky jobs to nightlies (#3717)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-10-07 23:34:26 +08:00
Andrew Purtell 18b9fa8a3c HBASE-26259 Fallback support to pure Java compression (#3691)
This change introduces provided compression codecs to HBase as
new Maven modules. Each module provides compression codec support
that formerly required Hadoop native codecs, which in turn relies
on native code integration, which may or may not be available on
a given hardware platform or in an operational environment. We
now provide codecs in the HBase distribution for users whom for
whatever reason cannot or do not wish to deploy the Hadoop native
codecs.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestCompressedWAL.java
2021-10-06 13:48:33 -07:00
Peter Somogyi 025b42bb28 HBASE-26329 Upgrade commons-io to 2.11.0 (#3722)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-10-06 10:22:02 +02:00