Commit Graph

345 Commits

Author SHA1 Message Date
Ujjawal fa3bd7feb9
HBASE-27353 - opentelemetry-context jar missing at runtime causes MR jobs to fail (#4759)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2022-09-07 10:49:53 -07:00
Michael Stack 1b5403cf7d
HBASE-27340 Artifacts with resolved profiles (#4740) (#4761)
(Forward port from branch-2; simplified by the fact that there
is no hadoop-2.0 profile on master branch)

    Make it so our published poms carry the minimum needed to run
    an hbase; the published pom has no profiles -- the profiles
    specified at build time are resolved, their dependencies inlined,
    and then they are stripped -- and no build-time, or plugins
    dependencies or properties, etc. Resultant poms have explicit
    hadoop lib versions baked in -- no more being able to choose
    hbase with hadoop2 or haddop3 at downstream build time by setting
    a '-Dhadoop.profile=X.0'.

    Pattern is to add profiles when none in sub-modules when
    the flatten plugin complains it can't resolve an hadoop
    dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
    Adding the profile in the sub-module make it so the flatten
    plugin can figure 'hadoop.version' definitively.
    (In master there is only the hadoop-3.0 profile).

    Another spin on the above happens when profiles already exist
    in submodule but the flatten plugin is complaining it can't
    figure figure version on an hadoop dependency NOT under
    profiles. Below, we move the delinquent hadoop dependency under
    existing profiles (minikdc was the usual dependency outside
    profiles in sub-modules that flatten complained about).

    Sometimes, moving an hadoop dependency under a profile, there
    would be excludes on the local dependency. If the parent pom
    excludes section was missing the local excludes, we added them
    up to the parent module so all excluding is done up there in
    the parent profile dependencyManagement section.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-09-06 23:06:13 +08:00
Duo Zhang acf144717b HBASE-27220 Apply the spotless format change in HBASE-27208 to our code base
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-07-19 10:00:43 +08:00
Duo Zhang 6f4a0476f2 HBASE-27148 Addendum uncomment the shutdown code in TestVerifyReplicationCrossDiffHdfs 2022-07-06 14:48:08 +08:00
Duo Zhang 41972cb460
HBASE-27148 Move minimum hadoop 3 support version to 3.2.3 (#4561)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-07-06 12:49:47 +08:00
Duo Zhang 9bcb7493e0
HBASE-27165 Set version as 3.0.0-alpha-4-SNAPSHOT in master (#4582)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-06-29 11:04:21 +08:00
Tak Lon (Stephen) Wu 33ea96cdf7
HBASE-27143 Add hbase-unsafe as a dependency for a MR job triggered by hbase shell (#4554)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-06-21 13:38:09 -07:00
XinSun b365748485
HBASE-26956 ExportSnapshot tool supports removing TTL (#4351)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-06-15 15:04:17 +08:00
Duo Zhang 02990894ab
HBASE-27121 Set version as 3.0.0-alpha-3 in master in prep for first RC of 3.0.0-alpha-3 (#4535)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-06-15 10:51:31 +08:00
wenwj0 b7065c1cf5
HBASE-26923 PerformanceEvaluation support encryption option (#4489)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-06-11 20:10:25 +08:00
litao c269dc5711
HBASE-27003 Optimize log format for PerformanceEvaluation (#4411)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2022-05-10 22:31:44 -07:00
Duo Zhang 9c8c9e7fbf HBASE-26899 Run spotless:apply
Closes #4312
2022-05-01 22:15:09 +08:00
Nick Dimiduk 8d8f956099
HBASE-26954 Compilation of master vs hadoop-3.3.2 fails (#4363)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2022-04-21 11:54:01 +02:00
Duo Zhang 3ae0d9012c
HBASE-26922 Fix LineLength warnings as much as possible if it can not be fixed by spotless (#4324)
Signed-off-by: Yulin Niu <niuyulin@apache.org
2022-04-09 21:38:41 +08:00
Duo Zhang c4ff355915
HBASE-26916 Fix missing braces warnings in DefaultVisibilityExpressionResolver (#4313)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-04-03 22:42:17 +08:00
Bryan Beaudreault 2a3ac5b33c
HBASE-26878 TableInputFormatBase should cache RegionSizeCalculator (#4271)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-24 14:54:49 -07:00
xicm ef280d4d8d
HBASE-26720 ExportSnapshot should validate the source snapshot before copying files (#4216)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-16 20:03:28 +08:00
Andrew Purtell 10471944bd
HBASE-26582 Prune use of Random and SecureRandom objects (#4118)
Avoid the pattern where a Random object is allocated, used once or twice, and
then left for GC. This pattern triggers warnings from some static analysis tools
because this pattern leads to poor effective randomness. In a few cases we were
legitimately suffering from this issue; in others a change is still good to
reduce noise in analysis results.

Use ThreadLocalRandom where there is no requirement to set the seed to gain
good reuse.

Where useful relax use of SecureRandom to simply Random or ThreadLocalRandom,
which are unlikely to block if the system entropy pool is low, if we don't need
crypographically strong randomness for the use case. The exception to this is
normalization of use of Bytes#random to fill byte arrays with randomness.
Because Bytes#random may be used to generate key material it must be backed by
SecureRandom.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-08 13:49:02 -08:00
Mike Drob 06f06cbd86 HBASE-26622 Update error-prone to 2.10
Author:    Mike Drob <mdrob@apache.org>
Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-02 16:07:23 +01:00
Duo Zhang e1c2c16214
HBASE-26690 Modify FSTableDescriptors to not rely on renaming when writing TableDescriptor (#4054)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-01-27 13:28:56 +08:00
meiyi f1e263a8fd
HBASE-26646 WALPlayer should obtain token from filesystem (#4003)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-01-07 17:37:29 +08:00
meiyi 8f0c2dabbb
HBASE-26625 ExportSnapshot tool failed to copy data files for tables with merge region (#3981)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-12-30 10:40:33 +08:00
Yutong Xiao a599d52820
HBASE-26564 Retire the method visitLogEntryBeforeWrite without RegionInfo in WALActionListner (#3938)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-12-27 23:59:13 +08:00
Duo Zhang 3f59f21be0
HBASE-26621 Set version as 3.0.0-alpha-3-SNAPSHOT in master (#3978)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-12-24 14:20:32 +08:00
Duo Zhang 8bca21b47d
HBASE-26558 Set version as 3.0.0-alpha-2 in master in prep for first RC of 3.0.0-alpha-2 (#3935)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
2021-12-11 20:52:35 +08:00
Ruanhui 32cb580ba2
HBASE-26485 Introduce a method to clean restore directory after Snapshot Scan (#3877)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-11-27 20:35:24 +08:00
Andrew Purtell cda5a87d85
HBASE-26446 CellCounter should report serialized cell size counts too (#3841)
Add the following stats for a given table:
 - 7. Total size of serialized cells of each CF.
 - 8. Total size of serialized cells of each qualifier.
 - 9. Total size of serialized cells across all rows.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-11-15 16:43:05 -08:00
Istvan Toth 60254bc184
HBASE-26398 CellCounter fails for large tables filling up local disk (#3798)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-10-28 09:42:42 +01:00
Andrew Purtell 8a6fed7b19
HBASE-26316 Per-table or per-CF compression codec setting overrides (#3730)
We get and retain Compressor instances in HFileBlockDefaultEncodingContext,
and could in theory call Compressor#reinit when setting up the context,
to update compression parameters like level and buffer size, but we do
not plumb through the CompoundConfiguration from the Store into the
encoding context. As a consequence we can only update codec parameters
globally in system site conf files.

Fine grained configurability is important for algorithms like ZStandard
(ZSTD), which offers more than 20 compression levels, where at level 1
it is almost as fast as LZ4, and where at higher levels it utilizes
computationally expensive techniques to rival LZMA at compression ratio
but trades off significantly for reduced compresson throughput. The ZSTD
level that should be set for a given column family or table will vary by
use case.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-19 12:04:25 -07:00
Andrew Purtell 42fe5e5baa
HBASE-26259 Fallback support to pure Java compression (#3691)
This change introduces provided compression codecs to HBase as
new Maven modules. Each module provides compression codec support
that formerly required Hadoop native codecs, which in turn relies
on native code integration, which may or may not be available on
a given hardware platform or in an operational environment. We
now provide codecs in the HBase distribution for users whom for
whatever reason cannot or do not wish to deploy the Hadoop native
codecs.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-10-06 13:17:18 -07:00
bitterfox 748a6827c4
HBASE-26238 Short message by Result#compareResults for VerifyReplication to avoid OOME (#3647)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-28 21:48:32 +08:00
Josh Elser 86bc640c17 HBASE-26273 Force ReadType.STREAM when the user does not explicitly set a ReadType on the Scan for a Snapshot-based Job
HBase 2 moved over Scans to use PREAD by default instead of STREAM like
HBase 1. In the context of a MapReduce job, we can generally expect that
clients using the InputFormat (batch job) would be reading most of the
data for a job. Cater to them, but still give users who want PREAD the
ability to do so.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
2021-09-13 19:09:35 -04:00
Wellington Ramos Chevreuil ee632bdcae
HBASE-26276 Allow HashTable/SyncTable to perform rawScan when comparing cells (#3673)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-09-10 16:26:10 +01:00
huaxiangsun b8ba3f788b
HBASE-26255 Add an option to use region location from meta table in TableSnapshotInputFormat (#3661)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2021-09-08 09:46:34 -07:00
Rushabh Shah ec747bcb29
HBASE-26106 AbstractFSWALProvider#getArchivedLogPath doesn't look for wal file in all oldWALs directory. (#3636)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-09-02 09:16:54 -07:00
bitterfox f62caa5c88
HBASE-26205 Use specified cluster conf for UserProvider in TableMRUtil#initCredentialsForCluster (#3592)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Rushabh Shah <shahrs87@gmail.com>
2021-09-01 20:30:41 +08:00
bitterfox d781113a08
HBASE-26204 Obtain credential for VerifyReplication with peerQuorumAddress (#3591)
Signed-off-by: Rushabh Shah <shahrs87@gmail.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-08-24 12:17:02 +01:00
Yechao Chen dc8196faf4
HBASE-24842 make export snapshot report size can be config (#2233)
Signed-off-by: stack <stack@apache.org>
2021-08-16 17:15:00 -07:00
bitterfox ad7e2cefc8
HBASE-26196 Support configuration override for remote cluster of HFileOutputFormat locality sensitive (#3582)
Signed-off-by: stack <stack@apache.org>
2021-08-16 10:26:25 -07:00
Duo Zhang 5f0950558f
HBASE-26096 Cleanup the deprecated methods in HBTU related classes and format code (#3503)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-07-29 10:18:38 +08:00
huaxiangsun d15f3cbea3
HBASE-26108 add option to disable scanMetrics in TableSnapshotInputFormat (#3516)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2021-07-22 11:20:15 -07:00
Duo Zhang 16721239e7
HBASE-26100 Set version as 3.0.0-alpha-2-SNAPSHOT in master (#3508)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-07-20 23:04:08 +08:00
Duo Zhang d30cc27097
HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate (#3478)
Signed-off-by: Michael Stack <stack@apache.org>
2021-07-19 09:29:08 +08:00
Duo Zhang 5118321ec9
HBASE-26059 Set version as 3.0.0-alpha-1 in master in prep for first RC of 3.0.0-alpha-1 (#3453)
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2021-07-02 07:50:41 +08:00
Andrew Purtell 335305e0cf
HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime (#3302)
We introduced EnvironmentEdgeManager as a way to inject alternate clocks
for unit tests. In order for this to be effective, all callers that would
otherwise use System.currentTimeMillis() must call
EnvironmentEdgeManager.currentTime() instead, except the implementers of
EnvironmentEdge.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-06-01 09:57:48 -07:00
Baiqiang Zhao d69d5c24b1
HBASE-25861 Correct the usage of Configuration#addDeprecation (#3249)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-05-14 09:31:06 -07:00
Michael Stack 2382f68b23
HBASE-25792 Filter out o.a.hadoop.thirdparty building shaded jars (#3184)
Need to add to allowed-licenses list too....

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Nick Dimiduk <ndimiduk@apache.org>
2021-04-27 08:37:25 -07:00
Duo Zhang 8d2a0efb7a
HBASE-25811 The client integration test is failing after HBASE-22120 merged (#3201)
move opentelemetry jars to client-facing-thirdparty
add opentelemetry jars when init map reduce job dependencies

Signed-off-by: Xin Sun <ddupgs@gmail.com>
2021-04-27 11:42:48 +08:00
Duo Zhang 302d9ea8b8 HBASE-25373 Remove HTrace completely in code base and try to make use of OpenTelemetry
Signed-off-by: stack <stack@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang ba3610d097
HBASE-19577 Use log4j2 instead of log4j for logging (#1708)
Signed-off-by: stack <stack@apache.org>
2021-03-20 09:21:25 +08:00