19296 Commits

Author SHA1 Message Date
Duo Zhang
31db8e3b3b
HBASE-26858 Refactor TestMasterRegionOnTwoFileSystems to avoid dead loop (#4238)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-18 23:09:09 +08:00
huiruan
84acdf98ce HBASE-26859 Split TestSnapshotProcedure to several smaller tests (#4239)
Closes #4239

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-18 17:52:15 +08:00
tianhang
eccd12a460
HBASE-26851 [Documentation] Point out compatibility issues with WAL Compression and Replication in doc (#4237)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-18 15:27:57 +08:00
Duo Zhang
bd65ad42a2 HBASE-26833 Addendum fix NPE in TestReplicationSource.testTerminateClearsBuffer 2022-03-17 23:13:41 +08:00
Duo Zhang
caea843e94 HBASE-26848 Addendum make tmpdir absolute 2022-03-17 13:02:14 +08:00
Nick Dimiduk
b1c161ea8e
HBASE-26813 Remove javax.ws.rs-api dependency (#4191)
This is no longer needed since we've transitioned to the shaded Jersey shipped in
hbase-thirdparty. Also drop supplemental models entry.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-16 16:46:15 +01:00
Duo Zhang
fec5478226 HBASE-26828 Addendum fix syntax error 2022-03-16 21:48:58 +08:00
xicm
ef280d4d8d
HBASE-26720 ExportSnapshot should validate the source snapshot before copying files (#4216)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-16 20:03:28 +08:00
xicm
b9c59438a4
Hbase 26275 update error message when executing deleteall with ROWPREFIXFILTER in meta table (#4205)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-16 20:02:38 +08:00
Xiaolin Ha
c7773adeed
HBASE-26680 Close and do not write trailer for the broken WAL writer (#4174)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-16 19:58:45 +08:00
Duo Zhang
5b28d76652
HBASE-26837 Set SFT config when creating TableDescriptor in TestCloneSnapshotProcedure (#4226)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-16 19:34:30 +08:00
Duo Zhang
cc13624232
HBASE-26848 Set java.io.tmpdir on mvn command when running jenkins job (#4231)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-03-16 17:52:36 +08:00
Nick Dimiduk
c4ec5230f2
HBASE-25896 Implement a Region Visualization on Master WebUI (#4228)
This is a demonstration of visualization of regions on the cluster. The visualization is a stacked
bar chart showing total storefile size per table per region server, with the x-axis being server
names, the y-axis being storfile size, and the bars stacked per table. The visualization is
generated entirely on the fly from within the browser, implemented using Vega Lite. So far, Vega
appears to handle rendering this visualization for a cluster of over 700 region servers with
approximately 300,000 regions.

Per [0], include an update to the top-level LICENSE.txt. Also update LICENSE files in all binary
distributions (i.e., jars), by way of LICENSE.vm. Vega uses a BSD 3-clause variant without
advertising clause, and as such is a "Category A" license, per [1].

No changes are made to the NOTICE files, as per the existing example of bundling the minified
JQuery, which is also a Category A license.

[0]: https://infra.apache.org/licensing-howto.html
[1]: https://www.apache.org/legal/resolved.html#category-a

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-15 21:01:57 +01:00
Nick Dimiduk
2507dc1b01 Revert "HBASE-25896 Implement a Region Visualization on Master WebUI"
This reverts commit d76e636da04b54781b53aea6dc9564559fae25ba.
2022-03-15 20:43:46 +01:00
Nick Dimiduk
d76e636da0 HBASE-25896 Implement a Region Visualization on Master WebUI
This is a demonstration of visualization of regions on the cluster. The visualization is a stacked
bar chart showing total storefile size per table per region server, with the x-axis being server
names, the y-axis being storfile size, and the bars stacked per table. The visualization is
generated entirely on the fly from within the browser, implemented using Vega Lite. So far, Vega
appears to handle rendering this visualization for a cluster of over 700 region servers with
approximately 300,000 regions.

Per [0], include an update to the top-level LICENSE.txt. Also update LICENSE files in all binary
distributions (i.e., jars), by way of LICENSE.vm. Vega uses a BSD 3-clause variant without
advertising clause, and as such is a "Category A" license, per [1].

No changes are made to the NOTICE files, as per the existing example of bundling the minified
JQuery, which is also a Category A license.

[0]: https://infra.apache.org/licensing-howto.html
[1]: https://www.apache.org/legal/resolved.html#category-a

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-15 20:31:14 +01:00
Xiaolin Ha
d050217a3d
HBASE-26833 Avoid waiting to clear buffer usage of ReplicationSourceShipper when aborting the RS (#4218)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-15 22:04:40 +08:00
Xiaolin Ha
fb09cb9e7b
HBASE-26840 Fix NPE in the retry of logroller (#4224)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-15 21:59:09 +08:00
Duo Zhang
a816204793
HBASE-26611 Changing SFT implementation on disabled table is dangerous (#4082)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-15 20:13:46 +08:00
Duo Zhang
5ad51e8b2b
HBASE-26835 Rewrite TestLruAdaptiveBlockCache to make it more stable (#4219)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-15 20:06:05 +08:00
Duo Zhang
cc2c22072f
HBASE-26831 TestAdminShell2 is failing because of we referenced a deleted method (#4217)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-15 14:26:07 +08:00
Nick Dimiduk
f640b7fd27 HBASE-25895 Implement a Cluster Metrics JSON endpoint (addendum) 2022-03-14 17:27:53 +01:00
Nick Dimiduk
36a5f86c50 HBASE-26531 Trace coprocessor exec endpoints
Trace table ExecService invocations as table operations. Ensure span relationships for both table
and master invocations.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-14 11:27:30 +01:00
Duo Zhang
a49d147d49
HBASE-26828 Increase the concurrency when running UTs in pre commit job (#4213)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2022-03-14 12:01:51 +08:00
meiyi
d542f471d1
HBASE-26670 HFileLinkCleaner should be added even if snapshot is disabled (#4032)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-14 11:47:23 +08:00
Duo Zhang
c3d0cc7040
HBASE-26830 Rewrite TestLruBlockCache to make it more stable (#4212)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-14 11:04:15 +08:00
Ruanhui
fd301ad55f
HBASE-26323 Introduce a Snapshot Procedure (#4115)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-12 23:21:22 +08:00
Duo Zhang
c2485215de
HBASE-25465 Use javac --release option for supporting cross version compilation (#4164)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-12 16:51:49 +08:00
Duo Zhang
f4866a49dc
HBASE-26822 Revert the changes on hbase-daemon.sh after switching to log4j2 properties file (#4209)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-12 16:48:06 +08:00
Andrew Purtell
71ff3ec323
HBASE-26761 TestMobStoreScanner (testGetMassive) can OOME (#4204)
Change the timing of the test but not the objective by storing three large
MOB values into the row with three separate puts, each randomizing the
data. Increases running time of the testGetMassive case 2x but avoids OOME
in the test environment where the OOME was consistently reproductable.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2022-03-11 14:44:42 +08:00
Andrew Purtell
f3faa26047
HBASE-26824 TestHBaseTestingUtil.testResolvePortConflict failing after HBASE-26582 (#4203)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-11 14:43:00 +08:00
Xiaolin Ha
dbf56819c3
HBASE-26816 Fix CME in ReplicationSourceManager (#4187)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-11 11:06:14 +08:00
Andrew Purtell
addace21b6
HBASE-26821 Bump dependencies in /dev-support/git-jira-release-audit (#4193)
Bumps urllib3 from 1.25.8 to 1.26.5 to resolve two dependabot warnings

  CRLF injection (Moderate)
  urllib3 (pip) · dev-support/git-jira-release-audit/requirements.txt

  Catastrophic backtracking in URL authority parser when passed URL containing many @ characters (High)
  urllib3 (pip) · dev-support/git-jira-release-audit/requirements.txt

Bumps cryptography from 2.8 to 3.3.2 to resolve one dependabot warning

  RSA decryption vulnerable to Bleichenbacher timing vulnerability (Moderate)
  cryptography (pip) · dev-support/git-jira-release-audit/requirements.txt

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-03-10 12:25:33 -08:00
Andrew Purtell
e5dbbd20ac
HBASE-26715 Blocked on SyncFuture in AsyncProtobufLogWriter#write (#4184)
Co-authored-by: Bryan Beaudreault <bbeaudreault@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-10 11:55:38 -08:00
Bryan Beaudreault
4da53efcf9
HBASE-26814 Addendum: Fix spotbugs warning and clarify intent (#4201)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-10 10:09:51 -08:00
Nick Dimiduk
be0afbf23a HBASE-25895 Implement a Cluster Metrics JSON endpoint
Publishes a set of JSON endpoints following a RESTful structure, which expose a subset of the
`o.a.h.h.ClusterMetrics` object tree. The URI structure is as follows

    /api/v1/admin/cluster_metrics
    /api/v1/admin/cluster_metrics/live_servers
    /api/v1/admin/cluster_metrics/dead_servers

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-10 14:58:05 +01:00
Nick Dimiduk
5851400a46 HBASE-26819 Minor code cleanup in and around RpcScheduler
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-10 14:45:04 +01:00
Nick Dimiduk
068031ea82 HBASE-26817 Mark RpcExecutor as IA.LimitedPrivate COPROC and PHOENIX
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-10 14:36:39 +01:00
xicm
9293d6af2c
HBASE-26815 TestFanOutOneBlockAsyncDFSOutput is flakey (#4186)
Trigger BlockReports in MiniDFSCluster after restarting datanode

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-10 16:06:12 +08:00
chenglei
1057da8be6
HBASE-26768 Avoid unnecessary replication suspending in RegionReplica… (#4127) 2022-03-10 10:58:47 +08:00
Bryan Beaudreault
1dd29dba5c
HBASE-26814 Default StoreHotnessProtector to off, with logs to guide when to turn it on (#4194)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-09 14:43:58 -08:00
Andrew Purtell
10471944bd
HBASE-26582 Prune use of Random and SecureRandom objects (#4118)
Avoid the pattern where a Random object is allocated, used once or twice, and
then left for GC. This pattern triggers warnings from some static analysis tools
because this pattern leads to poor effective randomness. In a few cases we were
legitimately suffering from this issue; in others a change is still good to
reduce noise in analysis results.

Use ThreadLocalRandom where there is no requirement to set the seed to gain
good reuse.

Where useful relax use of SecureRandom to simply Random or ThreadLocalRandom,
which are unlikely to block if the system entropy pool is low, if we don't need
crypographically strong randomness for the use case. The exception to this is
normalization of use of Bytes#random to fill byte arrays with randomness.
Because Bytes#random may be used to generate key material it must be backed by
SecureRandom.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-08 13:49:02 -08:00
Bryan Beaudreault
39ecaa1975
HBASE-26784 Use HIGH_QOS for ResultScanner.close requests (#4146)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-08 10:48:57 +08:00
Sean Busbey
bcd9a9acef
HBASE-26760 LICENSE handling should not allow non-aggregated apache-2.0 (#4172)
- correct the license name for the Xerial Snappy wrapper.
- correct the license name for the version of GSON brought in with Hadoop 3.3.2+

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2022-03-07 10:31:02 -06:00
Nick Dimiduk
f3f2aa9631 HBASE-26782 Minor code cleanup in and around RpcExecutor
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-07 11:55:36 +01:00
Xiaolin Ha
591f7819f3
HBASE-26552 Introduce retry to logroller to avoid abort (#4038)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-07 12:20:01 +08:00
Xiaolin Ha
f3a48d1910
HBASE-25709 Close region may stuck when region is compacting and skipped most cells read (#3117)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-07 11:21:39 +08:00
Duo Zhang
fee9bb0c2f
HBASE-26803 Ban reload4j and logback dependencies (#4167)
Signed-off-by: Sean Busbey <busbey@apache.org>
2022-03-07 09:54:20 +08:00
Duo Zhang
19ad3d6654 HBASE-26805 Addendum forward port the test improvements in HBASE-26304 for branch-2 2022-03-07 08:16:21 +08:00
Duo Zhang
480db06e46
HBASE-26805 TestRegionHDFSBlockLocationFinder is flaky (#4169)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-03-07 08:08:04 +08:00
Rushabh Shah
2b205d2230
HBASE-26792 Implement ScanInfo#toString (#4153)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-05 17:45:37 +08:00