Commit Graph

17323 Commits

Author SHA1 Message Date
Jan Hentschel 12f1c7c1ce
HBASE-23777 Removed deprecated createTableDescriptor(String, int, int, int, KeepDeletedCells) from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 00:24:19 +01:00
Jan Hentschel 5a3ad6f08f
HBASE-23776 Removed deprecated method createLocalHTU() from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
2020-02-02 00:28:25 +01:00
stack 9b6d25066d HBASE-23764: Switch to IP address for ZK ensemble
ADDENDUM removing comment just to trigger full test suite in pom.xml
2020-02-01 14:59:01 -08:00
Bharath Vissapragada d9453203ae HBASE-23764: Switch to IP address for ZK ensemble
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 14:57:51 -08:00
Josh Elser 70699365ad HBASE-23774 Add the user-zh mailing list 2020-01-31 19:33:59 -05:00
stack ba9f026d69 HBASE-23770 [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart 2020-01-31 08:39:46 -08:00
stack 1a6bdc7c50 HBASE-23771 [Flakey Tests] Test TestSplitTransactionOnCluster Again 2020-01-31 08:35:58 -08:00
abhinaba.sarkar 77229c79e3 HBASE-23350 Make compaction files cacheonWrite configurable based on threshold
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-31 11:35:11 +05:30
Nick Dimiduk 85b0c8e6bc
HBASE-23763 Add 'new on release line report' for git/jira audit tool (#1104)
Codify building the summary of what's new on a release line
branch (i.e., `branch-2`), but not yet released on earlier release
branches of that line.

Builds a cvs report that looks like https://home.apache.org/~ndimiduk/new_for_branch-2.csv
2020-01-30 09:31:01 -08:00
Josh Elser 8b00f9f0b1 HBASE-17115 Define UI admins via an ACL
The Hadoop AccessControlList allows us to specify admins of the webUI
via a list of users and/or groups. Admins of the WebUI can mutate the
system, potentially seeing sensitive data or modifying the system.

hbase.security.authentication.spnego.admin.users is a comma-separated
list of users who are admins.
hbase.security.authentication.spnego.admin.groups is a comma-separated
list of groups whose membership are admins. Either of these
configuration properties may also contain an asterisk (*) which denotes
"any entity" (e.g user, group).

Previously, when a user was denied from some endpoint that was
designated for admins, they received an HTTP/401. In this case, it is
more correct to return HTTP/403 as they were correctly authenticated,
but they were disallowed from fetching the given resource. This commit
incorporates this change.

hbase.security.authentication.ui.config.protected also exists for users
who have sensitive information stored in the Hadoop service
configuration and want to limit access to this endpoint. By default,
the Hadoop configuration endpoint is not protected and any
authenticated user can access it.

The test is based off of work by Nihal Jain in HBASE-20472.

Co-authored-by: Nihal Jain <nihaljain.cs@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-29 16:36:55 -05:00
Viraj Jasani 66d198d35f
HBASE-23749 : TestHFileWriterV3 for all DataBlock Encoders
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-29 23:54:18 +05:30
Jan Hentschel 60b80b6298
HBASE-23736 Removed deprecated getTimeStampOfLastAppliedOp from MetricsSink
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2020-01-29 17:02:09 +01:00
stack 98cff8a26a HBASE-23753 Update of errorprone generated failures
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>

hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
 Complains about mismatch in types when Compare. Implement Compare in
 base Interface.

hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
 Complains pbs never return null.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java
 Needed redo because errorprone complains can't mock Service from guava.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Unrelated...adding one-liner debug statements chasing other test
 failures.
2020-01-28 18:54:35 -08:00
stack 92a4a1d7b7 HBASE-23746 [Flakey Tests] Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush and hsync 2020-01-28 09:45:08 -08:00
Jan Hentschel e836f5684f
HBASE-23738 Removed deprecated createLocalHRegion from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-28 12:40:00 +01:00
stack 85d0208baa HBASE-23751 Move core to hbase-thirdparty 3.2.0 2020-01-27 22:09:32 -08:00
Nick Dimiduk fb06ca5ba4
HBASE-23707 Add IntelliJ check style plugin configuration (#1064)
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-27 10:40:24 -08:00
Nick Dimiduk 5bada0b5d0
HBASE-22853 Git/Jira Release Audit Tool (#1088)
* HBASE-22853 Git/Jira Release Audit Tool

This is an application for performing an audit between the histories
on our git branches and the `fixVersion` field set on issues in
JIRA. It does this by building a Sqlite database from the commits
found on each git branch, identifying Jira IDs and release tags, and
then requesting information about those issues from Jira. Once both
sources have been collected, queries can be performed against the
database to look for discrepancies between the sources of truth (and,
possibly, bugs in this script).

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-27 10:28:11 -08:00
stack 375e01dac4 HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time
Second part of this issue -- changes to the test.
2020-01-26 10:41:42 -08:00
stack d82c3a5ebd HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time; AMENDMENT
This is actual fix; previous added debug to test.
2020-01-26 10:31:59 -08:00
stack 1690414263 HBASE-23735 [Flakey Tests] TestClusterRestartFailover & TestClusterRestartFailoverSplitWithoutZk 2020-01-24 18:21:33 -08:00
Michael Stack 753cc99814 HBASE-23726 Forward-port HBASE-21345 to branch-2.2, 2.3 & master as well.
HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'.
Just remove the check state from the getClusterStatus call.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthi@apache.org>
(cherry picked from commit dd8496a546)
2020-01-24 12:00:20 -08:00
stack 3c1bccb0f8 HBASE-23733 [Flakey Tests] TestSplitTransactionOnCluster 2020-01-24 11:25:02 -08:00
stack eda5df7eac HBASE-23729 [Flakeys] TestRSGroupsBasics#testClearNotProcessedDeadServer fails most of the time 2020-01-23 16:45:41 -08:00
Geoffrey Jacoby 2d6bb81dff HBASE-23710 - Priority configuration for system coprocessors (#1077)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 16:23:28 -08:00
stack 7c61c3992a HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM
AND.... undo thirdparty testing version update.
2020-01-23 14:22:30 -08:00
stack 44e66fc040 HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM
Remove staging repo added by mistake.
2020-01-23 14:16:58 -08:00
Josh Elser fd9e19c12e HBASE-23722 Real user might be null in non-proxy-user case
Closes #1085

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 16:02:01 -05:00
stack 0da08258a0 HBASE-23069 periodic dependency bump for Sep 2019 (#1082); ADDENDUM
For asciidoctor, s/1.5.8/1.5.8.1/.
2020-01-23 12:00:04 -08:00
Michael Stack 373857854a
HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… (#1012)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 09:34:54 -08:00
Michael Stack 988d34746d HBASE-23069 periodic dependency bump for Sep 2019 (#1082)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-23 09:27:27 -08:00
Wellington Ramos Chevreuil d6ac8b34eb
HBASE-23715 MasterFileSystem should not create MasterProcWALs dir on … (#1078)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-01-23 09:25:31 +00:00
Michael Stack 6cdc4b1f05
HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use.  Backfill.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
 Adds a new compareRows with default implementation that takes a ByteBuffer.
 Needed by the index in a block encoder implementation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
 Adds implementation for meta of new compareRows method. Adds utility
 method for figuring comparator based off tablename.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
 Comparator is in context. Remove redundant handling.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
 Comparator is in context. Remove redundant handling. Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
 Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
 Cache context so can use it to get comparator to use later.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Cache cellcomparator to use. Javdoc on diff between HFileContext and
 HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
 Add CellComparator

M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
 Remove comparator caching. Get from context instead.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
 Skip a reflection if we can.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
 Javadoc. Removed unused filed.

Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 20:00:21 -08:00
stack a58f2a4a55 HBASE-23720 [create-release] Update yetus version used from 0.11.0 to 0.11.1 2020-01-22 18:07:53 -08:00
Peter Somogyi 11b7ecb3af
HBASE-23719 Add 1.5.0 release to Downloads (#1083)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 15:46:37 +01:00
Josh Elser ae6a2de560 HBASE-23709 Unwrap the real user to properly dispatch proxy-user auth'n
REST and Thrift servers started failing because the check in
BuiltinProviderSelector wasn't checking the "real" user for kerberos
credentials. This resulted in the KerberosAuthnProvider not being
invoked when it should have been.

Closes #1080

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-01-22 10:12:06 +01:00
Geoffrey Jacoby bb56dfa974 HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL (#1079)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-22 00:07:15 -08:00
Michael Stack 2ed81c645b
HBASE-20516 Offheap read-path needs more detail (#1081)
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
2020-01-21 19:08:27 -08:00
Michael Stack ba3463d9de HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 15:02:40 -08:00
Peter Somogyi 00e64d83e8
HBASE-23347 Allow custom authentication methods for RPCs; addendum (#1060)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-21 18:47:13 +01:00
Guanghao Zhang 569ac1232a
HBASE-23156 start-hbase.sh failed with ClassNotFoundException when build with hadoop3 (#1067)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-20 13:22:04 +08:00
Jan Hentschel 50e264485d
HBASE-23686 Revert binary incompatible change in ByteRangeUtils and removed reflections in CommonFSUtils
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-19 18:06:31 +01:00
Jan Hentschel 75b85016eb
HBASE-23661 Reduced number of Checkstyle violations in hbase-rest
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-18 20:50:29 +01:00
Duo Zhang 167892ce64
HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
2020-01-18 20:30:47 +08:00
Peter Somogyi 65bcf55892
HBASE-23653 Expose content of meta table in web ui; addendum (#1061)
Fix error prone problem

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-18 09:33:21 +01:00
Nick Dimiduk 70c8a5d939
HBASE-23700 Upgrade checkstyle and plugin versions (#1056)
Bump checkstyle version to 8.28, maven-checkstyle-plugin to 3.1.0.

As per HBASE-23242 and the updated checkstyle docs[1], the LineLength
check should be placed under an instance of Checker.

[1] https://checkstyle.sourceforge.io/config_sizes.html#LineLength

Co-authored-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-17 16:58:49 -08:00
bzhaoopenstack 5480493f5f HBASE-23612 Add new profile to make hbase build success on ARM (#959) 2020-01-17 15:27:44 -08:00
Michael Stack 9e4323171a
HBASE-23690 Checkstyle plugin complains about our checkstyle.xml format; doc how to resolve mismatched version (#1044)
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-17 14:51:25 -08:00
Josh Elser df8f80a819 HBASE-23701 Try to converge automated checks around Category
We have a compile-time guarantee to either have a category array
of length zero or one because the category annotation is non-repeatable.

Also tries to clean up the checking for unit-test annotations.

Closes #1057

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-17 12:39:25 -05:00
Peter Somogyi c4395b52c9
HBASE-23703 Add HBase 2.2.3 documentation to website (#1059)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-17 16:31:13 +01:00