Commit Graph

1632 Commits

Author SHA1 Message Date
Bharath Vissapragada b930ea1ebb
HBASE-25528: Dedicated merge dispatch threadpool on master (#2910)
Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2.

Also adds additional logging that includes the number of split plans
and merge plans computed for each normalizer run.

(cherry picked from commit 36b4698cea)

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-28 16:09:18 -08:00
Duo Zhang f70229b193 HBASE-25509 ChoreService.cancelChore will not call ScheduledChore.cle… (#2890)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-20 23:24:16 +08:00
Baiqiang Zhao c5674e21dd
HBASE-25439 Add BYTE unit in PrettyPrinter.Unit (#2840)
Signed-off-by: stack <stack@apache.org>
2021-01-14 21:47:01 -08:00
申胜利 478c4e8d6a HBASE-25449 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
Revert of the revert -- re-applying HBASE-25449 with a change
of renaming the test hdfs XML configuration file as it was adversely
affecting tests using MiniDFS

This reverts commit c218e576fe.

Co-authored-by: Josh Elser <elserj@apache.org>

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-01-13 21:06:13 -05:00
Duo Zhang 51e8cf4d34 Revert "HBASE-25449 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml"
This reverts commit 661bea9135.
2021-01-12 16:09:27 +08:00
Viraj Jasani 0788547fea
HBASE-25474 : Bump HBase version on branch-2 (#2871)
Signed-off-by: stack <stack@apache.org>
2021-01-12 10:20:22 +05:30
申胜利 661bea9135 HBASE-25449 'dfs.client.read.shortcircuit' should not be set in hbase-default.xml
Signed-off-by: Josh Elser <elserj@apache.org>
2021-01-08 15:20:27 -05:00
申胜利 d2c1886bf4 Shenshengli hbase-25450 The parameter "hbase.bucketcache.size" is misdescribed (#2821)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@apache.org>
2021-01-03 09:31:10 -08:00
Pankaj 382ebf3f4e HBASE-25379 Make retry pause time configurable for regionserver short operation RPC (reportRegionStateTransition/reportProcedureDone) (#2757)
* HBASE-25379 Make retry pause time configurable for regionserver short operation RPC (reportRegionStateTransition/reportProcedureDone)
* HBASE-25379 RemoteProcedureResultReporter also should retry after the configured pause time
* Addressed the review comments

Signed-off-by: Yulin Niu <niuyulin@apache.org>
(cherry picked from commit c96fbf0407)
2020-12-29 22:33:31 +05:30
ramkrish86 1c41c9732e HBASE-24850 CellComparator perf improvement (#2776)
* Using ContiguousCellFormat as a marker alone

* Commit the new file

* Fix the comparator logic that was an oversight

* Fix the sequenceId check order

* Adding few more static methods that helps in scan flow like query
matcher where we have more cols

* Remove ContiguousCellFormat and ensure compare() can be inlined

* applying negation as per review comment

* Fix checkstyle comments

* fix review comments

* Address review comments

Signed-off-by: stack <stack@apache.org>
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-12-28 13:08:50 +05:30
stack b07b400b62 HBASE-25425 Some notes on RawCell 2020-12-21 08:13:42 -08:00
Adam 9c2f1f7a2b HBASE-25370 Fix flaky test TestClassFinder#testClassFinderDefaultsToOwnPackage (#2740)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-11 08:23:49 +08:00
shahrs87 65ac1dbeb0
HBASE-25328 : Add builder method to create Tags
Closes #2707

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-08 18:17:30 +05:30
Andrew Purtell ea331a66be HBASE-25292 Improve InetSocketAddress usage discipline (#2669)
Network identities should be bound late. Remote addresses should be
resolved at the last possible moment, just before connect(). Network
identity mappings can change, so our code should not inappropriately
cache them. Otherwise we might miss a change and fail to operate normally.

Revert "HBASE-14544 Allow HConnectionImpl to not refresh the dns on errors"
Removes hbase.resolve.hostnames.on.failure and related code. We always
resolve hostnames, as late as possible.

Preserve InetSocketAddress caching per RPC connection. Avoids potential
lookups per Call.

Replace InetSocketAddress with Address where used as a map key. If we want
to key by hostname and/or resolved address we should be explicit about it.
Using Address chooses mapping by hostname and port only.

Add metrics for potential nameservice resolution attempts, whenever an
InetSocketAddress is instantiated for connect; and metrics for failed
resolution, whenever InetSocketAddress#isUnresolved on the new instance
is true.

* Use ServerName directly to build a stub key

* Resolve and cache ISA on a RpcChannel as late as possible, at first call

* Remove now invalid unit test TestCIBadHostname

We resolve DNS at the latest possible time, at first call, and do not
resolve hostnames for creating stubs at all, so this unit test cannot
work now.

Reviewed-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-04 10:19:02 -08:00
Mate Szalay-Beko 1dbfe96d69 HBASE-25263 Various improvements of column family encryption
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:

1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.

Smaller issues fixed by this commit:

2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.

3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.

4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.

5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.

closes #2676

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-26 16:57:12 +01:00
Andrew Purtell 1b6399a480
HBASE-24640 [branch-2] Purge use of VisibleForTesting (#2696)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:15:32 -08:00
ramkrish86 988be9321d HBASE-25050 - We initialize Filesystems more than once. (#2419)
* HBASE-25050 - We initialize Filesystems more than once.

* Ensuring that calling the FS#get() will only ensure FS init.

* Fix for testfailures. We should pass the entire path and no the scheme
alone

* Cases where we don't have a scheme for the URI

* Address review comments

* Add some comments on why FS#get(URI, conf) is getting used

* Adding the comment as per Sean's review

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-11-24 15:31:55 +05:30
ramkrish86 a8e02c97bc HBASE-25187 Improve SizeCachedKV variants initialization (#2582)
* HBASE-25187 Improve SizeCachedKV variants initialization

* HBASE-25187 Improve SizeCachedKV variants initialization

* The BBKeyValue also can be optimized

* Change for SizeCachedKeyValue

* Addressing revew comments

* Fixing checkstyle and spot bugs comments

* Spot bug fix for hashCode

* Minor updates make the rowLen as short and some consturctor formatting

* Change two more places where there was a cast
2020-11-24 15:10:44 +05:30
Mate Szalay-Beko 9fe62c3ec6
HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
Prior to this patch hbase always used the MD5 hash algorithm to store a hash for encryption keys.
This hash is needed to verify the secret key of the subject. (e.g. making
sure that the same secrey key is used during encrypted HFile read and write).
The MD5 algorithm is considered weak, and can not be used in some
(e.g. FIPS compliant) clusters.

In this patch we:
- add a config parameter to globally enable/disable column family encryption (def enabled)
- introduce a backward compatible way of specifying the hash algorithm.
  This enable us to use newer and more secure hash algorithms like SHA-384
  or SHA-512 (which are FIPS compliant).
- add a config parameter to fail if an hfile is encountered that uses a
  different hash algorithm than the one currently configured to ease validation after
  migrating key hash algorithms (def disabled)

Closes #2539

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
(cherry picked from commit 6a5c928539)
2020-11-09 14:15:22 -06:00
gvprathyusha6 21b32d9a6f
HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Closes #2542

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-11-09 12:41:04 +05:30
WenFeiYi de74312d74
HBASE-25240 gson format of RpcServer.logResponse is abnormal
Closes #2623

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-05 19:59:34 +05:30
Nick Dimiduk 1c7d472537 HBASE-25167 Normalizer support for hot config reloading (#2523)
Wire up the `ConfigurationObserver` chain for
`RegionNormalizerManager`. The following configuration keys support
hot-reloading:
 * hbase.normalizer.throughput.max_bytes_per_sec
 * hbase.normalizer.split.enabled
 * hbase.normalizer.merge.enabled
 * hbase.normalizer.min.region.count
 * hbase.normalizer.merge.min_region_age.days
 * hbase.normalizer.merge.min_region_size.mb

Note that support for `hbase.normalizer.period` is not provided
here. Support would need to be implemented generally for the `Chore`
subsystem.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-10-30 15:32:36 -07:00
Nick Dimiduk b84e2f598b HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-10-30 10:43:43 -07:00
sanjeetnishad95 dd0a14538b HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into account configuration map. (#2484)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-23 08:43:31 +08:00
Duo Zhang 37c2ffdc2b HBASE-25164 Make ModifyTableProcedure support changing meta replica count (#2513)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-13 10:13:48 +08:00
Michael Stack 45e10d43fe HBASE-22976 [HBCK2] Add RecoveredEditsPlayer (#2504)
Make it so WALPlayer can replay recovered.edits files.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
 Allow for WAL files that do NOT have a startime in their name.
 Use the 'generic' WAL-filename parser instead of the one that
 used be local here. Implement support for 'startTime' filter.
 Previous was just not implemented.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
 Checkstyle.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
 Use the new general WAL name timestamp parser.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility for parsing timestamp from WAL filename.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
 Export attributes about the local recovered.edits test file
 so other tests can play with it.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-10-09 08:54:53 -07:00
Duo Zhang fe03a35ea2 HBASE-25066 Use FutureUtils.rethrow in AsyncTableResultScanner to better catch the stack trace (#2420)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-20 11:28:01 +08:00
Viraj Jasani 4316dc738c
HBASE-24528 : BalancerDecision queue implementation in HMaster with Admin API (#2411)
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto

Closes #2261

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-09-17 18:51:42 +05:30
Duo Zhang 7a3bb8aefe HBASE-25037 Lots of thread pool are changed to non daemon after HBASE-24750 which causes trouble when shutting down (#2407)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-16 22:03:42 +08:00
XinSun 4d012db00f HBASE-25014 ScheduledChore is never triggered when initalDelay > 1.5*period (#2395)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-16 09:01:37 +08:00
XinSun c7b930a9e8
HBASE-24759 Refuse to update configuration of default group (#2350)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-04 18:45:12 +08:00
Viraj Jasani 859bf3ea9a
HBASE-24968 : Move META_COMPARATOR to subclass MetaCellComparator (#2332)
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
  static initializer by moving META_COMPARATOR to subclass MetaCellComparator

Closes #2329

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-31 19:51:00 +05:30
Nick Dimiduk 9c85f3641f HBASE-24583 Normalizer can't actually merge empty regions...
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-08-12 10:11:40 -07:00
Viraj Jasani 8ccf643fdc
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder (#2214)
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: niuyulin <nyl353@163.com>
2020-08-12 15:57:53 +05:30
bsglz ebf493f075
HBASE-24694 Support flush a single column family of table (#2218)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-10 10:36:13 +01:00
Duo Zhang ef7b9eb36e HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762)
Signed-off-by: stack <stack@apache.org>
2020-08-10 10:54:30 +08:00
niuyulin e099aa5e20 HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-06 14:52:20 -07:00
Sean Busbey 0806349ada
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe
* refactor how we use connection to rely on the access method
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 86ebbdd8a2)
2020-08-02 00:51:03 -05:00
stack edb4cd534c HBASE-24632 Enable procedure-based log splitting as default in hbase3 Add deprecation of 'classic' zk-based WAL splitter.
Also fix three bugs:

 * We were trying to delete non-empty directory; weren't doing
 accounting for meta WALs where meta had moved off the server
 (successfully)
 * We were deleting split WALs rather than archiving them.
 * We were not handling corrupt files.

Deprecations and removal of tests of old system.
2020-07-27 08:23:29 -07:00
Viraj Jasani 6cb51cc0f0
HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold (#2136)
Closes #2127

Signed-off-by: stack <stack@apache.org>
2020-07-24 20:32:40 +05:30
Viraj Jasani ce4e692699
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2110)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-22 11:36:19 +05:30
Peter Somogyi e0c917b647 HBASE-24710 Incorrect checksum calculation in saveVersion.sh (#2056)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-07-21 10:41:38 +02:00
stack 620470607e Revert "HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)"
Causes TestAdminShell and TestThriftHBaseServiceHandler to fail 100% of the time in branch-2.

This reverts commit 8ae3480e70.
2020-07-20 14:50:03 -07:00
Viraj Jasani 8ae3480e70
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-17 19:57:05 +05:30
bsglz ec3beaf5a2
HBASE-24664 Some changing of split region by overall region size rather than only one store size (#2054)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-17 10:47:01 +01:00
Bo Cui 44d351915a
HBASE-24693 regioninfo#isLast() should point to EMPTY_END_ROW (readability improvement)
Closes #2051

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-14 15:43:17 +05:30
XinSun 5fcffae5db
HBASE-24431 RSGroupInfo add configuration map to store something extra (#2031)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-07-08 14:39:41 +08:00
pengmq1 2a12fd283e HBASE-24289 Heterogeneous Storage for Date Tiered Compaction (#1730)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-30 15:19:30 +08:00
Duo Zhang ed6104418c HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU (#1987)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-06-28 23:27:42 +08:00
Bharath Vissapragada a6c887012e
HBASE-24603: Make Zookeeper sync() call synchronous (#1945) (#1975)
Writing a test for this is tricky. There is enough coverage for
functional tests. Only concern is performance, but there is enough
logging for it to detect timed out/badly performing sync calls.

Additionally, this patch decouples the ZK event processing into it's
own thread rather than doing it in the EventThread's context. That
avoids deadlocks and stalls of the event thread.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 84e246f9b1)
2020-06-25 16:41:11 -07:00
Nick Dimiduk 3db4f00f24 HBASE-22504 Addendum: restore findCommonPrefix
Address incompatibility issue raised in 2.3.0RC0 vote thread.
2020-06-25 11:17:22 -07:00
Michael Stack c4042f4927
HBASE-23055 Alter hbase:meta (#1956)
Addendum to fix illegal removal of unused constant w/o
a deprecation cycle.
2020-06-22 20:20:42 -07:00
Anoop Sam John ed7dc9ed4c HBASE-24189 WALSplit recreates region dirs for deleted table with recovered edits data. 2020-06-11 21:28:32 +05:30
Andrew Purtell b67f896954
HBASE-24367 ScheduledChore log elapsed timespan in a human-friendly format (#1749)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-06-09 09:57:03 -07:00
XinSun 67fa367bb6 HBASE-24359 Optionally ignore edits for deleted CFs for replication (#1855)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-08 13:14:50 +08:00
Nick Dimiduk 4884773676 HBASE-24418 Consolidate Normalizer implementations
Simplify our Normalizer story to have just a single, configurable
implementation.

* fold the features of `MergeNormalizer` into
  `SimpleRegionNormalizer`, removing the intermediate abstract class.
* configuration keys for merge-only features now share a common
  structure.
* add configuration to selectively disable normalizer split/merge
  operations.
* `RegionNormalizer` now extends `Configurable` instead of creating a
  new instance of `HBaseConfiguration` or snooping one off of other
  fields.
* avoid the extra RPCs by using `MasterServices` instead of
  `MasterRpcServices`.
* boost test coverage of all the various flags and feature
  combinations.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-06-03 16:43:40 -07:00
niuyulin 571ef7e811 HBASE-24412 Canary support check only one column family per RegionTask (#1817)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-03 09:16:11 +08:00
Bharath Vissapragada d7f388518c
HBASE-24477: Move ConfigurationObserver and related classes to hbase-common
This utility is useful for any module that wants to detect
dynamic config changes. Having it to hbase-common makes it
accessible to all the other modules.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-31 14:14:30 -07:00
bsglz 22efe67758
HBASE-24451 Remove the HasThread because the related bug had been fixed since jdk7 (#1797)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-29 17:46:03 +05:30
Duo Zhang 58eed9a4bb HBASE-24408 Introduce a general 'local region' to store data on master (#1753)
Signed-off-by: stack <stack@apache.org>
2020-05-23 16:40:39 +08:00
Viraj Jasani 2383cc4242
HBASE-24405 : Document hbase:slowlog related operations (#1747)
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-05-22 20:58:04 +05:30
Viraj Jasani 6f5e5e4828
HBASE-23938 : System table hbase:slowlog to store complete slow/large… (#1681)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-05-20 15:30:30 +05:30
Mingliang Liu 5893c4f393
HBASE-23969 Meta browser should show all `info` columns (#1710)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-18 11:48:36 -07:00
Nick Dimiduk 1c32a8ba2e HBASE-24271 Set values in `conf/hbase-site.xml` that enable running on `LocalFileSystem` out of the box
Simplify the new user experience shipping a configuration that enables
a fresh checkout or tarball distribution to run in standalone mode
without direct user configuration. This change restores the behavior
we had when running on Hadoop 2.8 and earlier.

Patch for master includes an update to the book. This change will be
omitted when backporting to earlier branches.

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-05-14 14:05:40 -07:00
Nick Dimiduk 2ab394ddb1 Revert "HBASE-24086 Disable output stream capability enforcement when running on LocalFileSystem"
This reverts commit 4e31ada43b.
2020-05-14 14:05:40 -07:00
shahrs87 941cef71c2
HBASE-24190 : Make kerberos value of hbase.security.authentication property case insensitive (#1687)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-14 11:46:33 +05:30
Viraj Jasani b78f4367f7
Revert "Make kerberos value of hbase.security.authentication property case insensitive (#1687)"
This reverts commit c03ec837e7.
2020-05-14 11:45:23 +05:30
Duo Zhang dc2146069c
HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for … (#1697)
Signed-off-by: stack <stack@apache.org>
2020-05-13 17:59:21 +08:00
Sambit Mohapatra d85caaa195
HBASE-23832 Old config hbase.hstore.compactionThreshold is ignored (#1698)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-05-12 23:20:48 +05:30
shahrs87 c03ec837e7
Make kerberos value of hbase.security.authentication property case insensitive (#1687)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-11 11:10:29 +05:30
Duo Zhang 7f4683bafa HBASE-24265 Remove hedged rpc call support, implement the logic in MaterRegistry … (#1593)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-05-06 16:11:28 +08:00
Michael Stack 528e9d6454
HBASE-24307 [Flakey Tests] krb server for secure thrift tests throws BindException (#1656)
Move the random free port generate back into hbasecommontestingutility
  from hbasetestingutility.

  Add a create simple kdc server utility that will start a kdc server and
  if a bindexception, create a new one on a new random port in hbase-common.

  Add new BoundSocketMaker helpful when trying to manufacture
  BindExceptions because of port clash.

  Change thrift and http kdc tests to use this new utility (removes
  code duplication around kdc server setup).
2020-05-05 21:08:09 -07:00
Andrew Purtell 5dc10dbba5
HBASE-24322 UnsafeAvailChecker should also check that required methods are available (#1650)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-05-05 12:11:37 -07:00
Duo Zhang 1f556b3ad9 HBASE-24281 Remove some stale hadoop.version properties in the pom for sub modules (#1602)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-29 18:25:41 +08:00
Duo Zhang 379b48bb5d Revert "HBASE-24181 Remove some stale hadoop.version properties in the pom for sub modules (#1602)"
This reverts commit dc9608ec0b.
2020-04-29 18:25:23 +08:00
Duo Zhang dc9608ec0b HBASE-24181 Remove some stale hadoop.version properties in the pom for sub modules (#1602)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-29 18:08:33 +08:00
Duo Zhang 922921ee5f HBASE-24249 Move code in FSHDFSUtils to FSUtils and mark related clas… (#1586)
Signed-off-by: stack <stack@apache.org>
2020-04-29 11:31:32 +08:00
Viraj Jasani b40381857a
HBASE-24024 : Reject multi() requests with rows higher than threshold (#1560)
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-04-22 12:51:49 +05:30
Mark Robert Miller 8a7aa89506
HBASE-24185: Junit tests do not behave well with System.exit or Runtime.halt or JVM exits in general. (#1540)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-20 22:20:08 -07:00
Viraj Jasani ca1f7a9499
HBASE-24194 : Refactor anonymous inner classes of BufferedEncodedSeeker to named inner classes (#1522)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-04-16 19:17:44 +05:30
stack ba87ffed09 Revert "HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread."
Revert because applied with wrong JIRA number.

This reverts commit 9c60c5fddb.
2020-04-07 10:17:48 -07:00
stack 9c60c5fddb HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread.
Up the nproc in hbase-personality. Add test that prints out env.
2020-04-07 07:37:48 -07:00
Nick Dimiduk 4e31ada43b HBASE-24086 Disable output stream capability enforcement when running on LocalFileSystem
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-03 09:24:29 -07:00
Duo Zhang 7d842ca886
HBASE-24050 Deprecated PBType on all 2.x branches (#1354)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-27 10:43:14 +08:00
Duo Zhang 30eba2c24e HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 18:10:03 +08:00
Nick Dimiduk ffb2359146
HBASE-24013 Bump branch-2 version to 2.4.0-SNAPSHOT (#1309)
Increment version in poms with

```
$ mvn org.codehaus.mojo:versions-maven-plugin:2.7:set -DnewVersion=2.4.0-SNAPSHOT -DgenerateBackupPoms=false
```

Verified no dangling references with

```
$ find . -iname '*pom.xml' -exec grep -n '2.3.0-SNAPSHOT' {} +
```

Verified build with

```
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home mvn clean package -DskipTests
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home mvn clean package -DskipTests -Dhadoop.profile=3.0
```

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-19 08:01:43 -07:00
Nick Dimiduk 33cb9fc932 HBASE-23829 Get `-PrunSmallTests` passing on JDK11
Signed-off-by: stack <stack@apache.org>
2020-03-17 09:01:02 -07:00
Duo Zhang 7eeb6a0815 HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 11:42:23 +08:00
Aman Poonia c9c22fedaa HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-10 15:47:18 -07:00
Michael Stack a6700bc6b5
HBASE-23851 Log networks and bind addresses when multicast publisher/listener enabled (#1173)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-10 10:55:44 -07:00
Anoop Sam John 9ff3fe11c4
HBASE-23788 ROW_INDEX_V1 encoder should consider the secondary index size with the encoded data size tracking (#1241)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-03-06 06:15:15 +05:30
Viraj Jasani 6366b73134
HBASE-22978 : Online slow response log (#1228)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-03-01 00:32:35 +05:30
Bharath Vissapragada 5848b7d835 HBASE-23804: Fix default master addr hostname in master registry (#1137)
Master rpc server end point doesn't bind to localhost's
IP address by default. Instead, it looks up the hostname and
binds to the endpoint to which it resolves. MasterRegistry should
do the same when building the default server end point to talk to.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit c34dbc3c29)
2020-02-27 10:33:42 -08:00
Bharath Vissapragada bd13416275 HBASE-23731: De-flake TestFromClientSide (#1091)
There were a couple of issues.

- There was a leak of a file descriptor for hbck lock file. This
was contributing to all the "ConnectionRefused" stack traces since
it was trying to renew lease for an already expired mini dfs cluster.
This issue was there for a while, just that we noticed it now.

- After upgrade to JUnit 4.13, it looks like the behavior for test
timeouts has changed. Earlier the timeout seems to have applied for
each parameterized run, but now it looks like it is applied across
all the runs.

This patch fixes both the issues.

Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
(cherry picked from commit 5c88672d54)
2020-02-27 10:33:42 -08:00
Bharath Vissapragada 69e3e0e2ef HBASE-23647: Make MasterRegistry the default impl. (#1039)
Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit 229b8aaaf3)
2020-02-27 10:33:42 -08:00
Bharath Vissapragada be30d43a6c HBASE-23305: Master based registry implementation (#954)
Implements a master based registry for clients.

 - Supports hedged RPCs (fan out configured via configs).
 - Parameterized existing client tests to run with multiple registry combinations.
 - Added unit-test coverage for the new registry implementation.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit 62da419b23)
2020-02-27 10:33:42 -08:00
stack 8e11a5caa0 HBASE-23789 [Flakey Tests] ERROR [Time-limited test] balancer.HeterogeneousRegionCountCostFunction(199): cannot read rules file located at ' /tmp/hbase-balancer.rules ' 2020-02-04 16:37:56 -08:00
Mark Robert Miller e385fd97e0 HBASE-23783: Address tests writing and reading SSL/Security files in a common location. (#1116)
This is causing me issues with parallel test runs.

Also allow setting the surefire reports and temp directories via command line.

Signed-off-by: stack <stack@apache.org>
2020-02-04 14:40:52 -08:00
Michael Stack 12f4e0977c
HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.


Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 09:16:47 -08:00
Bharath Vissapragada 4de06915b8 HBASE-23764: Switch to IP address for ZK ensemble #1106
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 15:13:54 -08:00
stack 62875a229a Revert "Revert "HBASE-23705 Add CellComparator to HFileContext (#1062)""
Revert of a revert, i.e., putting this patch back.

This reverts commit f4b0ad9f6a.
2020-01-27 07:40:56 -08:00
stack f4b0ad9f6a Revert "HBASE-23705 Add CellComparator to HFileContext (#1062)"
This reverts commit f817293fb4.
2020-01-26 20:49:03 -08:00
Jan Hentschel bfa4b0c4c1
HBASE-23686 Revert binary incompatible change in ByteRangeUtils and removed reflections in CommonFSUtils
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-24 20:28:01 +01:00
Michael Stack f817293fb4 HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use.  Backfill.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
 Adds a new compareRows with default implementation that takes a ByteBuffer.
 Needed by the index in a block encoder implementation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
 Adds implementation for meta of new compareRows method. Adds utility
 method for figuring comparator based off tablename.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
 Comparator is in context. Remove redundant handling.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
 Comparator is in context. Remove redundant handling. Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
 Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
 Cache context so can use it to get comparator to use later.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Cache cellcomparator to use. Javdoc on diff between HFileContext and
 HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
 Add CellComparator

M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
 Remove comparator caching. Get from context instead.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
 Skip a reflection if we can.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
 Javadoc. Removed unused filed.

Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 20:38:24 -08:00
Michael Stack ac852c2b66
HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 13:17:27 -08:00
Jan Hentschel c16569996b
HBASE-23625 Reduced number of Checkstyle violations in hbase-common
Signed-off-by: stack <stack@apache.org>
2020-01-20 22:24:01 +01:00
Duo Zhang 70c69ba765 HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
2020-01-18 20:52:29 +08:00
Josh Elser 1ea389a834 HBASE-23701 Try to converge automated checks around Category
We have a compile-time guarantee to either have a category array
of length zero or one because the category annotation is non-repeatable.

Also tries to clean up the checking for unit-test annotations.

Closes #1057

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-17 12:49:50 -05:00
Josh Elser bef1eb33f4 HBASE-23347 Allow custom authentication methods for RPCs
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.

Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.

Closes #884

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-01-16 11:04:08 -05:00
Josh Elser 0935ba4e16 HBASE-23695 Fail gracefully if no category is present
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Closes #1052
2020-01-16 10:24:53 -05:00
Bharath Vissapragada 31b4fbdee5 HBASE-23665: Split unit tests from TestTableName into a separate test-only class. (#1032)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-15 13:56:51 -08:00
stack da782e4233 Revert "HBASE-23055 Alter hbase:meta"
This reverts commit d64b0e3612.
2020-01-11 09:21:15 -08:00
stack d64b0e3612 HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
    was hardcoded ENABLED. Make it dynamic. State is now kept in
    current active Master. It is transient so falls back to default
    if Master crashes. Add to registry a getMetaTableState
    which reads mirrored state from zookeeper (NOT from Master and
    defaults ENABLED if no implementation or error fetching state).
    hbase:meta schema will be bootstrapped from the filesystem.
    Changes to filesystem schema are atomic so we should be ok if
    Master fails mid-edit (TBD). Undoes a bunch of guards that
    prevented our being able to edit hbase:meta.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     Add fetching of hbase:meta table state from registry. Adds cache of
     tablestates w/ a ttl of 1 second (adjustable).

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
      Add querying registry for hbase:meta table state.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
      Add querying of mirrored table state for hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta. Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-09 09:52:21 -08:00
Jan Hentschel 4f7de9e27e
HBASE-23645 Fixed remaining Checkstyle violations in hbase-common tests
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-07 09:07:56 +01:00
Nick Dimiduk f98085e44e HBASE-23333 Include Call.toShortString() in sendCall exceptions 2020-01-03 12:23:37 -08:00
Guanghao Zhang a693a8fd95 HBASE-23286 Improve MTTR: Split WAL to HFile (#820)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-03 16:39:06 +08:00
Duo Zhang bee7f4e08c HBASE-23624 Add a tool to dump the procedure info in HFile (#975)
Signed-off-by: stack <stack@apache.org>
2020-01-02 11:32:29 +08:00
Viraj Jasani 17652a7b32
HBASE-23590 : Update maxStoreFileRefCount to maxCompactedStoreFileRefCount for auto region recovery based on old reader references
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-01-01 22:50:37 +05:30
Jan Hentschel 71a0c30caf
HBASE-23626 Reduced number of Checkstyle violations in tests in hbase-common
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-29 22:42:41 +01:00
Jan Hentschel f3e11d7691
HBASE-23622 Reduced the number of Checkstyle violations in hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-28 20:05:50 +01:00
Jan Hentschel 2382d009a6
HBASE-23621 Reduced the number of Checkstyle violations in tests of hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-28 18:51:33 +01:00
belugabehr d6eec74273 HBASE-23556: Minor ChoreService Cleanup (#927) 2019-12-10 16:21:11 -08:00
chenxu14 46c090ce45 HBASE-23293 [REPLICATION] make ship edits timeout configurable (#882)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2019-11-28 11:01:18 +08:00
Nick Dimiduk d742112ed9
HBASE-23259: Populate master address end points in cluster/rs configs (#807) (#857)
All the clients need to know the master RPC end points while using master
based registry for creating cluster connections. This patch amends the
test cluster utility to populate these configs in the base configuration
object used to spin up the cluster.

The config key added here ("hbase.master.addrs") is used in the subsequent
patches for HBASE-18095.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2019-11-21 09:43:53 -08:00
Sean Busbey baf8849050 HBASE-19450 Addendum Limit logging of chore execution time at INFO to once per 5 minutes.
* Ensure MovingAverage related classes are IA.Private
* Move trace logging into MovingAverage class

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-16 09:40:20 -06:00
Reid Chan f5db86ca8f HBASE-19450 Add log about average execution time for ScheduledChore
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-11-14 22:07:02 -06:00
Mingliang Liu d22e6f55d8 HBASE-23283 Provide clear and consistent logging about the period of enabled chores
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-11-14 20:47:01 -06:00
Geoffrey Jacoby da06aa7f1f HBASE-23251 - Add Column Family and Table Names to HFileContext and use in HFileWriterImpl logging (#796)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Zheng Hu <openinx@gmail.com>
2019-11-11 16:22:09 +08:00
Viraj Jasani 29c27e3d04
HBASE-23212 Dynamically reload configs for Region Recovery chore (#802)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-11-06 15:54:57 -08:00
chenxu14 26663585de HBASE-23184 The HeapAllocation in WebUI is not accurate (#730)
Signed-off-by: stack <stack@apache.org>
2019-10-31 17:57:52 +08:00
Michael Stack 471538ca9e HBASE-23221 Polish the WAL interface after HBASE-23181 (#774)
Removes the closeRegion flag added by HBASE-23181 and instead
relies on reading meta WALEdit content. Modified how qualifier is
written when the meta WALEdit is for a RegionEventDescriptor
so the 'type' is added to the qualifer so can figure type
w/o having to deserialize protobuf value content: e.g.
HBASE::REGION_EVENT::REGION_CLOSE

Added doc on WALEdit and tried to formalize the 'meta' WALEdit
type and how it works. Needs complete redo in part as suggested
by HBASE-8457. Meantime, some doc and cleanup.

Also changed the LogRoller constructor to remove redundant param.
Because of constructor change, need to change also
TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL &
WALPerformanceEvaluation.java

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-10-30 22:10:51 -07:00
Duo Zhang 3dba799837 HBASE-23181 Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us" (#753)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-10-26 20:47:29 +08:00
Viraj Jasani 33e8156ebc HBASE-22460 : Reopen regions with very high Store Ref Counts (#750)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2019-10-24 21:20:24 +05:30
chenxu14 b0b7e5f5b8 HBASE-23107 Avoid temp byte array creation when doing cacheDataOnWrite (#678)
Signed-off-by: huzheng <openinx@gmail.com>
Signed-off-by: stack <stack@apache.org>
2019-10-17 09:47:14 +08:00
Michael Stack d8ad2a0bb5 HBASE-23129 Move core to use hbase-thirdparty-3.1.1 (#695)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-10-11 14:32:19 +02:00
haxiaolin fff0f33c5a HBASE-22767 System table RIT STUCK if their RSGroup has no highest version RSes
Signed-off-by: stack <stack@apache.org>
2019-10-08 16:32:07 -07:00
Istvan Toth e3078aa5aa HBASE-23119 ArrayIndexOutOfBoundsException in PrivateCellUtil#qualifierStartsWith (#688)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2019-10-04 15:14:30 +02:00
stack 61bc57f525 Revert " HBASE-23055 Alter hbase:meta (#655)"
"Too radical for branch-2"

This reverts commit ff217d4269.
2019-10-01 08:34:09 -07:00
Michael Stack ff217d4269
HBASE-23055 Alter hbase:meta (#655)
Make it so hbase:meta can be altered. TableState for hbase:meta
    is kept in Master. State is in-memory transient so if  Master
    fails, hbase:meta is ENABLED again. hbase:meta schema will be
    bootstrapped from the filesystem. Changes to filesystem schema
    are atomic so we should be ok if Master fails mid-edit (TBD)
    Undoes a bunch of guards that prevented our being able to edit
    hbase:meta. At minimmum, need to add in a bunch of WARNING.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     For table state, go to master rather than go to meta direct. Going
     to meta won't work for hbase;meta state. Puts load on Master.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
     Change isTableDisabled/Enabled implementation to ask the Master instead.
     This will give the Master's TableStateManager's opinion rather than
     client figuring it for themselves reading meta table direct.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
     TODO: Cleanup in here. Go to master for state, not to meta.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
     Logging cleanup.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta.
     Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.


   Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
2019-09-30 09:27:41 -07:00
chenxu14 be932487e8 HBASE-22965 RS Crash due to DBE reference to an reused ByteBuff (#603)
Signed-off-by: huzheng <openinx@gmail.com>
2019-09-30 09:20:35 +08:00
Sanjeet Nishad 6def152819 HBASE-23038 Provide consistent and clear logging about disabling chores
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit b45c0d0efa)
2019-09-27 20:13:18 -05:00
Viraj Jasani faed3e457d HBASE-23024 Replace initcause with Constructor arg (#627)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-09-17 13:40:07 +02:00
Viraj Jasani 63568854b6 HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity (#618) 2019-09-13 10:37:22 +05:30
chenxu14 4282d70212 HBASE-22802 Avoid temp ByteBuffer allocation in FileIOEngine#read (#583) 2019-09-09 17:44:49 +08:00
Josh Elser 2feca0abde HBASE-22701 Disable the DynamicClassLoader when it fails to initialize
Signed-off-by: Ankit Singhal <ankitsinghal59@gmail.com>
2019-09-04 18:22:21 -04:00
康智冬 576e688cec HBASE-22962 Fix typo in javadoc description (#569)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-09-01 20:16:05 +08:00
linkaline 84b4a61e88 HBASE-22881 Fix non-daemon threads in hbase server implementation (#512) (#536)
Signed-off-by: stack <stack@apache.org>
2019-08-28 09:48:27 -07:00
ravowlga123 c60748cba1 HBASE-22893 Change the comment in HBaseClassTestRule to reflect change in default test timeouts (#545)
Signed-off-by: Sakthi <sakthi@apache.org>
2019-08-27 01:36:26 -07:00
openinx 53522f36cc HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/restoring snapshot (addendum - use the old config key) (#517) 2019-08-22 11:53:46 +08:00
Huon Wilson 8cbe20c382 HBASE-22841 Add more factory functions to TimeRange
These functions make it easier to possible to use
`org.apache.hadoop.hbase.client.Table.CheckAndMutateBuilder#timeRange`
with more interesting ranges, without being forced to use the
deprecated constructors.

Signed-off-by: huzheng <openinx@gmail.com>
2019-08-17 10:12:48 +08:00
openinx 6f9db4660a HBASE-22810 Initialize an separate ThreadPoolExecutor for taking/restoring snapshot (#486) 2019-08-15 11:22:08 +08:00
stack 225c6cd5bc HBASE-22777 Add a multi-region merge (for fixing overlaps)
Makes MergeTableRegionsProcedure do more than just two regions at a
time. Compatible as MTRP was done considering one day it'd do more than
two at a time.

Changes hardcoded assumption that merge parent regions are named
mergeA and mergeB in a column on the resultant region. Instead
can have N columns on the merged region, one for each parent
merged. Column qualifiers all being with 'merge'.

Most of code below is undoing the assumption that there are two
parents on a merge only.
2019-08-07 16:58:06 -07:00
Jan Hentschel 3971773189
HBASE-22790 Added deprecation versions for fields in ByteBuffAllocator
Signed-off-by: stack <stack@apache.org>
2019-08-05 09:01:03 +02:00
Braavos 89e2cc6281
HBASE-22677 Add unit tests for org.apache.hadoop.hbase.util.ByteRangeUtils and Classes
Co-authored-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-08-05 03:32:32 +02:00
stack 4565e9f43b HBASE-22723 Have CatalogJanitor report holes and overlaps; i.e. problems it sees when doing its regular scan of hbase:meta
Refactor of CatalogJanitor so it generates a
Report on the state of hbase:meta when it runs. Also
refactor so CJ runs even if RIT (previous it would
punt on running if RIT) so it can generate a 'Report'
on the interval regardless. If RIT, it just doesn't
go on to do the merge/split GC as it used to.

If report finds an issue, dump as a WARN message
to the master log.

Follow-on is to make the Report actionable/available
for the Master to pull when it goes to draw the hbck
UI page (could also consider shipping the Report as
part of ClusterMetrics?)

Adds new, fatter Visitor to CJ, one that generates
Report on each run keeping around more findings as
it runs.

Moved some methods around so class reads better;
previous methods were randomly ordered in the class.

M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
 Make a few handy methods public.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
 Add utility as defaults on the Inteface; i.e. is this the first region
 in table, is it last, does a passed region come next, or does passed
 region overlap this region (added tests for this new stuff).

M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
 Bugfix... handle case where buffer passed is null.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
 Lots of change, reorg., but mostly adding consistency checking
 to the visitor used scanning hbase:meta on a period and the
 generation of a Report on what the scan has found traversing
 hbase:meta. Added a main so could try the CatalogJanitor against
 a running cluster.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorCluster.java
 Fat ugly test for CatalogJanitor consistency checking.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
 Add tests for new functionality in RI.

M hbase-shell/src/main/ruby/hbase/table.rb
 Bug fix for case where meta has a null regioninfo; scan was aborting.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-07-29 16:07:10 -07:00
Wellington Chevreuil f160dd3c10 HBASE-21773 rowcounter utility should respond to pleas for help
Signed-off-by: stack <stack@apache.org>
2019-07-26 21:29:48 +01:00
Viraj Jasani 3a72ccefbd
HBASE-22648 Snapshot TTL (#371)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-client/src/main/java/org/apache/hadoop/hbase/client/Admin.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
2019-07-22 15:08:20 -07:00
openinx 6fb03e63b6 HBASE-22663 The HeapAllocationRatio in WebUI is not accurate because all of the heap allocation will happen in another separated allocator named HEAP (#365) 2019-07-20 23:36:21 +08:00
毛蛤丝 dfebcbec38 HBASE-22700:incorrect timeout in recommended ZooKeeper configuration
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-07-17 22:12:07 +08:00
syedmurtazahassan 955e643e1d
HBASE-22586 Javadoc Warnings related to @param tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 16:56:53 +02:00
Peter Somogyi ed44b60cb1 HBASE-22651 ErrorProne issue in TestByteBufferArray
Signed-off-by: Michael Stack <stack@apache.org>
2019-07-02 16:37:46 +02:00
syedmurtazahassan 8fa7800389 HBASE-22572 Javadoc Warnings: @link reference not found (#306)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2019-07-01 21:18:11 -07:00
openinx 345f80ea5f HBASE-20060 Add details of off heap memstore into book. (#334) 2019-06-27 16:54:48 +08:00
huzheng 6a3aa789d8 HBASE-22612 Address the final overview reviewing comments of HBASE-21879 2019-06-26 15:40:23 +08:00
openinx af0e23c359 HBASE-22547 Align the config keys and add document for offheap read in HBase Book. (#301) 2019-06-25 22:25:30 +08:00
openinx d34c4a0481 HBASE-22598 Deprecated the hbase.ipc.server.reservoir.initial.buffer.size & hbase.ipc.server.reservoir.initial.max for HBase2.x compatibility (#318) 2019-06-25 22:22:29 +08:00
openinx 3199175932 HBASE-22491 Separate the heap HFileBlock and offheap HFileBlock because the heap block won't need refCnt and save into prevBlocks list before shipping (#268) 2019-06-25 22:22:16 +08:00
huzheng 5ff317fe7b HBASE-22504 Optimize the MultiByteBuff#get(ByteBuffer, offset, len) 2019-06-25 22:22:04 +08:00
openinx 4dd728dfff HBASE-22483 It's better to use 65KB as the default buffer size in ByteBuffAllocator (#279) 2019-06-25 22:20:58 +08:00
huzheng 9e5fc2b379 HBASE-22435 Add a UT to address the HFileBlock#heapSize() in TestHeapSize 2019-06-25 22:14:57 +08:00
huzheng b00360731a HBASE-22412 Improve the metrics in ByteBuffAllocator 2019-06-25 22:14:50 +08:00
huzheng d7009d258c HBASE-21921 Notify users if the ByteBufAllocator is always allocating ByteBuffers from heap which means the increacing GC pressure 2019-06-25 22:14:36 +08:00
huzheng 39a6a357ff HBASE-21937 Make the Compression#decompress can accept ByteBuff as input 2019-06-25 22:11:51 +08:00
huzheng fc460136e0 HBASE-21957 Unify refCount of BucketEntry and refCount of hbase.nio.ByteBuff into one 2019-06-25 22:07:16 +08:00
huzheng bd8260c07e HBASE-22159 ByteBufferIOEngine should support write off-heap ByteBuff to the bufferArray 2019-06-25 21:28:02 +08:00
huzheng 081dc03e12 HBASE-22127 Ensure that the block cached in the LRUBlockCache offheap is allocated from heap 2019-06-25 21:23:54 +08:00
huzheng 0d294d23a9 HBASE-22005 Use ByteBuff's refcnt to track the life cycle of data block 2019-06-25 21:16:48 +08:00
huzheng c19592a971 HBASE-21916 Abstract an ByteBuffAllocator to allocate/free ByteBuffer in ByteBufferPool 2019-06-25 21:12:37 +08:00
Duo Zhang 974e1086d7 HBASE-22617 Recovered WAL directories not getting cleaned up (#330)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-06-25 17:32:39 +08:00
Sean Busbey 23a2f8abab HBASE-22615 Make TestChoreService more robust to timing
* phrase fudge factor "deltas" in terms of the original period
* increase the delta allowed for chore timing from 5% to 20%
* improve some assertions

Closes #328

Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Sakthi <sakthivel.azhaku@gmail.com>
(cherry picked from commit 9aee88e03a)
2019-06-22 00:29:04 -05:00
Mingliang Liu bc737d1ed9
HBASE-21284 Default limits for PressureAwareCompactionThroughputController are too low
This is a forward port HBASE-21000 from branch-1.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-06-20 12:44:14 -07:00
Jan Hentschel 6cff25cc23
HBASE-22344 Documented the deprecation of public and limited private APIs (#208)
Signed-off-by: stack <stack@apache.org>
2019-06-17 22:29:51 +02:00
Murtaza Hassan a2b22a6807
HBASE-22565 Javadoc Warnings: @see cannot be used in inline documentation 2019-06-13 23:14:58 +02:00
Sakthi 38139350a3 HBASE-22458: TestClassFinder fails when run on JDK11
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 0445186f1a)
2019-06-13 09:02:20 -05:00
Sakthi 35d3612f66 HBASE-22534 TestCellUtil fails when run on JDK11
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 9b23e1d00c)
2019-06-04 14:51:20 -05:00
Duo Zhang 1a2632b796 HBASE-22316 Record the stack trace for current thread in FutureUtils.get 2019-05-30 11:27:43 +08:00
Wellington Chevreuil ea41133e44 HBASE-22496 UnsafeAccess.unsafeCopy should not copy more than UNSAFE_COPY_THRESHOLD on each iteration
Change-Id: I259bc54a0a5b3474d3c455639c3e9cb1e95c8438

Signed-off-by: huzheng <openinx@gmail.com>
2019-05-30 10:21:43 +08:00
Artem Ervits e9e680f3d1 HBASE-22469 replace md5 checksum in saveVersion script with sha512
Signed-off-by: Josh Elser <elserj@apache.org>
2019-05-28 15:06:37 -04:00
Sean Busbey 16f18d5c2d HBASE-22148 Provide a LimitedPrivate(COPROC) API for setting Cell timestamp without copying.
Signed-off-by: Josh Elser <elserj@apache.org>
2019-05-28 12:18:48 -05:00
Andrew Purtell 2c55bd9344
HBASE-22449 https everywhere in Maven metadata (#247) 2019-05-21 12:38:42 -07:00
OrDTesters b585dc42ba HBASE-21467 Fix flaky test TestCoprocessorClassLoader.testCleanupOldJars 2019-05-06 09:45:51 -07:00
Peter Somogyi 32462d5690 HBASE-22340 Corrupt KeyValue is silently ignored (#207) 2019-05-02 11:22:05 +02:00
Sakthi 018dadc3c1 HBASE-22054: Space Quota: Compaction is not working for super user in case of NO_WRITES_COMPACTIONS
Signed-off-by: Josh Elser <elserj@apache.org>
2019-04-30 13:41:08 -04:00
Sean Busbey 4862a596ef HBASE-22083 move eclipse settings into a profile.
Signed-off-by: stack <stack@apache.org>

 Conflicts:
	hbase-backup/pom.xml
	hbase-hadoop-compat/pom.xml
	hbase-protocol/pom.xml
2019-04-25 14:38:38 -05:00
Duo Zhang b117d5b4c2 HBASE-22307 Deprecated Preemptive Fail Fast 2019-04-25 14:20:49 +08:00
李小保 1bd5b5cf7b HBASE-22250 The same constants used in many places should be placed in constant classes
Signed-off-by: stack <stack@apache.org>
2019-04-23 21:24:46 -07:00
zouxiangwen 4138fce370 HBASE-22292 PreemptiveFastFailInterceptor clean repeatedFailuresMap issue
Signed-off-by: stack <stack@apache.org>
2019-04-23 08:02:48 -07:00
Jan Hentschel c40e6e2339 HBASE-22231 Removed unused and '*' import 2019-04-23 12:53:52 +02:00
Ben Manes cadf321d53
HBASE-15560 W-TinyLFU based BlockCache
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-04-17 11:10:55 -07:00
Andrew Purtell f3532e4c26
HBASE-22235 OperationStatus.{SUCCESS|FAILURE|NOT_RUN} are not visible to 3rd party coprocessors 2019-04-16 13:19:56 -07:00