Commit Graph

1615 Commits

Author SHA1 Message Date
Michael Stack 56f1dbd116
HBASE-25425 Some notes on RawCell (#2797)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-21 08:08:22 -08:00
Duo Zhang f098461a55 HBASE-25370 Addendum fix checkstyle issue and dependencies 2020-12-12 21:08:51 +08:00
Adam c62c18dca7
HBASE-25370 Fix flaky test TestClassFinder#testClassFinderDefaultsToOwnPackage (#2740)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-10 22:28:21 +08:00
shahrs87 5016219d3c
HBASE-25328 : Add builder method to create Tags
Closes #2707

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-08 18:15:45 +05:30
Andrew Purtell 679dd7b8f9
HBASE-25292 Improve InetSocketAddress usage discipline (#2669)
Network identities should be bound late. Remote addresses should be
resolved at the last possible moment, just before connect(). Network
identity mappings can change, so our code should not inappropriately
cache them. Otherwise we might miss a change and fail to operate normally.

Revert "HBASE-14544 Allow HConnectionImpl to not refresh the dns on errors"
Removes hbase.resolve.hostnames.on.failure and related code. We always
resolve hostnames, as late as possible.

Preserve InetSocketAddress caching per RPC connection. Avoids potential
lookups per Call.

Replace InetSocketAddress with Address where used as a map key. If we want
to key by hostname and/or resolved address we should be explicit about it.
Using Address chooses mapping by hostname and port only.

Add metrics for potential nameservice resolution attempts, whenever an
InetSocketAddress is instantiated for connect; and metrics for failed
resolution, whenever InetSocketAddress#isUnresolved on the new instance
is true.

* Use ServerName directly to build a stub key

* Resolve and cache ISA on a RpcChannel as late as possible, at first call

* Remove now invalid unit test TestCIBadHostname

We resolve DNS at the latest possible time, at first call, and do not
resolve hostnames for creating stubs at all, so this unit test cannot
work now.

Reviewed-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-28 21:01:22 +08:00
Mate Szalay-Beko 451a4b06b1 HBASE-25263 Various improvements of column family encryption
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:

1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.

Smaller issues fixed by this commit:

2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.

3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.

4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.

5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.

closes #2676

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-26 16:44:34 +01:00
Andrew Purtell 1726160839
HBASE-24640 Purge use of VisibleForTesting (#2695)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:10:12 -08:00
ramkrish86 f221d11227
HBASE-25050 - We initialize Filesystems more than once. (#2419)
* HBASE-25050 - We initialize Filesystems more than once.

* Ensuring that calling the FS#get() will only ensure FS init.

* Fix for testfailures. We should pass the entire path and no the scheme
alone

* Cases where we don't have a scheme for the URI

* Address review comments

* Add some comments on why FS#get(URI, conf) is getting used

* Adding the comment as per Sean's review

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-11-24 15:31:04 +05:30
ramkrish86 57d9cae480
HBASE-25187 Improve SizeCachedKV variants initialization (#2582)
* HBASE-25187 Improve SizeCachedKV variants initialization

* HBASE-25187 Improve SizeCachedKV variants initialization

* The BBKeyValue also can be optimized

* Change for SizeCachedKeyValue

* Addressing revew comments

* Fixing checkstyle and spot bugs comments

* Spot bug fix for hashCode

* Minor updates make the rowLen as short and some consturctor formatting

* Change two more places where there was a cast
2020-11-11 17:39:39 +05:30
Mate Szalay-Beko 6a5c928539
HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
Prior to this patch hbase always used the MD5 hash algorithm to store a hash for encryption keys.
This hash is needed to verify the secret key of the subject. (e.g. making
sure that the same secrey key is used during encrypted HFile read and write).
The MD5 algorithm is considered weak, and can not be used in some
(e.g. FIPS compliant) clusters.

In this patch we:
- add a config parameter to globally enable/disable column family encryption (def enabled)
- introduce a backward compatible way of specifying the hash algorithm.
  This enable us to use newer and more secure hash algorithms like SHA-384
  or SHA-512 (which are FIPS compliant).
- add a config parameter to fail if an hfile is encountered that uses a
  different hash algorithm than the one currently configured to ease validation after
  migrating key hash algorithms (def disabled)

Closes #2539

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2020-11-09 13:53:50 -06:00
gvprathyusha6 5c7432f4a9
HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Closes #2542

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-09 12:27:34 +05:30
WenFeiYi 0356e8efd1
HBASE-25240 gson format of RpcServer.logResponse is abnormal
Closes #2623

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-05 19:58:05 +05:30
Duo Zhang f37cd05c32
HBASE-25235 Cleanup the deprecated methods in TimeRange (#2616)
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-11-04 11:02:28 +08:00
Nick Dimiduk d790bdedde
HBASE-25167 Normalizer support for hot config reloading (#2523)
Wire up the `ConfigurationObserver` chain for
`RegionNormalizerManager`. The following configuration keys support
hot-reloading:
 * hbase.normalizer.throughput.max_bytes_per_sec
 * hbase.normalizer.split.enabled
 * hbase.normalizer.merge.enabled
 * hbase.normalizer.min.region.count
 * hbase.normalizer.merge.min_region_age.days
 * hbase.normalizer.merge.min_region_size.mb

Note that support for `hbase.normalizer.period` is not provided
here. Support would need to be implemented generally for the `Chore`
subsystem.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-10-30 10:41:56 -07:00
Nick Dimiduk 12d0397015 HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-10-29 13:37:23 -07:00
sanjeetnishad95 d8de24c311
HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into account configuration map. (#2484)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-23 08:42:00 +08:00
Jan Hentschel 6c8f5f30d5
HBASE-25196 Add deprecation documentation to HConstants (#2559)
Add the documentation when
HConstants#REPLICATION_DROP_ON_DELETED_TABLE_KEY was deprecated and when
it is expected to be removed.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-21 09:18:35 +02:00
Duo Zhang 92c3bcd9fb
HBASE-25164 Make ModifyTableProcedure support changing meta replica count (#2513)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-13 09:43:56 +08:00
niuyulin 3d1aaa6632
HBASE-25175 Remove the constructors of HBaseConfiguration (#2530)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Jan Hentschel <janh@apache.org>
2020-10-11 17:11:32 +02:00
Michael Stack 665a8767a0
HBASE-22976 [HBCK2] Add RecoveredEditsPlayer (#2504)
Make it so WALPlayer can replay recovered.edits files.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
 Allow for WAL files that do NOT have a startime in their name.
 Use the 'generic' WAL-filename parser instead of the one that
 used be local here. Implement support for 'startTime' filter.
 Previous was just not implemented.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
 Checkstyle.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
 Use the new general WAL name timestamp parser.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility for parsing timestamp from WAL filename.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
 Export attributes about the local recovered.edits test file
 so other tests can play with it.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-10-09 08:46:05 -07:00
Guanghao Zhang 8828643bb2
HBASE-25086 Refactor Replication: move the default ReplicationSinkService implementation out (#2444)
Signed-off-by: meiyi <myimeiyi@gmail.com>
2020-09-24 17:25:34 +08:00
Duo Zhang 0956c34658
HBASE-25066 Use FutureUtils.rethrow in AsyncTableResultScanner to better catch the stack trace (#2420)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-20 11:25:22 +08:00
Viraj Jasani 35d59dc570
HBASE-24528 : BalancerDecision queue implementation in HMaster with Admin API
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto

Closes #2261

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-09-16 20:56:54 +05:30
XinSun ca96f96461
HBASE-25014 ScheduledChore is never triggered when initalDelay > 1.5*period (#2395)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-16 09:00:21 +08:00
XinSun c2721c8235
HBASE-24759 Refuse to update configuration of default group (#2126)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-04 08:19:39 +08:00
Viraj Jasani 1c568ec95d
HBASE-24968 : Move META_COMPARATOR to subclass MetaCellComparator
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
  static initializer by moving META_COMPARATOR to subclass MetaCellComparator

Closes #2329

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-30 22:03:57 +05:30
Nick Dimiduk 2e8120a63f HBASE-24583 Normalizer can't actually merge empty regions...
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-08-12 10:04:45 -07:00
Viraj Jasani ea130249ae
HBASE-24750 : Adding default UncaughtExceptionHandler for Thread factories (ADDENDUM)
Closes #2231
2020-08-11 17:18:47 +05:30
Duo Zhang 726756a8b9
HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762)
Signed-off-by: stack <stack@apache.org>
2020-08-10 09:11:31 +08:00
Viraj Jasani 0b604d921a
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
2020-08-07 20:24:36 +05:30
bsglz 485e0d2fa4
HBASE-24694 Support flush a single column family of table (#2179)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-07 11:28:53 +01:00
niuyulin f710d2d654
HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-06 14:50:32 -07:00
Sean Busbey 86ebbdd8a2
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe
* refactor how we use connection and async connection to rely on their access methods
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-01 23:19:07 -05:00
Michael Stack 345b77a4ca
HBASE-24632 Enable procedure-based log splitting as default in hbase3 Add deprecation of 'classic' zk-based WAL splitter. (#2156)
Also fix three bugs:

 * We were trying to delete non-empty directory; weren't doing
 accounting for meta WALs where meta had moved off the server
 (successfully)
 * We were deleting split WALs rather than archiving them.
 * We were not handling corrupt files.

Deprecations and removal of tests of old system.

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-07-29 07:37:28 -07:00
Viraj Jasani 09e7ccd42d
HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold
Closes #2127

Signed-off-by: stack <stack@apache.org>
2020-07-24 13:16:02 +05:30
Duo Zhang 0b85729da4
HBASE-24762 Purge protobuf java 2.5.0 dependency (#2128)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-07-24 11:48:35 +08:00
Viraj Jasani f35c5eaadd
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2109)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-21 20:24:13 +05:30
bsglz fccd03ffa2
HBASE-24555 Correct the description of hbase.hregion.max.filesize (#1895)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-21 11:04:42 +01:00
Peter Somogyi ea96743dd2
HBASE-24710 Incorrect checksum calculation in saveVersion.sh (#2056)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-07-21 10:40:22 +02:00
Viraj Jasani f2c087caeb
Revert "HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)"
This reverts commit 16f306b4a1.

TestAdminShell and TestThriftHBaseServiceHandler are failing consistently
2020-07-21 13:50:27 +05:30
Viraj Jasani 16f306b4a1
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-17 19:16:22 +05:30
Bo Cui d132c4b53d
HBASE-24693 regioninfo#isLast() should point to EMPTY_END_ROW (readability improvement)
Closes #2051

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-14 15:41:44 +05:30
bsglz 3bd54217a0
HBASE-24664 Some changing of split region by overall region size rath… (#2011)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-07-13 10:00:48 +01:00
pengmq1 be57e40f36
HBASE-24289 Heterogeneous Storage for Date Tiered Compaction (#1730)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-30 15:10:04 +08:00
Duo Zhang 46bf8944fd
HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU (#1987)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-06-28 23:26:05 +08:00
Michael Stack 0197438564
HBASE-24638 Edit doc on (offheap) memory management (#1978) 2020-06-26 09:11:04 -07:00
Bharath Vissapragada 84e246f9b1
HBASE-24603: Make Zookeeper sync() call synchronous (#1945)
Writing a test for this is tricky. There is enough coverage for
functional tests. Only concern is performance, but there is enough
logging for it to detect timed out/badly performing sync calls.

Additionally, this patch decouples the ZK event processing into it's
own thread rather than doing it in the EventThread's context. That
avoids deadlocks and stalls of the event thread.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-06-25 13:27:38 -07:00
XinSun 4b89ee24f3
HBASE-24431 RSGroupInfo add configuration map to store something extra (#1782)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-24 18:46:09 +08:00
Michael Stack 500a1df379 HBASE-23055 Alter hbase:meta (#1956)
Addendum to fix illegal removal of unused constant w/o
a deprecation cycle.
2020-06-22 20:25:15 -07:00
Anoop Sam John 4005e42a62
HBASE-24189 WALSplit recreates region dirs for deleted table with recovered edits data. 2020-06-11 21:27:21 +05:30
Andrew Purtell 474d200daa
HBASE-24367 ScheduledChore log elapsed timespan in a human-friendly format (#1749)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-06-09 09:55:20 -07:00
XinSun 59031ab661
HBASE-24359 Optionally ignore edits for deleted CFs for replication (#1722)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-04 17:52:52 +08:00
niuyulin f71f1cdfa0
HBASE-24412 Canary support check only one column family per RegionTask (#1817)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-03 09:15:10 +08:00
Nick Dimiduk cb5c0d2782 HBASE-24418 Consolidate Normalizer implementations
Simplify our Normalizer story to have just a single, configurable
implementation.

* fold the features of `MergeNormalizer` into
  `SimpleRegionNormalizer`, removing the intermediate abstract class.
* configuration keys for merge-only features now share a common
  structure.
* add configuration to selectively disable normalizer split/merge
  operations.
* `RegionNormalizer` now extends `Configurable` instead of creating a
  new instance of `HBaseConfiguration` or snooping one off of other
  fields.
* avoid the extra RPCs by using `MasterServices` instead of
  `MasterRpcServices`.
* boost test coverage of all the various flags and feature
  combinations.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-06-02 11:38:39 -07:00
Bharath Vissapragada 4f49a96258
HBASE-24477: Move ConfigurationObserver and related classes to hbase-common (#1815)
This utility is useful for any module that wants to detect
dynamic config changes. Having it to hbase-common makes it
accessible to all the other modules.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-31 11:18:16 -07:00
bsglz 7b7e25a50b
HBASE-24451 Remove the HasThread because the related bug had been fixed since jdk7 (#1797)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-29 17:33:18 +05:30
Bharath Vissapragada fe1fc25fba
HBASE-24449: Generate version.h include file during compilation. (#1794)
This is needed for the hbase-native-client to compile. Also ships
the includes in the assembly tar.

Signed-off-by: Marc <phrocker@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-05-28 15:26:53 -07:00
Duo Zhang c303f9d329
HBASE-24408 Introduce a general 'local region' to store data on master (#1753)
Signed-off-by: stack <stack@apache.org>
2020-05-23 15:59:51 +08:00
Viraj Jasani e1d7dc82ea
HBASE-24405 : Document hbase:slowlog related operations (#1747)
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-05-22 18:19:57 +05:30
Viraj Jasani 86a2692dc4
HBASE-23938 : System table hbase:slowlog to store complete slow/large… (#1681)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-05-20 15:10:29 +05:30
Nick Dimiduk 7e295e767f HBASE-24271 Set values in `conf/hbase-site.xml` that enable running on `LocalFileSystem` out of the box
Simplify the new user experience shipping a configuration that enables
a fresh checkout or tarball distribution to run in standalone mode
without direct user configuration. This change restores the behavior
we had when running on Hadoop 2.8 and earlier.

Patch for master includes an update to the book. This change will be
omitted when backporting to earlier branches.

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-05-14 13:18:43 -07:00
Nick Dimiduk 8148c5858b Revert "HBASE-24086 Disable output stream capability enforcement when running on LocalFileSystem"
This reverts commit 10cbb3fb73.
2020-05-14 13:18:43 -07:00
Mingliang Liu a40a0322a7
HBASE-23969 Meta browser should show all `info` columns (#1485)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-13 11:50:53 -07:00
Sambit Mohapatra c4e892e3f1
HBASE-23832 Old config hbase.hstore.compactionThreshold is ignored
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-05-12 23:19:32 +05:30
Duo Zhang 8601416ee8
HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for modules other than hbase-assembly (#1640)
Signed-off-by: stack <stack@apache.org>
2020-05-12 12:03:30 +08:00
shahrs87 ff85daf1e2
[HBASE-24190] Make kerberos value of hbase.security.authentication property case insensitive (#1687)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-11 10:54:45 +05:30
Jan Hentschel 03e5a1402d
HBASE-24305 Removed deprecations in ServerName (#1666)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-05-08 09:41:55 +02:00
Duo Zhang c1cb22f0b3
HBASE-24265 Remove hedged rpc call support, implement the logic in MaterRegistry … (#1593)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-05-06 15:55:26 +08:00
Michael Stack 33eadb5bb1 HBASE-24307 [Flakey Tests] krb server for secure thrift tests throws BindException (#1656)
Move the random free port generate back into hbasecommontestingutility
  from hbasetestingutility.

  Add a create simple kdc server utility that will start a kdc server and
  if a bindexception, create a new one on a new random port in hbase-common.

  Add new BoundSocketMaker helpful when trying to manufacture
  BindExceptions because of port clash.

  Change thrift and http kdc tests to use this new utility (removes
  code duplication around kdc server setup).
2020-05-05 21:36:59 -07:00
Yu Wang 3d96007c37
HBASE-24137:The max merge count of metafixer may be remind in hbase-site.xml (#1478) 2020-05-05 16:14:14 -07:00
Andrew Purtell 404849ab36
HBASE-24322 UnsafeAvailChecker should also check that required methods are available (#1650)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-05-05 12:07:56 -07:00
Duo Zhang a2e373dc32 HBASE-24281 Remove some stale hadoop.version properties in the pom for sub modules (#1602)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-29 18:24:52 +08:00
Duo Zhang a51a1f4d10 Revert "HBASE-24181 Remove some stale hadoop.version properties in the pom for sub modules (#1602)"
This reverts commit af40bff8be.
2020-04-29 18:24:41 +08:00
Duo Zhang af40bff8be
HBASE-24181 Remove some stale hadoop.version properties in the pom for sub modules (#1602)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-29 18:02:07 +08:00
Duo Zhang 9f52e6b725
HBASE-24249 Move code in FSHDFSUtils to FSUtils and mark related clas… (#1586)
Signed-off-by: stack <stack@apache.org>
2020-04-29 10:44:34 +08:00
Viraj Jasani 4ed15e51d9
HBASE-24024 : Reject multi() requests with rows higher than threshold (#1560)
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-04-22 12:46:40 +05:30
Mark Robert Miller f48b50964c HBASE-24185: Junit tests do not behave well with System.exit or Runtime.halt or JVM exits in general. (#1540)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-20 22:20:45 -07:00
Viraj Jasani 24e15c08ed
HBASE-24194 : Refactor anonymous inner classes of BufferedEncodedSeeker to named inner classes (#1522)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-04-16 19:16:00 +05:30
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
Reid Chan 22675d8070
HBASE-24112 [RSGroup] Support renaming rsgroup (#1435)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-13 14:04:32 +08:00
Sean Busbey eb7df0498c
HBASE-23723 Ensure MOB compaction works in optimized mode after snapshot clone (#1446)
* Reorganize MOB compaction tests for more reuse.
* Add tests for mob compaction after snapshot clone operations
* note the original table used to write a given mob hfile and use that to find it later.

Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2020-04-08 14:16:43 -05:00
Nick Dimiduk 10cbb3fb73 HBASE-24086 Disable output stream capability enforcement when running on LocalFileSystem
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-03 09:17:03 -07:00
Duo Zhang f3ee9b8aa3
HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 17:05:31 +08:00
Duo Zhang 76e3db6720
HBASE-23981 PBType which is marked as IA.Public references protobuf Message (#1323)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 10:38:41 +08:00
Nick Dimiduk 059c189451 HBASE-23829 Get `-PrunSmallTests` passing on JDK11
Signed-off-by: stack <stack@apache.org>
2020-03-16 15:21:37 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Michael Stack e1e8f396ca HBASE-23851 Log networks and bind addresses when multicast publisher/listener enabled (#1173)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-10 11:00:00 -07:00
Abhinaba Sarkar b6eefcaeb7
HBASE-23788 ROW_INDEX_V1 encoder should consider the secondary index size with the encoded data size tracking (#1219)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-03-04 23:14:43 +05:30
Duo Zhang b3399ef365 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 3709f335a8 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Viraj Jasani b24ea32b3b
HBASE-22978 : Online slow response log
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-28 19:47:59 +05:30
Bharath Vissapragada c34dbc3c29 HBASE-23804: Fix default master addr hostname in master registry (#1137)
Master rpc server end point doesn't bind to localhost's
IP address by default. Instead, it looks up the hostname and
binds to the endpoint to which it resolves. MasterRegistry should
do the same when building the default server end point to talk to.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 5c88672d54 HBASE-23731: De-flake TestFromClientSide (#1091)
There were a couple of issues.

- There was a leak of a file descriptor for hbck lock file. This
was contributing to all the "ConnectionRefused" stack traces since
it was trying to renew lease for an already expired mini dfs cluster.
This issue was there for a while, just that we noticed it now.

- After upgrade to JUnit 4.13, it looks like the behavior for test
timeouts has changed. Earlier the timeout seems to have applied for
each parameterized run, but now it looks like it is applied across
all the runs.

This patch fixes both the issues.

Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 229b8aaaf3 HBASE-23647: Make MasterRegistry the default impl. (#1039)
Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 1aa6a4efb9 HBASE-23305: Master based registry implementation (#954)
Implements a master based registry for clients.

 - Supports hedged RPCs (fan out configured via configs).
 - Parameterized existing client tests to run with multiple registry combinations.
 - Added unit-test coverage for the new registry implementation.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Vladimir Rodionov b8194b4902 HBASE-22749 Distributed MOB compactions
- MOB compaction is now handled in-line with per-region compaction on region
  servers
- regions with mob data store per-hfile metadata about which mob hfiles are
  referenced
- admin requested major compaction will also rewrite MOB files; periodic RS
  initiated major compaction will not
- periodically a chore in the master will initiate a major compaction that
  will rewrite MOB values to ensure it happens. controlled by
  'hbase.mob.compaction.chore.period'. default is weekly
- control how many RS the chore requests major compaction on in parallel
  with 'hbase.mob.major.compaction.region.batch.size'. default is as
  parallel as possible.
- periodic chore in master will scan backing hfiles from regions to get the
  set of referenced mob hfiles and archive those that are no longer
  referenced. control period with 'hbase.master.mob.cleaner.period'
- Optionally, RS that are compacting mob files can limit write
  amplification by not rewriting values from mob hfiles over a certain size
  limit. opt-in by setting 'hbase.mob.compaction.type' to 'optimized'.
  control threshold by 'hbase.mob.compactions.max.file.size'.
  default is 1GiB
- Should smoothly integrate with existing MOB users via rolling upgrade.
  will delay old MOB file cleanup until per-region compaction has managed
  to compact each region at least once so that used mob hfile metadata can
  be gathered.
2020-02-19 16:06:38 -06:00
stack 9cc7a711e6 HBASE-23789 [Flakey Tests] ERROR [Time-limited test] balancer.HeterogeneousRegionCountCostFunction(199): cannot read rules file located at ' /tmp/hbase-balancer.rules ' 2020-02-04 16:40:50 -08:00
Mark Robert Miller 299b6bebc5
HBASE-23783: Address tests writing and reading SSL/Security files in a common location. (#1116)
This is causing me issues with parallel test runs.

Also allow setting the surefire reports and temp directories via command line.

Signed-off-by: stack <stack@apache.org>
2020-02-04 14:38:56 -08:00
Michael Stack 661abeb730 HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 10:28:19 -08:00
Bharath Vissapragada d9453203ae HBASE-23764: Switch to IP address for ZK ensemble
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 14:57:51 -08:00
Michael Stack 6cdc4b1f05
HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use.  Backfill.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
 Adds a new compareRows with default implementation that takes a ByteBuffer.
 Needed by the index in a block encoder implementation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
 Adds implementation for meta of new compareRows method. Adds utility
 method for figuring comparator based off tablename.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
 Comparator is in context. Remove redundant handling.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
 Comparator is in context. Remove redundant handling. Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
 Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
 Cache context so can use it to get comparator to use later.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Cache cellcomparator to use. Javdoc on diff between HFileContext and
 HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
 Add CellComparator

M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
 Remove comparator caching. Get from context instead.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
 Skip a reflection if we can.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
 Javadoc. Removed unused filed.

Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 20:00:21 -08:00
Michael Stack ba3463d9de HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 15:02:40 -08:00
Jan Hentschel 50e264485d
HBASE-23686 Revert binary incompatible change in ByteRangeUtils and removed reflections in CommonFSUtils
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-19 18:06:31 +01:00
Duo Zhang 167892ce64
HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
2020-01-18 20:30:47 +08:00
Josh Elser df8f80a819 HBASE-23701 Try to converge automated checks around Category
We have a compile-time guarantee to either have a category array
of length zero or one because the category annotation is non-repeatable.

Also tries to clean up the checking for unit-test annotations.

Closes #1057

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-17 12:39:25 -05:00
Josh Elser 04d789f1a7 HBASE-23347 Allow custom authentication methods for RPCs
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.

Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.

Closes #884

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-01-16 10:27:28 -05:00
Josh Elser edc53683c4 HBASE-23695 Fail gracefully if no category is present
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Closes #1052
2020-01-16 10:24:46 -05:00
Bharath Vissapragada fd05aabf02 HBASE-23665: Split unit tests from TestTableName into a separate test-only class. (#1032)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-15 09:16:28 -08:00
stack c8496bc694 Revert "HBASE-23055 Alter hbase:meta"
This reverts commit 9abdb7b5ae.
2020-01-11 09:17:17 -08:00
stack 9abdb7b5ae HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
    was hardcoded ENABLED. Make it dynamic. State is now kept in
    current active Master. It is transient so falls back to default
    if Master crashes. Add to registry a getMetaTableState
    which reads mirrored state from zookeeper (NOT from Master and
    defaults ENABLED if no implementation or error fetching state).
    hbase:meta schema will be bootstrapped from the filesystem.
    Changes to filesystem schema are atomic so we should be ok if
    Master fails mid-edit (TBD). Undoes a bunch of guards that
    prevented our being able to edit hbase:meta.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     Add fetching of hbase:meta table state from registry. Adds cache of
     tablestates w/ a ttl of 1 second (adjustable).

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
      Add querying registry for hbase:meta table state.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
      Add querying of mirrored table state for hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta. Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-09 16:25:16 -08:00
Aman Poonia 12b79a3907 HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-09 11:49:48 -08:00
Jan Hentschel 1eafba5603
HBASE-23645 Fixed remaining Checkstyle violations in hbase-common tests
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-07 08:54:04 +01:00
Nick Dimiduk af3bb1feca HBASE-23333 Include Call.toShortString() in sendCall exceptions 2020-01-03 11:08:55 -08:00
Duo Zhang 4cb952ce34
HBASE-23624 Add a tool to dump the procedure info in HFile (#975)
Signed-off-by: stack <stack@apache.org>
2020-01-02 11:23:55 +08:00
Jan Hentschel e0eb98c963
HBASE-23625 Reduced number of Checkstyle violations in hbase-common
Signed-off-by: stack <stack@apache.org>
2020-01-01 22:32:37 +01:00
Viraj Jasani e32dbe8ed2
HBASE-23590 : Update maxStoreFileRefCount to maxCompactedStoreFileRefCount for auto region recovery based on old reader references
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-01-01 22:16:54 +05:30
Jan Hentschel 923ba7763e
HBASE-23626 Reduced number of Checkstyle violations in tests in hbase-common
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-29 22:39:09 +01:00
Jan Hentschel 2ca1e46d74
HBASE-23622 Reduced the number of Checkstyle violations in hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-28 20:03:58 +01:00
Jan Hentschel 703ee60c90
HBASE-23621 Reduced the number of Checkstyle violations in tests of hbase-common
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2019-12-28 23:03:05 +05:30
Guanghao Zhang def9ac7c45
HBASE-23286 Improve MTTR: Split WAL to HFile (#820)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-12-27 15:59:23 +08:00
belugabehr 72ab1c1193 HBASE-23556: Minor ChoreService Cleanup (#927) 2019-12-10 14:20:11 -08:00
Viraj Jasani 142997c04e HBASE-23342 : Handle NPE while closing compressingStream (#877)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2019-12-02 10:08:42 +05:30
chenxu14 17468efb88 HBASE-23293 [REPLICATION] make ship edits timeout configurable (#825)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-11-26 19:05:53 +08:00
Bharath Vissapragada 834ccb4bf6 HBASE-23259: Populate master address end points in cluster/rs configs (#807)
All the clients need to know the master RPC end points while using master
based registry for creating cluster connections. This patch amends the
test cluster utility to populate these configs in the base configuration
object used to spin up the cluster.

The config key added here ("hbase.master.addrs") is used in the subsequent
patches for HBASE-18095.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2019-11-20 10:15:43 -08:00
belugabehr 33bedf8d4d HBASE-23308: Review of NullPointerExceptions (#836)
Signed-off-by: stack <stack@apache.org>
2019-11-19 13:51:11 -08:00
Sean Busbey b292ffd14d HBASE-19450 Addendum Limit logging of chore execution time at INFO to once per 5 minutes.
* Ensure MovingAverage related classes are IA.Private
* Move trace logging into MovingAverage class

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-16 09:38:44 -06:00
Reid Chan 67f9afc722 HBASE-19450 Add log about average execution time for ScheduledChore
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-11-14 21:50:08 -06:00
Mingliang Liu 73f3e1d4a4 HBASE-23283 Provide clear and consistent logging about the period of enabled chores
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-11-14 20:32:10 -06:00
Geoffrey Jacoby 77490f815a HBASE-23251 - Add Column Family and Table Names to HFileContext and use in HFileWriterImpl logging (#796)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Zheng Hu <openinx@gmail.com>
2019-11-11 16:18:43 +08:00
Viraj Jasani 10cc64a7d6 HBASE-23212 : Dynamically reload configs for Region Recovery chore (#773)
* HBASE-23212 : Dynamically reload configs for Region Recovery chore

* remove redundant volatile
2019-11-04 11:36:38 -08:00
Michael Stack e1b4a2a895 HBASE-23221 Polish the WAL interface after HBASE-23181 (#774)
Removes the closeRegion flag added by HBASE-23181 and instead
relies on reading meta WALEdit content. Modified how qualifier is
written when the meta WALEdit is for a RegionEventDescriptor
so the 'type' is added to the qualifer so can figure type
w/o having to deserialize protobuf value content: e.g.
HBASE::REGION_EVENT::REGION_CLOSE

Added doc on WALEdit and tried to formalize the 'meta' WALEdit
type and how it works. Needs complete redo in part as suggested
by HBASE-8457. Meantime, some doc and cleanup.

Also changed the LogRoller constructor to remove redundant param.
Because of constructor change, need to change also
TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL &
WALPerformanceEvaluation.java

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-10-31 18:52:04 +08:00
chenxu14 255be63e1d HBASE-23184 The HeapAllocation in WebUI is not accurate (#730)
Signed-off-by: stack <stack@apache.org>
2019-10-31 17:53:24 +08:00
Duo Zhang 7b5cd0152f
HBASE-23181 Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us" (#753)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-10-26 20:37:37 +08:00
Viraj Jasani 14dcf1d0c6 HBASE-22460 : Reopen regions with very high Store Ref Counts (#600)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2019-10-23 21:16:47 +05:30
Viraj Jasani 2ad62b0162 HBASE-22679 : Revamping CellUtil (#735)
* HBASE-22679 : Revamping CellUtil

* checkstyle fix

* incorporating review

* minor indentation
2019-10-21 23:04:21 -07:00
chenxu14 0043dfebc5 HBASE-23107 Avoid temp byte array creation when doing cacheDataOnWrite (#678)
Signed-off-by: huzheng <openinx@gmail.com>
Signed-off-by: stack <stack@apache.org>
2019-10-17 09:44:40 +08:00
chenxu14 1aee5f0552 HBASE-23083 Collect Executor status info periodically and report to (#664)
metrics system
Signed-off-by: stack <stack@apache.org>
2019-10-11 12:05:53 -07:00
Michael Stack abc38d80e4 HBASE-23129 Move core to use hbase-thirdparty-3.1.1 (#695)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-10-11 14:30:28 +02:00
haxiaolin 92bf07a98d HBASE-22767 System table RIT STUCK if their RSGroup has no highest version RSes
Signed-off-by: stack <stack@apache.org>
2019-10-08 16:31:22 -07:00
Istvan Toth 800c35a30e HBASE-23119 ArrayIndexOutOfBoundsException in PrivateCellUtil#qualifierStartsWith (#688)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2019-10-04 15:13:03 +02:00
stack 2ebdcbc3b0 Revert " HBASE-23055 Alter hbase:meta (#667)"
Revert because new feedback and requested survey of master usage
figuring table state.

This reverts commit 52176184df.
2019-10-01 09:24:10 -07:00
Michael Stack 52176184df
HBASE-23055 Alter hbase:meta (#667)
Make it so hbase:meta can be altered. TableState for hbase:meta
    is kept in Master. State is in-memory transient so if  Master
    fails, hbase:meta is ENABLED again. hbase:meta schema will be
    bootstrapped from the filesystem. Changes to filesystem schema
    are atomic so we should be ok if Master fails mid-edit (TBD)
    Undoes a bunch of guards that prevented our being able to edit
    hbase:meta. At minimmum, need to add in a bunch of WARNING.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     For table state, go to master rather than go to meta direct. Going
     to meta won't work for hbase;meta state. Puts load on Master.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
     Change isTableDisabled/Enabled implementation to ask the Master instead.
     This will give the Master's TableStateManager's opinion rather than
     client figuring it for themselves reading meta table direct.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
     TODO: Cleanup in here. Go to master for state, not to meta.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
     Logging cleanup.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta.
     Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

 Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
2019-09-30 09:29:51 -07:00
chenxu14 ce0fbc2af9 HBASE-22965 RS Crash due to DBE reference to an reused ByteBuff (#603)
Signed-off-by: huzheng <openinx@gmail.com>
2019-09-30 09:17:07 +08:00
Sanjeet Nishad b45c0d0efa HBASE-23038 Provide consistent and clear logging about disabling chores
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-09-27 15:01:15 -05:00
Viraj Jasani 0dbae8f0e1 HBASE-23024 Replace initcause with Constructor arg (#627)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-09-17 13:39:26 +02:00
Viraj Jasani 1dcc8ee50c HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity (#617) 2019-09-13 10:34:01 +05:30
Duo Zhang 936bb82908 Revert "HBASE-22695 Store the rsgroup of a table in table configuration (#426)"
This reverts commit eab7d3d502.
2019-09-10 11:14:17 +08:00
Duo Zhang 64581fa33b Revert "HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)"
This reverts commit 9196a15de9.
2019-09-10 11:13:58 +08:00
Duo Zhang 9196a15de9 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-09-10 10:47:44 +08:00
Duo Zhang eab7d3d502 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-09-10 10:47:44 +08:00