Commit Graph

17834 Commits

Author SHA1 Message Date
Norbert Kalmar ec3f843b23
HBASE-20598 Upgrade to JRuby 9.2
- upgrade our default jruby to 9.2.13.0
- this major JRuby version update changes the Ruby compatibility from Ruby 2.3 to Ruby 2.5
- use a custom IRB prompt to convey similar information to before
- update the joni and jcoding dependencies to match this version of jruby-complete

closes #2308

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit f0c430aed2)
2020-11-09 16:50:34 -06:00
Mate Szalay-Beko 9fe62c3ec6
HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
Prior to this patch hbase always used the MD5 hash algorithm to store a hash for encryption keys.
This hash is needed to verify the secret key of the subject. (e.g. making
sure that the same secrey key is used during encrypted HFile read and write).
The MD5 algorithm is considered weak, and can not be used in some
(e.g. FIPS compliant) clusters.

In this patch we:
- add a config parameter to globally enable/disable column family encryption (def enabled)
- introduce a backward compatible way of specifying the hash algorithm.
  This enable us to use newer and more secure hash algorithms like SHA-384
  or SHA-512 (which are FIPS compliant).
- add a config parameter to fail if an hfile is encountered that uses a
  different hash algorithm than the one currently configured to ease validation after
  migrating key hash algorithms (def disabled)

Closes #2539

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
(cherry picked from commit 6a5c928539)
2020-11-09 14:15:22 -06:00
gvprathyusha6 21b32d9a6f
HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Closes #2542

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-11-09 12:41:04 +05:30
Duo Zhang 7a07dfe059 HBASE-25254 Rewrite TestMultiLogThreshold to remove the LogDelegate in RSRpcServices (#2631)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-08 22:16:03 +08:00
Duo Zhang 38512e6f38 HBASE-25252 Move HMaster inner classes out (#2628)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-07 20:14:39 +08:00
Michael Stack df0009d879
HBASE-25238 Upgrading HBase from 2.2.0 to 2.3.x fails because of “Message missing required fields: state” (#2625)
Make protobuf fields add post-2.0.0 release marked 'required' instead
be 'optional' so migrations from 2.0.x to 2.1+ or 2.2+ succeeds.
2020-11-05 08:36:55 -08:00
WenFeiYi de74312d74
HBASE-25240 gson format of RpcServer.logResponse is abnormal
Closes #2623

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-05 19:59:34 +05:30
niuyulin 3d8152b635 HBASE-25053 WAL replay should ignore 0-length files (#2437)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-04 08:22:14 -08:00
niuyulin 770098128a HBASE-25210 RegionInfo.isOffline is now a duplication with RegionInfo.isSplit (#2580)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-11-04 08:11:00 -08:00
Duo Zhang 04293ad6dd HBASE-25216 The client zk syncer should deal with meta replica count change (#2614)
Signed-off-by: Yu Li <liyu@apache.org>
2020-11-04 18:25:08 +08:00
Andrew Purtell 1cc39ffd59
HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close (#2575)
If hbase.regionserver.close.wait.abort is set to true, interrupt RPC
handler threads holding the region close lock.

Until requests in progress can be aborted, wait on the region close lock for
a configurable interval (specified by hbase.regionserver.close.wait.time.ms,
default 60000 (1 minute)). If we have failed to acquire the close lock after
this interval elapses, if allowed (also specified by
hbase.regionserver.close.wait.abort), abort the regionserver.

We will attempt to interrupt any running handlers every
hbase.regionserver.close.wait.interval.ms (default 10000 (10 seconds)) until
either the close lock is acquired or we reach the maximum wait time.

Define a subset of region operations as interruptible. Track threads holding
the close lock transiting those operations. Set the thread interrupt status
of tracked threads when trying to close the region. Use the thread interrupt
status where safe to break out of request processing.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-03 15:21:32 -08:00
Sandeep Pal 729518af9d
HBASE-24859: Optimize in-memory representation of HBase map reduce table splits (#2610)
Patch fixes the single table input format case.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-11-03 07:40:25 -08:00
Nick Dimiduk ca45b1b45c HBASE-25228 Delete dev-support/jenkins_precommit_jira_yetus.sh (#2611)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-11-02 13:18:57 -08:00
bsglz 3a45fc17e5
HBASE-24967 The table.jsp cost long time to load if the table include closed regions (#2613) 2020-11-01 21:11:15 +08:00
Nick Dimiduk 1c7d472537 HBASE-25167 Normalizer support for hot config reloading (#2523)
Wire up the `ConfigurationObserver` chain for
`RegionNormalizerManager`. The following configuration keys support
hot-reloading:
 * hbase.normalizer.throughput.max_bytes_per_sec
 * hbase.normalizer.split.enabled
 * hbase.normalizer.merge.enabled
 * hbase.normalizer.min.region.count
 * hbase.normalizer.merge.min_region_age.days
 * hbase.normalizer.merge.min_region_size.mb

Note that support for `hbase.normalizer.period` is not provided
here. Support would need to be implemented generally for the `Chore`
subsystem.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-10-30 15:32:36 -07:00
Nick Dimiduk 5e3fa7df65 HBASE-24200 Upgrade to Yetus 0.12.0
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-30 14:00:11 -07:00
Nick Dimiduk b84e2f598b HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-10-30 10:43:43 -07:00
Pankaj 40608c33ef HBASE-24977 Meta table shouldn't be modified as read only (#2537)
Signed-off-by: stack <stack@apache.org>
2020-10-29 16:35:07 -07:00
GeorryHuang 69f282e21a
HBASE-25090 CompactionConfiguration logs unrealistic store file sizes (#2595)
Signed-off-by: stack <stack@apache.org>
2020-10-29 13:09:18 -07:00
Peter Somogyi b155c62f38 HBASE-25224 Maximize sleep for checking meta and namespace regions availability (#2593)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-28 18:09:16 +01:00
Minji Kim 970aaf38bd HBASE-25223 Use try-with-resources statement (#2592)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-10-28 09:20:35 -07:00
niuyulin bfe92ea53b HBASE-25201 YouAreDeadException should be moved to hbase-server module (#2581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 15:02:30 +08:00
niuyulin 1bf68514ea HBASE-25176 MasterStoppedException should be moved to hbase-client module (#2538)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 15:02:20 +08:00
Reid Chan 70631e901f HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level (#2549)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-25 17:47:58 +08:00
Toshihiro Suzuki f7e6143f49 HBASE-25206 Data loss can happen if a cloned table loses original split region(delete table) (#2569)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-24 23:04:33 +08:00
Sandeep Pal 3862dfc930
HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Closes #2556

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-23 12:47:31 +05:30
sanjeetnishad95 dd0a14538b HBASE-25128 RSGroupInfo's toString() and hashCode() does not take into account configuration map. (#2484)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-23 08:43:31 +08:00
Duo Zhang fdea847130 HBASE-25207 Revisit the implementation and usage of RegionStates.include (#2571)
Remove the RegionStates.include method as its name is ambiguous.
Add more comments to describe the logic on why we filter region like
this.

Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
2020-10-22 22:01:17 +08:00
Duo Zhang 83b189fb46
HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCleaner (#2577)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-22 10:54:57 +08:00
Duo Zhang 9d61afae6e
HBASE-25186 Addendum fix compile error (#2572)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-21 16:47:56 +08:00
Guanghao Zhang aa888f7e38 HBASE-25204 Nightly job failed as the name of jdk and maven changed (#2567)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-20 17:21:08 +08:00
Duo Zhang 02e77e0ba7 HBASE-25203 Change the reference url to flaky list in our jenkins jobs (#2566)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-20 17:02:35 +08:00
Duo Zhang 0c20f2b212 HBASE-25194 Do not publish workspace in flaky find job (#2564)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-10-20 12:31:30 +08:00
shahrs87 06bb12ed83
HBASE-25179 : Fix Assert format in HFilePerformanceEvaluation class
Closes #2551

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-16 12:09:10 +05:30
Nick Dimiduk fc4957f456 HBASE-24628 Region normalizer now respects a rate limit (HMaster chore shutdown NPE ADDENDUM) (#2540)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-15 22:06:40 +08:00
Duo Zhang 2f566dc234 HBASE-25186 TestMasterRegionOnTwoFileSystems is failing after HBASE-25065 (#2544)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-10-15 10:45:09 +08:00
XinSun 4e6c52c319 HBASE-25117 ReplicationSourceShipper thread can not be finished (#2521)
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>

(cherry picked from commit 78b7244091)
2020-10-14 18:11:10 +01:00
Wellington Ramos Chevreuil 5d65bd45fa
HBASE-24877 Add option to avoid aborting RS process upon uncaught exceptions happen on replication source (#2399)
Signed-off-by: stack <stack@apache.org>
2020-10-14 10:54:23 +01:00
Sean Busbey b862c3e9c9
HBASE-14067 bundle ruby files for hbase shell into a jar.
* removes some cruft from the hbase-shell pom that appears to be from coping the hbase-server pom long ago
* puts the ruby scripts into the hbase-shell jar following the guide from jruby for packaging
* removes hard coding the location of the implementation scripts from our runtime
* removes hard coding the load path for the implementation from the test code (leaves hard coding the test code location)
* provides a work around for a name conflict between our shell and the ruby stdlib shell.

closes #2515

Signed-off-by: Michael Stack <stack@apache.org>
(cherry picked from commit fd0ecadbb9)
2020-10-13 16:29:55 -05:00
bsglz dd6974fda8
HBASE-24875 Remove the force param for unassign since it dose not take effect any more (#2254)
Modified compared to main branch to deprecate obviated MasterObserver interface methods instead of remove them.

Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit c5ca191921)

 Conflicts:
	hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestWithDisabledAuthorization.java
2020-10-13 16:29:03 -05:00
Elliot 29bbc7de45
HBASE-24806 Small Updates to Functionality of Shell IRB Workspace (#2232)
* HBASE-24806 Small Updates to Functionality of Shell IRB Workspace

- Move exception handler from Shell::Shell#eval_io to new method,
  Shell::Shell#exception_handler
- Add unit tests for Shell::Shell#exception_handler
- Change Shell::Shell#eval_io to no longer raise SystemExit when any error is
  seen and update unit test
- Update ruby test runner to catch SystemExit and fail to avoid tests that
  cause the test runner to incorrectly exit successfully
- Add Hbase::Loader module to find ruby scripts in the $LOAD_PATH and classpath
  using JRuby's loader.
- In hbase-shell, install IRB commands before exporting HBase commands. The
  HBase commands will override the IRB commands, and no warning will be
  printed.

* Remove unused variables from shell_test

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
(cherry picked from commit 98e35842eb)
2020-10-13 16:27:21 -05:00
Elliot 588e8d2263
HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace (#2141)
* HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace

- Refactor Shell.export_commands to define commands using ruby lambdas.
  Additionally, this change stores a reference to shell_inst in scope so that
  we no longer need to assume the existance of the variable @shell.
- Add logic to Shell class for constructing an IRB workspace with its own
  binding and non-global receiver. This workspace is loaded with all HBase and
  IRB commands.
- Create new method on Shell for evaluating input from an arbitrary IO instance
  within the created IRB workspace. This is based on work by Hsieh that was
  previously in bin/hirb.rb. This method is more generic and more testable.
  This single pattern can be used for both executing files and reading from
  stdin, therefore reducing complexity.
- Move special 'help' and 'tools' command definitions to shell.rb. These
  commands are tightly linked with an instance of the shell, so it is easiest
  to have them defined together.
- Remove all global includes of HBaseConstants from ruby test files. Before
  this change, tests were loading these constants into the top level, which
  could cause tests to pass that should really fail.
- Try to reduce the number of places that constants are included. I think it's
  best to reference each ruby constant's full name, but where that would cause
  a big diff I instead moved the include to the innermost Module or Class.
- Update docs and comments
- Remove unneccessary includes
- Add shell --top-level-cmds compatibility flag. Since this PR removes all the
  HBase symbols from the top-level receiver (ie. main Object), it is possible
  (albeit unlikely) that this will break operator scripts. This flag will
  export all the commands at the top-level like the shell previously did.

* HBASE-11686 Light refactoring with added unit tests

- Fixes some constants references by admin test 2
- Install HBase commands as singleton methods on recevier instances so that
  multiple receivers may exist.
- Rename new flag from --top-level-cmds to --top-level-defs to be more
  semantically accurate.
- Create new helper method Shell::Shell#export_all to install @hbase, @shell,
  constants, and all hbase commands to a target receiver. As a result, the
  HBaseReceiver became very simple and could be moved to shell.rb.
- Add unit tests for Shell::Shell#eval_io and Shell::Shell#export_all
- Add @hbase and @shell to hbase-shell IRB workspace
- Fix robocop issues within patch

* Typo s/is/if/

(cherry picked from commit 7eff07d6bf)
2020-10-13 16:27:03 -05:00
niuyulin b0d140f40a
HBASE-25093 the RSGroupBasedLoadBalancer#retainAssignment throws NPE (#2534)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-13 21:25:09 +08:00
Duo Zhang a32303290d HBASE-25169 Update documentation about meta region replica (#2528)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-13 10:14:05 +08:00
Duo Zhang 37c2ffdc2b HBASE-25164 Make ModifyTableProcedure support changing meta replica count (#2513)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-13 10:13:48 +08:00
ramkrish86 ed7aa8e369
HBASE-25065 - WAL archival to be done by a separate thread (#2531)
* HBASE-25065 - WAL archival to be done by a separate thread

* Fix checkstyle comments

* Fix compile issue

* Fix checkstyle and make the failing test more reliable

* Remove unused import
2020-10-12 21:57:25 +05:30
Qi Yu 3beae80c60 HBASE-25162 Make flaky tests run more aggressively (#2525)
Change the trigger interval from 12h to 4h

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-10 16:10:29 +08:00
Duo Zhang 841556f91b HBASE-25163 Increase the timeout value for nightly jobs (#2512)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-10 08:32:03 +08:00
Mohammad Arshad 74e111633e
HBASE-24025:Improve performance of move_servers_rsgroup and move_tables_rsgroup by using async region move API (#1321) 2020-10-09 12:12:09 -07:00
Nick Dimiduk 2253800ee1 HBASE-24628 Region normalizer now respects a rate limit
Implement a rate limiter for the normalizer. Implemented in terms of
MB/sec of affacted region size (the same metrics used to make
normalization decisions). Uses Guava `RateLimiter` to perform the
resource accounting. `RateLimiter` works by blocking (uninterruptible
😖) the calling thread. Thus, the whole construction of the normalizer
subsystem needed refactoring. See the provided `package-info.java` for
an overview of this new structure.

Introduces a new configuration,
`hbase.normalizer.throughput.max_bytes_per_sec`, for specifying a
limit on the throughput of actions executed by the normalizer. Note
that while this configuration value is in bytes, the minimum honored
valued `1_000_000`. Supports values configured using the
human-readable suffixes honored by `Configuration.getLongBytes`

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-09 09:47:45 -07:00