The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
Remove the RegionStates.include method as its name is ambiguous.
Add more comments to describe the logic on why we filter region like
this.
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
* removes some cruft from the hbase-shell pom that appears to be from coping the hbase-server pom long ago
* puts the ruby scripts into the hbase-shell jar following the guide from jruby for packaging
* removes hard coding the location of the implementation scripts from our runtime
* removes hard coding the load path for the implementation from the test code (leaves hard coding the test code location)
* provides a work around for a name conflict between our shell and the ruby stdlib shell.
closes#2515
Signed-off-by: Michael Stack <stack@apache.org>
(cherry picked from commit fd0ecadbb9)
Modified compared to main branch to deprecate obviated MasterObserver interface methods instead of remove them.
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit c5ca191921)
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestWithDisabledAuthorization.java
* HBASE-24806 Small Updates to Functionality of Shell IRB Workspace
- Move exception handler from Shell::Shell#eval_io to new method,
Shell::Shell#exception_handler
- Add unit tests for Shell::Shell#exception_handler
- Change Shell::Shell#eval_io to no longer raise SystemExit when any error is
seen and update unit test
- Update ruby test runner to catch SystemExit and fail to avoid tests that
cause the test runner to incorrectly exit successfully
- Add Hbase::Loader module to find ruby scripts in the $LOAD_PATH and classpath
using JRuby's loader.
- In hbase-shell, install IRB commands before exporting HBase commands. The
HBase commands will override the IRB commands, and no warning will be
printed.
* Remove unused variables from shell_test
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
(cherry picked from commit 98e35842eb)
* HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace
- Refactor Shell.export_commands to define commands using ruby lambdas.
Additionally, this change stores a reference to shell_inst in scope so that
we no longer need to assume the existance of the variable @shell.
- Add logic to Shell class for constructing an IRB workspace with its own
binding and non-global receiver. This workspace is loaded with all HBase and
IRB commands.
- Create new method on Shell for evaluating input from an arbitrary IO instance
within the created IRB workspace. This is based on work by Hsieh that was
previously in bin/hirb.rb. This method is more generic and more testable.
This single pattern can be used for both executing files and reading from
stdin, therefore reducing complexity.
- Move special 'help' and 'tools' command definitions to shell.rb. These
commands are tightly linked with an instance of the shell, so it is easiest
to have them defined together.
- Remove all global includes of HBaseConstants from ruby test files. Before
this change, tests were loading these constants into the top level, which
could cause tests to pass that should really fail.
- Try to reduce the number of places that constants are included. I think it's
best to reference each ruby constant's full name, but where that would cause
a big diff I instead moved the include to the innermost Module or Class.
- Update docs and comments
- Remove unneccessary includes
- Add shell --top-level-cmds compatibility flag. Since this PR removes all the
HBase symbols from the top-level receiver (ie. main Object), it is possible
(albeit unlikely) that this will break operator scripts. This flag will
export all the commands at the top-level like the shell previously did.
* HBASE-11686 Light refactoring with added unit tests
- Fixes some constants references by admin test 2
- Install HBase commands as singleton methods on recevier instances so that
multiple receivers may exist.
- Rename new flag from --top-level-cmds to --top-level-defs to be more
semantically accurate.
- Create new helper method Shell::Shell#export_all to install @hbase, @shell,
constants, and all hbase commands to a target receiver. As a result, the
HBaseReceiver became very simple and could be moved to shell.rb.
- Add unit tests for Shell::Shell#eval_io and Shell::Shell#export_all
- Add @hbase and @shell to hbase-shell IRB workspace
- Fix robocop issues within patch
* Typo s/is/if/
(cherry picked from commit 7eff07d6bf)
* HBASE-25065 - WAL archival to be done by a separate thread
* Fix checkstyle comments
* Fix compile issue
* Fix checkstyle and make the failing test more reliable
* Remove unused import
Implement a rate limiter for the normalizer. Implemented in terms of
MB/sec of affacted region size (the same metrics used to make
normalization decisions). Uses Guava `RateLimiter` to perform the
resource accounting. `RateLimiter` works by blocking (uninterruptible
😖) the calling thread. Thus, the whole construction of the normalizer
subsystem needed refactoring. See the provided `package-info.java` for
an overview of this new structure.
Introduces a new configuration,
`hbase.normalizer.throughput.max_bytes_per_sec`, for specifying a
limit on the throughput of actions executed by the normalizer. Note
that while this configuration value is in bytes, the minimum honored
valued `1_000_000`. Supports values configured using the
human-readable suffixes honored by `Configuration.getLongBytes`
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: Michael Stack <stack@apache.org>
Make it so WALPlayer can replay recovered.edits files.
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
Allow for WAL files that do NOT have a startime in their name.
Use the 'generic' WAL-filename parser instead of the one that
used be local here. Implement support for 'startTime' filter.
Previous was just not implemented.
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
Checkstyle.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
Use the new general WAL name timestamp parser.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
Utility for parsing timestamp from WAL filename.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
Export attributes about the local recovered.edits test file
so other tests can play with it.
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Change the test to wait for evidence that the active master has seen
that the backup master killed by the test has gone away. This is done
before proceeding to validate that the dead backup is correctly
omitted from the ClusterStatus report.
Also, minor fixup to several assertions, using `assertEquals` instead
of `assertTrue(...equals(...))` and correcting expected vs. actual
ordering of assertion arguments.
Signed-off-by: Michael Stack <stack@apache.org>
Nick Dimiduk
Pankaj
GeorryHuang
Peter Somogyi
Minji Kim
niuyulin
Reid Chan
Toshihiro Suzuki
Sandeep Pal
sanjeetnishad95
Duo Zhang
Guanghao Zhang
shahrs87
XinSun
Wellington Ramos Chevreuil
Sean Busbey
bsglz
Elliot
ramkrish86
Qi Yu
Mohammad Arshad
Michael Stack
Semen Komissarov
Andor Molnár
申胜利
Viraj Jasani
Joseph295