* removes some cruft from the hbase-shell pom that appears to be from coping the hbase-server pom long ago
* puts the ruby scripts into the hbase-shell jar following the guide from jruby for packaging
* removes hard coding the location of the implementation scripts from our runtime
* removes hard coding the load path for the implementation from the test code (leaves hard coding the test code location)
* provides a work around for a name conflict between our shell and the ruby stdlib shell.
closes#2515
Signed-off-by: Michael Stack <stack@apache.org>
(cherry picked from commit fd0ecadbb9)
Modified compared to main branch to deprecate obviated MasterObserver interface methods instead of remove them.
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit c5ca191921)
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestWithDisabledAuthorization.java
* HBASE-24806 Small Updates to Functionality of Shell IRB Workspace
- Move exception handler from Shell::Shell#eval_io to new method,
Shell::Shell#exception_handler
- Add unit tests for Shell::Shell#exception_handler
- Change Shell::Shell#eval_io to no longer raise SystemExit when any error is
seen and update unit test
- Update ruby test runner to catch SystemExit and fail to avoid tests that
cause the test runner to incorrectly exit successfully
- Add Hbase::Loader module to find ruby scripts in the $LOAD_PATH and classpath
using JRuby's loader.
- In hbase-shell, install IRB commands before exporting HBase commands. The
HBase commands will override the IRB commands, and no warning will be
printed.
* Remove unused variables from shell_test
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
(cherry picked from commit 98e35842eb)
* HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace
- Refactor Shell.export_commands to define commands using ruby lambdas.
Additionally, this change stores a reference to shell_inst in scope so that
we no longer need to assume the existance of the variable @shell.
- Add logic to Shell class for constructing an IRB workspace with its own
binding and non-global receiver. This workspace is loaded with all HBase and
IRB commands.
- Create new method on Shell for evaluating input from an arbitrary IO instance
within the created IRB workspace. This is based on work by Hsieh that was
previously in bin/hirb.rb. This method is more generic and more testable.
This single pattern can be used for both executing files and reading from
stdin, therefore reducing complexity.
- Move special 'help' and 'tools' command definitions to shell.rb. These
commands are tightly linked with an instance of the shell, so it is easiest
to have them defined together.
- Remove all global includes of HBaseConstants from ruby test files. Before
this change, tests were loading these constants into the top level, which
could cause tests to pass that should really fail.
- Try to reduce the number of places that constants are included. I think it's
best to reference each ruby constant's full name, but where that would cause
a big diff I instead moved the include to the innermost Module or Class.
- Update docs and comments
- Remove unneccessary includes
- Add shell --top-level-cmds compatibility flag. Since this PR removes all the
HBase symbols from the top-level receiver (ie. main Object), it is possible
(albeit unlikely) that this will break operator scripts. This flag will
export all the commands at the top-level like the shell previously did.
* HBASE-11686 Light refactoring with added unit tests
- Fixes some constants references by admin test 2
- Install HBase commands as singleton methods on recevier instances so that
multiple receivers may exist.
- Rename new flag from --top-level-cmds to --top-level-defs to be more
semantically accurate.
- Create new helper method Shell::Shell#export_all to install @hbase, @shell,
constants, and all hbase commands to a target receiver. As a result, the
HBaseReceiver became very simple and could be moved to shell.rb.
- Add unit tests for Shell::Shell#eval_io and Shell::Shell#export_all
- Add @hbase and @shell to hbase-shell IRB workspace
- Fix robocop issues within patch
* Typo s/is/if/
(cherry picked from commit 7eff07d6bf)
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto
Closes#2261
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
- Update all hbase-shell commands to return help text implicitly
- Add hbase-shell test to ensure all help methods return unindented text
- Set the outermost indentation to zero for the help text of all hbase-shell
commands by using the squiggly heredoc added in Ruby 2.3.0.
- Document return value of hbck_chore_run and add usage examples to truncate commands
- Update docs to reflect current return value of list command
Signed-off-by: stack <stack@apache.org>
- In HBase::Table, the instance variable @converters is used to map column
names to converters. This patch fixes how HBase::Table#_get_internal and
HBase::Table#_scan_internal generate the column name key used to access
@converters.
- Refactor parsing of family:qualifier:converter specifications so that the
code is more readable and reusable. As part of this change, I added two
private methods and marked HBase::Table#set_converter as deprecated for
removal in HBase 4.0.0.
- Add unit testing for the fixed bug
Signed-off-by: stack <stack@apache.org>
- Prior to this commit, there were 13 commands that unintentionally return the
number of lines they print (usually one). This commit ensures that they
return the value documented by the help text, or nil if there is not a simple
logical value to return.
- Fixes 6 hbase-shell commands that return String rather than TrueClass or
FalseClass
- Use double-bang to cast truthy values to TrueClass and FalseClass so that
ruby's to_s can reliably print true or false without using ternary operators
- Updates tests for is_disabled, is_enabled, disable_rpc_throttle,
enable_rpc_throttle, disable_exceed_throttle_quota,
enable_exceed_throttle_quota, clear_deadservers, snapshot_cleanup_switch,
snapshot_cleanup_enabled, and balancer to check return values
- Adds new tests for balance_switch, balancer_enabled, normalizer_switch,
normalizer_enabled, catalog_janitor_switch, catalogjanitor_enabled,
cleaner_chore_switch, cleaner_chore_enabled, splitormerge_switch, and
splitormerge_enabled
signed-off-by: stack <stack@apache.org>
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
Have client and server use loopback instead of 'localhost'
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
Rename the threads we create in here so they are NOT named same was
threads created by Hadoop RPC.
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
Allow configuring eventloopgroup thread count (so can override for
tests)
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
Enable socket resuse.
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
Enable socket resuse and config for how many threads to use.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
Thread name edit; drop the redundant 'Thread' suffix.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
Make closeable and shutdown executor when called.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
Call close on HFileReplicator
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
HDFS creates lots of threads. Use less of it so less threads overall.
hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
Constrain resources when running in test context.
hbase-server/src/test/resources/log4j.properties
Enable debug on netty to see netty configs in our log
pom.xml
Add system properties when we launch JVMs to constrain thread counts in
tests
Signed-off-by: Duo Zhang <zhangduo@apache.org>
mergeRegionsAsync admin API with two regions as parameters deprecated since 2.3.0 and removed from 4.0
merge_region shell command now supports multiple regions merge since 2.3.0 & 3.0.0
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
(cherry picked from commit 5f61df4792)
Make hbase:meta region schema dynamic.
Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shut down access to internals and removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Cleanup/refactor section on replica-handling.
M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Get hbase:meta schema from filesystem rather than from hard-coding.
Make it so hbase:meta can be altered. TableState for hbase:meta
was hardcoded ENABLED. Make it dynamic. State is now kept in
current active Master. It is transient so falls back to default
if Master crashes. Add to registry a getMetaTableState
which reads mirrored state from zookeeper (NOT from Master and
defaults ENABLED if no implementation or error fetching state).
hbase:meta schema will be bootstrapped from the filesystem.
Changes to filesystem schema are atomic so we should be ok if
Master fails mid-edit (TBD). Undoes a bunch of guards that
prevented our being able to edit hbase:meta.
TODO: Tests, more clarity around hbase:meta table state, and undoing
references to hard-coded hbase:meta regioninfo.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Throw illegal access exception if you try to use MetaTableAccessor
getting state of the hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
Add fetching of hbase:meta table state from registry. Adds cache of
tablestates w/ a ttl of 1 second (adjustable).
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
Add querying registry for hbase:meta table state.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
Add querying of mirrored table state for hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shutdown access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
Just cleanup.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
Add state holder for hbase:meta. Removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
Shut down access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
Allow hbase:meta to be disabled.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Allow hbase:meta to be enabled.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Sean Busbey
bsglz
Elliot
Viraj Jasani
Bo Cui
Guanghao Zhang
Nick Dimiduk
Pankaj
XinSun
Wellington Ramos Chevreuil
Duo Zhang
Michael Stack
Reid Chan
EEPROM
stack
Sakthi
Karthik Palanisamy
Mingliang Liu