Commit Graph

511 Commits

Author SHA1 Message Date
Elliot 7974a1e9bf
HBASE-11676 Scan FORMATTER is not applied for columns using non-printable name in shell (#2161)
- In HBase::Table, the instance variable @converters is used to map column
  names to converters. This patch fixes how HBase::Table#_get_internal and
  HBase::Table#_scan_internal generate the column name key used to access
  @converters.
- Refactor parsing of family:qualifier:converter specifications so that the
  code is more readable and reusable. As part of this change, I added two
  private methods and marked HBase::Table#set_converter as deprecated for
  removal in HBase 4.0.0.
- Add unit testing for the fixed bug

Signed-off-by: stack <stack@apache.org>
2020-07-28 20:43:19 -07:00
Pankaj 4471a644f6
HBASE-24738 [Shell] processlist command fails with ERROR: Unexpected end of file from server when SSL enabled (#2123)
Signed-off-by: Elliot Miller <elliot_miller@apple.com>
2020-07-28 09:10:42 -07:00
Elliot 7eff07d6bf
HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace (#2141)
* HBASE-11686 Shell code should create a binding / irb workspace instead of polluting the root namespace

- Refactor Shell.export_commands to define commands using ruby lambdas.
  Additionally, this change stores a reference to shell_inst in scope so that
  we no longer need to assume the existance of the variable @shell.
- Add logic to Shell class for constructing an IRB workspace with its own
  binding and non-global receiver. This workspace is loaded with all HBase and
  IRB commands.
- Create new method on Shell for evaluating input from an arbitrary IO instance
  within the created IRB workspace. This is based on work by Hsieh that was
  previously in bin/hirb.rb. This method is more generic and more testable.
  This single pattern can be used for both executing files and reading from
  stdin, therefore reducing complexity.
- Move special 'help' and 'tools' command definitions to shell.rb. These
  commands are tightly linked with an instance of the shell, so it is easiest
  to have them defined together.
- Remove all global includes of HBaseConstants from ruby test files. Before
  this change, tests were loading these constants into the top level, which
  could cause tests to pass that should really fail.
- Try to reduce the number of places that constants are included. I think it's
  best to reference each ruby constant's full name, but where that would cause
  a big diff I instead moved the include to the innermost Module or Class.
- Update docs and comments
- Remove unneccessary includes
- Add shell --top-level-cmds compatibility flag. Since this PR removes all the
  HBase symbols from the top-level receiver (ie. main Object), it is possible
  (albeit unlikely) that this will break operator scripts. This flag will
  export all the commands at the top-level like the shell previously did.

* HBASE-11686 Light refactoring with added unit tests

- Fixes some constants references by admin test 2
- Install HBase commands as singleton methods on recevier instances so that
  multiple receivers may exist.
- Rename new flag from --top-level-cmds to --top-level-defs to be more
  semantically accurate.
- Create new helper method Shell::Shell#export_all to install @hbase, @shell,
  constants, and all hbase commands to a target receiver. As a result, the
  HBaseReceiver became very simple and could be moved to shell.rb.
- Add unit tests for Shell::Shell#eval_io and Shell::Shell#export_all
- Add @hbase and @shell to hbase-shell IRB workspace
- Fix robocop issues within patch

* Typo s/is/if/
2020-07-27 20:56:09 -07:00
Guanghao Zhang 32c7012ad7
HBASE-24743 Reject to add a peer which replicate to itself earlier (#2122)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-24 08:08:20 +08:00
Viraj Jasani f35c5eaadd
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2109)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-21 20:24:13 +05:30
Elliot 4b3ef815eb
HBASE-24722 Update commands with unintentional return values (#2058)
- Prior to this commit, there were 13 commands that unintentionally return the
  number of lines they print (usually one). This commit ensures that they
  return the value documented by the help text, or nil if there is not a simple
  logical value to return.
- Fixes 6 hbase-shell commands that return String rather than TrueClass or
  FalseClass
- Use double-bang to cast truthy values to TrueClass and FalseClass so that
  ruby's to_s can reliably print true or false without using ternary operators
- Updates tests for is_disabled, is_enabled, disable_rpc_throttle,
  enable_rpc_throttle, disable_exceed_throttle_quota,
  enable_exceed_throttle_quota, clear_deadservers, snapshot_cleanup_switch,
  snapshot_cleanup_enabled, and balancer to check return values
- Adds new tests for balance_switch, balancer_enabled, normalizer_switch,
  normalizer_enabled, catalog_janitor_switch, catalogjanitor_enabled,
  cleaner_chore_switch, cleaner_chore_enabled, splitormerge_switch, and
  splitormerge_enabled

signed-off-by: stack <stack@apache.org>
2020-07-17 11:44:30 -07:00
bsglz 724f0478ed
HBASE-24404 Support flush a single column family of region (#2032)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-10 18:13:50 +01:00
Elliot 83f27b5287
HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module (#1959)
* HBASE-20819 Use TableDescriptor to replace HTableDescriptor in hbase-shell module

* Resolve whitespace issues

* Use removeValue rather than setValue where possible in hbase-shell

* Add String removeValue overload to TableDescriptorBuilder

* Convert remaining usages of HTableDescriptor in security.rb

* Rename hcd (HColumnDescriptor) to cfd (ColumnFamilyDescriptor)

* Add back 3 constants to HBaseConstants in hbase-shell

* Fix bad constant reference

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-26 12:39:34 -07:00
XinSun 4b89ee24f3
HBASE-24431 RSGroupInfo add configuration map to store something extra (#1782)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-24 18:46:09 +08:00
Duo Zhang c91829bb41
HBASE-24491 Remove HRegionInfo (#1830)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-06-05 22:19:01 +08:00
bsglz 3963f62453
HBASE-24486 Remove the unused method online in admin.rb (#1824)
Signed-off-by: clarax98007@gmail.com
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-02 11:22:29 -07:00
Wellington Ramos Chevreuil e5345b3a7c
HBASE-21406 "status 'replication'" should not show SINK if the cluste… (#1761)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-06-02 09:27:16 +01:00
Viraj Jasani f0f9a20cd7
HBASE-23941 : FilterBy operator support in get_slowlog_responses API (#1793)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-05-30 12:59:59 +05:30
Wellington Ramos Chevreuil 9d9f07be36
HBASE-24413 HBASE-22259 Removed deprecated getTimeStampOfLastShippedOp method from ReplicationLoadSource, but there were still references to this method on ruby admin.rb (#1754)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
2020-05-22 10:46:33 +01:00
Duo Zhang 8601416ee8
HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for modules other than hbase-assembly (#1640)
Signed-off-by: stack <stack@apache.org>
2020-05-12 12:03:30 +08:00
bsglz 2cafe81e9c
HBASE-24335 Support deleteall with ts but without column in shell mode (#1668)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-05-07 18:18:57 +01:00
Michael Stack 5488124be0 HBASE-24284 [h3/jdk11] REST server won't start Exclude transitive includes of jax-rs 1.x and then explicitly include jax-rs 2.x glassfish impl for REST context when hadoop3. (#1625) 2020-05-05 15:36:01 -07:00
Duo Zhang a2e373dc32 HBASE-24281 Remove some stale hadoop.version properties in the pom for sub modules (#1602)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-29 18:24:52 +08:00
Duo Zhang a51a1f4d10 Revert "HBASE-24181 Remove some stale hadoop.version properties in the pom for sub modules (#1602)"
This reverts commit af40bff8be.
2020-04-29 18:24:41 +08:00
Duo Zhang af40bff8be
HBASE-24181 Remove some stale hadoop.version properties in the pom for sub modules (#1602)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-29 18:02:07 +08:00
Duo Zhang 6928674eb8
HBASE-24228 Merge the code in hbase-hadoop2-compat module to hbase-hadoop-compat (#1563)
Signed-off-by: stack <stack@apache.org>
2020-04-29 10:34:53 +08:00
Reid Chan 9cddac01ba
HBASE-24196 [Shell] Add rename rsgroup command in hbase shell (#1551)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-21 10:22:33 +08:00
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
EEPROM 35ea4acf48
HBASE-24176 user_permission '.*' command failed to show all table permissions (#1499)
* user_permission failed when show table permissions with '.*' argument

Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2020-04-14 10:14:21 +08:00
Jan Hentschel 4ddf55d20a
HBASE-23843 Removed deprecated Scan(byte[]) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-02 11:44:34 +02:00
Viraj Jasani 0dcbf80583
HBASE-23937 : Support Online LargeLogs similar to SlowLogs APIs (#1346)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-01 18:22:22 +05:30
Jan Hentschel fb17ecdd07
HBASE-23842 Removed deprecated Scan(byte[], byte[]) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-01 13:11:21 +02:00
Michael Stack 3e1cf00c71 HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:26:06 -07:00
stack f66cbe1a40 HBASE-23992 Fix TestAdminShell and TestQuotasShell mistakenly broken by parent commit 2020-03-14 15:18:21 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
stack cf121227aa HBASE-23930 Shell should attempt to format `timestamp` attributes as ISO-8601
Make display of Cell timestamp be ISO8601 format instead of pure milliseconds.
2020-03-11 17:15:15 -07:00
stack 0b18a7dc26 HBASE-23929 Shell formatter for for meta table should pretty-print values of info:merge columns 2020-03-11 16:41:30 -07:00
Michael Stack aaf9a148a0 HBASE-23956 Use less resources running tests (#1266)
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
 Rename the threads we create in here so they are NOT named same was
 threads created by Hadoop RPC.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
 Allow configuring eventloopgroup thread count (so can override for
 tests)

hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
 Enable socket resuse.

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
 Enable socket resuse and config for how many threads to use.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
 Thread name edit; drop the redundant 'Thread' suffix.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
 Make closeable and shutdown executor when called.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 Call close on HFileReplicator

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
 HDFS creates lots of threads. Use less of it so less threads overall.

hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
 Constrain resources when running in test context.

hbase-server/src/test/resources/log4j.properties
 Enable debug on netty to see netty configs in our log

pom.xml
 Add system properties when we launch JVMs to constrain thread counts in
 tests

 Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-11 10:37:50 -07:00
Viraj Jasani 4ef326be15
HBASE-23909 : list_regions raise correct error if it is being splitted/merged/transitioning (#1238)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-06 13:01:17 +05:30
Duo Zhang 7f2d823164 HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the code base (#1152)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 37e87aeca7 HBASE-23807 Make rsgroup related shell command to use the new admin methods (#1148)
Signed-off-by: stack <stack@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang f0a13bb74d HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2020-03-02 15:43:40 +08:00
Viraj Jasani b24ea32b3b
HBASE-22978 : Online slow response log
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-28 19:47:59 +05:30
Sakthi 5f61df4792
HBASE-22827 Expose multi-region merge in shell and Admin API (#1138)
mergeRegionsAsync admin API with two regions as parameters deprecated since 2.3.0 and removed from 4.0
merge_region shell command now supports multiple regions merge since 2.3.0 & 3.0.0

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-13 14:40:46 -08:00
Duo Zhang 4479cb2313 HBASE-23809 Addendum remove rs group code in AbstractTestShell 2020-02-08 22:46:08 +08:00
Duo Zhang 546828685e
HBASE-23809 The RSGroup shell test is missing (#1142)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-08 17:10:00 +08:00
Michael Stack ba3463d9de HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 15:02:40 -08:00
stack c8496bc694 Revert "HBASE-23055 Alter hbase:meta"
This reverts commit 9abdb7b5ae.
2020-01-11 09:17:17 -08:00
stack 9abdb7b5ae HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
    was hardcoded ENABLED. Make it dynamic. State is now kept in
    current active Master. It is transient so falls back to default
    if Master crashes. Add to registry a getMetaTableState
    which reads mirrored state from zookeeper (NOT from Master and
    defaults ENABLED if no implementation or error fetching state).
    hbase:meta schema will be bootstrapped from the filesystem.
    Changes to filesystem schema are atomic so we should be ok if
    Master fails mid-edit (TBD). Undoes a bunch of guards that
    prevented our being able to edit hbase:meta.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     Add fetching of hbase:meta table state from registry. Adds cache of
     tablestates w/ a ttl of 1 second (adjustable).

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
      Add querying registry for hbase:meta table state.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
      Add querying of mirrored table state for hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta. Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-09 16:25:16 -08:00
stack b0233a294f HBASE-23554 Encoded regionname to regionname utility (#923); ADDENDUM 2019-12-11 11:02:59 -08:00
Michael Stack 8742265d85 HBASE-23554 Encoded regionname to regionname utility (#923)
Adds shell command regioninfo:

  hbase(main):001:0>  regioninfo '0e6aa5c19ae2b2627649dc7708ce27d0'
  {ENCODED => 0e6aa5c19ae2b2627649dc7708ce27d0, NAME => 'TestTable,,1575941375972.0e6aa5c19ae2b2627649dc7708ce27d0.', STARTKEY => '', ENDKEY => '00000000000000000000299441'}
  Took 0.4737 seconds

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-12-11 08:57:59 -08:00
Karthik Palanisamy a553b78c1c HBASE-23360 [CLI] Fix help command 'set_quota' for removing limits (#896) 2019-12-11 10:06:40 -06:00
Karthik Palanisamy df9cbd4254 HBASE-23336 [CLI] Incorrect row(s) count 'clear_deadservers' (#875)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-11-26 19:20:33 +08:00
Mingliang Liu 4cc899d8e3 HBASE-23290 shell processlist command is broken 2019-11-14 16:03:29 -08:00
Karthik Palanisamy 0f910f0c32 HBASE-23176 delete_all_snapshot does not work with regex (#725) 2019-10-17 20:35:56 +08:00