The Hadoop AccessControlList allows us to specify admins of the webUI
via a list of users and/or groups. Admins of the WebUI can mutate the
system, potentially seeing sensitive data or modifying the system.
hbase.security.authentication.spnego.admin.users is a comma-separated
list of users who are admins.
hbase.security.authentication.spnego.admin.groups is a comma-separated
list of groups whose membership are admins. Either of these
configuration properties may also contain an asterisk (*) which denotes
"any entity" (e.g user, group).
Previously, when a user was denied from some endpoint that was
designated for admins, they received an HTTP/401. In this case, it is
more correct to return HTTP/403 as they were correctly authenticated,
but they were disallowed from fetching the given resource. This commit
incorporates this change.
hbase.security.authentication.ui.config.protected also exists for users
who have sensitive information stored in the Hadoop service
configuration and want to limit access to this endpoint. By default,
the Hadoop configuration endpoint is not protected and any
authenticated user can access it.
The test is based off of work by Nihal Jain in HBASE-20472.
Co-authored-by: Nihal Jain <nihaljain.cs@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
Complains about mismatch in types when Compare. Implement Compare in
base Interface.
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Complains pbs never return null.
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java
Needed redo because errorprone complains can't mock Service from guava.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
Unrelated...adding one-liner debug statements chasing other test
failures.
HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'.
Just remove the check state from the getClusterStatus call.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthi@apache.org>
(cherry picked from commit dd8496a546)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Codecs don't have access to what CellComparator to use. Backfill.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
Adds a new compareRows with default implementation that takes a ByteBuffer.
Needed by the index in a block encoder implementation.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
Adds implementation for meta of new compareRows method. Adds utility
method for figuring comparator based off tablename.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
Comparator is in context. Remove redundant handling.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
Comparator is in context. Remove redundant handling. Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
Cache context so can use it to get comparator to use later.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
Cache cellcomparator to use. Javdoc on diff between HFileContext and
HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
Add CellComparator
M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
Remove comparator caching. Get from context instead.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
Skip a reflection if we can.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
Javadoc. Removed unused filed.
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
Make hbase:meta region schema dynamic.
Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shut down access to internals and removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Cleanup/refactor section on replica-handling.
M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Get hbase:meta schema from filesystem rather than from hard-coding.
Adds a display of the content of 'hbase:meta' to the Master's
table.jsp, when that table is selected. Supports basic pagination,
filtering, &c.
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.
Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.
Closes#884
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
The current process for an operator, after fixing holes in meta, is to
manually disable and enable the whole table. Let's try to avoid
bringing the whole table offline if we can. Have the master attempt to
queue up assignment procedures for any new regions it creates.
Signed-off-by: stack <stack@apache.org>
The cleanupBulkLoad method is only called for the first Region in the
table which was being bulk loaded into. This means that potentially N-1
other RegionServers (where N is the number of RegionServers) will leak
one FileSystem object into the FileSystem cache which will never be
cleaned up. We need to do this clean-up as a part of secureBulkLoadHFiles
otherwise we cannot guarantee that heap usage won't grow unbounded.
Closes#1029
Signed-off-by: Sean Busbey <busbey@apache.org>
Make it so hbase:meta can be altered. TableState for hbase:meta
was hardcoded ENABLED. Make it dynamic. State is now kept in
current active Master. It is transient so falls back to default
if Master crashes. Add to registry a getMetaTableState
which reads mirrored state from zookeeper (NOT from Master and
defaults ENABLED if no implementation or error fetching state).
hbase:meta schema will be bootstrapped from the filesystem.
Changes to filesystem schema are atomic so we should be ok if
Master fails mid-edit (TBD). Undoes a bunch of guards that
prevented our being able to edit hbase:meta.
TODO: Tests, more clarity around hbase:meta table state, and undoing
references to hard-coded hbase:meta regioninfo.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Throw illegal access exception if you try to use MetaTableAccessor
getting state of the hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
Add fetching of hbase:meta table state from registry. Adds cache of
tablestates w/ a ttl of 1 second (adjustable).
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
Add querying registry for hbase:meta table state.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
Add querying of mirrored table state for hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shutdown access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
Just cleanup.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
Add state holder for hbase:meta. Removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
Shut down access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
Allow hbase:meta to be disabled.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Allow hbase:meta to be enabled.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Master force-closes unknown/incorrect Regions OPEN on RS
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Added a note and small refactor.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Fix an NPE when CJ ran.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Minor clean up of log message; make it clearer.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
Make it so closeRegionSilentlyAndWait can be used w/o timeout.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
If a RegionServer Report notes a Region is OPEN and the Master does not
know of said Region, close it (We used to crash out the RegionServer)
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
Minor tweak of toString -- label should be state, not rit (confusing).
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
Doc.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java
Add region name to exception.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java
Be more careful about which Regions we queue up for reassign. This
procedure is run by the operator so could happen at any time. We
will likely be running this when Master has some accounting of
cluster members so check its answers for what Regions were on
server before running.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Doc and we were misrepresenting the case where a Region as not in RIT
when we got CLOSE -- we were reporting it as though it was already
trying to CLOSE.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Change its behavior so it will only look in hbase:meta
if the call to the super class turns up zero references.
Only then will it search hbase:meta for references to
'Unknown Servers'. Normal operation where we read Master
context is usual and sufficient. The scan of hbase:meta
is only for case where Master state has been corrupted
and we need to clear out 'Unknown Servers'.
Having it as static means the test cannot be parameterized (ran into
this issue in HBASE-23305). That happens because the field is not
reset between parameterized runs.
* Adds a new MapReduce job that builds a report of health of mob files
* Also builds background information on mob system use
* add a basic mob architecture in the ref guide to explain how mob takes the reference hfile value and finds the actual cell contents
* add a troubleshooting mob section to the ref guide to explain how to do a mob reference scan.
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* Clean up JavaDocs
* Clean up logging and error messages
* Remove superfluous code
* Replace static code with library call
* Do not swallow Interrupted Exceptions
* Use try-with-resources
* User multi-Exception catches to reduce code size
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Removes a bunch of dead code and fixes some checkstyle nits.
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
* Handling the BAD value in info:state columns in hbase:meta
* Adding a unit test and region encoded name in the LOG
* Adding a null check for region state to complete the test scenario and fixing the nit
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: GuangxuCheng <guangxucheng@gmail.com>
Signed-off-by: stack <stack@apache.org>
All the clients need to know the master RPC end points while using master
based registry for creating cluster connections. This patch amends the
test cluster utility to populate these configs in the base configuration
object used to spin up the cluster.
The config key added here ("hbase.master.addrs") is used in the subsequent
patches for HBASE-18095.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* Add a bit of javadoc around SerialReplicationChecker.
* Miniscule edit to the profiler jsp page and then a bit of doc on how to make it work that might help.
* Add some detail if NPE getting BitSetNode to help w/ debug.
* Change HbckChore to log region names instead of encoded names; helps doing diagnostics; can take region name and query in shell to find out all about the region according to hbase:meta.
* Add some fix-it help inline in the HBCK Report page – how to fix.
* Add counts in procedures page so can see if making progress; move listing of WALs to end of the page.
Have the existing scheduleRecoveries launch a new HBCKSCP
instead of SCP. It gets regions to recover from Master
in-memory context AND from a scan of hbase:meta. This
new HBCKSCP is For processing 'Unknown Servers', servers that
are 'dead' and purged but still have references in
hbase:meta. Rare occurance but needs tooling to address.
Later have catalogjanitor take care of these deviations
between Master in-memory and hbase:meta content (usually
because of overdriven cluster with failed RPCs to hbase:meta,
etc)
Changed expireServers in ServerManager so could pass in
custom reaction to expired server.... This is how we
run our custom HBCKSCP while keeping all other aspects
of expiring services (rather than try replicate it
externally).
This patch implements a simple cache that all the masters
can lookup to serve cluster ID to clients. Active HMaster
is still responsible for creating it but all the masters
will read it from fs to serve clients.
RPCs exposing it will come in a separate patch as a part of
HBASE-18095.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
* Clean up a bunch of private variable leakage into other
classes. Reduces visibility as much as possible, providing getters
where access remains necessary or making use of getters that
already exist. There remains an insidious relationship between
`HRegionServer` and `RSRpcServices`.
* Rename `fs` to `dataFs`, `rootDir` as `dataRootDir` so as to
distinguish from the new `walFs`, `walRootDir` (and make it easier
to spot bugs).
* Cleanup or delete a bunch of lack-luster javadoc comments.
* Delete a handful of methods that are unused according to static
analysis.
* Reduces the warning count as reported by IntelliJ from 100 to 7.
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 577db5d7e5 - Test only, removed changes from no more exisiting ScannerCallableWithReplicas class)
Removes the closeRegion flag added by HBASE-23181 and instead
relies on reading meta WALEdit content. Modified how qualifier is
written when the meta WALEdit is for a RegionEventDescriptor
so the 'type' is added to the qualifer so can figure type
w/o having to deserialize protobuf value content: e.g.
HBASE::REGION_EVENT::REGION_CLOSE
Added doc on WALEdit and tried to formalize the 'meta' WALEdit
type and how it works. Needs complete redo in part as suggested
by HBASE-8457. Meantime, some doc and cleanup.
Also changed the LogRoller constructor to remove redundant param.
Because of constructor change, need to change also
TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL &
WALPerformanceEvaluation.java
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
Adds logging of row and complaint if consistency check fails during CJ
checking. Adds a few more null checks. Does edit on the 'HBCK Report'
top line.
Signed-off-by: Reid Chan <reidchan@apache.org>
* better logging on MOB compaction process
* HFileCleanerDelegate to optionally halt removal of mob hfiles
* use archiving when removing committed mob file after bulkload ref failure
closes#763
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
Introducing property hbase.regionserver.user.metrics.enabled(Default:true)
to disable user metrics in case it accounts for any performance issues
Close#661
Signed-off-by: Josh Elser <elserj@apache.org>
This commit adds table name to the logging context when
StochasticLoadBalancer is configured "per table". Added some
test coverage with per-table balancer enabled and manually
verified the logs to make sure the table name is formatted
correctly.
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.com>
Make it so hbase:meta can be altered. TableState for hbase:meta
is kept in Master. State is in-memory transient so if Master
fails, hbase:meta is ENABLED again. hbase:meta schema will be
bootstrapped from the filesystem. Changes to filesystem schema
are atomic so we should be ok if Master fails mid-edit (TBD)
Undoes a bunch of guards that prevented our being able to edit
hbase:meta. At minimmum, need to add in a bunch of WARNING.
TODO: Tests, more clarity around hbase:meta table state, and undoing
references to hard-coded hbase:meta regioninfo.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Throw illegal access exception if you try to use MetaTableAccessor
getting state of the hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
For table state, go to master rather than go to meta direct. Going
to meta won't work for hbase;meta state. Puts load on Master.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
Change isTableDisabled/Enabled implementation to ask the Master instead.
This will give the Master's TableStateManager's opinion rather than
client figuring it for themselves reading meta table direct.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
TODO: Cleanup in here. Go to master for state, not to meta.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
Logging cleanup.
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shutdown access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
Just cleanup.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
Add state holder for hbase:meta.
Removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
Shut down access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
Allow hbase:meta to be disabled.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Allow hbase:meta to be enabled.
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Space quotas has a feature which intends to avoid enacting a space quota
violation policy when only a subset of the Regions for that Table have
reported their space usage (under the assumption that we cannot make an
informed decision if we do not include all regions in our calculations).
This had the unintended side-effect, when a table is disabled as a part
of a violation policy, of causing the regions for that table to not be
reported which disables the violation policy and enables the table.
Need to make sure that when a table is disabled because of a violation
policy that the code does not automatically move that table out of
violation because region sizes are not being reported (because those
regions are not open).
Closes#572
Signed-off-by: Josh Elser <elserj@apache.org>
* Add chaos monkey action for suspend/resume region servers
* Add chaos monkey action for graceful rolling restart
* Add these to relevant chaos monkeys
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
There was a bug in which we would not drop the RegionSizes
for a table in a namespace, where the namespace had a quota
on it. This allowed a scenario in which recreation of a table
inside of a namespace would unintentionally move into violation
despite the table being empty. Need to make sure the RegionSizes
are dropped on table deletion if there is _any_ quota applying
to that table.
Closes#598
Signed-off-by: Josh Elser <elserj@apache.org>
During startup, it's possible that quotas are enabled but the Master has
not yet created the hbase:quotas table.
Closes#559
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
* HBase-22027: Split non-MR related parts of TokenUtil off into a ClientTokenUtil, and move ClientTokenUtil to hbase-client
* Replace uses of deprecated TokenUtil methods with ClientTokenUtil methods. Make methods that don't need to be public package-private
* Don't use reflection where not necessary in TestClientTokenUtil
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Add section to the bulk load complete tool on how it can be
used 'adopting' stray 'orphan' data turned up by hbck2 or
the new reporting facility in the Master UI.
Did a cleanup of BulkLoadHFileTool mostly around usage
pointing back to this new documentation.
Check if overlap is split parent.
Cleaned up the HBCK Report page too with some notes that it is made of
two reports; have the two sections display the same.
Adds deprecations on HBaseFsck and on supporting classes such as
the reporting Interface. Provides alternatives in FSUtils for
progress reporting and deprecates methods that use hbck1 facility.
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* HBASE-22941 merge operation returns parent regions in random order
store and return the merge parent regions in ascending order
remove left over check for exactly two merged regions
add unit test
* use SortedMap type to emphasise that the Map is sorted.
* use regionCount consistently and checkstyle fixes
* Delete tests that expect multiregion merges to fail.
Signed-off-by: stack <stack@apache.org>
* HBASE-22922 Lock all regions to be merged in case of multi region merge
* HBASE-22922 Lock all regions to be merged in case of multi region merge
(addendum)
fix off-by-one error in patch
Signed-off-by: stack <stack@apache.org>
* address checkstyle issue
* change threadPool threads to daemon in server implementation
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
These functions make it easier to possible to use
`org.apache.hadoop.hbase.client.Table.CheckAndMutateBuilder#timeRange`
with more interesting ranges, without being forced to use the
deprecated constructors.
Signed-off-by: huzheng <openinx@gmail.com>
* HBASE-22833: MultiRowRangeFilter should provide a method for creating a filter which is functionally equivalent to multiple prefix filters
* Delete superfluous comments
* Add description for MultiRowRangeFilter constructor
* Add null check for rowKeyPrefixes
* Fix checkstyle
Signed-off-by: huzheng <openinx@gmail.com>
Makes MergeTableRegionsProcedure do more than just two regions at a
time. Compatible as MTRP was done considering one day it'd do more than
two at a time.
Changes hardcoded assumption that merge parent regions are named
mergeA and mergeB in a column on the resultant region. Instead
can have N columns on the merged region, one for each parent
merged. Column qualifiers all being with 'merge'.
Most of code below is undoing the assumption that there are two
parents on a merge only.
This is a first cut at this patch. Implements hold fixing only
currently.
Add a fixMeta method to Hbck Interface.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Bug fix. If hole is on end of last table, I wasn't seeing it.
A hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
Add a general meta fixer class. Explains up top why this stuff doesn't
belong inside MetaTableAccessor.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Break out the filesystem messing so don't have to copy it nor do more
than is needed doing fixup for Region holes.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
Change behavious slightly. If directory exists, don't fail as we did
but try and keep going and create .regioninfo file if missing (or
overwrite if in place). This should make it idempotent. Can rerun
command. Lets see if any repercussions in test suite.
A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMetaFixer.java
Add test.
Signed-off-by: Zheng Hu <openinx@gmail.com>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
This is a first cut at this patch. Implements hold fixing only
currently.
Add a fixMeta method to Hbck Interface.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Bug fix. If hole is on end of last table, I wasn't seeing it.
A hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
Add a general meta fixer class. Explains up top why this stuff doesn't
belong inside MetaTableAccessor.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Break out the filesystem messing so don't have to copy it nor do more
than is needed doing fixup for Region holes.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
Change behavious slightly. If directory exists, don't fail as we did
but try and keep going and create .regioninfo file if missing (or
overwrite if in place). This should make it idempotent. Can rerun
command. Lets see if any repercussions in test suite.
A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMetaFixer.java
Add test.
Signed-off-by: Zheng Hu <openinx@gmail.com>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Refactor of CatalogJanitor so it generates a
Report on the state of hbase:meta when it runs. Also
refactor so CJ runs even if RIT (previous it would
punt on running if RIT) so it can generate a 'Report'
on the interval regardless. If RIT, it just doesn't
go on to do the merge/split GC as it used to.
If report finds an issue, dump as a WARN message
to the master log.
Follow-on is to make the Report actionable/available
for the Master to pull when it goes to draw the hbck
UI page (could also consider shipping the Report as
part of ClusterMetrics?)
Adds new, fatter Visitor to CJ, one that generates
Report on each run keeping around more findings as
it runs.
Moved some methods around so class reads better;
previous methods were randomly ordered in the class.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Make a few handy methods public.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
Add utility as defaults on the Inteface; i.e. is this the first region
in table, is it last, does a passed region come next, or does passed
region overlap this region (added tests for this new stuff).
M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
Bugfix... handle case where buffer passed is null.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Lots of change, reorg., but mostly adding consistency checking
to the visitor used scanning hbase:meta on a period and the
generation of a Report on what the scan has found traversing
hbase:meta. Added a main so could try the CatalogJanitor against
a running cluster.
A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorCluster.java
Fat ugly test for CatalogJanitor consistency checking.
M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
Add tests for new functionality in RI.
M hbase-shell/src/main/ruby/hbase/table.rb
Bug fix for case where meta has a null regioninfo; scan was aborting.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-22610 [BucketCache] Rename hbase.offheapcache.minblocksize.
* HBASE-22610 [BucketCache] Rename hbase.offheapcache.minblocksize. Deprecated old attribute and introduced a new one
* HBASE-22610 [BucketCache] Rename hbase.offheapcache.minblocksize. Removed unnecessary import
* HBASE-22610 added two import configs and removed one
Signed-off-by: Reid Chan <reidchan@apache.org>
Jan Hentschel
Bharath Vissapragada
stack
abhinaba.sarkar
Josh Elser
Viraj Jasani
Jan Hentschel
Geoffrey Jacoby
Michael Stack
Wellington Ramos Chevreuil
Peter Somogyi
Duo Zhang
Nick Dimiduk
dingbaosheng
WenFeiYi
Guanghao Zhang
belugabehr
Aman Poonia
binlijin
Yiran Wu
chenxu14
BukrosSzabolcs
Toshihiro Suzuki
Sean Busbey
Ankit Singhal
ramkrish86
David Mollitor
meiyi
ramkrish86
Sandeep Pal
Lars Hofhansl
Karthik Palanisamy
Junegunn Choi
XinSun
Viraj Jasani
xuqinya1
Karthik Palanisamy
bsglz
Baiqiang Zhao
Udai Bhan Kashyap
lijinbin
ravowlga123
Ankit Singhal
Pierre Zemb
Andrew Purtell
Caroline Zhou
Guangxu Cheng
Lijin Bin
OlegPt
haxiaolin
Sanjeet Nishad
Bharath Vissapragada
Rushabh
shardul-cr7
BukrosSzabolcs
Pankaj
wuqiong
sreenivasulureddy
Stig Døssing
johnhomsea
康智冬
Istvan Toth
Guanghao Zhang
David Manning
Bing Xiao
Sakthi
Tai-Ying Lee
openinx
shahrs87
satanson
Huon Wilson
Daisuke Kobayashi
Itsuki Toyota
syedmurtazahassan
Andor Molnár
Biju Nair
Venkatesh Sridharan
Jeongdae Kim
Reid Chan
Viraj Jasani
Viraj Jasani
Mate Szalay-Beko
Mingliang Liu