This is causing me issues with parallel test runs.
Also allow setting the surefire reports and temp directories via command line.
Signed-off-by: stack <stack@apache.org>
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
The Hadoop AccessControlList allows us to specify admins of the webUI
via a list of users and/or groups. Admins of the WebUI can mutate the
system, potentially seeing sensitive data or modifying the system.
hbase.security.authentication.spnego.admin.users is a comma-separated
list of users who are admins.
hbase.security.authentication.spnego.admin.groups is a comma-separated
list of groups whose membership are admins. Either of these
configuration properties may also contain an asterisk (*) which denotes
"any entity" (e.g user, group).
Previously, when a user was denied from some endpoint that was
designated for admins, they received an HTTP/401. In this case, it is
more correct to return HTTP/403 as they were correctly authenticated,
but they were disallowed from fetching the given resource. This commit
incorporates this change.
hbase.security.authentication.ui.config.protected also exists for users
who have sensitive information stored in the Hadoop service
configuration and want to limit access to this endpoint. By default,
the Hadoop configuration endpoint is not protected and any
authenticated user can access it.
The test is based off of work by Nihal Jain in HBASE-20472.
Co-authored-by: Nihal Jain <nihaljain.cs@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
Complains about mismatch in types when Compare. Implement Compare in
base Interface.
hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Complains pbs never return null.
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java
Needed redo because errorprone complains can't mock Service from guava.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
Unrelated...adding one-liner debug statements chasing other test
failures.
HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'.
Just remove the check state from the getClusterStatus call.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthi@apache.org>
(cherry picked from commit dd8496a546)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Codecs don't have access to what CellComparator to use. Backfill.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
Adds a new compareRows with default implementation that takes a ByteBuffer.
Needed by the index in a block encoder implementation.
M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
Adds implementation for meta of new compareRows method. Adds utility
method for figuring comparator based off tablename.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
Comparator is in context. Remove redundant handling.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
Comparator is in context. Remove redundant handling. Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
Clean javadoc.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
Cache context so can use it to get comparator to use later.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
Cache cellcomparator to use. Javdoc on diff between HFileContext and
HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
Add CellComparator
M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
Remove comparator caching. Get from context instead.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
Skip a reflection if we can.
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
Javadoc. Removed unused filed.
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
Make hbase:meta region schema dynamic.
Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shut down access to internals and removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Cleanup/refactor section on replica-handling.
M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
Get hbase:meta schema from filesystem rather than from hard-coding.
Adds a display of the content of 'hbase:meta' to the Master's
table.jsp, when that table is selected. Supports basic pagination,
filtering, &c.
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.
Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.
Closes#884
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
The current process for an operator, after fixing holes in meta, is to
manually disable and enable the whole table. Let's try to avoid
bringing the whole table offline if we can. Have the master attempt to
queue up assignment procedures for any new regions it creates.
Signed-off-by: stack <stack@apache.org>
The cleanupBulkLoad method is only called for the first Region in the
table which was being bulk loaded into. This means that potentially N-1
other RegionServers (where N is the number of RegionServers) will leak
one FileSystem object into the FileSystem cache which will never be
cleaned up. We need to do this clean-up as a part of secureBulkLoadHFiles
otherwise we cannot guarantee that heap usage won't grow unbounded.
Closes#1029
Signed-off-by: Sean Busbey <busbey@apache.org>
Make it so hbase:meta can be altered. TableState for hbase:meta
was hardcoded ENABLED. Make it dynamic. State is now kept in
current active Master. It is transient so falls back to default
if Master crashes. Add to registry a getMetaTableState
which reads mirrored state from zookeeper (NOT from Master and
defaults ENABLED if no implementation or error fetching state).
hbase:meta schema will be bootstrapped from the filesystem.
Changes to filesystem schema are atomic so we should be ok if
Master fails mid-edit (TBD). Undoes a bunch of guards that
prevented our being able to edit hbase:meta.
TODO: Tests, more clarity around hbase:meta table state, and undoing
references to hard-coded hbase:meta regioninfo.
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Throw illegal access exception if you try to use MetaTableAccessor
getting state of the hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
Add fetching of hbase:meta table state from registry. Adds cache of
tablestates w/ a ttl of 1 second (adjustable).
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
Add querying registry for hbase:meta table state.
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
Add querying of mirrored table state for hbase:meta table.
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
Shutdown access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
Just cleanup.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
Add state holder for hbase:meta. Removed unused methods.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
Shut down access.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
Allow hbase:meta to be disabled.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
Allow hbase:meta to be enabled.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Master force-closes unknown/incorrect Regions OPEN on RS
M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
Added a note and small refactor.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
Fix an NPE when CJ ran.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Minor clean up of log message; make it clearer.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
Make it so closeRegionSilentlyAndWait can be used w/o timeout.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
If a RegionServer Report notes a Region is OPEN and the Master does not
know of said Region, close it (We used to crash out the RegionServer)
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
Minor tweak of toString -- label should be state, not rit (confusing).
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
Doc.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java
Add region name to exception.
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java
Be more careful about which Regions we queue up for reassign. This
procedure is run by the operator so could happen at any time. We
will likely be running this when Master has some accounting of
cluster members so check its answers for what Regions were on
server before running.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Doc and we were misrepresenting the case where a Region as not in RIT
when we got CLOSE -- we were reporting it as though it was already
trying to CLOSE.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Change its behavior so it will only look in hbase:meta
if the call to the super class turns up zero references.
Only then will it search hbase:meta for references to
'Unknown Servers'. Normal operation where we read Master
context is usual and sufficient. The scan of hbase:meta
is only for case where Master state has been corrupted
and we need to clear out 'Unknown Servers'.
Having it as static means the test cannot be parameterized (ran into
this issue in HBASE-23305). That happens because the field is not
reset between parameterized runs.
* Adds a new MapReduce job that builds a report of health of mob files
* Also builds background information on mob system use
* add a basic mob architecture in the ref guide to explain how mob takes the reference hfile value and finds the actual cell contents
* add a troubleshooting mob section to the ref guide to explain how to do a mob reference scan.
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
* Clean up JavaDocs
* Clean up logging and error messages
* Remove superfluous code
* Replace static code with library call
* Do not swallow Interrupted Exceptions
* Use try-with-resources
* User multi-Exception catches to reduce code size
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Removes a bunch of dead code and fixes some checkstyle nits.
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
Mark Robert Miller
Duo Zhang
stack
Jan Hentschel
Michael Stack
Bharath Vissapragada
abhinaba.sarkar
Josh Elser
Viraj Jasani
Jan Hentschel
Geoffrey Jacoby
Wellington Ramos Chevreuil
Peter Somogyi
Nick Dimiduk
dingbaosheng
WenFeiYi
Guanghao Zhang
belugabehr
Aman Poonia
binlijin
Yiran Wu
chenxu14
BukrosSzabolcs
Toshihiro Suzuki
Sean Busbey
Ankit Singhal
ramkrish86
David Mollitor
meiyi
ramkrish86