94 Commits

Author SHA1 Message Date
Jerry He
2765b9d9d9 HBASE-16667 Building with JDK 8: ignoring option MaxPermSize=256m (Niels Basjes) 2016-09-24 16:07:25 -07:00
stack
45bb6180a3 REVERT of revert of "HBASE-16308 Contain protobuf references Gather up the pb references into a few locations only rather than have pb references distributed all about the code base."
This is a revert of a revert; i.e. we are adding back the change only adding
back with fixes for the broken unit test; was a real issue on a test that
went in just at same time as this commit; I was getting a new nonce on each
retry rather than getting one for the mutation.

Other changes since revert are more hiding of RpcController. Use
accessor method rather than always pass in a RpcController

Walked back retrying operations that used to be single-shot (though
code comment said need a retry) because it opens a can of worms where
we retry stuff like bad column family when we shouldn't (needs
work adding in DoNotRetryIOEs)

Changed name of class from PayloadCarryingServerCallable to
CancellableRegionServerCallable.

Fix javadoc and findbugs warnings.

Fix case of not initializing the ScannerCallable RpcController.

Below is original commit message:

 Remove mention of ServiceException and other protobuf classes from all over the codebase.
 Purge TimeLimitedRpcController. Lets just have one override of RpcController.
        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable.java
         Cleanup. Make it clear this is an odd class for async hbase intro.
        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
         Refactor of RegionServerCallable allows me clean up a bunch of
         boilerplate in here and remove protobuf references.
        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          Purge protobuf references everywhere except a reference to a throw of a
          ServiceException in method checkHBaseAvailable. I deprecated it in favor
          of new available method (the SE is not actually needed)
        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/PayloadCarryingServerCallable.java
         Move the RetryingTimeTracker instance in here from HTable.
         Allows me to contain tracker and remove a repeated code in HTable.
        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java
         Clean up move set up of rpc in here rather than have it repeat in HTable.
         Allows me to remove protobuf references from a bunch of places.
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/FlushRegionCallable.java
     Make use of the push of boilerplate up into RegionServerCallable
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/PayloadCarryingServerCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/SecureBulkLoadClient.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
     Move boilerplate up into superclass.
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingTimeTracker.java
     Cleanup
    M hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/PayloadCarryingRpcController.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
     Factor in TimeLimitedRpcController. Just have one RpcController override.
    D hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/TimeLimitedRpcController.java
     Removed. Lets have one override of pb rpccontroller only.
    M hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
     (handleRemoteException) added
     (toText) added
2016-08-10 10:12:06 -07:00
stack
0206dc67d6 Revert "HBASE-16308 Contain protobuf references Gather up the pb references into a few locations only rather than have pb references distributed all about the code base."
This reverts commit ed87a81b4b61c4842c12572a47c97ae23773012f.
2016-08-05 15:18:48 -07:00
stack
ed87a81b4b HBASE-16308 Contain protobuf references Gather up the pb references into a few locations only rather than have pb references distributed all about the code base.
Purge ServiceException from Callable subclasses by pushing SE handling
up into the parent Callable class (varies by context but this is basic
patten). Allows us remove a bunch of boilerplate.
Do this in the public facing classes in particular (though if
an API has SE in it -- which a few do, this patch leaves these
untouched -- for now.) Make it so HBaseAdmin and HTable have no
direct pb imports (except for endpoint processor API).

Change a few of the HBaseAdmin calls to be retrying where comments
ask that we do retry rather than one time.

Purge TimeLimitedRpcController. Lets just have one override of RpcController.

        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable.java
         Cleanup. Make it clear this is an odd class for async hbase intro.

        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
         Refactor of RegionServerCallable allows me clean up a bunch of
         boilerplate in here and remove protobuf references.

        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
          Purge protobuf references everywhere except a reference to a throw of a
          ServiceException in method checkHBaseAvailable. I deprecated it in favor
          of new available method (the SE is not actually needed)

        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/PayloadCarryingServerCallable.java
         Move the RetryingTimeTracker instance in here from HTable.
         Allows me to contain tracker and remove a repeated code in HTable.

        M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionServerCallable.java
         Clean up move set up of rpc in here rather than have it repeat in HTable.
         Allows me to remove protobuf references from a bunch of places.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/FlushRegionCallable.java
     Make use of the push of boilerplate up into RegionServerCallable

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/MultiServerCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/PayloadCarryingServerCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionAdminServiceCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/SecureBulkLoadClient.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java
     Move boilerplate up into superclass.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RetryingTimeTracker.java
     Cleanup

    M hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/PayloadCarryingRpcController.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALEditsReplaySink.java
    M hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
     Factor in TimeLimitedRpcController. Just have one RpcController override.

    D hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/TimeLimitedRpcController.java
     Removed. Lets have one override of pb rpccontroller only.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
     (handleRemoteException) added
     (toText) added

Signed-off-by: stack <stack@apache.org>
2016-08-05 10:13:58 -07:00
Sean Busbey
c59f76485e HBASE-16340 exclude Xerces iplementation jars from coming in transitively.
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2016-08-05 10:16:29 -05:00
Sean Busbey
60b79e2daa HBASE-16321 ensure no findbugs-jsr305
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2016-08-05 10:16:29 -05:00
stack
ff7d0082b8 HBASE-16306 Add specific imports to avoid namespace clash in defaultSource.scala (Sai Teja Ranuva) 2016-07-29 13:43:16 -07:00
stack
9d740f7b8b HBASE-16263 Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
Signed-off-by: stack <stack@apache.org>
2016-07-21 10:02:05 -07:00
Apekshit
4ffea7711a HBASE-15944 Spark test flooding mvn output. Redirect test logs to file. This doesn't fix the problem fully as I still see few logs being dumped in stdout. But it cleans up majority of the earlier dump. (Apekshit)
Change-Id: I6893301d154078a7cfb6b9af2eedc744deafb8d7

Signed-off-by: stack <stack@apache.org>
2016-06-02 08:44:08 -07:00
Jurriaan Mous
cdd532da8a HBASE-15610 Remove deprecated HConnection for 2.0 thus removing all PB references for 2.0
Signed-off-by: stack <stack@apache.org>
2016-05-29 07:50:55 -07:00
tedyu
084b036cb2 HBASE-15825 Fix the null pointer in DynamicLogicExpressionSuite (Zhan Zhang) 2016-05-16 09:04:50 -07:00
Jonathan M Hsieh
b353e388bb HBASE-15333 [hbase-spark] Enhance the dataframe filters to handle naively encoded short, integer, long, float and double (Zhan Zhang) 2016-05-13 07:00:41 -07:00
tedyu
ebb5d421f9 HBASE-15707 ImportTSV bulk output does not support tags with hfile.format.version=3 (huaxiang sun) 2016-04-26 11:21:29 -07:00
Apekshit
7efb9edecb HBASE-15296 Break out writer and reader from StoreFile. Done using Intellij15 Refactor > Move. (Apekshit)
Change-Id: Ie719569cc3393e0b5361e9d462c3cf125ad5144e

Signed-off-by: stack <stack@apache.org>
2016-04-13 22:43:03 -07:00
Weiqing Yang
58177c103f HBASE-15572 Adding optional timestamp semantics to HBase-Spark
4 parameters, "timestamp", "minTimestamp", "maxiTimestamp" and
"maxVersions" are added to HBaseSparkConf. Users can select a
timestamp, they can also select a time range with minimum timestamp and
maximum timestamp.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Ted Yu <tedyu@apache.org>
Signed-off-by: Jerry He <jerryjch@apache.org>
2016-04-13 22:39:14 -05:00
Sean Busbey
6905d272d3 Revert "HBASE-15572 Adding optional timestamp semantics to HBase-Spark (Weiqing Yang)"
This reverts commit eec27ad7ef7b5078f705301bd3042991d4d4b4d9.
2016-03-31 21:40:50 -05:00
tedyu
eec27ad7ef HBASE-15572 Adding optional timestamp semantics to HBase-Spark (Weiqing Yang) 2016-03-31 19:08:33 -07:00
tedyu
2b8a7f8d7b HBASE-15334 Add avro support for spark hbase connector (Zhan Zhang) 2016-03-17 09:11:38 -07:00
tedyu
f6945c4631 HBASE-15336 Support Dataframe writer to the spark connector (Zhan Zhang) 2016-03-10 06:44:29 -08:00
Jonathan M Hsieh
97cce850fe HBASE-14801 Enhance the Spark-HBase connector catalog with json format (Zhan Zhang) 2016-03-09 10:41:56 -08:00
Ted Malaska
b29ce7f114 HBASE-15271 Spark bulk load should write to temporary location and then rename on success.
Signed-off-by: Ted Yu <tedyu@apache.org>
Signed-off-by: Jonathan Hsieh <jon@cloudera.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2016-03-08 17:28:38 -08:00
Jonathan M Hsieh
20e14f449a HBASE-15310 hbase-spark module has compilation failures with clover profile 2016-02-24 11:54:43 -08:00
tedyu
00248656ee HBASE-15184 SparkSQL Scan operation doesn't work on kerberos cluster (Ted Malaska) 2016-02-23 16:52:13 -08:00
Jonathan M Hsieh
f352f3c371 HBASE-15282 Bump hbase-spark to use Spark 1.6.0 2016-02-18 17:31:42 -08:00
Jerry He
3aff98c75b HBASE-15201 Add hbase-spark to hbase assembly 2016-02-08 14:13:46 -08:00
Apekshit
5e89ebcc2f HBASE-14159 Add dependency on hadoop-mapreduce-client-jobclient from hbase-spark.
Adds a test-scope dependency to get rid of following error:

[WARNING] warning: Class org.apache.hadoop.mapred.MiniMRCluster not found - continuing with a stub.
[WARNING] one warning found

Signed-off-by: Sean Busbey <busbey@cloudera.com>
2016-01-12 07:48:42 -06:00
tedyu
6868c63660 HBASE-14796 Enhance the Gets in the connector (Zhan Zhang) 2015-12-28 15:48:10 -08:00
tedyu
e75e26e3c6 HBASE-14849 Add option to set block cache to false on SparkSQL executions (Zhan Zhang) 2015-12-19 15:14:58 -08:00
Jonathan M Hsieh
bbfff0d072 HBASE-14991 Fix the '-feature' warning in scala build (Zhan Zhang) 2015-12-17 14:19:36 -08:00
tedyu
676ce01c82 HBASE-14795 Enhance the spark-hbase scan operations (Zhan Zhang) 2015-12-13 18:26:54 -08:00
tedyu
2317ec583d HBASE-14894 Fix misspellings of threshold in log4j.properties files for tests (Youngjoon Kim) 2015-11-28 10:08:38 -08:00
Daniel Vimont
6a493ddff7 Commit for HBASE-14825 -- corrections of typos, misspellings, and mangled links
Commit for HBASE-14825 -- corrections of typos, misspellings, and mangled links - addition commit for line lengths
2015-11-25 07:12:36 +10:00
Matteo Bertozzi
55087ce888 HBASE-14863 Add missing test/resources/log4j files in hbase modules 2015-11-22 07:31:16 -08:00
Andrew Purtell
ca1048415b HBASE-14340 Add second bulk load option to Spark Bulk Load to send puts as the value (Ted Malaska) 2015-11-17 13:48:47 -08:00
stack
950050e9d6 Revert "HBASE-14728 TestRowCounter is broken in master (Abhishek Singh Chouhan)"
Revert because I overcommitted, included files from another patch.

This reverts commit 47659200ac1d53590379ff431c428dffc07fd06a.
2015-11-02 08:06:34 -08:00
stack
47659200ac HBASE-14728 TestRowCounter is broken in master (Abhishek Singh Chouhan) 2015-11-01 22:22:59 -08:00
tedyu
dae1775a11 HBASE-14406 The dataframe datasource filter is wrong, and will result in data loss or unexpected behavior (Ted Malaska) 2015-10-16 11:22:07 -07:00
Sean Busbey
e3383fd146 HBASE-14515 make sure spark tests are skippable like the other modules. 2015-09-30 08:35:11 -05:00
stack
a47ff1d998 HBASE-14447 Spark tests failing: bind exception when putting up info server 2015-09-17 09:12:48 -07:00
Sean Busbey
fda317cebb HBASE-14377 JavaHBaseContextSuite isn't being run. 2015-09-11 00:34:55 -05:00
Ted Malaska
e95358a7fc HBASE-14181 Add Spark DataFrame DataSource to HBase-Spark Module
Signed-off-by: Sean Busbey <busbey@cloudera.com>
2015-09-07 09:44:10 -05:00
tedyu
72a48a1333 HBASE-14150 Add BulkLoad functionality to HBase-Spark Module (Ted Malaska) 2015-08-12 08:29:03 -07:00
tedyu
e53d2481ee HBASE-14184 Fix indention and type-o in JavaHBaseContext (Ted Malaska) 2015-08-07 11:02:24 -07:00
Ted Malaska
30f7d127c3 HBASE-13992 Integrate SparkOnHBase into HBase
Signed-off-by: Sean Busbey <busbey@cloudera.com>
2015-07-28 11:45:23 -05:00