Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-19 00:20:43 +00:00
Code Issues Packages Projects Releases Wiki Activity
7,581 Commits 116 Branches 508 Tags
Commit Graph

817 Commits

Author SHA1 Message Date
Jonathan Wei
880a021a7a Fix missed travis failures from PR 3567 and 2798 (#3761) 
* Fix checkstyle failures from PR 3567

* Fix GranularityPathSpecTest compile failure
 2016-12-07 19:07:31 -08:00
Navis Ryu
f794246ec1 Trimming out outside of given interval (#2798) 
* Trimming out outside of given interval (Fix for #2659)

* addressed comments
 2016-12-07 18:05:50 -08:00
Roman Leventov
c070b4a816 Fix concurrency defects, remove unnecessary volatiles (#3701) 2016-11-22 16:42:28 -08:00
Erik Dubbelboer
7d36f540e8 WIP: Add Google Storage support (#2458) 
Also excludes the correct artifacts from #2741
 2016-11-16 14:06:45 +05:30
Gian Merlino
bcd20441be Make buildV9Directly the default. (#3688) 2016-11-14 09:29:32 -08:00
praveev
52a74cf84f Use timestamp in millis as Map key instead of DateTime object (#3674) 
* Use Long timestamp as key instead of DateTime.

DateTime representation is screwed up when you store with an obj
and read with a different DateTime obj.

For example: The code below fails when you use DateTime as key
```
        DateTime odt = DateTime.now(DateTimeUtils.getZone(DateTimeZone.forID("America/Los_Angeles")));
        HashMap<DateTime, String> map = new HashMap<>();
        map.put(odt, "abc");
        DateTime dt = new DateTime(odt.getMillis());
        System.out.println(map.get(dt));
```

* Respect timezone when creating the file.

* Update docs with timezone caveat in granularity spec

* Remove unused imports
 2016-11-11 10:20:20 -08:00
Himanshu
b76b3f8d85 reset-cluster command to clean up druid state stored on metadata and deep storage (#3670) 2016-11-09 11:07:01 -06:00
Gian Merlino
89d9c61894 Deprecate Aggregator.getName and AggregatorFactory.getAggregatorStartValue. (#3572) 2016-10-31 15:24:30 -07:00
Akash Dwivedi
6a845e1f7b Adding getDelegate() to directly access delegate. (#3616) 
👍
 2016-10-27 15:57:36 -07:00
Akash Dwivedi
4b3bd8bd63 Migrating java-util from Metamarkets. (#3585) 
* Migrating java-util from Metamarkets.

* checkstyle and updated license on java-util files.

* Removed unused imports from whole project.

* cherry pick metamx/java-util@826021f.

* Copyright changes on java-util pom, address review comments.
 2016-10-21 14:57:07 -07:00
Gian Merlino
dd0bb6da1e Unit test for #3544: Avoid exceptions for dataSource spec when using s3. (#3571) 2016-10-17 12:41:43 -07:00
Navis Ryu
4554c1214b Avoid exceptions for dataSource spec when using s3 (#3544) 2016-10-14 18:24:19 -07:00
Akash Dwivedi
078de4fcf9 Use explicit version from HadoopIngestionSpec. (#3554) 2016-10-07 13:59:14 -07:00
praveev
43cdc675c7 Add support for timezone in segment granularity (#3528) 
* Add support for timezone in segment granularity

* CR feedback. Handle null timezone during equals check.

* Include timezone in docs.
Add timezone for ArbitraryGranularitySpec.
 2016-10-03 08:15:42 -07:00
Gian Merlino
40f2fe7893 Bump versions to 0.9.3-SNAPSHOT (#3524) 2016-09-29 13:53:32 -07:00
Fokko Driesprong
67920c114e Fixed info message (#3481) 2016-09-21 15:50:29 -07:00
Gian Merlino
27bd5cb13a Add forceExtendableShardSpecs option to Hadoop indexing, IndexTask. (#3473) 
Fixes #3241.
 2016-09-21 13:40:04 -06:00
Slim
ba6ddf307e Adding hadoop kerberos authentification. (#3419) 
* adding kerberos authentication

* make the 2 functions identical
 2016-09-13 10:42:50 -07:00
Jonathan Wei
df766b2bbd Add dimension handling interface for ingestion and segment creation (#3217) 
* Add dimension handling interface for ingestion and segment creation

* update javadocs for DimensionHandler/DimensionIndexer

* Move IndexIO row validation into DimensionHandler

* Fix null column skipping in mergerV9

* Add deprecation note for 'numeric_dims' filename pattern in IndexIO v8->v9 conversion

* Fix java7 test failure
 2016-09-12 12:54:02 -07:00
Himanshu
3b6c81e7c0 fix cleanup of hadoop ingestion intermediate path (#3385) 2016-09-08 01:36:56 +05:30
Dave Li
c4e8440c22 Adds long compression methods (#3148) 
* add read

* update deprecated guava calls

* add write and vsizeserde

* add benchmark

* separate encoding and compression

* add header and reformat

* update doc

* address PR comment

* fix buffer order

* generate benchmark files

* separate encoding strategy and format

* fix benchmark

* modify supplier write to channel

* add float NONE handling

* address PR comment

* address PR comment 2
 2016-08-30 16:17:46 -07:00
Hamlet Lee
e4f0eac8e6 Fix issue #2707 (#2708) 2016-08-16 12:19:44 -05:00
Gian Merlino
a2bcd97512 IncrementalIndex: Fix multi-value dimensions returned from iterators. (#3344) 
They had arrays as values, which MapBasedRow doesn't understand and
toStrings rather than converting to lists.
 2016-08-10 08:47:29 -07:00
Gian Merlino
9437a7a313 HLL: Avoid some allocations when possible. (#3314) 
- HLLC.fold avoids duplicating the other buffer by saving and restoring its position.
- HLLC.makeCollector(buffer) no longer duplicates incoming BBs.
- Updated call sites where appropriate to duplicate BBs passed to HLLC.
 2016-08-03 18:08:52 -07:00
kaijianding
50d52a24fc ability to not rollup at index time, make pre aggregation an option (#3020) 
* ability to not rollup at index time, make pre aggregation an option

* rename getRowIndexForRollup to getPriorIndex

* fix doc misspelling

* test query using no-rollup indexes

* fix benchmark fail due to jmh bug
 2016-08-02 11:13:05 -07:00
kaijianding
3dc2974894 Add timestampSpec to metadata.drd and SegmentMetadataQuery (#3227) 
* save TimestampSpec in metadata.drd

* add timestampSpec info in SegmentMetadataQuery
 2016-07-25 15:45:30 -07:00
Navis Ryu
cd7337fc8a Calculate max split size based on numMapTask in DatasourceInputFormat (#2882) 
* Calculate max split size based on numMapTask

* updated docs & fixed possible ArithmeticException
 2016-07-20 16:53:51 -07:00
Hyukjin Kwon
55e7a52475 Replace deprecated usage for StringInputRowParser and JSONParseSpec (#3215) 2016-07-14 09:19:17 -07:00
Gian Merlino
ea03906fcf Configurable compressRunOnSerialization for Roaring bitmaps. (#3228) 
Defaults to true, which is a change in behavior (this used to be false and unconfigurable).
 2016-07-08 10:24:19 +05:30
Hyukjin Kwon
45f553fc28 Replace the deprecated usage of NoneShardSpec (#3166) 2016-06-25 10:27:25 -07:00
Nishant
2696b0c451 Retry for transient exceptions while doing cleanup for Hadoop Jobs (#3177) 
* fix 1828

fixes https://github.com/druid-io/druid/issues/1828

* remove unused import

* Review comment
 2016-06-23 13:38:47 -07:00
Nishant
6f330dc816 Better handling for parseExceptions for Batch Ingestion (#3171) 
* Better handling for parseExceptions

* make parseException handling consistent with Realtime

* change combiner default val to true

* review comments

* review comments
 2016-06-22 16:38:29 -07:00
Gian Merlino
ebf890fe79 Update master version to 0.9.2-SNAPSHOT. (#3133) 2016-06-13 13:10:38 -07:00
Nishant
778f97a80e attempt to fix-2906 (#2985) 
* attempt to fix-2984

* review comments

* Add test
 2016-05-18 15:12:38 -05:00
Charles Allen
15ccf451f9 Move QueryGranularity static fields to QueryGranularities (#2980) 
* Move QueryGranularity static fields to QueryGranularityUtil
* Fixes #2979

* Add test showing #2979

* change name to QueryGranularities
 2016-05-17 16:23:48 -07:00
David Lim
b489f63698 Supervisor for KafkaIndexTask (#2656) 
* supervisor for kafka indexing tasks

* cr changes
 2016-05-04 23:13:13 -07:00
Navis Ryu
49ef4d96cb Merge pull request #2802 from navis/optimize_multiplepath_concat 
Optimize adding lots of paths to pathspec
 2016-04-11 23:35:28 -05:00
jon-wei
0e481d6f93 Allow filters to use extraction functions 2016-04-05 13:24:56 -07:00
Gian Merlino
977e867ad8 Downgrade geoip2, exclude com.google.http-client. 
Reverts "Update com.maxmind.geoip2 to 2.6.0" and exclude the google http client
from com.maxmind.geoip2. This should satisfy the original need from #2646 (wanting
to run Druid along with an upgraded com.google.http-client) while preventing
Jackson conflicts pointed out in #2717.

Fixes #2717.

This reverts commit 21b7572533592f1700f86379483d87e9e340f2a7.
 2016-03-25 14:43:22 -07:00
Gian Merlino
ff25325f3b Improved docs for multi-value dimensions. 
- Add central doc for multi-value dimensions, with some content from other docs.
- Link to multi-value dimension doc from topN and groupBy docs.
- Fixes a broken link from dimensionspecs.md, which was presciently already
  linking to this nonexistent doc.
- Resolve inconsistent naming in docs & code (sometimes "multi-valued", sometimes
  "multi-value") in favor of "multi-value".
 2016-03-22 14:40:55 -07:00
Himanshu
00d7021291 Merge pull request #2607 from jon-wei/dim_schema 
Support use of DimensionSchema class in DimensionsSpec
 2016-03-22 11:53:46 -05:00
Himanshu
3220b109ad Merge pull request #2570 from binlijin/single_dimension_partitioning 
Single dimension hash-based partitioning
 2016-03-22 11:51:06 -05:00
binlijin
bce600f5d5 Single dimension hash-based partitioning 2016-03-22 13:15:33 +08:00
jon-wei
a59c9ee1b1 Support use of DimensionSchema class in DimensionsSpec 2016-03-21 13:12:04 -07:00
Gian Merlino
738dcd8cd9 Update version to 0.9.1-SNAPSHOT. 
Fixes #2462
 2016-03-17 10:34:20 -07:00
Himanshu
ea3281ad78 Merge pull request #2645 from atomx/gs-scheme 
Add gs:// hdfs support
 2016-03-14 22:15:42 -05:00
Erik Dubbelboer
375620cfb3 Add gs:// hdfs support 
Used to access google cloud storage
 2016-03-12 08:57:57 +00:00
Gian Merlino
187569e702 DataSource metadata. 
Geared towards supporting transactional inserts of new segments. This involves an
interface "DataSourceMetadata" that allows combining of partially specified metadata
(useful for partitioned ingestion).

DataSource metadata is stored in a new "dataSource" table.
 2016-03-10 17:41:50 -08:00
Fangjin Yang
1e49092ce7 Merge pull request #2627 from himanshug/fix_datasource_inputformat_locations 
fix regression - bug in DatasourceInputFormat best effort split location finder code
 2016-03-10 13:46:04 -08:00
Himanshu Gupta
eab8a0b54d in DatasourceInputFormat code for determining segment block locations avoid the split calulation by helper TextInputFormat 2016-03-10 14:28:53 -06:00