Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
13,494 Commits 113 Branches 504 Tags 724 MiB
Commit Graph

392 Commits

Author SHA1 Message Date
praveev c3bf40108d One granularity (#3850) 
* Refactor Segment Granularity

* Beginning of one granularity

* Copy the fix for custom periods in segment-grunalrity over here.

* Remove the custom serialization for now.

* Compilation cleanup

* Reformat code

* Fixing unit tests

* Unify to use a single iterable

* Backward compatibility for rolling upgrade

* Minor check style. Cosmetic changes.

* Rename length and millis to duration

* CR feedback

* Minor changes.
 2017-02-25 01:02:29 -06:00
Parag Jain 1f263fe50b alert when resetting offsets (#3931) 
* alert when resetting offsets

* add more data to alerts
 2017-02-13 13:49:24 -08:00
Parag Jain 8e31a465ad report hand off count finite appenderator driver (#3925) 2017-02-13 10:41:24 -08:00
Gian Merlino 12317fd001 Bump version to 0.10.0-SNAPSHOT. (#3913) 2017-02-06 17:54:35 -08:00
Parag Jain 1aabb45a09 auto reset option for Kafka Indexing service (#3842) 
* auto reset option for Kafka Indexing service in case message at the offset being fetched is not present anymore at kafka brokers

* review comments

* review comments

* reverted last change

* review comments

* review comments

* fix typo
 2017-02-02 14:57:45 -06:00
Parag Jain b3dae0efc3 catch all errors (#3844) 2017-01-24 18:01:30 -07:00
Gian Merlino 6440ddcbca Fix #3795 (Java 7 compatibility). (#3796) 
* Fix #3795 (Java 7 compatibility).

Also introduce Animal Sniffer checks during build, which would
have caught the original problems.

* Add Animal Sniffer on caffeine-cache for JDK8.
 2016-12-21 10:19:13 -08:00
David Lim 0b9dff0bc1 fix worker thread pool exhaustion bug (#3760) 
* fix worker thread pool exhaustion bug

* code review changes

* code review changes
 2016-12-09 15:23:11 -08:00
David Lim 7f087cdd3b allow Kafka consumer group.id to be overriden by config (#3765) 2016-12-08 15:53:13 -08:00
Parag Jain 7ee6bb7410 option to reset offest automatically in case of OffsetOutOfRangeException (#3678) 
* option to reset offset automatically in case of OffsetOutOfRangeException
if the next offset is less than the earliest available offset for that partition

* review comments

* refactoring

* refactor

* review comments
 2016-11-21 16:29:46 -06:00
Gian Merlino bcd20441be Make buildV9Directly the default. (#3688) 2016-11-14 09:29:32 -08:00
Akash Dwivedi 4b3bd8bd63 Migrating java-util from Metamarkets. (#3585) 
* Migrating java-util from Metamarkets.

* checkstyle and updated license on java-util files.

* Removed unused imports from whole project.

* cherry pick metamx/java-util@826021f.

* Copyright changes on java-util pom, address review comments.
 2016-10-21 14:57:07 -07:00
David Lim c2ae734848 KafkaIndexTask: Allow run thread to stop gracefully instead of interrupting (#3534) 
* allow run thread to gracefully complete instead of interrupting when stopGracefully() is called

* add comments
 2016-10-17 10:52:19 -04:00
Parag Jain 592903571a add context to kafka supervisor for the kafka indexing task (#3464) 2016-10-04 20:08:43 -05:00
Parag Jain e419407eba handle supervisor spec metadata failures (#3456) 
close kafka consumer in case supervisor start fails
 2016-10-04 10:15:28 -07:00
Gian Merlino 40f2fe7893 Bump versions to 0.9.3-SNAPSHOT (#3524) 2016-09-29 13:53:32 -07:00
Parag Jain 15c9918c65 log exceptions while trying to pause task (#3504) 2016-09-23 16:53:23 -07:00
David Lim 9226d4af3c configurable shutdownTimeout for Kakfa supervisor (#3497) 
* configurable shutdownTimeout

* cr change
 2016-09-23 13:26:45 -06:00
David Lim ca9114b41b add supervisor reset API (#3484) 
* add supervisor reset API

* CR doc changes and kill running tasks / clear offsets from supervisor
 2016-09-22 17:51:06 -07:00
Himanshu 05ea88df5c fix kafka-indexing-service pom to not reference specific version but parent version for druid core dependencies (#3472) 2016-09-20 15:18:21 -07:00
David Lim 96fcca18ea update KafkaSupervisor to make HTTP requests to tasks in parallel where possible (#3452) 2016-09-20 22:51:15 +05:30
Gian Merlino bcff08826b KafkaIndexTask: Treat null values as unparseable. (#3453) 2016-09-13 10:56:38 -07:00
Jonathan Wei df766b2bbd Add dimension handling interface for ingestion and segment creation (#3217) 
* Add dimension handling interface for ingestion and segment creation

* update javadocs for DimensionHandler/DimensionIndexer

* Move IndexIO row validation into DimensionHandler

* Fix null column skipping in mergerV9

* Add deprecation note for 'numeric_dims' filename pattern in IndexIO v8->v9 conversion

* Fix java7 test failure
 2016-09-12 12:54:02 -07:00
David Lim 146a17de48 KafkaIndexTask: allow pause to break out of retry loop (#3401) 2016-09-06 22:29:37 -06:00
David Lim 5b1ae21bd1 retry calls to getStartTime (#3429) 2016-09-06 14:02:22 -07:00
David Lim ed924bf214 allow registrants to opt out of announcing themselves when registering as a chat handler (#3360) 2016-08-16 10:51:28 +05:30
Gian Merlino 8030f1cb67 Be more respectful of maxRowsInMemory. (#3284) 
- Appenderator: Respect maxRowsInMemory across all sinks.
- KafkaIndexTask: Respect maxRowsInMemory across all partitions.
 2016-07-26 15:02:35 -06:00
David Lim 1d40df4bb7 fix kafka consumer concurrent access during shutdown (#3193) 2016-06-28 13:23:17 -07:00
Gian Merlino ebf890fe79 Update master version to 0.9.2-SNAPSHOT. (#3133) 2016-06-13 13:10:38 -07:00
David Lim 4faa298977 update kafka client for kafka indexing service to 0.9.0.1 (#3109) 2016-06-08 06:51:03 -07:00
David Lim a2290a8f05 support seamless config changes (#3051) 2016-06-03 13:50:19 -07:00
David Lim f6c39cc844 Kafka task minimum message time (#3035) 
* add KafkaIndexTask support for minimumMessageTime

* add Kafka supervisor support for lateMessageRejectionPeriod
 2016-05-31 11:37:00 -07:00
David Lim 3ef24c03b3 Validate X-Druid-Task-Id header in request/response and support retrying on outdated TaskLocation information, add KafkaIndexTaskClient unit tests (#3006) 
* validate X-Druid-Task-Id header in request and add header to response

* modify KafkaIndexTaskClient to take a TaskLocationProvider as the TaskLocation may not remain constant
 2016-05-25 22:05:18 -07:00
Charles Allen 15ccf451f9 Move QueryGranularity static fields to QueryGranularities (#2980) 
* Move QueryGranularity static fields to QueryGranularityUtil
* Fixes #2979

* Add test showing #2979

* change name to QueryGranularities
 2016-05-17 16:23:48 -07:00
David Lim b489f63698 Supervisor for KafkaIndexTask (#2656) 
* supervisor for kafka indexing tasks

* cr changes
 2016-05-04 23:13:13 -07:00
Gian Merlino f8ddfb9a4b Split SegmentInsertAction and SegmentTransactionalInsertAction for backwards compat. (#2922) 
Fixes #2912.
 2016-05-04 13:54:34 -07:00
David Lim 7641f2628f add control and status endpoints to KafkaIndexTask (#2730) 2016-04-21 15:34:59 -07:00
Gian Merlino 08c784fbf6 KafkaIndexTask: Use a separate sequence per Kafka partition in order to make (#2844) 
segment creation deterministic.

This means that each segment will contain data from just one Kafka
partition. So, users will probably not want to have a super high number
of Kafka partitions...

Fixes #2703.
 2016-04-18 22:29:52 -07:00
Gian Merlino 7e7a886f65 Move druid-api into the druid repo. 
This is from druid-api-0.3.17, as of commit 51884f1d05d5512cacaf62cedfbb28c6ab2535cf
in the druid-api repo.
 2016-03-24 11:04:34 -07:00
jon-wei a59c9ee1b1 Support use of DimensionSchema class in DimensionsSpec 2016-03-21 13:12:04 -07:00
Gian Merlino 738dcd8cd9 Update version to 0.9.1-SNAPSHOT. 
Fixes #2462
 2016-03-17 10:34:20 -07:00
Gian Merlino f22fb2c2cf KafkaIndexTask. 
Reads a specific offset range from specific partitions, and can use dataSource metadata
transactions to guarantee exactly-once ingestion.

Each task has a finite lifecycle, so it is expected that some process will be supervising
existing tasks and creating new ones when needed.
 2016-03-10 18:41:43 -08:00