Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
10,469 Commits 111 Branches 503 Tags 728 MiB
Commit Graph

288 Commits

Author SHA1 Message Date
Parag Jain 1aabb45a09 auto reset option for Kafka Indexing service (#3842) 
* auto reset option for Kafka Indexing service in case message at the offset being fetched is not present anymore at kafka brokers

* review comments

* review comments

* reverted last change

* review comments

* review comments

* fix typo
 2017-02-02 14:57:45 -06:00
Parag Jain b3dae0efc3 catch all errors (#3844) 2017-01-24 18:01:30 -07:00
Gian Merlino 6440ddcbca Fix #3795 (Java 7 compatibility). (#3796) 
* Fix #3795 (Java 7 compatibility).

Also introduce Animal Sniffer checks during build, which would
have caught the original problems.

* Add Animal Sniffer on caffeine-cache for JDK8.
 2016-12-21 10:19:13 -08:00
David Lim 0b9dff0bc1 fix worker thread pool exhaustion bug (#3760) 
* fix worker thread pool exhaustion bug

* code review changes

* code review changes
 2016-12-09 15:23:11 -08:00
David Lim 7f087cdd3b allow Kafka consumer group.id to be overriden by config (#3765) 2016-12-08 15:53:13 -08:00
Parag Jain 7ee6bb7410 option to reset offest automatically in case of OffsetOutOfRangeException (#3678) 
* option to reset offset automatically in case of OffsetOutOfRangeException
if the next offset is less than the earliest available offset for that partition

* review comments

* refactoring

* refactor

* review comments
 2016-11-21 16:29:46 -06:00
Gian Merlino bcd20441be Make buildV9Directly the default. (#3688) 2016-11-14 09:29:32 -08:00
Akash Dwivedi 4b3bd8bd63 Migrating java-util from Metamarkets. (#3585) 
* Migrating java-util from Metamarkets.

* checkstyle and updated license on java-util files.

* Removed unused imports from whole project.

* cherry pick metamx/java-util@826021f.

* Copyright changes on java-util pom, address review comments.
 2016-10-21 14:57:07 -07:00
David Lim c2ae734848 KafkaIndexTask: Allow run thread to stop gracefully instead of interrupting (#3534) 
* allow run thread to gracefully complete instead of interrupting when stopGracefully() is called

* add comments
 2016-10-17 10:52:19 -04:00
Parag Jain 592903571a add context to kafka supervisor for the kafka indexing task (#3464) 2016-10-04 20:08:43 -05:00
Parag Jain e419407eba handle supervisor spec metadata failures (#3456) 
close kafka consumer in case supervisor start fails
 2016-10-04 10:15:28 -07:00
Gian Merlino 40f2fe7893 Bump versions to 0.9.3-SNAPSHOT (#3524) 2016-09-29 13:53:32 -07:00
Parag Jain 15c9918c65 log exceptions while trying to pause task (#3504) 2016-09-23 16:53:23 -07:00
David Lim 9226d4af3c configurable shutdownTimeout for Kakfa supervisor (#3497) 
* configurable shutdownTimeout

* cr change
 2016-09-23 13:26:45 -06:00
David Lim ca9114b41b add supervisor reset API (#3484) 
* add supervisor reset API

* CR doc changes and kill running tasks / clear offsets from supervisor
 2016-09-22 17:51:06 -07:00
Himanshu 05ea88df5c fix kafka-indexing-service pom to not reference specific version but parent version for druid core dependencies (#3472) 2016-09-20 15:18:21 -07:00
David Lim 96fcca18ea update KafkaSupervisor to make HTTP requests to tasks in parallel where possible (#3452) 2016-09-20 22:51:15 +05:30
Gian Merlino bcff08826b KafkaIndexTask: Treat null values as unparseable. (#3453) 2016-09-13 10:56:38 -07:00
Jonathan Wei df766b2bbd Add dimension handling interface for ingestion and segment creation (#3217) 
* Add dimension handling interface for ingestion and segment creation

* update javadocs for DimensionHandler/DimensionIndexer

* Move IndexIO row validation into DimensionHandler

* Fix null column skipping in mergerV9

* Add deprecation note for 'numeric_dims' filename pattern in IndexIO v8->v9 conversion

* Fix java7 test failure
 2016-09-12 12:54:02 -07:00
David Lim 146a17de48 KafkaIndexTask: allow pause to break out of retry loop (#3401) 2016-09-06 22:29:37 -06:00
David Lim 5b1ae21bd1 retry calls to getStartTime (#3429) 2016-09-06 14:02:22 -07:00
David Lim ed924bf214 allow registrants to opt out of announcing themselves when registering as a chat handler (#3360) 2016-08-16 10:51:28 +05:30
Gian Merlino 8030f1cb67 Be more respectful of maxRowsInMemory. (#3284) 
- Appenderator: Respect maxRowsInMemory across all sinks.
- KafkaIndexTask: Respect maxRowsInMemory across all partitions.
 2016-07-26 15:02:35 -06:00
David Lim 1d40df4bb7 fix kafka consumer concurrent access during shutdown (#3193) 2016-06-28 13:23:17 -07:00
Gian Merlino ebf890fe79 Update master version to 0.9.2-SNAPSHOT. (#3133) 2016-06-13 13:10:38 -07:00
David Lim 4faa298977 update kafka client for kafka indexing service to 0.9.0.1 (#3109) 2016-06-08 06:51:03 -07:00
David Lim a2290a8f05 support seamless config changes (#3051) 2016-06-03 13:50:19 -07:00
David Lim f6c39cc844 Kafka task minimum message time (#3035) 
* add KafkaIndexTask support for minimumMessageTime

* add Kafka supervisor support for lateMessageRejectionPeriod
 2016-05-31 11:37:00 -07:00
David Lim 3ef24c03b3 Validate X-Druid-Task-Id header in request/response and support retrying on outdated TaskLocation information, add KafkaIndexTaskClient unit tests (#3006) 
* validate X-Druid-Task-Id header in request and add header to response

* modify KafkaIndexTaskClient to take a TaskLocationProvider as the TaskLocation may not remain constant
 2016-05-25 22:05:18 -07:00
Charles Allen 15ccf451f9 Move QueryGranularity static fields to QueryGranularities (#2980) 
* Move QueryGranularity static fields to QueryGranularityUtil
* Fixes #2979

* Add test showing #2979

* change name to QueryGranularities
 2016-05-17 16:23:48 -07:00
David Lim b489f63698 Supervisor for KafkaIndexTask (#2656) 
* supervisor for kafka indexing tasks

* cr changes
 2016-05-04 23:13:13 -07:00
Gian Merlino f8ddfb9a4b Split SegmentInsertAction and SegmentTransactionalInsertAction for backwards compat. (#2922) 
Fixes #2912.
 2016-05-04 13:54:34 -07:00
David Lim 7641f2628f add control and status endpoints to KafkaIndexTask (#2730) 2016-04-21 15:34:59 -07:00
Gian Merlino 08c784fbf6 KafkaIndexTask: Use a separate sequence per Kafka partition in order to make (#2844) 
segment creation deterministic.

This means that each segment will contain data from just one Kafka
partition. So, users will probably not want to have a super high number
of Kafka partitions...

Fixes #2703.
 2016-04-18 22:29:52 -07:00
Gian Merlino 7e7a886f65 Move druid-api into the druid repo. 
This is from druid-api-0.3.17, as of commit 51884f1d05d5512cacaf62cedfbb28c6ab2535cf
in the druid-api repo.
 2016-03-24 11:04:34 -07:00
jon-wei a59c9ee1b1 Support use of DimensionSchema class in DimensionsSpec 2016-03-21 13:12:04 -07:00
Gian Merlino 738dcd8cd9 Update version to 0.9.1-SNAPSHOT. 
Fixes #2462
 2016-03-17 10:34:20 -07:00
Gian Merlino f22fb2c2cf KafkaIndexTask. 
Reads a specific offset range from specific partitions, and can use dataSource metadata
transactions to guarantee exactly-once ingestion.

Each task has a finite lifecycle, so it is expected that some process will be supervising
existing tasks and creating new ones when needed.
 2016-03-10 18:41:43 -08:00