Commit Graph

7167 Commits

Author SHA1 Message Date
Gian Merlino 79a95f7789 WorkerTaskMonitor: stop() waits for mainLoop to exit.
Fixes #2637.
2016-03-11 11:40:13 -08:00
Himanshu 61cd838d6b Merge pull request #2640 from guobingkun/fix_doc
fix broken link for Tasks
2016-03-11 12:55:04 -06:00
Bingkun Guo 96c981cd0a fix broken link for Tasks 2016-03-11 11:36:34 -06:00
Gian Merlino 05397a9b4f ThreadPoolTaskRunner: Make graceful shutdown logs less scary.
- It's okay to suppress InterruptedException during graceful shutdown, as
  tasks may use it to accelerate their own shutdown.
- It's okay to ignore return statuses during graceful shutdown (which may
  be FAILED!) because it actually doesn't matter what they are.
2016-03-11 07:49:29 -08:00
Fangjin Yang f4ab1c2e52 Merge pull request #2632 from gianm/examples-druid-provided
examples: Switch druid-server, druid-common to "provided".
2016-03-10 19:57:32 -08:00
Gian Merlino d63473e0d5 examples: Switch druid-server, druid-common to "provided". 2016-03-10 18:43:29 -08:00
Gian Merlino f22fb2c2cf KafkaIndexTask.
Reads a specific offset range from specific partitions, and can use dataSource metadata
transactions to guarantee exactly-once ingestion.

Each task has a finite lifecycle, so it is expected that some process will be supervising
existing tasks and creating new ones when needed.
2016-03-10 18:41:43 -08:00
Gian Merlino 187569e702 DataSource metadata.
Geared towards supporting transactional inserts of new segments. This involves an
interface "DataSourceMetadata" that allows combining of partially specified metadata
(useful for partitioned ingestion).

DataSource metadata is stored in a new "dataSource" table.
2016-03-10 17:41:50 -08:00
Gian Merlino 3d2214377d Appenderatoring.
Appenderators are a way of getting more control over the ingestion process
than a Plumber allows. The idea is that existing Plumbers could be implemented
using Appenderators, but you could also implement things that Plumbers can't do.

FiniteAppenderatorDrivers help simplify indexing a finite stream of data.

Also:
- Sink: Ability to consider itself "finished" vs "still writable".
- Sink: Ability to return the number of rows contained within the sink.
2016-03-10 17:41:50 -08:00
Gian Merlino 08284fea62 Publish test-jar for indexing-service. 2016-03-10 16:50:37 -08:00
Gian Merlino 92c828f904 Make SegmentHandoffNotifier Closeable. 2016-03-10 16:50:37 -08:00
Gian Merlino ad5ffdf483 Nix Committers.supplierOf; Suppliers.ofInstance is good enough. 2016-03-10 16:50:37 -08:00
Gian Merlino 8a11161b20 Plumbers: Move plumber.add out of try/catch for ParseException.
The incremental indexes handle that now so it's not necessary.

Also, add debug logging and more detailed exceptions to the incremental
indexes for the case where there are parse exceptions during aggregation.
2016-03-10 16:39:26 -08:00
Himanshu Gupta dc0214bddb while GroupBy merging use unsorted facts in IncrementalIndex wherever possible 2016-03-10 16:11:48 -06:00
Himanshu Gupta 02dfd5cd80 update IncrementalIndex to support unsorted facts map that can be used in groupBy merging to improve performance 2016-03-10 16:11:48 -06:00
Fangjin Yang 1e49092ce7 Merge pull request #2627 from himanshug/fix_datasource_inputformat_locations
fix regression - bug in DatasourceInputFormat best effort split location finder code
2016-03-10 13:46:04 -08:00
Xavier Léauté 90d7409e1a Merge pull request #2611 from himanshug/gp_by_max_limit
only allow lowering maxResults and maxIntermediateRows from groupBy query context
2016-03-10 13:44:13 -08:00
Himanshu 863aa66808 Merge pull request #2597 from metamx/forwardPortMMX23
Forward port - Fix dependency problems
2016-03-10 14:56:52 -06:00
Himanshu Gupta eab8a0b54d in DatasourceInputFormat code for determining segment block locations avoid the split calulation by helper TextInputFormat 2016-03-10 14:28:53 -06:00
Charles Allen 7b1bfbf704 Add documentation to modules about what should be excluded. 2016-03-10 10:18:33 -08:00
Charles Allen d299540efc Make HadoopTask load hadoop dependency classes LAST for local isolated classrunner 2016-03-10 10:18:23 -08:00
Nishant ba1185963b Fix a bunch of dependencies
* Eliminate exclusion groups from pull-deps
* Only consider dependency nodes in pull-deps if they are not in the following scopes
	* provided
	* test
	* system
* Fix a bunch of `<scope>provided</scope>` missing tags
* Better exclusions for a couple of problematic libs
2016-03-10 10:18:08 -08:00
Fangjin Yang cf3965c82e Merge pull request #2625 from gianm/clarify-parser-docs
Clarify parser docs.
2016-03-10 09:44:23 -08:00
Gian Merlino a2b1652787 Clarify parser docs.
- Clarify what parseSpecs are used for.
- Avro, Protobuf should use timeAndDims parseSpecs.
- Hadoop jobs should use hadoopyString string parsers.
2016-03-10 08:45:04 -08:00
Fangjin Yang 68cffe1d91 Merge pull request #2615 from gianm/timeseries-skipEmptyBuckets-cache
Fix caching of skipEmptyBuckets for TimeseriesQuery.
2016-03-09 18:45:59 -08:00
Bingkun Guo 5bb9fcf785 Merge pull request #2620 from guobingkun/fix_log
log exception correctly in DatasourceInputFormat and IndexGeneratorJob
2016-03-09 17:33:48 -06:00
Xavier Léauté 0f3a7c94c6 Merge pull request #2616 from gianm/getContextBoolean
Make specifying query context booleans more consistent.
2016-03-09 14:23:10 -08:00
Bingkun Guo c20d7682a9 log exceptions correctly in DatasourceInputFormat and IndexGeneratorJob 2016-03-09 13:41:31 -06:00
Charles Allen 4c3a3f8da6 Merge pull request #2537 from druid-io/refactor-ext
refactor extensions into core and contrib
2016-03-09 08:18:42 -08:00
Fangjin Yang 94da1f8f59 Merge pull request #2608 from gaodayue/hdfs_federation_support
support Federationed HDFS
2016-03-08 20:41:13 -08:00
Gian Merlino 708bc674fa Make specifying query context booleans more consistent.
Before, some needed to be strings and some needed to be real booleans. Now
they can all be either one.
2016-03-08 19:38:26 -08:00
Gian Merlino 40dad6dff4 Fix caching of skipEmptyBuckets for TimeseriesQuery. 2016-03-08 19:22:12 -08:00
fjy e3e932a4d4 refactor extensions into core and contrib 2016-03-08 17:12:09 -08:00
Xavier Léauté 163e536415 Merge pull request #2601 from navis/fix-combine-sequence
Relay final value to yielder in CombineSequence (Fix for #2586)
2016-03-08 15:59:08 -08:00
Himanshu Gupta ca5de3f583 only allow lowering maxResults and maxIntermediateRows from groupBy query context 2016-03-08 15:03:59 -06:00
Fangjin Yang 8e36e6fa43 Merge pull request #2610 from dclim/add-combineText-doc
add combineText property and cleanup batch ingestion doc
2016-03-08 12:54:16 -08:00
Fangjin Yang 9e31e2ce0e Merge pull request #2605 from himanshug/robust_hdfs_kill
let hdfs segment kill be a success when segment file does not exist
2016-03-08 12:43:59 -08:00
Fangjin Yang 9c2420a1bc Merge pull request #2599 from himanshug/datasource_isolation
make coordinator db polling for list of segments more robust
2016-03-08 12:43:49 -08:00
Fangjin Yang e7018f524f Merge pull request #2598 from himanshug/handoff_timeout
optional ability to configure handoff wait timeout on realtime tasks
2016-03-08 12:43:36 -08:00
Fangjin Yang de869f6efc Merge pull request #2584 from himanshug/gp_by_max_limit
allow groupBy maxRows limit be overridable by context
2016-03-08 12:43:25 -08:00
dclim df29667a89 add combineText property and cleanup batch ingestion doc 2016-03-08 13:10:34 -07:00
Fangjin Yang db11d8bbe5 Merge pull request #2609 from metamx/showBadLog
Add LogTest to show bad log behavior
2016-03-08 10:53:27 -08:00
Slim 3afa9e916b Merge pull request #2578 from b-slim/fix_metric
override metric name
2016-03-08 12:04:50 -06:00
Charles Allen 908eb7eb4d Add LogTest to show bad log behavior 2016-03-08 09:46:26 -08:00
navis.ryu 1b3fd8a8aa added more tests and fixed concat+combine 2016-03-09 02:08:52 +09:00
Slim Bouguerra c72438ead0 override metric name 2016-03-08 10:58:12 -06:00
gaodayue a6dc3703ca use ISODataTimeFormat for both hdfs and viewfs schema to support Federationed HDFS 2016-03-08 13:55:05 +08:00
navis.ryu 4ff1620131 Relay final value to yielder in CombineSequence (Fix for #2586) 2016-03-08 10:31:15 +09:00
Himanshu Gupta ea408f531e let the hdfs segment kill be success when segment file does not exist 2016-03-07 16:51:41 -06:00
Himanshu Gupta 099acb4966 allow groupBy max[Intermediate]Rows limit be overridable by context 2016-03-07 15:22:41 -06:00