Commit Graph

6874 Commits

Author SHA1 Message Date
Fangjin Yang 8f97b1e40c Merge pull request #2554 from guobingkun/improve_include_extensions
improve doc on including druid and hadoop extensions
2016-02-26 20:05:57 -08:00
Fangjin Yang dd060eb826 Merge pull request #2558 from gianm/hdfs-log-overwrite
HdfsTaskLogs: Allow overwriting existing logs.
2016-02-26 15:50:51 -08:00
Gian Merlino 7e70881101 HdfsTaskLogs: Allow overwriting existing logs.
Necessary because ForkingTaskRunner pushes logs when gracefully stopping,
but it may need to re-push those logs when the task finishes for real after
restoring.
2016-02-26 15:10:19 -08:00
Charles Allen ca1bf648d1 Merge pull request #2533 from metamx/javaOptsArray
Allow specifying peon javaOpts as an array
2016-02-26 14:05:49 -08:00
Charles Allen c6803c4364 Allow specifying peon javaOpts as an array 2016-02-26 13:24:35 -08:00
Bingkun Guo 18f9e05f0f improve doc on including druid and hadoop extensions 2016-02-26 13:53:08 -06:00
Gian Merlino eb13d7afe3 Merge pull request #2521 from himanshug/fix_2497
RTR has multiple threads for assignment of pending tasks now
2016-02-26 08:14:15 -08:00
Nishant 9f8faabddb Merge pull request #2469 from pdeva/patch-10
correct service names
2016-02-26 21:15:58 +05:30
Himanshu Gupta bc156effe7 RTR has multiple threads for assignment of pending tasks now. 2016-02-26 09:27:03 -06:00
Charles Allen 1fe277ee29 Merge pull request #2367 from se7entyse7en/feature-rackspace-cloud-files-static-firehose
Adds support to use Rackspace's cloudfiles as static firehose
2016-02-25 17:31:06 -08:00
Fangjin Yang 29d29ba98d Merge pull request #2263 from jon-wei/flex_dims3
Allow IncrementalIndex to store Long/Float dimensions
2016-02-25 17:23:02 -08:00
Fangjin Yang 3a9fe2aad0 Merge pull request #2231 from lizhanhui/pull_request
Add druid-rocketmq module
2016-02-25 17:19:57 -08:00
Fangjin Yang 3691fbd849 Merge pull request #2548 from metamx/moreLoggingJDBCNamespace
More logging for JDBCExtractionNamespaceFunctionFactory
2016-02-25 17:13:20 -08:00
Fangjin Yang 53a5f07c14 Merge pull request #2544 from metamx/fixMaxPort
Limit PortFinder to 0xFFFF
2016-02-25 17:12:53 -08:00
Fangjin Yang 143e85eaa5 Merge pull request #2419 from gianm/task-hostports
Plumb task peon host/ports back out to the overlord.
2016-02-25 17:11:53 -08:00
Gian Merlino e987e2d73d Merge pull request #2552 from gianm/fix-dumb-bug
Fix parsing of druid.indexer.server.maxChatRequests.
2016-02-25 15:49:24 -08:00
Gian Merlino b331fb4a83 Fix parsing of druid.indexer.server.maxChatRequests. 2016-02-25 14:47:15 -08:00
Charles Allen a8820d837b More logging for JDBCExtractionNamespaceFunctionFactory 2016-02-25 09:48:56 -08:00
Nishant 4c522a8159 Merge pull request #2543 from metamx/fixUnregisterChildPorts
UnRegister port in ForkingTaskRunner
2016-02-25 22:32:31 +05:30
Charles Allen 3fa7a7ebfe Limit PortFinder to 0xFFFF 2016-02-25 08:16:40 -08:00
Charles Allen 187b788089 UnRegister port in ForkingTaskRunner 2016-02-25 08:04:25 -08:00
Gian Merlino cf0bc905fb Plumb task peon host/ports back out to the overlord.
- Add TaskLocation class
- Add registerListener to TaskRunner
- Add getLocation to TaskRunnerWorkItem
- Implement location tracking in existing TaskRunners
- Rework WorkerTaskMonitor to do management out of a single thread so it can
  handle status and location updates more simply.
2016-02-24 15:13:10 -08:00
Gian Merlino 917e7e58f5 Merge pull request #2536 from jon-wei/regex_extraction_rename
Rename 'replaceMissingValues...' parameters in RegexExtractionFn
2016-02-24 14:41:06 -08:00
jon-wei c17ce02467 Allow IncrementalIndex to store Long/Float dimensions 2016-02-24 13:51:57 -08:00
jon-wei fd3782522c Rename 'replaceMissingValues...' parameters in RegexExtractionFn 2016-02-24 13:12:56 -08:00
Nishant 6ac32c5518 Merge pull request #2530 from himanshug/fix_plumber
In persistAndMerge, increase the scope of try-catch block so that any…
2016-02-24 22:41:32 +05:30
Nishant fb7eae34ed Merge pull request #2249 from metamx/workerExpanded
Use Worker instead of ZkWorker whenever possible
2016-02-24 13:23:22 +05:30
Himanshu Gupta a3b37e9225 In persistAndMerge, increase the scope of try-catch block so that any exception while persisting hydrants is caught and consequently that sink is abandoned or the task will forever wait for handoff to happen. 2016-02-23 22:22:33 -06:00
Nishant 6c9e1a28ad Merge pull request #2519 from gianm/unparseable-handling
Better handling of ParseExceptions.
2016-02-24 04:46:29 +05:30
Charles Allen ac13a5942a Use Worker instead of ZkWorker whenver possible
* Moves last run task state information to Worker
* Makes WorkerTaskRunner a TaskRunner which has interfaces to help with getting information about a Worker
2016-02-23 15:02:03 -08:00
Fangjin Yang 93540c0631 Merge pull request #2503 from gianm/jetty-qos
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
2016-02-23 10:35:53 -08:00
Fangjin Yang 4f300cfe49 Merge pull request #2526 from druid-io/b-slim-patch-1
fix docs about sketches
2016-02-23 10:23:53 -08:00
Gian Merlino 3534483433 Better handling of ParseExceptions.
Two changes:
- Allow IncrementalIndex to suppress ParseExceptions on "aggregate".
- Add "reportParseExceptions" option to realtime tuning configs. By default this is "false".

Behavior of the counters should now be:

- processed: Number of rows indexed, including rows where some fields could be parsed and some could not.
- thrownAway: Number of rows thrown away due to rejection policy.
- unparseable: Number of rows thrown away due to being completely unparseable (no fields salvageable at all).

If "reportParseExceptions" is true then "unparseable" will always be zero (because a parse error would
cause an exception to be thrown). In addition, "processed" will only include fully parseable rows
(because even partial parse failures will cause exceptions to be thrown).

Fixes #2510.
2016-02-23 10:11:43 -08:00
Fangjin Yang 11c9a63ee3 Merge pull request #2522 from himanshug/doc_update
fix thetaSketchSetOp doc
2016-02-23 09:33:55 -08:00
Slim 86c4900347 fix thetaSketch post aggregator doc 2016-02-23 10:43:54 -06:00
Himanshu Gupta c7cb5bff14 fix thetaSketchSetOp doc 2016-02-23 09:17:49 -06:00
Fangjin Yang 0c984f9e32 Merge pull request #2109 from himanshug/segments_in_delta_ingestion
idempotent batch delta ingestion
2016-02-22 14:00:45 -08:00
Slim 79614bd159 Merge pull request #2514 from himanshug/doc_update
updating thetaSketchSetOp post agg documentation to reflect the possibility of nesting
2016-02-22 13:13:58 -06:00
Fangjin Yang 3bdd757024 Merge pull request #1773 from b-slim/log_details
Adding downstream source when throwing QueryInterruptedException
2016-02-22 10:16:07 -08:00
Himanshu Gupta f7679dd5a9 updating thetaSketchSetOp post agg documentation to reflect the possibility of nesting 2016-02-22 09:38:58 -06:00
Himanshu Gupta 21b0b8a07d new coordinator endpoint to get list of used segment given a dataSource and list of intervals 2016-02-21 23:17:58 -06:00
Himanshu Gupta 09ffcae4ae give user the option to specify the segments for dataSource inputSpec 2016-02-21 23:15:31 -06:00
Slim Bouguerra 77925cc061 adding downstream source of QueryInterruptedException 2016-02-20 13:05:14 -06:00
Fangjin Yang 8ee81947cd Merge pull request #2494 from himanshug/fix_timeseries
do not drop post-aggs in TimeseriesQueryToolChest.makePreComputeManipulatorFn
2016-02-20 10:37:32 -08:00
Gian Merlino 23c993c9e7 Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
- Add druid.indexer.server.maxChatRequests, which sets up a QoSFilter on the main Jetty server.
- Deprecate druid.indexer.runner.separateIngestionEndpoint
- Deprecate druid.indexer.server.chathandler.*
2016-02-19 13:36:09 -08:00
Fangjin Yang 8a41f03377 Merge pull request #2496 from gianm/hll-finalizing-comparator
Add comparator to HyperUniquesFinalizingPostAggregator.
2016-02-19 13:03:43 -08:00
Bingkun Guo 499288ff4b Merge pull request #2509 from metamx/hadoopIsolatorTest
Add hadoop classloader isolation tests for HadoopTask
2016-02-19 14:23:22 -06:00
Fangjin Yang a3c29b91cc Merge pull request #2505 from gianm/rt-exceptions
Harmonize realtime indexing loop across the task and standalone nodes.
2016-02-19 11:23:14 -08:00
Charles Allen 9dff0e5dbd Add hadoop classloader isolation tests for HadoopTask 2016-02-19 11:15:53 -08:00
Fangjin Yang 44e925c98f Merge pull request #2506 from metamx/failTravisFaster
Fail travis builds faster
2016-02-19 11:01:37 -08:00