Commit Graph

2845 Commits

Author SHA1 Message Date
Gian Merlino 243ac5399b Harmonize realtime indexing loop across the task and standalone nodes.
- Both now catch ParseExceptions on plumber.add (see https://groups.google.com/d/topic/druid-user/wmiRDvx2RvM/discussion)
- Standalone now treats IndexSizeExceededException as fatal (previously only the task did)
2016-02-19 07:34:15 -08:00
Gian Merlino e0c049c0b0 Make startup properties logging optional.
Off by default, but enabled in the example config files. See also #2452.
2016-02-12 14:12:16 -08:00
Fangjin Yang 1430bc2c88 Merge pull request #2276 from harshjain2/feature-2021
Fix for issue 2021.
2016-02-10 17:04:45 -08:00
Gian Merlino fa92b77f5a Harmonize znode writing code in RTR and Worker.
- Throw most exceptions rather than suppressing them, which should help
  detect problems. Continue suppressing exceptions that make sense to
  suppress.
- Handle payload length checks consistently, and improve error message.
- Remove unused WorkerCuratorCoordinator.announceTaskAnnouncement method.
- Max znode length should be int, not long.
- Add tests.
2016-02-10 14:52:00 -08:00
Harsh Jain a3eb863c8e Fix for issue 2021 2016-02-10 22:19:12 +05:30
Himanshu Gupta d1cb17d3f7 at broker - only add segments from specific tiers to the timeline 2016-02-09 22:33:22 -06:00
Himanshu Gupta b40c342cd1 make Global stupid pool cache size configurable 2016-02-05 14:18:06 -06:00
Parag Jain 9002548eeb increase test time out and general clean up 2016-02-03 13:26:37 -06:00
Charles Allen 5111fd52f2 Add check for log4j-core in Log4jShutterDownerModule 2016-02-02 15:56:48 -08:00
Himanshu dc89cdd0f9 Merge pull request #2336 from himanshug/fix_2331
limit size of X-Druid-Response-Context header to 7K
2016-02-02 12:06:59 -06:00
navis.ryu c03918f89a AsyncQueryForwardingServletTest#testDeleteBroadcast sometimes fails by port conflict 2016-01-29 19:28:58 +09:00
Himanshu Gupta f6b4dbd697 bug fix and unit tests for DruidCoordinatorSegmentKiller 2016-01-28 14:10:17 -06:00
Himanshu Gupta ab3edfa8fc moving DruidCoordinatorSegmentKiller class out of DruidCoordinator 2016-01-28 14:03:56 -06:00
Nishant 3880f54b87 Merge pull request #2332 from himanshug/configurable_partial
make populateUncoveredIntervals a configuration in query context
2016-01-28 10:34:35 +05:30
Himanshu Gupta a7bde8f4da limit size of X-Druid-Response-Context header to 7K due to https://github.com/druid-io/druid/issues/2331 2016-01-27 15:18:08 -06:00
Xavier Léauté 5a3642bb93 Merge pull request #2247 from metamx/pedanticBuild
Enable strict building in travis
2016-01-27 10:27:03 -08:00
Charles Allen 508734c8b0 Long constant reformatting in tests `l` --> `L` 2016-01-27 08:59:19 -08:00
Nishant fd6bf3fe22 Use interval comparator instead of bucketMonthComparator
fix when two segments have same interval

review comments
2016-01-27 17:35:43 +05:30
Himanshu Gupta 3719b6e3c8 make populateUncoveredIntervals a configuration in query context 2016-01-26 15:13:45 -06:00
Harsh Jain 41730b96d4 Fix for issue 2021 2016-01-25 02:48:22 +05:30
Himanshu 7a6109f0ca Merge pull request #2321 from gianm/pending-index
Replace two-column index on pendingSegments table with one-column index.
2016-01-22 13:38:15 -06:00
Gian Merlino 0bd9bff075 Replace two-column index on pendingSegments table with one-column index.
Fixes #2319.
2016-01-22 10:50:21 -08:00
Himank Chaudhary 1a5d4e714c Adding custom mapper for json processing exception to return bad request instead of 500 2016-01-22 09:48:52 -08:00
Fangjin Yang 04d3054353 Merge pull request #2303 from CHOIJAEHONG1/localfirehouse-basedir-npe
Throw an IAE when baseDir is null in LocalFireHose
2016-01-21 07:58:52 -08:00
Nishant dcb7830330 Merge pull request #984 from drcrallen/thread-priority-rebase
Use thread priorities. (aka set `nice` values for background-like tasks)
2016-01-21 15:02:34 +05:30
Charles Allen 2e1d6aaf3d Use thread priorities. (aka set `nice` values for background-like tasks)
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
 * Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
 * Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
 * Add options to ForkingTaskRunner
    * Add "-XX:+UseThreadPriorities" default option
    * Add "-XX:ThreadPriorityPolicy=42" default option
 * AbstractTask - Removed unneded @JsonIgnore on priority
 * Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
 * Add persistThreadPriority and mergeThreadPriority to realtime tuning config
2016-01-20 14:00:31 -08:00
Nishant 61aca6f9cc remove wrong checks
sink never have null hydrants and hydrants never have null adapters
2016-01-20 23:43:53 +05:30
Jaehong Choi 7132428bba throw IAE when baseDir is null in LocalFireHose 2016-01-21 01:27:32 +09:00
Nishant 59ea186af7 fix reference counting for segments 2016-01-20 17:24:21 +05:30
jon-wei 747343e621 Preserve dimension order across indexes during ingestion 2016-01-19 13:34:11 -08:00
Jonathan Wei df2906a91c Merge pull request #2290 from gianm/index-merger-v9-stuff
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
2016-01-19 13:04:00 -08:00
Fangjin Yang 0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata
Store AggregatorFactory[] in segment metadata
2016-01-19 12:55:49 -08:00
Himanshu fe841fd961 Merge pull request #2118 from guobingkun/fix_segment_loading
Fix loading segment for historical
2016-01-19 14:25:48 -06:00
Himanshu Gupta a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
Gian Merlino 1dcf22edb7 Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes.
Also parameterize some tests to run with/without buildV9Directly:

- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
2016-01-19 12:15:06 -08:00
Bingkun Guo c4ad50f92c Fix loading segment for historical
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
2016-01-19 10:29:49 -06:00
Himanshu Gupta 164b0aad7a removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class
instead of a Map to store segment metadata
2016-01-18 22:03:46 -06:00
Himanshu Gupta 637d2605e7 kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller 2016-01-18 21:51:04 -06:00
Harsh Jain 71f1cd5e34 Fix for issue 2021 2016-01-17 16:10:04 +05:30
Fangjin Yang f6a1a4ae20 Merge pull request #2138 from KurtYoung/feature-build-v9
build v9 directly
2016-01-16 13:35:46 -06:00
Kurt Young 82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Fangjin Yang 2e54553a8f Merge pull request #1990 from himanshug/schedule_kill_task
support periodic hard delete of segments
2016-01-15 15:22:33 -06:00
David Lim 7c65880e55 Merge pull request #2270 from rasahner/warnOfChatHandlerNoop
if chathandler is noop, log using warn instead of info
2016-01-15 13:29:27 -07:00
Harsh Jain 6ec6835b5d Fix for issue 2021. 2016-01-16 00:58:33 +05:30
David Lim 34cd8f8c72 Merge pull request #2258 from fjy/acl-zk
acl for zookeeper is added
2016-01-15 10:27:08 -07:00
Fangjin Yang a54c726726 Merge pull request #2266 from anubhgup/fix-announce
Fix loss in segment announcements when segments do not fit in zNode
2016-01-14 17:10:43 -08:00
Xavier Léauté dc1a62c3d9 Merge pull request #2248 from metamx/druidNodeHashEquals
Add hashCode and equals to DruidNode
2016-01-14 16:04:58 -08:00
Anubhav Gupta 6d09ab839f Fix for loss in segment announcements when segments do not fit in the znodes during compress mode.
Added unit test (from Navis).
2016-01-14 14:44:23 -08:00
Robin 7361cd173f if chathandler is noop, log using warn instead of info 2016-01-14 10:12:42 -06:00
Nikita Geer 1908d63162 acl for zookeeper is added 2016-01-13 14:56:05 -08:00
navis.ryu 18479bb757 time-descending result of timeseries queries 2016-01-13 12:23:01 +09:00
Himanshu Gupta eb2d251ac8 support periodic hard delete of segments 2016-01-12 16:55:05 -06:00
Himanshu 01a0715ee2 Merge pull request #2161 from metamx/query-metrics-timeout
Fix Query metrics for query timeout
2016-01-12 09:50:40 -06:00
Himanshu Gupta ef3cddabe9 adding UTs for DruidCoordinatorConfig 2016-01-11 22:03:25 -06:00
Charles Allen b0e04a9162 Add hashCode and equals to DruidNode 2016-01-11 15:23:45 -08:00
Charles Allen ea623e43d2 Merge pull request #2240 from metamx/fix-load-rule
Fix loadRule when one of the tiers had no available servers
2016-01-11 10:05:31 -08:00
Nishant 32bc2f776e Fix loadRule when one of the tiers had no servers
When one of the tiers have no servers, LoadRule should ignore that tier
and continue to load/drop segments in other available tiers.

the bug also causes whacky behavior with LoadRule with non existent
tier where the segment balancer keeps on moving segments to other nodes
in existing tiers but the extra segment copies are never dropped
eventually leading to all the tiers getting full .
2016-01-11 15:53:14 +05:30
Himanshu d255f4baac Merge pull request #2234 from pjain1/emit_realtime_metrics
emit handoff count metrics
2016-01-08 14:24:16 -06:00
Parag Jain 9dba0f67e7 emit handoff count metrics 2016-01-08 12:36:13 -06:00
Fangjin Yang 15fc070232 Merge pull request #2213 from himanshug/fix_curtator_test_base
[wip] trying/finding fix for announcer test failures
2016-01-07 18:23:49 -08:00
Nishant 1bfb4e3988 Emit query/time for failed and timeout queries
emit query/time metric also

add success flag

fix success flag for router metrics

review comments

formatting.
2016-01-07 19:41:54 +05:30
Himanshu Gupta 7ab810f3eb do not ignore exceptions from curator cleanup in CuratorTestBase 2016-01-06 10:42:53 -06:00
Nishant 14989f272d Add metrics for ingest/bytes/received for EventReceiverFirehose
review comments

review comments
2016-01-05 20:06:09 +05:30
fjy 57d91d754d Comment out buggy unit tests, fix #2185 2016-01-03 09:50:16 -08:00
fjy 8424b2b456 Fix announcer test bad check 2016-01-01 19:55:22 -08:00
Himanshu Gupta fa5c3bb014 adding decorate(DimensionSelector) to DimensionSpec to enable support for arbitrary filtering/transformations to returned dimension values 2015-12-30 15:06:24 -06:00
Bingkun Guo 3c107c5757 Merge pull request #2150 from himanshug/emit_query_bytes
emit query/bytes metric
2015-12-30 13:44:19 -06:00
Fangjin Yang 7ffa706655 Merge pull request #2152 from metamx/add-taskId
Add taskId to realtimeMetrics
2015-12-29 10:33:40 -08:00
Fangjin Yang b1261035a7 Merge pull request #1861 from guobingkun/insert_segment_tool
insert-segment tool
2015-12-29 10:06:07 -08:00
fjy 38b0f1fbc2 fix transient failures in unit tests 2015-12-28 20:03:30 -08:00
Fangjin Yang d94821998f Merge pull request #2153 from navis/fix-testfail-DruidCoordinatorTest
Another try to fix DruidCoordinatorTest#testCoordinatorRun
2015-12-27 17:03:46 -08:00
navis.ryu 10c19fe9f8 Another try to fix DruidCoordinatorTest#testCoordinatorRun 2015-12-24 03:02:37 +09:00
navis.ryu 2c3c4a3f8f Another try to fix xxServerViewTests 2015-12-24 02:13:40 +09:00
Fangjin Yang 508629916d Merge pull request #2137 from guobingkun/fix_serverview_transient_error2
[Don't merge]Another effort on fixing the transient error for XXXServerViewTest
2015-12-23 07:33:54 -08:00
Nishant 978a3fd8ae Add taskId to realtimeMetrics
Add task Id to Realtime Metrics
2015-12-23 18:05:25 +05:30
Himanshu Gupta 1a8546a682 emit query/bytes metric 2015-12-23 00:29:44 -06:00
Bingkun Guo 89e2a9a6d2 build at Tue Dec 22 21:30:00 CST 2015 2015-12-22 21:30:00 -06:00
Bingkun Guo a110eafff7 build at Tue Dec 22 17:30:00 CST 2015 2015-12-22 17:30:00 -06:00
Bingkun Guo 4b93528334 build at Tue Dec 22 17:00:01 CST 2015 2015-12-22 17:00:01 -06:00
Bingkun Guo b8d9860dbc build at Tue Dec 22 16:00:00 CST 2015 2015-12-22 16:00:00 -06:00
Bingkun Guo ffdb9407e5 build at Tue Dec 22 15:30:00 CST 2015 2015-12-22 15:30:00 -06:00
Bingkun Guo 6f54c1b60d build at Tue Dec 22 15:00:00 CST 2015 2015-12-22 15:00:01 -06:00
Bingkun Guo 4e61261c68 build at Tue Dec 22 14:30:00 CST 2015 2015-12-22 14:30:00 -06:00
Bingkun Guo 4961b8c0bd build at Tue Dec 22 14:00:00 CST 2015 2015-12-22 14:00:00 -06:00
Bingkun Guo 0a635a11d6 build at Tue Dec 22 13:40:00 CST 2015 2015-12-22 13:40:00 -06:00
Charles Allen 05c9e1b598 Reorder Before/After in JDBCExtractionNamespaceTest
* Fixes https://github.com/druid-io/druid/issues/2120
2015-12-22 11:39:46 -08:00
Bingkun Guo 849ab2d958 build at 2015-12-22 13:08:29 -06:00
Bingkun Guo 455980d659 address another node exists failure 2015-12-22 13:06:20 -06:00
Bingkun Guo 2ffeda5d25 build number 13 2015-12-22 10:19:39 -06:00
Charles Allen 7e64d5179f Merge pull request #2139 from himanshug/emit_query_node_bytes
emit query/node/bytes metric
2015-12-22 08:07:04 -08:00
Bingkun Guo bc573484f3 build number 12 2015-12-22 00:52:58 -06:00
Himanshu 7b5fd76058 Merge pull request #2119 from navis/fix-sporadic-testfail2
Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun
2015-12-21 23:28:32 -06:00
Himanshu Gupta b96f560255 emit query/node/bytes metric 2015-12-21 23:23:20 -06:00
Bingkun Guo 06c220ae7b build number 11 2015-12-21 19:40:47 -06:00
Bingkun Guo 93a91e5633 build number 10 2015-12-21 18:21:15 -06:00
Bingkun Guo 471dd7e164 build number 9 2015-12-21 17:35:18 -06:00
Bingkun Guo 627c311923 build #8 2015-12-21 17:06:37 -06:00
Bingkun Guo 293f2ad65d build #7 2015-12-21 16:09:19 -06:00
Bingkun Guo 9ed843ea21 build #6 2015-12-21 15:41:23 -06:00
Bingkun Guo f8342ca240 build #5 2015-12-21 15:10:37 -06:00
Bingkun Guo 52261c5539 build #4 2015-12-21 14:41:38 -06:00
Bingkun Guo 47a8f2d3f1 build #3 2015-12-21 13:56:43 -06:00
Bingkun Guo 5cc56cf9e7 build #2 2015-12-21 13:05:38 -06:00
Bingkun Guo f12cfa99fa another effort on fixing the transient error for Broker/CoordinatorServerViewTest happend on Travis build 2015-12-21 12:31:36 -06:00
Charles Allen c2a2d19d62 Merge pull request #2126 from guobingkun/remove_single_announcer
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
2015-12-21 08:55:07 -08:00
navis.ryu 7b24d1f355 Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun 2015-12-21 17:00:52 +09:00
Bingkun Guo 951a4e9b35 Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer 2015-12-21 00:05:53 -06:00
Bingkun Guo 89b477970f DataSegmentFinder tool
`insert-segment-to-db` is a tool that can insert segments into Druid metadata storage. It is intended to be used
to update the segment table in metadata storage after people manually migrate segments from one place to another.
It can also be used to insert missing segment into Druid, or even recover metadata storage by telling it where the
segments are stored.

Note: This tool expects users to have Druid cluster running in a "safe" mode, where there are no active tasks to interfere
the segments being inserted. Users can optionally bring down the cluster to make 100% sure nothing is interfering.
2015-12-21 00:02:04 -06:00
navis.ryu ad3312171b Fix sporadic fail of io.druid.client.CoordinatorServerViewTest 2015-12-21 11:39:32 +09:00
Fangjin Yang 7019d3c421 Merge pull request #2107 from jon-wei/fix_smq
More efficient SegmentMetadataQuery
2015-12-18 16:40:47 -08:00
Fangjin Yang e52c9d2b3c Merge pull request #2095 from guobingkun/fix_client_merge_task_and_serde
Fix getType() and Json serialization in ClientMergeQuery + serde tests
2015-12-18 11:40:13 -08:00
jon-wei 356b07c6c3 More efficient SegmentMetadataQuery 2015-12-17 12:46:23 -08:00
Bingkun Guo 1e5aa2f3ac fix getType() and Json serialization in ClientMergeQuery and add serde tests 2015-12-15 12:08:43 -06:00
Gian Merlino e6c2db89da Merge pull request #1921 from jon-wei/flat_json
Add docs and benchmark for JSON flattening parser
2015-12-14 20:53:54 -08:00
Xavier Léauté a2d0bea223 Merge pull request #2074 from guobingkun/add_comment
Add comment to BrokerServerViewTest
2015-12-09 16:19:27 -08:00
jon-wei c53bf85d83 Add docs and benchmark for JSON flattening parser 2015-12-09 16:13:30 -08:00
Bingkun Guo 31502a920c Add comment to BrokerServerViewTest
Explain why we need to do explicit checks on existences of zNodes before we create them. Address issue: https://github.com/druid-io/druid/issues/1512
2015-12-09 14:01:07 -06:00
Fangjin Yang ce79d707dd Merge pull request #2072 from guobingkun/fix_transient_error_brokerserverviewtest
An effort to fix transient error in BrokerServerViewTest
2015-12-09 11:12:35 -08:00
Bingkun Guo 161c4ce288 An effort to fix transient error in BrokerServerViewTest 2015-12-09 10:50:13 -06:00
Bingkun Guo c4de26ef09 fix typo in InitializationTest.java 2015-12-09 00:11:03 -06:00
Charles Allen 5bc6d61a25 Fix InitializationTest 2015-12-08 17:37:56 -08:00
Xavier Léauté dcd15735cd Merge pull request #2015 from metamx/handoff-notifier-coordinator
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information
2015-12-08 14:06:39 -08:00
Nishant a32906c7fd Remove FilteredServerView 2015-12-09 01:54:12 +05:30
Nishant 9491e8de3b Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView

review comment

fix realtime node guide injection

review comments

make test not rely on scheduled exec

fix compilation

fix import

review comment

introduce immutableSegmentLoadInfo

fix son reading

remove unnecessary logging
2015-12-09 01:54:09 +05:30
Fangjin Yang 56fd48c0e4 Merge pull request #2056 from guobingkun/fix_server_selector2
Bug fix: stale segment in ServerSelector
2015-12-07 16:57:33 -08:00
Fangjin Yang ba980dcadb Merge pull request #2058 from cheddar/report-uncovered-segments
Add "uncoveredIntervals" to responseContext
2015-12-07 16:42:42 -08:00
Eric Tschetter a36185926e Add "uncoveredIntervals" to responseContext
This change will cause the CachingClusteredClient to populate the "uncoveredIntervals"
key in the responseContext map.  The value will be any intervals that were requested in
the query but are not actually covered by the segments underlying the data source.

For unit testing, CachingClisteredClientTest is testing the caching behavior of the
object and it is pretty hard to adjust it to only test this new behavior, so I created
a new, parallel "CachingClusteredClientFunctionalityTest" to simplify testing just
basic functionality.
2015-12-07 16:16:57 -08:00
Bingkun Guo 970b9cfcaa fix stale segment in ServerSelector 2015-12-07 14:04:34 -06:00
Fangjin Yang 00416e26ff Merge pull request #2012 from himanshug/add_comment
adding more comments to why drops do not happen
2015-12-07 11:50:06 -08:00
Himanshu Gupta 62ba9ade37 unifying license header in all java files 2015-12-05 22:16:23 -06:00
刘麒贇 3781aa380f Added an injection test for ZkCoordinator. 2015-12-05 11:56:22 +08:00
Himanshu Gupta 221fb95d07 add support for getting used segments for multiple interval in IndexerMetadataStorageCoordinator 2015-12-03 21:28:04 -06:00
Nishant 1eb8211346 Add datasource and taskId to metrics emitted by peons
This PR adds the datasource and taskId to the jvm and sys metrics
emitted by the peons.

fix spelling

review comment

review comment
2015-12-01 23:20:59 +05:30
Himanshu Gupta 7c91219130 adding more comments to why drops do not happen when there are no segments in db 2015-11-25 16:26:51 -06:00
Fangjin Yang 8e83d800d6 Merge pull request #1881 from gianm/restartable-tasks
Restorable indexing tasks
2015-11-23 21:14:37 -08:00
Gian Merlino 13af260102 Merge pull request #1957 from metamx/skip-incremental-segment
Ability to skip Incremental Index during query using query context
2015-11-23 17:32:57 -08:00
Gian Merlino 501dcb43fa Some changes that make it possible to restart tasks on the same hardware.
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.

The main changes are,

1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
   stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
   one that currently implements this.

2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
   ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
   ForkingTaskRunner- close output stream to restorable tasks, wait for exit
   RemoteTaskRunner- do nothing special, we actually don't want to shutdown

3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
   Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
   a list of restorable tasks.

4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
   task and a zombie old task from stomping on each other.
2015-11-23 11:22:08 -08:00
Gian Merlino 36569094ce EventReceiverFirehose: Drain buffer when closed, until empty. 2015-11-20 19:12:59 -08:00
Fangjin Yang 21c84b5ff7 Merge pull request #1896 from gianm/allocate-segment
SegmentAllocateAction (fixes #1515)
2015-11-18 21:05:46 -08:00
Nishant 60f649dab1 Ability to skip Incremental Index during query using query context
This PR adds adds the ability to skip incremental index when querying
results from realtime nodes. default behaviour is to include
incrementalIndex in queries.

review comment
2015-11-19 00:29:41 +05:30
Charles Allen 8fcf2403e3 Merge pull request #1943 from metamx/realtime-caching
Enable caching on intermediate realtime persists
2015-11-17 15:06:43 -08:00
Charles Allen 8fa34ee671 Merge pull request #1973 from metamx/cachemonitor-optional-injection
CacheMonitor - make cache injection optional
2015-11-17 13:02:06 -08:00
Himanshu 9118b010c0 Merge pull request #1976 from noddi/feature/count-parameter-rules-history
Add count parameter to rules history endpoint.
2015-11-17 14:41:50 -06:00
Charles Allen dbe201aeed Merge pull request #1929 from pjain1/jetty_threads
separate ingestion and query thread pool
2015-11-17 12:14:25 -08:00
Parag Jain 6c498b7d4a separate ingestion and query thread pool 2015-11-17 13:42:41 -06:00
Bartosz Ługowski b0a444eafc Add count parameter to rules history endpoint. 2015-11-17 20:14:12 +01:00
Xavier Léauté d7eb2f717e enable query caching on intermediate realtime persists 2015-11-17 10:58:00 -08:00
Xavier Léauté 71376efd57 make cache injection optional
allows enabling the CacheMonitor for all index tasks, even if some don't
bind any cache instance.
2015-11-16 22:40:01 -08:00
Fangjin Yang 4f46d457f1 Merge pull request #1947 from noddi/feature/count-parameter-history-endpoints
Add count parameter to history endpoints
2015-11-12 10:23:44 -08:00
Fangjin Yang 465cbcf9a7 Merge pull request #1956 from metamx/remove-unused-imports
Cleanup + remove unused imports
2015-11-11 17:36:47 -08:00
Gian Merlino e4e5f0375b SegmentAllocateAction (fixes #1515)
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).

The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.

The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
2015-11-11 16:54:35 -08:00
Charles Allen dc4ae59aaf Merge pull request #1865 from noddi/bugfix/issue-560
Fix #560 - datasource segments summary shouldn't include replication.
2015-11-11 15:29:08 -08:00
Bartosz Ługowski 6e5d2c6745 Add count parameter to history endpoints. 2015-11-11 23:03:57 +01:00
Gian Merlino 8e743b70c6 SegmentIdentifier, like a pre-DataSegment. 2015-11-11 12:25:43 -08:00
Xavier Léauté e26f7fca36 cleanup cache interface 2015-11-11 12:25:22 -08:00
Xavier Léauté fa6142e217 cleanup and remove unused imports 2015-11-11 12:25:21 -08:00
Gian Merlino f2c271e500 Better error message when LocalDataSegmentPusher cannot create its directory. 2015-11-10 19:49:01 -08:00
Xavier Léauté cf779946ef Merge pull request #1791 from guobingkun/event_receiver_firehose_monitor
EventReceiverFirehoseMonitor
2015-11-10 11:09:42 -08:00
Xavier Léauté a57cbfd2c3 Merge pull request #1387 from metamx/enableShutdownLogging
Add special handler to allow logger messages during shutdown
2015-11-09 17:20:09 -08:00
Charles Allen 1df4baf489 Move Jackson Guice adapters into io.druid
* Removes access to protected methods in com.fasterxml
* Eliminates druid-common's use of foreign package com.fasterxml
2015-11-09 10:50:45 -08:00
Fangjin Yang fe6efb6367 Merge pull request #1932 from gianm/data-schema-exclude-metric-names
DataSchema: Exclude metric names from dimension list.
2015-11-07 15:22:33 -08:00
Bartosz Ługowski 8b5de492cc Fix #560 - datasource segments summary shouldn't include replication. 2015-11-07 21:43:47 +01:00
Gian Merlino 6b908a5061 DataSchema: Exclude metric names from dimension list.
Otherwise we could end up with a metric and a dimension with the same column name.
2015-11-07 12:25:36 -08:00
Xavier Léauté e9533db987 Merge pull request #1850 from metamx/friendlyBardCache
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-11-06 10:25:49 -08:00
Xavier Léauté c896818241 Update curator to 2.9.1
Lots of bugfixes since 2.8.0
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12333324
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12332392
2015-11-05 15:53:01 -08:00
fjy 4efc90dda3 fix zkcoordinator inject 2015-11-04 17:30:17 -08:00
fjy 8f231fd3e3 cleanup druid codebase 2015-11-04 13:59:53 -08:00
Xavier Léauté 3b3d88bd03 Merge pull request #1857 from noddi/feature/intervals-route
Queryable intervals
2015-11-03 13:19:04 -08:00
Himanshu Gupta 8b67417ac8 make methods in Index[Merger,Maker,IO] non-static so that they can have
appropriate ObjectMapper injected instead of creating one statically
2015-11-02 23:24:26 -06:00
Fangjin Yang f90ddfdb89 Merge pull request #1745 from himanshug/numbered_to_elastic
making NumberedShardSpec elastic
2015-10-30 16:01:03 -07:00
Bingkun Guo c3b6fcce9d Add EventReceiverFirehoseMonitor
add an EventReceiverFirehoseMonitor so that we can monitor how many
events have been queued in the EventReceiverFirehose and get a sense
about whether the firehose is under too much pressure.
2015-10-30 11:40:02 -05:00
Bartosz Ługowski 6de51e022d Coordinator - add intervals route. 2015-10-30 11:51:42 +01:00
Xavier Léauté 149333d77d forward cancellation request to all brokers, fixes #1802 2015-10-29 16:58:38 -07:00
Charles Allen dfce14ed17 Allow setting upper limit on the number of cache segments a broker will try to fetch. 2015-10-29 11:50:00 -07:00
Xavier Léauté 59872bd0cd Merge pull request #1809 from metamx/fifoPriorityExecutorService
Make PrioritizedExecutorService optionally FIFO
2015-10-27 15:19:32 -07:00
Fangjin Yang ea2267e08c Merge pull request #1868 from gianm/fix-announcements
Historical and MiddleManager server announcements should not remove parents.
2015-10-27 14:50:05 -07:00
Gian Merlino 7df7370935 Merge pull request #1862 from metamx/indexingServiceMMGone
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 14:38:01 -07:00
Charles Allen 7a2ceef690 Add special handler to allow logger messages during shutdown
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
2015-10-27 14:33:36 -07:00
Charles Allen ecdafa87c5 Make PrioritizedExecutorService optionally FIFO 2015-10-27 14:16:22 -07:00
Charles Allen 67d58f6d8d Update CoordinatorRuleManager for http-client 1.0.4 2015-10-27 14:08:45 -07:00
Gian Merlino 4b92752deb Historical and MiddleManager server announcements should not remove parents.
Removing parent paths causes watchers of the "announcements" path to get stuck
and stop seeing new updates.
2015-10-27 08:06:11 -07:00
Fangjin Yang 5a082b2f5e Merge pull request #1824 from metamx/UniformGranularitySpecHashEquals
Add hashCode and equals to UniformGranularitySpec
2015-10-26 09:34:01 -07:00
Fangjin Yang 5f23703216 Merge pull request #1638 from guobingkun/remove_maven_client_code
Remove Maven client at runtime + Provide a way to load Druid extensions through local file system
2015-10-26 09:30:05 -07:00
Xavier Léauté 72c408cf2d Merge pull request #1770 from metamx/merge-time
Add segment merge time as a metric
2015-10-22 22:03:41 -07:00
Himanshu 853f95f64c Merge pull request #1845 from metamx/moreVerboseRuleAuditLogging
Add more verbose logging to SQLMetadataRuleManager
2015-10-22 16:12:46 -05:00
Nishant 7cecc55045 Add segment merge time as a metric
Add merge and persist cpu time

Fix typo

review comment

move cpu time measuring to VMUtils

review comments.
2015-10-22 12:28:03 +05:30
Charles Allen 6ceab03fc4 Add debug logging to start of segment manager polling 2015-10-21 16:17:49 -07:00
Charles Allen 1cad571354 Add more verbose logging to SQLMetadataRuleManager 2015-10-21 16:11:40 -07:00
Bingkun Guo 4914925d65 New extension loading mechanism
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
2015-10-21 14:22:36 -05:00
Xavier Léauté e4ac78e43d bump next snapshot to 0.9.0 2015-10-20 13:46:13 -07:00
Himanshu 0292db648d Merge pull request #1833 from metamx/improve-log
Improve alert message in CachingClusteredClient - Add Datasource info.
2015-10-20 15:40:57 -05:00
Nishant bf214f23b2 Improve message for alert - Add Datasource info. 2015-10-16 19:43:21 +05:30
Xavier Léauté 4c2c7a2c37 update version to 0.8.3 2015-10-14 21:40:55 -07:00
Charles Allen f432b8e3f9 Add hashCode and equals to UniformGranularitySpec
* Also add hashCode != 0 to AllGranularity and NoneGranularity
2015-10-13 16:42:21 -07:00
Charles Allen bf11723a52 Update usages of io.druid.client.selector.Server to build URL or URI directly instead of using String.format 2015-10-12 12:30:56 -07:00
Charles Allen e450877a78 Make ServerDiscoverySelector more IPv6 friendly 2015-10-08 10:03:50 -07:00
Charles Allen 693e14a14d Add unit tests for ipv6 in ServerDiscoverySelectorTest 2015-10-08 10:03:22 -07:00
Charles Allen 020a706ac4 Merge pull request #1800 from metamx/hybrid-cache
Hybrid L1/L2 cache
2015-10-06 15:18:07 -07:00
Xavier Léauté b11c0859e8 hybrid l1/l2 cache to combine local and remote cache 2015-10-06 14:32:08 -07:00
Fangjin Yang b5737a233a Merge pull request #1801 from guobingkun/clientInfoResource
Fix ClientInfoResource
2015-10-06 07:53:44 -07:00
Charles Allen 2bc4aed1e5 Add CPUTimeMetricQueryRunner to ClientQuerySegmentWalker 2015-10-05 10:09:22 -07:00
Bingkun Guo 57c8e56451 Fix ClientInfoResource
Fix ClientInfoResource so that it doesn't return empty list for overshadowed segments
2015-10-02 14:12:55 -05:00
Himanshu 166c4fcf46 Merge pull request #1795 from metamx/announcerTestTransactions
Try and make AnnouncerTest a bit more predictable
2015-10-02 13:04:33 -05:00
Charles Allen 7d635a2ce2 Try and make AnnouncerTest a bit more predictable 2015-10-02 09:19:40 -07:00
Xavier Léauté 5db4c5a089 move cache bindings into a dedicated module 2015-10-02 00:29:05 -04:00
Himanshu Gupta b9740a727b unit test for IngestSegmentFirehose 2015-10-01 16:06:04 -05:00
Charles Allen 2d847ad654 Merge pull request #1730 from metamx/union-queries-fix
fix #1727 - Union bySegment queries fix
2015-09-29 12:23:25 -07:00
Nishant 573aa96bd6 fix #1727 - Union bySegment queries fix
Fixes #1727.
revert to doing merging for results for union queries on broker.

revert unrelated changes

Add test for union query runner

Add test

remove unused imports

fix imports

fix renamed file

fix test

update docs.
2015-09-29 23:32:36 +05:30
Gian Merlino 348172203f OverlordRedirectInfo: Fix ability to detect that there is no leader. 2015-09-25 09:30:09 -07:00
Gian Merlino 906e99d631 RedirectFilter: User agents are more likely to preserve method on 307. 2015-09-25 08:54:23 -07:00
Gian Merlino 3aba401ee0 SQLMetadataConnector: Retry table creation, in case something goes wrong.
Also rejigger table creation methods to not take a DBI. It's already available
inside the connector, and everyone was just using that one anyway.
2015-09-24 21:39:36 -07:00
Charles Allen 4308aa93e2 Add meter metrics for memcached 2015-09-22 12:50:54 -07:00
Charles Allen 46ba60f10f Enable logging for memcached in factory 2015-09-22 11:13:47 -07:00
Xavier Léauté 1c86eac9e1 Merge pull request #1760 from metamx/memcached082Updates
Update memcached client for better concurrency in metrics.
2015-09-22 10:59:43 -07:00
Xavier Léauté 35caa753aa Merge pull request #1761 from gianm/local-path-for-hadoop
LocalDataSegmentPusher: Fix for Hadoop + relative paths.
2015-09-22 08:57:44 -07:00
Gian Merlino 4efbe649a3 LocalDataSegmentPusher: Fix for Hadoop + relative paths. 2015-09-22 07:44:12 -07:00
Charles Allen e6f07a832d Update memcached client for better concurrency in metrics. Also fixes another injection problem 2015-09-22 07:40:24 -07:00
Himanshu 34a8fbcd68 Merge pull request #1755 from metamx/update-druid-api
update druid-api for timestamp parsing speedup
2015-09-21 19:43:13 -05:00
Gian Merlino 490b1aea2a Merge pull request #1758 from metamx/fixMemcachedInjection
Fix memcached cache provider injection and add test
2015-09-21 15:36:29 -07:00
Charles Allen 37eb83a1d0 Fix memcached cache provider injection and add test 2015-09-21 15:00:22 -07:00
Xavier Léauté 8f489c1f66 fix type cast warnings 2015-09-21 09:57:29 -07:00
Xavier Léauté af86c0e6ea update druid-api + java-util for timstamp parsing speedup 2015-09-21 09:57:29 -07:00
Himanshu Gupta 117a6754b5 improving msging when indexing service is not found 2015-09-18 23:07:44 -05:00
Fangjin Yang 6605a60ef0 Merge pull request #1747 from jon-wei/realtime_corrupt_sink_fix
Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4)
2015-09-18 15:22:50 -07:00
jon-wei 9f6bb03ef4 Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4) 2015-09-18 14:53:37 -07:00
Xavier Léauté df4c8a3aa5 balance memcached cache across multiple connections 2015-09-18 09:52:16 -07:00
Himanshu Gupta a7b1cacbbf make NumberedShardSpec "elastic" and UTs 2015-09-17 08:38:48 -05:00
Gian Merlino 9705c5139b Merge pull request #1732 from jon-wei/segmentmeta
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-16 12:36:25 -07:00
Himanshu Gupta 74f4572bd4 Lazily deserialize "parser" to InputRowParser in DataSchema
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
2015-09-16 10:58:13 -05:00
Xavier Léauté b464da438c Merge pull request #1688 from metamx/moreMemcachedMetrics
More memcached metrics
2015-09-15 15:33:51 -07:00
jon-wei 193fb4fdfc Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups 2015-09-14 19:41:42 -07:00
Himanshu 5ff92664f8 Merge pull request #1696 from metamx/cpuTimeReporting
Add CPU time to metrics for segment scanning.
2015-09-14 10:53:55 -05:00
Himanshu Gupta ebdb612933 composing emitter module to use multiple emitters together 2015-09-09 16:45:50 -05:00
Charles Allen 5813856819 More memcached metrics 2015-09-08 13:34:58 -07:00
Charles Allen fcf5cae81d Add CPU time to metrics for segment scanning. 2015-09-08 13:34:19 -07:00
Gian Merlino 062a47fba4 Modify Plumbers in these ways,
1) Persist using Committer instead of Runnable. (Although the metadata object
   is ignored in this patch)

2) Remove the getSink method.

3) Plumbers are now responsible for time-based and hydrant-full-based periodic
   committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
2015-09-03 11:13:06 -07:00
Nishant e79572b4bc Merge pull request #1637 from metamx/peonPlaysNiceOnShutdown
Make jetty scheduler threads daemon thread
2015-09-03 00:03:20 +05:30
Nishant 0096e6a0a0 Merge pull request #1658 from metamx/cleanupJDBCExtractionNamespaceTest
Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest
2015-09-02 23:49:49 +05:30
nishant 0835e12f2a add endpoint to fetch rule history for all datasources.
review comments

Review comment

fix compilation
2015-08-31 14:05:52 +05:30
Fangjin Yang dfe3be0f6d Merge pull request #1689 from metamx/lz4TranscoderUpdate
Update LZ4Transcoder to match Compressed strategy factory type.
2015-08-29 12:31:59 -07:00
Charles Allen 413c0d1c9e Update LZ4Transcoder to match Compressed strategy factory type. 2015-08-28 19:19:48 -07:00
Gian Merlino 940e1aa3eb Replace funky imports with standard ones.
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
2015-08-28 18:02:05 -07:00
Himanshu ceaa49ec4f Merge pull request #1681 from gianm/ingest-segment-overlapping
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat
2015-08-28 11:12:51 -05:00
Gian Merlino e575ef2b20 Remove unused Indexer interface. 2015-08-28 08:40:47 -07:00
Gian Merlino 414a6fb477 Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat.
Fixes #1678. IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
2015-08-28 07:32:41 -07:00
Himanshu Gupta 2e0dd1d792 adding UTs and addressing review comments to
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
2015-08-27 20:50:46 -05:00
lvjq 2237a8cf0f kafka 8 simple consumer firehose 2015-08-27 20:50:46 -05:00
Charles Allen 10af233b37 Merge pull request #1622 from metamx/smile-eventreceiverfirehose
Support for JSON Smile format for EventReceiverFirehoseFactory
2015-08-27 10:42:25 -07:00
Charles Allen ac8e32b58e Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest 2015-08-26 23:05:51 -07:00
Gian Merlino 5b67ec6cda Merge pull request #1672 from metamx/derbyRule
Migrate TestDerbyConnector to a JUnit @Rule
2015-08-26 22:56:18 -07:00
Charles Allen e38cf54bc8 Migrate TestDerbyConnector to a JUnit @Rule 2015-08-26 21:47:40 -07:00
Xavier Léauté e9237611c5 Merge pull request #1668 from metamx/cachingClusteredClientCodeCleanup1
Code cleanup for CachingClusteredClientTest
2015-08-26 17:13:55 -07:00
Xavier Léauté c325b194c0 Merge pull request #1661 from pjain1/poll_rules
Poll rules immediately after change
2015-08-25 13:46:55 -07:00
Charles Allen 85aeda834c Code cleanup for CachingClusteredClientTest 2015-08-25 12:49:55 -07:00
Gian Merlino 2bf9a70bfa Consolidate SQL retrying by moving logic into the connectors.
Also change boolean removeLock to void addLock in MetadataStorageActionHandler.
2015-08-25 12:42:29 -07:00
Parag Jain 073c32f01b Poll rules immediately after change 2015-08-24 16:08:25 -05:00
Fangjin Yang e3964cd9f6 Merge pull request #1653 from metamx/fix-success-flag
fix success flag emitted in router query metrics
2015-08-21 07:56:57 -07:00
Nishant 6d85a62ed2 fix success flag emitted in router query metrics 2015-08-21 12:25:57 +05:30
Nishant e6b20db62d Support for JSON Smile format for EventReceiverFirehoseFactory
Support for Smile format for EventReceiverFireshoseFactory.
would help in decreasing network load on middle managers ingesting high
data volumes.

review comments
2015-08-20 19:52:45 +05:30
Fangjin Yang 6a0389297c Merge pull request #1634 from metamx/IrcFirehoseFixes
Allow IrcFirehoseFactory to shutdown cleanly
2015-08-18 21:06:34 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Charles Allen 13773865cf Make jetty scheduler threads daemon thread 2015-08-18 13:08:19 -07:00
Charles Allen cd2c3776f8 Allow IrcFirehoseFactory to shutdown cleanly 2015-08-17 13:24:41 -07:00
Himanshu Gupta 4d4aa8bfc6 refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable
Conflicts:
	indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
2015-08-14 14:44:22 -05:00
Charles Allen db19d2d547 Revert "Update to guice 4.0" 2015-08-14 09:26:07 -07:00
Charles Allen be89105621 Merge pull request #1602 from metamx/more-code-cleanup
Some perf Improvements in Broker
2015-08-11 13:51:49 -07:00
Charles Allen 7e61216287 Update to guice 4.0
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
2015-08-10 13:57:18 -07:00
Nishant b33c644da0 avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
profiling shows it took upto 6-12% of cpu time in
JodaUtils.condenseIntervals
2015-08-06 13:51:17 +05:30
Charles Allen 9dd18de1a5 Merge pull request #1596 from metamx/console-artifact
make console a maven dependency instead of filedump
2015-08-04 15:28:45 -07:00
Xavier Léauté 07ee1acca2 make console a maven dependency instead of filedump 2015-08-04 09:49:34 -07:00
Fangjin Yang b05be0a104 Merge pull request #1593 from metamx/run-tests-offline
Fix test so that it can run offline
2015-08-04 09:23:40 -07:00
Nishant f490547faf Fix test so that it can run offline too
Test was getting stuck for me when I tried to run it offline while
resolving the host. creating unresolved SocketAddress works around
this.
2015-08-03 23:37:04 +05:30
Bingkun Guo 479b09c2dd fix transient error in BrokerServerViewTest 2015-07-31 13:33:55 -05:00
Fangjin Yang 22567946cf Merge pull request #1259 from metamx/queryTimeLookup
Query Time Lookup
2015-07-28 11:43:05 -10:00
Charles Allen ba59f8afc4 Merge pull request #1550 from himanshug/optionally_log_all_requests
print *all* HTTP requests to log if configured
2015-07-28 11:19:36 -07:00
Charles Allen 86ede702b1 Add namespaced lookups as extensions
* Adds kafka, URI, and JDBC namespace defintions
* Add ability to explicitly rename using a "namespace" which is a particular data collection that is loaded on all realtime, historic nodes, and brokers. If any of these nodes has the namespace extension, ALL nodes have the namespace extension.
* Add namespace caching and populating (can be on heap or off heap)
* Add NamespaceExtractionCacheManager for handling caches
* Added ExtractionNamespace for handling metadata on the extraction namespaces
* Added ExtractionNamespaceUpdate for handling metadata related to updates
* Add extension which caches renames from a kafka stream (requires kafka8)
* Added README.md for the namespace kafka extension
* Added docs
* Added namespace/size, namespace/count, namespace/deltaTasksStarted metrics

Add static config for namespaces via `druid.query.extraction.namespace`
* This is a rebase of https://github.com/b-slim/druid/tree/static_config_only
2015-07-28 11:14:14 -07:00
Himanshu Gupta 90b475975e Log all the requests to debug level in logger io.druid.jetty.RequestLog 2015-07-28 12:56:04 -05:00
nishant efb6e0649e simplify merging of results on broker.
add benchmark

formatting and review comments

organize imports
2015-07-27 11:46:11 -07:00
Xavier Léauté 6e8da5de86 Merge pull request #1441 from metamx/queryResourceFlusher
Add flush prior to close on output stream in QueryResource
2015-07-17 16:32:52 -07:00
Fangjin Yang e21195f987 Merge pull request #1469 from guobingkun/table_config
Inconsistent property names for "druid.metadata.storage.tables.xxx"
2015-07-17 07:43:19 -07:00
Charles Allen 456ad9ffba Merge pull request #1529 from metamx/update-versions
inrement version
2015-07-15 13:25:31 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Fangjin Yang a4cb0332fb Merge pull request #1524 from metamx/peonShutdown
Make CliPeon run shutdown hooks properly
2015-07-15 10:09:33 -07:00
Charles Allen 92d5c250ab Remove comment 2015-07-14 16:20:36 -07:00
Charles Allen 5eadd395e2 Move lots of executor service creation to Execs 2015-07-14 15:38:49 -07:00
Charles Allen 2b87ad307d Make CliPeon run shutdown hooks properly
* Also make jetty threads daemon
2015-07-14 15:15:26 -07:00
Fangjin Yang da8e86146e Merge pull request #1505 from metamx/improve-test
Modify test to check for multiple shards for same interval
2015-07-14 08:50:47 -07:00
Fangjin Yang 3f7ba58227 Merge pull request #1504 from metamx/fix-1447
fix for #1447
2015-07-14 08:50:08 -07:00
Himanshu 1affdf7bec Merge pull request #1502 from samjhecht/poll-alerting
add alert on errors polling for rules
2015-07-13 18:37:37 -05:00
samjhecht fc438e5a72 add alert on errors polling for rules
reset retrystarttime after alerting
2015-07-13 09:03:51 -07:00
Himanshu 725086cc89 Merge pull request #1506 from gianm/realtime-plumber-nulls
Consider null inputRows and parse errors as unparseable during realtime ingestion.
2015-07-13 10:12:12 -05:00
Gian Merlino 9068bcd062 Consider null inputRows and parse errors as unparseable during realtime ingestion.
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
2015-07-11 20:40:03 -07:00
Himanshu e9b627bf55 Merge pull request #1476 from pjain1/improved_tests
all the review comments have been taken care of, so merging.
2015-07-09 16:25:30 -05:00
Nishant 1766341648 Modify test to check for multiple shards for same interval
Modify test to check for multiple shards for same interval.
2015-07-09 20:35:31 +05:30
Parag Jain 58cc3954ff improved coordinator and datasourceresource tests 2015-07-09 09:46:54 -05:00
Nishant 5fe27fe4ad fix for #1447
fixes #1447
2015-07-09 19:05:48 +05:30
Nishant 184b12bee8 fix groupBy caching to work with renamed aggregators
Issue - while storing results in cache we store the event map which
contains aggregator names mapped to values. Now when someone fire same
query after renaming aggs, the cache key will be same but the event
will contain metric values mapped to older names which leads to wrong
results.
Fix - modify cache to not store raw event but the actual list of values
only.

review comments + fix dimension renaming

review comment
2015-07-09 11:48:26 +05:30
Nishant fe36f7f852 emit alert for DB polling error 2015-07-07 19:51:36 +05:30
Fangjin Yang d846f3ced6 Merge pull request #1498 from rasahner/doc_localFirehose
say explicitly that local firehose searches dirs recursively for files
2015-07-05 15:34:36 -07:00
sahner acd20e8c00 say explicitly that local firehose searches directories recursively for files 2015-07-05 14:46:44 -05:00