Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,718 Commits 111 Branches 503 Tags 726 MiB
Commit Graph

990 Commits

Author SHA1 Message Date
Charles Allen 2a69a58570 Merge pull request #2149 from binlijin/master 
Do persist IncrementalIndex in another thread in IndexGeneratorReducer
 2016-01-20 17:06:42 -08:00
Charles Allen 2e1d6aaf3d Use thread priorities. (aka set `nice` values for background-like tasks) 
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
 * Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
 * Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
 * Add options to ForkingTaskRunner
    * Add "-XX:+UseThreadPriorities" default option
    * Add "-XX:ThreadPriorityPolicy=42" default option
 * AbstractTask - Removed unneded @JsonIgnore on priority
 * Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
 * Add persistThreadPriority and mergeThreadPriority to realtime tuning config
 2016-01-20 14:00:31 -08:00
Logan Linn c3bdaefe1f Update batch-ingestion.md 
Fix documented type of the `dataGranularity` config
 2016-01-19 17:20:47 -08:00
binlijin 8e43e2c446 Do persist IncrementalIndex in another thread in IndexGeneratorReducer 2016-01-20 09:20:09 +08:00
Fangjin Yang 0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata 
Store AggregatorFactory[] in segment metadata
 2016-01-19 12:55:49 -08:00
Himanshu Gupta a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
zhxiaog 3459a202ce fixed #1873, add ability to express CONCAT as an extractionFn 2016-01-18 15:03:17 -08:00
Fangjin Yang 592638f6b3 Merge pull request #2209 from sirpkt/cascade_extraction_functions 
support cascade execution of extraction filters in extraction dimension spec
 2016-01-18 07:59:36 -08:00
Gian Merlino 42ee473052 ZooKeeper 3.4.7 was unreleased, revert to 3.4.6. 2016-01-17 21:31:29 -08:00
Keuntae Park 238dd3be3c support cascade execution of extraction filters in extraction dimension spec 2016-01-18 11:10:19 +09:00
Gian Merlino 2d3f6e7705 Some more multitenancy docs 2016-01-17 17:47:49 -08:00
Harsh Jain 71f1cd5e34 Fix for issue 2021 2016-01-17 16:10:04 +05:30
Fangjin Yang f6a1a4ae20 Merge pull request #2138 from KurtYoung/feature-build-v9 
build v9 directly
 2016-01-16 13:35:46 -06:00
Kurt Young 82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Fangjin Yang 2e54553a8f Merge pull request #1990 from himanshug/schedule_kill_task 
support periodic hard delete of segments
 2016-01-15 15:22:33 -06:00
David Lim 34cd8f8c72 Merge pull request #2258 from fjy/acl-zk 
acl for zookeeper is added
 2016-01-15 10:27:08 -07:00
Nikita Geer 1908d63162 acl for zookeeper is added 2016-01-13 14:56:05 -08:00
navis.ryu 18479bb757 time-descending result of timeseries queries 2016-01-13 12:23:01 +09:00
Fangjin Yang d7ad93debc Merge pull request #2221 from binlijin/topN_minTopNThreshold 
Allow change minTopNThreshold per topN query
 2016-01-12 16:22:20 -08:00
Himanshu Gupta eb2d251ac8 support periodic hard delete of segments 2016-01-12 16:55:05 -06:00
binlijin 2751f785f8 add doc 2016-01-12 11:25:11 +08:00
Fangjin Yang fafc7c07ba Merge pull request #2238 from rasahner/docServerviewEndpoint 
doc: add information about new serverview coordinator endpoint
 2016-01-11 10:44:18 -08:00
Robin c1b813011c doc: add information about new serverview coordinator endpoint 2016-01-10 19:47:23 -06:00
Himanshu d255f4baac Merge pull request #2234 from pjain1/emit_realtime_metrics 
emit handoff count metrics
 2016-01-08 14:24:16 -06:00
Parag Jain 9dba0f67e7 emit handoff count metrics 2016-01-08 12:36:13 -06:00
Himanshu Gupta 0d5f82aee7 document size attribute in thetaSketchSetOp post aggregator 2016-01-07 23:59:03 -06:00
Fangjin Yang 3048b1f0a5 Merge pull request #2174 from metamx/ingest-size-metrics 
Add metrics for ingest/bytes/received for EventReceiverFirehose
 2016-01-06 22:05:55 -08:00
Fangjin Yang 4cc81d3eff Merge pull request #2096 from b-slim/add_use_case_unapply 
Add use case unapply
 2016-01-06 21:58:12 -08:00
Fangjin Yang dd262f0451 Merge pull request #2215 from pjain1/fix_doc_metrics 
correct metric name - segment/added/count -> segment/assigned/count
 2016-01-06 16:21:54 -08:00
Parag Jain 768d07b702 correct metric name - segment/added/count -> segment/assigned/count 2016-01-06 15:55:11 -06:00
Slim Bouguerra 032d3bf6e6 Optimization of extraction filter by reversing the lookup 2016-01-06 11:16:11 -06:00
Himanshu 82bdfbbbf1 Merge pull request #2155 from metamx/taskConfigTmpdir 
Make TaskConfig pull from java.io.tmpdir
 2016-01-05 13:58:39 -06:00
Nishant 14989f272d Add metrics for ingest/bytes/received for EventReceiverFirehose 
review comments

review comments
 2016-01-05 20:06:09 +05:30
Zhao Weinan 5e57ddb8cc Adding avro support to realtime & hadoop batch indexing. 2016-01-05 10:21:27 +08:00
Robin 0961c0b703 trivial documentation fix 2016-01-04 12:39:10 -06:00
Himanshu ebccea6bb0 Merge pull request #2188 from druid-io/doc-fix 
small fixes to documentation
 2016-01-04 12:25:26 -06:00
Charles Allen e18301d99c Make TaskConfig pull from java.io.tmpdir 
* Also makes paths built off of java.nio.file.Paths instead of String.format
 2016-01-04 10:17:08 -08:00
Robin e280ab5f07 update zookeeper version to 3.4.7 2016-01-04 11:47:02 -06:00
fjy d3d2ee03ce minor fixes to docs 2016-01-03 11:37:06 -08:00
Gian Merlino 5a63c3dd63 Merge pull request #2186 from druid-io/dev-docs2 
Add intro developer docs
 2016-01-03 11:36:41 -05:00
fjy 88f6b9b5ad Multiple improvements for docs 2016-01-02 21:54:54 -08:00
fjy 06a8e14820 Add intro developer docs 2016-01-02 14:44:45 -08:00
Himanshu Gupta 48de9dfafa doc update to make it easy to find how to do re-indexing or delta ingestion 2015-12-30 23:58:09 -06:00
Himanshu Gupta e1ea93b6fc documenting querying behavior on multi-valued dimensions 2015-12-30 18:14:25 -06:00
Himanshu Gupta b47d807738 Add support for filtering at DimensionSpec level so that multivalued dimensions can be filtered correctly 
also adding UTs for multi-valued dimensions
 2015-12-30 17:59:47 -06:00
Bingkun Guo 3c107c5757 Merge pull request #2150 from himanshug/emit_query_bytes 
emit query/bytes metric
 2015-12-30 13:44:19 -06:00
Fangjin Yang e14ad74088 Merge pull request #1936 from b-slim/between_range_with_predicat 
adding Upper/Lower Bound Filter
 2015-12-29 10:11:22 -08:00
Fangjin Yang b1261035a7 Merge pull request #1861 from guobingkun/insert_segment_tool 
insert-segment tool
 2015-12-29 10:06:07 -08:00
Fangjin Yang 5042b20eb4 Merge pull request #2164 from pdeva/patch-7 
correct docs
 2015-12-27 14:43:51 -08:00
pdeva 77863285e9 fix typo 2015-12-27 14:28:23 -08:00
pdeva b308a13483 correct docs 2015-12-27 14:27:20 -08:00
pdeva 98fc08b0de List who can access metadata storage 
This is useful to know which ip addresses you need to allow to access your RDS instance for example
 2015-12-27 14:25:17 -08:00
Himanshu Gupta 1a8546a682 emit query/bytes metric 2015-12-23 00:29:44 -06:00
Fangjin Yang daaa24d7ad Merge pull request #2145 from gianm/restoring-option 
druid.indexer.task.restoreTasksOnRestart configuration.
 2015-12-22 15:22:32 -08:00
Gian Merlino bad270b6c4 druid.indexer.task.restoreTasksOnRestart configuration. 2015-12-22 10:59:15 -08:00
Himanshu Gupta b96f560255 emit query/node/bytes metric 2015-12-21 23:23:20 -06:00
Charles Allen c2a2d19d62 Merge pull request #2126 from guobingkun/remove_single_announcer 
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
 2015-12-21 08:55:07 -08:00
Bingkun Guo 951a4e9b35 Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer 2015-12-21 00:05:53 -06:00
Bingkun Guo 89b477970f DataSegmentFinder tool 
`insert-segment-to-db` is a tool that can insert segments into Druid metadata storage. It is intended to be used
to update the segment table in metadata storage after people manually migrate segments from one place to another.
It can also be used to insert missing segment into Druid, or even recover metadata storage by telling it where the
segments are stored.

Note: This tool expects users to have Druid cluster running in a "safe" mode, where there are no active tasks to interfere
the segments being inserted. Users can optionally bring down the cluster to make 100% sure nothing is interfering.
 2015-12-21 00:02:04 -06:00
Fangjin Yang 7019d3c421 Merge pull request #2107 from jon-wei/fix_smq 
More efficient SegmentMetadataQuery
 2015-12-18 16:40:47 -08:00
fjy 398a3ec620 add docs for more specs 2015-12-17 18:06:30 -08:00
jon-wei 356b07c6c3 More efficient SegmentMetadataQuery 2015-12-17 12:46:23 -08:00
Jonathan Wei f8cf84f466 Merge pull request #1995 from himanshug/num_rows_seg_metadata_query 
add numRows to segment metadata query response
 2015-12-17 12:23:46 -08:00
Fangjin Yang cb085f99d3 Merge pull request #2097 from metamx/update-coordinator-docs 
update the coordinator API docs to better explain how to format ISO86…
 2015-12-15 17:35:48 -08:00
desaianuj 2f32cb1fb6 update the coordinator API docs to better explain how to format ISO8601 intervals in the request url 2015-12-15 16:17:54 -08:00
Gian Merlino e6c2db89da Merge pull request #1921 from jon-wei/flat_json 
Add docs and benchmark for JSON flattening parser
 2015-12-14 20:53:54 -08:00
Fangjin Yang e7f06cf61c Merge pull request #2075 from jon-wei/regex_extract 
Configurable value replacement on match failure for RegexExtractionFn
 2015-12-14 19:10:50 -08:00
jon-wei c88f75df7c Configurable value replacement on match failure for RegexExtractionFn 2015-12-14 17:57:41 -08:00
Steve M 2b5a010332 Change sample worker config spec with host:port instead of ip:port. 
Also extend description of the 'affinity' property of the worker strategy
fillCapacityWithAffinity and fix a couple typos of middle manager (to
be more consistent throughout the page).

Add additional verbiage about appropriate middle manager host value.
 2015-12-14 14:59:23 -08:00
Slim Bouguerra 77afdf25e3 adding Bound Filter 2015-12-10 08:47:21 -06:00
jon-wei c53bf85d83 Add docs and benchmark for JSON flattening parser 2015-12-09 16:13:30 -08:00
Fangjin Yang b0ab363022 Merge pull request #2052 from gianm/service-names 
Change service names in docs, examples to match defaults in the code.
 2015-12-08 15:40:35 -08:00
Xavier Léauté dcd15735cd Merge pull request #2015 from metamx/handoff-notifier-coordinator 
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information
 2015-12-08 14:06:39 -08:00
Nishant 9491e8de3b Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs 
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView

review comment

fix realtime node guide injection

review comments

make test not rely on scheduled exec

fix compilation

fix import

review comment

introduce immutableSegmentLoadInfo

fix son reading

remove unnecessary logging
 2015-12-09 01:54:09 +05:30
Nishant 6c23d8edb4 Merge pull request #2043 from mangeshpardeshiyahoo/master 
Add dimension selector support for groupby/having filters
 2015-12-08 12:08:53 +05:30
Mangesh Pardeshi d7ce120929 Add dimension selector support for groupby/having quries 2015-12-08 01:51:11 +00:00
michael.schiff 4ed0128b3b removed documentation about (removed) DeleteTask 2015-12-07 15:56:46 -08:00
Fangjin Yang 00416e26ff Merge pull request #2012 from himanshug/add_comment 
adding more comments to why drops do not happen
 2015-12-07 11:50:06 -08:00
Fangjin Yang fe05545f58 Merge pull request #2055 from jkukul/documentation-fixes 
Minor documentation fixes.
 2015-12-07 10:50:38 -08:00
Jakub Kukul 8248f7f40c Minor docs fixes in 'Evaluate Druid' section. 2015-12-07 19:42:52 +01:00
AdrieanKhisbe 5d10088e2d Update Tutorial with new Zookeper Version and instructions 2015-12-07 06:50:42 +01:00
Gian Merlino 8e594a2e72 Change service names in docs, examples to match defaults in the code. 2015-12-06 10:04:21 -08:00
Himanshu Gupta efe3c9f4a5 update the examples for batch reindexing/delta ingestion to use "intervals" instead of deprecated "interval" 2015-12-06 00:22:20 -06:00
Himanshu Gupta 61aaa09012 support multiple intervals in dataSource input spec 2015-12-03 21:28:04 -06:00
Himanshu 0eab8417cb Merge pull request #2008 from codingwhatever/regex-search-query 
Regex search query
 2015-12-03 09:57:34 -06:00
Sam Groth 596b7ebd9a Adding RegexSearchQuerySpec 2015-12-03 09:16:02 -06:00
Clint Wylie 68ef5f437a fixes #1874 adding a substring extraction function, tests, and documentation 2015-12-01 23:50:32 -08:00
Himanshu Gupta 7c91219130 adding more comments to why drops do not happen when there are no segments in db 2015-11-25 16:26:51 -06:00
jon-wei 95dca4440f Update data formats doc with info about JSON multi-value dimensions 2015-11-24 14:38:06 -08:00
Fangjin Yang 8e83d800d6 Merge pull request #1881 from gianm/restartable-tasks 
Restorable indexing tasks
 2015-11-23 21:14:37 -08:00
Gian Merlino 501dcb43fa Some changes that make it possible to restart tasks on the same hardware. 
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.

The main changes are,

1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
   stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
   one that currently implements this.

2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
   ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
   ForkingTaskRunner- close output stream to restorable tasks, wait for exit
   RemoteTaskRunner- do nothing special, we actually don't want to shutdown

3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
   Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
   a list of restorable tasks.

4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
   task and a zombie old task from stomping on each other.
 2015-11-23 11:22:08 -08:00
Gian Merlino c0580bf063 Merge pull request #1928 from druid-io/new-compares 
New comparisons for Druid
 2015-11-20 16:40:59 -08:00
Himanshu Gupta 7a89b2e1a6 add numRows to segment metadata query response 2015-11-20 01:25:02 -06:00
Himanshu Gupta fde9df2720 update to sketches-core-0.2.2 . 
adds support for "cardinality" aggregator.
do not create sketch per event at ingestion time to make realtime ingestion faster
 2015-11-19 01:05:59 -06:00
Fangjin Yang 87c43fbf02 Merge pull request #1981 from rasahner/fixFormattingSchemaDesign 
fix formatting in schema-design
 2015-11-17 17:51:16 -08:00
Charles Allen 8fcf2403e3 Merge pull request #1943 from metamx/realtime-caching 
Enable caching on intermediate realtime persists
 2015-11-17 15:06:43 -08:00
sahner a4ed2ce2d1 fix formatting in schema-design 2015-11-17 16:50:53 -06:00
Himanshu 9118b010c0 Merge pull request #1976 from noddi/feature/count-parameter-rules-history 
Add count parameter to rules history endpoint.
 2015-11-17 14:41:50 -06:00
Charles Allen dbe201aeed Merge pull request #1929 from pjain1/jetty_threads 
separate ingestion and query thread pool
 2015-11-17 12:14:25 -08:00
Parag Jain 6c498b7d4a separate ingestion and query thread pool 2015-11-17 13:42:41 -06:00