mirror of https://github.com/apache/druid.git
Finish converting docs over to something that displays properly
This commit is contained in:
cheddar
parent
154b58defd
commit
65bb68c7de
|
|
@ -0,0 +1,51 @@
|
|||
|
||||
<!-- Start page_footer include -->
|
||||
<div class="container">
|
||||
<footer>
|
||||
<div class="container">
|
||||
<hr>
|
||||
<div class="row">
|
||||
<div class="col-md-4">
|
||||
<address>
|
||||
<strong>CONTACT US</strong>
|
||||
<a href="mailto:info@druid.io">info@druid.io</a>
|
||||
</address>
|
||||
<address>
|
||||
<strong>Metamarkets</strong>
|
||||
625 2nd Street, Suite #230<br>
|
||||
San Francisco, CA 94017<br>
|
||||
<div class="soc">
|
||||
<a href="https://twitter.com/druidio"></a>
|
||||
<a href="https://github.com/metamx/druid" class="github"></a>
|
||||
<a href="http://www.meetup.com/Open-Druid/" class="meet"></a>
|
||||
<a href="http://druid.io/feed/" class="rss" target="_blank"></a>
|
||||
</div>
|
||||
</div>
|
||||
<ul class="col-md-4 list-unstyled">
|
||||
<li><a href="/"><strong>DRUID</strong></a></li>
|
||||
<li><a href="/druid.html">What is Druid?</a></li>
|
||||
<li><a href="/downloads.html">Downloads</a></li>
|
||||
<li><a target="_blank" href="https://github.com/metamx/druid/wiki">Documentation</a></li>
|
||||
</ul>
|
||||
<ul class="col-md-4 list-unstyled">
|
||||
<li><a href="/community.html"><strong>SUPPORT</strong></a></li>
|
||||
<li><a href="/community.html">Community</a></li>
|
||||
<li><a href="/faq.html">FAQ</a></li>
|
||||
<li><a href="/licensing.html">Licensing</a></li>
|
||||
<li><a href="/blog"><strong>BLOG</strong></a></li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
</footer>
|
||||
</div>
|
||||
<script type="text/javascript">
|
||||
var gaJsHost = (("https:" == document.location.protocol) ? "https://ssl." : "http://www.");
|
||||
document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E"));
|
||||
</script>
|
||||
<script type="text/javascript">
|
||||
try {
|
||||
var pageTracker = _gat._getTracker("UA-40280432-1");
|
||||
pageTracker._trackPageview();
|
||||
} catch(err) {}
|
||||
</script>
|
||||
<!-- stop page_footer include -->
|
||||
|
|
@ -23,6 +23,7 @@
|
|||
</div>
|
||||
</div>
|
||||
|
||||
{% include page_footer.html %}
|
||||
<script src="http://code.jquery.com/jquery.js"></script>
|
||||
<script src="http://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js"></script>
|
||||
<script>
|
||||
|
|
|
|||
|
|
@ -4,17 +4,23 @@ layout: doc_page
|
|||
Aggregations are specifications of processing over metrics available in Druid.
|
||||
Available aggregations are:
|
||||
|
||||
### Count aggregator
|
||||
|
||||
`count` computes the row count that match the filters
|
||||
|
||||
```json
|
||||
{ "type" : "count", "name" : <output_name> }
|
||||
```
|
||||
|
||||
### Sum aggregators
|
||||
|
||||
#### `longSum` aggregator
|
||||
|
||||
computes the sum of values as a 64-bit, signed integer
|
||||
|
||||
<code>{
|
||||
"type" : "longSum",
|
||||
"name" : <output_name>,
|
||||
"fieldName" : <metric_name>
|
||||
}</code>
|
||||
```json
|
||||
{ "type" : "longSum", "name" : <output_name>, "fieldName" : <metric_name> }
|
||||
```
|
||||
|
||||
`name` – output name for the summed value
|
||||
`fieldName` – name of the metric column to sum over
|
||||
|
|
@ -23,20 +29,9 @@ computes the sum of values as a 64-bit, signed integer
|
|||
|
||||
Computes the sum of values as 64-bit floating point value. Similar to `longSum`
|
||||
|
||||
<code>{
|
||||
"type" : "doubleSum",
|
||||
"name" : <output_name>,
|
||||
"fieldName" : <metric_name>
|
||||
}</code>
|
||||
|
||||
### Count aggregator
|
||||
|
||||
`count` computes the row count that match the filters
|
||||
|
||||
<code>{
|
||||
"type" : "count",
|
||||
"name" : <output_name>,
|
||||
}</code>
|
||||
```json
|
||||
{ "type" : "doubleSum", "name" : <output_name>, "fieldName" : <metric_name> }
|
||||
```
|
||||
|
||||
### Min / Max aggregators
|
||||
|
||||
|
|
@ -44,21 +39,17 @@ Computes the sum of values as 64-bit floating point value. Similar to `longSum`
|
|||
|
||||
`min` computes the minimum metric value
|
||||
|
||||
<code>{
|
||||
"type" : "min",
|
||||
"name" : <output_name>,
|
||||
"fieldName" : <metric_name>
|
||||
}</code>
|
||||
```json
|
||||
{ "type" : "min", "name" : <output_name>, "fieldName" : <metric_name> }
|
||||
```
|
||||
|
||||
#### `max` aggregator
|
||||
|
||||
`max` computes the maximum metric value
|
||||
|
||||
<code>{
|
||||
"type" : "max",
|
||||
"name" : <output_name>,
|
||||
"fieldName" : <metric_name>
|
||||
}</code>
|
||||
```json
|
||||
{ "type" : "max", "name" : <output_name>, "fieldName" : <metric_name> }
|
||||
```
|
||||
|
||||
### JavaScript aggregator
|
||||
|
||||
|
|
@ -66,25 +57,27 @@ Computes an arbitrary JavaScript function over a set of columns (both metrics an
|
|||
|
||||
All JavaScript functions must return numerical values.
|
||||
|
||||
<code>{
|
||||
"type": "javascript",
|
||||
"name": "<output_name>",
|
||||
```json
|
||||
{ "type": "javascript", "name": "<output_name>",
|
||||
"fieldNames" : [ <column1>, <column2>, ... ],
|
||||
"fnAggregate" : "function(current, column1, column2, ...) {
|
||||
<updates partial aggregate (current) based on the current row values>
|
||||
return <updated partial aggregate>
|
||||
}"
|
||||
"fnCombine" : "function(partialA, partialB) { return <combined partial results>; }"
|
||||
}",
|
||||
"fnCombine" : "function(partialA, partialB) { return <combined partial results>; }",
|
||||
"fnReset" : "function() { return <initial value>; }"
|
||||
}</code>
|
||||
}
|
||||
```
|
||||
|
||||
**Example**
|
||||
|
||||
<code>{
|
||||
```json
|
||||
{
|
||||
"type": "javascript",
|
||||
"name": "sum(log(x)/y) + 10",
|
||||
"fieldNames": ["x", "y"],
|
||||
"fnAggregate" : "function(current, a, b) { return current + (Math.log(a) * b); }"
|
||||
"fnCombine" : "function(partialA, partialB) { return partialA + partialB; }"
|
||||
"fnAggregate" : "function(current, a, b) { return current + (Math.log(a) * b); }",
|
||||
"fnCombine" : "function(partialA, partialB) { return partialA + partialB; }",
|
||||
"fnReset" : "function() { return 10; }"
|
||||
}</code>
|
||||
}
|
||||
```
|
||||
|
|
|
|||
|
|
@ -18,14 +18,14 @@ HadoopDruidIndexer
|
|||
|
||||
Located at `com.metamx.druid.indexer.HadoopDruidIndexerMain` can be run like
|
||||
|
||||
<code>
|
||||
java -cp hadoop_config_path:druid_indexer_selfcontained_jar_path com.metamx.druid.indexer.HadoopDruidIndexerMain <config_file>
|
||||
</code>
|
||||
```
|
||||
java -cp hadoop_config_path:druid_indexer_selfcontained_jar_path com.metamx.druid.indexer.HadoopDruidIndexerMain <config_file>
|
||||
```
|
||||
|
||||
The interval is the [ISO8601 interval](http://en.wikipedia.org/wiki/ISO_8601#Time_intervals) of the data you are processing. The config\_file is a path to a file (the “specFile”) that contains JSON and an example looks like:
|
||||
The interval is the [ISO8601 interval](http://en.wikipedia.org/wiki/ISO_8601#Time_intervals) of the data you are processing. The config\_file is a path to a file (the "specFile") that contains JSON and an example looks like:
|
||||
|
||||
<code>
|
||||
{
|
||||
```
|
||||
{
|
||||
"dataSource": "the_data_source",
|
||||
"timestampColumn": "ts",
|
||||
"timestampFormat": "<iso, millis, posix, auto or any Joda time format>",
|
||||
|
|
@ -62,8 +62,8 @@ The interval is the [ISO8601 interval](http://en.wikipedia.org/wiki/ISO_8601#Tim
|
|||
"password":"passmeup",
|
||||
"segmentTable":"segments"
|
||||
}
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
### Hadoop indexer config
|
||||
|
||||
|
|
@ -100,10 +100,12 @@ Is a type of data loader that expects data to be laid out in a specific path for
|
|||
|
||||
For example, if the sample config were run with the interval 2012-06-01/2012-06-02, it would expect data at the paths
|
||||
|
||||
s3n://billy-bucket/the/data/is/here/y=2012/m=06/d=01/H=00
|
||||
s3n://billy-bucket/the/data/is/here/y=2012/m=06/d=01/H=01
|
||||
...
|
||||
s3n://billy-bucket/the/data/is/here/y=2012/m=06/d=01/H=23
|
||||
```
|
||||
s3n://billy-bucket/the/data/is/here/y=2012/m=06/d=01/H=00
|
||||
s3n://billy-bucket/the/data/is/here/y=2012/m=06/d=01/H=01
|
||||
...
|
||||
s3n://billy-bucket/the/data/is/here/y=2012/m=06/d=01/H=23
|
||||
```
|
||||
|
||||
### Rollup specification
|
||||
|
||||
|
|
@ -116,7 +118,7 @@ The indexing process has the ability to roll data up as it processes the incomin
|
|||
|
||||
### Partitioning specification
|
||||
|
||||
Segments are always partitioned based on timestamp (according to the granularitySpec) and may be further partitioned in some other way. For example, data for a day may be split by the dimension “last\_name” into two segments: one with all values from A-M and one with all values from N-Z.
|
||||
Segments are always partitioned based on timestamp (according to the granularitySpec) and may be further partitioned in some other way. For example, data for a day may be split by the dimension "last\_name" into two segments: one with all values from A-M and one with all values from N-Z.
|
||||
|
||||
To use this option, the indexer must be given a target partition size. It can then find a good set of partition ranges on its own.
|
||||
|
||||
|
|
@ -132,7 +134,7 @@ This is a specification of the properties that tell the job how to update metada
|
|||
|
||||
|property|description|required?|
|
||||
|--------|-----------|---------|
|
||||
|type|“db” is the only value available|yes|
|
||||
|type|"db" is the only value available|yes|
|
||||
|connectURI|a valid JDBC url to MySQL|yes|
|
||||
|user|username for db|yes|
|
||||
|password|password for db|yes|
|
||||
|
|
|
|||
|
|
@ -20,5 +20,6 @@ LICENSE client eclipse_formatting.xml index-common merger realtime
|
|||
```
|
||||
|
||||
You can find the example executables in the examples/bin directory:
|
||||
|
||||
* run_example_server.sh
|
||||
* run_example_client.sh
|
||||
|
|
|
|||
|
|
@ -4,12 +4,11 @@ layout: doc_page
|
|||
Concepts and Terminology
|
||||
========================
|
||||
|
||||
- **Aggregators:** A mechanism for combining records during realtime incremental indexing, Hadoop batch indexing, and in queries.
|
||||
- **DataSource:** A table-like view of data; specified in a “specFile” and in a query.
|
||||
- **Granularity:** The time interval corresponding to aggregation by time.
|
||||
- The *indexGranularity* setting in a schema is used to aggregate input (ingest) records within an interval into a single output (internal) record.
|
||||
- The *segmentGranularity* is the interval specifying how internal records are stored together in a single file.
|
||||
|
||||
- **Segment:** A collection of (internal) records that are stored and processed together.
|
||||
- **Shard:** A unit of partitioning data across machine. TODO: clarify; by time or other dimensions?
|
||||
- **specFile** is specification for services in JSON format; see [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html)
|
||||
* **Aggregators**: A mechanism for combining records during realtime incremental indexing, Hadoop batch indexing, and in queries.
|
||||
* **DataSource**: A table-like view of data; specified in a "specFile" and in a query.
|
||||
* **Granularity**: The time interval corresponding to aggregation by time.
|
||||
* **indexGranularity**: specifies the granularity used to bucket timestamps within a segment.
|
||||
* **segmentGranularity**: specifies the granularity of the segment, i.e. the amount of time a segment will represent
|
||||
* **Segment**: A collection of (internal) records that are stored and processed together.
|
||||
* **Shard**: A sub-partition of the data in a segment. It is possible to have multiple segments represent all data for a given segmentGranularity.
|
||||
* **specFile**: is specification for services in JSON format; see [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html)
|
||||
|
|
|
|||
|
|
@ -1,14 +1,14 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
This describes the basic server configuration that is loaded by all the server processes; the same file is loaded by all. See also the json “specFile” descriptions in [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html).
|
||||
This describes the basic server configuration that is loaded by all the server processes; the same file is loaded by all. See also the json "specFile" descriptions in [Realtime](Realtime.html) and [Batch-ingestion](Batch-ingestion.html).
|
||||
|
||||
JVM Configuration Best Practices
|
||||
================================
|
||||
|
||||
There are three JVM parameters that we set on all of our processes:
|
||||
|
||||
1. `-Duser.timezone=UTC` This sets the doc_page timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs
|
||||
1. `-Duser.timezone=UTC` This sets the default timezone of the JVM to UTC. We always set this and do not test with other default timezones, so local timezones might work, but they also might uncover weird and interesting bugs
|
||||
2. `-Dfile.encoding=UTF-8` This is similar to timezone, we test assuming UTF-8. Local encodings might work, but they also might result in weird and interesting bugs
|
||||
3. `-Djava.io.tmpdir=<a path>` Various parts of the system that interact with the file system do it via temporary files, these files can get somewhat large. Many production systems are setup to have small (but fast) `/tmp` directories, these can be problematic with Druid so we recommend pointing the JVM’s tmp directory to something with a little more meat.
|
||||
|
||||
|
|
@ -17,7 +17,7 @@ Basic Service Configuration
|
|||
|
||||
Configuration of the various nodes is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath. Note: as a future item, I’d like to consolidate all of the various configuration into a yaml/JSON based configuration files.
|
||||
|
||||
The periodic time intervals (like “PT1M”) are [ISO8601 intervals](http://en.wikipedia.org/wiki/ISO_8601#Time_intervals)
|
||||
The periodic time intervals (like "PT1M") are [ISO8601 intervals](http://en.wikipedia.org/wiki/ISO_8601#Time_intervals)
|
||||
|
||||
An example runtime.properties is as follows:
|
||||
|
||||
|
|
@ -50,7 +50,7 @@ druid.server.maxSize=300000000000
|
|||
druid.zk.service.host=
|
||||
# ZK path prefix for Druid-usage of zookeeper, Druid will create multiple paths underneath this znode
|
||||
druid.zk.paths.base=/druid
|
||||
# ZK path for discovery, the only path not to doc_page to anything
|
||||
# ZK path for discovery, the only path not to default to anything
|
||||
druid.zk.paths.discoveryPath=/druid/discoveryPath
|
||||
|
||||
# the host:port as advertised to clients
|
||||
|
|
@ -91,7 +91,7 @@ These properties are for connecting with S3 and using it to pull down segments.
|
|||
|
||||
### JDBC connection
|
||||
|
||||
These properties specify the jdbc connection and other configuration around the “segments table” database. The only processes that connect to the DB with these properties are the [Master](Master.html) and [Indexing service](Indexing-service.html). This is tested on MySQL.
|
||||
These properties specify the jdbc connection and other configuration around the "segments table" database. The only processes that connect to the DB with these properties are the [Master](Master.html) and [Indexing service](Indexing-service.html). This is tested on MySQL.
|
||||
|
||||
|Property|Description|Default|
|
||||
|--------|-----------|-------|
|
||||
|
|
@ -142,7 +142,7 @@ These are properties that the compute nodes use
|
|||
|Property|Description|Default|
|
||||
|--------|-----------|-------|
|
||||
|`druid.server.maxSize`|The maximum number of bytes worth of segment that the node wants assigned to it. This is not a limit that the compute nodes actually enforce, they just publish it to the master and trust the master to do the right thing|none|
|
||||
|`druid.server.type`|Specifies the type of the node. This is published via ZK and depending on the value the node will be treated specially by the Master/Broker. Allowed values are “realtime” or “historical”. This is a configuration parameter because the plan is to allow for a more configurable cluster composition. At the current time, all realtime nodes should just be “realtime” and all compute nodes should just be “compute”|none|
|
||||
|`druid.server.type`|Specifies the type of the node. This is published via ZK and depending on the value the node will be treated specially by the Master/Broker. Allowed values are "realtime" or "historical". This is a configuration parameter because the plan is to allow for a more configurable cluster composition. At the current time, all realtime nodes should just be "realtime" and all compute nodes should just be "compute"|none|
|
||||
|
||||
### Emitter Properties
|
||||
|
||||
|
|
@ -150,7 +150,7 @@ The Druid servers emit various metrics and alerts via something we call an [Emit
|
|||
|
||||
|Property|Description|Default|
|
||||
|--------|-----------|-------|
|
||||
|`com.metamx.emitter.logging`|Set to “true” to use the logging emitter|none|
|
||||
|`com.metamx.emitter.logging`|Set to "true" to use the logging emitter|none|
|
||||
|`com.metamx.emitter.logging.level`|Sets the level to log at|debug|
|
||||
|`com.metamx.emitter.logging.class`|Sets the class to log at|com.metamx.emiter.core.LoggingEmitter|
|
||||
|
||||
|
|
|
|||
|
|
@ -7,14 +7,16 @@ The currently supported types of deep storage follow.
|
|||
|
||||
## S3-compatible
|
||||
|
||||
S3-compatible deep storage is basically either S3 or something like riak-cs which exposes the same API as S3. This is the doc_page deep storage implementation.
|
||||
S3-compatible deep storage is basically either S3 or something like riak-cs which exposes the same API as S3. This is the default deep storage implementation.
|
||||
|
||||
S3 configuration parameters are
|
||||
|
||||
com.metamx.aws.accessKey=<S3 access key>
|
||||
com.metamx.aws.secretKey=<S3 secret_key>
|
||||
druid.pusher.s3.bucket=<bucket to store in>
|
||||
druid.pusher.s3.baseKey=<base key prefix to use, i.e. what directory>
|
||||
```
|
||||
com.metamx.aws.accessKey=<S3 access key>
|
||||
com.metamx.aws.secretKey=<S3 secret_key>
|
||||
druid.pusher.s3.bucket=<bucket to store in>
|
||||
druid.pusher.s3.baseKey=<base key prefix to use, i.e. what directory>
|
||||
```
|
||||
|
||||
## HDFS
|
||||
|
||||
|
|
@ -22,8 +24,10 @@ As of 0.4.0, HDFS can be used for storage of segments as well.
|
|||
|
||||
In order to use hdfs for deep storage, you need to set the following configuration on your realtime nodes.
|
||||
|
||||
druid.pusher.hdfs=true
|
||||
druid.pusher.hdfs.storageDirectory=<directory for storing segments>
|
||||
```
|
||||
druid.pusher.hdfs=true
|
||||
druid.pusher.hdfs.storageDirectory=<directory for storing segments>
|
||||
```
|
||||
|
||||
If you are using the Hadoop indexer, set your output directory to be a location on Hadoop and it will work
|
||||
|
||||
|
|
@ -34,8 +38,10 @@ A local mount can be used for storage of segments as well. This allows you to u
|
|||
|
||||
In order to use a local mount for deep storage, you need to set the following configuration on your realtime nodes.
|
||||
|
||||
druid.pusher.local=true
|
||||
druid.pusher.local.storageDirectory=<directory for storing segments>
|
||||
```
|
||||
druid.pusher.local=true
|
||||
druid.pusher.local.storageDirectory=<directory for storing segments>
|
||||
```
|
||||
|
||||
Note that you should generally set `druid.pusher.local.storageDirectory` to something different from `druid.paths.indexCache`.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,13 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
|
||||
For a comprehensive look at the architecture of Druid, read the [White Paper](http://static.druid.io/docs/druid.pdf).
|
||||
|
||||
What is Druid?
|
||||
==============
|
||||
|
||||
Druid is a system built to allow fast (“real-time”) access to large sets of seldom-changing data. It was designed with the intent of being a service and maintaining 100% uptime in the face of code deployments, machine failures and other eventualities of a production system. It can be useful for back-office use cases as well, but design decisions were made explicitly targetting an always-up service.
|
||||
Druid is a system built to allow fast ("real-time") access to large sets of seldom-changing data. It was designed with the intent of being a service and maintaining 100% uptime in the face of code deployments, machine failures and other eventualities of a production system. It can be useful for back-office use cases as well, but design decisions were made explicitly targetting an always-up service.
|
||||
|
||||
Druid currently allows for single-table queries in a similar manner to [Dremel](http://research.google.com/pubs/pub36632.html) and [PowerDrill](http://www.vldb.org/pvldb/vol5/p1436_alexanderhall_vldb2012.pdf). It adds to the mix
|
||||
|
||||
|
|
@ -18,20 +19,21 @@ Druid currently allows for single-table queries in a similar manner to [Dremel](
|
|||
|
||||
As far as a comparison of systems is concerned, Druid sits in between PowerDrill and Dremel on the spectrum of functionality. It implements almost everything Dremel offers (Dremel handles arbitrary nested data structures while Druid only allows for a single level of array-based nesting) and gets into some of the interesting data layout and compression methods from PowerDrill.
|
||||
|
||||
Druid is a good fit for products that require real-time data ingestion of a single, large data stream. Especially if you are targetting no-downtime operation and are building your product on top of a time-oriented summarization of the incoming data stream. Druid is probably not the right solution if you care more about query flexibility and raw data access than query speed and no-downtime operation. When talking about query speed it is important to clarify what “fast” means, with Druid it is entirely within the realm of possibility (we have done it) to achieve queries that run in single-digit seconds across a 6TB data set.
|
||||
Druid is a good fit for products that require real-time data ingestion of a single, large data stream. Especially if you are targetting no-downtime operation and are building your product on top of a time-oriented summarization of the incoming data stream. Druid is probably not the right solution if you care more about query flexibility and raw data access than query speed and no-downtime operation. When talking about query speed it is important to clarify what "fast" means, with Druid it is entirely within the realm of possibility (we have done it) to achieve queries that run in single-digit seconds across a 6TB data set.
|
||||
|
||||
### Architecture
|
||||
|
||||
Druid is architected as a grouping of systems each with a distinct role and together they form a working system. The name comes from the Druid class in many role-playing games: it is a shape-shifter, capable of taking many different forms to fulfill various different roles in a group.
|
||||
|
||||
The node types that currently exist are:
|
||||
\* **Compute** nodes are the workhorses that handle storage and querying on “historical” data (non-realtime)
|
||||
\* **Realtime** nodes ingest data in real-time, they are in charge of listening to a stream of incoming data and making it available immediately inside the Druid system. As data they have ingested ages, they hand it off to the compute nodes.
|
||||
\* **Master** nodes act as coordinators. They look over the grouping of computes and make sure that data is available, replicated and in a generally “optimal” configuration.
|
||||
\* **Broker** nodes understand the topology of data across all of the other nodes in the cluster and re-write and route queries accordingly
|
||||
\* **Indexer** nodes form a cluster of workers to load batch and real-time data into the system as well as allow for alterations to the data stored in the system (also known as the Indexing Service)
|
||||
|
||||
This separation allows each node to only care about what it is best at. By separating Compute and Realtime, we separate the memory concerns of listening on a real-time stream of data and processing it for entry into the system. By separating the Master and Broker, we separate the needs for querying from the needs for maintaining “good” data distribution across the cluster.
|
||||
* **Compute** nodes are the workhorses that handle storage and querying on "historical" data (non-realtime)
|
||||
* **Realtime** nodes ingest data in real-time, they are in charge of listening to a stream of incoming data and making it available immediately inside the Druid system. As data they have ingested ages, they hand it off to the compute nodes.
|
||||
* **Master** nodes act as coordinators. They look over the grouping of computes and make sure that data is available, replicated and in a generally "optimal" configuration.
|
||||
* **Broker** nodes understand the topology of data across all of the other nodes in the cluster and re-write and route queries accordingly
|
||||
* **Indexer** nodes form a cluster of workers to load batch and real-time data into the system as well as allow for alterations to the data stored in the system (also known as the Indexing Service)
|
||||
|
||||
This separation allows each node to only care about what it is best at. By separating Compute and Realtime, we separate the memory concerns of listening on a real-time stream of data and processing it for entry into the system. By separating the Master and Broker, we separate the needs for querying from the needs for maintaining "good" data distribution across the cluster.
|
||||
|
||||
All nodes can be run in some highly available fashion. Either as symmetric peers in a share-nothing cluster or as hot-swap failover nodes.
|
||||
|
||||
|
|
@ -39,7 +41,7 @@ Aside from these nodes, there are 3 external dependencies to the system:
|
|||
|
||||
1. A running [ZooKeeper](http://zookeeper.apache.org/) cluster for cluster service discovery and maintenance of current data topology
|
||||
2. A MySQL instance for maintenance of metadata about the data segments that should be served by the system
|
||||
3. A “deep storage” LOB store/file system to hold the stored segments
|
||||
3. A "deep storage" LOB store/file system to hold the stored segments
|
||||
|
||||
### Data Storage
|
||||
|
||||
|
|
@ -53,9 +55,9 @@ Getting data into the Druid system requires an indexing process. This gives the
|
|||
- Bitmap compression
|
||||
- RLE (on the roadmap, but not yet implemented)
|
||||
|
||||
The output of the indexing process is stored in a “deep storage” LOB store/file system ([Deep Storage](Deep Storage.html) for information about potential options). Data is then loaded by compute nodes by first downloading the data to their local disk and then memory mapping it before serving queries.
|
||||
The output of the indexing process is stored in a "deep storage" LOB store/file system ([Deep Storage](Deep Storage.html) for information about potential options). Data is then loaded by compute nodes by first downloading the data to their local disk and then memory mapping it before serving queries.
|
||||
|
||||
If a compute node dies, it will no longer serve its segments, but given that the segments are still available on the “deep storage” any other node can simply download the segment and start serving it. This means that it is possible to actually remove all compute nodes from the cluster and then re-provision them without any data loss. It also means that if the “deep storage” is not available, the nodes can continue to serve the segments they have already pulled down (i.e. the cluster goes stale, not down).
|
||||
If a compute node dies, it will no longer serve its segments, but given that the segments are still available on the "deep storage" any other node can simply download the segment and start serving it. This means that it is possible to actually remove all compute nodes from the cluster and then re-provision them without any data loss. It also means that if the "deep storage" is not available, the nodes can continue to serve the segments they have already pulled down (i.e. the cluster goes stale, not down).
|
||||
|
||||
In order for a segment to exist inside of the cluster, an entry has to be added to a table in a MySQL instance. This entry is a self-describing bit of metadata about the segment, it includes things like the schema of the segment, the size, and the location on deep storage. These entries are what the Master uses to know what data **should** be available on the cluster.
|
||||
|
||||
|
|
@ -65,7 +67,7 @@ In order for a segment to exist inside of the cluster, an entry has to be added
|
|||
- **Master** Can be run in a hot fail-over configuration. If no masters are running, then changes to the data topology will stop happening (no new data and no data balancing decisions), but the system will continue to run.
|
||||
- **Broker** Can be run in parallel or in hot fail-over.
|
||||
- **Realtime** Depending on the semantics of the delivery stream, multiple of these can be run in parallel processing the exact same stream. They periodically checkpoint to disk and eventually push out to the Computes. Steps are taken to be able to recover from process death, but loss of access to the local disk can result in data loss if this is the only method of adding data to the system.
|
||||
- **“deep storage” file system** If this is not available, new data will not be able to enter the cluster, but the cluster will continue operating as is.
|
||||
- **"deep storage" file system** If this is not available, new data will not be able to enter the cluster, but the cluster will continue operating as is.
|
||||
- **MySQL** If this is not available, the master will be unable to find out about new segments in the system, but it will continue with its current view of the segments that should exist in the cluster.
|
||||
- **ZooKeeper** If this is not available, data topology changes will not be able to be made, but the Brokers will maintain their most recent view of the data topology and continue serving requests accordingly.
|
||||
|
||||
|
|
@ -77,12 +79,8 @@ For filters at a more granular level than what the Broker can prune based on, th
|
|||
|
||||
Once it knows the rows that match the current query, it can access the columns it cares about for those rows directly without having to load data that it is just going to throw away.
|
||||
|
||||
The following diagram shows the data flow for queries without showing batch indexing:
|
||||
|
||||
|
||||
|
||||
### In-memory?
|
||||
|
||||
Druid is not always and only in-memory. When we first built it, it is true that it was all in-memory all the time, but as time went on the price-performance tradeoff ended up swinging towards keeping all of our customers data in memory all the time a non-starter. We then added the ability to memory map data and allow the OS to handle paging data in and out of memory on demand. Our production cluster is primarily configured to operate with this memory mapping behavior and we are definitely over-subscribed in terms of memory available vs. data a node is serving.
|
||||
|
||||
As you read some of the old blog posts or other literature about the project, you will see “in-memory” often touted as that is the history of where Druid came from, but the technical reality is that there is a spectrum of price vs. performance and being able to slide along it from all in-memory (high cost, great performance) to mostly on disk (low cost, low performance) is the important knob to be able to adjust.
|
||||
As you read some of the old blog posts or other literature about the project, you will see "in-memory" often touted as that is the history of where Druid came from, but the technical reality is that there is a spectrum of price vs. performance and being able to slide along it from all in-memory (high cost, great performance) to mostly on disk (low cost, low performance) is the important knob to be able to adjust.
|
||||
|
|
|
|||
|
|
@ -68,7 +68,7 @@ This guide walks you through the steps to create the cluster and then how to cre
|
|||
|
||||
**`http://IPAddressDruidMaster:8082/druid/v3/demoServlet`**
|
||||
|
||||
As you can see from the image below, there are doc_page values in the Dimensions and Granularity fields. Clicking **Execute** will produce a basic query result.
|
||||
As you can see from the image below, there are default values in the Dimensions and Granularity fields. Clicking **Execute** will produce a basic query result.
|
||||
|
||||
|
||||
1. Note: when the Query is in running the **Execute** button will be disabled and read: **Fetching…**
|
||||
|
|
|
|||
|
|
@ -3,4 +3,4 @@ layout: doc_page
|
|||
---
|
||||
Druid is a complementary addition to Hadoop. Hadoop is great at storing and making accessible large amounts of individually low-value data. Unfortunately, Hadoop is not great at providing query speed guarantees on top of that data, nor does it have very good operational characteristics for a customer-facing production system. Druid, on the other hand, excels at taking high-value summaries of the low-value data on Hadoop, making it available in a fast and always-on fashion, such that it could be exposed directly to a customer.
|
||||
|
||||
Druid also requires some infrastructure to exist for “deep storage”. HDFS is one of the implemented options for this “deep storage”.
|
||||
Druid also requires some infrastructure to exist for [deep storage](Deep-Storage.html). HDFS is one of the implemented options for this [deep storage](Deep-Storage.html).
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ It’s write semantics aren’t as fluid and does not support joins. ParAccel is
|
|||
|
||||
###Data distribution model
|
||||
|
||||
Druid’s data distribution, is segment based which exists on highly available “deep” storage, like S3 or HDFS. Scaling up (or down) does not require massive copy actions or downtime; in fact, losing any number of compute nodes does not result in data loss because new compute nodes can always be brought up by reading data from “deep” storage.
|
||||
Druid’s data distribution, is segment based which exists on highly available "deep" storage, like S3 or HDFS. Scaling up (or down) does not require massive copy actions or downtime; in fact, losing any number of compute nodes does not result in data loss because new compute nodes can always be brought up by reading data from "deep" storage.
|
||||
|
||||
To contrast, ParAccel’s data distribution model is hash-based. Expanding the cluster requires re-hashing the data across the nodes, making it difficult to perform without taking downtime. Amazon’s Redshift works around this issue with a multi-step process:
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,6 @@ How does Druid compare to Vertica?
|
|||
|
||||
Vertica is similar to ParAccel/Redshift ([Druid-vs-Redshift](Druid-vs-Redshift.html)) described above in that it wasn’t built for real-time streaming data ingestion and it supports full SQL.
|
||||
|
||||
The other big difference is that instead of employing indexing, Vertica tries to optimize processing by leveraging run-length encoding (RLE) and other compression techniques along with a “projection” system that creates materialized copies of the data in a different sort order (to maximize the effectiveness of RLE).
|
||||
The other big difference is that instead of employing indexing, Vertica tries to optimize processing by leveraging run-length encoding (RLE) and other compression techniques along with a "projection" system that creates materialized copies of the data in a different sort order (to maximize the effectiveness of RLE).
|
||||
|
||||
We are unclear about how Vertica handles data distribution and replication, so we cannot speak to if/how Druid is different.
|
||||
|
|
|
|||
|
|
@ -69,4 +69,4 @@ In another terminal window:
|
|||
```
|
||||
|
||||
|
||||
The result of the client query is in JSON format. The client makes a REST request using the program `curl` which is usually installed on Linux, Unix, and OSX by doc_page.
|
||||
The result of the client query is in JSON format. The client makes a REST request using the program `curl` which is usually installed on Linux, Unix, and OSX by default.
|
||||
|
|
|
|||
|
|
@ -9,12 +9,9 @@ The simplest filter is a selector filter. The selector filter will match a speci
|
|||
|
||||
The grammar for a SELECTOR filter is as follows:
|
||||
|
||||
<code>"filter": {
|
||||
"type": "selector",
|
||||
"dimension": <dimension_string>,
|
||||
"value": <dimension_value_string>
|
||||
}
|
||||
</code>
|
||||
``` json
|
||||
"filter": { "type": "selector", "dimension": <dimension_string>, "value": <dimension_value_string> }
|
||||
```
|
||||
|
||||
This is the equivalent of `WHERE <dimension_string> = '<dimension_value_string>'`.
|
||||
|
||||
|
|
@ -22,12 +19,9 @@ This is the equivalent of `WHERE <dimension_string> = '<dimension_value_string>'
|
|||
|
||||
The regular expression filter is similar to the selector filter, but using regular expressions. It matches the specified dimension with the given pattern. The pattern can be any standard [Java regular expression](http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html).
|
||||
|
||||
<code>"filter": {
|
||||
"type": "regex",
|
||||
"dimension": <dimension_string>,
|
||||
"pattern": <pattern_string>
|
||||
}
|
||||
</code>
|
||||
``` json
|
||||
"filter": { "type": "regex", "dimension": <dimension_string>, "pattern": <pattern_string> }
|
||||
```
|
||||
|
||||
### Logical expression filters
|
||||
|
||||
|
|
@ -35,11 +29,9 @@ The regular expression filter is similar to the selector filter, but using regul
|
|||
|
||||
The grammar for an AND filter is as follows:
|
||||
|
||||
<code>"filter": {
|
||||
"type": "and",
|
||||
"fields": [<filter>, <filter>, ...]
|
||||
}
|
||||
</code>
|
||||
``` json
|
||||
"filter": { "type": "and", "fields": [<filter>, <filter>, ...] }
|
||||
```
|
||||
|
||||
The filters in fields can be any other filter defined on this page.
|
||||
|
||||
|
|
@ -47,11 +39,9 @@ The filters in fields can be any other filter defined on this page.
|
|||
|
||||
The grammar for an OR filter is as follows:
|
||||
|
||||
<code>"filter": {
|
||||
"type": "or",
|
||||
"fields": [<filter>, <filter>, ...]
|
||||
}
|
||||
</code>
|
||||
``` json
|
||||
"filter": { "type": "or", "fields": [<filter>, <filter>, ...] }
|
||||
```
|
||||
|
||||
The filters in fields can be any other filter defined on this page.
|
||||
|
||||
|
|
@ -59,11 +49,9 @@ The filters in fields can be any other filter defined on this page.
|
|||
|
||||
The grammar for a NOT filter is as follows:
|
||||
|
||||
<code>"filter": {
|
||||
"type": "not",
|
||||
"field": <filter>
|
||||
}
|
||||
</code>
|
||||
```json
|
||||
"filter": { "type": "not", "field": <filter> }
|
||||
```
|
||||
|
||||
The filter specified at field can be any other filter defined on this page.
|
||||
|
||||
|
|
@ -73,19 +61,21 @@ The JavaScript filter matches a dimension against the specified JavaScript funct
|
|||
|
||||
The function takes a single argument, the dimension value, and returns either true or false.
|
||||
|
||||
<code>{
|
||||
```json
|
||||
{
|
||||
"type" : "javascript",
|
||||
"dimension" : <dimension_string>,
|
||||
"function" : "function(value) { <...> }"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
**Example**
|
||||
The following matches any dimension values for the dimension `name` between `'bar'` and `'foo'`
|
||||
|
||||
<code>{
|
||||
```json
|
||||
{
|
||||
"type" : "javascript",
|
||||
"dimension" : "name",
|
||||
"function" : "function(x) { return(x >= 'bar' && x <= 'foo') }"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
|
@ -11,7 +11,7 @@ We describe the configuration of the Kafka firehose from the example below, but
|
|||
|
||||
- `consumerProps` is a map of properties for the Kafka consumer. The JSON object is converted into a Properties object and passed along to the Kafka consumer.
|
||||
- `feed` is the feed that the Kafka consumer should read from.
|
||||
- `parser` represents a parser that knows how to convert from String representations into the required `InputRow` representation that Druid uses. This is a potentially reusable piece that can be found in many of the firehoses that are based on text streams. The spec in the example describes a JSON feed (new-line delimited objects), with a timestamp column called “timestamp” in ISO8601 format and that it should not include the dimension “value” when processing. More information about the options available for the parser are available [here](https://github.com/metamx/druid/wiki/Firehose#parsing-data).
|
||||
- `parser` represents a parser that knows how to convert from String representations into the required `InputRow` representation that Druid uses. This is a potentially reusable piece that can be found in many of the firehoses that are based on text streams. The spec in the example describes a JSON feed (new-line delimited objects), with a timestamp column called "timestamp" in ISO8601 format and that it should not include the dimension "value" when processing. More information about the options available for the parser are available [here](https://github.com/metamx/druid/wiki/Firehose#parsing-data).
|
||||
|
||||
Available Firehoses
|
||||
-------------------
|
||||
|
|
|
|||
|
|
@ -10,20 +10,25 @@ It can be specified either as a string for simple granularities or as an object
|
|||
Simple granularities are specified as a string and bucket timestamps by their UTC time (i.e. days start at 00:00 UTC).
|
||||
|
||||
Supported granularity strings are: `all`, `none`, `minute`, `fifteen_minute`, `thirty_minute`, `hour` and `day`
|
||||
\* **`all`** buckets everything into a single bucket
|
||||
\* **`none`** does not bucket data (it actually uses the granularity of the index - minimum here is `none` which means millisecond granularity). Using `none` in a [timeseries query|TimeSeriesQuery](timeseries query|TimeSeriesQuery.html) is currently not recommended (the system will try to generate 0 values for all milliseconds that didn’t exist, which is often a lot).
|
||||
|
||||
* `all` buckets everything into a single bucket
|
||||
* `none` does not bucket data (it actually uses the granularity of the index - minimum here is `none` which means millisecond granularity). Using `none` in a [TimeSeriesQuery](TimeSeriesQuery.html) is currently not recommended (the system will try to generate 0 values for all milliseconds that didn’t exist, which is often a lot).
|
||||
|
||||
### Duration Granularities
|
||||
|
||||
Duration granularities are specified as an exact duration in milliseconds and timestamps are returned as UTC.
|
||||
|
||||
They also support specifying an optional origin, which defines where to start counting time buckets from (doc_pages to 1970-01-01T00:00:00Z).
|
||||
They also support specifying an optional origin, which defines where to start counting time buckets from (defaults to 1970-01-01T00:00:00Z).
|
||||
|
||||
<code>{"type": "duration", "duration": "7200000"}</code>
|
||||
```
|
||||
{"type": "duration", "duration": "7200000"}
|
||||
```
|
||||
|
||||
This chunks up every 2 hours.
|
||||
|
||||
<code>{"type": "duration", "duration": "3600000", "origin": "2012-01-01T00:30:00Z"}</code>
|
||||
```
|
||||
{"type": "duration", "duration": "3600000", "origin": "2012-01-01T00:30:00Z"}
|
||||
```
|
||||
|
||||
This chunks up every hour on the half-hour.
|
||||
|
||||
|
|
@ -33,16 +38,20 @@ Period granularities are specified as arbitrary period combinations of years, mo
|
|||
|
||||
They support specifying a time zone which determines where period boundaries start and also determines the timezone of the returned timestamps.
|
||||
|
||||
By doc_page years start on the first of January, months start on the first of the month and weeks start on Mondays unless an origin is specified.
|
||||
By default years start on the first of January, months start on the first of the month and weeks start on Mondays unless an origin is specified.
|
||||
|
||||
Time zone is optional (doc_pages to UTC)
|
||||
Origin is optional (doc_pages to 1970-01-01T00:00:00 in the given time zone)
|
||||
Time zone is optional (defaults to UTC)
|
||||
Origin is optional (defaults to 1970-01-01T00:00:00 in the given time zone)
|
||||
|
||||
<code>{"type": "period", "period": "P2D", "timeZone": "America/Los_Angeles"}</code>
|
||||
```
|
||||
{"type": "period", "period": "P2D", "timeZone": "America/Los_Angeles"}
|
||||
```
|
||||
|
||||
This will bucket by two day chunks in the Pacific timezone.
|
||||
|
||||
<code>{"type": "period", "period": "P3M", "timeZone": "America/Los_Angeles", "origin": "2012-02-01T00:00:00-08:00"}</code>
|
||||
```
|
||||
{"type": "period", "period": "P3M", "timeZone": "America/Los_Angeles", "origin": "2012-02-01T00:00:00-08:00"}
|
||||
```
|
||||
|
||||
This will bucket by 3 month chunks in the Pacific timezone where the three-month quarters are defined as starting from February.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,92 +5,49 @@ These types of queries take a groupBy query object and return an array of JSON o
|
|||
|
||||
An example groupBy query object is shown below:
|
||||
|
||||
<pre>
|
||||
<code>
|
||||
``` json
|
||||
{
|
||||
[queryType]() “groupBy”,
|
||||
[dataSource]() “sample\_datasource”,
|
||||
[granularity]() “day”,
|
||||
[dimensions]() [“dim1”, “dim2”],
|
||||
[limitSpec]() {
|
||||
[type]() “doc_page”,
|
||||
[limit]() 5000,
|
||||
[columns]() [“dim1”, “metric1”]
|
||||
},
|
||||
[filter]() {
|
||||
[type]() “and”,
|
||||
[fields]() [
|
||||
{
|
||||
[type]() “selector”,
|
||||
[dimension]() “sample\_dimension1”,
|
||||
[value]() “sample\_value1”
|
||||
},
|
||||
{
|
||||
[type]() “or”,
|
||||
[fields]() [
|
||||
{
|
||||
[type]() “selector”,
|
||||
[dimension]() “sample\_dimension2”,
|
||||
[value]() “sample\_value2”
|
||||
},
|
||||
{
|
||||
[type]() “selector”,
|
||||
[dimension]() “sample\_dimension3”,
|
||||
[value]() “sample\_value3”
|
||||
}
|
||||
"queryType": "groupBy",
|
||||
"dataSource": "sample_datasource",
|
||||
"granularity": "day",
|
||||
"dimensions": ["dim1", "dim2"],
|
||||
"limitSpec": { "type": "default", "limit": 5000, "columns": ["dim1", "metric1"] },
|
||||
"filter": {
|
||||
"type": "and",
|
||||
"fields": [
|
||||
{ "type": "selector", "dimension": "sample_dimension1", "value": "sample_value1" },
|
||||
{ "type": "or",
|
||||
"fields": [
|
||||
{ "type": "selector", "dimension": "sample_dimension2", "value": "sample_value2" },
|
||||
{ "type": "selector", "dimension": "sample_dimension3", "value": "sample_value3" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
[aggregations]() [
|
||||
{
|
||||
[type]() “longSum”,
|
||||
[name]() “sample\_name1”,
|
||||
[fieldName]() “sample\_fieldName1”
|
||||
},
|
||||
{
|
||||
[type]() “doubleSum”,
|
||||
[name]() “sample\_name2”,
|
||||
[fieldName]() “sample\_fieldName2”
|
||||
}
|
||||
"aggregations": [
|
||||
{ "type": "longSum", "name": "sample_name1", "fieldName": "sample_fieldName1" },
|
||||
{ "type": "doubleSum", "name": "sample_name2", "fieldName": "sample_fieldName2" }
|
||||
],
|
||||
[postAggregations]() [
|
||||
{
|
||||
[type]() “arithmetic”,
|
||||
[name]() “sample\_divide”,
|
||||
[fn]() “/”,
|
||||
[fields]() [
|
||||
{
|
||||
[type]() “fieldAccess”,
|
||||
[name]() “sample\_name1”,
|
||||
[fieldName]() “sample\_fieldName1”
|
||||
},
|
||||
{
|
||||
[type]() “fieldAccess”,
|
||||
[name]() “sample\_name2”,
|
||||
[fieldName]() “sample\_fieldName2”
|
||||
}
|
||||
"postAggregations": [
|
||||
{ "type": "arithmetic",
|
||||
"name": "sample_divide",
|
||||
"fn": "/",
|
||||
"fields": [
|
||||
{ "type": "fieldAccess", "name": "sample_name1", "fieldName": "sample_fieldName1" },
|
||||
{ "type": "fieldAccess", "name": "sample_name2", "fieldName": "sample_fieldName2" }
|
||||
]
|
||||
}
|
||||
],
|
||||
[intervals]() [
|
||||
“2012-01-01T00:00:00.000/2012-01-03T00:00:00.000”
|
||||
],
|
||||
[having]() {
|
||||
[type]() “greaterThan”,
|
||||
[aggregation]() “sample\_name1”,
|
||||
[value]() 0
|
||||
}
|
||||
"intervals": [ "2012-01-01T00:00:00.000/2012-01-03T00:00:00.000" ],
|
||||
"having": { "type": "greaterThan", "aggregation": "sample_name1", "value": 0 }
|
||||
}
|
||||
|
||||
</pre>
|
||||
</code>
|
||||
```
|
||||
|
||||
There are 9 main parts to a groupBy query:
|
||||
|
||||
|property|description|required?|
|
||||
|--------|-----------|---------|
|
||||
|queryType|This String should always be “groupBy”; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|queryType|This String should always be "groupBy"; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|dataSource|A String defining the data source to query, very similar to a table in a relational database|yes|
|
||||
|dimensions|A JSON list of dimensions to do the groupBy over|yes|
|
||||
|orderBy|See [OrderBy](OrderBy.html).|no|
|
||||
|
|
@ -102,33 +59,32 @@ There are 9 main parts to a groupBy query:
|
|||
|intervals|A JSON Object representing ISO-8601 Intervals. This defines the time ranges to run the query over.|yes|
|
||||
|context|An additional JSON Object which can be used to specify certain flags.|no|
|
||||
|
||||
To pull it all together, the above query would return *n\*m* data points, up to a maximum of 5000 points, where n is the cardinality of the “dim1” dimension, m is the cardinality of the “dim2” dimension, each day between 2012-01-01 and 2012-01-03, from the “sample\_datasource” table. Each data point contains the (long) sum of sample\_fieldName1 if the value of the data point is greater than 0, the (double) sum of sample\_fieldName2 and the (double) the result of sample\_fieldName1 divided by sample\_fieldName2 for the filter set for a particular grouping of “dim1” and “dim2”. The output looks like this:
|
||||
To pull it all together, the above query would return *n\*m* data points, up to a maximum of 5000 points, where n is the cardinality of the "dim1" dimension, m is the cardinality of the "dim2" dimension, each day between 2012-01-01 and 2012-01-03, from the "sample_datasource" table. Each data point contains the (long) sum of sample_fieldName1 if the value of the data point is greater than 0, the (double) sum of sample_fieldName2 and the (double) the result of sample_fieldName1 divided by sample_fieldName2 for the filter set for a particular grouping of "dim1" and "dim2". The output looks like this:
|
||||
|
||||
<pre>
|
||||
<code>
|
||||
[ {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-01-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“dim1” : <some_dim1_value>,
|
||||
“dim2” : <some_dim2_value>,
|
||||
“sample\_name1” : <some_sample_name1_value>,
|
||||
“sample\_name2” :<some_sample_name2_value>,
|
||||
“sample\_divide” : <some_sample_divide_value>
|
||||
```json
|
||||
[
|
||||
{
|
||||
"version" : "v1",
|
||||
"timestamp" : "2012-01-01T00:00:00.000Z",
|
||||
"event" : {
|
||||
"dim1" : <some_dim_value_one>,
|
||||
"dim2" : <some_dim_value_two>,
|
||||
"sample_name1" : <some_sample_name_value_one>,
|
||||
"sample_name2" :<some_sample_name_value_two>,
|
||||
"sample_divide" : <some_sample_divide_value>
|
||||
}
|
||||
}, {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-01-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“dim1” : <some_other_dim1_value>,
|
||||
“dim2” : <some_other_dim2_value>,
|
||||
“sample\_name1” : <some_other_sample_name1_value>,
|
||||
“sample\_name2” :<some_other_sample_name2_value>,
|
||||
“sample\_divide” : <some_other_sample_divide_value>
|
||||
},
|
||||
{
|
||||
"version" : "v1",
|
||||
"timestamp" : "2012-01-01T00:00:00.000Z",
|
||||
"event" : {
|
||||
"dim1" : <some_other_dim_value_one>,
|
||||
"dim2" : <some_other_dim_value_two>,
|
||||
"sample_name1" : <some_other_sample_name_value_one>,
|
||||
"sample_name2" :<some_other_sample_name_value_two>,
|
||||
"sample_divide" : <some_other_sample_divide_value>
|
||||
}
|
||||
},
|
||||
…
|
||||
},
|
||||
...
|
||||
]
|
||||
|
||||
</pre>
|
||||
</code>
|
||||
```
|
||||
|
|
@ -17,12 +17,13 @@ Numeric filters can be used as the base filters for more complex boolean express
|
|||
The equalTo filter will match rows with a specific aggregate value.
|
||||
The grammar for an `equalTo` filter is as follows:
|
||||
|
||||
<code>"having": {
|
||||
```json
|
||||
{
|
||||
"type": "equalTo",
|
||||
"aggregation": <aggregate_metric>,
|
||||
"value": <numeric_value>
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
This is the equivalent of `HAVING <aggregate> = <value>`.
|
||||
|
||||
|
|
@ -31,12 +32,13 @@ This is the equivalent of `HAVING <aggregate> = <value>`.
|
|||
The greaterThan filter will match rows with aggregate values greater than the given value.
|
||||
The grammar for a `greaterThan` filter is as follows:
|
||||
|
||||
<code>"having": {
|
||||
```json
|
||||
{
|
||||
"type": "greaterThan",
|
||||
"aggregation": <aggregate_metric>,
|
||||
"value": <numeric_value>
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
This is the equivalent of `HAVING <aggregate> > <value>`.
|
||||
|
||||
|
|
@ -45,12 +47,13 @@ This is the equivalent of `HAVING <aggregate> > <value>`.
|
|||
The lessThan filter will match rows with aggregate values less than the specified value.
|
||||
The grammar for a `greaterThan` filter is as follows:
|
||||
|
||||
<code>"having": {
|
||||
```json
|
||||
{
|
||||
"type": "lessThan",
|
||||
"aggregation": <aggregate_metric>,
|
||||
"value": <numeric_value>
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
This is the equivalent of `HAVING <aggregate> < <value>`.
|
||||
|
||||
|
|
@ -60,11 +63,12 @@ This is the equivalent of `HAVING <aggregate> < <value>`.
|
|||
|
||||
The grammar for an AND filter is as follows:
|
||||
|
||||
<code>"having": {
|
||||
```json
|
||||
{
|
||||
"type": "and",
|
||||
"havingSpecs": [<having clause>, <having clause>, ...]
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
The having clauses in `havingSpecs` can be any other having clause defined on this page.
|
||||
|
||||
|
|
@ -72,11 +76,12 @@ The having clauses in `havingSpecs` can be any other having clause defined on th
|
|||
|
||||
The grammar for an OR filter is as follows:
|
||||
|
||||
<code>"having": {
|
||||
```json
|
||||
{
|
||||
"type": "or",
|
||||
"havingSpecs": [<having clause>, <having clause>, ...]
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
The having clauses in `havingSpecs` can be any other having clause defined on this page.
|
||||
|
||||
|
|
@ -84,10 +89,11 @@ The having clauses in `havingSpecs` can be any other having clause defined on th
|
|||
|
||||
The grammar for a NOT filter is as follows:
|
||||
|
||||
<code>"having": {
|
||||
```json
|
||||
{
|
||||
"type": "not",
|
||||
"havingSpec": <having clause>
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
The having clause specified at `havingSpec` can be any other having clause defined on this page.
|
||||
|
|
|
|||
|
|
@ -21,27 +21,37 @@ The indexer coordinator node exposes HTTP endpoints where tasks can be submitted
|
|||
|
||||
Tasks can be submitted via POST requests to:
|
||||
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task
|
||||
```
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task
|
||||
```
|
||||
|
||||
Tasks can cancelled via POST requests to:
|
||||
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task/{taskId}/shutdown
|
||||
```
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task/{taskId}/shutdown
|
||||
```
|
||||
|
||||
Issuing the cancel request once sends a graceful shutdown request. Graceful shutdowns may not stop a task right away, but instead issue a safe stop command at a point deemed least impactful to the system. Issuing the cancel request twice in succession will kill –9 the task.
|
||||
|
||||
Task statuses can be retrieved via GET requests to:
|
||||
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task/{taskId}/status
|
||||
```
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task/{taskId}/status
|
||||
```
|
||||
|
||||
Task segments can be retrieved via GET requests to:
|
||||
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task/{taskId}/segments
|
||||
```
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/task/{taskId}/segments
|
||||
```
|
||||
|
||||
When a task is submitted, the coordinator creates a lock over the data source and interval of the task. The coordinator also stores the task in a MySQL database table. The database table is read at startup time to bootstrap any tasks that may have been submitted to the coordinator but may not yet have been executed.
|
||||
|
||||
The coordinator also exposes a simple UI to show what tasks are currently running on what nodes at
|
||||
|
||||
http://<COORDINATOR_IP>:<port>/static/console.html
|
||||
```
|
||||
http://<COORDINATOR_IP>:<port>/static/console.html
|
||||
```
|
||||
|
||||
#### Task Execution
|
||||
|
||||
|
|
@ -55,11 +65,14 @@ The Autoscaling mechanisms currently in place are tightly coupled with our deplo
|
|||
|
||||
The Coordinator node controls the number of workers in the cluster according to a worker setup spec that is submitted via a POST request to the indexer at:
|
||||
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/worker/setup
|
||||
```
|
||||
http://<COORDINATOR_IP>:<port>/druid/indexer/v1/worker/setup
|
||||
```
|
||||
|
||||
A sample worker setup spec is shown below:
|
||||
|
||||
<code>{
|
||||
```
|
||||
{
|
||||
"minVersion":"some_version",
|
||||
"minNumWorkers":"0",
|
||||
"maxNumWorkers":"10",
|
||||
|
|
@ -78,8 +91,8 @@ A sample worker setup spec is shown below:
|
|||
"version":"druid_version",
|
||||
"type":"sample_cluster/worker"
|
||||
}
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
Issuing a GET request at the same URL will return the current worker setup spec that is currently in place. The worker setup spec list above is just a sample and it is possible to write worker setup specs for other deployment environments. A description of the worker setup spec is shown below.
|
||||
|
||||
|
|
@ -101,19 +114,21 @@ Indexer Coordinator nodes can be run using the `com.metamx.druid.indexing.coordi
|
|||
|
||||
Indexer Coordinator nodes require [basic service configuration](https://github.com/metamx/druid/wiki/Configuration#basic-service-configuration). In addition, there are several extra configurations that are required.
|
||||
|
||||
-Ddruid.zk.paths.indexer.announcementsPath=/druid/indexer/announcements
|
||||
-Ddruid.zk.paths.indexer.leaderLatchPath=/druid/indexer/leaderLatchPath
|
||||
-Ddruid.zk.paths.indexer.statusPath=/druid/indexer/status
|
||||
-Ddruid.zk.paths.indexer.tasksPath=/druid/demo/indexer/tasks
|
||||
```
|
||||
-Ddruid.zk.paths.indexer.announcementsPath=/druid/indexer/announcements
|
||||
-Ddruid.zk.paths.indexer.leaderLatchPath=/druid/indexer/leaderLatchPath
|
||||
-Ddruid.zk.paths.indexer.statusPath=/druid/indexer/status
|
||||
-Ddruid.zk.paths.indexer.tasksPath=/druid/demo/indexer/tasks
|
||||
|
||||
-Ddruid.indexer.runner=remote
|
||||
-Ddruid.indexer.taskDir=/mnt/persistent/task/
|
||||
-Ddruid.indexer.configTable=sample_config
|
||||
-Ddruid.indexer.workerSetupConfigName=worker_setup
|
||||
-Ddruid.indexer.strategy=ec2
|
||||
-Ddruid.indexer.hadoopWorkingPath=/tmp/druid-indexing
|
||||
-Ddruid.indexer.logs.s3bucket=some_bucket
|
||||
-Ddruid.indexer.logs.s3prefix=some_prefix
|
||||
-Ddruid.indexer.runner=remote
|
||||
-Ddruid.indexer.taskDir=/mnt/persistent/task/
|
||||
-Ddruid.indexer.configTable=sample_config
|
||||
-Ddruid.indexer.workerSetupConfigName=worker_setup
|
||||
-Ddruid.indexer.strategy=ec2
|
||||
-Ddruid.indexer.hadoopWorkingPath=/tmp/druid-indexing
|
||||
-Ddruid.indexer.logs.s3bucket=some_bucket
|
||||
-Ddruid.indexer.logs.s3prefix=some_prefix
|
||||
```
|
||||
|
||||
The indexing service requires some additional Zookeeper configs.
|
||||
|
||||
|
|
@ -128,7 +143,7 @@ There’s several additional configs that are required to run tasks.
|
|||
|
||||
|Property|Description|Default|
|
||||
|--------|-----------|-------|
|
||||
|`druid.indexer.runner`|Indicates whether tasks should be run locally or in a distributed environment. “local” or “remote”.|local|
|
||||
|`druid.indexer.runner`|Indicates whether tasks should be run locally or in a distributed environment. "local" or "remote".|local|
|
||||
|`druid.indexer.taskDir`|Intermediate temporary directory that tasks may use.|none|
|
||||
|`druid.indexer.configTable`|The MySQL config table where misc configs live.|none|
|
||||
|`druid.indexer.strategy`|The autoscaling strategy to use.|noop|
|
||||
|
|
@ -140,7 +155,9 @@ There’s several additional configs that are required to run tasks.
|
|||
|
||||
The indexer console can be used to view pending tasks, running tasks, available workers, and recent worker creation and termination. The console can be accessed at:
|
||||
|
||||
http://<COORDINATOR_IP>:8080/static/console.html
|
||||
```
|
||||
http://<COORDINATOR_IP>:8080/static/console.html
|
||||
```
|
||||
|
||||
Worker Node
|
||||
-----------
|
||||
|
|
@ -155,29 +172,31 @@ Worker nodes can be run using the `com.metamx.druid.indexing.worker.http.WorkerM
|
|||
|
||||
Worker nodes require [basic service configuration](https://github.com/metamx/druid/wiki/Configuration#basic-service-configuration). In addition, there are several extra configurations that are required.
|
||||
|
||||
-Ddruid.worker.version=0
|
||||
-Ddruid.worker.capacity=3
|
||||
```
|
||||
-Ddruid.worker.version=0
|
||||
-Ddruid.worker.capacity=3
|
||||
|
||||
-Ddruid.indexer.threads=3
|
||||
-Ddruid.indexer.taskDir=/mnt/persistent/task/
|
||||
-Ddruid.indexer.hadoopWorkingPath=/tmp/druid-indexing
|
||||
-Ddruid.indexer.threads=3
|
||||
-Ddruid.indexer.taskDir=/mnt/persistent/task/
|
||||
-Ddruid.indexer.hadoopWorkingPath=/tmp/druid-indexing
|
||||
|
||||
-Ddruid.worker.masterService=druid:sample_cluster:indexer
|
||||
-Ddruid.worker.masterService=druid:sample_cluster:indexer
|
||||
|
||||
-Ddruid.indexer.fork.hostpattern=<IP>:%d
|
||||
-Ddruid.indexer.fork.startport=8080
|
||||
-Ddruid.indexer.fork.main=com.metamx.druid.indexing.worker.executor.ExecutorMain
|
||||
-Ddruid.indexer.fork.opts="-server -Xmx1g -Xms1g -XX:NewSize=256m -XX:MaxNewSize=256m"
|
||||
-Ddruid.indexer.fork.property.druid.service=druid/sample_cluster/executor
|
||||
-Ddruid.indexer.fork.hostpattern=<IP>:%d
|
||||
-Ddruid.indexer.fork.startport=8080
|
||||
-Ddruid.indexer.fork.main=com.metamx.druid.indexing.worker.executor.ExecutorMain
|
||||
-Ddruid.indexer.fork.opts="-server -Xmx1g -Xms1g -XX:NewSize=256m -XX:MaxNewSize=256m"
|
||||
-Ddruid.indexer.fork.property.druid.service=druid/sample_cluster/executor
|
||||
|
||||
# These configs are the same configs you would set for basic service configuration, just with a different prefix
|
||||
-Ddruid.indexer.fork.property.druid.monitoring.monitorSystem=false
|
||||
-Ddruid.indexer.fork.property.druid.computation.buffer.size=268435456
|
||||
-Ddruid.indexer.fork.property.druid.indexer.taskDir=/mnt/persistent/task/
|
||||
-Ddruid.indexer.fork.property.druid.processing.formatString=processing-%s
|
||||
-Ddruid.indexer.fork.property.druid.processing.numThreads=1
|
||||
-Ddruid.indexer.fork.property.druid.server.maxSize=0
|
||||
-Ddruid.indexer.fork.property.druid.request.logging.dir=request_logs/
|
||||
# These configs are the same configs you would set for basic service configuration, just with a different prefix
|
||||
-Ddruid.indexer.fork.property.druid.monitoring.monitorSystem=false
|
||||
-Ddruid.indexer.fork.property.druid.computation.buffer.size=268435456
|
||||
-Ddruid.indexer.fork.property.druid.indexer.taskDir=/mnt/persistent/task/
|
||||
-Ddruid.indexer.fork.property.druid.processing.formatString=processing-%s
|
||||
-Ddruid.indexer.fork.property.druid.processing.numThreads=1
|
||||
-Ddruid.indexer.fork.property.druid.server.maxSize=0
|
||||
-Ddruid.indexer.fork.property.druid.request.logging.dir=request_logs/
|
||||
```
|
||||
|
||||
Many of the configurations for workers are similar to those for basic service configuration":https://github.com/metamx/druid/wiki/Configuration\#basic-service-configuration, but with a different config prefix. Below we describe the unique worker configs.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,9 +1,6 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
### R
|
||||
|
||||
- [RDruid](https://github.com/metamx/RDruid) - Druid connector for R
|
||||
|
||||
Community Libraries
|
||||
-------------------
|
||||
|
|
@ -11,13 +8,18 @@ Community Libraries
|
|||
Some great folks have written their own libraries to interact with Druid
|
||||
|
||||
#### Ruby
|
||||
\* [madvertise/ruby-druid](https://github.com/madvertise/ruby-druid) - A ruby client for Druid
|
||||
|
||||
* [madvertise/ruby-druid](https://github.com/madvertise/ruby-druid) - A ruby client for Druid
|
||||
|
||||
#### Python
|
||||
\* [metamx/pydruid](https://github.com/metamx/pydruid) - A python client for Druid
|
||||
|
||||
* [metamx/pydruid](https://github.com/metamx/pydruid) - A python client for Druid
|
||||
|
||||
#### R
|
||||
|
||||
- [RDruid](https://github.com/metamx/RDruid) - Druid connector for R
|
||||
|
||||
#### Helper Libraries
|
||||
|
||||
- [madvertise/druid-dumbo](https://github.com/madvertise/druid-dumbo) - Scripts to help generate batch configs for the ingestion of data into Druid
|
||||
|
||||
- [housejester/druid-test-harness](https://github.com/housejester/druid-test-harness) - A set of scripts to simplify standing up some servers and seeing how things work
|
||||
* [madvertise/druid-dumbo](https://github.com/madvertise/druid-dumbo) - Scripts to help generate batch configs for the ingestion of data into Druid
|
||||
* [housejester/druid-test-harness](https://github.com/housejester/druid-test-harness) - A set of scripts to simplify standing up some servers and seeing how things work
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ Before any unassigned segments are serviced by compute nodes, the available comp
|
|||
Rules
|
||||
-----
|
||||
|
||||
Segments are loaded and dropped from the cluster based on a set of rules. Rules indicate how segments should be assigned to different compute node tiers and how many replicants of a segment should exist in each tier. Rules may also indicate when segments should be dropped entirely from the cluster. The master loads a set of rules from the database. Rules may be specific to a certain datasource and/or a doc_page set of rules can be configured. Rules are read in order and hence the ordering of rules is important. The master will cycle through all available segments and match each segment with the first rule that applies. Each segment may only match a single rule
|
||||
Segments are loaded and dropped from the cluster based on a set of rules. Rules indicate how segments should be assigned to different compute node tiers and how many replicants of a segment should exist in each tier. Rules may also indicate when segments should be dropped entirely from the cluster. The master loads a set of rules from the database. Rules may be specific to a certain datasource and/or a default set of rules can be configured. Rules are read in order and hence the ordering of rules is important. The master will cycle through all available segments and match each segment with the first rule that applies. Each segment may only match a single rule
|
||||
|
||||
For more information on rules, see [Rule Configuration](Rule-Configuration.html).
|
||||
|
||||
|
|
@ -39,61 +39,91 @@ The master node exposes several HTTP endpoints for interactions.
|
|||
|
||||
### GET
|
||||
|
||||
/info/master - returns the current true master of the cluster as a JSON object. E.g. A GET request to <IP>:8080/info/master will yield JSON of the form {[host]("IP"})
|
||||
* `/info/master`
|
||||
|
||||
/info/cluster - returns JSON data about every node and segment in the cluster. E.g. A GET request to <IP>:8080/info/cluster will yield JSON data organized by nodes. Information about each node and each segment on each node will be returned.
|
||||
Returns the current true master of the cluster as a JSON object.
|
||||
|
||||
/info/servers (optional param ?full) - returns all segments in the cluster if the full flag is not set, otherwise returns full metadata about all servers in the cluster
|
||||
* `/info/cluster`
|
||||
|
||||
/info/servers/{serverName} - returns full metadata about a specific server
|
||||
Returns JSON data about every node and segment in the cluster. Information about each node and each segment on each node will be returned.
|
||||
|
||||
/info/servers/{serverName}/segments (optional param ?full) - returns a list of all segments for a server if the full flag is not set, otherwise returns all segment metadata
|
||||
* `/info/servers`
|
||||
|
||||
/info/servers/{serverName}/segments/{segmentId} - returns full metadata for a specific segment
|
||||
Returns information about servers in the cluster. Set the `?full` query parameter to get full metadata about all servers and their segments in the cluster.
|
||||
|
||||
/info/segments (optional param ?full)- returns all segments in the cluster as a list if the full flag is not set, otherwise returns all metadata about segments in the cluster
|
||||
* `/info/servers/{serverName}`
|
||||
|
||||
/info/segments/{segmentId} - returns full metadata for a specific segment
|
||||
Returns full metadata about a specific server.
|
||||
|
||||
/info/datasources (optional param ?full) - returns a list of datasources in the cluster if the full flag is not set, otherwise returns all the metadata for every datasource in the cluster
|
||||
* `/info/servers/{serverName}/segments`
|
||||
|
||||
/info/datasources/{dataSourceName} - returns full metadata for a datasource
|
||||
Returns a list of all segments for a server. Set the `?full` query parameter to get all segment metadata included
|
||||
|
||||
/info/datasources/{dataSourceName}/segments (optional param ?full) - returns a list of all segments for a datasource if the full flag is not set, otherwise returns full segment metadata for a datasource
|
||||
* `/info/servers/{serverName}/segments/{segmentId}`
|
||||
|
||||
/info/datasources/{dataSourceName}/segments/{segmentId} - returns full segment metadata for a specific segment
|
||||
Returns full metadata for a specific segment.
|
||||
|
||||
/info/rules - returns all rules for all data sources in the cluster including the doc_page datasource.
|
||||
* `/info/segments`
|
||||
|
||||
/info/rules/{dataSourceName} - returns all rules for a specified datasource
|
||||
Returns all segments in the cluster as a list. Set the `?full` flag to get all metadata about segments in the cluster
|
||||
|
||||
* `/info/segments/{segmentId}`
|
||||
|
||||
Returns full metadata for a specific segment
|
||||
|
||||
* `/info/datasources`
|
||||
|
||||
Returns a list of datasources in the cluster. Set the `?full` flag to get all metadata for every datasource in the cluster
|
||||
|
||||
* `/info/datasources/{dataSourceName}`
|
||||
|
||||
Returns full metadata for a datasource
|
||||
|
||||
* `/info/datasources/{dataSourceName}/segments`
|
||||
|
||||
Returns a list of all segments for a datasource. Set the `?full` flag to get full segment metadata for a datasource
|
||||
|
||||
* `/info/datasources/{dataSourceName}/segments/{segmentId}`
|
||||
|
||||
Returns full segment metadata for a specific segment
|
||||
|
||||
* `/info/rules`
|
||||
|
||||
Returns all rules for all data sources in the cluster including the default datasource.
|
||||
|
||||
* `/info/rules/{dataSourceName}`
|
||||
|
||||
Returns all rules for a specified datasource
|
||||
|
||||
### POST
|
||||
|
||||
/info/rules/{dataSourceName} - POST with a list of rules in JSON form to update rules.
|
||||
* `/info/rules/{dataSourceName}`
|
||||
|
||||
POST with a list of rules in JSON form to update rules.
|
||||
|
||||
The Master Console
|
||||
------------------
|
||||
|
||||
The Druid master exposes a web GUI for displaying cluster information and rule configuration. After the master starts, the console can be accessed at http://HOST:PORT/static/. There exists a full cluster view, as well as views for individual compute nodes, datasources and segments themselves. Segment information can be displayed in raw JSON form or as part of a sortable and filterable table.
|
||||
|
||||
The master console also exposes an interface to creating and editing rules. All valid datasources configured in the segment database, along with a doc_page datasource, are available for configuration. Rules of different types can be added, deleted or edited.
|
||||
The master console also exposes an interface to creating and editing rules. All valid datasources configured in the segment database, along with a default datasource, are available for configuration. Rules of different types can be added, deleted or edited.
|
||||
|
||||
FAQ
|
||||
---
|
||||
|
||||
1. **Do clients ever contact the master node?**
|
||||
|
||||
The master is not involved in the lifecycle of a query.
|
||||
The master is not involved in a query.
|
||||
|
||||
Compute nodes never directly contact the master node. The Druid master tells the compute nodes to load/drop data via Zookeeper, but the compute nodes are completely unaware of the master.
|
||||
Compute nodes never directly contact the master node. The Druid master tells the compute nodes to load/drop data via Zookeeper, but the compute nodes are completely unaware of the master.
|
||||
|
||||
Brokers also never contact the master. Brokers base their understanding of the data topology on metadata exposed by the compute nodes via ZK and are completely unaware of the master.
|
||||
Brokers also never contact the master. Brokers base their understanding of the data topology on metadata exposed by the compute nodes via ZK and are completely unaware of the master.
|
||||
|
||||
2. **Does it matter if the master node starts up before or after other processes?**
|
||||
|
||||
No. If the Druid master is not started up, no new segments will be loaded in the cluster and outdated segments will not be dropped. However, the master node can be started up at any time, and after a configurable delay, will start running master tasks.
|
||||
No. If the Druid master is not started up, no new segments will be loaded in the cluster and outdated segments will not be dropped. However, the master node can be started up at any time, and after a configurable delay, will start running master tasks.
|
||||
|
||||
This also means that if you have a working cluster and all of your masters die, the cluster will continue to function, it just won’t experience any changes to its data topology.
|
||||
This also means that if you have a working cluster and all of your masters die, the cluster will continue to function, it just won’t experience any changes to its data topology.
|
||||
|
||||
Running
|
||||
-------
|
||||
|
|
|
|||
|
|
@ -10,11 +10,12 @@ This is dictated by the `druid.database.segmentTable` property (Note that these
|
|||
|
||||
This table stores metadata about the segments that are available in the system. The table is polled by the [Master](Master.html) to determine the set of segments that should be available for querying in the system. The table has two main functional columns, the other columns are for indexing purposes.
|
||||
|
||||
The `used` column is a boolean “tombstone”. A 1 means that the segment should be “used” by the cluster (i.e. it should be loaded and available for requests). A 0 means that the segment should not be actively loaded into the cluster. We do this as a means of removing segments from the cluster without actually removing their metadata (which allows for simpler rolling back if that is ever an issue).
|
||||
The `used` column is a boolean "tombstone". A 1 means that the segment should be "used" by the cluster (i.e. it should be loaded and available for requests). A 0 means that the segment should not be actively loaded into the cluster. We do this as a means of removing segments from the cluster without actually removing their metadata (which allows for simpler rolling back if that is ever an issue).
|
||||
|
||||
The `payload` column stores a JSON blob that has all of the metadata for the segment (some of the data stored in this payload is redundant with some of the columns in the table, that is intentional). This looks something like
|
||||
|
||||
{
|
||||
```
|
||||
{
|
||||
"dataSource":"wikipedia",
|
||||
"interval":"2012-05-23T00:00:00.000Z/2012-05-24T00:00:00.000Z",
|
||||
"version":"2012-05-24T00:10:00.046Z",
|
||||
|
|
@ -27,7 +28,8 @@ The `payload` column stores a JSON blob that has all of the metadata for the seg
|
|||
"binaryVersion":9,
|
||||
"size":size_of_segment,
|
||||
"identifier":"wikipedia_2012-05-23T00:00:00.000Z_2012-05-24T00:00:00.000Z_2012-05-23T00:10:00.046Z"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note that the format of this blob can and will change from time-to-time.
|
||||
|
||||
|
|
|
|||
|
|
@ -5,23 +5,23 @@ The orderBy field provides the functionality to sort and limit the set of result
|
|||
|
||||
### DefaultLimitSpec
|
||||
|
||||
The doc_page limit spec takes a limit and the list of columns to do an orderBy operation over. The grammar is:
|
||||
The default limit spec takes a limit and the list of columns to do an orderBy operation over. The grammar is:
|
||||
|
||||
<code>
|
||||
{
|
||||
"type" : "doc_page",
|
||||
```json
|
||||
{
|
||||
"type" : "default",
|
||||
"limit" : <integer_value>,
|
||||
"columns" : [list of OrderByColumnSpec],
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
#### OrderByColumnSpec
|
||||
|
||||
OrderByColumnSpecs indicate how to do order by operations. Each order by condition can be a <code>String</code> or a map of the following form:
|
||||
OrderByColumnSpecs indicate how to do order by operations. Each order by condition can be a `jsonString` or a map of the following form:
|
||||
|
||||
<code>
|
||||
{
|
||||
"dimension" : "<Any dimension or metric>",
|
||||
```json
|
||||
{
|
||||
"dimension" : <Any dimension or metric>,
|
||||
"direction" : "ASCENDING OR DESCENDING"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
The Plumber is the thing that handles generated segments both while they are being generated and when they are “done”. This is also technically a pluggable interface and there are multiple implementations, but there are a lot of details handled by the plumber such that it is expected that there will only be a few implementations and only more advanced third-parties will implement their own. See [here](https://github.com/metamx/druid/wiki/Plumber#available-plumbers) for a description of the plumbers included with Druid.
|
||||
The Plumber is the thing that handles generated segments both while they are being generated and when they are "done". This is also technically a pluggable interface and there are multiple implementations, but there are a lot of details handled by the plumber such that it is expected that there will only be a few implementations and only more advanced third-parties will implement their own.
|
||||
|
||||
|Field|Type|Description|Required|
|
||||
|-----|----|-----------|--------|
|
||||
|
|
@ -9,8 +9,8 @@ The Plumber is the thing that handles generated segments both while they are bei
|
|||
|
||||
We provide a brief description of the example to exemplify the types of things that are configured on the plumber.
|
||||
|
||||
- `windowPeriod` is the amount of lag time to allow events. This is configured with a 10 minute window, meaning that any event more than 10 minutes ago will be thrown away and not included in the segment generated by the realtime server.
|
||||
- `basePersistDirectory` is the directory to put things that need persistence. The plumber is responsible for the actual intermediate persists and this tells it where to store those persists.
|
||||
* `windowPeriod` is the amount of lag time to allow events. This is configured with a 10 minute window, meaning that any event more than 10 minutes ago will be thrown away and not included in the segment generated by the realtime server.
|
||||
* `basePersistDirectory` is the directory to put things that need persistence. The plumber is responsible for the actual intermediate persists and this tells it where to store those persists.
|
||||
|
||||
Available Plumbers
|
||||
------------------
|
||||
|
|
|
|||
|
|
@ -13,83 +13,63 @@ Supported functions are `+`, `-`, `*`, and `/`
|
|||
|
||||
The grammar for an arithmetic post aggregation is:
|
||||
|
||||
<code>postAggregation : {
|
||||
```json
|
||||
postAggregation : {
|
||||
"type" : "arithmetic",
|
||||
"name" : <output_name>,
|
||||
"fn" : <arithmetic_function>,
|
||||
"fields": [<post_aggregator>, <post_aggregator>, ...]
|
||||
}</code>
|
||||
}
|
||||
```
|
||||
|
||||
### Field accessor post-aggregator
|
||||
|
||||
This returns the value produced by the specified [aggregator|Aggregations](aggregator|Aggregations.html).
|
||||
This returns the value produced by the specified [aggregator](Aggregations.html).
|
||||
|
||||
`fieldName` refers to the output name of the aggregator given in the [aggregations|Aggregations](aggregations|Aggregations.html) portion of the query.
|
||||
`fieldName` refers to the output name of the aggregator given in the [aggregations](Aggregations.html) portion of the query.
|
||||
|
||||
<code>field_accessor : {
|
||||
"type" : "fieldAccess",
|
||||
"fieldName" : <aggregator_name>
|
||||
}</code>
|
||||
```json
|
||||
{ "type" : "fieldAccess", "fieldName" : <aggregator_name> }
|
||||
```
|
||||
|
||||
### Constant post-aggregator
|
||||
|
||||
The constant post-aggregator always returns the specified value.
|
||||
|
||||
<code>constant : {
|
||||
"type" : "constant",
|
||||
"name" : <output_name>,
|
||||
"value" : <numerical_value>,
|
||||
}</code>
|
||||
```json
|
||||
{ "type" : "constant", "name" : <output_name>, "value" : <numerical_value> }
|
||||
```
|
||||
|
||||
### Example Usage
|
||||
|
||||
In this example, let’s calculate a simple percentage using post aggregators. Let’s imagine our data set has a metric called “total”.
|
||||
In this example, let’s calculate a simple percentage using post aggregators. Let’s imagine our data set has a metric called "total".
|
||||
|
||||
The format of the query JSON is as follows:
|
||||
|
||||
<code>
|
||||
{
|
||||
```json
|
||||
{
|
||||
...
|
||||
"aggregations" : [
|
||||
{
|
||||
"type" : "count",
|
||||
"name" : "rows"
|
||||
},
|
||||
{
|
||||
"type" : "doubleSum",
|
||||
"name" : "tot",
|
||||
"fieldName" : "total"
|
||||
}
|
||||
{ "type" : "count", "name" : "rows" },
|
||||
{ "type" : "doubleSum", "name" : "tot", "fieldName" : "total" }
|
||||
],
|
||||
"postAggregations" : {
|
||||
"type" : "arithmetic",
|
||||
"name" : "average",
|
||||
"fn" : "*",
|
||||
"fields" : [
|
||||
{
|
||||
"type" : "arithmetic",
|
||||
{ "type" : "arithmetic",
|
||||
"name" : "div",
|
||||
"fn" : "/",
|
||||
"fields" : [
|
||||
{
|
||||
"type" : "fieldAccess",
|
||||
"name" : "tot",
|
||||
"fieldName" : "tot"
|
||||
},
|
||||
{
|
||||
"type" : "fieldAccess",
|
||||
"name" : "rows",
|
||||
"fieldName" : "rows"
|
||||
}
|
||||
{ "type" : "fieldAccess", "name" : "tot", "fieldName" : "tot" },
|
||||
{ "type" : "fieldAccess", "name" : "rows", "fieldName" : "rows" }
|
||||
]
|
||||
},
|
||||
{
|
||||
"type" : "constant",
|
||||
"name": "const",
|
||||
"value" : 100
|
||||
}
|
||||
{ "type" : "constant", "name": "const", "value" : 100 }
|
||||
]
|
||||
}
|
||||
...
|
||||
}
|
||||
</code>
|
||||
}
|
||||
|
||||
```
|
||||
|
|
|
|||
|
|
@ -8,86 +8,100 @@ Queries are made using an HTTP REST style request to a [Broker](Broker.html), [C
|
|||
|
||||
We start by describing an example query with additional comments that mention possible variations. Query operators are also summarized in a table below.
|
||||
|
||||
Example Query “rand”
|
||||
Example Query "rand"
|
||||
--------------------
|
||||
|
||||
Here is the query in the examples/rand subproject (file is query.body), followed by a commented version of the same.
|
||||
|
||||
\`\`\`javascript
|
||||
```javascript
|
||||
{
|
||||
[queryType]() “groupBy”,
|
||||
[dataSource]() “randSeq”,
|
||||
[granularity]() “all”,
|
||||
[dimensions]() [],
|
||||
[aggregations]() [
|
||||
{ [type]() “count”, [name]() “rows” },
|
||||
{ [type]() “doubleSum”, [fieldName]() “events”, [name]() “e” },
|
||||
{ [type]() “doubleSum”, [fieldName]() “outColumn”, [name]() “randomNumberSum” }
|
||||
"queryType": "groupBy",
|
||||
"dataSource": "randSeq",
|
||||
"granularity": "all",
|
||||
"dimensions": [],
|
||||
"aggregations": [
|
||||
{ "type": "count", "name": "rows" },
|
||||
{ "type": "doubleSum", "fieldName": "events", "name": "e" },
|
||||
{ "type": "doubleSum", "fieldName": "outColumn", "name": "randomNumberSum" }
|
||||
],
|
||||
[postAggregations]() [{
|
||||
[type]() “arithmetic”,
|
||||
[name]() “avg\_random”,
|
||||
[fn]() “/”,
|
||||
[fields]() [
|
||||
{ [type]() “fieldAccess”, [fieldName]() “randomNumberSum” },
|
||||
{ [type]() “fieldAccess”, [fieldName]() “rows” }
|
||||
"postAggregations": [{
|
||||
"type": "arithmetic",
|
||||
"name": "avg_random",
|
||||
"fn": "/",
|
||||
"fields": [
|
||||
{ "type": "fieldAccess", "fieldName": "randomNumberSum" },
|
||||
{ "type": "fieldAccess", "fieldName": "rows" }
|
||||
]
|
||||
}],
|
||||
[intervals]() [“2012-10-01T00:00/2020-01-01T00”]
|
||||
"intervals": ["2012-10-01T00:00/2020-01-01T00"]
|
||||
}
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
This query could be submitted via curl like so (assuming the query object is in a file “query.json”).
|
||||
This query could be submitted via curl like so (assuming the query object is in a file "query.json").
|
||||
|
||||
curl -X POST "http://host:port/druid/v2/?pretty" -H 'content-type: application/json' -d @query.json
|
||||
```
|
||||
curl -X POST "http://host:port/druid/v2/?pretty" -H 'content-type: application/json' -d @query.json
|
||||
```
|
||||
|
||||
The “pretty” query parameter gets the results formatted a bit nicer.
|
||||
The "pretty" query parameter gets the results formatted a bit nicer.
|
||||
|
||||
Details of Example Query “rand”
|
||||
Details of Example Query "rand"
|
||||
-------------------------------
|
||||
|
||||
The queryType JSON field identifies which kind of query operator is to be used, in this case it is groupBy, the most frequently used kind (which corresponds to an internal implementation class GroupByQuery registered as “groupBy”), and it has a set of required fields that are also part of this query. The queryType can also be “search” or “timeBoundary” which have similar or different required fields summarized below:
|
||||
\`\`\`javascript
|
||||
The queryType JSON field identifies which kind of query operator is to be used, in this case it is groupBy, the most frequently used kind (which corresponds to an internal implementation class GroupByQuery registered as "groupBy"), and it has a set of required fields that are also part of this query. The queryType can also be "search" or "timeBoundary" which have similar or different required fields summarized below:
|
||||
|
||||
```javascript
|
||||
{
|
||||
[queryType]() “groupBy”,
|
||||
\`\`\`
|
||||
The dataSource JSON field shown next identifies where to apply the query. In this case, randSeq corresponds to the examples/rand/rand\_realtime.spec file schema:
|
||||
\`\`\`javascript
|
||||
[dataSource]() “randSeq”,
|
||||
\`\`\`
|
||||
The granularity JSON field specifies the bucket size for values. It could be a built-in time interval like “second”, “minute”, “fifteen\_minute”, “thirty\_minute”, “hour” or “day”. It can also be an expression like `{"type": "period", "period":"PT6m"}` meaning “6 minute buckets”. See [Granularities](Granularities.html) for more information on the different options for this field. In this example, it is set to the special value “all” which means [bucket all data points together into the same time bucket]()
|
||||
\`\`\`javascript
|
||||
[granularity]() “all”,
|
||||
\`\`\`
|
||||
"queryType": "groupBy",
|
||||
```
|
||||
|
||||
The dataSource JSON field shown next identifies where to apply the query. In this case, randSeq corresponds to the examples/rand/rand_realtime.spec file schema:
|
||||
|
||||
```javascript
|
||||
"dataSource": "randSeq",
|
||||
```
|
||||
|
||||
The granularity JSON field specifies the bucket size for values. It could be a built-in time interval like "second", "minute", "fifteen_minute", "thirty_minute", "hour" or "day". It can also be an expression like `{"type": "period", "period":"PT6m"}` meaning "6 minute buckets". See [Granularities](Granularities.html) for more information on the different options for this field. In this example, it is set to the special value "all" which means [bucket all data points together into the same time bucket]()
|
||||
|
||||
```javascript
|
||||
"granularity": "all",
|
||||
```
|
||||
|
||||
The dimensions JSON field value is an array of zero or more fields as defined in the dataSource spec file or defined in the input records and carried forward. These are used to constrain the grouping. If empty, then one value per time granularity bucket is requested in the groupBy:
|
||||
\`\`\`javascript
|
||||
[dimensions]() [],
|
||||
\`\`\`
|
||||
A groupBy also requires the JSON field “aggregations” (See [Aggregations](Aggregations.html)), which are applied to the column specified by fieldName and the output of the aggregation will be named according to the value in the “name” field:
|
||||
\`\`\`javascript
|
||||
[aggregations]() [
|
||||
{ [type]() “count”, [name]() “rows” },
|
||||
{ [type]() “doubleSum”, [fieldName]() “events”, [name]() “e” },
|
||||
{ [type]() “doubleSum”, [fieldName]() “outColumn”, [name]() “randomNumberSum” }
|
||||
|
||||
```javascript
|
||||
"dimensions": [],
|
||||
```
|
||||
|
||||
A groupBy also requires the JSON field "aggregations" (See [Aggregations](Aggregations.html)), which are applied to the column specified by fieldName and the output of the aggregation will be named according to the value in the "name" field:
|
||||
|
||||
```javascript
|
||||
"aggregations": [
|
||||
{ "type": "count", "name": "rows" },
|
||||
{ "type": "doubleSum", "fieldName": "events", "name": "e" },
|
||||
{ "type": "doubleSum", "fieldName": "outColumn", "name": "randomNumberSum" }
|
||||
],
|
||||
\`\`\`
|
||||
You can also specify postAggregations, which are applied after data has been aggregated for the current granularity and dimensions bucket. See [Post Aggregations](Post Aggregations.html) for a detailed description. In the rand example, an arithmetic type operation (division, as specified by “fn”) is performed with the result “name” of “avg\_random”. The “fields” field specifies the inputs from the aggregation stage to this expression. Note that identifiers corresponding to “name” JSON field inside the type “fieldAccess” are required but not used outside this expression, so they are prefixed with “dummy” for clarity:
|
||||
\`\`\`javascript
|
||||
[postAggregations]() [{
|
||||
[type]() “arithmetic”,
|
||||
[name]() “avg\_random”,
|
||||
[fn]() “/”,
|
||||
[fields]() [
|
||||
{ [type]() “fieldAccess”, [fieldName]() “randomNumberSum” },
|
||||
{ [type]() “fieldAccess”, [fieldName]() “rows” }
|
||||
```
|
||||
|
||||
You can also specify postAggregations, which are applied after data has been aggregated for the current granularity and dimensions bucket. See [Post Aggregations](Post Aggregations.html) for a detailed description. In the rand example, an arithmetic type operation (division, as specified by "fn") is performed with the result "name" of "avg_random". The "fields" field specifies the inputs from the aggregation stage to this expression. Note that identifiers corresponding to "name" JSON field inside the type "fieldAccess" are required but not used outside this expression, so they are prefixed with "dummy" for clarity:
|
||||
|
||||
```javascript
|
||||
"postAggregations": [{
|
||||
"type": "arithmetic",
|
||||
"name": "avg_random",
|
||||
"fn": "/",
|
||||
"fields": [
|
||||
{ "type": "fieldAccess", "fieldName": "randomNumberSum" },
|
||||
{ "type": "fieldAccess", "fieldName": "rows" }
|
||||
]
|
||||
}],
|
||||
\`\`\`
|
||||
```
|
||||
The time range(s) of the query; data outside the specified intervals will not be used; this example specifies from October 1, 2012 until January 1, 2020:
|
||||
\`\`\`javascript
|
||||
[intervals]() [“2012-10-01T00:00/2020-01-01T00”]
|
||||
|
||||
```javascript
|
||||
"intervals": ["2012-10-01T00:00/2020-01-01T00"]
|
||||
}
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
Query Operators
|
||||
---------------
|
||||
|
|
@ -99,12 +113,12 @@ The following table summarizes query properties.
|
|||
|timeseries, groupBy, search, timeBoundary|dataSource|query is applied to this data source|yes|
|
||||
|timeseries, groupBy, search|intervals|range of time series to include in query|yes|
|
||||
|timeseries, groupBy, search, timeBoundary|context|This is a key-value map that can allow the query to alter some of the behavior of a query. It is primarily used for debugging, for example if you include `"bySegment":true` in the map, you will get results associated with the data segment they came from.|no|
|
||||
|timeseries, groupBy, search|filter|Specifies the filter (the “WHERE” clause in SQL) for the query. See [Filters](Filters.html)|no|
|
||||
|timeseries, groupBy, search|granularity|the timestamp granularity to bucket results into (i.e. “hour”). See [Granularities](Granularities.html) for more information.|no|
|
||||
|timeseries, groupBy, search|filter|Specifies the filter (the "WHERE" clause in SQL) for the query. See [Filters](Filters.html)|no|
|
||||
|timeseries, groupBy, search|granularity|the timestamp granularity to bucket results into (i.e. "hour"). See [Granularities](Granularities.html) for more information.|no|
|
||||
|groupBy|dimensions|constrains the groupings; if empty, then one value per time granularity bucket|yes|
|
||||
|timeseries, groupBy|aggregations|aggregations that combine values in a bucket. See [Aggregations](Aggregations.html).|yes|
|
||||
|timeseries, groupBy|postAggregations|aggregations of aggregations. See [Post Aggregations](Post Aggregations.html).|yes|
|
||||
|search|limit|maximum number of results (doc_page is 1000), a system-level maximum can also be set via `com.metamx.query.search.maxSearchLimit`|no|
|
||||
|search|limit|maximum number of results (default is 1000), a system-level maximum can also be set via `com.metamx.query.search.maxSearchLimit`|no|
|
||||
|search|searchDimensions|Dimensions to apply the search query to. If not specified, it will search through all dimensions.|no|
|
||||
|search|query|The query portion of the search query. This is essentially a predicate that specifies if something matches.|yes|
|
||||
|
||||
|
|
|
|||
|
|
@ -23,12 +23,12 @@ Configuration
|
|||
|
||||
Realtime nodes take a mix of base server configuration and spec files that describe how to connect, process and expose the realtime feed. See [Configuration](Configuration.html) for information about general server configuration.
|
||||
|
||||
### Realtime “specFile”
|
||||
### Realtime "specFile"
|
||||
|
||||
The property `druid.realtime.specFile` has the path of a file (absolute or relative path and file name) with realtime specifications in it. This “specFile” should be a JSON Array of JSON objects like the following:
|
||||
The property `druid.realtime.specFile` has the path of a file (absolute or relative path and file name) with realtime specifications in it. This "specFile" should be a JSON Array of JSON objects like the following:
|
||||
|
||||
<code>
|
||||
[{
|
||||
```json
|
||||
[{
|
||||
"schema" : { "dataSource":"dataSourceName",
|
||||
"aggregators":[ {"type":"count", "name":"events"},
|
||||
{"type":"doubleSum","name":"outColumn","fieldName":"inColumn"} ],
|
||||
|
|
@ -53,8 +53,8 @@ The property `druid.realtime.specFile` has the path of a file (absolute or relat
|
|||
"windowPeriod" : "PT10m",
|
||||
"segmentGranularity":"hour",
|
||||
"basePersistDirectory" : "/tmp/realtime/basePersist" }
|
||||
}]
|
||||
</code>
|
||||
}]
|
||||
```
|
||||
|
||||
This is a JSON Array so you can give more than one realtime stream to a given node. The number you can put in the same process depends on the exact configuration. In general, it is best to think of each realtime stream handler as requiring 2-threads: 1 thread for data consumption and aggregation, 1 thread for incremental persists and other background tasks.
|
||||
|
||||
|
|
@ -68,7 +68,7 @@ This describes the data schema for the output Druid segment. More information ab
|
|||
|-----|----|-----------|--------|
|
||||
|aggregators|Array of Objects|The list of aggregators to use to aggregate colliding rows together.|yes|
|
||||
|dataSource|String|The name of the dataSource that the segment belongs to.|yes|
|
||||
|indexGranularity|String|The granularity of the data inside the segment. E.g. a value of “minute” will mean that data is aggregated at minutely granularity. That is, if there are collisions in the tuple (minute(timestamp), dimensions), then it will aggregate values together using the aggregators instead of storing individual rows.|yes|
|
||||
|indexGranularity|String|The granularity of the data inside the segment. E.g. a value of "minute" will mean that data is aggregated at minutely granularity. That is, if there are collisions in the tuple (minute(timestamp), dimensions), then it will aggregate values together using the aggregators instead of storing individual rows.|yes|
|
||||
|segmentGranularity|String|The granularity of the segment as a whole. This is generally larger than the index granularity and describes the rate at which the realtime server will push segments out for historical servers to take over.|yes|
|
||||
|shardSpec|Object|This describes the shard that is represented by this server. This must be specified properly in order to have multiple realtime nodes indexing the same data stream in a sharded fashion.|no|
|
||||
|
||||
|
|
@ -94,7 +94,8 @@ Constraints
|
|||
|
||||
The following tables summarizes constraints between settings in the spec file for the Realtime subsystem.
|
||||
|
||||
|*. Name |*. Effect |*. Minimum |*. Recommended |
|
||||
|Name|Effect|Minimum|Recommended|
|
||||
|----|------|-------|-----------|
|
||||
| windowPeriod| when reading an InputRow, events with timestamp older than now minus this window are discarded | time jitter tolerance | use this to reject outliers |
|
||||
| segmentGranularity| time granularity (minute, hour, day, week, month) for loading data at query time | equal to indexGranularity| more than indexGranularity|
|
||||
| indexGranularity| time granularity (minute, hour, day, week, month) of indexes | less than segmentGranularity| minute, hour, day, week, month |
|
||||
|
|
@ -115,8 +116,8 @@ Extending the code
|
|||
|
||||
Realtime integration is intended to be extended in two ways:
|
||||
|
||||
1. Connect to data streams from varied systems ([Firehose](https://github.com/metamx/druid/blob/master/realtime/src/main/java/com/metamx/druid/realtime/FirehoseFactory.java))
|
||||
2. Adjust the publishing strategy to match your needs ([Plumber](https://github.com/metamx/druid/blob/master/realtime/src/main/java/com/metamx/druid/realtime/PlumberSchool.java))
|
||||
1. Connect to data streams from varied systems ([Firehose](https://github.com/metamx/druid/blob/druid-0.5.x/realtime/src/main/java/com/metamx/druid/realtime/firehose/FirehoseFactory.java))
|
||||
2. Adjust the publishing strategy to match your needs ([Plumber](https://github.com/metamx/druid/blob/druid-0.5.x/realtime/src/main/java/com/metamx/druid/realtime/plumber/PlumberSchool.java))
|
||||
|
||||
The expectations are that the former will be very common and something that users of Druid will do on a fairly regular basis. Most users will probably never have to deal with the latter form of customization. Indeed, we hope that all potential use cases can be packaged up as part of Druid proper without requiring proprietary customization.
|
||||
|
||||
|
|
@ -124,9 +125,9 @@ Given those expectations, adding a firehose is straightforward and completely en
|
|||
|
||||
We will do our best to accept contributions from the community of new Firehoses and Plumbers, but we also understand the requirement for being able to plug in your own proprietary implementations. The model for doing this is by embedding the druid code in another project and writing your own `main()` method that initializes a RealtimeNode object and registers your proprietary objects with it.
|
||||
|
||||
<code>
|
||||
public class MyRealtimeMain
|
||||
{
|
||||
```java
|
||||
public class MyRealtimeMain
|
||||
{
|
||||
private static final Logger log = new Logger(MyRealtimeMain.class);
|
||||
|
||||
public static void main(String[] args) throws Exception
|
||||
|
|
@ -151,7 +152,7 @@ We will do our best to accept contributions from the community of new Firehoses
|
|||
|
||||
lifecycle.join();
|
||||
}
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
Pluggable pieces of the system are either handled by a setter on the RealtimeNode object, or they are configuration driven and need to be setup to allow for [Jackson polymorphic deserialization](http://wiki.fasterxml.com/JacksonPolymorphicDeserialization) and registered via the relevant methods on the RealtimeNode object.
|
||||
|
|
|
|||
|
|
@ -12,33 +12,33 @@ Load rules indicate how many replicants of a segment should exist in a server ti
|
|||
|
||||
Interval load rules are of the form:
|
||||
|
||||
<code>
|
||||
{
|
||||
```json
|
||||
{
|
||||
"type" : "loadByInterval",
|
||||
"interval" : "2012-01-01/2013-01-01",
|
||||
"tier" : "hot"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
type - this should always be “loadByInterval”
|
||||
interval - A JSON Object representing ISO-8601 Intervals
|
||||
tier - the configured compute node tier
|
||||
* `type` - this should always be "loadByInterval"
|
||||
* `interval` - A JSON Object representing ISO-8601 Intervals
|
||||
* `tier` - the configured compute node tier
|
||||
|
||||
### Period Load Rule
|
||||
|
||||
Period load rules are of the form:
|
||||
|
||||
<code>
|
||||
{
|
||||
"type" : "loadByInterval",
|
||||
```json
|
||||
{
|
||||
"type" : "loadByPeriod",
|
||||
"period" : "P1M",
|
||||
"tier" : "hot"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
type - this should always be “loadByPeriod”
|
||||
period - A JSON Object representing ISO-8601 Periods
|
||||
tier - the configured compute node tier
|
||||
* `type` - this should always be "loadByPeriod"
|
||||
* `period` - A JSON Object representing ISO-8601 Periods
|
||||
* `tier` - the configured compute node tier
|
||||
|
||||
The interval of a segment will be compared against the specified period. The rule matches if the period overlaps the interval.
|
||||
|
||||
|
|
@ -51,15 +51,15 @@ Drop rules indicate when segments should be dropped from the cluster.
|
|||
|
||||
Interval drop rules are of the form:
|
||||
|
||||
<code>
|
||||
{
|
||||
```json
|
||||
{
|
||||
"type" : "dropByInterval",
|
||||
"interval" : "2012-01-01/2013-01-01"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
type - this should always be “dropByInterval”
|
||||
interval - A JSON Object representing ISO-8601 Periods
|
||||
* `type` - this should always be "dropByInterval"
|
||||
* `interval` - A JSON Object representing ISO-8601 Periods
|
||||
|
||||
A segment is dropped if the interval contains the interval of the segment.
|
||||
|
||||
|
|
@ -67,14 +67,14 @@ A segment is dropped if the interval contains the interval of the segment.
|
|||
|
||||
Period drop rules are of the form:
|
||||
|
||||
<code>
|
||||
{
|
||||
```json
|
||||
{
|
||||
"type" : "dropByPeriod",
|
||||
"period" : "P1M"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
type - this should always be “dropByPeriod”
|
||||
period - A JSON Object representing ISO-8601 Periods
|
||||
* `type` - this should always be "dropByPeriod"
|
||||
* `period` - A JSON Object representing ISO-8601 Periods
|
||||
|
||||
The interval of a segment will be compared against the specified period. The period is from some time in the past to the current time. The rule matches if the period contains the interval.
|
||||
|
|
|
|||
|
|
@ -28,14 +28,14 @@ There are several main parts to a search query:
|
|||
|
||||
|property|description|required?|
|
||||
|--------|-----------|---------|
|
||||
|queryType|This String should always be “search”; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|queryType|This String should always be "search"; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|dataSource|A String defining the data source to query, very similar to a table in a relational database|yes|
|
||||
|granularity|Defines the granularity of the query. See [Granularities](Granularities.html)|yes|
|
||||
|filter|See [Filters](Filters.html)|no|
|
||||
|intervals|A JSON Object representing ISO-8601 Intervals. This defines the time ranges to run the query over.|yes|
|
||||
|searchDimensions|The dimensions to run the search over. Excluding this means the search is run over all dimensions.|no|
|
||||
|query|See [SearchQuerySpec](SearchQuerySpec.html).|yes|
|
||||
|sort|How the results of the search should sorted. Two possible types here are “lexicographic” and “strlen”.|yes|
|
||||
|sort|How the results of the search should sorted. Two possible types here are "lexicographic" and "strlen".|yes|
|
||||
|context|An additional JSON Object which can be used to specify certain flags.|no|
|
||||
|
||||
The format of the result is:
|
||||
|
|
|
|||
|
|
@ -1,26 +1,28 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
Search query specs define how a “match” is defined between a search value and a dimension value. The available search query specs are:
|
||||
Search query specs define how a "match" is defined between a search value and a dimension value. The available search query specs are:
|
||||
|
||||
InsensitiveContainsSearchQuerySpec
|
||||
----------------------------------
|
||||
|
||||
If any part of a dimension value contains the value specified in this search query spec, regardless of case, a “match” occurs. The grammar is:
|
||||
If any part of a dimension value contains the value specified in this search query spec, regardless of case, a "match" occurs. The grammar is:
|
||||
|
||||
<code>{
|
||||
```json
|
||||
{
|
||||
"type" : "insensitive_contains",
|
||||
"value" : "some_value"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
FragmentSearchQuerySpec
|
||||
-----------------------
|
||||
|
||||
If any part of a dimension value contains any of the values specified in this search query spec, regardless of case, a “match” occurs. The grammar is:
|
||||
If any part of a dimension value contains any of the values specified in this search query spec, regardless of case, a "match" occurs. The grammar is:
|
||||
|
||||
<code>{
|
||||
```json
|
||||
{
|
||||
"type" : "fragment",
|
||||
"values" : ["fragment1", "fragment2"]
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
|
|
|||
|
|
@ -2,25 +2,27 @@
|
|||
layout: doc_page
|
||||
---
|
||||
Segment metadata queries return per segment information about:
|
||||
\* Cardinality of all columns in the segment
|
||||
\* Estimated byte size for the segment columns in TSV format
|
||||
\* Interval the segment covers
|
||||
\* Column type of all the columns in the segment
|
||||
\* Estimated total segment byte size in TSV format
|
||||
\* Segment id
|
||||
|
||||
<code>{
|
||||
* Cardinality of all columns in the segment
|
||||
* Estimated byte size for the segment columns in TSV format
|
||||
* Interval the segment covers
|
||||
* Column type of all the columns in the segment
|
||||
* Estimated total segment byte size in TSV format
|
||||
* Segment id
|
||||
|
||||
```json
|
||||
{
|
||||
"queryType":"segmentMetadata",
|
||||
"dataSource":"sample_datasource",
|
||||
"intervals":["2013-01-01/2014-01-01"],
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
There are several main parts to a segment metadata query:
|
||||
|
||||
|property|description|required?|
|
||||
|--------|-----------|---------|
|
||||
|queryType|This String should always be “segmentMetadata”; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|queryType|This String should always be "segmentMetadata"; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|dataSource|A String defining the data source to query, very similar to a table in a relational database|yes|
|
||||
|intervals|A JSON Object representing ISO-8601 Intervals. This defines the time ranges to run the query over.|yes|
|
||||
|merge|Merge all individual segment metadata results into a single result|no|
|
||||
|
|
@ -28,31 +30,16 @@ There are several main parts to a segment metadata query:
|
|||
|
||||
The format of the result is:
|
||||
|
||||
<code>[ {
|
||||
```json
|
||||
[ {
|
||||
"id" : "some_id",
|
||||
"intervals" : [ "2013-05-13T00:00:00.000Z/2013-05-14T00:00:00.000Z" ],
|
||||
"columns" : {
|
||||
"__time" : {
|
||||
"type" : "LONG",
|
||||
"size" : 407240380,
|
||||
"cardinality" : null
|
||||
},
|
||||
"dim1" : {
|
||||
"type" : "STRING",
|
||||
"size" : 100000,
|
||||
"cardinality" : 1944
|
||||
},
|
||||
"dim2" : {
|
||||
"type" : "STRING",
|
||||
"size" : 100000,
|
||||
"cardinality" : 1504
|
||||
},
|
||||
"metric1" : {
|
||||
"type" : "FLOAT",
|
||||
"size" : 100000,
|
||||
"cardinality" : null
|
||||
}
|
||||
"__time" : { "type" : "LONG", "size" : 407240380, "cardinality" : null },
|
||||
"dim1" : { "type" : "STRING", "size" : 100000, "cardinality" : 1944 },
|
||||
"dim2" : { "type" : "STRING", "size" : 100000, "cardinality" : 1504 },
|
||||
"metric1" : { "type" : "FLOAT", "size" : 100000, "cardinality" : null }
|
||||
},
|
||||
"size" : 300000
|
||||
} ]
|
||||
</code>
|
||||
} ]
|
||||
```
|
||||
|
|
|
|||
|
|
@ -14,36 +14,28 @@ Naming Convention
|
|||
Identifiers for segments are typically constructed using the segment datasource, interval start time (in ISO 8601 format), interval end time (in ISO 8601 format), and a version. If data is additionally sharded beyond a time range, the segment identifier will also contain a partition number.
|
||||
|
||||
An example segment identifier may be:
|
||||
datasource\_intervalStart\_intervalEnd\_version\_partitionNum
|
||||
datasource_intervalStart_intervalEnd_version_partitionNum
|
||||
|
||||
Segment Components
|
||||
------------------
|
||||
|
||||
A segment is compromised of several files, listed below.
|
||||
|
||||
### `version.bin`
|
||||
* `version.bin`
|
||||
|
||||
4 bytes representing the current segment version as an integer. E.g., for v9 segments, the version is 0x0, 0x0, 0x0, 0x9
|
||||
4 bytes representing the current segment version as an integer. E.g., for v9 segments, the version is 0x0, 0x0, 0x0, 0x9
|
||||
|
||||
### `meta.smoosh`
|
||||
* `meta.smoosh`
|
||||
|
||||
A file with metadata (filenames and offsets) about the contents of the other `smoosh` files
|
||||
A file with metadata (filenames and offsets) about the contents of the other `smoosh` files
|
||||
|
||||
### `XXXXX.smoosh`
|
||||
* `XXXXX.smoosh`
|
||||
|
||||
There are some number of these files, which are concatenated binary data
|
||||
There are some number of these files, which are concatenated binary data
|
||||
|
||||
The `smoosh` files represent multiple files “smooshed” together in order to minimize the number of file descriptors that must be open to house the data. They are files of up to 2GB in size (to match the limit of a memory mapped ByteBuffer in Java). The `smoosh` files house individual files for each of the columns in the data as well as an `index.drd` file with extra metadata about the segment.
|
||||
The `smoosh` files represent multiple files "smooshed" together in order to minimize the number of file descriptors that must be open to house the data. They are files of up to 2GB in size (to match the limit of a memory mapped ByteBuffer in Java). The `smoosh` files house individual files for each of the columns in the data as well as an `index.drd` file with extra metadata about the segment.
|
||||
|
||||
There is also a special column called `__time` that refers to the time column of the segment. This will hopefully become less and less special as the code evolves, but for now it’s as special as my Mommy always told me I am.
|
||||
|
||||
### `index.drd`
|
||||
|
||||
The `index.drd` file houses 3 pieces of data in order
|
||||
|
||||
1. The names of all of the columns of the data
|
||||
2. The names of the “dimensions” of the data (these are the dictionary-encoded, string columns. This is here to support some legacy APIs and will be superfluous in the future)
|
||||
3. The data interval represented by this segment stored as the start and end timestamps as longs
|
||||
There is also a special column called `__time` that refers to the time column of the segment. This will hopefully become less and less special as the code evolves, but for now it’s as special as my Mommy always told me I am.
|
||||
|
||||
Format of a column
|
||||
------------------
|
||||
|
|
|
|||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
Tasks are run on workers and always operate on a single datasource. Once an indexer coordinator node accepts a task, a lock is created for the datasource and interval specified in the task. Tasks do not need to explicitly release locks, they are released upon task completion. Tasks may potentially release locks early if they desire. Tasks ids are unique by naming them using UUIDs or the timestamp in which the task was created. Tasks are also part of a “task group”, which is a set of tasks that can share interval locks.
|
||||
Tasks are run on workers and always operate on a single datasource. Once an indexer coordinator node accepts a task, a lock is created for the datasource and interval specified in the task. Tasks do not need to explicitly release locks, they are released upon task completion. Tasks may potentially release locks early if they desire. Tasks ids are unique by naming them using UUIDs or the timestamp in which the task was created. Tasks are also part of a "task group", which is a set of tasks that can share interval locks.
|
||||
|
||||
There are several different types of tasks.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
|
||||
YourKit supports the Druid open source projects with its
|
||||
full-featured Java Profiler.
|
||||
YourKit, LLC is the creator of innovative and intelligent tools for profiling
|
||||
|
|
|
|||
|
|
@ -3,27 +3,29 @@ layout: doc_page
|
|||
---
|
||||
Time boundary queries return the earliest and latest data points of a data set. The grammar is:
|
||||
|
||||
<code>{
|
||||
```json
|
||||
{
|
||||
"queryType" : "timeBoundary",
|
||||
"dataSource": "sample_datasource"
|
||||
}
|
||||
</code>
|
||||
}
|
||||
```
|
||||
|
||||
There are 3 main parts to a time boundary query:
|
||||
|
||||
|property|description|required?|
|
||||
|--------|-----------|---------|
|
||||
|queryType|This String should always be “timeBoundary”; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|queryType|This String should always be "timeBoundary"; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|dataSource|A String defining the data source to query, very similar to a table in a relational database|yes|
|
||||
|context|An additional JSON Object which can be used to specify certain flags.|no|
|
||||
|
||||
The format of the result is:
|
||||
|
||||
<code>[ {
|
||||
```json
|
||||
[ {
|
||||
"timestamp" : "2013-05-09T18:24:00.000Z",
|
||||
"result" : {
|
||||
"minTime" : "2013-05-09T18:24:00.000Z",
|
||||
"maxTime" : "2013-05-09T18:37:00.000Z"
|
||||
}
|
||||
} ]
|
||||
</code>
|
||||
} ]
|
||||
```
|
||||
|
|
|
|||
|
|
@ -8,81 +8,46 @@ These types of queries take a timeseries query object and return an array of JSO
|
|||
|
||||
An example timeseries query object is shown below:
|
||||
|
||||
<pre>
|
||||
<code>
|
||||
```json
|
||||
{
|
||||
[queryType]() “timeseries”,
|
||||
[dataSource]() “sample\_datasource”,
|
||||
[granularity]() “day”,
|
||||
[filter]() {
|
||||
[type]() “and”,
|
||||
[fields]() [
|
||||
{
|
||||
[type]() “selector”,
|
||||
[dimension]() “sample\_dimension1”,
|
||||
[value]() “sample\_value1”
|
||||
},
|
||||
{
|
||||
[type]() “or”,
|
||||
[fields]() [
|
||||
{
|
||||
[type]() “selector”,
|
||||
[dimension]() “sample\_dimension2”,
|
||||
[value]() “sample\_value2”
|
||||
},
|
||||
{
|
||||
[type]() “selector”,
|
||||
[dimension]() “sample\_dimension3”,
|
||||
[value]() “sample\_value3”
|
||||
}
|
||||
"queryType": "timeseries",
|
||||
"dataSource": "sample_datasource",
|
||||
"granularity": "day",
|
||||
"filter": {
|
||||
"type": "and",
|
||||
"fields": [
|
||||
{ "type": "selector", "dimension": "sample_dimension1", "value": "sample_value1" },
|
||||
{ "type": "or",
|
||||
"fields": [
|
||||
{ "type": "selector", "dimension": "sample_dimension2", "value": "sample_value2" },
|
||||
{ "type": "selector", "dimension": "sample_dimension3", "value": "sample_value3" }
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
[aggregations]() [
|
||||
{
|
||||
[type]() “longSum”,
|
||||
[name]() “sample\_name1”,
|
||||
[fieldName]() “sample\_fieldName1”
|
||||
},
|
||||
{
|
||||
[type]() “doubleSum”,
|
||||
[name]() “sample\_name2”,
|
||||
[fieldName]() “sample\_fieldName2”
|
||||
}
|
||||
"aggregations": [
|
||||
{ "type": "longSum", "name": "sample_name1", "fieldName": "sample_fieldName1" },
|
||||
{ "type": "doubleSum", "name": "sample_name2", "fieldName": "sample_fieldName2" }
|
||||
],
|
||||
[postAggregations]() [
|
||||
{
|
||||
[type]() “arithmetic”,
|
||||
[name]() “sample\_divide”,
|
||||
[fn]() “/”,
|
||||
[fields]() [
|
||||
{
|
||||
[type]() “fieldAccess”,
|
||||
[name]() “sample\_name1”,
|
||||
[fieldName]() “sample\_fieldName1”
|
||||
},
|
||||
{
|
||||
[type]() “fieldAccess”,
|
||||
[name]() “sample\_name2”,
|
||||
[fieldName]() “sample\_fieldName2”
|
||||
}
|
||||
"postAggregations": [
|
||||
{ "type": "arithmetic",
|
||||
"name": "sample_divide",
|
||||
"fn": "/",
|
||||
"fields": [
|
||||
{ "type": "fieldAccess", "name": "sample_name1", "fieldName": "sample_fieldName1" },
|
||||
{ "type": "fieldAccess", "name": "sample_name2", "fieldName": "sample_fieldName2" }
|
||||
]
|
||||
}
|
||||
],
|
||||
[intervals]() [
|
||||
“2012-01-01T00:00:00.000/2012-01-03T00:00:00.000”
|
||||
]
|
||||
"intervals": [ "2012-01-01T00:00:00.000/2012-01-03T00:00:00.000" ]
|
||||
}
|
||||
|
||||
</pre>
|
||||
</code>
|
||||
```
|
||||
|
||||
There are 7 main parts to a timeseries query:
|
||||
|
||||
|property|description|required?|
|
||||
|--------|-----------|---------|
|
||||
|queryType|This String should always be “timeseries”; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|queryType|This String should always be "timeseries"; this is the first thing Druid looks at to figure out how to interpret the query|yes|
|
||||
|dataSource|A String defining the data source to query, very similar to a table in a relational database|yes|
|
||||
|granularity|Defines the granularity of the query. See [Granularities](Granularities.html)|yes|
|
||||
|filter|See [Filters](Filters.html)|no|
|
||||
|
|
@ -91,28 +56,17 @@ There are 7 main parts to a timeseries query:
|
|||
|intervals|A JSON Object representing ISO-8601 Intervals. This defines the time ranges to run the query over.|yes|
|
||||
|context|An additional JSON Object which can be used to specify certain flags.|no|
|
||||
|
||||
To pull it all together, the above query would return 2 data points, one for each day between 2012-01-01 and 2012-01-03, from the “sample\_datasource” table. Each data point would be the (long) sum of sample\_fieldName1, the (double) sum of sample\_fieldName2 and the (double) the result of sample\_fieldName1 divided by sample\_fieldName2 for the filter set. The output looks like this:
|
||||
To pull it all together, the above query would return 2 data points, one for each day between 2012-01-01 and 2012-01-03, from the "sample\_datasource" table. Each data point would be the (long) sum of sample\_fieldName1, the (double) sum of sample\_fieldName2 and the (double) the result of sample\_fieldName1 divided by sample\_fieldName2 for the filter set. The output looks like this:
|
||||
|
||||
<pre>
|
||||
<code>
|
||||
```json
|
||||
[
|
||||
{
|
||||
[timestamp]() “2012-01-01T00:00:00.000Z”,
|
||||
[result]() {
|
||||
[sample\_name1]() <some_value>,
|
||||
[sample\_name2]() <some_value>,
|
||||
[sample\_divide]() <some_value>
|
||||
}
|
||||
"timestamp": "2012-01-01T00:00:00.000Z",
|
||||
"result": { "sample_name1": <some_value>, "sample_name2": <some_value>, "sample_divide": <some_value> }
|
||||
},
|
||||
{
|
||||
[timestamp]() “2012-01-02T00:00:00.000Z”,
|
||||
[result]() {
|
||||
[sample\_name1]() <some_value>,
|
||||
[sample\_name2]() <some_value>,
|
||||
[sample\_divide]() <some_value>
|
||||
}
|
||||
"timestamp": "2012-01-02T00:00:00.000Z",
|
||||
"result": { "sample_name1": <some_value>, "sample_name2": <some_value>, "sample_divide": <some_value> }
|
||||
}
|
||||
]
|
||||
|
||||
</pre>
|
||||
</code>
|
||||
```
|
||||
|
|
@ -6,7 +6,7 @@ Greetings! This tutorial will help clarify some core Druid concepts. We will use
|
|||
About the data
|
||||
--------------
|
||||
|
||||
The data source we’ll be working with is Wikipedia edits. Each time an edit is made in Wikipedia, an event gets pushed to an IRC channel associated with the language of the Wikipedia page. We scrape IRC channels for several different languages and load this data into Druid.
|
||||
The data source we'll be working with is Wikipedia edits. Each time an edit is made in Wikipedia, an event gets pushed to an IRC channel associated with the language of the Wikipedia page. We scrape IRC channels for several different languages and load this data into Druid.
|
||||
|
||||
Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and where the user resides), and a list of metrics associated with the event (such as the number of characters added and deleted).
|
||||
|
||||
|
|
@ -47,7 +47,7 @@ There are two ways to setup Druid: download a tarball, or [Build From Source](Bu
|
|||
|
||||
### Download a Tarball
|
||||
|
||||
We’ve built a tarball that contains everything you’ll need. You’ll find it [here](http://static.druid.io/artifacts/releases/druid-services-0.5.54-bin.tar.gz)
|
||||
We've built a tarball that contains everything you'll need. You'll find it [here](http://static.druid.io/artifacts/releases/druid-services-0.5.54-bin.tar.gz)
|
||||
Download this file to a directory of your choosing.
|
||||
|
||||
You can extract the awesomeness within by issuing:
|
||||
|
|
@ -56,7 +56,7 @@ You can extract the awesomeness within by issuing:
|
|||
tar -zxvf druid-services-*-bin.tar.gz
|
||||
```
|
||||
|
||||
Not too lost so far right? That’s great! If you cd into the directory:
|
||||
Not too lost so far right? That's great! If you cd into the directory:
|
||||
|
||||
```
|
||||
cd druid-services-0.5.54
|
||||
|
|
@ -71,7 +71,7 @@ You should see a bunch of files:
|
|||
Running Example Scripts
|
||||
-----------------------
|
||||
|
||||
Let’s start doing stuff. You can start a Druid [Realtime](Realtime.html) node by issuing:
|
||||
Let's start doing stuff. You can start a Druid [Realtime](Realtime.html) node by issuing:
|
||||
|
||||
```
|
||||
./run_example_server.sh
|
||||
|
|
@ -90,13 +90,13 @@ Once the node starts up you will see a bunch of logs about setting up properties
|
|||
|
||||
The Druid real time-node ingests events in an in-memory buffer. Periodically, these events will be persisted to disk. If you are interested in the details of our real-time architecture and why we persist indexes to disk, I suggest you read our [White Paper](http://static.druid.io/docs/druid.pdf).
|
||||
|
||||
Okay, things are about to get real-time. To query the real-time node you’ve spun up, you can issue:
|
||||
Okay, things are about to get real-time. To query the real-time node you've spun up, you can issue:
|
||||
|
||||
```
|
||||
./run_example_client.sh
|
||||
```
|
||||
|
||||
Select "wikipedia" once again. This script issues [GroupByQuery](GroupByQuery.html)s to the data we’ve been ingesting. The query looks like this:
|
||||
Select "wikipedia" once again. This script issues [GroupByQuery](GroupByQuery.html)s to the data we've been ingesting. The query looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
|
|
@ -137,7 +137,7 @@ The result looks something like this:
|
|||
...
|
||||
```
|
||||
|
||||
This groupBy query is a bit complicated and we’ll return to it later. For the time being, just make sure you are getting some blocks of data back. If you are having problems, make sure you have [curl](http://curl.haxx.se/) installed. Control+C to break out of the client script.
|
||||
This groupBy query is a bit complicated and we'll return to it later. For the time being, just make sure you are getting some blocks of data back. If you are having problems, make sure you have [curl](http://curl.haxx.se/) installed. Control+C to break out of the client script.
|
||||
|
||||
h2. Querying Druid
|
||||
|
||||
|
|
@ -159,7 +159,7 @@ Druid queries are JSON blobs which are relatively painless to create programmati
|
|||
The [TimeBoundaryQuery](TimeBoundaryQuery.html) is one of the simplest Druid queries. To run the query, you can issue:
|
||||
|
||||
```
|
||||
curl -X POST ‘http://localhost:8083/druid/v2/?pretty’ -H 'content-type: application/json' -d @time_boundary_query.body
|
||||
curl -X POST 'http://localhost:8083/druid/v2/?pretty' -H 'content-type: application/json' -d @time_boundary_query.body
|
||||
```
|
||||
|
||||
We get something like this JSON back:
|
||||
|
|
@ -255,9 +255,9 @@ This gives us something like the following:
|
|||
Solving a Problem
|
||||
-----------------
|
||||
|
||||
One of Druid’s main powers is to provide answers to problems, so let’s pose a problem. What if we wanted to know what the top pages in the US are, ordered by the number of edits over the last few minutes you’ve been going through this tutorial? To solve this problem, we have to return to the query we introduced at the very beginning of this tutorial, the [GroupByQuery](GroupByQuery.html). It would be nice if we could group by results by dimension value and somehow sort those results... and it turns out we can!
|
||||
One of Druid's main powers is to provide answers to problems, so let's pose a problem. What if we wanted to know what the top pages in the US are, ordered by the number of edits over the last few minutes you've been going through this tutorial? To solve this problem, we have to return to the query we introduced at the very beginning of this tutorial, the [GroupByQuery](GroupByQuery.html). It would be nice if we could group by results by dimension value and somehow sort those results... and it turns out we can!
|
||||
|
||||
Let’s create the file:
|
||||
Let's create the file:
|
||||
|
||||
```
|
||||
group_by_query.body
|
||||
|
|
@ -272,7 +272,7 @@ and put the following in there:
|
|||
"granularity": "all",
|
||||
"dimensions": [ "page" ],
|
||||
"orderBy": {
|
||||
"type": "doc_page",
|
||||
"type": "default",
|
||||
"columns": [ { "dimension": "edit_count", "direction": "DESCENDING" } ],
|
||||
"limit": 10
|
||||
},
|
||||
|
|
@ -284,7 +284,7 @@ and put the following in there:
|
|||
}
|
||||
```
|
||||
|
||||
Woah! Our query just got a way more complicated. Now we have these [Filters](Filters.html) things and this [OrderBy](OrderBy.html) thing. Fear not, it turns out the new objects we’ve introduced to our query can help define the format of our results and provide an answer to our question.
|
||||
Woah! Our query just got a way more complicated. Now we have these [Filters](Filters.html) things and this [OrderBy](OrderBy.html) thing. Fear not, it turns out the new objects we've introduced to our query can help define the format of our results and provide an answer to our question.
|
||||
|
||||
If you issue the query:
|
||||
|
||||
|
|
|
|||
|
|
@ -92,7 +92,7 @@ Let's start up a few nodes and download our data. First things though, let's cre
|
|||
mkdir config
|
||||
```
|
||||
|
||||
If you are interested in learning more about Druid configuration files, check out this [link](https://github.com/metamx/druid/wiki/Configuration). Many aspects of Druid are customizable. For the purposes of this tutorial, we are going to use doc_page values for most things.
|
||||
If you are interested in learning more about Druid configuration files, check out this [link](https://github.com/metamx/druid/wiki/Configuration). Many aspects of Druid are customizable. For the purposes of this tutorial, we are going to use default values for most things.
|
||||
|
||||
### Start a Master Node ###
|
||||
|
||||
|
|
|
|||
|
|
@ -6,126 +6,140 @@ Greetings! This tutorial will help clarify some core Druid concepts. We will use
|
|||
About the data
|
||||
--------------
|
||||
|
||||
The data source we’ll be working with is the Bit.ly USA Government website statistics stream. You can see the stream [here](http://developer.usa.gov/1usagov), and read about the stream [here](http://www.usa.gov/About/developer-resources/1usagov.shtml) . This is a feed of json data that gets updated whenever anyone clicks a bit.ly shortened USA.gov website. A typical event might look something like this:
|
||||
\`\`\`json
|
||||
The data source we'll be working with is the Bit.ly USA Government website statistics stream. You can see the stream [here](http://developer.usa.gov/1usagov), and read about the stream [here](http://www.usa.gov/About/developer-resources/1usagov.shtml) . This is a feed of json data that gets updated whenever anyone clicks a bit.ly shortened USA.gov website. A typical event might look something like this:
|
||||
|
||||
```json
|
||||
{
|
||||
[user\_agent]() “Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)”,
|
||||
[country]() “US”,
|
||||
[known\_user]() 1,
|
||||
[timezone]() “America/New\_York”,
|
||||
[geo\_region]() “DC”,
|
||||
[global\_bitly\_hash]() “17ctAFs”,
|
||||
[encoding\_user\_bitly\_hash]() “17ctAFr”,
|
||||
[encoding\_user\_login]() “senrubiopress”,
|
||||
[aaccept\_language]() “en-US”,
|
||||
[short\_url\_cname]() “1.usa.gov”,
|
||||
[referring\_url]() “http://t.co/4Av4NUFAYq”,
|
||||
[long\_url]() “http://www.rubio.senate.gov/public/index.cfm/fighting-for-florida?ID=c8357d12-9da8-4e9d-b00d-7168e1bf3599”,
|
||||
[timestamp]() 1372190407,
|
||||
[timestamp of time hash was created]() 1372190097,
|
||||
[city]() “Washington”,
|
||||
[latitude\_longitude]() [
|
||||
38.893299,
|
||||
~~77.014603
|
||||
]
|
||||
"user_agent": "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)",
|
||||
"country": "US",
|
||||
"known_user": 1,
|
||||
"timezone": "America/New_York",
|
||||
"geo_region": "DC",
|
||||
"global_bitly_hash": "17ctAFs",
|
||||
"encoding_user_bitly_hash": "17ctAFr",
|
||||
"encoding_user_login": "senrubiopress",
|
||||
"aaccept_language": "en-US",
|
||||
"short_url_cname": "1.usa.gov",
|
||||
"referring_url": "http://t.co/4Av4NUFAYq",
|
||||
"long_url": "http://www.rubio.senate.gov/public/index.cfm/fighting-for-florida?ID=c8357d12-9da8-4e9d-b00d-7168e1bf3599",
|
||||
"timestamp": 1372190407,
|
||||
"timestamp of time hash was created": 1372190097,
|
||||
"city": "Washington",
|
||||
"latitude_longitude": [ 38.893299, -77.014603 ]
|
||||
}
|
||||
\`\`\`
|
||||
The “known\_user” field is always 1 or 0. It is 1 if the user is known to the server, and 0 otherwise. We will use this field extensively in this demo.
|
||||
```
|
||||
|
||||
The "known_user" field is always 1 or 0. It is 1 if the user is known to the server, and 0 otherwise. We will use this field extensively in this demo.
|
||||
|
||||
h2. Setting Up
|
||||
There are two ways to setup Druid: download a tarball, or ]. You only need to do one of these.
|
||||
|
||||
There are two ways to setup Druid: download a tarball, or [Build From Source](Build-From-Source.html). You only need to do one of these.
|
||||
|
||||
h3. Download a Tarball
|
||||
We’ve built a tarball that contains everything you’ll need. You’ll find it [here](http://static.druid.io/artifacts/releases/druid-services-0.5.50-bin.tar.gz)
|
||||
|
||||
We've built a tarball that contains everything you'll need. You'll find it [here](http://static.druid.io/artifacts/releases/druid-services-0.5.50-bin.tar.gz)
|
||||
Download this file to a directory of your choosing.
|
||||
You can extract the awesomeness within by issuing:
|
||||
\<pre\>tar~~zxvf druid-services~~**~~bin.tar.gz\</pre\>
|
||||
Not too lost so far right? That’s great! If you cd into the directory:
|
||||
\<pre\>cd druid-services-0.5.50\</pre\>
|
||||
|
||||
```
|
||||
tar zxvf druid-services-*-bin.tar.gz
|
||||
```
|
||||
|
||||
Not too lost so far right? That's great! If you cd into the directory:
|
||||
|
||||
```
|
||||
cd druid-services-0.5.50
|
||||
```
|
||||
|
||||
You should see a bunch of files:
|
||||
\* run\_example\_server.sh
|
||||
\* run\_example\_client.sh
|
||||
\* LICENSE, config, examples, lib directories
|
||||
|
||||
* run_example_server.sh
|
||||
* run_example_client.sh
|
||||
* LICENSE, config, examples, lib directories
|
||||
|
||||
h2. Running Example Scripts
|
||||
Let’s start doing stuff. You can start a Druid ] node by issuing:
|
||||
\<pre\>./run\_example\_server.sh\</pre\>
|
||||
Select “webstream”.
|
||||
Let's start doing stuff. You can start a Druid [Realtime](Realtime.html) node by issuing:
|
||||
|
||||
```
|
||||
./run_example_server.sh
|
||||
```
|
||||
|
||||
Select "webstream".
|
||||
Once the node starts up you will see a bunch of logs about setting up properties and connecting to the data source. If everything was successful, you should see messages of the form shown below.
|
||||
\<pre\><code>
|
||||
|
||||
```
|
||||
2013-07-19 21:54:05,154 INFO com.metamx.druid.realtime.RealtimeNode~~ Starting Jetty
|
||||
2013-07-19 21:54:05,154 INFO org.mortbay.log - jetty-6.1.x
|
||||
2013-07-19 21:54:05,171 INFO com.metamx.druid.realtime.plumber.RealtimePlumberSchool - Expect to run at
|
||||
2013-07-19 21:54:05,246 INFO org.mortbay.log - Started SelectChannelConnector@0.0.0.0:8083
|
||||
</code>\</pre\>
|
||||
```
|
||||
|
||||
The Druid real time-node ingests events in an in-memory buffer. Periodically, these events will be persisted to disk. If you are interested in the details of our real-time architecture and why we persist indexes to disk, I suggest you read our [White Paper](http://static.druid.io/docs/druid.pdf).
|
||||
Okay, things are about to get real. To query the real-time node you’ve spun up, you can issue:
|
||||
\<pre\>./run\_example\_client.sh\</pre\>
|
||||
Select “webstream” once again. This script issues ]s to the data we’ve been ingesting. The query looks like this:
|
||||
\`\`\`json
|
||||
Okay, things are about to get real. To query the real-time node you've spun up, you can issue:
|
||||
|
||||
```
|
||||
./run_example_client.sh
|
||||
```
|
||||
|
||||
Select "webstream" once again. This script issues [GroupByQuery](GroupByQuery.html)s to the data we've been ingesting. The query looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
[queryType]() “groupBy”,
|
||||
[dataSource]() “webstream”,
|
||||
[granularity]() “minute”,
|
||||
[dimensions]() [
|
||||
“timezone”
|
||||
"queryType": "groupBy",
|
||||
"dataSource": "webstream",
|
||||
"granularity": "minute",
|
||||
"dimensions": [ "timezone" ],
|
||||
"aggregations": [
|
||||
{ "type": "count", "name": "rows" },
|
||||
{ "type": "doubleSum", "fieldName": "known_users", "name": "known_users" }
|
||||
],
|
||||
[aggregations]() [
|
||||
{
|
||||
[type]() “count”,
|
||||
[name]() “rows”
|
||||
},
|
||||
{
|
||||
[type]() “doubleSum”,
|
||||
[fieldName]() “known\_users”,
|
||||
[name]() “known\_users”
|
||||
}
|
||||
],
|
||||
[filter]() {
|
||||
[type]() “selector”,
|
||||
[dimension]() “country”,
|
||||
[value]() “US”
|
||||
},
|
||||
[intervals]() [
|
||||
“2013-06-01T00:00/2020-01-01T00”
|
||||
]
|
||||
"filter": { "type": "selector", "dimension": "country", "value": "US" },
|
||||
"intervals": [ "2013-06-01T00:00/2020-01-01T00" ]
|
||||
}
|
||||
\`\`\`
|
||||
This is a****groupBy**\* query, which you may be familiar with from SQL. We are grouping, or aggregating, via the **dimensions** field: . We are **filtering** via the **“country”** dimension, to only look at website hits in the US. Our **aggregations** are what we are calculating: a row count, and the sum of the number of known users in our data.
|
||||
```
|
||||
This is a `groupBy` query, which you may be familiar with from SQL. We are grouping, or aggregating, via the `dimensions` field: . We are **filtering** via the `"country"` dimension, to only look at website hits in the US. Our **aggregations** are what we are calculating: a row count, and the sum of the number of known users in our data.
|
||||
|
||||
The result looks something like this:
|
||||
\`\`\`json
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
[version]() “v1”,
|
||||
[timestamp]() “2013-07-18T19:39:00.000Z”,
|
||||
[event]() {
|
||||
[timezone]() “America/Chicago”,
|
||||
[known\_users]() 10,
|
||||
[rows]() 15
|
||||
}
|
||||
"version": "v1",
|
||||
"timestamp": "2013-07-18T19:39:00.000Z",
|
||||
"event": { "timezone": "America/Chicago", "known_users": 10, "rows": 15 }
|
||||
},
|
||||
{
|
||||
[version]() “v1”,
|
||||
[timestamp]() “2013-07-18T19:39:00.000Z”,
|
||||
[event]() {
|
||||
[timezone]() “America/Los\_Angeles”,
|
||||
[known\_users]() 0,
|
||||
[rows]() 3
|
||||
}
|
||||
"version": "v1",
|
||||
"timestamp": "2013-07-18T19:39:00.000Z",
|
||||
"event": { "timezone": "America/Los_Angeles", "known_users": 0, "rows": 3 }
|
||||
},
|
||||
…
|
||||
\`\`\`
|
||||
This groupBy query is a bit complicated and we’ll return to it later. For the time being, just make sure you are getting some blocks of data back. If you are having problems, make sure you have [curl](http://curl.haxx.se/) installed. Control+C to break out of the client script.
|
||||
...
|
||||
```
|
||||
|
||||
This groupBy query is a bit complicated and we'll return to it later. For the time being, just make sure you are getting some blocks of data back. If you are having problems, make sure you have [curl](http://curl.haxx.se/) installed. Control+C to break out of the client script.
|
||||
|
||||
h2. Querying Druid
|
||||
|
||||
In your favorite editor, create the file:
|
||||
\<pre\>time\_boundary\_query.body\</pre\>
|
||||
|
||||
```
|
||||
time_boundary_query.body
|
||||
```
|
||||
|
||||
Druid queries are JSON blobs which are relatively painless to create programmatically, but an absolute pain to write by hand. So anyway, we are going to create a Druid query by hand. Add the following to the file you just created:
|
||||
\<pre\><code>
|
||||
|
||||
```
|
||||
{
|
||||
[queryType]() “timeBoundary”,
|
||||
[dataSource]() “webstream”
|
||||
"queryType": "timeBoundary",
|
||||
"dataSource": "webstream"
|
||||
}
|
||||
</code>\</pre\>
|
||||
The ] is one of the simplest Druid queries. To run the query, you can issue:
|
||||
\<pre\><code> curl~~X POST ‘http://localhost:8083/druid/v2/?pretty’ ~~H ‘content-type: application/json’~~d ```` time_boundary_query.body</code></pre>
|
||||
```
|
||||
|
||||
The [TimeBoundaryQuery](TimeBoundaryQuery.html) is one of the simplest Druid queries. To run the query, you can issue:
|
||||
|
||||
```
|
||||
curl -X POST 'http://localhost:8083/druid/v2/?pretty' -H 'content-type: application/json' -d time_boundary_query.body
|
||||
```
|
||||
|
||||
We get something like this JSON back:
|
||||
|
||||
|
|
@ -143,203 +157,151 @@ We get something like this JSON back:
|
|||
As you can probably tell, the result is indicating the maximum and minimum timestamps we've seen thus far (summarized to a minutely granularity). Let's explore a bit further.
|
||||
|
||||
Return to your favorite editor and create the file:
|
||||
<pre>timeseries_query.body</pre>
|
||||
|
||||
```
|
||||
timeseries_query.body
|
||||
```
|
||||
|
||||
We are going to make a slightly more complicated query, the [TimeseriesQuery](TimeseriesQuery.html). Copy and paste the following into the file:
|
||||
<pre><code>
|
||||
|
||||
```json
|
||||
{
|
||||
"queryType": "timeseries",
|
||||
"dataSource": "webstream",
|
||||
"intervals": [
|
||||
"2010-01-01/2020-01-01"
|
||||
],
|
||||
"intervals": [ "2010-01-01/2020-01-01" ],
|
||||
"granularity": "all",
|
||||
"aggregations": [
|
||||
{
|
||||
"type": "count",
|
||||
"name": "rows"
|
||||
},
|
||||
{
|
||||
"type": "doubleSum",
|
||||
"fieldName": "known_users",
|
||||
"name": "known_users"
|
||||
}
|
||||
{ "type": "count", "name": "rows" },
|
||||
{ "type": "doubleSum", "fieldName": "known_users", "name": "known_users" }
|
||||
]
|
||||
}
|
||||
</code></pre>
|
||||
```
|
||||
|
||||
You are probably wondering, what are these [Granularities](Granularities.html) and [Aggregations](Aggregations.html) things? What the query is doing is aggregating some metrics over some span of time.
|
||||
To issue the query and get some results, run the following in your command line:
|
||||
<pre><code>curl -X POST 'http://localhost:8083/druid/v2/?pretty' -H 'content-type: application/json' -d ````timeseries\_query.body</code>
|
||||
|
||||
</pre>
|
||||
```
|
||||
curl -X POST 'http://localhost:8083/druid/v2/?pretty' -H 'content-type: application/json' -d timeseries_query.body
|
||||
```
|
||||
|
||||
Once again, you should get a JSON blob of text back with your results, that looks something like this:
|
||||
|
||||
\`\`\`json
|
||||
```json
|
||||
[
|
||||
{
|
||||
“timestamp” : “2013-07-18T19:39:00.000Z”,
|
||||
“result” : {
|
||||
“known\_users” : 787.0,
|
||||
“rows” : 2004
|
||||
{
|
||||
"timestamp" : "2013-07-18T19:39:00.000Z",
|
||||
"result" : { "known_users" : 787.0, "rows" : 2004 }
|
||||
}
|
||||
}
|
||||
]
|
||||
\`\`\`
|
||||
```
|
||||
|
||||
If you issue the query again, you should notice your results updating.
|
||||
|
||||
Right now all the results you are getting back are being aggregated into a single timestamp bucket. What if we wanted to see our aggregations on a per minute basis? What field can we change in the query to accomplish this?
|
||||
|
||||
If you loudly exclaimed “we can change granularity to minute”, you are absolutely correct! We can specify different granularities to bucket our results, like so:
|
||||
If you loudly exclaimed "we can change granularity to minute", you are absolutely correct! We can specify different granularities to bucket our results, like so:
|
||||
|
||||
<code>
|
||||
```json
|
||||
{
|
||||
"queryType": "timeseries",
|
||||
"dataSource": "webstream",
|
||||
"intervals": [
|
||||
"2010-01-01/2020-01-01"
|
||||
],
|
||||
"intervals": [ "2010-01-01/2020-01-01" ],
|
||||
"granularity": "minute",
|
||||
"aggregations": [
|
||||
{
|
||||
"type": "count",
|
||||
"name": "rows"
|
||||
},
|
||||
{
|
||||
"type": "doubleSum",
|
||||
"fieldName": "known_users",
|
||||
"name": "known_users"
|
||||
}
|
||||
{ "type": "count", "name": "rows" },
|
||||
{ "type": "doubleSum", "fieldName": "known_users", "name": "known_users" }
|
||||
]
|
||||
}
|
||||
</code>
|
||||
```
|
||||
|
||||
This gives us something like the following:
|
||||
|
||||
\`\`\`json
|
||||
```json
|
||||
[
|
||||
{
|
||||
[timestamp]() “2013-07-18T19:39:00.000Z”,
|
||||
[result]() {
|
||||
[known\_users]() 33,
|
||||
[rows]() 76
|
||||
}
|
||||
"timestamp": "2013-07-18T19:39:00.000Z",
|
||||
"result": { "known_users": 33, "rows": 76 }
|
||||
},
|
||||
{
|
||||
[timestamp]() “2013-07-18T19:40:00.000Z”,
|
||||
[result]() {
|
||||
[known\_users]() 105,
|
||||
[rows]() 221
|
||||
}
|
||||
"timestamp": "2013-07-18T19:40:00.000Z",
|
||||
"result": { "known_users": 105, "rows": 221 }
|
||||
},
|
||||
{
|
||||
[timestamp]() “2013-07-18T19:41:00.000Z”,
|
||||
[result]() {
|
||||
[known\_users]() 53,
|
||||
[rows]() 167
|
||||
}
|
||||
"timestamp": "2013-07-18T19:41:00.000Z",
|
||||
"result": { "known_users": 53, "rows": 167 }
|
||||
},
|
||||
…
|
||||
\`\`\`
|
||||
...
|
||||
```
|
||||
|
||||
Solving a Problem
|
||||
-----------------
|
||||
|
||||
One of Druid’s main powers is to provide answers to problems, so let’s pose a problem. What if we wanted to know what the top states in the US are, ordered by the number of visits by known users over the last few minutes? To solve this problem, we have to return to the query we introduced at the very beginning of this tutorial, the [GroupByQuery](GroupByQuery.html). It would be nice if we could group by results by dimension value and somehow sort those results… and it turns out we can!
|
||||
One of Druid's main powers is to provide answers to problems, so let's pose a problem. What if we wanted to know what the top states in the US are, ordered by the number of visits by known users over the last few minutes? To solve this problem, we have to return to the query we introduced at the very beginning of this tutorial, the [GroupByQuery](GroupByQuery.html). It would be nice if we could group by results by dimension value and somehow sort those results… and it turns out we can!
|
||||
|
||||
Let’s create the file:
|
||||
Let's create the file:
|
||||
|
||||
group_by_query.body</pre>
|
||||
and put the following in there:
|
||||
<pre><code>
|
||||
{
|
||||
```
|
||||
group_by_query.body
|
||||
```
|
||||
|
||||
and put the following in there:
|
||||
|
||||
```
|
||||
{
|
||||
"queryType": "groupBy",
|
||||
"dataSource": "webstream",
|
||||
"granularity": "all",
|
||||
"dimensions": [
|
||||
"geo_region"
|
||||
],
|
||||
"dimensions": [ "geo_region" ],
|
||||
"orderBy": {
|
||||
"type": "doc_page",
|
||||
"type": "default",
|
||||
"columns": [
|
||||
{
|
||||
"dimension": "known_users",
|
||||
"direction": "DESCENDING"
|
||||
}
|
||||
{ "dimension": "known_users", "direction": "DESCENDING" }
|
||||
],
|
||||
"limit": 10
|
||||
},
|
||||
"aggregations": [
|
||||
{
|
||||
"type": "count",
|
||||
"name": "rows"
|
||||
},
|
||||
{
|
||||
"type": "doubleSum",
|
||||
"fieldName": "known_users",
|
||||
"name": "known_users"
|
||||
}
|
||||
{ "type": "count", "name": "rows" },
|
||||
{ "type": "doubleSum", "fieldName": "known_users", "name": "known_users" }
|
||||
],
|
||||
"filter": {
|
||||
"type": "selector",
|
||||
"dimension": "country",
|
||||
"value": "US"
|
||||
},
|
||||
"intervals": [
|
||||
"2012-10-01T00:00/2020-01-01T00"
|
||||
]
|
||||
}
|
||||
</code>
|
||||
"filter": { "type": "selector", "dimension": "country", "value": "US" },
|
||||
"intervals": [ "2012-10-01T00:00/2020-01-01T00" ]
|
||||
}
|
||||
```
|
||||
|
||||
Woah! Our query just got a way more complicated. Now we have these [Filters](Filters.html) things and this [OrderBy](OrderBy.html) thing. Fear not, it turns out the new objects we’ve introduced to our query can help define the format of our results and provide an answer to our question.
|
||||
Woah! Our query just got a way more complicated. Now we have these [Filters](Filters.html) things and this [OrderBy](OrderBy.html) thing. Fear not, it turns out the new objects we've introduced to our query can help define the format of our results and provide an answer to our question.
|
||||
|
||||
If you issue the query:
|
||||
|
||||
<code>curl -X POST 'http://localhost:8083/druid/v2/?pretty' -H 'content-type: application/json' -d @group_by_query.body</code>
|
||||
```
|
||||
curl -X POST 'http://localhost:8083/druid/v2/?pretty' -H 'content-type: application/json' -d @group_by_query.body
|
||||
```
|
||||
|
||||
You should see an answer to our question. For my stream, it looks like this:
|
||||
|
||||
\`\`\`json
|
||||
```json
|
||||
[
|
||||
{
|
||||
[version]() “v1”,
|
||||
[timestamp]() “2012-10-01T00:00:00.000Z”,
|
||||
[event]() {
|
||||
[geo\_region]() “RI”,
|
||||
[known\_users]() 359,
|
||||
[rows]() 143
|
||||
}
|
||||
"version": "v1",
|
||||
"timestamp": "2012-10-01T00:00:00.000Z",
|
||||
"event": { "geo_region": "RI", "known_users": 359, "rows": 143 }
|
||||
},
|
||||
{
|
||||
[version]() “v1”,
|
||||
[timestamp]() “2012-10-01T00:00:00.000Z”,
|
||||
[event]() {
|
||||
[geo\_region]() “NY”,
|
||||
[known\_users]() 187,
|
||||
[rows]() 322
|
||||
}
|
||||
"version": "v1",
|
||||
"timestamp": "2012-10-01T00:00:00.000Z",
|
||||
"event": { "geo_region": "NY", "known_users": 187, "rows": 322 }
|
||||
},
|
||||
{
|
||||
[version]() “v1”,
|
||||
[timestamp]() “2012-10-01T00:00:00.000Z”,
|
||||
[event]() {
|
||||
[geo\_region]() “CA”,
|
||||
[known\_users]() 145,
|
||||
[rows]() 466
|
||||
}
|
||||
"version": "v1",
|
||||
"timestamp": "2012-10-01T00:00:00.000Z",
|
||||
"event": { "geo_region": "CA", "known_users": 145, "rows": 466 }
|
||||
},
|
||||
{
|
||||
[version]() “v1”,
|
||||
[timestamp]() “2012-10-01T00:00:00.000Z”,
|
||||
[event]() {
|
||||
[geo\_region]() “IL”,
|
||||
[known\_users]() 121,
|
||||
[rows]() 185
|
||||
}
|
||||
"version": "v1",
|
||||
"timestamp": "2012-10-01T00:00:00.000Z",
|
||||
"event": { "geo_region": "IL", "known_users": 121, "rows": 185 }
|
||||
},
|
||||
…
|
||||
\`\`\`
|
||||
...
|
||||
```
|
||||
|
||||
Feel free to tweak other query parameters to answer other questions you may have about the data.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,333 +0,0 @@
|
|||
---
|
||||
layout: doc_page
|
||||
---
|
||||
Greetings! We see you’ve taken an interest in Druid. That’s awesome! Hopefully this tutorial will help clarify some core Druid concepts. We will go through one of the Real-time [Examples](Examples.html), and issue some basic Druid queries. The data source we’ll be working with is the [Twitter spritzer stream](https://dev.twitter.com/docs/streaming-apis/streams/public). If you are ready to explore Druid, brave its challenges, and maybe learn a thing or two, read on!
|
||||
|
||||
Setting Up
|
||||
----------
|
||||
|
||||
There are two ways to setup Druid: download a tarball, or build it from source.
|
||||
|
||||
### Download a Tarball
|
||||
|
||||
We’ve built a tarball that contains everything you’ll need. You’ll find it [here](http://static.druid.io/data/examples/druid-services-0.4.6.tar.gz).
|
||||
Download this bad boy to a directory of your choosing.
|
||||
|
||||
You can extract the awesomeness within by issuing:
|
||||
|
||||
tar -zxvf druid-services-0.4.6.tar.gz
|
||||
|
||||
Not too lost so far right? That’s great! If you cd into the directory:
|
||||
|
||||
cd druid-services-0.4.6-SNAPSHOT
|
||||
|
||||
You should see a bunch of files:
|
||||
|
||||
* run_example_server.sh
|
||||
* run_example_client.sh
|
||||
* LICENSE, config, examples, lib directories
|
||||
|
||||
### Clone and Build from Source
|
||||
|
||||
The other way to setup Druid is from source via git. To do so, run these commands:
|
||||
|
||||
```
|
||||
git clone git@github.com:metamx/druid.git
|
||||
cd druid
|
||||
git checkout druid-0.4.32-branch
|
||||
./build.sh
|
||||
```
|
||||
|
||||
You should see a bunch of files:
|
||||
|
||||
```
|
||||
DruidCorporateCLA.pdf README common examples indexer pom.xml server
|
||||
DruidIndividualCLA.pdf build.sh doc group_by.body install publications services
|
||||
LICENSE client eclipse_formatting.xml index-common merger realtime
|
||||
```
|
||||
|
||||
You can find the example executables in the examples/bin directory:
|
||||
* run_example_server.sh
|
||||
* run_example_client.sh
|
||||
|
||||
Running Example Scripts
|
||||
-----------------------
|
||||
|
||||
Let’s start doing stuff. You can start a Druid [Realtime](Realtime.html) node by issuing:
|
||||
|
||||
./run_example_server.sh
|
||||
|
||||
Select “twitter”.
|
||||
|
||||
You’ll need to register a new application with the twitter API, which only takes a minute. Go to [https://twitter.com/oauth_clients/new](https://twitter.com/oauth_clients/new) and fill out the form and submit. Don’t worry, the home page and callback url can be anything. This will generate keys for the Twitter example application. Take note of the values for consumer key/secret and access token/secret.
|
||||
|
||||
Enter your credentials when prompted.
|
||||
|
||||
Once the node starts up you will see a bunch of logs about setting up properties and connecting to the data source. If everything was successful, you should see messages of the form shown below. If you see crazy exceptions, you probably typed in your login information incorrectly.
|
||||
|
||||
```
|
||||
2013-05-17 23:04:40,934 INFO [main] org.mortbay.log - Started SelectChannelConnector@0.0.0.0:8080
|
||||
2013-05-17 23:04:40,935 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.druid.http.FileRequestLogger.start()] on object[com.metamx.druid.http.FileRequestLogger@42bb0406].
|
||||
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Connection established.
|
||||
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] druid.examples.twitter.TwitterSpritzerFirehoseFactory - Connected_to_Twitter
|
||||
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Receiving status stream.
|
||||
```
|
||||
|
||||
Periodically, you’ll also see messages of the form:
|
||||
|
||||
```
|
||||
2013-05-17 23:04:59,793 INFO [chief-twitterstream] druid.examples.twitter.TwitterSpritzerFirehoseFactory - nextRow() has returned 1,000 InputRows
|
||||
```
|
||||
|
||||
These messages indicate you are ingesting events. The Druid real time-node ingests events in an in-memory buffer. Periodically, these events will be persisted to disk. Persisting to disk generates a whole bunch of logs:
|
||||
|
||||
```
|
||||
2013-05-17 23:06:40,918 INFO [chief-twitterstream] com.metamx.druid.realtime.plumber.RealtimePlumberSchool - Submitting persist runnable for dataSource[twitterstream]
|
||||
2013-05-17 23:06:40,920 INFO [twitterstream-incremental-persist] com.metamx.druid.realtime.plumber.RealtimePlumberSchool - DataSource[twitterstream], Interval[2013-05-17T23:00:00.000Z/2013-05-18T00:00:00.000Z], persisting Hydrant[FireHydrant{index=com.metamx.druid.index.v1.IncrementalIndex@126212dd, queryable=com.metamx.druid.index.IncrementalIndexSegment@64c47498, count=0}]
|
||||
2013-05-17 23:06:40,937 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Starting persist for interval[2013-05-17T23:00:00.000Z/2013-05-17T23:07:00.000Z], rows[4,666]
|
||||
2013-05-17 23:06:41,039 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - outDir[/tmp/example/twitter_realtime/basePersist/twitterstream/2013-05-17T23:00:00.000Z_2013-05-18T00:00:00.000Z/0/v8-tmp] completed index.drd in 11 millis.
|
||||
2013-05-17 23:06:41,070 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - outDir[/tmp/example/twitter_realtime/basePersist/twitterstream/2013-05-17T23:00:00.000Z_2013-05-18T00:00:00.000Z/0/v8-tmp] completed dim conversions in 31 millis.
|
||||
2013-05-17 23:06:41,275 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.CompressedPools - Allocating new chunkEncoder[1]
|
||||
2013-05-17 23:06:41,332 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - outDir[/tmp/example/twitter_realtime/basePersist/twitterstream/2013-05-17T23:00:00.000Z_2013-05-18T00:00:00.000Z/0/v8-tmp] completed walk through of 4,666 rows in 262 millis.
|
||||
2013-05-17 23:06:41,334 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Starting dimension[htags] with cardinality[634]
|
||||
2013-05-17 23:06:41,381 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Completed dimension[htags] in 49 millis.
|
||||
2013-05-17 23:06:41,382 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Starting dimension[lang] with cardinality[19]
|
||||
2013-05-17 23:06:41,398 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Completed dimension[lang] in 17 millis.
|
||||
2013-05-17 23:06:41,398 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Starting dimension[utc_offset] with cardinality[32]
|
||||
2013-05-17 23:06:41,413 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - Completed dimension[utc_offset] in 15 millis.
|
||||
2013-05-17 23:06:41,413 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexMerger - outDir[/tmp/example/twitter_realtime/basePersist/twitterstream/2013-05-17T23:00:00.000Z_2013-05-18T00:00:00.000Z/0/v8-tmp] completed inverted.drd in 81 millis.
|
||||
2013-05-17 23:06:41,425 INFO [twitterstream-incremental-persist] com.metamx.druid.index.v1.IndexIO$DefaultIndexIOHandler - Converting v8[/tmp/example/twitter_realtime/basePersist/twitterstream/2013-05-17T23:00:00.000Z_2013-05-18T00:00:00.000Z/0/v8-tmp] to v9[/tmp/example/twitter_realtime/basePersist/twitterstream/2013-05-17T23:00:00.000Z_2013-05-18T00:00:00.000Z/0]
|
||||
2013-05-17 23:06:41,426 INFO [twitterstream-incremental-persist]
|
||||
... ETC
|
||||
```
|
||||
|
||||
Select “twitter” once again. This script issues [GroupByQuery]:(./GroupByQuery.html)s to the twitter data we’ve been ingesting. The query looks like this:
|
||||
|
||||
```json
|
||||
{
|
||||
"queryType": "groupBy",
|
||||
"dataSource": "twitterstream",
|
||||
"granularity": "all",
|
||||
"dimensions": ["lang", "utc_offset"],
|
||||
"aggregations":[
|
||||
{ "type": "count", "name": "rows"},
|
||||
{ "type": "doubleSum", "fieldName": "tweets", "name": "tweets"}
|
||||
],
|
||||
"filter": { "type": "selector", "dimension": "lang", "value": "en" },
|
||||
"intervals":["2012-10-01T00:00/2020-01-01T00"]
|
||||
}
|
||||
```
|
||||
|
||||
This is a **groupBy** query, which you may be familiar with from SQL. We are grouping, or aggregating, via the **dimensions** field: . We are **filtering** via the **“lang”** dimension, to only look at english tweets. Our **aggregations** are what we are calculating: a row count, and the sum of the tweets in our data.
|
||||
The result looks something like this:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"version": "v1",
|
||||
"timestamp": "2012-10-01T00:00:00.000Z",
|
||||
"event": {
|
||||
"utc_offset": "-10800",
|
||||
"tweets": 90,
|
||||
"lang": "en",
|
||||
"rows": 81
|
||||
}
|
||||
},
|
||||
{
|
||||
"version": "v1",
|
||||
"timestamp": "2012-10-01T00:00:00.000Z",
|
||||
"event": {
|
||||
"utc_offset": "-14400",
|
||||
"tweets": 177,
|
||||
"lang": "en",
|
||||
"rows": 154
|
||||
}
|
||||
},
|
||||
...
|
||||
```
|
||||
|
||||
This data, plotted in a time series/distribution, looks something like this:
|
||||
|
||||
This groupBy query is a bit complicated and we’ll return to it later. For the time being, just make sure you are getting some blocks of data back. If you are having problems, make sure you have [curl](http://curl.haxx.se/) installed. Control+C to break out of the client script.
|
||||
h2. Querying Druid
|
||||
In your favorite editor, create the file:
|
||||
|
||||
`time_boundary_query.body`
|
||||
|
||||
Druid queries are JSON blobs which are relatively painless to create programmatically, but an absolute pain to write by hand. So anyway, we are going to create a Druid query by hand. Add the following to the file you just created:
|
||||
|
||||
\<pre\><code>
|
||||
</code>\</pre\>
|
||||
The ] is one of the simplest Druid queries. To run the query, you can issue:
|
||||
\<pre\><code> curl~~X POST ‘http://localhost:8080/druid/v2/?pretty’ ~~H ‘content-type: application/json’~~d ```` time_boundary_query.body</code></pre>
|
||||
|
||||
We get something like this JSON back:
|
||||
|
||||
```json
|
||||
[ {
|
||||
"timestamp" : "2013-06-10T19:09:00.000Z",
|
||||
"result" : {
|
||||
"minTime" : "2013-06-10T19:09:00.000Z",
|
||||
"maxTime" : "2013-06-10T20:50:00.000Z"
|
||||
}
|
||||
} ]
|
||||
```
|
||||
That's the result. What information do you think the result is conveying?
|
||||
...
|
||||
If you said the result is indicating the maximum and minimum timestamps we've seen thus far (summarized to a minutely granularity), you are absolutely correct. I can see you are a person legitimately interested in learning about Druid. Let's explore a bit further.
|
||||
|
||||
Return to your favorite editor and create the file:
|
||||
<pre>timeseries_query.body</pre>
|
||||
|
||||
We are going to make a slightly more complicated query, the [TimeseriesQuery](TimeseriesQuery.html). Copy and paste the following into the file:
|
||||
<pre><code>{
|
||||
"queryType":"timeseries",
|
||||
"dataSource":"twitterstream",
|
||||
"intervals":["2010-01-01/2020-01-01"],
|
||||
"granularity":"all",
|
||||
"aggregations":[
|
||||
{ "type": "count", "name": "rows"},
|
||||
{ "type": "doubleSum", "fieldName": "tweets", "name": "tweets"}
|
||||
]
|
||||
}
|
||||
</code></pre>
|
||||
|
||||
You are probably wondering, what are these [Granularities](Granularities.html) and [Aggregations](Aggregations.html) things? What the query is doing is aggregating some metrics over some span of time.
|
||||
To issue the query and get some results, run the following in your command line:
|
||||
<pre><code>curl -X POST 'http://localhost:8080/druid/v2/?pretty' -H 'content-type: application/json' -d ````timeseries_query.body</code>
|
||||
|
||||
</pre>
|
||||
Once again, you should get a JSON blob of text back with your results, that looks something like this:
|
||||
|
||||
```json
|
||||
[ {
|
||||
“timestamp” : “2013-06-10T19:09:00.000Z”,
|
||||
“result” : {
|
||||
“tweets” : 358562.0,
|
||||
“rows” : 272271
|
||||
}
|
||||
} ]
|
||||
```
|
||||
|
||||
If you issue the query again, you should notice your results updating.
|
||||
|
||||
Right now all the results you are getting back are being aggregated into a single timestamp bucket. What if we wanted to see our aggregations on a per minute basis? What field can we change in the query to accomplish this?
|
||||
|
||||
If you loudly exclaimed “we can change granularity to minute”, you are absolutely correct again! We can specify different granularities to bucket our results, like so:
|
||||
|
||||
```json
|
||||
{
|
||||
[queryType]("timeseries"),
|
||||
[dataSource]("twitterstream"),
|
||||
[intervals](["2010-01-01/2020-01-01"]),
|
||||
[granularity]("minute"),
|
||||
[aggregations]([)
|
||||
{ [type]() “count”, [name]() “rows”},
|
||||
{ [type]() “doubleSum”, [fieldName]() “tweets”, [name]() “tweets”}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This gives us something like the following:
|
||||
|
||||
```json
|
||||
[ {
|
||||
“timestamp” : “2013-06-10T19:09:00.000Z”,
|
||||
“result” : {
|
||||
“tweets” : 2650.0,
|
||||
“rows” : 2120
|
||||
}
|
||||
}, {
|
||||
“timestamp” : “2013-06-10T19:10:00.000Z”,
|
||||
“result” : {
|
||||
“tweets” : 3401.0,
|
||||
“rows” : 2609
|
||||
}
|
||||
}, {
|
||||
“timestamp” : “2013-06-10T19:11:00.000Z”,
|
||||
“result” : {
|
||||
“tweets” : 3472.0,
|
||||
“rows” : 2610
|
||||
}
|
||||
},
|
||||
…
|
||||
```
|
||||
|
||||
Solving a Problem
|
||||
-----------------
|
||||
|
||||
One of Druid’s main powers (see what we did there?) is to provide answers to problems, so let’s pose a problem. What if we wanted to know what the top hash tags are, ordered by the number tweets, where the language is english, over the last few minutes you’ve been reading this tutorial? To solve this problem, we have to return to the query we introduced at the very beginning of this tutorial, the [GroupByQuery](GroupByQuery.html). It would be nice if we could group by results by dimension value and somehow sort those results… and it turns out we can!
|
||||
|
||||
Let’s create the file:
|
||||
|
||||
group_by_query.body</pre>
|
||||
and put the following in there:
|
||||
<pre><code>{
|
||||
"queryType": "groupBy",
|
||||
"dataSource": "twitterstream",
|
||||
"granularity": "all",
|
||||
"dimensions": ["htags"],
|
||||
"orderBy": {"type":"doc_page", "columns":[{"dimension": "tweets", "direction":"DESCENDING"}], "limit":5},
|
||||
"aggregations":[
|
||||
{ "type": "longSum", "fieldName": "tweets", "name": "tweets"}
|
||||
],
|
||||
"filter": {"type": "selector", "dimension": "lang", "value": "en" },
|
||||
"intervals":["2012-10-01T00:00/2020-01-01T00"]
|
||||
}
|
||||
</code>
|
||||
|
||||
Woah! Our query just got a way more complicated. Now we have these [Filters](Filters.html) things and this [OrderBy](OrderBy.html) thing. Fear not, it turns out the new objects we’ve introduced to our query can help define the format of our results and provide an answer to our question.
|
||||
|
||||
If you issue the query:
|
||||
|
||||
<code>curl -X POST 'http://localhost:8080/druid/v2/?pretty' -H 'content-type: application/json' -d @group_by_query.body</code>
|
||||
|
||||
You should hopefully see an answer to our question. For my twitter stream, it looks like this:
|
||||
|
||||
```json
|
||||
[ {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-10-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“tweets” : 2660,
|
||||
“htags” : “android”
|
||||
}
|
||||
}, {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-10-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“tweets” : 1944,
|
||||
“htags” : “E3”
|
||||
}
|
||||
}, {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-10-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“tweets” : 1927,
|
||||
“htags” : “15SueñosPendientes”
|
||||
}
|
||||
}, {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-10-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“tweets” : 1717,
|
||||
“htags” : “ipad”
|
||||
}
|
||||
}, {
|
||||
“version” : “v1”,
|
||||
“timestamp” : “2012-10-01T00:00:00.000Z”,
|
||||
“event” : {
|
||||
“tweets” : 1515,
|
||||
“htags” : “IDidntTextYouBackBecause”
|
||||
}
|
||||
} ]
|
||||
```
|
||||
|
||||
Feel free to tweak other query parameters to answer other questions you may have about the data.
|
||||
|
||||
Additional Information
|
||||
----------------------
|
||||
|
||||
This tutorial is merely showcasing a small fraction of what Druid can do. Next, continue on to [Loading Your Data](Loading Your Data.html).
|
||||
|
||||
And thus concludes our journey! Hopefully you learned a thing or two about Druid real-time ingestion, querying Druid, and how Druid can be used to solve problems. If you have additional questions, feel free to post in our [google groups page](http://www.groups.google.com/forum/#!forum/druid-development).
|
||||
|
|
@ -8,13 +8,13 @@ Versioning Strategy
|
|||
|
||||
We generally follow [semantic versioning](http://semver.org/). The general idea is
|
||||
|
||||
- “Major” version (leftmost): backwards incompatible, no guarantees exist about APIs between the versions
|
||||
- “Minor” version (middle number): you can move forward from a smaller number to a larger number, but moving backwards *might* be incompatible.
|
||||
- “bug-fix” version (“patch” or the rightmost): Interchangeable. The higher the number, the more things are fixed (hopefully), but the programming interfaces are completely compatible and you should be able to just drop in a new jar and have it work.
|
||||
* "Major" version (leftmost): backwards incompatible, no guarantees exist about APIs between the versions
|
||||
* "Minor" version (middle number): you can move forward from a smaller number to a larger number, but moving backwards *might* be incompatible.
|
||||
* "bug-fix" version ("patch" or the rightmost): Interchangeable. The higher the number, the more things are fixed (hopefully), but the programming interfaces are completely compatible and you should be able to just drop in a new jar and have it work.
|
||||
|
||||
Note that this is defined in terms of programming API, **not** in terms of functionality. It is possible that a brand new awesome way of doing something is introduced in a “bug-fix” release version if it doesn’t add to the public API or change it.
|
||||
Note that this is defined in terms of programming API, **not** in terms of functionality. It is possible that a brand new awesome way of doing something is introduced in a "bug-fix" release version if it doesn’t add to the public API or change it.
|
||||
|
||||
One exception for right now, while we are still in major version 0, we are considering the APIs to be in beta and are conflating “major” and “minor” so a minor version increase could be backwards incompatible for as long as we are at major version 0. These will be communicated via email on the group.
|
||||
One exception for right now, while we are still in major version 0, we are considering the APIs to be in beta and are conflating "major" and "minor" so a minor version increase could be backwards incompatible for as long as we are at major version 0. These will be communicated via email on the group.
|
||||
|
||||
For external deployments, we recommend running the stable release tag. Releases are considered stable after we have deployed them into our production environment and they have operated bug-free for some time.
|
||||
|
||||
|
|
|
|||
|
|
@ -4,54 +4,68 @@ layout: doc_page
|
|||
Druid uses ZooKeeper (ZK) for management of current cluster state. The operations that happen over ZK are
|
||||
|
||||
1. [Master](Master.html) leader election
|
||||
2. Segment “publishing” protocol from [Compute](Compute.html) and [Realtime](Realtime.html)
|
||||
2. Segment "publishing" protocol from [Compute](Compute.html) and [Realtime](Realtime.html)
|
||||
3. Segment load/drop protocol between [Master](Master.html) and [Compute](Compute.html)
|
||||
|
||||
### Property Configuration
|
||||
|
||||
ZooKeeper paths are set via the `runtime.properties` configuration file. Druid will automatically create paths that do not exist, so typos in config files is a very easy way to become split-brained.
|
||||
|
||||
There is a prefix path that is required and can be used as the only (well, kinda, see the note below) path-related zookeeper configuration parameter (everything else will be a doc_page based on the prefix):
|
||||
There is a prefix path that is required and can be used as the only (well, kinda, see the note below) path-related zookeeper configuration parameter (everything else will be a default based on the prefix):
|
||||
|
||||
druid.zk.paths.base
|
||||
```
|
||||
druid.zk.paths.base
|
||||
```
|
||||
|
||||
You can also override each individual path (doc_pages are shown below):
|
||||
You can also override each individual path (defaults are shown below):
|
||||
|
||||
druid.zk.paths.propertiesPath=${druid.zk.paths.base}/properties
|
||||
druid.zk.paths.announcementsPath=${druid.zk.paths.base}/announcements
|
||||
druid.zk.paths.servedSegmentsPath=${druid.zk.paths.base}/servedSegments
|
||||
druid.zk.paths.loadQueuePath=${druid.zk.paths.base}/loadQueue
|
||||
druid.zk.paths.masterPath=${druid.zk.paths.base}/master
|
||||
druid.zk.paths.indexer.announcementsPath=${druid.zk.paths.base}/indexer/announcements
|
||||
druid.zk.paths.indexer.tasksPath=${druid.zk.paths.base}/indexer/tasks
|
||||
druid.zk.paths.indexer.statusPath=${druid.zk.paths.base}/indexer/status
|
||||
druid.zk.paths.indexer.leaderLatchPath=${druid.zk.paths.base}/indexer/leaderLatchPath
|
||||
```
|
||||
druid.zk.paths.propertiesPath=${druid.zk.paths.base}/properties
|
||||
druid.zk.paths.announcementsPath=${druid.zk.paths.base}/announcements
|
||||
druid.zk.paths.servedSegmentsPath=${druid.zk.paths.base}/servedSegments
|
||||
druid.zk.paths.loadQueuePath=${druid.zk.paths.base}/loadQueue
|
||||
druid.zk.paths.masterPath=${druid.zk.paths.base}/master
|
||||
druid.zk.paths.indexer.announcementsPath=${druid.zk.paths.base}/indexer/announcements
|
||||
druid.zk.paths.indexer.tasksPath=${druid.zk.paths.base}/indexer/tasks
|
||||
druid.zk.paths.indexer.statusPath=${druid.zk.paths.base}/indexer/status
|
||||
druid.zk.paths.indexer.leaderLatchPath=${druid.zk.paths.base}/indexer/leaderLatchPath
|
||||
```
|
||||
|
||||
NOTE: We also use Curator’s service discovery module to expose some services via zookeeper. This also uses a zookeeper path, but this path is **not** affected by `druid.zk.paths.base` and **must** be specified separately. This property is
|
||||
|
||||
druid.zk.paths.discoveryPath
|
||||
```
|
||||
druid.zk.paths.discoveryPath
|
||||
```
|
||||
|
||||
### Master Leader Election
|
||||
|
||||
We use the Curator LeadershipLatch recipe to do leader election at path
|
||||
|
||||
${druid.zk.paths.masterPath}/_MASTER
|
||||
```
|
||||
${druid.zk.paths.masterPath}/_MASTER
|
||||
```
|
||||
|
||||
### Segment “publishing” protocol from Compute and Realtime
|
||||
### Segment "publishing" protocol from Compute and Realtime
|
||||
|
||||
The `announcementsPath` and `servedSegmentsPath` are used for this.
|
||||
|
||||
All [Compute](Compute.html) and [Realtime](Realtime.html) nodes publish themselves on the `announcementsPath`, specifically, they will create an ephemeral znode at
|
||||
|
||||
${druid.zk.paths.announcementsPath}/${druid.host}
|
||||
```
|
||||
${druid.zk.paths.announcementsPath}/${druid.host}
|
||||
```
|
||||
|
||||
Which signifies that they exist. They will also subsequently create a permanent znode at
|
||||
|
||||
${druid.zk.paths.servedSegmentsPath}/${druid.host}
|
||||
```
|
||||
${druid.zk.paths.servedSegmentsPath}/${druid.host}
|
||||
```
|
||||
|
||||
And as they load up segments, they will attach ephemeral znodes that look like
|
||||
|
||||
${druid.zk.paths.servedSegmentsPath}/${druid.host}/_segment_identifier_
|
||||
```
|
||||
${druid.zk.paths.servedSegmentsPath}/${druid.host}/_segment_identifier_
|
||||
```
|
||||
|
||||
Nodes like the [Master](Master.html) and [Broker](Broker.html) can then watch these paths to see which nodes are currently serving which segments.
|
||||
|
||||
|
|
@ -61,6 +75,8 @@ The `loadQueuePath` is used for this.
|
|||
|
||||
When the [Master](Master.html) decides that a [Compute](Compute.html) node should load or drop a segment, it writes an ephemeral znode to
|
||||
|
||||
${druid.zk.paths.loadQueuePath}/_host_of_compute_node/_segment_identifier
|
||||
```
|
||||
${druid.zk.paths.loadQueuePath}/_host_of_compute_node/_segment_identifier
|
||||
```
|
||||
|
||||
This node will contain a payload that indicates to the Compute node what it should do with the given segment. When the Compute node is done with the work, it will delete the znode in order to signify to the Master that it is complete.
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
}
|
||||
|
||||
.doc-content p {
|
||||
padding: 8px 0 8px 0;
|
||||
margin: 18px 0 18px 0;
|
||||
}
|
||||
|
||||
/*** HACK: This is a horrible hack, but I have no clue why images don't want to stay in the container **/
|
||||
|
|
@ -21,6 +21,11 @@
|
|||
background-color: transparent;
|
||||
}
|
||||
|
||||
.doc-content table,
|
||||
.doc-content pre {
|
||||
margin: 35px 0 35px 0;
|
||||
}
|
||||
|
||||
.doc-content table,
|
||||
.doc-content table > thead > tr > th,
|
||||
.doc-content table > tbody > tr > th,
|
||||
|
|
|
|||
Loading…
Reference in New Issue