mirror of https://github.com/apache/druid.git
Remove Tranquility configs and certain doc references. (#8793)
Since it hasn't received updates or community interest in a while, it makes sense to de-emphasize it in the distribution and most documentation (outside of simple mentions of its existence).
This commit is contained in:
Gian Merlino
Fangjin Yang
parent
c922d2c3c9
commit
7605c23354
|
|
@ -31,7 +31,7 @@ A recommended way of organizing Druid configuration files can be seen in the `co
|
|||
|
||||
```
|
||||
$ ls -R conf
|
||||
druid tranquility
|
||||
druid
|
||||
|
||||
conf/druid:
|
||||
_common broker coordinator historical middleManager overlord
|
||||
|
|
@ -53,9 +53,6 @@ jvm.config runtime.properties
|
|||
|
||||
conf/druid/overlord:
|
||||
jvm.config runtime.properties
|
||||
|
||||
conf/tranquility:
|
||||
kafka.json server.json
|
||||
```
|
||||
|
||||
Each directory has a `runtime.properties` file containing configuration properties for the specific Druid process corresponding to the directory (e.g., `historical`).
|
||||
|
|
|
|||
|
|
@ -43,47 +43,7 @@ If you plan to read LZO-compressed Thrift files, you will need to download versi
|
|||
| thriftJar | String | path of thrift jar, if not provided, it will try to find the thrift class in classpath. Thrift jar in batch ingestion should be uploaded to HDFS first and configure `jobProperties` with `"tmpjars":"/path/to/your/thrift.jar"` | no |
|
||||
| thriftClass | String | classname of thrift | yes |
|
||||
|
||||
- Realtime Ingestion (tranquility example)
|
||||
|
||||
```json
|
||||
{
|
||||
"dataSources": [{
|
||||
"spec": {
|
||||
"dataSchema": {
|
||||
"dataSource": "book",
|
||||
"granularitySpec": { },
|
||||
"parser": {
|
||||
"type": "thrift",
|
||||
"thriftClass": "org.apache.druid.data.input.thrift.Book",
|
||||
"protocol": "compact",
|
||||
"parseSpec": {
|
||||
"format": "json",
|
||||
...
|
||||
}
|
||||
},
|
||||
"metricsSpec": [...]
|
||||
},
|
||||
"tuningConfig": {...}
|
||||
},
|
||||
"properties": {...}
|
||||
}],
|
||||
"properties": {...}
|
||||
}
|
||||
```
|
||||
|
||||
To use it with tranquility,
|
||||
|
||||
```bash
|
||||
bin/tranquility kafka \
|
||||
-configFile $jsonConfig \
|
||||
-Ddruid.extensions.directory=/path/to/extensions \
|
||||
-Ddruid.extensions.loadList='["druid-thrift-extensions"]'
|
||||
```
|
||||
|
||||
Hadoop-client is also needed, you may copy all the hadoop-client dependency jars into directory `druid-thrift-extensions` to make is simple.
|
||||
|
||||
|
||||
- Batch Ingestion - `inputFormat` and `tmpjars` should be set.
|
||||
- Batch Ingestion example - `inputFormat` and `tmpjars` should be set.
|
||||
|
||||
This is for batch ingestion using the HadoopDruidIndexer. The inputFormat of inputSpec in ioConfig could be one of `"org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat"` and `com.twitter.elephantbird.mapreduce.input.LzoThriftBlockInputFormat`. Be careful, when `LzoThriftBlockInputFormat` is used, thrift class must be provided twice.
|
||||
|
||||
|
|
|
|||
|
|
@ -26,9 +26,8 @@ sidebar_label: "Apache Kafka"
|
|||
|
||||
The Kafka indexing service enables the configuration of *supervisors* on the Overlord, which facilitate ingestion from
|
||||
Kafka by managing the creation and lifetime of Kafka indexing tasks. These indexing tasks read events using Kafka's own
|
||||
partition and offset mechanism and are therefore able to provide guarantees of exactly-once ingestion. They are also
|
||||
able to read non-recent events from Kafka and are not subject to the window period considerations imposed on other
|
||||
ingestion mechanisms using Tranquility. The supervisor oversees the state of the indexing tasks to coordinate handoffs,
|
||||
partition and offset mechanism and are therefore able to provide guarantees of exactly-once ingestion.
|
||||
The supervisor oversees the state of the indexing tasks to coordinate handoffs,
|
||||
manage failures, and ensure that the scalability and replication requirements are maintained.
|
||||
|
||||
This service is provided in the `druid-kafka-indexing-service` core Apache Druid (incubating) extension (see
|
||||
|
|
|
|||
|
|
@ -26,9 +26,8 @@ sidebar_label: "Amazon Kinesis"
|
|||
|
||||
Similar to the [Kafka indexing service](./kafka-ingestion.md), the Kinesis indexing service enables the configuration of *supervisors* on the Overlord, which facilitate ingestion from
|
||||
Kinesis by managing the creation and lifetime of Kinesis indexing tasks. These indexing tasks read events using Kinesis's own
|
||||
Shards and Sequence Number mechanism and are therefore able to provide guarantees of exactly-once ingestion. They are also
|
||||
able to read non-recent events from Kinesis and are not subject to the window period considerations imposed on other
|
||||
ingestion mechanisms using Tranquility. The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures,
|
||||
Shards and Sequence Number mechanism and are therefore able to provide guarantees of exactly-once ingestion.
|
||||
The supervisor oversees the state of the indexing tasks to coordinate handoffs, manage failures,
|
||||
and ensure that the scalability and replication requirements are maintained.
|
||||
|
||||
The Kinesis indexing service is provided as the `druid-kinesis-indexing-service` core Apache Druid (incubating) extension (see
|
||||
|
|
|
|||
|
|
@ -41,7 +41,7 @@ This example demonstrates how to load Protobuf messages from Kafka. Please read
|
|||
|
||||
Files used in this example are found at `./examples/quickstart/protobuf` in your Druid directory.
|
||||
|
||||
- We will use [Kafka Indexing Service](./kafka-ingestion.md) instead of Tranquility.
|
||||
- We will use [Kafka Indexing Service](./kafka-ingestion.md).
|
||||
- Kafka broker host is `localhost:9092`.
|
||||
- Kafka topic is `metrics_pb` instead of `metrics`.
|
||||
- datasource name is `metrics-kafka-pb` instead of `metrics-kafka` to avoid the confusion.
|
||||
|
|
|
|||
|
|
@ -294,7 +294,7 @@ Once `forceTimeChunkLock` is unset, the task will choose a proper lock type to u
|
|||
Please note that segment lock is not always available. The most common use case where time chunk lock is enforced is
|
||||
when an overwriting task changes the segment granularity.
|
||||
Also, the segment locking is supported by only native indexing tasks and Kafka/Kinesis indexing tasks.
|
||||
The Hadoop indexing tasks and realtime indexing tasks (with [Tranquility](tranquility.md)) don't support it yet.
|
||||
Hadoop indexing tasks and `index_realtime` tasks (used by [Tranquility](tranquility.md)) don't support it yet.
|
||||
|
||||
`forceTimeChunkLock` in the task context is only applied to individual tasks.
|
||||
If you want to unset it for all tasks, you would want to set `druid.indexer.tasklock.forceTimeChunkLock` to false in the [overlord configuration](../configuration/index.html#overlord-operations).
|
||||
|
|
|
|||
|
|
@ -60,13 +60,8 @@ With batch indexing, you can use [single-dimension partitioning](../ingestion/ha
|
|||
to partition your data by tenant_id. Druid always partitions by time first, but the secondary partition within each
|
||||
time bucket will be on tenant_id.
|
||||
|
||||
With realtime indexing, you have a couple of options.
|
||||
|
||||
1. Partition on tenant_id upfront. You'd do this by tweaking the stream you send to Druid. If you're using Kafka then
|
||||
you can have your Kafka producer partition your topic by a hash of tenant_id. If you're using Tranquility then you can
|
||||
define a custom [Partitioner](http://static.druid.io/tranquility/api/latest/#com.metamx.tranquility.partition.Partitioner).
|
||||
2. Reindex your older data periodically. You can do this with the ["dataSource" input spec](../ingestion/hadoop.html#datasource).
|
||||
You can use this in concert with single-dimension partitioning to repartition your data.
|
||||
With realtime indexing, you'd do this by tweaking the stream you send to Druid. For example, if you're using Kafka then
|
||||
you can have your Kafka producer partition your topic by a hash of tenant_id.
|
||||
|
||||
## Customizing data distribution
|
||||
|
||||
|
|
|
|||
|
|
@ -405,9 +405,6 @@ inbound connections on the following:
|
|||
- 8082 (Broker)
|
||||
- 8088 (Router, if used)
|
||||
|
||||
### Other
|
||||
- 8200 (Tranquility Server, if used)
|
||||
|
||||
> In production, we recommend deploying ZooKeeper and your metadata store on their own dedicated hardware,
|
||||
> rather than on the Master server.
|
||||
|
||||
|
|
@ -455,24 +452,6 @@ You can add more Data servers as needed.
|
|||
> For clusters with complex resource allocation needs, you can break apart Historicals and MiddleManagers and scale the components individually.
|
||||
> This also allows you take advantage of Druid's built-in MiddleManager autoscaling facility.
|
||||
|
||||
### Tranquility
|
||||
|
||||
If you are doing push-based stream ingestion with Kafka or over HTTP, you can also start Tranquility Server on the Data server.
|
||||
|
||||
For large scale production, Data server processes and the Tranquility Server can still be co-located.
|
||||
|
||||
If you are running Tranquility (not server) with a stream processor, you can co-locate Tranquility with the stream processor and not require Tranquility Server.
|
||||
|
||||
First install Tranquility:
|
||||
|
||||
```bash
|
||||
curl http://static.druid.io/tranquility/releases/tranquility-distribution-0.8.3.tgz -o tranquility-distribution-0.8.3.tgz
|
||||
tar -xzf tranquility-distribution-0.8.3.tgz
|
||||
mv tranquility-distribution-0.8.3 tranquility
|
||||
```
|
||||
|
||||
Afterwards, in `conf/supervise/cluster/data.conf`, uncomment out the `tranquility-server` line, and restart the Data server processes.
|
||||
|
||||
## Start Query Server
|
||||
|
||||
Copy the Druid distribution and your edited configurations to your Query servers.
|
||||
|
|
|
|||
|
|
@ -2,6 +2,3 @@
|
|||
|
||||
historical bin/run-druid historical conf/druid/cluster/data
|
||||
middleManager bin/run-druid middleManager conf/druid/cluster/data
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -8,6 +8,3 @@ broker bin/run-druid broker conf/druid/single-server/large
|
|||
router bin/run-druid router conf/druid/single-server/large
|
||||
historical bin/run-druid historical conf/druid/single-server/large
|
||||
!p90 middleManager bin/run-druid middleManager conf/druid/single-server/large
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -8,6 +8,3 @@ broker bin/run-druid broker conf/druid/single-server/medium
|
|||
router bin/run-druid router conf/druid/single-server/medium
|
||||
historical bin/run-druid historical conf/druid/single-server/medium
|
||||
!p90 middleManager bin/run-druid middleManager conf/druid/single-server/medium
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -8,6 +8,3 @@ broker bin/run-druid broker conf/druid/single-server/micro-quickstart
|
|||
router bin/run-druid router conf/druid/single-server/micro-quickstart
|
||||
historical bin/run-druid historical conf/druid/single-server/micro-quickstart
|
||||
!p90 middleManager bin/run-druid middleManager conf/druid/single-server/micro-quickstart
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -8,6 +8,3 @@ broker bin/run-druid broker conf/druid/single-server/nano-quickstart
|
|||
router bin/run-druid router conf/druid/single-server/nano-quickstart
|
||||
historical bin/run-druid historical conf/druid/single-server/nano-quickstart
|
||||
!p90 middleManager bin/run-druid middleManager conf/druid/single-server/nano-quickstart
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -8,6 +8,3 @@ broker bin/run-druid broker conf/druid/single-server/small
|
|||
router bin/run-druid router conf/druid/single-server/small
|
||||
historical bin/run-druid historical conf/druid/single-server/small
|
||||
!p90 middleManager bin/run-druid middleManager conf/druid/single-server/small
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -8,6 +8,3 @@ broker bin/run-druid broker conf/druid/single-server/xlarge
|
|||
router bin/run-druid router conf/druid/single-server/xlarge
|
||||
historical bin/run-druid historical conf/druid/single-server/xlarge
|
||||
!p90 middleManager bin/run-druid middleManager conf/druid/single-server/xlarge
|
||||
|
||||
# Uncomment to use Tranquility Server
|
||||
#!p95 tranquility-server tranquility/bin/tranquility server -configFile conf/tranquility/wikipedia-server.json -Ddruid.extensions.loadList=[]
|
||||
|
|
|
|||
|
|
@ -1,76 +0,0 @@
|
|||
{
|
||||
"dataSources" : {
|
||||
"metrics-kafka" : {
|
||||
"spec" : {
|
||||
"dataSchema" : {
|
||||
"dataSource" : "metrics-kafka",
|
||||
"parser" : {
|
||||
"type" : "string",
|
||||
"parseSpec" : {
|
||||
"timestampSpec" : {
|
||||
"column" : "timestamp",
|
||||
"format" : "auto"
|
||||
},
|
||||
"dimensionsSpec" : {
|
||||
"dimensions" : [],
|
||||
"dimensionExclusions" : [
|
||||
"timestamp",
|
||||
"value"
|
||||
]
|
||||
},
|
||||
"format" : "json"
|
||||
}
|
||||
},
|
||||
"granularitySpec" : {
|
||||
"type" : "uniform",
|
||||
"segmentGranularity" : "hour",
|
||||
"queryGranularity" : "none"
|
||||
},
|
||||
"metricsSpec" : [
|
||||
{
|
||||
"type" : "count",
|
||||
"name" : "count"
|
||||
},
|
||||
{
|
||||
"name" : "value_sum",
|
||||
"type" : "doubleSum",
|
||||
"fieldName" : "value"
|
||||
},
|
||||
{
|
||||
"fieldName" : "value",
|
||||
"name" : "value_min",
|
||||
"type" : "doubleMin"
|
||||
},
|
||||
{
|
||||
"type" : "doubleMax",
|
||||
"name" : "value_max",
|
||||
"fieldName" : "value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"ioConfig" : {
|
||||
"type" : "realtime"
|
||||
},
|
||||
"tuningConfig" : {
|
||||
"type" : "realtime",
|
||||
"intermediatePersistPeriod" : "PT10M",
|
||||
"windowPeriod" : "PT10M"
|
||||
}
|
||||
},
|
||||
"properties" : {
|
||||
"task.partitions" : "1",
|
||||
"task.replicants" : "1",
|
||||
"topicPattern" : "metrics"
|
||||
}
|
||||
}
|
||||
},
|
||||
"properties" : {
|
||||
"zookeeper.connect" : "master.example.com",
|
||||
"druid.discovery.curator.path" : "/druid/discovery",
|
||||
"druid.selectors.indexing.serviceName" : "druid/overlord",
|
||||
"commit.periodMillis" : "15000",
|
||||
"consumer.numThreads" : "2",
|
||||
"kafka.zookeeper.connect" : "kafka-zk.example.com",
|
||||
"kafka.group.id" : "tranquility-kafka"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
{
|
||||
"dataSources" : {
|
||||
"metrics" : {
|
||||
"spec" : {
|
||||
"dataSchema" : {
|
||||
"dataSource" : "metrics",
|
||||
"parser" : {
|
||||
"type" : "string",
|
||||
"parseSpec" : {
|
||||
"timestampSpec" : {
|
||||
"column" : "timestamp",
|
||||
"format" : "auto"
|
||||
},
|
||||
"dimensionsSpec" : {
|
||||
"dimensions" : [],
|
||||
"dimensionExclusions" : [
|
||||
"timestamp",
|
||||
"value"
|
||||
]
|
||||
},
|
||||
"format" : "json"
|
||||
}
|
||||
},
|
||||
"granularitySpec" : {
|
||||
"type" : "uniform",
|
||||
"segmentGranularity" : "hour",
|
||||
"queryGranularity" : "none"
|
||||
},
|
||||
"metricsSpec" : [
|
||||
{
|
||||
"type" : "count",
|
||||
"name" : "count"
|
||||
},
|
||||
{
|
||||
"name" : "value_sum",
|
||||
"type" : "doubleSum",
|
||||
"fieldName" : "value"
|
||||
},
|
||||
{
|
||||
"fieldName" : "value",
|
||||
"name" : "value_min",
|
||||
"type" : "doubleMin"
|
||||
},
|
||||
{
|
||||
"type" : "doubleMax",
|
||||
"name" : "value_max",
|
||||
"fieldName" : "value"
|
||||
}
|
||||
]
|
||||
},
|
||||
"ioConfig" : {
|
||||
"type" : "realtime"
|
||||
},
|
||||
"tuningConfig" : {
|
||||
"type" : "realtime",
|
||||
"intermediatePersistPeriod" : "PT10M",
|
||||
"windowPeriod" : "PT10M"
|
||||
}
|
||||
},
|
||||
"properties" : {
|
||||
"task.partitions" : "1",
|
||||
"task.replicants" : "1"
|
||||
}
|
||||
}
|
||||
},
|
||||
"properties" : {
|
||||
"zookeeper.connect" : "master.example.com",
|
||||
"druid.discovery.curator.path" : "/druid/discovery",
|
||||
"druid.selectors.indexing.serviceName" : "druid/overlord",
|
||||
"http.port" : "8200",
|
||||
"http.threads" : "8"
|
||||
}
|
||||
}
|
||||
|
|
@ -1,84 +0,0 @@
|
|||
{
|
||||
"dataSources" : {
|
||||
"wikipedia" : {
|
||||
"spec" : {
|
||||
"dataSchema" : {
|
||||
"dataSource" : "wikipedia",
|
||||
"parser" : {
|
||||
"type" : "string",
|
||||
"parseSpec" : {
|
||||
"format" : "json",
|
||||
"dimensionsSpec" : {
|
||||
"dimensions" : [
|
||||
"channel",
|
||||
"cityName",
|
||||
"comment",
|
||||
"countryIsoCode",
|
||||
"countryName",
|
||||
"isAnonymous",
|
||||
"isMinor",
|
||||
"isNew",
|
||||
"isRobot",
|
||||
"isUnpatrolled",
|
||||
"metroCode",
|
||||
"namespace",
|
||||
"page",
|
||||
"regionIsoCode",
|
||||
"regionName",
|
||||
"user"
|
||||
]
|
||||
},
|
||||
"timestampSpec": {
|
||||
"column": "time",
|
||||
"format": "iso"
|
||||
}
|
||||
}
|
||||
},
|
||||
"metricsSpec" : [
|
||||
{
|
||||
"name" : "added",
|
||||
"type" : "longSum",
|
||||
"fieldName" : "added"
|
||||
},
|
||||
{
|
||||
"name" : "deleted",
|
||||
"type" : "longSum",
|
||||
"fieldName" : "deleted"
|
||||
},
|
||||
{
|
||||
"name" : "delta",
|
||||
"type" : "longSum",
|
||||
"fieldName" : "delta"
|
||||
}
|
||||
],
|
||||
"granularitySpec" : {
|
||||
"type" : "uniform",
|
||||
"segmentGranularity" : "day",
|
||||
"queryGranularity" : "none",
|
||||
"intervals" : ["2015-09-12/2015-09-13"],
|
||||
"rollup" : false
|
||||
}
|
||||
},
|
||||
"ioConfig" : {
|
||||
"type" : "realtime"
|
||||
},
|
||||
"tuningConfig" : {
|
||||
"type" : "realtime",
|
||||
"intermediatePersistPeriod" : "PT10M",
|
||||
"windowPeriod" : "P3650D"
|
||||
}
|
||||
},
|
||||
"properties" : {
|
||||
"task.partitions" : "1",
|
||||
"task.replicants" : "1"
|
||||
}
|
||||
}
|
||||
},
|
||||
"properties" : {
|
||||
"zookeeper.connect" : "localhost",
|
||||
"druid.discovery.curator.path" : "/druid/discovery",
|
||||
"druid.selectors.indexing.serviceName" : "druid/overlord",
|
||||
"http.port" : "8200",
|
||||
"http.threads" : "9"
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue