Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2315 from rasahner/docUpdateBuild 

docs: edits related to loadList
This commit is contained in:
Fangjin Yang 2016-01-21 09:52:27 -08:00
parent 04d3054353 c9368702fa
commit ab635c5cd3
9 changed files with 55 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/content/configuration/production-cluster.md
View File

@ -24,7 +24,7 @@ To setup a local Druid cluster, see [Simple Cluster Configuration](../configurat
```
# Extensions
druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions", "io.druid.extensions:druid-histogram", "io.druid.extensions:mysql-metadata-storage"]
druid.extensions.loadList=["druid-s3-extensions", "druid-histogram", "mysql-metadata-storage"]
# Zookeeper
druid.zk.service.host=#{ZK_IPs}

2
docs/content/configuration/simple-cluster.md
View File

@ -10,7 +10,7 @@ This simple Druid cluster configuration can be used for initially experimenting
```
# Extensions
-Ddruid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight"]
-Ddruid.extensions.loadList=["druid-kafka-eight"]
# Zookeeper (defaults to localhost)

2
docs/content/dependencies/cassandra-deep-storage.md
View File

@ -32,7 +32,7 @@ First create the schema above. I use a new keyspace called `druid` for this purp
Then, add the following to your historical and realtime runtime properties files to enable a Cassandra backend.
```properties
druid.extensions.coordinates=["io.druid.extensions:druid-cassandra-storage:<druid version>"]
druid.extensions.loadList=["druid-cassandra-storage"]
druid.storage.type=c*
druid.storage.host=localhost:9160
druid.storage.keyspace=druid

51
docs/content/dependencies/metadata-storage.md
View File

@ -11,18 +11,28 @@ a number of tables used for various purposes described below.
The following metadata storage engines are supported:
* Derby (default - only works if you have all processes running on the same node)
* MySQL (io.druid.extensions:mysql-metadata-storage)
* PostgreSQL (io.druid.extensions:postgresql-metadata-storage)
* Derby (default, but not suitable for production)
* MySQL
* PostgreSQL
To choose a metadata storage,
Even though Derby is the default, it works only if you have all Druid
processes running on the same host, and should be used only for experimentation.
For production, MySQL or PostgreSQL should be used.
1. Make sure Druid can pick up the extension files from either classpath or
extensions directory, see [Including Extensions](../operations/including-extensions.html) for more information.
2. set the `druid.extensions` configuration to include the extension for the
metadata storage you plan to use. See below.
To choose the metadata storage type, set `druid.metadata.storage.type` to
`mysql`, `postgres` or `derby`.
Set other `druid.metadata.storage` configuration
keywords as shown below to give Druid information about how to connect to
the database.
As discussed in [Including Extensions](../operations/including-extensions.html),
there are two ways for giving Druid the extension files it needs for the
database you are using.
The first is to put the extension files in the classpath. The second is to
put the extension files in a subdirectory of
`druid.extensions.directory` (by default `extensions` under the Druid working directory) and list the subdirectory name in
`druid.extensions.loadList`. The example properties below show the second
way.
## Setting up MySQL
@ -57,20 +67,20 @@ metadata storage you plan to use. See below.
3. Configure your Druid metadata storage extension:
Add the following parameters to your Druid configuration, replacing `<host>`
with the hostname of the database.
with the location (host name and port) of the database.
```properties
druid.extensions.loadList=["mysql-metadata-storage"]
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://<host>/druid_test
druid.metadata.storage.connector.connectURI=jdbc:mysql://<host>/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=diurd
```
Note: metadata storage extension is not packaged within the main Druid tarball, it is
Note: the metadata storage extension is not packaged within the main Druid tarball; it is
packaged in a separate tarball that can be downloaded from [here](http://druid.io/downloads.html).
However, you can always get it using [pull-deps](../pull-deps.html), or you can even build
it from source code, see [Build from Source](../development/build.html)
You can also get it using [pull-deps](../pull-deps.html), or you can build
it from source code; see [Build from Source](../development/build.html).
## Setting up PostgreSQL
@ -104,16 +114,25 @@ metadata storage you plan to use. See below.
3. Configure your Druid metadata storage extension:
Add the following parameters to your Druid configuration, replacing `<host>`
with the hostname of the database.
with the location (host name and port) of the database.
```properties
druid.extensions.loadList=["postgresql-metadata-storage"]
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid_test
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<host>/druid
druid.metadata.storage.connector.user=druid
druid.metadata.storage.connector.password=diurd
```
## Using derby
Add the following to your Druid configuration.
```properties
druid.metadata.storage.type=derby
druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527//home/y/var/druid_state/derby;create=true
```
## Metadata Storage Tables
### Segments Table

8
docs/content/development/build.md
View File

@ -4,9 +4,9 @@ layout: doc_page
### Build from Source
The other way to setup Druid is from source via git.
Druid can be set up by building from source via git.
Builing Druid requires the following:
Building Druid requires the following:
- [JDK 7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html)
or [JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html)
- [Maven version 3.x](http://maven.apache.org/download.cgi)
@ -22,8 +22,8 @@ mvn clean install
This will compile the project and create the Druid binary distribution tar under
`distribution/target/druid-VERSION-bin.tar.gz`.
This will also create a tarball that contains `mysql-metadata-storage` extension under
`distribution/target/mysql-metadata-storage-bin.tar.gz`. If you want Druid to load `mysql-metadata-storage`, you can first untar `druid-VERSION-bin.tar.gz`, then go to ```druid-<version>/extensions```, untar `mysql-metadata-storage-bin.tar.gz` there. Now just specifiy `mysql-metadata-storage` in `druid.extensions.loadList` so that Druid will pick it up. See [Including Extensions](../operations/including-extensions.html) for more infomation.
This will also create `distribution/target/mysql-metadata-storage-bin.tar.gz`,
which is a tarball that contains the `mysql-metadata-storage` extension.
You can find the example executables in the examples/bin directory:

6
docs/content/development/experimental.md
View File

@ -5,10 +5,10 @@ layout: doc_page
Experimental features are features we have developed but have not fully tested in a production environment. If you choose to try them out, there will likely be edge cases that we have not covered. We would love feedback on any of these features, whether they are bug reports, suggestions for improvement, or letting us know they work as intended.
To enable experimental features, include their artifacts in the configuration runtime.properties file. Eg-
To enable experimental features, include their artifacts in the configuration runtime.properties file, e.g.,
```
druid.extensions.coordinates=["io.druid.extensions:druid-histogram:{VERSION}"]
druid.extensions.loadList=["druid-histogram"]
```
The configuration for all the indexer and query nodes need to be updated with this.
The configuration files for all the indexer and query nodes need to be updated with this.

10
docs/content/operations/including-extensions.md
View File

@ -19,9 +19,9 @@ If you don't want to fiddle with classpath, you can create an extension director
To let Druid load your extensions, follow the steps below
1) Specify `druid.extensions.directory` (root directory for normal Druid extensions). If you don' specify it, Druid will use their default value, see [Configuration](../configuration/index.html).
1) Specify `druid.extensions.directory` (root directory for normal Druid extensions). If you don't specify it, Druid will use the default value, see [Configuration](../configuration/index.html).
2) Prepare normal extension directories under root extension directory. Under the root extension directory, you should create sub-directories for each extension you might want to load. Inside each sub-directory, you can put extension related files in it. (If you don't want to manually setup the extension directory, Druid also provides a [pull-deps](../pull-deps.html) tool that can help you generate these directories automatically)
2) Under the root extension directory, create sub-directories for each extension you might want to load. Inside each sub-directory, put extension-related files. (If you don't want to manually set up the extension directory, you can use Druid's [pull-deps](../pull-deps.html) tool to help you generate these directories automatically.)
Example:
@ -63,8 +63,8 @@ extensions/
As you can see, under ```extensions``` there are three sub-directories ```druid-examples```, ```druid-kafka-eight``` and ```mysql-metadata-storage```, each sub-directory denotes an extension that Druid might load.
3) Tell Druid which extensions to load. Now you have prepared your extension directories, if you want Druid to load a specific list of extensions under root extension directory, you need to specify `druid.extensions.loadList`. Using the example above, if you want Druid to load ```druid-kafka-eight``` and ```mysql-metadata-storage```, you can specify `druid.extensions.loadList=["druid-kafka-eight", "mysql-metadata-storage"]`.
3) To have Druid load a specific list of extensions present under the root extension directory, set `druid.extensions.loadList` to the list of extensions to load. Using the example above, if you want Druid to load ```druid-kafka-eight``` and ```mysql-metadata-storage```, you can specify `druid.extensions.loadList=["druid-kafka-eight", "mysql-metadata-storage"]`.
If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system.
If you specify `druid.extensions.loadList=[]`, Druid won't load any extensions from the file system.
If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory.
If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under the root extension directory.

6
docs/content/tutorials/tutorial-a-first-look-at-druid.md
View File

@ -12,9 +12,9 @@ About the data
The data source we'll be working with is Wikipedia edits. Each time an edit is made in Wikipedia, an event gets pushed to an IRC channel associated with the language of the Wikipedia page. We scrape IRC channels for several different languages and load this data into Druid.
Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and where the user is a bot), and a list of metrics associated with the event (such as the number of characters added and deleted).
Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and whether the user is a bot), and a list of metrics associated with the event (such as the number of characters added and deleted).
Specifically. the data schema looks like so:
Specifically. the data schema looks like this:
Dimensions (things to filter on):
@ -53,7 +53,7 @@ We've built a tarball that contains everything you'll need. You'll find it [here
### Build From Source
Follow the [Build From Source](../development/build.html) guide to build from source. Then grab the tarball from services/target/druid-<version>-bin.tar.gz.
Follow the [Build From Source](../development/build.html) guide to build from source. Then grab the tarball from distribution/target/druid-<version>-bin.tar.gz.
### Unpack the Tarball

4
docs/content/tutorials/tutorial-the-druid-cluster.md
View File

@ -12,7 +12,7 @@ In this tutorial, we will set up other types of Druid nodes and external depende
## Downloading Druid
If you followed the first tutorial, you should already have Druid downloaded. If not, let's go back and do that first.
You can download the latest version of druid [here](http://druid.io/downloads.html). You can also [Build From Source](../development/build.html) and grab the tarball from services/target/druid-<version>-bin.tar.gz.
You can download the latest version of druid [here](http://druid.io/downloads.html). You can also [Build From Source](../development/build.html) and grab the tarball from distribution/target/druid-<version>-bin.tar.gz.
Either way, once you have the tarball, untar the contents within by issuing:
@ -95,7 +95,7 @@ In the directory, there should be a `common.runtime.properties` file with the fo
```
# Extensions
druid.extensions.coordinates=["io.druid.extensions:druid-examples","io.druid.extensions:druid-kafka-eight"]
druid.extensions.loadList=["druid-examples","druid-kafka-eight"]
# Zookeeper
druid.zk.service.host=localhost