diff --git a/docs/content/configuration/production-cluster.md b/docs/content/configuration/production-cluster.md index 1bf9a40b381..0981385bb3b 100644 --- a/docs/content/configuration/production-cluster.md +++ b/docs/content/configuration/production-cluster.md @@ -24,7 +24,7 @@ To setup a local Druid cluster, see [Simple Cluster Configuration](../configurat ``` # Extensions -druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions", "io.druid.extensions:druid-histogram", "io.druid.extensions:mysql-metadata-storage"] +druid.extensions.loadList=["druid-s3-extensions", "druid-histogram", "mysql-metadata-storage"] # Zookeeper druid.zk.service.host=#{ZK_IPs} diff --git a/docs/content/configuration/simple-cluster.md b/docs/content/configuration/simple-cluster.md index 2f434db6551..783d80d693c 100644 --- a/docs/content/configuration/simple-cluster.md +++ b/docs/content/configuration/simple-cluster.md @@ -10,7 +10,7 @@ This simple Druid cluster configuration can be used for initially experimenting ``` # Extensions --Ddruid.extensions.coordinates=["io.druid.extensions:druid-kafka-eight"] +-Ddruid.extensions.loadList=["druid-kafka-eight"] # Zookeeper (defaults to localhost) diff --git a/docs/content/dependencies/cassandra-deep-storage.md b/docs/content/dependencies/cassandra-deep-storage.md index 15d409e52a1..77b37ac3c09 100644 --- a/docs/content/dependencies/cassandra-deep-storage.md +++ b/docs/content/dependencies/cassandra-deep-storage.md @@ -32,7 +32,7 @@ First create the schema above. I use a new keyspace called `druid` for this purp Then, add the following to your historical and realtime runtime properties files to enable a Cassandra backend. ```properties -druid.extensions.coordinates=["io.druid.extensions:druid-cassandra-storage:"] +druid.extensions.loadList=["druid-cassandra-storage"] druid.storage.type=c* druid.storage.host=localhost:9160 druid.storage.keyspace=druid diff --git a/docs/content/dependencies/metadata-storage.md b/docs/content/dependencies/metadata-storage.md index 8c1046334c1..b1470b71986 100644 --- a/docs/content/dependencies/metadata-storage.md +++ b/docs/content/dependencies/metadata-storage.md @@ -11,18 +11,28 @@ a number of tables used for various purposes described below. The following metadata storage engines are supported: -* Derby (default - only works if you have all processes running on the same node) -* MySQL (io.druid.extensions:mysql-metadata-storage) -* PostgreSQL (io.druid.extensions:postgresql-metadata-storage) +* Derby (default, but not suitable for production) +* MySQL +* PostgreSQL -To choose a metadata storage, +Even though Derby is the default, it works only if you have all Druid +processes running on the same host, and should be used only for experimentation. +For production, MySQL or PostgreSQL should be used. -1. Make sure Druid can pick up the extension files from either classpath or -extensions directory, see [Including Extensions](../operations/including-extensions.html) for more information. - -2. set the `druid.extensions` configuration to include the extension for the -metadata storage you plan to use. See below. +To choose the metadata storage type, set `druid.metadata.storage.type` to +`mysql`, `postgres` or `derby`. +Set other `druid.metadata.storage` configuration +keywords as shown below to give Druid information about how to connect to +the database. +As discussed in [Including Extensions](../operations/including-extensions.html), +there are two ways for giving Druid the extension files it needs for the +database you are using. +The first is to put the extension files in the classpath. The second is to +put the extension files in a subdirectory of +`druid.extensions.directory` (by default `extensions` under the Druid working directory) and list the subdirectory name in +`druid.extensions.loadList`. The example properties below show the second +way. ## Setting up MySQL @@ -57,20 +67,20 @@ metadata storage you plan to use. See below. 3. Configure your Druid metadata storage extension: Add the following parameters to your Druid configuration, replacing `` - with the hostname of the database. + with the location (host name and port) of the database. ```properties druid.extensions.loadList=["mysql-metadata-storage"] druid.metadata.storage.type=mysql - druid.metadata.storage.connector.connectURI=jdbc:mysql:///druid_test + druid.metadata.storage.connector.connectURI=jdbc:mysql:///druid druid.metadata.storage.connector.user=druid druid.metadata.storage.connector.password=diurd ``` - Note: metadata storage extension is not packaged within the main Druid tarball, it is + Note: the metadata storage extension is not packaged within the main Druid tarball; it is packaged in a separate tarball that can be downloaded from [here](http://druid.io/downloads.html). - However, you can always get it using [pull-deps](../pull-deps.html), or you can even build - it from source code, see [Build from Source](../development/build.html) + You can also get it using [pull-deps](../pull-deps.html), or you can build + it from source code; see [Build from Source](../development/build.html). ## Setting up PostgreSQL @@ -104,16 +114,25 @@ metadata storage you plan to use. See below. 3. Configure your Druid metadata storage extension: Add the following parameters to your Druid configuration, replacing `` - with the hostname of the database. + with the location (host name and port) of the database. ```properties druid.extensions.loadList=["postgresql-metadata-storage"] druid.metadata.storage.type=postgresql - druid.metadata.storage.connector.connectURI=jdbc:postgresql:///druid_test + druid.metadata.storage.connector.connectURI=jdbc:postgresql:///druid druid.metadata.storage.connector.user=druid druid.metadata.storage.connector.password=diurd ``` +## Using derby + + Add the following to your Druid configuration. + + ```properties + druid.metadata.storage.type=derby + druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527//home/y/var/druid_state/derby;create=true + ``` + ## Metadata Storage Tables ### Segments Table diff --git a/docs/content/development/build.md b/docs/content/development/build.md index c40d0343002..62b2dbbf37c 100644 --- a/docs/content/development/build.md +++ b/docs/content/development/build.md @@ -4,9 +4,9 @@ layout: doc_page ### Build from Source -The other way to setup Druid is from source via git. +Druid can be set up by building from source via git. -Builing Druid requires the following: +Building Druid requires the following: - [JDK 7](http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html) or [JDK 8](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) - [Maven version 3.x](http://maven.apache.org/download.cgi) @@ -22,8 +22,8 @@ mvn clean install This will compile the project and create the Druid binary distribution tar under `distribution/target/druid-VERSION-bin.tar.gz`. -This will also create a tarball that contains `mysql-metadata-storage` extension under - `distribution/target/mysql-metadata-storage-bin.tar.gz`. If you want Druid to load `mysql-metadata-storage`, you can first untar `druid-VERSION-bin.tar.gz`, then go to ```druid-/extensions```, untar `mysql-metadata-storage-bin.tar.gz` there. Now just specifiy `mysql-metadata-storage` in `druid.extensions.loadList` so that Druid will pick it up. See [Including Extensions](../operations/including-extensions.html) for more infomation. +This will also create `distribution/target/mysql-metadata-storage-bin.tar.gz`, +which is a tarball that contains the `mysql-metadata-storage` extension. You can find the example executables in the examples/bin directory: diff --git a/docs/content/development/experimental.md b/docs/content/development/experimental.md index 16295428bdd..5d52309fd2f 100644 --- a/docs/content/development/experimental.md +++ b/docs/content/development/experimental.md @@ -5,10 +5,10 @@ layout: doc_page Experimental features are features we have developed but have not fully tested in a production environment. If you choose to try them out, there will likely be edge cases that we have not covered. We would love feedback on any of these features, whether they are bug reports, suggestions for improvement, or letting us know they work as intended. -To enable experimental features, include their artifacts in the configuration runtime.properties file. Eg- +To enable experimental features, include their artifacts in the configuration runtime.properties file, e.g., ``` -druid.extensions.coordinates=["io.druid.extensions:druid-histogram:{VERSION}"] +druid.extensions.loadList=["druid-histogram"] ``` -The configuration for all the indexer and query nodes need to be updated with this. \ No newline at end of file +The configuration files for all the indexer and query nodes need to be updated with this. diff --git a/docs/content/operations/including-extensions.md b/docs/content/operations/including-extensions.md index 23a045d3167..7081bca1fc7 100644 --- a/docs/content/operations/including-extensions.md +++ b/docs/content/operations/including-extensions.md @@ -19,9 +19,9 @@ If you don't want to fiddle with classpath, you can create an extension director To let Druid load your extensions, follow the steps below -1) Specify `druid.extensions.directory` (root directory for normal Druid extensions). If you don' specify it, Druid will use their default value, see [Configuration](../configuration/index.html). +1) Specify `druid.extensions.directory` (root directory for normal Druid extensions). If you don't specify it, Druid will use the default value, see [Configuration](../configuration/index.html). -2) Prepare normal extension directories under root extension directory. Under the root extension directory, you should create sub-directories for each extension you might want to load. Inside each sub-directory, you can put extension related files in it. (If you don't want to manually setup the extension directory, Druid also provides a [pull-deps](../pull-deps.html) tool that can help you generate these directories automatically) +2) Under the root extension directory, create sub-directories for each extension you might want to load. Inside each sub-directory, put extension-related files. (If you don't want to manually set up the extension directory, you can use Druid's [pull-deps](../pull-deps.html) tool to help you generate these directories automatically.) Example: @@ -63,8 +63,8 @@ extensions/ As you can see, under ```extensions``` there are three sub-directories ```druid-examples```, ```druid-kafka-eight``` and ```mysql-metadata-storage```, each sub-directory denotes an extension that Druid might load. -3) Tell Druid which extensions to load. Now you have prepared your extension directories, if you want Druid to load a specific list of extensions under root extension directory, you need to specify `druid.extensions.loadList`. Using the example above, if you want Druid to load ```druid-kafka-eight``` and ```mysql-metadata-storage```, you can specify `druid.extensions.loadList=["druid-kafka-eight", "mysql-metadata-storage"]`. +3) To have Druid load a specific list of extensions present under the root extension directory, set `druid.extensions.loadList` to the list of extensions to load. Using the example above, if you want Druid to load ```druid-kafka-eight``` and ```mysql-metadata-storage```, you can specify `druid.extensions.loadList=["druid-kafka-eight", "mysql-metadata-storage"]`. -If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system. +If you specify `druid.extensions.loadList=[]`, Druid won't load any extensions from the file system. -If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory. +If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under the root extension directory. diff --git a/docs/content/tutorials/tutorial-a-first-look-at-druid.md b/docs/content/tutorials/tutorial-a-first-look-at-druid.md index 08995fda35f..858858cba78 100644 --- a/docs/content/tutorials/tutorial-a-first-look-at-druid.md +++ b/docs/content/tutorials/tutorial-a-first-look-at-druid.md @@ -12,9 +12,9 @@ About the data The data source we'll be working with is Wikipedia edits. Each time an edit is made in Wikipedia, an event gets pushed to an IRC channel associated with the language of the Wikipedia page. We scrape IRC channels for several different languages and load this data into Druid. -Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and where the user is a bot), and a list of metrics associated with the event (such as the number of characters added and deleted). +Each event has a timestamp indicating the time of the edit (in UTC time), a list of dimensions indicating various metadata about the event (such as information about the user editing the page and whether the user is a bot), and a list of metrics associated with the event (such as the number of characters added and deleted). -Specifically. the data schema looks like so: +Specifically. the data schema looks like this: Dimensions (things to filter on): @@ -53,7 +53,7 @@ We've built a tarball that contains everything you'll need. You'll find it [here ### Build From Source -Follow the [Build From Source](../development/build.html) guide to build from source. Then grab the tarball from services/target/druid--bin.tar.gz. +Follow the [Build From Source](../development/build.html) guide to build from source. Then grab the tarball from distribution/target/druid--bin.tar.gz. ### Unpack the Tarball diff --git a/docs/content/tutorials/tutorial-the-druid-cluster.md b/docs/content/tutorials/tutorial-the-druid-cluster.md index 01f516209c3..524addb6a3d 100644 --- a/docs/content/tutorials/tutorial-the-druid-cluster.md +++ b/docs/content/tutorials/tutorial-the-druid-cluster.md @@ -12,7 +12,7 @@ In this tutorial, we will set up other types of Druid nodes and external depende ## Downloading Druid If you followed the first tutorial, you should already have Druid downloaded. If not, let's go back and do that first. -You can download the latest version of druid [here](http://druid.io/downloads.html). You can also [Build From Source](../development/build.html) and grab the tarball from services/target/druid--bin.tar.gz. +You can download the latest version of druid [here](http://druid.io/downloads.html). You can also [Build From Source](../development/build.html) and grab the tarball from distribution/target/druid--bin.tar.gz. Either way, once you have the tarball, untar the contents within by issuing: @@ -95,7 +95,7 @@ In the directory, there should be a `common.runtime.properties` file with the fo ``` # Extensions -druid.extensions.coordinates=["io.druid.extensions:druid-examples","io.druid.extensions:druid-kafka-eight"] +druid.extensions.loadList=["druid-examples","druid-kafka-eight"] # Zookeeper druid.zk.service.host=localhost