Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

fix according to code review and fix broken examples

This commit is contained in:
fjy 2013-11-07 15:42:48 -08:00
parent e2e10fae1f
commit aeb411a3a3
30 changed files with 68 additions and 36 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/content/Configuration.md
View File

@ -15,7 +15,7 @@ There are three JVM parameters that we set on all of our processes:
Modules
=======
As of Druid v0.6.2, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules.
As of Druid v0.6, most core Druid functionality has been compartmentalized into modules. There are a set of default modules that may apply to any node type, and there are specific modules for the different node types. Default modules are __lazily instantiated__. Each module has its own set of configuration. This page will describe the configuration of the default modules.
Configuration of the various modules is done via Java properties. These can either be provided as `-D` system properties on the java command line or they can be passed in via a file called `runtime.properties` that exists on the classpath.

4
docs/content/Tutorial:-The-Druid-Cluster.md
View File

@ -202,7 +202,7 @@ Usually, when new segments are created, these MySQL entries are created directly
``` sql
use druid;
INSERT INTO segments (id, dataSource, created_date, start, end, partitioned, version, used, payload) VALUES ('wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z', 'wikipedia', '2013-08-08T21:26:23.799Z', '2013-08-01T00:00:00.000Z', '2013-08-02T00:00:00.000Z', '0', '2013-08-08T21:22:48.989Z', '1', '{\"dataSource\":\"wikipedia\",\"interval\":\"2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z\",\"version\":\"2013-08-08T21:22:48.989Z\",\"loadSpec\":{\"type\":\"s3_zip\",\"bucket\":\"static.druid.io\",\"key\":\"data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip\"},\"dimensions\":\"dma_code,continent_code,geo,area_code,robot,country_name,network,city,namespace,anonymous,unpatrolled,page,postal_code,language,newpage,user,region_lookup\",\"metrics\":\"count,delta,variation,added,deleted\",\"shardSpec\":{\"type\":\"none\"},\"binaryVersion\":9,\"size\":24664730,\"identifier\":\"wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z\"}');
INSERT INTO druid_segments (id, dataSource, created_date, start, end, partitioned, version, used, payload) VALUES ('wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z', 'wikipedia', '2013-08-08T21:26:23.799Z', '2013-08-01T00:00:00.000Z', '2013-08-02T00:00:00.000Z', '0', '2013-08-08T21:22:48.989Z', '1', '{\"dataSource\":\"wikipedia\",\"interval\":\"2013-08-01T00:00:00.000Z/2013-08-02T00:00:00.000Z\",\"version\":\"2013-08-08T21:22:48.989Z\",\"loadSpec\":{\"type\":\"s3_zip\",\"bucket\":\"static.druid.io\",\"key\":\"data/segments/wikipedia/20130801T000000.000Z_20130802T000000.000Z/2013-08-08T21_22_48.989Z/0/index.zip\"},\"dimensions\":\"dma_code,continent_code,geo,area_code,robot,country_name,network,city,namespace,anonymous,unpatrolled,page,postal_code,language,newpage,user,region_lookup\",\"metrics\":\"count,delta,variation,added,deleted\",\"shardSpec\":{\"type\":\"none\"},\"binaryVersion\":9,\"size\":24664730,\"identifier\":\"wikipedia_2013-08-01T00:00:00.000Z_2013-08-02T00:00:00.000Z_2013-08-08T21:22:48.989Z\"}');
```
If you look in your coordinator node logs, you should, after a maximum of a minute or so, see logs of the following form:
@ -237,6 +237,8 @@ druid.port=8083
druid.zk.service.host=localhost
druid.extensions.coordinates=["io.druid.extensions:druid-examples:0.6.2-SNAPSHOT"]
druid.db.connector.connectURI=jdbc\:mysql\://localhost\:3306/druid
druid.db.connector.user=druid
druid.db.connector.password=diurd

4
docs/content/Twitter-Tutorial.textile
View File

@ -61,12 +61,12 @@ Once the node starts up you will see a bunch of logs about setting up properties
<pre><code>2013-05-17 23:04:40,934 INFO [main] org.mortbay.log - Started SelectChannelConnector@0.0.0.0:8080
2013-05-17 23:04:40,935 INFO [main] com.metamx.common.lifecycle.Lifecycle$AnnotationBasedHandler - Invoking start method[public void com.metamx.druid.http.FileRequestLogger.start()] on object[com.metamx.druid.http.FileRequestLogger@42bb0406].
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Connection established.
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] druid.examples.twitter.TwitterSpritzerFirehoseFactory - Connected_to_Twitter
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] io.druid.examples.twitter.TwitterSpritzerFirehoseFactory - Connected_to_Twitter
2013-05-17 23:04:41,578 INFO [Twitter Stream consumer-1[Establishing connection]] twitter4j.TwitterStreamImpl - Receiving status stream.
</code></pre>
Periodically, you'll also see messages of the form:
<pre><code>2013-05-17 23:04:59,793 INFO [chief-twitterstream] druid.examples.twitter.TwitterSpritzerFirehoseFactory - nextRow() has returned 1,000 InputRows
<pre><code>2013-05-17 23:04:59,793 INFO [chief-twitterstream] io.druid.examples.twitter.TwitterSpritzerFirehoseFactory - nextRow() has returned 1,000 InputRows
</code></pre>
These messages indicate you are ingesting events. The Druid real time-node ingests events in an in-memory buffer. Periodically, these events will be persisted to disk. Persisting to disk generates a whole bunch of logs:

2
examples/bin/run_example_client.sh
View File

@ -18,7 +18,7 @@ EXAMPLE=$1
if [ -z ${EXAMPLE} ] ; then
echo "Please specify an example type."
echo "Examples availables:"
echo `ls ${EXAMPLES_DIR}`
echo `ls ${EXAMPLES_DIR} | grep -v indexing`
read -p "> " EXAMPLE
echo " "
fi

2
examples/bin/run_example_server.sh
View File

@ -19,7 +19,7 @@ EXAMPLE=$1
if [ -z ${EXAMPLE} ] ; then
echo "Please specify an example type."
echo "Examples availables:"
echo `ls ${EXAMPLES_DIR}`
echo `ls ${EXAMPLES_DIR} | grep -v indexing`
read -p "> " EXAMPLE
echo " "
fi

2
examples/config/historical/runtime.properties
View File

@ -4,7 +4,7 @@ druid.port=8081
druid.zk.service.host=localhost
druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions:0.6.2"]
druid.extensions.coordinates=["io.druid.extensions:druid-s3-extensions:0.6.2-SNAPSHOT"]
# Dummy read only AWS account (used to download example data)
druid.s3.secretKey=QyyfVZ7llSiRg6Qcrql1eEUG7buFpAK6T6engr1b

2
examples/config/realtime/runtime.properties
View File

@ -4,6 +4,8 @@ druid.port=8083
druid.zk.service.host=localhost
druid.extensions.coordinates=["io.druid.extensions:druid-examples:0.6.2-SNAPSHOT"]
druid.db.connector.connectURI=jdbc\:mysql\://localhost\:3306/druid
druid.db.connector.user=druid
druid.db.connector.password=diurd

2
examples/pom.xml
View File

@ -20,7 +20,7 @@
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>io.druid</groupId>
<groupId>io.druid.extensions</groupId>
<artifactId>druid-examples</artifactId>
<name>druid-examples</name>
<description>druid-examples</description>

10
examples/src/main/java/druid/examples/ExamplesDruidModule.java → examples/src/main/java/io/druid/examples/ExamplesDruidModule.java
View File

@ -17,16 +17,16 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples;
package io.druid.examples;
import com.fasterxml.jackson.databind.Module;
import com.fasterxml.jackson.databind.jsontype.NamedType;
import com.fasterxml.jackson.databind.module.SimpleModule;
import com.google.inject.Binder;
import druid.examples.flights.FlightsFirehoseFactory;
import druid.examples.rand.RandomFirehoseFactory;
import druid.examples.twitter.TwitterSpritzerFirehoseFactory;
import druid.examples.web.WebFirehoseFactory;
import io.druid.examples.flights.FlightsFirehoseFactory;
import io.druid.examples.rand.RandomFirehoseFactory;
import io.druid.examples.twitter.TwitterSpritzerFirehoseFactory;
import io.druid.examples.web.WebFirehoseFactory;
import io.druid.initialization.DruidModule;
import java.util.Arrays;

2
examples/src/main/java/druid/examples/flights/FlightsConverter.java → examples/src/main/java/io/druid/examples/flights/FlightsConverter.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.flights;
package io.druid.examples.flights;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.io.Closeables;

2
examples/src/main/java/druid/examples/flights/FlightsFirehoseFactory.java → examples/src/main/java/io/druid/examples/flights/FlightsFirehoseFactory.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.flights;
package io.druid.examples.flights;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

2
examples/src/main/java/druid/examples/rabbitmq/RabbitMQProducerMain.java → examples/src/main/java/io/druid/examples/rabbitmq/RabbitMQProducerMain.java
View File

@ -1,4 +1,4 @@
package druid.examples.rabbitmq;
package io.druid.examples.rabbitmq;
import com.rabbitmq.client.Channel;
import com.rabbitmq.client.Connection;

2
examples/src/main/java/druid/examples/rand/RandomFirehoseFactory.java → examples/src/main/java/io/druid/examples/rand/RandomFirehoseFactory.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.rand;
package io.druid.examples.rand;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

2
examples/src/main/java/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java → examples/src/main/java/io/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.twitter;
package io.druid.examples.twitter;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

2
examples/src/main/java/druid/examples/web/InputSupplierUpdateStream.java → examples/src/main/java/io/druid/examples/web/InputSupplierUpdateStream.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.fasterxml.jackson.core.JsonParseException;
import com.fasterxml.jackson.core.type.TypeReference;

2
examples/src/main/java/druid/examples/web/InputSupplierUpdateStreamFactory.java → examples/src/main/java/io/druid/examples/web/InputSupplierUpdateStreamFactory.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.io.InputSupplier;

2
examples/src/main/java/druid/examples/web/RenamingKeysUpdateStream.java → examples/src/main/java/io/druid/examples/web/RenamingKeysUpdateStream.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.collect.Maps;

2
examples/src/main/java/druid/examples/web/RenamingKeysUpdateStreamFactory.java → examples/src/main/java/io/druid/examples/web/RenamingKeysUpdateStreamFactory.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import java.util.Map;

2
examples/src/main/java/druid/examples/web/UpdateStream.java → examples/src/main/java/io/druid/examples/web/UpdateStream.java
View File

@ -16,7 +16,7 @@
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import java.util.Map;
import java.util.concurrent.TimeUnit;

2
examples/src/main/java/druid/examples/web/UpdateStreamFactory.java → examples/src/main/java/io/druid/examples/web/UpdateStreamFactory.java
View File

@ -16,7 +16,7 @@
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
public interface UpdateStreamFactory
{

2
examples/src/main/java/druid/examples/web/WebFirehoseFactory.java → examples/src/main/java/io/druid/examples/web/WebFirehoseFactory.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.fasterxml.jackson.annotation.JsonCreator;
import com.fasterxml.jackson.annotation.JsonProperty;

2
examples/src/main/java/druid/examples/web/WebJsonSupplier.java → examples/src/main/java/io/druid/examples/web/WebJsonSupplier.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.io.InputSupplier;
import com.metamx.emitter.EmittingLogger;

1
examples/src/main/resources/META-INF/services/io.druid.initialization.DruidModule Normal file
View File

@ -0,0 +1 @@
io.druid.examples.ExamplesDruidModule

2
examples/src/test/java/druid/examples/web/InputSupplierUpdateStreamTest.java → examples/src/test/java/io/druid/examples/web/InputSupplierUpdateStreamTest.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.io.InputSupplier;
import junit.framework.Assert;

2
examples/src/test/java/druid/examples/web/RenamingKeysUpdateStreamTest.java → examples/src/test/java/io/druid/examples/web/RenamingKeysUpdateStreamTest.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.io.InputSupplier;
import junit.framework.Assert;

2
examples/src/test/java/druid/examples/web/TestCaseSupplier.java → examples/src/test/java/io/druid/examples/web/TestCaseSupplier.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.io.InputSupplier;

2
examples/src/test/java/druid/examples/web/WebFirehoseFactoryTest.java → examples/src/test/java/io/druid/examples/web/WebFirehoseFactoryTest.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Lists;

2
examples/src/test/java/druid/examples/web/WebJsonSupplierTest.java → examples/src/test/java/io/druid/examples/web/WebJsonSupplierTest.java
View File

@ -17,7 +17,7 @@
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*/
package druid.examples.web;
package io.druid.examples.web;
import org.junit.Test;

24
indexing-service/src/main/java/io/druid/indexing/common/index/EventReceiverFirehoseFactory.java
View File

@ -63,6 +63,9 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
private final MapInputRowParser parser;
private final Optional<ChatHandlerProvider> chatHandlerProvider;
@Deprecated
private final EventReceiverFirehoseFactoryConfig config;
@JsonCreator
public EventReceiverFirehoseFactory(
@JsonProperty("serviceName") String serviceName,
@ -75,14 +78,17 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
{
// This code is here for backwards compatibility
if (serviceName == null) {
serviceName = String.format("%s:%s",
this.serviceName = String.format(
"%s:%s",
config.getFirehoseIdPrefix(),
Preconditions.checkNotNull(firehoseId, "firehoseId")
);
this.config = config;
} else {
this.serviceName = Preconditions.checkNotNull(serviceName, "serviceName");
this.config = null;
}
this.serviceName = Preconditions.checkNotNull(serviceName, "serviceName");
this.bufferSize = bufferSize == null || bufferSize <= 0 ? DEFAULT_BUFFER_SIZE : bufferSize;
this.parser = Preconditions.checkNotNull(parser, "parser");
this.chatHandlerProvider = Optional.fromNullable(chatHandlerProvider);
@ -98,7 +104,6 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
if (chatHandlerProvider.isPresent()) {
log.info("Found chathandler of class[%s]", chatHandlerProvider.get().getClass().getName());
chatHandlerProvider.get().register(serviceName, firehose);
chatHandlerProvider.get().register(serviceName.replaceAll(".*:", ""), firehose); // rofl
} else {
log.info("No chathandler detected");
}
@ -112,6 +117,17 @@ public class EventReceiverFirehoseFactory implements FirehoseFactory
return serviceName;
}
@Deprecated
@JsonProperty("firehoseId")
public String getFirehoseId()
{
if (config != null) {
return serviceName.replaceFirst(config.getFirehoseIdPrefix(), "");
}
return null;
}
@JsonProperty
public int getBufferSize()
{

13
services/pom.xml
View File

@ -17,7 +17,8 @@
~ along with this program; if not, write to the Free Software
~ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>io.druid</groupId>
<artifactId>druid-services</artifactId>
@ -82,6 +83,16 @@
<outputFile>
${project.build.directory}/${project.artifactId}-${project.version}-selfcontained.jar
</outputFile>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>