f123a14373
Merge branch 'master' into az
...
Conflicts:
docs/content/Realtime.md
server/src/main/java/io/druid/client/selector/ServerSelector.java
2014-01-09 19:09:38 -08:00
632d6b0a78
make server selection configurable, add back random server selection
2014-01-09 14:37:33 -08:00
f44509b530
queries working at this point, tests pass, still need more tests and fix rules interface
2014-01-07 16:24:53 -08:00
e11952f3b6
bridge is kinda working for mastery stuff right now, tests pass
2014-01-06 17:01:50 -08:00
95d92915bf
more tests for Initialization + dependency cleanup
2013-12-30 16:31:10 -08:00
bbe1b82347
Update versions
2013-12-20 12:26:27 -08:00
4bc8ff4d6a
Update versions
2013-12-20 11:19:36 -08:00
fba6caf7fd
Merge pull request #326 from metamx/versions
...
StatusResource
2013-12-18 22:44:50 -08:00
fc4dd51194
Update versions
2013-12-18 15:33:25 -08:00
d112b2d211
prepare for next release
2013-12-17 17:47:04 -08:00
0051877f84
fix status endpoint
2013-12-17 17:46:16 -08:00
07875dd30b
prepare for next release
2013-12-17 16:28:11 -08:00
69f33b0acd
remove commented line
2013-12-17 16:24:15 -08:00
bbb2754a7c
fix redirects in druid
2013-12-17 16:22:36 -08:00
f2241c8885
prepare for next release
2013-12-16 16:16:00 -08:00
ac2ca0e46c
separate move and archive tasks
2013-12-16 14:00:55 -08:00
a417cd5df2
add archive task
2013-12-16 13:59:15 -08:00
84233238b1
Merge branch 'master' of github.com:metamx/druid
2013-12-16 12:01:55 -08:00
0998d835b0
fix redirection issues with master and overlord
2013-12-16 12:01:52 -08:00
7759ed4517
Coordinator: Link service announcement to leadership
2013-12-16 11:28:28 -08:00
3752122a7b
implementing comments from review
2013-12-16 15:16:28 +05:30
87b83bceb1
fix task storage config serde and prepare for next release
2013-12-13 16:55:22 -08:00
01f9c1df31
fix broken task storage config and prepare for next release
2013-12-13 16:45:32 -08:00
9a5fd6b0e9
prepare for next release
2013-12-13 16:18:34 -08:00
600dc7546f
Configurability of recency threshold
2013-12-13 16:02:54 -08:00
407f8caf94
remove server monitor from list of default monitors
2013-12-13 09:04:32 -08:00
25ebba2c38
fix default start delay of indexing service and prepare for next release
2013-12-12 18:51:25 -08:00
19eafe0e96
prepare for next release
2013-12-12 17:26:50 -08:00
a58f90a689
prepare for next release
2013-12-12 14:51:14 -08:00
70c153592f
CliPeon: Fix local mode
2013-12-12 14:22:57 -08:00
c4b8c8bc6f
Rework indexing service internals to hopefully be more reliable.
...
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.
Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).
Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
2013-12-11 15:05:16 -08:00
f3cfd1d781
Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service
2013-12-10 17:39:43 -08:00
6cc0860657
fix s3 module problems and prepare for next release
2013-12-09 20:05:55 -08:00
ab1f02b2bc
fix version brokenness and prepare for next release
2013-12-09 18:38:22 -08:00
976affb3af
prepare for next release
2013-12-09 18:28:31 -08:00
303f6ff334
fix worker config setup problems
2013-12-09 18:25:29 -08:00
8aba4ad3d2
Merge pull request #311 from metamx/versions
...
Add versions to static page
2013-12-09 17:18:16 -08:00
b5f6dbc32f
Code refactoring, one place!
2013-12-10 02:09:41 +05:30
8117fff4e2
Some minor fixes
2013-12-10 00:39:10 +05:30
fbad1e2d7a
fix convert properties with infoPath and prepare for next release
2013-12-04 18:55:35 -08:00
32200b9b41
removing debug line.
2013-12-04 11:00:47 +05:30
47278ad109
Merge relevant changes from 'druid-version-info'
2013-12-04 10:21:19 +05:30
05f72601c4
fix request time metric and prepare for next release
2013-12-02 14:20:59 -08:00
ddb6e83f17
fix realtime default and prepare for next deploy
2013-11-25 17:53:17 -08:00
fbb65988bd
make query resource eagerly instantiate and prepare for next release
2013-11-20 11:58:25 -08:00
c234dafee1
prepare for next release
2013-11-20 10:43:37 -08:00
84a6262e04
prepare for next release
2013-11-20 10:27:09 -08:00
e7167ae00a
add new query resource
2013-11-20 10:02:43 -08:00
984e24c1bb
fix broken links in docs, prepare for next release
2013-11-19 09:57:56 -08:00
67a93ccd4d
fix broken validation query config port
2013-11-18 15:53:49 -08:00
1820110818
clean up docs and fix server monitor double emitting metrics
2013-11-18 15:37:30 -08:00
a86d3af17e
fix docs
2013-11-14 18:29:27 -08:00
346cf0e04c
fix out of order in urls for hadoop classpath
2013-11-14 18:18:39 -08:00
5d00e4075e
prepare for next release
2013-11-14 15:06:26 -08:00
6b41681424
fix ordering of urls, ugh, need to write more tests
2013-11-14 15:06:07 -08:00
23ebca6d32
more hadoop dependency hell, getting the right urls to hadoop is hard
2013-11-14 14:29:42 -08:00
61367c6b13
prepare for next release
2013-11-14 13:16:38 -08:00
c51eed060f
actually use a class loader pull out dependencies from extension modules
2013-11-14 11:52:21 -08:00
fcf0c6ce06
change the classpath ordering for batch processing and prepare for next release
2013-11-14 10:52:51 -08:00
60300bdad0
fix doc version
2013-11-13 13:34:57 -08:00
c1009f0872
prepare for next release
2013-11-13 11:36:50 -08:00
d0f990129a
fix docs
2013-11-13 10:59:03 -08:00
9fe9ca3163
the emitter library is broken and also prepare for next release
2013-11-08 15:54:16 -08:00
822a30658d
fix broken http post emitter and prepare for next release
2013-11-08 14:01:16 -08:00
64b93bf448
fix broken autoscaling and prepare for next release
2013-11-08 11:29:42 -08:00
d70bcc5657
repair broken pom
2013-11-07 23:27:06 -08:00
a0eedf7342
fix broken docs
2013-11-07 23:11:30 -08:00
d1c3176bf5
update docs ver
2013-11-07 19:47:36 -08:00
a049b42674
fix an issue with task tables not getting created automatically and prepare for next release
2013-11-07 18:01:35 -08:00
d49554185a
fix broken default table names
2013-11-07 17:38:00 -08:00
6b573c76f1
more fixes
2013-11-07 14:40:45 -08:00
33561031a2
update docs version to match next release
2013-11-05 15:30:41 -08:00
dce6e7e0eb
hack to support remote repository authentication
2013-11-01 09:33:07 -07:00
ef88f37bbe
add tools pull-deps to cli
2013-10-30 18:16:27 -07:00
21235ae533
fix configuration conversion bugs and docs and hdfs configs
2013-10-29 17:34:24 -04:00
2e9c46867f
Fixes for indexing service.
...
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
2013-10-18 11:14:33 -07:00
a1c09df17f
make the hadoop index task work again
2013-10-16 09:45:17 -07:00
9796a40b92
port docs over to 0.6 and a bunch of misc fixes
2013-10-11 18:38:53 -07:00
a9a723bd11
clean up poms, add a new loading your own data tutorial, add new validation, clean up logs
2013-10-09 15:42:39 -07:00
4e509d1d09
Merge branch 'master' into is-docs
2013-10-09 14:05:10 -07:00
c47fe202c7
Fix HadoopDruidIndexer to work with the new way of things
...
There are multiple and sundry changes in here.
First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.
In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module. I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.
In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api. They are all API-level things, however, so they really belong in druid-api instead.
Lastly, I removed the druid-realtime module and put it all in druid-server.
2013-10-09 15:15:44 -05:00
4ec4b8e024
rewrite indexing service docs
2013-10-08 16:34:58 -07:00
ac330f72bb
first set of changes to standarize the naming convention we use in druid
2013-10-03 16:36:48 -07:00
bf157ea8ef
fix the example assembly and have better error logging for rabbitmq
2013-10-03 10:54:25 -07:00
17874eeb67
make the CliPeon actually able to run on its own
2013-10-02 15:55:10 -07:00
bc8db7daa5
1) make chat handler resource work again
...
2) add more default configs
3) make examples work again
2013-10-02 14:22:39 -07:00
384dcda7e4
Chat handlers still don't work, but, they're closer maybe.
2013-10-01 17:45:53 -07:00
5d0d71250b
fix chat handler resources not correctly registering themselves
2013-10-01 11:25:39 -07:00
d6445cd8f3
remove imports
2013-10-01 09:42:28 -07:00
53698a135a
add interface to new firehose as per code review comments
2013-09-30 18:00:59 -07:00
f55a5199b1
add a firehose module to remove so much copy and pasted code
2013-09-30 16:29:20 -07:00
ed9e0cf9f6
add a local firehose for indexing local files
2013-09-30 16:03:26 -07:00
4a7a28a7f6
Merge branch 'master' into fix-config
2013-09-27 17:17:46 -07:00
dc5dab8747
Fixes for property conversion, firehose registration, and the indexing service
2013-09-27 17:09:59 -07:00
a79ad7bab4
make dynamic master resource configuration work again
2013-09-27 15:00:40 -07:00
0b04325ee8
fix things up according to code review comments
2013-09-27 10:17:45 -07:00
e404295c1f
make indexing service work
2013-09-26 17:44:21 -07:00
8bc56daa66
fix things up according to code review comments
2013-09-26 11:35:45 -07:00
87259321b6
port hadoop druid indexer to new guice framework
2013-09-26 11:04:42 -07:00
19276f6bad
fix spacing issues and other code review comments
2013-09-24 14:34:09 -07:00
be5bb7f2eb
fix lifecycle startup/stop ordering problem with discovery module and make druid able to load local extensions
2013-09-24 14:20:56 -07:00
1ff04412a2
clean up realtime module and fix breakage in broker paths
2013-09-23 16:26:05 -07:00
1750b702d7
Add missed com.metamx.druid.emitter.period propery to ConvertProperties
2013-09-23 16:46:37 -05:00
cc632cdb30
Make Jetty Server lazy
...
JettyServerModule had been eagerly instantiating the Server object, which was causing things that didn't care about an HTTP interface to all of a sudden require host and port parameters. The change makes the JettyServerModule only setup the bindings without eagerly instantiating the Jetty Server. Each cli needs to register the Server class with the Lifecycle in order to make it actually get used.
2013-09-23 16:02:24 -05:00
dc8a119787
fix broken unit tests are a result of the last merge
2013-09-23 12:56:01 -07:00
fefc71e34f
Refix the things that were lost in the merge.
2013-09-20 18:01:08 -05:00
b13ae3f405
Merge branch 'master' into fix-extensionization
...
Conflicts:
services/src/main/java/io/druid/cli/CliBroker.java
services/src/main/java/io/druid/cli/CliCoordinator.java
2013-09-20 17:49:13 -05:00
5712b29c8c
Fix issues with bindings and handling extensions
...
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about. This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need. Guice tries to fail-fast and verifies that all
bindings exist before it does anything, which is a problem because the extension bind
some objects that don't necessarily have all of their dependencies bound in all processes.
The fix for this is to build a single Injector with all bindings in it and let each of the
processes only load the things that they care about. This also requires the use of
Module overrides and other such interesting things, which are node done.
In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided. This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.
In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
2013-09-20 17:45:01 -05:00
f26e13c61a
fix broken http endpoints
2013-09-19 17:33:58 -07:00
6bfe2310bb
Merge branch 'master' into guice
...
Conflicts:
client/pom.xml
client/src/main/java/com/metamx/druid/query/segment/LegacySegmentSpec.java
indexing-common/pom.xml
indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolboxFactory.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/task/RealtimeIndexTask.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskQueueTest.java
realtime/src/main/java/com/metamx/druid/realtime/RealtimeNode.java
realtime/src/main/java/com/metamx/druid/realtime/firehose/IrcFirehoseFactory.java
realtime/src/main/java/com/metamx/druid/realtime/firehose/WikipediaIrcDecoder.java
realtime/src/main/java/com/metamx/druid/realtime/plumber/RealtimePlumberSchool.java
server/src/main/java/com/metamx/druid/BaseServerNode.java
server/src/main/java/com/metamx/druid/http/ComputeNode.java
server/src/main/java/com/metamx/druid/index/v1/IncrementalIndexStorageAdapter.java
server/src/main/java/com/metamx/druid/query/group/GroupByQueryEngine.java
2013-09-18 16:46:54 -05:00
f7df951610
broker needs to announce itself in service discovery
2013-09-18 09:23:46 -07:00
cabae7993d
port over multi threaded realtime and also fix broken realtime nodes that can't start up
2013-09-16 16:03:47 -07:00
5a8b71ddec
fix how druid services is assembled
2013-09-13 18:41:33 -07:00
e0738b3b22
1) Add tool to convert properties from the old settings to the new ones.
2013-09-12 17:59:23 -05:00
a2dcc45a8e
1) Remove SingleSegmentLoader and replace with OmniSegmentLoader
2013-09-12 11:47:03 -05:00
3e4a4f5566
1) Adjustments to allow for the addition of new Runnables via modules. Modules should implement CliCommandCreator and add their stuff there.
2013-09-11 17:41:20 -05:00
6c9a107356
1) remove duplicate package initialization.initialization
2013-09-09 17:02:57 -05:00
a11eae4144
Edited assembly.xml to exclude the selfcontained jars, and edited path of run_example_server to use the lib/ directory instead of selfcontained jars, thus reducing the size of the assembly tarball from 150MB to 50MB.
2013-09-04 12:53:38 -07:00
740c70ed06
1) It's good when unit tests actually pass...
2013-08-31 17:27:15 -05:00
5ab671050e
No more com.metamx.druid, it is now all io.druid!
2013-08-30 19:42:12 -05:00
bd0756e360
More stuff moved, things still compiling and tests still passing. Yay!
2013-08-30 18:58:35 -05:00
56e2b956d0
OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.!
2013-08-30 18:21:04 -05:00
3617ac17fc
1) Eliminate ExecutorMain and have it run using the new Main!
2013-08-27 14:11:05 -05:00
269997dc94
1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful
2013-08-26 18:08:41 -05:00
55dbda2046
1) Worker appears to be running! It's also now known as the MiddleManager
2013-08-23 17:59:48 -05:00
b897c2cb22
1) IndexCoordinator appears to work as the CliOverlord now, yay!
2013-08-23 14:11:34 -05:00
6a8c160740
update code according to code review
2013-08-22 10:46:05 -07:00
85ee8bb267
port realtime to guice
2013-08-13 17:08:45 -07:00
51f648ec11
1) Rename QueryToolChestModule back
2013-08-06 12:05:56 -07:00
69f7831d82
1) Rename QueryToolChestModule because there was a case sensitivity issue, grrr
2013-08-06 12:04:04 -07:00
9f71d42314
1) Broker works with Guice
...
2) Extract ServerViewModule to handle various ServerView thingies
3) Extract QueryToolChestModule and QueryRunnerFactoryModule to reuse code for configuration of Query stuff
4) Extract QueryJettyServerInitializer to reuse between Historical and Broker nodes
5) Remove ClientMain, BrokerMain and BrokerNode
2013-08-06 12:01:52 -07:00
293c0e8418
1) Clean up the servlet registration stuff, resource registrations on CliHistorical weren't working, they do now.
2013-08-05 13:58:45 -07:00
d64bc2f762
1) Eliminate Old mains() that aren't needed anymore.
2013-08-05 13:33:43 -07:00
9e78bb38f5
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/QueryableNode.java
client/src/main/java/com/metamx/druid/client/ServerInventoryView.java
client/src/main/java/com/metamx/druid/coordination/SingleDataSegmentAnnouncer.java
client/src/main/java/com/metamx/druid/initialization/CuratorDiscoveryConfig.java
client/src/main/java/com/metamx/druid/query/MetricsEmittingExecutorService.java
indexing-hadoop/src/test/java/com/metamx/druid/indexer/HadoopDruidIndexerConfigTest.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/http/IndexerCoordinatorNode.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/http/WorkerNode.java
pom.xml
server/src/main/java/com/metamx/druid/coordination/ServerManager.java
server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
server/src/main/java/com/metamx/druid/db/DatabaseRuleManager.java
server/src/main/java/com/metamx/druid/db/DatabaseSegmentManager.java
server/src/main/java/com/metamx/druid/http/ComputeNode.java
server/src/main/java/com/metamx/druid/http/MasterMain.java
server/src/main/java/com/metamx/druid/loading/SegmentLoaderConfig.java
server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-08-01 16:42:47 -07:00
019bb5d453
1) Another whole bunch of changes to annotate things and create Modules and bind stuff. But OMFG, the compute node actually appears to be working!
...
2) The compute node works with Guice
3) The compute node fires up with Guice and appears to work
4) Did I mention that the compute node, now called historical node, fires up with Guice and appears to work?
2013-08-01 15:28:08 -07:00
f4fc8b2f6e
1) A whole boondoggle of changes to Guicify the instantiation of things for the Historical nodes
...
2) Add GuiceInjectableValues that supports @JacksonInject resolution via Guice
3) Refactor RequestLogger stuff and setup polymorphic deserialization method of creating objects
2013-07-31 18:36:47 -07:00
97fb971f84
1) Start the journey of making the historical nodes work with Guice. Doesn't work yet, but at least it builds.
2013-07-30 11:58:05 -07:00
2b587cba85
1) Introduce airlift/airline for CLI stuff and add a Coordinator CLI verb to make it all happiness
2013-07-29 15:14:20 -07:00
28af252bca
Include multiple configurations, one per node type, in assembly package for examples.
2013-07-01 16:36:30 -07:00
5a9bf98b6c
Some changes to automatically build a kit with samples.
2013-05-04 20:04:16 -07:00
12fe145cc8
1) Setup assembly to grab all dependencies -- it's broken right now, http://jira.codehaus.org/browse/MASSEMBLY-646
2013-05-02 16:35:06 -05:00
d1a8b93b3f
1) druid-services -> services
2013-05-01 17:37:38 -05:00
fjy
Xavier Léauté
Gian Merlino
fjy
Himadri Singh
cheddar
Russell Jurney
Steve Harris