6192602893
fix extensions config not getting picked up in hadoop index task
2013-10-16 16:52:23 -07:00
a1c09df17f
make the hadoop index task work again
2013-10-16 09:45:17 -07:00
9796a40b92
port docs over to 0.6 and a bunch of misc fixes
2013-10-11 18:38:53 -07:00
4e509d1d09
Merge branch 'master' into is-docs
2013-10-09 14:05:10 -07:00
c47fe202c7
Fix HadoopDruidIndexer to work with the new way of things
...
There are multiple and sundry changes in here.
First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.
In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module. I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.
In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api. They are all API-level things, however, so they really belong in druid-api instead.
Lastly, I removed the druid-realtime module and put it all in druid-server.
2013-10-09 15:15:44 -05:00
4ec4b8e024
rewrite indexing service docs
2013-10-08 16:34:58 -07:00
703b674800
add availability zone info to autoscaling
2013-10-07 12:16:50 -07:00
ac330f72bb
first set of changes to standarize the naming convention we use in druid
2013-10-03 16:36:48 -07:00
17874eeb67
make the CliPeon actually able to run on its own
2013-10-02 15:55:10 -07:00
bc8db7daa5
1) make chat handler resource work again
...
2) add more default configs
3) make examples work again
2013-10-02 14:22:39 -07:00
384dcda7e4
Chat handlers still don't work, but, they're closer maybe.
2013-10-01 17:45:53 -07:00
59f2d0711d
Merge branch 'master' into local-index
2013-10-01 13:21:43 -07:00
30df53671e
remove line iterator factory because it is not needed
2013-10-01 13:21:20 -07:00
62eda5020c
ShardSpec: Remove isInChunk(Map<String, String>)
2013-10-01 12:50:08 -07:00
5d0d71250b
fix chat handler resources not correctly registering themselves
2013-10-01 11:25:39 -07:00
53698a135a
add interface to new firehose as per code review comments
2013-09-30 18:00:59 -07:00
f55a5199b1
add a firehose module to remove so much copy and pasted code
2013-09-30 16:29:20 -07:00
ed9e0cf9f6
add a local firehose for indexing local files
2013-09-30 16:03:26 -07:00
dc5dab8747
Fixes for property conversion, firehose registration, and the indexing service
2013-09-27 17:09:59 -07:00
0b04325ee8
fix things up according to code review comments
2013-09-27 10:17:45 -07:00
e404295c1f
make indexing service work
2013-09-26 17:44:21 -07:00
8bc56daa66
fix things up according to code review comments
2013-09-26 11:35:45 -07:00
87259321b6
port hadoop druid indexer to new guice framework
2013-09-26 11:04:42 -07:00
15843c3978
refactor how server service discovery is done
2013-09-24 10:36:26 -07:00
dc8a119787
fix broken unit tests are a result of the last merge
2013-09-23 12:56:01 -07:00
5712b29c8c
Fix issues with bindings and handling extensions
...
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about. This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need. Guice tries to fail-fast and verifies that all
bindings exist before it does anything, which is a problem because the extension bind
some objects that don't necessarily have all of their dependencies bound in all processes.
The fix for this is to build a single Injector with all bindings in it and let each of the
processes only load the things that they care about. This also requires the use of
Module overrides and other such interesting things, which are node done.
In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided. This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.
In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
2013-09-20 17:45:01 -05:00
cabae7993d
port over multi threaded realtime and also fix broken realtime nodes that can't start up
2013-09-16 16:03:47 -07:00
f7c10e3594
rework tests in indexing service to be more unit testy
2013-09-12 16:37:58 -07:00
a2dcc45a8e
1) Remove SingleSegmentLoader and replace with OmniSegmentLoader
2013-09-12 11:47:03 -05:00
6c9a107356
1) remove duplicate package initialization.initialization
2013-09-09 17:02:57 -05:00
3c39f90c89
1) Move Firehose interface and dependencies to druid-api
...
2) Move DataSegment* interfaces and dependencies to druid-api
2013-08-31 16:43:28 -05:00
5ab671050e
No more com.metamx.druid, it is now all io.druid!
2013-08-30 19:42:12 -05:00
bd0756e360
More stuff moved, things still compiling and tests still passing. Yay!
2013-08-30 18:58:35 -05:00
56e2b956d0
OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.!
2013-08-30 18:21:04 -05:00
cb90ed05b0
Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper.
2013-08-29 16:45:03 -05:00
2a46086e20
1) Didn't remove the io.druid files from client. Remove those and make sure things compile
...
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
2013-08-29 15:25:36 -05:00
9c30ced5ea
1) Move various "api" classes to io.druid packages and make sure things compile and stuff
2013-08-28 15:51:02 -05:00
ee1e73cfa1
1) Make it compile again after the merge
2013-08-27 14:36:01 -05:00
8097450d8c
Some things that didn't get committed with the merge for some reason!?
2013-08-27 14:29:03 -05:00
5fa944dd26
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
pom.xml
server/src/main/java/com/metamx/druid/http/MasterMain.java
server/src/main/java/com/metamx/druid/http/MasterServletModule.java
server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
2013-08-27 14:27:32 -05:00
3617ac17fc
1) Eliminate ExecutorMain and have it run using the new Main!
2013-08-27 14:11:05 -05:00
269997dc94
1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful
2013-08-26 18:08:41 -05:00
6636ef1ea8
Remove unused files again
2013-08-23 18:00:56 -05:00
55dbda2046
1) Worker appears to be running! It's also now known as the MiddleManager
2013-08-23 17:59:48 -05:00
613ebd54b5
1) Delete unused things
2013-08-23 14:32:14 -05:00
b897c2cb22
1) IndexCoordinator appears to work as the CliOverlord now, yay!
2013-08-23 14:11:34 -05:00
d92ab8bb58
more logs for RTR
2013-08-21 21:47:59 -07:00
54f00479cc
add explicit null check for moving tasks from pending to running
2013-08-21 13:02:35 -07:00
88661b26a0
bug fix for RTR removing workers race condition and partition chunks not being sorted by chunk number
2013-08-21 11:14:54 -07:00
e283de6831
fix another bug with RTR to remove things correctly from running tasks
2013-08-20 19:34:30 -07:00
d02be16245
fix RTR closing PCC too early
2013-08-20 19:25:16 -07:00
70ab225770
Add missing license headers
2013-08-20 17:50:10 -07:00
455645e723
Workers announce TaskAnnouncement rather than TaskStatus
2013-08-20 16:14:36 -07:00
9609314765
ForkingTaskRunner: Make TaskInfo into ForkingTaskRunnerWorkItem
...
This allows the API/GUI to return reasonable results when the primary
task runner is a ForkingTaskRunner.
2013-08-20 14:04:28 -07:00
4e8325f963
Better tests and error messages for TaskResource
2013-08-20 14:01:38 -07:00
b102d67173
Fix getResourceManagementScheduler for non-autoscaling configs
2013-08-20 13:39:05 -07:00
25e330780c
Simplify AbstractTask constructor
2013-08-20 13:38:52 -07:00
d8493f8e26
RealtimeIndexTask: Fix "resource" serde
2013-08-20 13:02:52 -07:00
5c600c7012
change error msg to alert
2013-08-15 13:27:41 -07:00
4e7dac18b9
fix condition where status would be returned even if worker not running task
2013-08-15 13:12:57 -07:00
1fb6107a37
fix the case where RTR does not clean up a completed task on startup
2013-08-15 13:09:02 -07:00
8d7a4f4493
Retries for S3TaskLogs, S3DataSegmentPusher
2013-08-12 14:27:34 -07:00
df883a9823
learn to type
2013-08-06 21:18:36 -07:00
795657aedf
fix bug where workers with same capacity would not be unique
2013-08-06 21:04:13 -07:00
ebf1ac47f0
Merge branch 'master' of github.com:metamx/druid
2013-08-06 15:38:25 -07:00
9d0e4a94f0
alert when task fails in RTR assign
2013-08-06 15:38:17 -07:00
4a64ce37ed
Finish the merging, wtf IntelliJ?
2013-08-06 13:34:24 -07:00
eee1efdcb5
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/client/DruidServerConfig.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/index/ChatHandlerProvider.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/TaskMasterLifecycle.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
2013-08-06 13:33:31 -07:00
a1904c9b3b
ChatHandlerResource: Fix Guice type errors
2013-08-05 19:56:03 -07:00
d1b2a5a4b3
fix indexer console serde of running tasks
2013-08-05 18:22:12 -07:00
479f0cefca
fix bug with RTR not assigning tasks when a new worker is available
2013-08-05 17:57:59 -07:00
626cf14a6e
fix bug where the curator config name was changed in one place but not another; make some info msgs into debug msgs; fix zkworker serialization
2013-08-05 16:02:26 -07:00
66c658305f
Merge branch 'master' of github.com:metamx/druid
2013-08-05 14:44:09 -07:00
35f89d7232
make RTR idempotent to multiple run requests for same task, because higher level things in the indexing service require this behaviour
2013-08-05 14:44:01 -07:00
efd34f3a8b
TaskRunner: Fix comment
2013-08-05 14:20:31 -07:00
2361e0112a
Make it all compile again...
2013-08-02 10:14:46 -07:00
c33f2f06ff
fix logic of how to assign tasks to workers
2013-08-02 09:01:02 -07:00
584ccac833
move scanning of workers and tasks into RTR start, simplify bootstrap, make tests better
2013-08-01 17:50:05 -07:00
9e78bb38f5
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/QueryableNode.java
client/src/main/java/com/metamx/druid/client/ServerInventoryView.java
client/src/main/java/com/metamx/druid/coordination/SingleDataSegmentAnnouncer.java
client/src/main/java/com/metamx/druid/initialization/CuratorDiscoveryConfig.java
client/src/main/java/com/metamx/druid/query/MetricsEmittingExecutorService.java
indexing-hadoop/src/test/java/com/metamx/druid/indexer/HadoopDruidIndexerConfigTest.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/http/IndexerCoordinatorNode.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/http/WorkerNode.java
pom.xml
server/src/main/java/com/metamx/druid/coordination/ServerManager.java
server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
server/src/main/java/com/metamx/druid/db/DatabaseRuleManager.java
server/src/main/java/com/metamx/druid/db/DatabaseSegmentManager.java
server/src/main/java/com/metamx/druid/http/ComputeNode.java
server/src/main/java/com/metamx/druid/http/MasterMain.java
server/src/main/java/com/metamx/druid/loading/SegmentLoaderConfig.java
server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-08-01 16:42:47 -07:00
019bb5d453
1) Another whole bunch of changes to annotate things and create Modules and bind stuff. But OMFG, the compute node actually appears to be working!
...
2) The compute node works with Guice
3) The compute node fires up with Guice and appears to work
4) Did I mention that the compute node, now called historical node, fires up with Guice and appears to work?
2013-08-01 15:28:08 -07:00
a4edc2221d
fix RTR comments
2013-07-31 15:28:52 -07:00
215d147a69
Merge branch 'worker-resource' of github.com:metamx/druid into worker-resource
2013-07-31 15:23:49 -07:00
e2b5cd6067
Merge branch 'master' of github.com:metamx/druid into worker-resource
...
Conflicts:
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
2013-07-31 15:23:13 -07:00
eaddce06d5
Call TaskRunner.bootstrap immediately after starting it
2013-07-30 15:26:11 -07:00
50836798fa
toggle between compressed and non compressed service discovery
2013-07-29 15:40:45 -07:00
ad65c8111d
fix logs
2013-07-29 11:41:42 -07:00
4ae8395538
1) on bootstrap, load all initial data and do a compare with bootstrapped tasks, delete any that are extra out there
...
2) change autoscaling logic such that it only works with remote task runnrs
3) zk workers use their own status caches to determine what they are running
2013-07-26 14:32:08 -07:00
6b9963b472
1) Move the @Json and @Smile annotations to com.metamx.druid.guice.annotations
...
2) Changed ServiceAnnouncer to take a DruidNode object
3) Remove ServiceInstanceFactory interface and implementations
4) Add registrations to DiscoveryModule so that you can register nodes that should be announced on startup
5) Make the set of default Monitors configurable
2013-07-26 14:03:51 -07:00
a1262760b2
Merge branch 'master' into worker-resource
2013-07-26 10:02:15 -07:00
4dd42c7177
RemoteTaskActionClient: Log service uri when submitting action
2013-07-25 14:48:25 -07:00
5d44f0f15b
ForkingTaskRunner: Use guava Closer to closer stuff
2013-07-25 14:37:45 -07:00
952b8ce06b
RemoteTaskActionClient: Fix http path
2013-07-25 14:37:20 -07:00
f38d7cf964
Fix busted condition
2013-07-24 16:33:01 -07:00
6d1cb1bfc1
IndexerCoordinatorResource: Log when remote task action fails
2013-07-24 16:13:12 -07:00
30d98f56c1
RealtimeIndexTask: Allow configurable rejection policies
2013-07-24 16:12:54 -07:00
0eebb0a149
Add RealtimeMetricsMonitor to RealtimeIndexTask
2013-07-24 14:39:59 -07:00
71704f7ee8
Replace graceful/hard shutdown combo with simple exit
2013-07-24 10:27:21 -07:00
b9578a1ada
1) remove retry logic from RTR
...
2) simplify configs
3) introduce task resource
4) make worker versions match coordinator version by default
2013-07-12 12:51:12 -07:00
d3eb491155
Merge branch 'batch-zk' into worker-resource
...
Conflicts:
pom.xml
server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
2013-07-02 16:24:59 -07:00
e21aa41bdd
Merge branch 'master' into worker-resource
...
Conflicts:
server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
2013-07-02 16:23:53 -07:00
5ac2d1db0b
Merge branch 'master' of github.com:metamx/druid into batch-zk
...
Conflicts:
server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
2013-07-02 15:57:31 -07:00
7219ed15d3
fix according to code review
2013-07-02 15:56:12 -07:00
797a083b69
1) Adjust SingleSegmentLoader to allow for storing segments on multiple different mount points. The specification language is really janky right now, so this is remaining a stealth feature for the time being.
2013-07-01 14:56:57 -07:00
a6787ab201
fix up configs
2013-06-27 16:40:17 -07:00
caa68e101a
first commit; things working right now
2013-06-19 15:56:45 -07:00
11ea15fc1a
1) Introduce Jetty 8
...
2) Fix up modules based on exceptions at startup for MasterMain
2013-06-17 10:53:50 -07:00
2f56c24259
1) Inject IndexingServiceClient
...
2) Switch all the DBI references to IDBI
2013-06-07 17:37:33 -07:00
37a3994003
add delegating executor service and fix bug with query priortization
2013-06-07 14:08:51 -07:00
f68df7ab69
1) Make tests work and continue trying to make the DruidMaster start up with just Guice
2013-06-07 12:01:46 -07:00
d69fb61a6e
Fix getImplicitLockInterval check
2013-06-06 18:22:41 -07:00
14f9b0409e
fix pushing down too many properties
2013-06-06 17:58:40 -07:00
9df458a065
1) Initial commit of conversion to using Guice modules for bootstrapping. Things don't actually completely work yet.
2013-06-06 15:43:15 -07:00
e4ea357b52
multiple bug fixes for indexing service; scv quotes
2013-06-06 14:10:18 -07:00
451d3d358b
some fixes according to code review comments
2013-06-06 11:02:08 -07:00
06931ee0f5
introduce availability groups
2013-06-04 17:12:19 -07:00
42cc87a294
Merge branch 'master' into refactor-indexing
...
Conflicts:
indexing-service/src/main/java/com/metamx/druid/indexing/common/task/IndexTask.java
pom.xml
2013-05-31 17:28:59 -07:00
c6ad0753f0
bug fixes
2013-05-16 16:02:20 -07:00
6213c0b63c
Merge branch 'master' into refactor-indexing
2013-05-15 17:14:40 -07:00
20ae1d8b6b
lots of cleanups and refactorings
2013-05-15 15:37:04 -07:00
fjy
cheddar
Gian Merlino