fjy
87259321b6
port hadoop druid indexer to new guice framework
2013-09-26 11:04:42 -07:00
fjy
15843c3978
refactor how server service discovery is done
2013-09-24 10:36:26 -07:00
fjy
dc8a119787
fix broken unit tests are a result of the last merge
2013-09-23 12:56:01 -07:00
cheddar
5712b29c8c
Fix issues with bindings and handling extensions
...
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about. This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need. Guice tries to fail-fast and verifies that all
bindings exist before it does anything, which is a problem because the extension bind
some objects that don't necessarily have all of their dependencies bound in all processes.
The fix for this is to build a single Injector with all bindings in it and let each of the
processes only load the things that they care about. This also requires the use of
Module overrides and other such interesting things, which are node done.
In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided. This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.
In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
2013-09-20 17:45:01 -05:00
fjy
cabae7993d
port over multi threaded realtime and also fix broken realtime nodes that can't start up
2013-09-16 16:03:47 -07:00
fjy
f7c10e3594
rework tests in indexing service to be more unit testy
2013-09-12 16:37:58 -07:00
cheddar
a2dcc45a8e
1) Remove SingleSegmentLoader and replace with OmniSegmentLoader
2013-09-12 11:47:03 -05:00
cheddar
6c9a107356
1) remove duplicate package initialization.initialization
2013-09-09 17:02:57 -05:00
cheddar
3c39f90c89
1) Move Firehose interface and dependencies to druid-api
...
2) Move DataSegment* interfaces and dependencies to druid-api
2013-08-31 16:43:28 -05:00
cheddar
5ab671050e
No more com.metamx.druid, it is now all io.druid!
2013-08-30 19:42:12 -05:00
cheddar
bd0756e360
More stuff moved, things still compiling and tests still passing. Yay!
2013-08-30 18:58:35 -05:00
cheddar
56e2b956d0
OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.!
2013-08-30 18:21:04 -05:00
cheddar
cb90ed05b0
Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper.
2013-08-29 16:45:03 -05:00
cheddar
2a46086e20
1) Didn't remove the io.druid files from client. Remove those and make sure things compile
...
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
2013-08-29 15:25:36 -05:00
cheddar
9c30ced5ea
1) Move various "api" classes to io.druid packages and make sure things compile and stuff
2013-08-28 15:51:02 -05:00
cheddar
ee1e73cfa1
1) Make it compile again after the merge
2013-08-27 14:36:01 -05:00
cheddar
8097450d8c
Some things that didn't get committed with the merge for some reason!?
2013-08-27 14:29:03 -05:00
cheddar
5fa944dd26
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
pom.xml
server/src/main/java/com/metamx/druid/http/MasterMain.java
server/src/main/java/com/metamx/druid/http/MasterServletModule.java
server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
2013-08-27 14:27:32 -05:00
cheddar
3617ac17fc
1) Eliminate ExecutorMain and have it run using the new Main!
2013-08-27 14:11:05 -05:00
cheddar
269997dc94
1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful
2013-08-26 18:08:41 -05:00
cheddar
6636ef1ea8
Remove unused files again
2013-08-23 18:00:56 -05:00
cheddar
55dbda2046
1) Worker appears to be running! It's also now known as the MiddleManager
2013-08-23 17:59:48 -05:00
cheddar
613ebd54b5
1) Delete unused things
2013-08-23 14:32:14 -05:00
cheddar
b897c2cb22
1) IndexCoordinator appears to work as the CliOverlord now, yay!
2013-08-23 14:11:34 -05:00
fjy
d92ab8bb58
more logs for RTR
2013-08-21 21:47:59 -07:00
fjy
54f00479cc
add explicit null check for moving tasks from pending to running
2013-08-21 13:02:35 -07:00
fjy
88661b26a0
bug fix for RTR removing workers race condition and partition chunks not being sorted by chunk number
2013-08-21 11:14:54 -07:00
fjy
e283de6831
fix another bug with RTR to remove things correctly from running tasks
2013-08-20 19:34:30 -07:00
fjy
d02be16245
fix RTR closing PCC too early
2013-08-20 19:25:16 -07:00
Gian Merlino
70ab225770
Add missing license headers
2013-08-20 17:50:10 -07:00
Gian Merlino
455645e723
Workers announce TaskAnnouncement rather than TaskStatus
2013-08-20 16:14:36 -07:00
Gian Merlino
9609314765
ForkingTaskRunner: Make TaskInfo into ForkingTaskRunnerWorkItem
...
This allows the API/GUI to return reasonable results when the primary
task runner is a ForkingTaskRunner.
2013-08-20 14:04:28 -07:00
Gian Merlino
4e8325f963
Better tests and error messages for TaskResource
2013-08-20 14:01:38 -07:00
Gian Merlino
b102d67173
Fix getResourceManagementScheduler for non-autoscaling configs
2013-08-20 13:39:05 -07:00
Gian Merlino
25e330780c
Simplify AbstractTask constructor
2013-08-20 13:38:52 -07:00
Gian Merlino
d8493f8e26
RealtimeIndexTask: Fix "resource" serde
2013-08-20 13:02:52 -07:00
fjy
5c600c7012
change error msg to alert
2013-08-15 13:27:41 -07:00
fjy
4e7dac18b9
fix condition where status would be returned even if worker not running task
2013-08-15 13:12:57 -07:00
fjy
1fb6107a37
fix the case where RTR does not clean up a completed task on startup
2013-08-15 13:09:02 -07:00
Gian Merlino
8d7a4f4493
Retries for S3TaskLogs, S3DataSegmentPusher
2013-08-12 14:27:34 -07:00
fjy
df883a9823
learn to type
2013-08-06 21:18:36 -07:00
fjy
795657aedf
fix bug where workers with same capacity would not be unique
2013-08-06 21:04:13 -07:00
fjy
ebf1ac47f0
Merge branch 'master' of github.com:metamx/druid
2013-08-06 15:38:25 -07:00
fjy
9d0e4a94f0
alert when task fails in RTR assign
2013-08-06 15:38:17 -07:00
cheddar
4a64ce37ed
Finish the merging, wtf IntelliJ?
2013-08-06 13:34:24 -07:00
cheddar
eee1efdcb5
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/client/DruidServerConfig.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/index/ChatHandlerProvider.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/TaskMasterLifecycle.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
2013-08-06 13:33:31 -07:00
Gian Merlino
a1904c9b3b
ChatHandlerResource: Fix Guice type errors
2013-08-05 19:56:03 -07:00
fjy
d1b2a5a4b3
fix indexer console serde of running tasks
2013-08-05 18:22:12 -07:00
fjy
479f0cefca
fix bug with RTR not assigning tasks when a new worker is available
2013-08-05 17:57:59 -07:00
fjy
626cf14a6e
fix bug where the curator config name was changed in one place but not another; make some info msgs into debug msgs; fix zkworker serialization
2013-08-05 16:02:26 -07:00
