Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,342 Commits 111 Branches 503 Tags 728 MiB
Commit Graph

121 Commits

Author SHA1 Message Date
fjy d49554185a fix broken default table names 2013-11-07 17:38:00 -08:00
fjy 560981acbf a ton of fixes to docs 2013-10-10 15:05:01 -07:00
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
fjy bc8db7daa5 1) make chat handler resource work again 
2) add more default configs
3) make examples work again
 2013-10-02 14:22:39 -07:00
fjy 87259321b6 port hadoop druid indexer to new guice framework 2013-09-26 11:04:42 -07:00
cheddar 5712b29c8c Fix issues with bindings and handling extensions 
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about.  This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need.  Guice tries to fail-fast and verifies that all
 bindings exist before it does anything, which is a problem because the extension bind
 some objects that don't necessarily have all of their dependencies bound in all processes.

The fix for this is to build a single Injector with all bindings in it and let each of the
 processes only load the things that they care about.  This also requires the use of
 Module overrides and other such interesting things, which are node done.

 In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided.  This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.

 In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
 2013-09-20 17:45:01 -05:00
fjy 64f3821b4e throw exception for annotated members 2013-09-16 17:36:31 -07:00
cheddar 3e4a4f5566 1) Adjustments to allow for the addition of new Runnables via modules. Modules should implement CliCommandCreator and add their stuff there. 2013-09-11 17:41:20 -05:00
cheddar dadd9bc5ce 1) Rename MetricSelector(Factory) to ColumnSelector(Factory) 
2) Hard coded plugin as a first pass, won't actually work for anyone else.
 2013-09-04 11:19:46 -05:00
cheddar 3c39f90c89 1) Move Firehose interface and dependencies to druid-api 
2) Move DataSegment* interfaces and dependencies to druid-api
 2013-08-31 16:43:28 -05:00
cheddar 5ab671050e No more com.metamx.druid, it is now all io.druid! 2013-08-30 19:42:12 -05:00
cheddar bd0756e360 More stuff moved, things still compiling and tests still passing. Yay! 2013-08-30 18:58:35 -05:00
cheddar 56e2b956d0 OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.! 2013-08-30 18:21:04 -05:00
cheddar ca2c56d827 1) Remove PostAggregator and put it in druid-api 
2) Remove io.druid files from client
 2013-08-29 16:50:09 -05:00
cheddar cb90ed05b0 Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper. 2013-08-29 16:45:03 -05:00
cheddar 2a46086e20 1) Didn't remove the io.druid files from client. Remove those and make sure things compile 
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
 2013-08-29 15:25:36 -05:00
cheddar 1f37e962f6 1) Delete the files that have moved to druid-api and get them through the druid-api dependency. Still need to fix Searchable on StorageAdapter, but other than that, everything should compile. 2013-08-28 16:18:40 -05:00
cheddar bfe96739a3 1) Move various "api" classes to io.druid packages and make sure things compile and stuff -- continued! Come classes didn't get committed? 2013-08-28 15:52:55 -05:00
cheddar 9c30ced5ea 1) Move various "api" classes to io.druid packages and make sure things compile and stuff 2013-08-28 15:51:02 -05:00
cheddar 5fa944dd26 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
	client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
	client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	pom.xml
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/http/MasterServletModule.java
	server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
	server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
	server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
 2013-08-27 14:27:32 -05:00
cheddar 269997dc94 1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful 2013-08-26 18:08:41 -05:00
cheddar 55dbda2046 1) Worker appears to be running! It's also now known as the MiddleManager 2013-08-23 17:59:48 -05:00
cheddar b897c2cb22 1) IndexCoordinator appears to work as the CliOverlord now, yay! 2013-08-23 14:11:34 -05:00
Gian Merlino 9b9a2a9f53 Merge branch 'master' into num-shard-spec 
Conflicts:
	common/src/main/java/com/metamx/druid/partition/IntegerPartitionChunk.java
 2013-08-21 15:01:46 -07:00
fjy 54ad854471 Merge branch 'master' into rt-ut 2013-08-21 13:15:02 -07:00
fjy 88661b26a0 bug fix for RTR removing workers race condition and partition chunks not being sorted by chunk number 2013-08-21 11:14:54 -07:00
Gian Merlino 9c5fcc0984 Add missing license headers 2013-08-20 17:48:15 -07:00
Gian Merlino 8388cbf69d NumberedShardSpec 
Each shard has a number, and there are a fixed number of shards that will
form a complete set.
 2013-08-20 16:45:12 -07:00
fjy 9c145f5ce3 basic unit tets for realtime functinality 2013-08-19 15:00:10 -07:00
fjy 5f292143e0 disable merging segments with linear shard spec 2013-08-06 16:17:49 -07:00
cheddar d66af7625c 1) Eliminate the need for SingleSegmentLoader 
2) Setup the configuration glue for ServerInventoryViews and DataSegmentAnnouncer
3) Make processes run and work again!
 2013-08-02 17:05:01 -07:00
cheddar 9e78bb38f5 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
	client/src/main/java/com/metamx/druid/client/ServerInventoryView.java
	client/src/main/java/com/metamx/druid/coordination/SingleDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/initialization/CuratorDiscoveryConfig.java
	client/src/main/java/com/metamx/druid/query/MetricsEmittingExecutorService.java
	indexing-hadoop/src/test/java/com/metamx/druid/indexer/HadoopDruidIndexerConfigTest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/http/IndexerCoordinatorNode.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/http/WorkerNode.java
	pom.xml
	server/src/main/java/com/metamx/druid/coordination/ServerManager.java
	server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
	server/src/main/java/com/metamx/druid/db/DatabaseRuleManager.java
	server/src/main/java/com/metamx/druid/db/DatabaseSegmentManager.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/loading/SegmentLoaderConfig.java
	server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-08-01 16:42:47 -07:00
cheddar 019bb5d453 1) Another whole bunch of changes to annotate things and create Modules and bind stuff. But OMFG, the compute node actually appears to be working! 
2) The compute node works with Guice
3) The compute node fires up with Guice and appears to work
4) Did I mention that the compute node, now called historical node, fires up with Guice and appears to work?
 2013-08-01 15:28:08 -07:00
cheddar f4fc8b2f6e 1) A whole boondoggle of changes to Guicify the instantiation of things for the Historical nodes 
2) Add GuiceInjectableValues that supports @JacksonInject resolution via Guice
3) Refactor RequestLogger stuff and setup polymorphic deserialization method of creating objects
 2013-07-31 18:36:47 -07:00
cheddar 9de2e5b096 1) Adjust JsonConfigurator to be able to attach the proper property path if it is renamed via a @JsonProperty annotation 
2) JsonConfigurator now requires that fields be annotated with @JsonProperty.  It will fail fast if they aren't.
 2013-07-31 14:55:34 -07:00
cheddar 97fb971f84 1) Start the journey of making the historical nodes work with Guice. Doesn't work yet, but at least it builds. 2013-07-30 11:58:05 -07:00
cheddar 6b9963b472 1) Move the @Json and @Smile annotations to com.metamx.druid.guice.annotations 
2) Changed ServiceAnnouncer to take a DruidNode object
3) Remove ServiceInstanceFactory interface and implementations
4) Add registrations to DiscoveryModule so that you can register nodes that should be announced on startup
5) Make the set of default Monitors configurable
 2013-07-26 14:03:51 -07:00
cheddar 02ffd805c8 1) Move DbConnector into its own module 
2) Move JacksonConfig stuff into its own module
3) Fix LogEmitterModule to inject properly with a Supplier<>
4) Set Guice to require explicit bindings, to protect from weirdness like was initially experienced with LoggingEmitterConfig
5) Master fires up and appears to all work with Guice!  Whoot!
 2013-07-23 14:55:36 -07:00
cheddar 85a35201b7 1) The Master compiles and runs with Guice bindings!!!! 
2) The service discovery stuff really needs to be reworked.  It's unhappy now.
 2013-07-19 16:29:54 -07:00
cheddar b83bc14784 Merge pull request #173 from metamx/dhruv 
Add new demo firehose that is lower friction than twitter
 2013-07-09 17:39:51 -07:00
cheddar b42c5743c9 Merge pull request #177 from nebrera/master 
Postgresql adaptation
 2013-07-05 16:42:57 -07:00
fjy fde874ea56 fix according to code review 2013-07-02 16:11:12 -07:00
Pablo Nebrera e9ce6b596c PostgresSQL integration with druid 2013-06-28 12:49:24 +02:00
Dhruv Parthasarathy c15f18c343 removed NoopRunnable 2013-06-27 19:03:25 -07:00
fjy 15a4dae2b1 bunch of issue fixes 2013-06-26 10:45:54 -07:00
Dhruv Parthasarathy 4e4a0a7953 removed DS_Store files 2013-06-25 14:58:12 -07:00
cheddar 149c2e3560 1) A bunch more changes to make things actually run. Not quite done, but close! 2013-06-20 17:19:51 -07:00
Dhruv Parthasarathy 748695bf97 trying to recover my files on intellij 2013-06-20 14:59:52 -07:00
cheddar 11ea15fc1a 1) Introduce Jetty 8 
2) Fix up modules based on exceptions at startup for MasterMain
 2013-06-17 10:53:50 -07:00
cheddar 2f56c24259 1) Inject IndexingServiceClient 
2) Switch all the DBI references to IDBI
 2013-06-07 17:37:33 -07:00