Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,908 Commits 113 Branches 506 Tags 732 MiB
Commit Graph

51 Commits

Author SHA1 Message Date
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
cheddar 5712b29c8c Fix issues with bindings and handling extensions 
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about.  This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need.  Guice tries to fail-fast and verifies that all
 bindings exist before it does anything, which is a problem because the extension bind
 some objects that don't necessarily have all of their dependencies bound in all processes.

The fix for this is to build a single Injector with all bindings in it and let each of the
 processes only load the things that they care about.  This also requires the use of
 Module overrides and other such interesting things, which are node done.

 In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided.  This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.

 In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
 2013-09-20 17:45:01 -05:00
fjy cabae7993d port over multi threaded realtime and also fix broken realtime nodes that can't start up 2013-09-16 16:03:47 -07:00
cheddar 3c39f90c89 1) Move Firehose interface and dependencies to druid-api 
2) Move DataSegment* interfaces and dependencies to druid-api
 2013-08-31 16:43:28 -05:00
cheddar 5ab671050e No more com.metamx.druid, it is now all io.druid! 2013-08-30 19:42:12 -05:00
cheddar bd0756e360 More stuff moved, things still compiling and tests still passing. Yay! 2013-08-30 18:58:35 -05:00
cheddar 56e2b956d0 OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.! 2013-08-30 18:21:04 -05:00
cheddar 9c30ced5ea 1) Move various "api" classes to io.druid packages and make sure things compile and stuff 2013-08-28 15:51:02 -05:00
fjy 261ef7ce56 add some fixes 2013-08-22 10:56:50 -07:00
fjy 6a8c160740 update code according to code review 2013-08-22 10:46:05 -07:00
fjy 85ee8bb267 port realtime to guice 2013-08-13 17:08:45 -07:00
cheddar 2361e0112a Make it all compile again... 2013-08-02 10:14:46 -07:00
Dhruv Parthasarathy 05219689e8 added close methods on inputSupplier tests 2013-07-25 16:52:29 -07:00
Stefán Freyr Stefánsson 2842a40894 Fixed test that was failing on JDK 1.7 2013-07-24 23:41:47 +00:00
Stefán Freyr Stefánsson ae4132adba Adding a test producer application. 2013-07-18 19:42:07 +00:00
cheddar c519be5b9f 1) Fix stray imports from libraries that aren't depended on and fix annotations/methods accordingly. 2013-07-10 14:19:04 -07:00
cheddar 3778425250 1) Fix new WebStreamFirehose stuff to have tests in the tests directory 
2) Change the webStream package to the web package, because capital letters in packages suck
 2013-07-10 14:12:13 -07:00
cheddar b83bc14784 Merge pull request #173 from metamx/dhruv 
Add new demo firehose that is lower friction than twitter
 2013-07-09 17:39:51 -07:00
Dhruv Parthasarathy bd6dcd3973 interruption! 2013-07-09 17:39:19 -07:00
Dhruv Parthasarathy ba484fca5c catch error 2013-07-09 17:17:21 -07:00
Dhruv Parthasarathy 9f7284c801 fixed thread stuff 2013-07-09 16:59:49 -07:00
Dhruv Parthasarathy b3157c2752 made thread final 2013-07-09 16:42:35 -07:00
Dhruv Parthasarathy 3250c698bb fixed thread stuff and made tests cleaner 2013-07-09 16:34:55 -07:00
Dhruv Parthasarathy 6d000fc4c2 interfaces added and tests simplified 2013-07-09 15:31:03 -07:00
Dhruv Parthasarathy 41cb115d60 few more changes to RenamingKeysUpdateStreamFactory and test 2013-07-09 10:46:43 -07:00
Dhruv Parthasarathy 72fbc516bc added a RenameKeysUpdateStream wrapper 2013-07-08 18:53:08 -07:00
Dhruv Parthasarathy 142271aad2 better encapsulation 2013-07-08 16:46:47 -07:00
Dhruv Parthasarathy 01b4728c40 removed shared queue structure. Queue now encapsulated within updateStream 2013-07-08 14:27:35 -07:00
Dhruv Parthasarathy e7da31e42d seems to be working 2013-07-08 13:41:19 -07:00
Dhruv Parthasarathy 439e8ca4ad now take a map for renaming 2013-07-08 13:35:22 -07:00
Dhruv Parthasarathy c8c686c738 added druid license comments 2013-07-08 12:19:38 -07:00
Dhruv Parthasarathy adc00e6576 fixed with Erics suggestions 2013-07-08 12:13:25 -07:00
Dhruv Parthasarathy 6f6bf1996c log.info changed to log.debug 2013-06-27 19:09:35 -07:00
Dhruv Parthasarathy c15f18c343 removed NoopRunnable 2013-06-27 19:03:25 -07:00
Dhruv Parthasarathy 80f4ae25b5 updated with further cleanliness 2013-06-27 19:00:34 -07:00
Dhruv Parthasarathy 8bc4d7c436 added druid header 2013-06-26 15:16:21 -07:00
Dhruv Parthasarathy 235ecce522 fixed constants and made them static finals 2013-06-26 15:12:55 -07:00
Dhruv Parthasarathy e3f0b8770b updated with Fangjin comments 2013-06-26 15:05:52 -07:00
Dhruv Parthasarathy 039b06d646 made sure only edited files were pushed 2013-06-25 15:01:19 -07:00
Dhruv Parthasarathy 0e1857f046 Applied codestyle and optimized imports 2013-06-25 14:48:02 -07:00
Dhruv Parthasarathy 8750ee88f2 Webstream example with tests 2013-06-25 14:37:58 -07:00
fjy e242baacf4 Merge branch 'master' into batch-zk 2013-06-20 16:21:33 -07:00
Dhruv Parthasarathy 748695bf97 trying to recover my files on intellij 2013-06-20 14:59:52 -07:00
fjy caa68e101a first commit; things working right now 2013-06-19 15:56:45 -07:00
Russell Jurney 58780094a9 Upgraded to twitter4j 3.0.3, twitter example now works with twitter4j.properties in place that lists key pairs. 2013-06-13 17:59:17 -07:00
Brian O'Neill 10a96626d4 Documentation for C* 2013-05-07 16:53:12 -04:00
Brian O'Neill 863b8808cc Merge branch 'master' of https://github.com/metamx/druid into cassandra-segments 2013-05-07 11:36:49 -04:00
Brian O'Neill 79d802fede Working data segment pushed for C*. 2013-05-06 17:54:12 -04:00
Gian Merlino 8948c8d4ca Merge branch 'master' into event-push 
Conflicts:
	client/src/main/java/com/metamx/druid/initialization/Initialization.java
	examples/src/main/java/druid/examples/twitter/TwitterSpritzerFirehoseFactory.java
	index-common/src/main/java/com/metamx/druid/indexer/data/StringInputRowParser.java
	merger/src/main/java/com/metamx/druid/merger/common/task/RealtimeIndexTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/executor/ExecutorNode.java
 2013-05-02 16:34:14 +03:00
cheddar e1a2c43baf 1) Adjust the example code to all be in one repository. 
2) Adjust the run scripts to be able to switch between different examples.
 2013-05-01 12:51:26 -05:00