Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-09 03:24:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,934 Commits 116 Branches 507 Tags
Commit Graph

125 Commits

Author SHA1 Message Date
Gian Merlino
8660db93fc RemoteTaskRunner: Run taskComplete after a task times out 2013-11-04 10:56:26 -08:00
Gian Merlino
186bbd1cb6 WorkerTaskMonitor: Add log message 2013-11-04 10:25:16 -08:00
Gian Merlino
781673a8f8 ForkingTaskRunner: Fix pass-down of nodeType 2013-10-23 14:13:05 -07:00
Gian Merlino
b68b3526e8 IndexingServiceFirehoseModule: Add header 2013-10-18 13:27:40 -07:00
Gian Merlino
2e9c46867f Fixes for indexing service. 
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
 2013-10-18 11:14:33 -07:00
fjy
4862852b43 more docs about how to use different versions of hadoop in druid 2013-10-16 17:54:49 -07:00
fjy
6192602893 fix extensions config not getting picked up in hadoop index task 2013-10-16 16:52:23 -07:00
fjy
a1c09df17f make the hadoop index task work again 2013-10-16 09:45:17 -07:00
fjy
9796a40b92 port docs over to 0.6 and a bunch of misc fixes 2013-10-11 18:38:53 -07:00
fjy
4e509d1d09 Merge branch 'master' into is-docs 2013-10-09 14:05:10 -07:00
cheddar
c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
fjy
4ec4b8e024 rewrite indexing service docs 2013-10-08 16:34:58 -07:00
fjy
703b674800 add availability zone info to autoscaling 2013-10-07 12:16:50 -07:00
fjy
ac330f72bb first set of changes to standarize the naming convention we use in druid 2013-10-03 16:36:48 -07:00
fjy
17874eeb67 make the CliPeon actually able to run on its own 2013-10-02 15:55:10 -07:00
fjy
bc8db7daa5 1) make chat handler resource work again 
2) add more default configs
3) make examples work again
 2013-10-02 14:22:39 -07:00
Gian Merlino
384dcda7e4 Chat handlers still don't work, but, they're closer maybe. 2013-10-01 17:45:53 -07:00
fjy
59f2d0711d Merge branch 'master' into local-index 2013-10-01 13:21:43 -07:00
fjy
30df53671e remove line iterator factory because it is not needed 2013-10-01 13:21:20 -07:00
Gian Merlino
62eda5020c ShardSpec: Remove isInChunk(Map<String, String>) 2013-10-01 12:50:08 -07:00
fjy
5d0d71250b fix chat handler resources not correctly registering themselves 2013-10-01 11:25:39 -07:00
fjy
53698a135a add interface to new firehose as per code review comments 2013-09-30 18:00:59 -07:00
fjy
f55a5199b1 add a firehose module to remove so much copy and pasted code 2013-09-30 16:29:20 -07:00
fjy
ed9e0cf9f6 add a local firehose for indexing local files 2013-09-30 16:03:26 -07:00
Gian Merlino
dc5dab8747 Fixes for property conversion, firehose registration, and the indexing service 2013-09-27 17:09:59 -07:00
fjy
0b04325ee8 fix things up according to code review comments 2013-09-27 10:17:45 -07:00
fjy
e404295c1f make indexing service work 2013-09-26 17:44:21 -07:00
fjy
8bc56daa66 fix things up according to code review comments 2013-09-26 11:35:45 -07:00
fjy
87259321b6 port hadoop druid indexer to new guice framework 2013-09-26 11:04:42 -07:00
fjy
15843c3978 refactor how server service discovery is done 2013-09-24 10:36:26 -07:00
fjy
dc8a119787 fix broken unit tests are a result of the last merge 2013-09-23 12:56:01 -07:00
cheddar
5712b29c8c Fix issues with bindings and handling extensions 
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about.  This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need.  Guice tries to fail-fast and verifies that all
 bindings exist before it does anything, which is a problem because the extension bind
 some objects that don't necessarily have all of their dependencies bound in all processes.

The fix for this is to build a single Injector with all bindings in it and let each of the
 processes only load the things that they care about.  This also requires the use of
 Module overrides and other such interesting things, which are node done.

 In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided.  This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.

 In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
 2013-09-20 17:45:01 -05:00
fjy
cabae7993d port over multi threaded realtime and also fix broken realtime nodes that can't start up 2013-09-16 16:03:47 -07:00
fjy
f7c10e3594 rework tests in indexing service to be more unit testy 2013-09-12 16:37:58 -07:00
cheddar
a2dcc45a8e 1) Remove SingleSegmentLoader and replace with OmniSegmentLoader 2013-09-12 11:47:03 -05:00
cheddar
6c9a107356 1) remove duplicate package initialization.initialization 2013-09-09 17:02:57 -05:00
cheddar
3c39f90c89 1) Move Firehose interface and dependencies to druid-api 
2) Move DataSegment* interfaces and dependencies to druid-api
 2013-08-31 16:43:28 -05:00
cheddar
5ab671050e No more com.metamx.druid, it is now all io.druid! 2013-08-30 19:42:12 -05:00
cheddar
bd0756e360 More stuff moved, things still compiling and tests still passing. Yay! 2013-08-30 18:58:35 -05:00
cheddar
56e2b956d0 OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.! 2013-08-30 18:21:04 -05:00
cheddar
cb90ed05b0 Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper. 2013-08-29 16:45:03 -05:00
cheddar
2a46086e20 1) Didn't remove the io.druid files from client. Remove those and make sure things compile 
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
 2013-08-29 15:25:36 -05:00
cheddar
9c30ced5ea 1) Move various "api" classes to io.druid packages and make sure things compile and stuff 2013-08-28 15:51:02 -05:00
cheddar
ee1e73cfa1 1) Make it compile again after the merge 2013-08-27 14:36:01 -05:00
cheddar
8097450d8c Some things that didn't get committed with the merge for some reason!? 2013-08-27 14:29:03 -05:00
cheddar
5fa944dd26 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
	client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
	client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	pom.xml
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/http/MasterServletModule.java
	server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
	server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
	server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
 2013-08-27 14:27:32 -05:00
cheddar
3617ac17fc 1) Eliminate ExecutorMain and have it run using the new Main! 2013-08-27 14:11:05 -05:00
cheddar
269997dc94 1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful 2013-08-26 18:08:41 -05:00
cheddar
6636ef1ea8 Remove unused files again 2013-08-23 18:00:56 -05:00
cheddar
55dbda2046 1) Worker appears to be running! It's also now known as the MiddleManager 2013-08-23 17:59:48 -05:00