62c55eaf26
fix broken CombiningSequence.toYielder behavior
2014-05-14 17:08:15 -07:00
e0ff2aa0d6
make isPostgreSQL check URI instead of metadata
2014-03-27 12:13:10 -07:00
fa34f2ac59
more consistent placeholder references
2014-03-26 16:41:27 -07:00
e84128ebbe
remove database specific code from ConfigManager
2014-03-26 14:29:37 -07:00
aec6457a4d
fix config manager
2014-03-26 09:42:33 -07:00
70db460f97
Blocking Executors and maxPendingPersists, oh my!
...
- Execs.newBlockingSingleThreaded can now accept capacity = 0.
- Changed default maxPendingPersists from 2 to 0.
- Fixed serde of maxPendingPersists in RealtimeIndexTasks.
2014-03-05 10:55:12 -08:00
e40725d5f3
Added support for PostgreSQL on overlord nodes
2014-02-26 01:24:37 +01:00
a835db2a3c
Added support for task storage in PostgreSQL
2014-02-26 01:24:37 +01:00
d089f05127
Improved support for PostgreSQL in config management
2014-02-17 18:11:15 +00:00
8fac0b0b7a
Fixed automatic tables creation for PostgreSQL
2014-02-17 18:11:15 +00:00
53c5124b99
DbConnector: Indexes on tasks table
2014-02-05 15:59:14 -08:00
5c043803c6
review comments
2014-01-20 13:40:36 +05:30
874cdb1b5b
review comments
2014-01-17 12:51:28 +05:30
3db33c3268
review comments
2014-01-17 12:51:10 +05:30
6f5e1afbff
add blocking mechanism after 2 pending persist batches
2014-01-17 03:01:04 +05:30
d49554185a
fix broken default table names
2013-11-07 17:38:00 -08:00
560981acbf
a ton of fixes to docs
2013-10-10 15:05:01 -07:00
c47fe202c7
Fix HadoopDruidIndexer to work with the new way of things
...
There are multiple and sundry changes in here.
First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.
In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module. I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.
In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api. They are all API-level things, however, so they really belong in druid-api instead.
Lastly, I removed the druid-realtime module and put it all in druid-server.
2013-10-09 15:15:44 -05:00
bc8db7daa5
1) make chat handler resource work again
...
2) add more default configs
3) make examples work again
2013-10-02 14:22:39 -07:00
87259321b6
port hadoop druid indexer to new guice framework
2013-09-26 11:04:42 -07:00
5712b29c8c
Fix issues with bindings and handling extensions
...
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about. This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need. Guice tries to fail-fast and verifies that all
bindings exist before it does anything, which is a problem because the extension bind
some objects that don't necessarily have all of their dependencies bound in all processes.
The fix for this is to build a single Injector with all bindings in it and let each of the
processes only load the things that they care about. This also requires the use of
Module overrides and other such interesting things, which are node done.
In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided. This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.
In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
2013-09-20 17:45:01 -05:00
64f3821b4e
throw exception for annotated members
2013-09-16 17:36:31 -07:00
3e4a4f5566
1) Adjustments to allow for the addition of new Runnables via modules. Modules should implement CliCommandCreator and add their stuff there.
2013-09-11 17:41:20 -05:00
dadd9bc5ce
1) Rename MetricSelector(Factory) to ColumnSelector(Factory)
...
2) Hard coded plugin as a first pass, won't actually work for anyone else.
2013-09-04 11:19:46 -05:00
3c39f90c89
1) Move Firehose interface and dependencies to druid-api
...
2) Move DataSegment* interfaces and dependencies to druid-api
2013-08-31 16:43:28 -05:00
5ab671050e
No more com.metamx.druid, it is now all io.druid!
2013-08-30 19:42:12 -05:00
bd0756e360
More stuff moved, things still compiling and tests still passing. Yay!
2013-08-30 18:58:35 -05:00
56e2b956d0
OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.!
2013-08-30 18:21:04 -05:00
ca2c56d827
1) Remove PostAggregator and put it in druid-api
...
2) Remove io.druid files from client
2013-08-29 16:50:09 -05:00
cb90ed05b0
Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper.
2013-08-29 16:45:03 -05:00
2a46086e20
1) Didn't remove the io.druid files from client. Remove those and make sure things compile
...
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
2013-08-29 15:25:36 -05:00
1f37e962f6
1) Delete the files that have moved to druid-api and get them through the druid-api dependency. Still need to fix Searchable on StorageAdapter, but other than that, everything should compile.
2013-08-28 16:18:40 -05:00
bfe96739a3
1) Move various "api" classes to io.druid packages and make sure things compile and stuff -- continued! Come classes didn't get committed?
2013-08-28 15:52:55 -05:00
9c30ced5ea
1) Move various "api" classes to io.druid packages and make sure things compile and stuff
2013-08-28 15:51:02 -05:00
5fa944dd26
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
pom.xml
server/src/main/java/com/metamx/druid/http/MasterMain.java
server/src/main/java/com/metamx/druid/http/MasterServletModule.java
server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
2013-08-27 14:27:32 -05:00
269997dc94
1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful
2013-08-26 18:08:41 -05:00
55dbda2046
1) Worker appears to be running! It's also now known as the MiddleManager
2013-08-23 17:59:48 -05:00
b897c2cb22
1) IndexCoordinator appears to work as the CliOverlord now, yay!
2013-08-23 14:11:34 -05:00
9b9a2a9f53
Merge branch 'master' into num-shard-spec
...
Conflicts:
common/src/main/java/com/metamx/druid/partition/IntegerPartitionChunk.java
2013-08-21 15:01:46 -07:00
54ad854471
Merge branch 'master' into rt-ut
2013-08-21 13:15:02 -07:00
88661b26a0
bug fix for RTR removing workers race condition and partition chunks not being sorted by chunk number
2013-08-21 11:14:54 -07:00
9c5fcc0984
Add missing license headers
2013-08-20 17:48:15 -07:00
8388cbf69d
NumberedShardSpec
...
Each shard has a number, and there are a fixed number of shards that will
form a complete set.
2013-08-20 16:45:12 -07:00
9c145f5ce3
basic unit tets for realtime functinality
2013-08-19 15:00:10 -07:00
5f292143e0
disable merging segments with linear shard spec
2013-08-06 16:17:49 -07:00
d66af7625c
1) Eliminate the need for SingleSegmentLoader
...
2) Setup the configuration glue for ServerInventoryViews and DataSegmentAnnouncer
3) Make processes run and work again!
2013-08-02 17:05:01 -07:00
9e78bb38f5
Merge branch 'master' into guice
...
Conflicts:
client/src/main/java/com/metamx/druid/QueryableNode.java
client/src/main/java/com/metamx/druid/client/ServerInventoryView.java
client/src/main/java/com/metamx/druid/coordination/SingleDataSegmentAnnouncer.java
client/src/main/java/com/metamx/druid/initialization/CuratorDiscoveryConfig.java
client/src/main/java/com/metamx/druid/query/MetricsEmittingExecutorService.java
indexing-hadoop/src/test/java/com/metamx/druid/indexer/HadoopDruidIndexerConfigTest.java
indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/http/IndexerCoordinatorNode.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
indexing-service/src/main/java/com/metamx/druid/indexing/worker/http/WorkerNode.java
pom.xml
server/src/main/java/com/metamx/druid/coordination/ServerManager.java
server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
server/src/main/java/com/metamx/druid/db/DatabaseRuleManager.java
server/src/main/java/com/metamx/druid/db/DatabaseSegmentManager.java
server/src/main/java/com/metamx/druid/http/ComputeNode.java
server/src/main/java/com/metamx/druid/http/MasterMain.java
server/src/main/java/com/metamx/druid/loading/SegmentLoaderConfig.java
server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-08-01 16:42:47 -07:00
019bb5d453
1) Another whole bunch of changes to annotate things and create Modules and bind stuff. But OMFG, the compute node actually appears to be working!
...
2) The compute node works with Guice
3) The compute node fires up with Guice and appears to work
4) Did I mention that the compute node, now called historical node, fires up with Guice and appears to work?
2013-08-01 15:28:08 -07:00
f4fc8b2f6e
1) A whole boondoggle of changes to Guicify the instantiation of things for the Historical nodes
...
2) Add GuiceInjectableValues that supports @JacksonInject resolution via Guice
3) Refactor RequestLogger stuff and setup polymorphic deserialization method of creating objects
2013-07-31 18:36:47 -07:00
9de2e5b096
1) Adjust JsonConfigurator to be able to attach the proper property path if it is renamed via a @JsonProperty annotation
...
2) JsonConfigurator now requires that fields be annotated with @JsonProperty. It will fail fast if they aren't.
2013-07-31 14:55:34 -07:00
97fb971f84
1) Start the journey of making the historical nodes work with Guice. Doesn't work yet, but at least it builds.
2013-07-30 11:58:05 -07:00
6b9963b472
1) Move the @Json and @Smile annotations to com.metamx.druid.guice.annotations
...
2) Changed ServiceAnnouncer to take a DruidNode object
3) Remove ServiceInstanceFactory interface and implementations
4) Add registrations to DiscoveryModule so that you can register nodes that should be announced on startup
5) Make the set of default Monitors configurable
2013-07-26 14:03:51 -07:00
02ffd805c8
1) Move DbConnector into its own module
...
2) Move JacksonConfig stuff into its own module
3) Fix LogEmitterModule to inject properly with a Supplier<>
4) Set Guice to require explicit bindings, to protect from weirdness like was initially experienced with LoggingEmitterConfig
5) Master fires up and appears to all work with Guice! Whoot!
2013-07-23 14:55:36 -07:00
85a35201b7
1) The Master compiles and runs with Guice bindings!!!!
...
2) The service discovery stuff really needs to be reworked. It's unhappy now.
2013-07-19 16:29:54 -07:00
b83bc14784
Merge pull request #173 from metamx/dhruv
...
Add new demo firehose that is lower friction than twitter
2013-07-09 17:39:51 -07:00
b42c5743c9
Merge pull request #177 from nebrera/master
...
Postgresql adaptation
2013-07-05 16:42:57 -07:00
fde874ea56
fix according to code review
2013-07-02 16:11:12 -07:00
e9ce6b596c
PostgresSQL integration with druid
2013-06-28 12:49:24 +02:00
c15f18c343
removed NoopRunnable
2013-06-27 19:03:25 -07:00
15a4dae2b1
bunch of issue fixes
2013-06-26 10:45:54 -07:00
4e4a0a7953
removed DS_Store files
2013-06-25 14:58:12 -07:00
149c2e3560
1) A bunch more changes to make things actually run. Not quite done, but close!
2013-06-20 17:19:51 -07:00
748695bf97
trying to recover my files on intellij
2013-06-20 14:59:52 -07:00
11ea15fc1a
1) Introduce Jetty 8
...
2) Fix up modules based on exceptions at startup for MasterMain
2013-06-17 10:53:50 -07:00
2f56c24259
1) Inject IndexingServiceClient
...
2) Switch all the DBI references to IDBI
2013-06-07 17:37:33 -07:00
f68df7ab69
1) Make tests work and continue trying to make the DruidMaster start up with just Guice
2013-06-07 12:01:46 -07:00
9df458a065
1) Initial commit of conversion to using Guice modules for bootstrapping. Things don't actually completely work yet.
2013-06-06 15:43:15 -07:00
bd9e6fecc1
fix day 1 issue with VIT
2013-05-28 14:26:49 -07:00
343af872b5
DbConnectorConfig: Serialization is hard
2013-05-14 14:39:29 -07:00
43d3c53806
fix order by double sum agg
2013-05-08 10:14:40 -07:00
3c04f1635c
1) Fix bug with same name on multiple JSON properties
2013-05-02 18:00:50 -05:00
60b279b0d3
1) Add support for storing segments in HDFS
2013-04-30 12:47:43 -05:00
43d630c098
1) Replace InventoryManagers with Curator-based Inventory Managers
...
2) Replace ZkSegmentAnnouncer with CuratorSegmentAnnouncer
3) Adjust LoadQueuePeons to operate using Curator
4) Remove ZkPhoneBook
Things left to do:
1) Unit tests
2) Remove references to zkclient library
3) Remove references to PhoneBook interface
2013-04-26 19:19:23 -05:00
dbfa045df0
DbConnector: SQL and validation query tweaks
2013-04-19 14:51:22 -07:00
1a6594524f
Merger: Tweaks to DB tables. Create tables automatically.
2013-04-19 14:07:27 -07:00
1dfe133554
DbConnector: Optional validation query (default off for now)
2013-04-19 14:07:27 -07:00
66a52ed282
javascript aggregator now uses column selector
2013-04-16 09:33:50 -07:00
b66f69def6
specify javascript aggregator functions individually to support getCombiningFactory()
2013-04-16 09:33:50 -07:00
c83631928c
object column selectors
2013-04-16 09:33:50 -07:00
0c68bd1b1d
rename MetricSelectorFactory to ColumnSelectorFactory
2013-04-16 09:33:49 -07:00
55648c47a7
1) Adjust the GroupByQuery to also be able to merge results on the local node. Fixes #116
...
2) Make the GroupByQuery operate in a multi-threaded fashion by default (this is configurable via druid.query.groupBy.singleThreaded). Fixes #96
3) Fix up some post aggregation computation stuff. I believe #72 is fixed
4) Fix case sensitivity issue with post aggregations on GroupBy queries
2013-04-09 19:13:08 -07:00
6c27cd2e8e
cleanup - removed unused imports
2013-04-07 23:31:22 +02:00
a6dc29cf85
cleanup - removed extra empty lines
...
- there should not be any adjacent empty lines
2013-04-07 14:32:09 +02:00
8eec41f934
squigglies ftw
2013-03-20 11:42:17 -07:00
c68bd66945
add close method to aggregators
2013-03-19 16:03:48 -07:00
39449e6a3e
Merge branch 'master' of github.com:metamx/druid
2013-03-14 16:45:54 -05:00
df8e4d4061
Merge branch 'master' into realtime-index-task
2013-03-13 22:43:01 -07:00
1e0f2c2d92
1) Make log a bit more descriptive
2013-03-13 19:26:22 -05:00
fa66e1d2cc
1) SQL is hard
2013-03-13 16:29:34 -05:00
3fa46988f5
Realtime:
...
- MetadataUpdater now built from SegmentAnnouncer, SegmentPublisher instances.
- Sinks can take a version instead of always using interval.start. The realtime
plumber selects a version using a VersioningPolicy.
- Plumbers gained a startJob method.
- Realtime plumbers gained an implementation for finishJob.
2013-03-11 11:14:51 -07:00
a470f3fa6a
change name of druid.indexer.configTable to druid.database.configTable to be consistent
2013-03-08 10:35:13 -08:00
f70f71243d
1) Stop polling in ConfigManager when stop is called
...
2) Remove WorkSetupManager in favor of just using ConfigManager
2013-03-07 17:45:57 -06:00
1c3ef48f34
1) Adjust the Config stuff that WorkerSetupManager was using to be reusable
...
2) Use new ConfigManager to pull out a whitelist for the Master
2013-03-07 17:45:57 -06:00
f862d9205d
Merge branch 'master' into hadoop-index-task
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
2013-02-27 06:53:06 -08:00
14cf506c43
DefaultObjectMapper: Add GuavaModule
2013-02-25 17:05:01 -08:00
f8c54a72c2
1) Changes to allow for local storage
2013-02-21 15:47:01 -06:00
dc3459d3f9
1) Initial commit of refactorings on top of housejester's to simplify the zipping and simplify the creation of other methods of loading data
2013-02-21 15:47:01 -06:00
60da9ffddb
Merge branch 'master' into hadoop-index-task
...
Conflicts:
common/src/main/java/com/metamx/druid/utils/JodaUtils.java
indexer/src/main/java/com/metamx/druid/indexer/DbUpdaterJob.java
indexer/src/main/java/com/metamx/druid/indexer/HadoopDruidIndexerConfig.java
indexer/src/main/java/com/metamx/druid/indexer/IndexGeneratorJob.java
2013-02-15 13:22:31 -08:00
e042c29173
Fixed typos in comments, changed prefix on s3Paths to s3n from s3://, cleaned
...
up umbrellaIntervals code in JodaUtils, modified the DbUpdater to push segment
meta-data to the database in batches.
2013-02-15 11:39:49 -08:00
1b85eaad3e
simplify query toolchest interface
2013-02-13 21:25:16 -08:00
c5b78e75cf
1) Move up to Jackson 2.1
...
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
2013-02-13 18:51:32 -08:00
e10025c841
filter out unneeded segments for TimeBoundary query
2013-02-11 13:21:37 -08:00
6708029043
1) Remove class level comments auto-created by IntelliJ
2013-02-11 13:13:52 -06:00
4f70bbc687
adding linear shard spec and ACL for s3 bucket
2013-02-04 10:27:09 -08:00
7439a2e820
1) Make tests pass
2013-01-28 16:42:51 -06:00
fc07bc315e
Added umbrellaInterval method, which takes an Iterable of intervals and returns
...
a single interval spanning the entire range of input intervals.
2013-01-23 18:59:51 -08:00
55ae4c87dd
timezone support in groupby query
2013-01-23 08:51:02 -08:00
ac31afbce5
remove redundant index for primary key in config table
2013-01-18 16:45:23 -08:00
38b2041ad9
key/value config table
2013-01-17 14:56:48 -08:00
d1f4317af7
updates to autoscaling config based on code review comments
2013-01-14 14:55:04 -08:00
4c2da93389
bug fixes with ec2 auto scaling
2013-01-09 14:51:35 -08:00
3685839376
Merge branch 'master' into autoscaling
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
2013-01-07 18:04:14 -08:00
aff4a05ca3
db based configs for indexer workers
2013-01-07 13:41:26 -08:00
b184c46ae0
1) Introduce idea of value types and generic columns to the serialization format
...
2) Create SegmentLoader interface for the loading of segments!
3) Setup serialization of new format and conversion function inside IndexIO
4) Make some adjustments to the twitter demo to make it make a bit more sense based on some email feedback from someone kicking the tires.
2013-01-04 18:12:49 -06:00
6525d818ad
postAggregators now return field dependencies
2013-01-02 18:36:03 -08:00
db08640efc
reenable daylight saving test, now that we use joda-time 2
2012-12-31 14:10:43 -08:00
436828b67c
post aggs verify that they have all the fields they need
2012-12-12 10:36:46 -08:00
6827e48bd3
bug fixes for invalid query params
2012-12-11 15:09:16 -08:00
1ebd7a8a8a
fix wrong jquery version
2012-12-06 14:50:29 -08:00
4a1abfc5dc
gui for rule configuratin; introducing jquery ui to master console
2012-12-05 15:39:59 -08:00
5822f4f5f7
refactor master to run rules before cleaning up; more master stats; general improvements
2012-12-03 14:43:04 -08:00
aa8fb7b9a3
abstract the assign and drop logic; cleanups everywhere; http endpoints for getting and assigning rules
2012-11-30 11:28:10 -08:00
2e5e1ce989
first commit of tiers for compute nodes; working UT at this point
2012-11-28 17:37:08 -08:00
228b223509
1) Fix bug with Sinks that causes Realtime it to overwrite the first chunk of data when creating a segment to hand off to compute nodes.
...
2) Fix bug with RealtimePlumberSchool that can cause it to miss some previously-persisted segments when directory naming changes
3) Have the various runnables in RealtimePlumberSchool name the threads to make it a bit easier to debug
2012-11-21 18:38:57 -06:00
701cc9562b
1) Adjust the StorageAdapters to lowercase names of metrics and dimensions before looking them up.
...
2) Add some docs to InputRow/Row to indicate that column names passed into the methods are *always* lowercase and that the rows need to act accordingly. (fixes #29 , or at least clarifies the behavior...)
2012-11-19 17:01:17 -06:00
217caa3a04
HistogramVisual: put min. and max. in quantile array
2012-11-06 16:55:35 -08:00
0a42f18cc3
HistogramVisual: add support for quantiles
2012-11-05 14:33:15 -08:00
88a3fb842c
1) New attempt at making OpenJDK 6 happy
2012-11-02 14:24:43 -05:00
38ed1d8b75
1) Revert failed attempt at making OpenJDK6 happy.
2012-11-02 14:24:09 -05:00
63b2f502ac
1) Make TimelineEntry in VersionedIntervalTimeline a static class with its own generic parameters to hopefully make OpenJDK happier.
2012-11-02 11:35:23 -05:00
f07b15db1f
Undo massive import re-ordering
2012-11-01 18:38:27 -05:00
e24948319b
Removed some dead/unused code.
2012-10-31 00:00:30 -07:00
0d24df7628
Removed unused imports.
2012-10-30 22:22:46 -07:00
a34c5a100c
add buffered version of JavaScriptAggregator
2012-10-29 14:34:17 -07:00
27999caca0
1) Create LICENSE
...
2) Attach copyright and notice of license to files
2012-10-24 05:09:47 -04:00
9d41599967
Initial commit of OSS Druid Code
2012-10-24 03:39:51 -04:00
Xavier Léauté
fjy
Gian Merlino
Tugdual Saunier
nishantmonu51
cheddar
cheddar
Pablo Nebrera
Dhruv Parthasarathy
Eric Tschetter
René Scheibe
Nelson Ray
Deep Ganguli
Jae Hyeon Bae
Fangjin Yang
Ian Brandt