ab456cf3b1
[maven-release-plugin] prepare release druid-0.3.4
2013-02-26 15:15:36 -08:00
d32a6284ce
Merge branch 'task-stuff' into indexing_refactor
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/worker/config/WorkerConfig.java
2013-02-26 13:03:56 -08:00
d8fbddb9d4
Merger: Service discovery for worker -> master communication
2013-02-26 11:36:07 -08:00
2427e81874
Merger: Feedback from code review
2013-02-25 17:27:02 -08:00
3129e5451b
Merge branch 'master' into task-stuff
2013-02-25 12:33:45 -08:00
9964e7dfe4
[maven-release-plugin] prepare for next development iteration
2013-02-25 12:24:24 -08:00
99c4f9446c
[maven-release-plugin] prepare release druid-0.3.3
2013-02-25 12:24:15 -08:00
49700dffe9
Merge branch 'master' into task-stuff
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
2013-02-25 12:18:31 -08:00
e0f6df1a5c
[maven-release-plugin] prepare for next development iteration
2013-02-21 20:39:50 -06:00
846bc0e4f0
[maven-release-plugin] prepare release druid-0.3.2
2013-02-21 20:39:45 -06:00
9dbb8841a1
initial commit
2013-02-21 18:16:14 -08:00
699aadd2b1
[maven-release-plugin] prepare for next development iteration
2013-02-21 15:50:01 -06:00
606a2e4b01
[maven-release-plugin] prepare release druid-0.3.1
2013-02-21 15:49:55 -06:00
f8c54a72c2
1) Changes to allow for local storage
2013-02-21 15:47:01 -06:00
dc3459d3f9
1) Initial commit of refactorings on top of housejester's to simplify the zipping and simplify the creation of other methods of loading data
2013-02-21 15:47:01 -06:00
d1626576c0
Working toward making it easier to add new SegmentPullers.
...
1) Move the local cacheFile logic out of the S3 pullers into the SingleSegmentLoader
2) Make the S3SegmentPuller just pull down the file
3) Make the Loader do the unzip, ungzip, or rename
4) 2 and 3 make S3ZippedSegmentPuller not necessary (still there, just deprecated and empty)
4) Tweak the TaskToolbox so that the Pullers returned by getSegmentGetters behave the same as they did before
2013-02-21 15:47:01 -06:00
60da9ffddb
Merge branch 'master' into hadoop-index-task
...
Conflicts:
common/src/main/java/com/metamx/druid/utils/JodaUtils.java
indexer/src/main/java/com/metamx/druid/indexer/DbUpdaterJob.java
indexer/src/main/java/com/metamx/druid/indexer/HadoopDruidIndexerConfig.java
indexer/src/main/java/com/metamx/druid/indexer/IndexGeneratorJob.java
2013-02-15 13:22:31 -08:00
824e3c0eb2
[maven-release-plugin] prepare for next development iteration
2013-02-15 13:03:39 -08:00
6bbc992101
[maven-release-plugin] prepare release druid-0.3.0
2013-02-15 13:03:32 -08:00
e042c29173
Fixed typos in comments, changed prefix on s3Paths to s3n from s3://, cleaned
...
up umbrellaIntervals code in JodaUtils, modified the DbUpdater to push segment
meta-data to the database in batches.
2013-02-15 11:39:49 -08:00
bf0d9c601d
Merge branch 'task-stuff' into indexing_refactor
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/KillTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
merger/src/main/java/com/metamx/druid/merger/coordinator/TaskMasterLifecycle.java
merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunner.java
merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunnerWorkQueue.java
merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorServletModule.java
merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
2013-02-14 15:42:07 -08:00
ef768feeb1
new resource
2013-02-14 14:20:20 -08:00
de4fc98128
TaskMasterLifecycle: Stop leaderLifecycle if it dies while starting
2013-02-14 13:36:43 -08:00
18daa19fc9
Merge branch 'master' into task-stuff
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-02-14 13:10:29 -08:00
ff325b1f65
Merger: Changes to support rolling locks and a task-action API
...
Generally, TaskStatus is no longer used as the sole way for tasks to
communicate status and metadata update requests back to the
coordinator. It is still used to communicate success or failure, but a
TaskActionClient is used for all other communication. In remote mode
this uses an http api.
TaskContext no longer exists, as its purpose is now served by
the TaskActionClient.
The TaskQueue and TaskConsumer are no longer in charge of committing
stuff for a task. Instead, it happens when an action is sent in. Tasks
are expected to ensure that this is OK by making their behavior
idempotent (since they may be retried after partially completing).
One new kind of request afforded to tasks is the ability to release
and acquire arbitrary locks. This will be useful for long-running tasks.
Short-lived tasks can still be given a lock automatically upon startup
by providing a "fixed interval".
New tables when using db storage:
- tasklocks, to support persistence of arbitrary locks.
- tasklogs, to log actions taken by tasks
2013-02-14 12:46:40 -08:00
c5b78e75cf
1) Move up to Jackson 2.1
...
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
2013-02-13 18:51:32 -08:00
82f02a4fe4
Merge branch 'master' into indexing_refactor
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
2013-02-13 16:34:14 -08:00
1963bfe83f
cleanup
2013-02-13 16:04:43 -08:00
c7b4973700
bug fixes for RTR
2013-02-13 14:19:11 -08:00
a0b159fed7
1) Move up to Jackson 2.1
...
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
2013-02-12 20:58:17 -06:00
07b0b4611b
bump up to 0.3.0
2013-02-12 09:52:21 -08:00
b27a459f71
more refactoring and UT passing at this point
2013-02-11 18:13:01 -08:00
a9176e738a
Merge branch 'master' into indexing_refactor
2013-02-11 12:36:49 -08:00
e519f82e87
[maven-release-plugin] prepare for next development iteration
2013-02-11 10:58:51 -08:00
73f7468a23
[maven-release-plugin] prepare release druid-0.2.7
2013-02-11 10:58:43 -08:00
111df9a90e
Merge branch 'master' into indexing_refactor
...
Conflicts:
merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-02-07 10:35:32 -08:00
ec2b906fad
[maven-release-plugin] prepare for next development iteration
2013-02-06 18:43:42 -08:00
22d9607169
[maven-release-plugin] prepare release druid-0.2.6
2013-02-06 18:43:34 -08:00
e09c658352
[maven-release-plugin] prepare for next development iteration
2013-02-06 17:42:39 -08:00
668353e895
[maven-release-plugin] prepare release druid-0.2.5
2013-02-06 17:42:33 -08:00
d023d219ae
[maven-release-plugin] prepare for next development iteration
2013-02-06 16:54:21 -08:00
d0fdf4ccf0
[maven-release-plugin] prepare release druid-0.2.4
2013-02-06 16:54:16 -08:00
efd0dc0062
refactor scaling stuff
2013-02-06 11:09:49 -08:00
ea64eaabce
Merge branch 'task-stuff' of github.com:metamx/druid into task-stuff
2013-02-01 17:16:39 -08:00
d9f2e68f7f
better logs
2013-02-01 17:16:24 -08:00
34b4383934
1) Adjust DataSegment to have a binaryVersion field that indicates the expected binary version of the segment files
...
2) Eliminate race condition from RemoteTaskRunnerTest.testAlreadyExecutedTask()
2013-02-01 18:59:33 -06:00
4ec67f612e
Merger: Emit indexer/segment/bytes once per segment. No longer emit indexer/segment/count.
2013-02-01 16:29:58 -08:00
7ec3563e7b
[maven-release-plugin] prepare for next development iteration
2013-02-01 16:25:42 -08:00
a239f62158
[maven-release-plugin] prepare release druid-0.2.3
2013-02-01 16:25:35 -08:00
d252059e4a
Merge branch 'task-stuff' of github.com:metamx/druid into task-stuff
2013-02-01 12:59:04 -08:00
58bc305e26
changes to remote task runner to kill nodes better
2013-02-01 12:58:38 -08:00
393bec0539
TaskQueue: Behavior tweaks, simplification
...
- Lock tasks on add if possible
- Detect "already added" using exception test instead of bookkeeping map
- Update task status after commitRunnable instead of before commitRunnable
2013-02-01 12:16:06 -08:00
455bae2c76
Merge branch 'master' into task-stuff
2013-01-31 11:28:23 -08:00
f50c214514
Merge branch 'master' into task-stuff
2013-01-31 09:59:52 -08:00
f946fc3ee6
TaskConsumer: Commit to segment db in one transaction
2013-01-31 08:22:10 -08:00
1e35e6ad46
TaskQueue: Tweak to implementation of findTaskGroupForTask
2013-01-31 08:22:10 -08:00
34e76f868f
TaskQueue: Clarify "Naptime!" log message
2013-01-31 08:22:10 -08:00
86ec3a372e
TaskGroup: Add javadocs
2013-01-31 08:22:10 -08:00
65576cbade
TaskStorageQueryAdapter: Rename some methods and tweak javadocs for clarity
2013-01-31 08:22:10 -08:00
779c54d8f2
TaskToolbox: Replace IndexerCoordinatorConfig with TaskConfig
2013-01-31 08:22:09 -08:00
fe38ed2547
Introduce TaskExistsException, thrown by TaskStorage.insert when appropriate
2013-01-31 08:22:09 -08:00
0e469c6f4c
TaskStatus: Fix javadocs
2013-01-31 08:09:22 -08:00
00e6ac1e10
[maven-release-plugin] prepare for next development iteration
2013-01-30 17:09:23 -08:00
ef19de8270
[maven-release-plugin] prepare release druid-0.2.2
2013-01-30 17:09:18 -08:00
26ed96f05d
bug fix for autoscaling termination
2013-01-30 17:06:02 -08:00
fa733565e8
[maven-release-plugin] prepare for next development iteration
2013-01-30 16:28:28 -08:00
0b116a8fe4
[maven-release-plugin] prepare release druid-0.2.1
2013-01-30 16:28:22 -08:00
2a0de2090b
[maven-release-plugin] prepare for next development iteration
2013-01-30 13:47:15 -08:00
049ca2bad4
[maven-release-plugin] prepare release druid-0.2.0
2013-01-30 13:47:08 -08:00
74057600f9
fix worker node still using db connection
2013-01-27 09:57:37 -08:00
1cff766e2a
RemoteTaskRunner: Only cleanup paths during bootstrapping if task is complete
2013-01-25 16:01:17 -08:00
efdff7b9f1
fix unit test bug
2013-01-25 14:33:05 -08:00
0f4746450b
RemoteTaskRunner: null check on workerData
2013-01-25 13:40:20 -08:00
bb68091cef
Index service: TaskConsumer tests
2013-01-25 13:15:55 -08:00
82d77a8b72
Index service: Fix kill task
2013-01-25 13:15:49 -08:00
e6a618ca76
Merge branch 'killsegments' into task-stuff
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/IndexTask.java
merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
merger/src/main/java/com/metamx/druid/merger/coordinator/LocalTaskRunner.java
merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
merger/src/main/java/com/metamx/druid/merger/coordinator/exec/TaskConsumer.java
merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
2013-01-25 11:30:10 -08:00
a14200d779
Index service support for early returns and choice of commit semantics.
...
Task:
- Add TaskCallback to run method (for early returns)
TaskStatus:
- Remove CONTINUED status
- Add segmentsNuked (placeholder for future deletion support)
- Add more builder methods
- Add validations to constructor
TaskStorage:
- Add TaskStorageQueryAdapter, a concrete class that wraps TaskStorages and
provides various read-only convenience methods
- Add getTask method for benefit of TaskStorageQueryAdapter
TaskQueue:
- Rename "done" to "notify"
- notify is responsible for deciding if we should commit
- Add optional commitRunnable to "notify", which gets called when it's time to commit
- Allow nextTasks and commits to run early (statusCode RUNNING)
- Move getStatus, collapseStatus functionality to TaskStorageQueryAdapter
2013-01-25 11:05:34 -08:00
cb845e6f09
Addresed Gian's code review
2013-01-24 17:54:06 -08:00
017d4779d6
Implemented Hadoop Index Task which takes as input a HadoopDruidIndexConfig and
...
generates index segments.
The HadoopIndexTask run method wraps a HadoopDruidIndexerJob run method. The
key modifications to the HadoopDruidIndexerJob are as follows:
- The UpDaterJobSpec field of the config that is used to set up the indexer job
is set to null. This ensures that the job does not push a list of published
segments to the database, in order to allow the indexing service to handle this
later.
- Set the version field of the config file based on the TaskContext. Also
changed config.setVersion method to take a string (as opposed to a Date) as
input, and propogated this change where necessary.
- Set the SegmentOutputDir field of the config file based on the TaskToolbox,
to allow the indexing service to handle where to write the segments too.
- Added a method to IndexGeneratorJob called getPublishedSegments, that simply
returns a list of published segments without publishing this list to the
database.
2013-01-23 19:27:14 -08:00
272d737517
cleaning up some interactions with RTR and workers
2013-01-22 16:21:38 -08:00
bab9ee8827
Merge branch 'master' into killsegments
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
2013-01-21 14:47:49 -08:00
07131c51ed
Merge pull request #53 from metamx/autoscaling
...
Use a database to store configurations for indexer workers
2013-01-18 17:08:52 -08:00
7166534666
YeOldePlumberSchool: Tweak for IndexIO changes
2013-01-17 16:03:18 -08:00
5ce53eb2ac
Merge branch 'master' into yops-descriptor-dimensions
2013-01-17 16:02:32 -08:00
38b2041ad9
key/value config table
2013-01-17 14:56:48 -08:00
21613bc73b
initial commit to hard delete segments
2013-01-16 17:31:01 -08:00
c8cb96b006
1) Remove vast majority of usages of IndexIO.mapDir() and deprecated it. IndexIO.loadIndex() is the new IndexIO.mapDir()
...
2) Fix bug with IndexMerger and null columns
3) Add QueryableIndexIndexableAdapter so that QueryableIndexes can be merged
4) Adjust twitter example to have multiple values for each hash tag
5) Adjusted GroupByQueryEngine to just drop dimensions that don't exist instead of throwing an NPE
2013-01-16 17:10:33 -06:00
6fc350bfba
YeOldePlumberSchool: Populate dimension metadata in segment descriptor
2013-01-16 11:30:24 -08:00
3c0880fe28
Merge branch 'master' into autoscaling
2013-01-15 13:36:39 -08:00
7e074e8158
fix pom breakage
2013-01-15 12:04:12 -08:00
d1f4317af7
updates to autoscaling config based on code review comments
2013-01-14 14:55:04 -08:00
4c2da93389
bug fixes with ec2 auto scaling
2013-01-09 14:51:35 -08:00
3685839376
Merge branch 'master' into autoscaling
...
Conflicts:
merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
2013-01-07 18:04:14 -08:00
79e9ff94a6
add missing headers
2013-01-07 14:07:47 -08:00
aff4a05ca3
db based configs for indexer workers
2013-01-07 13:41:26 -08:00
b184c46ae0
1) Introduce idea of value types and generic columns to the serialization format
...
2) Create SegmentLoader interface for the loading of segments!
3) Setup serialization of new format and conversion function inside IndexIO
4) Make some adjustments to the twitter demo to make it make a bit more sense based on some email feedback from someone kicking the tires.
2013-01-04 18:12:49 -06:00
9627fdcd1c
[maven-release-plugin] prepare for next development iteration
2013-01-03 10:09:52 -08:00
f2c2d3f4cf
[maven-release-plugin] prepare release druid-0.1.25
2013-01-03 10:09:43 -08:00
2a188996cd
[maven-release-plugin] prepare for next development iteration
2012-12-31 14:56:10 -08:00
6cdcc2becc
[maven-release-plugin] prepare release druid-0.1.24
2012-12-31 14:56:03 -08:00
Fangjin Yang
Gian Merlino
Eric Tschetter
James Estes
Deep Ganguli
xvrl
cheddar