Commit Graph

4961 Commits

Author SHA1 Message Date
cheddar e1533b871e Merge pull request #88 from metamx/8to9-empty-column
Additional fix for columns with cardinality 0
2013-02-15 10:15:05 -08:00
Gian Merlino 548c901c0b Additional fix for columns with cardinality 0 2013-02-14 23:45:02 -08:00
Gian Merlino 3fed8ee0ce [maven-release-plugin] prepare for next development iteration 2013-02-14 21:23:46 -08:00
Gian Merlino ba7df5b709 [maven-release-plugin] prepare release druid-0.2.9 2013-02-14 21:23:41 -08:00
Gian Merlino 22d1f2a0c0 IndexMergerTest: Empty column test 2013-02-14 21:11:31 -08:00
Eric Tschetter 53e801ecb3 Merge branch 'master' of github.com:metamx/druid 2013-02-14 22:59:43 -06:00
Eric Tschetter 48538d0454 1) Fix bug in index converter when column has cardinality 0 2013-02-14 22:59:17 -06:00
Gian Merlino 152bf201ec [maven-release-plugin] prepare for next development iteration 2013-02-14 16:27:20 -08:00
Gian Merlino e56bc27f93 [maven-release-plugin] prepare release druid-0.2.8 2013-02-14 16:27:12 -08:00
Eric Tschetter 4f11eb5209 1) Fix alert sent from line DruidMasterBalancer:71 to not include all of the segments that a server is holding 2013-02-14 18:24:51 -06:00
Fangjin Yang bf0d9c601d Merge branch 'task-stuff' into indexing_refactor
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/KillTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskMasterLifecycle.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunnerWorkQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorServletModule.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
2013-02-14 15:42:07 -08:00
Gian Merlino dd44f5cbf0 Merge branch 'master' into task-stuff 2013-02-14 15:11:40 -08:00
Eric Tschetter ee4236dac2 1) Update to mmx build of jackson 2.1 branch to get bug-fix for jackson-databind #167 2013-02-14 17:10:27 -06:00
Fangjin Yang ef768feeb1 new resource 2013-02-14 14:20:20 -08:00
Gian Merlino ad1de9ba9d Merge branch 'master' into task-stuff 2013-02-14 14:16:46 -08:00
Fangjin Yang 77fbe974d8 fix version conflicts of old jackson 2013-02-14 13:52:19 -08:00
Gian Merlino de4fc98128 TaskMasterLifecycle: Stop leaderLifecycle if it dies while starting 2013-02-14 13:36:43 -08:00
Gian Merlino 18daa19fc9 Merge branch 'master' into task-stuff
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-02-14 13:10:29 -08:00
Gian Merlino ff325b1f65 Merger: Changes to support rolling locks and a task-action API
Generally, TaskStatus is no longer used as the sole way for tasks to
communicate status and metadata update requests back to the
coordinator. It is still used to communicate success or failure, but a
TaskActionClient is used for all other communication. In remote mode
this uses an http api.

TaskContext no longer exists, as its purpose is now served by
the TaskActionClient.

The TaskQueue and TaskConsumer are no longer in charge of committing
stuff for a task. Instead, it happens when an action is sent in. Tasks
are expected to ensure that this is OK by making their behavior
idempotent (since they may be retried after partially completing).

One new kind of request afforded to tasks is the ability to release
and acquire arbitrary locks. This will be useful for long-running tasks.
Short-lived tasks can still be given a lock automatically upon startup
by providing a "fixed interval".

New tables when using db storage:
- tasklocks, to support persistence of arbitrary locks.
- tasklogs, to log actions taken by tasks
2013-02-14 12:46:40 -08:00
xvrl 6998d604a2 fixes #80 2013-02-14 12:41:34 -08:00
Nelson Ray a616f86128 add DateTime import back 2013-02-14 09:09:22 -08:00
Nelson Ray 754b065c54 merge master 2013-02-14 09:08:25 -08:00
xvrl 98de0382d5 Merge branch 'master' into speedup-timeboundary 2013-02-13 21:27:27 -08:00
xvrl 1b85eaad3e simplify query toolchest interface 2013-02-13 21:25:16 -08:00
Gian Merlino e3822f6ca7 DeterminePartitionsJob: Fix docs 2013-02-13 18:51:33 -08:00
Gian Merlino ee193f0ca7 DeterminePartitionsJob: Select partition dimension to minimize segment size variance when cardinality is low 2013-02-13 18:51:32 -08:00
Eric Tschetter c5b78e75cf 1) Move up to Jackson 2.1
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
2013-02-13 18:51:32 -08:00
Fangjin Yang 82f02a4fe4 Merge branch 'master' into indexing_refactor
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
2013-02-13 16:34:14 -08:00
Fangjin Yang 1963bfe83f cleanup 2013-02-13 16:04:43 -08:00
Fangjin Yang c7b4973700 bug fixes for RTR 2013-02-13 14:19:11 -08:00
Gian Merlino ce1d90788d DeterminePartitionsJob: Fix docs 2013-02-13 10:16:03 -08:00
cheddar 0eb65f69d4 Merge pull request #86 from metamx/dimpartition-by-variance
Select partition dimension by variance when cardinality is low
2013-02-13 09:54:34 -08:00
Gian Merlino a665bfa2ef DeterminePartitionsJob: Select partition dimension to minimize segment size variance when cardinality is low 2013-02-13 09:51:04 -08:00
Nelson Ray 429e23d3b2 change break to continue in BalancerCostAnalyzer to skip calculating cost when it's not necessary 2013-02-13 08:57:39 -08:00
Eric Tschetter a0b159fed7 1) Move up to Jackson 2.1
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
2013-02-12 20:58:17 -06:00
Nelson Ray fcbac96193 revert change to findNewSegmentHome: now return single ServerHolder 2013-02-12 10:47:30 -08:00
xvrl 07b0b4611b bump up to 0.3.0 2013-02-12 09:52:21 -08:00
xvrl 68cd2e10a7 remove unnecessary segment grouping 2013-02-12 09:45:26 -08:00
xvrl 7ab2e0552a cleanup imports 2013-02-11 18:42:43 -08:00
xvrl 633cccce69 bump up to 0.3.0 2013-02-11 18:41:25 -08:00
xvrl d4009c8c1c refactor filterSegments to use `TimelineObjectHolder`s 2013-02-11 18:41:25 -08:00
Fangjin Yang b27a459f71 more refactoring and UT passing at this point 2013-02-11 18:13:01 -08:00
xvrl 6b87ef2921 assume segment intervals have already been filtered based on query interval 2013-02-11 16:56:52 -08:00
xvrl e9f546c4ca add default implementations to QueryToolChest methods 2013-02-11 16:51:40 -08:00
Nelson Ray 695b78b19c make findNewSegmentHome return a priority queue so assignment can assign replicants to other servers 2013-02-11 15:28:17 -08:00
xvrl e10025c841 filter out unneeded segments for TimeBoundary query 2013-02-11 13:21:37 -08:00
Fangjin Yang 69d0d98df6 t Merge branch 'master' into indexing_refactor 2013-02-11 12:37:20 -08:00
Fangjin Yang a9176e738a Merge branch 'master' into indexing_refactor 2013-02-11 12:36:49 -08:00
Nelson Ray 1b9764fffb add binary version to datasegments for balancer test 2013-02-11 11:36:45 -08:00
Nelson Ray e12da51057 merge master 2013-02-11 11:28:50 -08:00