Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,348 Commits 111 Branches 503 Tags 730 MiB
Commit Graph

280 Commits

Author SHA1 Message Date
Gian Merlino 18daa19fc9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-14 13:10:29 -08:00
xvrl 6998d604a2 fixes #80 2013-02-14 12:41:34 -08:00
xvrl 1b85eaad3e simplify query toolchest interface 2013-02-13 21:25:16 -08:00
Eric Tschetter c5b78e75cf 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-13 18:51:32 -08:00
Fangjin Yang 82f02a4fe4 Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-13 16:34:14 -08:00
Eric Tschetter a0b159fed7 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-12 20:58:17 -06:00
xvrl 68cd2e10a7 remove unnecessary segment grouping 2013-02-12 09:45:26 -08:00
xvrl 7ab2e0552a cleanup imports 2013-02-11 18:42:43 -08:00
xvrl d4009c8c1c refactor filterSegments to use `TimelineObjectHolder`s 2013-02-11 18:41:25 -08:00
xvrl 6b87ef2921 assume segment intervals have already been filtered based on query interval 2013-02-11 16:56:52 -08:00
xvrl e9f546c4ca add default implementations to QueryToolChest methods 2013-02-11 16:51:40 -08:00
xvrl e10025c841 filter out unneeded segments for TimeBoundary query 2013-02-11 13:21:37 -08:00
Fangjin Yang 69d0d98df6 t Merge branch 'master' into indexing_refactor 2013-02-11 12:37:20 -08:00
Eric Tschetter 6708029043 1) Remove class level comments auto-created by IntelliJ 2013-02-11 13:13:52 -06:00
cheddar 05168808c2 Merge pull request #77 from metacret/master 
Adding LinearShardSpec and ACL on S3 object
 2013-02-11 11:12:31 -08:00
Fangjin Yang 111df9a90e Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-07 10:35:32 -08:00
Fangjin Yang efd0dc0062 refactor scaling stuff 2013-02-06 11:09:49 -08:00
Jae Hyeon Bae 4f70bbc687 adding linear shard spec and ACL for s3 bucket 2013-02-04 10:27:09 -08:00
Eric Tschetter f4e843c8dc 1) Make BasePhoneBook not quite so zealous about throwing exceptions when things didn't line up exactly as it thought they should. 2013-02-03 11:02:38 -06:00
Eric Tschetter 34b4383934 1) Adjust DataSegment to have a binaryVersion field that indicates the expected binary version of the segment files 
2) Eliminate race condition from RemoteTaskRunnerTest.testAlreadyExecutedTask()
 2013-02-01 18:59:33 -06:00
xvrl 31743ed438 defaults + max memcached expiration is 30 days 2013-02-01 16:22:33 -08:00
Gian Merlino 9f73c1a428 Merge branch 'master' into task-stuff 2013-02-01 12:16:25 -08:00
xvrl 3aef020fe0 include actual key in value to detect improbable hash collisions 2013-01-31 15:54:52 -08:00
xvrl d5cf7cfdb3 remove references to 'broker' 2013-01-31 14:55:02 -08:00
xvrl 9179718db5 hash cache keys to stay under memcached limit of 250 characters 2013-01-31 14:53:50 -08:00
Fangjin Yang 41ca33c734 Merge branch 'master' of github.com:metamx/druid into task-stuff 2013-01-29 12:57:56 -08:00
Eric Tschetter da914b835e 1) Add caching for SegmentMetadataQuery 2013-01-29 13:04:57 -06:00
Eric Tschetter 08d3253f3d Merge branch 'master' of github.com:metamx/druid 2013-01-29 11:29:42 -06:00
Eric Tschetter f2cce28297 1) Make SegmentMetadataQuery work 2013-01-29 11:29:26 -06:00
Eric Tschetter 7439a2e820 1) Make tests pass 2013-01-28 16:42:51 -06:00
cheddar 5f18f368e3 Merge pull request #61 from metamx/fix-cache-timezone 
Cache did not preserve timezone information
 2013-01-28 14:39:00 -08:00
Gian Merlino 553738e1d8 Merge branch 'master' into task-stuff 2013-01-25 11:34:18 -08:00
Gian Merlino 37417cf22f QueryServlet: Add "host" (remote address) to alerts 2013-01-25 11:03:44 -08:00
Eric Tschetter ee7337fbb9 1) Adjust the Timeseries caching fixes to still store the long, but do the timezone adjustment on the way out. 
2) Store a reference to the granularity object instead of getting it every time
 2013-01-24 18:25:21 -06:00
xvrl 55ae4c87dd timezone support in groupby query 2013-01-23 08:51:02 -08:00
Fangjin Yang 272d737517 cleaning up some interactions with RTR and workers 2013-01-22 16:21:38 -08:00
xvrl d7ea8e9afc compare result timestamp based on millis + utcoffset 2013-01-21 17:01:41 -08:00
xvrl 8f38b775ae fix expected object type 2013-01-21 16:31:32 -08:00
xvrl 068ca67dba fix cache not preserving timezone information 2013-01-21 14:38:04 -08:00
xvrl 40c0bcad29 simplify MapCache 2013-01-18 18:25:51 -08:00
xvrl e0c34c3b97 rename cacheBroker -> cache 2013-01-18 15:22:56 -08:00
xvrl 86ca8967ca rework code pulling from cache to be more readable 2013-01-18 15:17:10 -08:00
xvrl a70ae15585 replace Pair<String, ByteBuffer> with NamedKey 2013-01-18 15:17:05 -08:00
xvrl 9032ef521b fix interrupted thread 2013-01-18 15:17:05 -08:00
xvrl 0bacb85a4a fix duplicate keys, shutdown gracefully and make sure we check all multiget keys in memcached benchmark 2013-01-16 19:18:14 -08:00
xvrl dcaa77a883 implement bulk get test 2013-01-16 19:15:43 -08:00
Fangjin Yang 21613bc73b initial commit to hard delete segments 2013-01-16 17:31:01 -08:00
xvrl e2788187fb don't let timeout skew benchmark stats 2013-01-16 16:02:51 -08:00
xvrl a2090411a3 modify cacheBroker interface to implement bulk-get 2013-01-16 14:49:50 -08:00
xvrl 700ce24106 fix JVM optimizing out the benchmarking code 2013-01-07 16:54:35 -08:00
Eric Tschetter b184c46ae0 1) Introduce idea of value types and generic columns to the serialization format 
2) Create SegmentLoader interface for the loading of segments!
3) Setup serialization of new format and conversion function inside IndexIO
4) Make some adjustments to the twitter demo to make it make a bit more sense based on some email feedback from someone kicking the tires.
 2013-01-04 18:12:49 -06:00
Eric Tschetter 4f5087b386 Merge branch 'master' of github.com:metamx/druid 2013-01-04 11:28:35 -06:00
xvrl 824997eb43 simplify code 2013-01-03 10:01:42 -08:00
xvrl 41edec62ce remove unused imports 2013-01-02 18:46:07 -08:00
xvrl 6525d818ad postAggregators now return field dependencies 2013-01-02 18:36:03 -08:00
Eric Tschetter 3bfbcbe95c 1) Create DirectClientQuerySegmentWalker to make it a bit easier to embed and use DirectDruidClient as a client. 2012-12-28 10:50:40 -06:00
Gian Merlino 5c1b2360d6 - Add optional query/wait metric to MetricsEmittingQueryRunner 
- Add MetricsEmittingExecutorService decorator, and use it on compute nodes
 2012-12-20 16:11:08 -08:00
Fangjin Yang 00ecf9d028 sorted db datasources for rules; smaller time boundary range 2012-12-18 16:16:03 -08:00
xvrl 8b55790900 add segmentMetadata query type 2012-12-18 10:15:23 -08:00
xvrl 54d8aff4d5 add parameter for benchmarking host 2012-12-17 16:44:40 -08:00
xvrl 2118252ef2 make memcached host configurable for benchmarking 2012-12-17 16:44:39 -08:00
Fangjin Yang 74a977504e bug fix for dropping segments in master 2012-12-14 17:06:03 -08:00
Eric Tschetter 62ea45f2c9 1) Register SearchQueryDimFilter with DimFilter so that it can actually be used. 2012-12-13 17:55:32 -06:00
Fangjin Yang 4bc721f3b3 revert last commit 2012-12-12 16:09:07 -08:00
Fangjin Yang 6828ad10d4 reduce boundaries for time boundary a bit; prevent bound errors 2012-12-12 16:07:47 -08:00
cheddar 55652861da Merge pull request #44 from metamx/segment-metadata-query 
Barebones segment metadata query
 2012-12-12 15:33:50 -08:00
Fangjin Yang dd25868919 add missing headers to files 2012-12-12 15:03:26 -08:00
cheddar f4a413c83f Merge pull request #40 from metamx/broker-cacheconfig 
configuration option to switch cache implementations
 2012-12-12 14:54:52 -08:00
Fangjin Yang 6db99c45a6 more specific exceptions in test 2012-12-12 14:51:19 -08:00
xvrl bfc9b63931 segment metadata query 2012-12-12 13:45:12 -08:00
Fangjin Yang 436828b67c post aggs verify that they have all the fields they need 2012-12-12 10:36:46 -08:00
Fangjin Yang 08eb3390c8 fix for caching bug with empty byte arrays 2012-12-12 09:30:48 -08:00
Fangjin Yang 6827e48bd3 bug fixes for invalid query params 2012-12-11 15:09:16 -08:00
xvrl 81bc2890dc configuration option to switch cache implementations 2012-12-10 17:10:20 -08:00
Fangjin Yang 4695fbfcd9 fix bug with comparing historical servers 2012-12-07 17:49:18 -08:00
Fangjin Yang 6028273332 defaults for rules to make startup a bit easier 2012-12-07 10:54:38 -08:00
cheddar 6e775a9ecd Merge pull request #38 from metamx/memcached-cachebroker 
memcached version of CacheBroker
 2012-12-06 16:08:42 -08:00
xvrl a0dbd233fb fix benchmarking code following removal of unnecessary method on MemcachedCacheBroker 2012-12-06 16:07:29 -08:00
xvrl 9ef46f5e65 remove unused method 2012-12-06 15:57:37 -08:00
xvrl c11334934e emit cache timeouts metric 2012-12-06 11:41:35 -08:00
xvrl 9b8037b0a0 properly propagate exceptions 2012-12-05 15:05:20 -08:00
xvrl 2145f78f78 add memcached cache benchmarking code 2012-12-04 16:32:44 -08:00
xvrl 1083ed9c11 make max cache object size configurable and disable client-side compressions 2012-12-04 16:28:27 -08:00
xvrl a3e46eb490 fix config property name 2012-12-04 15:36:05 -08:00
xvrl c6e641e819 fix invalid cache keys 2012-12-04 12:06:19 -08:00
xvrl 01dc787261 fix test case class name 2012-12-04 11:36:37 -08:00
xvrl 51a7adf4fe enable consistent hashing and optimizations + daemonize memcached IO thread 2012-12-04 11:18:51 -08:00
xvrl 98fbcfd7c2 file headers 2012-12-03 17:49:05 -08:00
xvrl a03f8527ea implement memcached CacheBroker 2012-12-03 17:44:17 -08:00
Fangjin Yang 5822f4f5f7 refactor master to run rules before cleaning up; more master stats; general improvements 2012-12-03 14:43:04 -08:00
Fangjin Yang 1fb3bf5cd8 remove defaults for druid tiers 2012-11-30 11:32:02 -08:00
Fangjin Yang aa8fb7b9a3 abstract the assign and drop logic; cleanups everywhere; http endpoints for getting and assigning rules 2012-11-30 11:28:10 -08:00
Fangjin Yang 2e5e1ce989 first commit of tiers for compute nodes; working UT at this point 2012-11-28 17:37:08 -08:00
xvrl 7d2778fae6 fix TimeBoundaryQuery to properly handle timestamps prior to 1970 2012-11-28 13:12:38 -08:00
Eric Tschetter 4de6ef5025 1) Remove the requirement on the segmentsPath zk property. 2012-11-28 11:54:59 -06:00
Eric Tschetter e2da8df3f4 1) Fix build 2012-11-21 19:15:08 -06:00
Eric Tschetter 9c732e75a1 1) Add new SearchQueryDimFilter/SearchQueryFilter that will do a filter using the same queries that can be used for SearchQuery 2012-11-21 19:06:32 -06:00
Eric Tschetter 06b8e4230c 1) Fix PostAggregations with GroupBy (Fixes #24) 
2) Add GroupByTimeseriesQueryRunnerTest which wraps a GroupByQueryRunner to look like a TimeseriesQueryRunner and leverages the tests there to help verify behavior (#8)
 2012-11-21 12:51:43 -06:00
Fangjin Yang 4131063049 Merge branch 'master' into fjy 2012-11-19 15:07:00 -08:00
Fangjin Yang b228c053e5 bug fix for too much data to indexer 2012-11-19 15:06:46 -08:00
Eric Tschetter 701cc9562b 1) Adjust the StorageAdapters to lowercase names of metrics and dimensions before looking them up. 
2) Add some docs to InputRow/Row to indicate that column names passed into the methods are *always* lowercase and that the rows need to act accordingly. (fixes #29, or at least clarifies the behavior...)
 2012-11-19 17:01:17 -06:00
Fangjin Yang cd1a812fff construct registering nodes with lists 2012-11-13 15:00:55 -08:00
Fangjin Yang 0ef40171a8 nodes no longer inherit from interfaces but instead extend classes 2012-11-13 13:18:31 -08:00
Fangjin Yang 7eccd76e9a add missing headers 2012-11-12 16:34:48 -08:00
Fangjin Yang 57468d39ef reverting some of the last changes 2012-11-12 16:14:48 -08:00
Fangjin Yang c20dccd0f4 modifying the way registering serdes works to hopefully be a bit easier to use 2012-11-12 13:58:43 -08:00
Fangjin Yang 2ef1136ab2 fix a bug where intervals were being returned as String instead of Interval objects by the compute nodes 2012-11-09 13:39:02 -08:00
Fangjin Yang 17fb93d049 fixes to init and more unit tests for timeseries 2012-11-08 14:43:02 -08:00
Fangjin Yang 8323feedb4 Merge branch 'master' into moarunittests 
Conflicts:
	client/src/main/java/com/metamx/druid/http/BrokerMain.java
 2012-11-08 10:37:05 -08:00
Fangjin Yang 7083821fbf no param for scaling provision and minor fix for initialization 2012-11-08 10:08:40 -08:00
Fangjin Yang 4d6cfdc774 Merge branch 'master' into autoscaling 2012-11-07 17:27:42 -08:00
Eric Tschetter b1475219a9 Merge branch 'master' of git://github.com/tralfamadude/druid-1 
Added DruidSetup.java and install/druid_setup.sh to run it for Druid ensemble setup (put properties to zk, create zk zpaths, prep db, and dump properties/paths in zk); property druid.zk.paths.base can establish a namespace for a druid ensemble allowing more than one ensemble to use the same zookeepers; this namespace is the base zpath prefix for paths with properties in the form druid.zk.paths.*Path which are normally set automatically now; if druid.zk.paths.*Path are explicitly set, then all of them must be set with absolute paths to ensure careful attention; ZkSetup is now deprecated, use DruidSetup instead; examples/twitter RealtimeStandaloneMain will exit properly now after a kill -15 (control-c or kill pid) instead of hanging on an unstopped daemon.
 2012-11-06 20:15:57 -08:00
Paul Baclace 86d2fb3550 added DruidSetup.java and install/druid_setup.sh to run it for Druid ensemble setup (put properties to zk, create zk zpaths, prep db, and dump properties/paths in zk); property druid.zk.paths.base can establish a namespace for a druid ensemble allowing more than one ensemble to use the same zookeepers; this namespace is the base zpath prefix for paths with properties in the form druid.zk.paths.*Path which are normally set automatically now; if druid.zk.paths.*Path are explicitly set, then all of them must be set with absolute paths to ensure careful attention; ZkSetup is now deprecated, use DruidSetup instead; examples/twitter RealtimeStandaloneMain will exit properly now after a kill -15 (control-c or kill pid) instead of hanging on an unstopped daemon. 2012-11-06 00:16:27 -08:00
Fangjin Yang 51cd361fbe some changes to how handlers are registered 2012-11-05 13:39:16 -08:00
Fangjin Yang 499263386c timeseries for druid + unit tests 2012-11-05 11:38:03 -08:00
Fangjin Yang a5c3eab2f1 fix merge break 2012-11-02 15:59:55 -07:00
Fangjin Yang b1b611735e Merge branch 'master' into autoscaling 
Conflicts:
	client/src/main/java/com/metamx/druid/http/BrokerMain.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
 2012-11-02 15:57:16 -07:00
Fangjin Yang 69b929fad5 Merge branch 'master' into moarunittests 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskInventoryManager.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/Worker.java
	merger/src/main/java/com/metamx/druid/merger/worker/WorkerCuratorCoordinator.java
 2012-11-01 16:41:13 -07:00
Fangjin Yang 09cc2e3353 Merge branch 'master' into moarunittests 
Conflicts:
	client/src/main/java/com/metamx/druid/http/BrokerMain.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskInventoryManager.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/Worker.java
	merger/src/main/java/com/metamx/druid/merger/worker/WorkerCuratorCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
 2012-11-01 16:39:28 -07:00
Eric Tschetter f07b15db1f Undo massive import re-ordering 2012-11-01 18:38:27 -05:00
Eric Tschetter 8dd6f5c059 1) Refactor the BrokerMain into a BrokerMain and a BrokerNode to allow for extensions 
2) Rename ServerMain to ComputeMain
3) Refactor ComputeMain to ComputeMain and ComputeNode to allow for extensions
4) Remove "TheSizeAdjuster", was old legacy stuff that's not relevant anymore
5) Fix bug with registering different IndexIO handlers
6) Adjust it so that when a query type is unknown, it returns an error message instead of just emitting an alert and returning nothing
7) Create super classes for the various *Node classes that allow for code sharing on pieces that are common
 2012-11-01 18:20:38 -05:00
Fangjin Yang 11c64593ae improvements to the indexing service and auto scaling 2012-10-31 14:35:09 -07:00
Fangjin Yang fe9cb397a1 remove unncessarily created file 2012-10-31 10:29:03 -07:00
Fangjin Yang d0f66c1fb3 bug fix for merging with empty strings 2012-10-31 10:28:24 -07:00
Ian Brandt e24948319b Removed some dead/unused code. 2012-10-31 00:00:30 -07:00
Ian Brandt 0d24df7628 Removed unused imports. 2012-10-30 22:22:46 -07:00
Eric Tschetter d7f85cfbc4 1) Fix configuration property naming issue. 2012-10-30 12:08:30 -05:00
Eric Tschetter f062e136d0 1) Fix bug with sharded realtime data ingestion 2012-10-30 10:59:07 -05:00
Eric Tschetter 27999caca0 1) Create LICENSE 
2) Attach copyright and notice of license to files
 2012-10-24 05:09:47 -04:00
Eric Tschetter 9d41599967 Initial commit of OSS Druid Code 2012-10-24 03:39:51 -04:00