Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
1,591 Commits 113 Branches 506 Tags 732 MiB
Commit Graph

309 Commits

Author SHA1 Message Date
cheddar 405eed7c60 Better name for ServerInventoryThingie 2013-04-29 11:48:42 -05:00
fjy 2849486251 simpler sequence limiting 2013-04-28 15:34:43 -07:00
fjy 67cafaa0b1 Merge branch 'master' into spatial 2013-04-26 17:56:18 -07:00
fjy 82a62e3ba0 bug fixes for spatial and groupby; threshold for groupby 2013-04-26 17:46:25 -07:00
cheddar 52ba60963e Screwed up the conflicts on rebasing. Fixorz orz orz 2013-04-26 19:25:23 -05:00
cheddar 74ff07ba3e Moar unit tests 2013-04-26 19:19:57 -05:00
cheddar 8768406069 1) Unit tests! 2013-04-26 19:19:57 -05:00
cheddar 046647c203 1) Make druid.zk.paths.base required 
2) Unit tests for announcer and fix race condition with session death
 2013-04-26 19:19:57 -05:00
cheddar 163a73bed1 More logging stuff 2013-04-26 19:19:57 -05:00
cheddar 19d80ccd9d Remove extraneous logs 2013-04-26 19:19:57 -05:00
cheddar 67ce1b6f26 1) Adjust all of the uses of zookeeper paths to be based on ZkPathsConfig 2013-04-26 19:19:56 -05:00
cheddar b6d72c0da4 1) Log when servers show up and when they load segments 
2) Adjust nodes to publish to ZK as the last part of their initialization
 2013-04-26 19:19:56 -05:00
cheddar 3e94204c44 1) Switch from com.netflix.curator to org.apache.curator 2013-04-26 19:19:55 -05:00
cheddar 9153cc94ac 1) Nicer logging 
2) Announcer should actually delete its zk entries when it is told to unannounce something
3) Skip over complex columns that we don't have a proper deserializer for
 2013-04-26 19:19:55 -05:00
cheddar b8ba9138ff 1) Fix bug with Master not starting up without an indexer specified 
2) Fix bug with PotentiallyGzippedCompressionProvider only catching ZipExceptions and not IOException (java 6 throws IO, java 7 throws Zip)
3) Create DruidServerMetadata and use that instead of DruidServer to represent the current server
 2013-04-26 19:19:55 -05:00
cheddar f71b941a1a 1) Refactor the announcement of segments to all exist inside the DataSegmentAnnouncer 
2) Adjust the ExecutorNodes to expose the correct nodeType given the refactorings
 2013-04-26 19:19:54 -05:00
cheddar 545b489093 1) Use PathChildrenCacheFactory to simplify creation of PathChildrenCache objects. 2013-04-26 19:19:53 -05:00
cheddar 57c31656d1 1) Eliminate PhoneBook 2013-04-26 19:19:53 -05:00
cheddar dde50a0d87 1) Remove references to zkclient 
2) Eradicate zkclient from the poms!
 2013-04-26 19:19:53 -05:00
cheddar 43d630c098 1) Replace InventoryManagers with Curator-based Inventory Managers 
2) Replace ZkSegmentAnnouncer with CuratorSegmentAnnouncer
3) Adjust LoadQueuePeons to operate using Curator
4) Remove ZkPhoneBook

Things left to do:
1) Unit tests
2) Remove references to zkclient library
3) Remove references to PhoneBook interface
 2013-04-26 19:19:23 -05:00
Gian Merlino f463b95256 Merger: Advertise event receivers through service discovery 2013-04-25 14:06:46 +03:00
xvrl e3c6bad117 add emitting request logger 2013-04-24 15:38:16 -07:00
fjy 7f43f37441 bug fixes 2013-04-23 18:06:41 -07:00
fjy 70937af7bd Merge branch 'master' of github.com:metamx/druid into spatial 
Conflicts:
	server/src/main/java/com/metamx/druid/index/v1/IndexStorageAdapter.java
	server/src/main/java/com/metamx/druid/index/v1/QueryableIndexStorageAdapter.java
 2013-04-23 11:40:44 -07:00
fjy f28abd4256 fix cache 2013-04-22 15:01:07 -07:00
Gian Merlino 4c4d30cf66 Initialization: Set service discovery address equal to druid.host (without port if any) 2013-04-19 15:45:00 -07:00
Fangjin Yang 71269d7e88 somehow tests are passing 2013-04-17 17:13:24 -07:00
Eric Tschetter 55648c47a7 1) Adjust the GroupByQuery to also be able to merge results on the local node. Fixes #116 
2) Make the GroupByQuery operate in a multi-threaded fashion by default (this is configurable via druid.query.groupBy.singleThreaded).  Fixes #96
3) Fix up some post aggregation computation stuff.  I believe #72 is fixed
4) Fix case sensitivity issue with post aggregations on GroupBy queries
 2013-04-09 19:13:08 -07:00
cheddar a678f08b67 Merge pull request #118 from darxriggs/master 
cleanup & refactoring
 2013-04-09 18:56:59 -07:00
Fangjin Yang 120b822c18 Merge branch 'master' of github.com:metamx/druid 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
 2013-04-08 11:24:17 -07:00
Fangjin Yang 925c104dd2 update emitter version and fix some NPEs 2013-04-08 11:23:08 -07:00
René Scheibe 6c27cd2e8e cleanup - removed unused imports 2013-04-07 23:31:22 +02:00
René Scheibe a6dc29cf85 cleanup - removed extra empty lines 
- there should not be any adjacent empty lines
 2013-04-07 14:32:09 +02:00
René Scheibe 488b3ef8ff refactoring - made json type infos consistent for all query types 2013-04-07 14:32:09 +02:00
xvrl 6d5869d8e4 remove unused imports 2013-04-05 14:36:41 -07:00
xvrl 1d050245fa add argument checks to dim filters 2013-04-04 16:30:33 -07:00
Fangjin Yang f9e5a10c70 fix query problem 2013-03-28 19:47:01 -07:00
Fangjin Yang 12e0dbce68 fix broker datasources path 2013-03-28 19:12:10 -07:00
Fangjin Yang 5bdcc3613c fix inconsistent druid http paths + fix master console bugs 2013-03-27 10:40:44 -07:00
Fangjin Yang 082ea59cf1 Merge branch 'master' of github.com:metamx/druid 2013-03-25 13:11:20 -07:00
Fangjin Yang 0ab7c315dd more logging in CEQR to track down NPE 2013-03-25 13:11:02 -07:00
xvrl c420fe3b56 fix output timestamps in groupby queries with granularity "all". 2013-03-23 23:42:43 -07:00
Fangjin Yang 5eaaabce7f fix for concurrent modifications in druid server 2013-03-22 17:05:39 -07:00
Eric Tschetter a933438e4e 1) Fix bugs with VersionConverterTask 
2) Fix bugs with NPEs on indexing
 2013-03-15 13:48:55 -05:00
cheddar 2a94bd508d Merge pull request #104 from metamx/javascript-dimfilters 
support for Javascript DimFilters
 2013-03-14 11:47:55 -07:00
Eric Tschetter 1c3ef48f34 1) Adjust the Config stuff that WorkerSetupManager was using to be reusable 
2) Use new ConfigManager to pull out a whitelist for the Master
 2013-03-07 17:45:57 -06:00
Eric Tschetter 6864007c05 1) Remove Task parameter from other methods off of TaskToolbox that required it 
2) Move getTaskDir() off of TaskConfig and onto TaskToolbox
3) Rename SegmentKiller interface to DataSegmentKiller
4) Change signature of "DataSegmentKiller.kill(Collection<DataSegment>) throws ServiceException" to just kill(DataSegment) throws SegmentLoadingException
5) Add various log messages
6) Update the version of the segment that has been converted
 2013-03-07 17:44:06 -06:00
xvrl a79c9ec183 fix some issues with negative numbers 2013-03-07 14:10:58 -08:00
cheddar a3279541c6 Merge pull request #102 from metamx/cache-fixes 
log but do not fail if memcached operations do not get queued in time
 2013-03-07 13:36:51 -08:00
cheddar a1cd3c3905 Merge pull request #103 from metamx/request-logs 
make request logs machine-readable
 2013-03-07 13:35:48 -08:00
xvrl b0349a6518 move objectMapper into requestLogger 2013-03-07 11:31:36 -08:00
xvrl 1b51848a89 add cache errorCount 2013-03-07 10:38:41 -08:00
xvrl 21d648cc39 make request logs machine-readable 2013-03-06 22:33:39 -08:00
xvrl 2837309075 do not fail on exceptions when pulling from cache 2013-03-06 19:22:15 -08:00
xvrl 8f6c313561 log but do not fail if memcached operations do not get queued in time 2013-03-06 17:28:09 -08:00
xvrl 4cb3ca00bc support for javascript dimfilters 2013-03-06 09:59:53 -08:00
xvrl 614d7801a4 very simplistic error handling 2013-03-05 12:41:44 -08:00
xvrl 28fe206404 output tsv format + better command line 2013-03-05 11:48:47 -08:00
xvrl 035986b3bc keep track of queried fields 2013-03-05 11:45:51 -08:00
xvrl ae682d2cbb fix multiple toplevel dimfilters 2013-03-04 16:44:54 -08:00
xvrl d1c128bdbf more elaborate example 2013-03-04 16:44:54 -08:00
xvrl 88515ec59f add support for regex filter 2013-03-04 16:44:54 -08:00
xvrl f5e3ff9692 add support for `!=` operator and `in ('a','b',...) expressions 2013-03-04 16:44:54 -08:00
xvrl 6a40072d50 toy SQL runner 2013-03-04 16:44:54 -08:00
xvrl 4e765f6754 remove obsolete comments 2013-03-04 16:44:54 -08:00
xvrl aa3822454a add support for 'NOT' filter and comments 2013-03-04 16:44:53 -08:00
xvrl eae5db6f57 friendlier aggregator naming 2013-03-04 16:44:53 -08:00
xvrl 5d1af7d884 support aliasing 2013-03-04 16:44:53 -08:00
xvrl f8a118dd98 better number handling 2013-03-04 16:44:53 -08:00
xvrl 3f7bd7d1c0 add group by 2013-03-04 16:44:53 -08:00
xvrl b64118b168 minimal SQL support 2013-03-04 16:44:52 -08:00
Fangjin Yang 7f32629429 fix a bug with client info resource with guice 2013-03-01 13:53:27 -08:00
Fangjin Yang fb87458056 cleaner code for dimension/metric exploration on broker 2013-03-01 11:41:28 -08:00
Gian Merlino 26dca5630f QueryServlet: Send alerts with remote address as "peer" instead of "host" 
"host" is used by the emitter. Both end up being emitted, which can be confusing.
 2013-03-01 06:59:25 -08:00
Gian Merlino d32a6284ce Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/worker/config/WorkerConfig.java
 2013-02-26 13:03:56 -08:00
Gian Merlino 49700dffe9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
 2013-02-25 12:18:31 -08:00
Eric Tschetter 0d99cee3c3 1) Whitespace ftw! 2013-02-15 14:25:44 -06:00
xvrl 780410446c even more simple 2013-02-15 11:19:13 -08:00
Fangjin Yang ef768feeb1 new resource 2013-02-14 14:20:20 -08:00
Gian Merlino 18daa19fc9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-14 13:10:29 -08:00
xvrl 6998d604a2 fixes #80 2013-02-14 12:41:34 -08:00
xvrl 1b85eaad3e simplify query toolchest interface 2013-02-13 21:25:16 -08:00
Eric Tschetter c5b78e75cf 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-13 18:51:32 -08:00
Fangjin Yang 82f02a4fe4 Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-13 16:34:14 -08:00
Eric Tschetter a0b159fed7 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-12 20:58:17 -06:00
xvrl 68cd2e10a7 remove unnecessary segment grouping 2013-02-12 09:45:26 -08:00
xvrl 7ab2e0552a cleanup imports 2013-02-11 18:42:43 -08:00
xvrl d4009c8c1c refactor filterSegments to use `TimelineObjectHolder`s 2013-02-11 18:41:25 -08:00
xvrl 6b87ef2921 assume segment intervals have already been filtered based on query interval 2013-02-11 16:56:52 -08:00
xvrl e9f546c4ca add default implementations to QueryToolChest methods 2013-02-11 16:51:40 -08:00
xvrl e10025c841 filter out unneeded segments for TimeBoundary query 2013-02-11 13:21:37 -08:00
Fangjin Yang 69d0d98df6 t Merge branch 'master' into indexing_refactor 2013-02-11 12:37:20 -08:00
Eric Tschetter 6708029043 1) Remove class level comments auto-created by IntelliJ 2013-02-11 13:13:52 -06:00
cheddar 05168808c2 Merge pull request #77 from metacret/master 
Adding LinearShardSpec and ACL on S3 object
 2013-02-11 11:12:31 -08:00
Fangjin Yang 111df9a90e Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-07 10:35:32 -08:00
Fangjin Yang efd0dc0062 refactor scaling stuff 2013-02-06 11:09:49 -08:00
Jae Hyeon Bae 4f70bbc687 adding linear shard spec and ACL for s3 bucket 2013-02-04 10:27:09 -08:00
Eric Tschetter f4e843c8dc 1) Make BasePhoneBook not quite so zealous about throwing exceptions when things didn't line up exactly as it thought they should. 2013-02-03 11:02:38 -06:00
Eric Tschetter 34b4383934 1) Adjust DataSegment to have a binaryVersion field that indicates the expected binary version of the segment files 
2) Eliminate race condition from RemoteTaskRunnerTest.testAlreadyExecutedTask()
 2013-02-01 18:59:33 -06:00
xvrl 31743ed438 defaults + max memcached expiration is 30 days 2013-02-01 16:22:33 -08:00
Gian Merlino 9f73c1a428 Merge branch 'master' into task-stuff 2013-02-01 12:16:25 -08:00
xvrl 3aef020fe0 include actual key in value to detect improbable hash collisions 2013-01-31 15:54:52 -08:00
xvrl d5cf7cfdb3 remove references to 'broker' 2013-01-31 14:55:02 -08:00
xvrl 9179718db5 hash cache keys to stay under memcached limit of 250 characters 2013-01-31 14:53:50 -08:00
Fangjin Yang 41ca33c734 Merge branch 'master' of github.com:metamx/druid into task-stuff 2013-01-29 12:57:56 -08:00
Eric Tschetter da914b835e 1) Add caching for SegmentMetadataQuery 2013-01-29 13:04:57 -06:00
Eric Tschetter 08d3253f3d Merge branch 'master' of github.com:metamx/druid 2013-01-29 11:29:42 -06:00
Eric Tschetter f2cce28297 1) Make SegmentMetadataQuery work 2013-01-29 11:29:26 -06:00
Eric Tschetter 7439a2e820 1) Make tests pass 2013-01-28 16:42:51 -06:00
cheddar 5f18f368e3 Merge pull request #61 from metamx/fix-cache-timezone 
Cache did not preserve timezone information
 2013-01-28 14:39:00 -08:00
Gian Merlino 553738e1d8 Merge branch 'master' into task-stuff 2013-01-25 11:34:18 -08:00
Gian Merlino 37417cf22f QueryServlet: Add "host" (remote address) to alerts 2013-01-25 11:03:44 -08:00
Eric Tschetter ee7337fbb9 1) Adjust the Timeseries caching fixes to still store the long, but do the timezone adjustment on the way out. 
2) Store a reference to the granularity object instead of getting it every time
 2013-01-24 18:25:21 -06:00
xvrl 55ae4c87dd timezone support in groupby query 2013-01-23 08:51:02 -08:00
Fangjin Yang 272d737517 cleaning up some interactions with RTR and workers 2013-01-22 16:21:38 -08:00
xvrl d7ea8e9afc compare result timestamp based on millis + utcoffset 2013-01-21 17:01:41 -08:00
xvrl 8f38b775ae fix expected object type 2013-01-21 16:31:32 -08:00
xvrl 068ca67dba fix cache not preserving timezone information 2013-01-21 14:38:04 -08:00
xvrl 40c0bcad29 simplify MapCache 2013-01-18 18:25:51 -08:00
xvrl e0c34c3b97 rename cacheBroker -> cache 2013-01-18 15:22:56 -08:00
xvrl 86ca8967ca rework code pulling from cache to be more readable 2013-01-18 15:17:10 -08:00
xvrl a70ae15585 replace Pair<String, ByteBuffer> with NamedKey 2013-01-18 15:17:05 -08:00
xvrl 9032ef521b fix interrupted thread 2013-01-18 15:17:05 -08:00
xvrl 0bacb85a4a fix duplicate keys, shutdown gracefully and make sure we check all multiget keys in memcached benchmark 2013-01-16 19:18:14 -08:00
xvrl dcaa77a883 implement bulk get test 2013-01-16 19:15:43 -08:00
Fangjin Yang 21613bc73b initial commit to hard delete segments 2013-01-16 17:31:01 -08:00
xvrl e2788187fb don't let timeout skew benchmark stats 2013-01-16 16:02:51 -08:00
xvrl a2090411a3 modify cacheBroker interface to implement bulk-get 2013-01-16 14:49:50 -08:00
xvrl 700ce24106 fix JVM optimizing out the benchmarking code 2013-01-07 16:54:35 -08:00
Eric Tschetter b184c46ae0 1) Introduce idea of value types and generic columns to the serialization format 
2) Create SegmentLoader interface for the loading of segments!
3) Setup serialization of new format and conversion function inside IndexIO
4) Make some adjustments to the twitter demo to make it make a bit more sense based on some email feedback from someone kicking the tires.
 2013-01-04 18:12:49 -06:00
Eric Tschetter 4f5087b386 Merge branch 'master' of github.com:metamx/druid 2013-01-04 11:28:35 -06:00
xvrl 824997eb43 simplify code 2013-01-03 10:01:42 -08:00
xvrl 41edec62ce remove unused imports 2013-01-02 18:46:07 -08:00
xvrl 6525d818ad postAggregators now return field dependencies 2013-01-02 18:36:03 -08:00
Eric Tschetter 3bfbcbe95c 1) Create DirectClientQuerySegmentWalker to make it a bit easier to embed and use DirectDruidClient as a client. 2012-12-28 10:50:40 -06:00
Gian Merlino 5c1b2360d6 - Add optional query/wait metric to MetricsEmittingQueryRunner 
- Add MetricsEmittingExecutorService decorator, and use it on compute nodes
 2012-12-20 16:11:08 -08:00
Fangjin Yang 00ecf9d028 sorted db datasources for rules; smaller time boundary range 2012-12-18 16:16:03 -08:00
xvrl 8b55790900 add segmentMetadata query type 2012-12-18 10:15:23 -08:00
xvrl 54d8aff4d5 add parameter for benchmarking host 2012-12-17 16:44:40 -08:00
xvrl 2118252ef2 make memcached host configurable for benchmarking 2012-12-17 16:44:39 -08:00
Fangjin Yang 74a977504e bug fix for dropping segments in master 2012-12-14 17:06:03 -08:00
Eric Tschetter 62ea45f2c9 1) Register SearchQueryDimFilter with DimFilter so that it can actually be used. 2012-12-13 17:55:32 -06:00
Fangjin Yang 4bc721f3b3 revert last commit 2012-12-12 16:09:07 -08:00
Fangjin Yang 6828ad10d4 reduce boundaries for time boundary a bit; prevent bound errors 2012-12-12 16:07:47 -08:00
cheddar 55652861da Merge pull request #44 from metamx/segment-metadata-query 
Barebones segment metadata query
 2012-12-12 15:33:50 -08:00
Fangjin Yang dd25868919 add missing headers to files 2012-12-12 15:03:26 -08:00
cheddar f4a413c83f Merge pull request #40 from metamx/broker-cacheconfig 
configuration option to switch cache implementations
 2012-12-12 14:54:52 -08:00
Fangjin Yang 6db99c45a6 more specific exceptions in test 2012-12-12 14:51:19 -08:00
xvrl bfc9b63931 segment metadata query 2012-12-12 13:45:12 -08:00
Fangjin Yang 436828b67c post aggs verify that they have all the fields they need 2012-12-12 10:36:46 -08:00