Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,529 Commits 111 Branches 503 Tags 726 MiB
Commit Graph

1008 Commits

Author SHA1 Message Date
Fangjin Yang 4f46d457f1 Merge pull request #1947 from noddi/feature/count-parameter-history-endpoints 
Add count parameter to history endpoints
 2015-11-12 10:23:44 -08:00
Gian Merlino e4e5f0375b SegmentAllocateAction (fixes #1515) 
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).

The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.

The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
 2015-11-11 16:54:35 -08:00
Bartosz Ługowski 6e5d2c6745 Add count parameter to history endpoints. 2015-11-11 23:03:57 +01:00
Xavier Léauté fa6142e217 cleanup and remove unused imports 2015-11-11 12:25:21 -08:00
Xavier Léauté a57cbfd2c3 Merge pull request #1387 from metamx/enableShutdownLogging 
Add special handler to allow logger messages during shutdown
 2015-11-09 17:20:09 -08:00
Charles Allen 1df4baf489 Move Jackson Guice adapters into io.druid 
* Removes access to protected methods in com.fasterxml
* Eliminates druid-common's use of foreign package com.fasterxml
 2015-11-09 10:50:45 -08:00
Fangjin Yang f90ddfdb89 Merge pull request #1745 from himanshug/numbered_to_elastic 
making NumberedShardSpec elastic
 2015-10-30 16:01:03 -07:00
Charles Allen 7a2ceef690 Add special handler to allow logger messages during shutdown 
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
 2015-10-27 14:33:36 -07:00
Xavier Léauté 72c408cf2d Merge pull request #1770 from metamx/merge-time 
Add segment merge time as a metric
 2015-10-22 22:03:41 -07:00
Nishant 7cecc55045 Add segment merge time as a metric 
Add merge and persist cpu time

Fix typo

review comment

move cpu time measuring to VMUtils

review comments.
 2015-10-22 12:28:03 +05:30
Charles Allen 1cad571354 Add more verbose logging to SQLMetadataRuleManager 2015-10-21 16:11:40 -07:00
Xavier Léauté e4ac78e43d bump next snapshot to 0.9.0 2015-10-20 13:46:13 -07:00
Xavier Léauté 4c2c7a2c37 update version to 0.8.3 2015-10-14 21:40:55 -07:00
Nishant 573aa96bd6 fix #1727 - Union bySegment queries fix 
Fixes #1727.
revert to doing merging for results for union queries on broker.

revert unrelated changes

Add test for union query runner

Add test

remove unused imports

fix imports

fix renamed file

fix test

update docs.
 2015-09-29 23:32:36 +05:30
Xavier Léauté 1f897257b5 add simple load balancing pool 2015-09-18 09:43:14 -07:00
Himanshu Gupta a7b1cacbbf make NumberedShardSpec "elastic" and UTs 2015-09-17 08:38:48 -05:00
Himanshu Gupta 22dfa8ecf7 change ParitionHolder.isComplete() to accept chunks beyond the "end" 2015-09-16 21:33:11 -05:00
nishant 0835e12f2a add endpoint to fetch rule history for all datasources. 
review comments

Review comment

fix compilation
 2015-08-31 14:05:52 +05:30
Gian Merlino 940e1aa3eb Replace funky imports with standard ones. 
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
 2015-08-28 18:02:05 -07:00
Himanshu Gupta 2e0dd1d792 adding UTs and addressing review comments to 
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
 2015-08-27 20:50:46 -05:00
lvjq 2237a8cf0f kafka 8 simple consumer firehose 2015-08-27 20:50:46 -05:00
Gian Merlino 2bf9a70bfa Consolidate SQL retrying by moving logic into the connectors. 
Also change boolean removeLock to void addLock in MetadataStorageActionHandler.
 2015-08-25 12:42:29 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Charles Allen db19d2d547 Revert "Update to guice 4.0" 2015-08-14 09:26:07 -07:00
Charles Allen 7e61216287 Update to guice 4.0 
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
 2015-08-10 13:57:18 -07:00
Charles Allen 7fe8562980 Remove locks from StupidPool 2015-08-05 19:24:56 -07:00
Charles Allen 86ede702b1 Add namespaced lookups as extensions 
* Adds kafka, URI, and JDBC namespace defintions
* Add ability to explicitly rename using a "namespace" which is a particular data collection that is loaded on all realtime, historic nodes, and brokers. If any of these nodes has the namespace extension, ALL nodes have the namespace extension.
* Add namespace caching and populating (can be on heap or off heap)
* Add NamespaceExtractionCacheManager for handling caches
* Added ExtractionNamespace for handling metadata on the extraction namespaces
* Added ExtractionNamespaceUpdate for handling metadata related to updates
* Add extension which caches renames from a kafka stream (requires kafka8)
* Added README.md for the namespace kafka extension
* Added docs
* Added namespace/size, namespace/count, namespace/deltaTasksStarted metrics

Add static config for namespaces via `druid.query.extraction.namespace`
* This is a rebase of https://github.com/b-slim/druid/tree/static_config_only
 2015-07-28 11:14:14 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Charles Allen a37d631cea Add log4j2 for debugging which can be specified with `-Dlog4j.configurationFile=/some/path/druid/common/src/main/resources/log4j2.debug.xml` during testing and development 2015-07-01 10:29:22 -07:00
Xavier Léauté 0a5bb909a2 [maven-release-plugin] prepare for next development iteration 2015-06-18 17:35:19 -07:00
Xavier Léauté 59c6b2b279 [maven-release-plugin] prepare release druid-0.8.0-rc1 2015-06-18 17:35:14 -07:00
fjy 7a6acf5c1b update pom to 0.8 2015-05-11 19:41:58 -06:00
nishant 50158357ff fixes #1330 
fixes #1330,
Avoid creating Period instance as creating a Period from Long.MAX_VALUE
throws arithmetic exception.
After this query metric will emit duration in seconds instead of
minutes.
 2015-05-04 20:34:28 +05:30
Xavier Léauté f0726f4d94 fix typing on StupidResourceHolder 2015-04-13 21:28:56 -07:00
Fangjin Yang 208e307915 Merge pull request #1251 from metamx/uriSegmentLoaders 
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
 2015-03-30 17:43:51 -07:00
fjy aea7f9d192 [maven-release-plugin] prepare for next development iteration 2015-03-30 16:35:24 -07:00
fjy 060d7aef03 [maven-release-plugin] prepare release druid-0.7.1 2015-03-30 16:35:20 -07:00
Charles Allen 1c6cbea89c Revert "Revert "Overhaul of SegmentPullers to add consistency and retries"" 
This reverts commit f904bc7858.
 2015-03-30 13:40:04 -07:00
Fangjin Yang f904bc7858 Revert "Overhaul of SegmentPullers to add consistency and retries" 2015-03-30 13:15:50 -07:00
Charles Allen 6d407e8677 Add URI handling to SegmentPullers 
* Requires https://github.com/druid-io/druid-api/pull/37
* Requires https://github.com/metamx/java-util/pull/22
* Moves the puller logic to use a more standard workflow going through java-util helpers instead of re-writing the handlers for each impl
  * General workflow goes like this: 1) LoadSpec makes sure the correct Puller is called with the correct parameters. 2) The Puller sets up general information like how to make an InputStream, how to find a file name (for .gz files for example), and when to retry. 3) CompressionUtils does most of the heavy lifting when it can
 2015-03-30 12:33:23 -07:00
Xavier Léauté 2c58e355fd Merge pull request #1239 from metamx/log4j2forRealz 
Exclude log4j from curator dependencies in favor of log4j-1.2-api
 2015-03-26 13:21:13 -07:00
Charles Allen 9cd6c08e96 Exclude log4j from curator dependencies in favor of log4j-1.2-api 2015-03-26 13:05:12 -07:00
Xavier Léauté 389ea4c32f Merge pull request #1245 from b-slim/fix_injector_plus_ut 
Bug fix @DruidSecondaryModule plus unit test
 2015-03-26 10:04:44 -07:00
Slim Bouguerra 1e6be7796e bug fix @DruidSecondaryModule plus unit test 2015-03-26 10:44:52 -05:00
nishantmonu51 638bf9d4e9 return sorted List of TimeLineObjectHolder 2015-03-26 11:51:09 +05:30
fjy b389cfe404 [maven-release-plugin] prepare for next development iteration 2015-03-19 12:38:17 -07:00
fjy 60e7d543cc [maven-release-plugin] prepare release druid-0.7.1-rc1 2015-03-19 12:38:13 -07:00
Fangjin Yang 6bcdf7f60f Merge pull request #1223 from metamx/AnnounceIdUUID 
Add consistency to UUIDUtils handling
 2015-03-17 16:09:58 -07:00
Charles Allen 868100c79c Add consistency to UUIDUtils handling 
* Use "_" instead of "-"
 2015-03-17 16:05:46 -07:00
Fangjin Yang 8d0dfd9af1 Merge pull request #1221 from metamx/AnnounceIdUUID 
Change announcement ID to a UUID instead of Timestamp
 2015-03-17 15:38:53 -07:00
Charles Allen 79b1443cc3 Change announcement ID to a UUID instead of Timestamp 
* Also add a UUIDUtils to the common package
 2015-03-17 15:34:04 -07:00
nishantmonu51 f9821d242f also log author ip address in audit log 2015-03-17 23:15:15 +05:30
Xavier Léauté 7fb4b1d2bb Merge pull request #1144 from metamx/javaLoggingManager 
Add log4j2 hooks to other loggings
 2015-03-12 14:38:51 -07:00
Charles Allen edfcea18d8 Add log4j2 hooks to standard java logging 2015-03-12 14:29:41 -07:00
nishantmonu51 3ec4a30ab5 initial commit 
review comments

more refactoring and cleaning of redundant code

add UT + docs + more refactoring

fixes + review comments

more cleanup

end points to fetch history

review comments

remove unnecessary changes

review comments rename header name

review comments + add test for MetadataRulesManager

review comments docs
 2015-03-12 22:50:29 +05:30
Himanshu Gupta e6ee98e2d2 UTs update for common 2015-02-25 15:40:26 -08:00
Xavier Léauté 0784d7e30e Merge pull request #1152 from himanshug/metastorage-pwd-provider 
support for metadata store PasswordProvider interface
 2015-02-25 15:19:37 -08:00
Himanshu Gupta 126262edce support for PasswordProvider interface to enable writing druid extension which can get metadata store password from secured location or anywhere instead of plain text properties file 2015-02-25 14:05:19 -06:00
Xavier Léauté b167dcf82c [maven-release-plugin] prepare for next development iteration 2015-02-23 14:28:06 -08:00
Xavier Léauté e81ac2ba43 [maven-release-plugin] prepare release druid-0.7.0 2015-02-23 14:27:58 -08:00
Xavier Léauté 78df7f6165 Move Druid release artifacts to Sonatype 
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
 2015-02-13 14:26:31 -08:00
fjy d29740ed9f [maven-release-plugin] prepare for next development iteration 2015-02-12 16:16:00 -08:00
fjy 211fd15b7e [maven-release-plugin] prepare release druid-0.7.0-rc3 2015-02-12 16:15:56 -08:00
nishantmonu51 e5fa8e0cb4 Fix NPE for union queries 
filter non-existing datasources for union queries.

handle review comment
 2015-02-10 23:21:58 +05:30
Fangjin Yang 25cf15824b Merge pull request #1085 from gianm/dsmrv-fix 
DataSourceMetadataResultValue fixes and JodaUtils adjustments.
 2015-02-03 17:51:33 -08:00
Gian Merlino 1298b4abd3 JodaUtils: Bring in MIN/MAX instants a bit. 
This prevents the difference overflowing a long.
 2015-02-03 17:34:33 -08:00
fjy 1f12c5b2f1 [maven-release-plugin] prepare for next development iteration 2015-02-03 12:06:49 -08:00
fjy e82d431be7 [maven-release-plugin] prepare release druid-0.7.0-rc2 2015-02-03 12:06:41 -08:00
fjy 3e5d338c8e Remove non friendly dependencies from Druid 2015-02-03 11:36:08 -08:00
Eric Tschetter 42eba986ce Towards consistent null handling 
This commit also includes
1) the addition of a context parameter on timeseries queries that allows it to ignore empty buckets instead of generating results for them
2) A cleanup of an unused method on an interface
 2015-02-02 12:53:07 -08:00
Fangjin Yang 92e616de11 Merge pull request #1077 from metamx/remove-unused-imports 
remove unused imports
 2015-02-02 10:45:27 -08:00
nishantmonu51 ba932bb1f2 remove unused imports 2015-02-02 21:53:39 +05:30
fjy d05032b98a towards a community led druid 2015-01-31 20:57:36 -08:00
fjy 1f94de22c6 [maven-release-plugin] prepare for next development iteration 2015-01-20 14:23:55 -08:00
fjy 17476edc31 [maven-release-plugin] prepare release druid-0.7.0-rc1 2015-01-20 14:23:51 -08:00
Charles Allen 3d27747f7e Upgrade to log4j2 
Default behavior is as before.
Added documentation for how to enable synchronous logging for select chatty classes:
* io.druid.client.ServerInventoryView
* io.druid.client.BatchServerInventoryView
* io.druid.curator.inventory.CuratorInventoryManager
* com.metamx.http.client.pool.ChannelResourceFactory
 2015-01-20 12:35:18 -08:00
fjy 84cc32ba32 Add more logging for when ingestion may be throttled 2015-01-08 14:51:46 -08:00
fjy 882874ce60 address cr 2014-12-16 11:37:38 -08:00
fjy b3999bbc6a update http client and fix log4j dependencies 2014-12-16 11:29:50 -08:00
fjy c0af7b2a5d fix broken ut 2014-12-15 18:29:15 -08:00
nishantmonu51 1a1b0e6f23 merge from master and review comments 2014-12-09 13:16:45 +05:30
Xavier Léauté 7871470491 fix jackson-annotations version dependencies 2014-12-08 15:31:58 -08:00
Xavier Léauté a79389a9e5 update joda-time and fix min/max instant 2014-12-02 17:27:22 -08:00
Charles Allen dc66e1708e Added src jar build to maven poms and re-formatted to conform to style guidelines. 2014-11-18 09:05:30 -08:00
nishantmonu51 cbffe3c648 merge from master and resolve conflicts 2014-11-17 18:07:08 +05:30
Xavier Léauté d19b895fb9 add tests for empty sequences 2014-11-13 14:49:13 -08:00
fjy 788a4a846b add encoding 2014-11-11 17:31:08 -08:00
Charles Allen a89b539b4f Merge pull request #823 from metamx/roaring 
Configurable bitmap indexes: roaring and concise
 2014-11-11 17:26:38 -08:00
fjy 1cc162727b address code review 2014-11-11 14:05:37 -08:00
Xavier Léauté 59f093882d fix Derby instantiation 2014-11-10 16:55:53 -08:00
fjy 8cbd21af90 fix defaults 2014-11-10 15:40:57 -08:00
fjy 98d05e63bf make derby instance only on coordinator 2014-11-10 15:34:49 -08:00
nishantmonu51 8bebb24fd5 Union Queries move merge to historical 2014-11-07 18:05:16 +05:30
Xavier Léauté 5bda4ee1dd global task entry type 2014-11-06 17:08:20 -08:00
Xavier Léauté ac753ed0f0 always use validation query 2014-11-06 17:08:01 -08:00
Xavier Léauté 10aa8a2d56 add explicit type names 2014-11-06 17:07:24 -08:00
Xavier Léauté 350bb09605 refactor sql storage to abstract task storage 2014-11-05 17:19:37 -08:00
Xavier Léauté 253c0a70b3 simplify interface + more tests 2014-11-04 15:03:13 -08:00
Xavier Léauté 696c9ba944 fix formatting 2014-10-31 13:32:18 -07:00
jisookim0513 a6016d70fc added deleted classes back 2014-10-31 13:10:09 -07:00