Commit Graph

7278 Commits

Author SHA1 Message Date
Gian Merlino cd5c5419bb Make docs deploying better. (#3040)
- Make redirects for old links based on _redirects.json
- Replace #{DRUIDVERSION} tokens in docs with current version
- Allow origins named something other than "origin"
- Can use either s3cmd or awscli, depending on availability
2016-05-31 15:34:58 -07:00
David Lim f6c39cc844 Kafka task minimum message time (#3035)
* add KafkaIndexTask support for minimumMessageTime

* add Kafka supervisor support for lateMessageRejectionPeriod
2016-05-31 11:37:00 -07:00
John Wang e662efa79f segment interface refactor for proposal 2965 (#2990) 2016-05-26 20:36:41 -07:00
John Wang a004f1e1c5 appenderator plumber work from gian's branch (#2913) 2016-05-26 14:46:32 -07:00
Charles Allen 847501a939 Add better messages around LookupCoordinatorManager failures (#3027)
* Add better messages around LookupCoordinatorManager failures
* Catches #3026

* A few more little tests

* Add more forceful shutdown
2016-05-26 14:32:35 -05:00
jaehong choi e2653a8cf4 handle a NPE in LookupCoordinatorManager.start() (#3026) 2016-05-26 09:55:33 -07:00
scusjs ebb6831770 rm , of jobProperties. jackson can not parse it (#3012) 2016-05-26 09:46:33 -07:00
David Lim 3ef24c03b3 Validate X-Druid-Task-Id header in request/response and support retrying on outdated TaskLocation information, add KafkaIndexTaskClient unit tests (#3006)
* validate X-Druid-Task-Id header in request and add header to response

* modify KafkaIndexTaskClient to take a TaskLocationProvider as the TaskLocation may not remain constant
2016-05-25 22:05:18 -07:00
Kurt Young b5bd406597 fix #2991: race condition in OnheapIncrementalIndex#addToFacts (#3002)
* fix #2991: race condition in OnheapIncrementalIndex#addToFacts

* add missing header

* handle parseExceptions when first doing first agg
2016-05-25 19:05:46 -07:00
Jonathan Wei b72c54c4f8 Add benchmark data generator, basic ingestion/persist/merge/query benchmarks (#2875) 2016-05-25 16:39:37 -07:00
Himanshu 7e67397b5a fix-3010: look through all versions to find the set with complete partitions (#3013) 2016-05-25 11:01:22 -07:00
Charles Allen 245077b47f Fix formatting in lookups-cached-global.md (#3009) 2016-05-24 17:28:39 -07:00
Charles Allen c738c0e1cd Silly Typo in docs 2016-05-24 13:31:58 -07:00
Charles Allen 8024b915e2 [QTL] Implement LookupExtractorFactory of namespaced lookup (#2926)
* support LookupReferencesManager registration of namespaced lookup and eliminate static configurations for lookup from namespecd lookup extensions

- druid-namespace-lookup and druid-kafka-extraction-namespace are modified
- However, druid-namespace-lookup still has configuration about ON/OFF
  HEAP cache manager selection, which is not namespace wide
  configuration but node wide configuration as multiple namespace shares
  the same cache manager

* update KafkaExtractionNamespaceTest to reflect argument signature changes

* Add more synchronization functionality to NamespaceLookupExtractorFactory

* Remove old way of using extraction namespaces

* resolve compile error by supporting LookupIntrospectHandler

* Remove kafka lookups

* Remove unused stuff

* Fix start and stop behavior to be consistent with new javadocs

* Remove unused strings

* Add timeout option

* Address comments on configurations and improve docs

* Add more options and update hash key and replaces

* Move monitoring to the overriding classes

* Add better start/stop logging

* Remove old docs about namespace names

* Fix bad comma

* Add `@JsonIgnore` to lookup factory

* Address code review comments

* Remove ExtractionNamespace from module json registration

* Fix problems with naming and initialization. Add tests

* Optimize imports / reformat

* Fix future not being properly cancelled on failed initial scheduling

* Fix delete returns

* Add more docs about whole introspection

* Add `/version` introspection point for lookups

* Add more tests and address comments

* Add StaticMap extraction namespace for testing. Also add a bunch of tests

* Move cache system property to `druid.lookup.namespace.cache.type`

* Make VERSION lower case

* Change poll period to 0ms  for StaticMap

* Move cache key to bytebuffer

* Change hashCode and equals on static map extraction fn

* Add more comments on StaticMap

* Address comments

* Make scheduleAndWait use a latch

* Sanity renames and fix imports

* Remove extra info in docs

* Fix review comments

* Strengthen failure on start from warn to error

* Address comments

* Rename namespace-lookup to lookups-cached-global

* Fix injective mis-naming
* Also add serde test
2016-05-24 10:56:40 -07:00
Nishant 0ac1b27d53 Allow manually setting of shutoffTime for EventReceiverFirehose (#2803)
* Allow dynamically setting of shutoffTime for EventReceiverFirehose

Allow dynamically setting shutoffTime for EventReceiverFirehose

review comments and tests

* shut down exec on close
2016-05-24 07:24:00 -07:00
Nishant dea4391a49 fix broken links (#3003) 2016-05-23 06:38:21 -07:00
Dave Li dcabd4b1ee Add lookup optimization for InDimFilter (#2938)
* Add lookup optimization for InDimFilter

* tests for in filter with lookup extraction fn

* refactor

* refactor2 and modified filter test

* make optimizeLookup private
2016-05-19 16:29:16 -07:00
Fangjin Yang 00de26c76a fix extensions docs (#2995)
* fix extensions docs

* fix mistakes
2016-05-19 14:01:06 -07:00
Charles Allen 99e8fb0216 Fix typo in CONTRIBUTING.md (#2988) 2016-05-18 13:46:53 -07:00
Nishant 778f97a80e attempt to fix-2906 (#2985)
* attempt to fix-2984

* review comments

* Add test
2016-05-18 15:12:38 -05:00
Charles Allen a8c2f07f1c Make formatting IntelliJ 2016 friendly (#2978) 2016-05-18 12:42:21 -07:00
Gian Merlino 970614875b Fix race where results from an IncrementalIndexSegment could be cached. (#2983) 2016-05-18 13:57:50 +05:30
Charles Allen 15ccf451f9 Move QueryGranularity static fields to QueryGranularities (#2980)
* Move QueryGranularity static fields to QueryGranularityUtil
* Fixes #2979

* Add test showing #2979

* change name to QueryGranularities
2016-05-17 16:23:48 -07:00
Charles Allen eaaad01de7 [QTL] Datasource as lookupTier (#2955)
* Datasource as lookup tier
* Adds an option to let indexing service tasks pull their lookup tier from the datasource they are working for.

* Fix bad docs for lookups lookupTier

* Add Datasource name holder

* Move task and datasource to be pulled from Task file

* Make LookupModule pull from bound dataSource

* Fix test

* Fix code style on imports

* Fix formatting

* Make naming better

* Address code comments about naming
2016-05-17 15:44:42 -07:00
Charles Allen fb01db4db7 [QTL] Allows RegisteredLookupExtractionFn to find its lookups lazily (#2971)
* Allows RegisteredLookupExtractionFn to find its lookups lazily

* Use raw variables instead of AtomicReference

* Make sure to use volatile

* Remove extra local variable.

* Move from BAOS to ByteBuffer
2016-05-17 11:29:39 -07:00
Xavier Léauté e79284da59 new interval based cost function (#2972)
* new interval based cost function

Addresses issues with balancing of segments in the existing cost function
- `gapPenalty` led to clusters of segments ~30 days apart
- `recencyPenalty` caused imbalance among recent segments
- size-based cost could be skewed by compression

New cost function is purely based on segment intervals:
- assumes each time-slice of a partition is a constant cost
- cost is additive, i.e. cost(A, B union C) = cost(A, B) + cost(A, C)
- cost decays exponentially based on distance between time-slices

* comments and formatting

* add more comments to explain the calculation
2016-05-17 09:56:00 -07:00
Charles Allen 2863adef60 Add jconsole.sh example script for connecting (#2947)
* Can change logging at run-time
2016-05-16 15:37:15 -07:00
Shekhar Gulati c41bfe50d0 Using quotes around the cp (#2934) 2016-05-16 15:16:48 -07:00
Gian Merlino a54381a084 Fix CombiningSequence.close on single element sequences. (#2969)
Regression introduced by #2892.
2016-05-13 23:12:30 -07:00
Parag Jain 681ffdb417 try to make DruidCoordinatorTest deterministic (#2967) 2016-05-13 14:43:28 -07:00
Nishant a9b721a01b Allow user to set cost balancer threads more than or equal to the number of cores. (#2964)
* Allow user to set cost balancer threads more than the number of cores.

Allow user to set cost balancer threads more than the number of cores.

* modify test
2016-05-13 13:27:42 -05:00
Himanshu d3e9c47a5f use correct ObjectMapper in Index[IO/Merger] in AggregationTestHelper and minor fix in theta sketch SketchMergeAggregatorFactory.getMergingFactory(..) (#2943) 2016-05-13 10:06:31 +05:30
Parag Jain e3ea842cd3 add available query granularity strings (#2960) 2016-05-12 18:49:31 -07:00
Himanshu d821144738 at historicals GpBy query mergeResults does not need merging as results are already merged by GroupByQueryRunnerFactory.mergeRunners(..) (#2962) 2016-05-12 17:41:24 -07:00
Slim abf64a13b0 reconnect to the graphite after transient disconnect (#2952)
* reconnect to the graphite after transient disconnect

* catch the socket exception and retry
2016-05-12 11:32:36 -07:00
Gian Merlino 01bebf432a GroupByQuery: Multi-value dimension tests. (#2959) 2016-05-12 11:31:50 -07:00
Charles Allen 81cab8a7bb Make lookups more idempotent on update requests. (#2954)
* No longer fails if an update fails but it shouldn't have replaced it
2016-05-11 11:22:35 -07:00
Jonathan Wei f2510cf125 Remove DataSchema equals() and hashcode() 2016-05-10 16:09:28 -07:00
Charles Allen 6332bd70f4 Add smile provider (#2951) 2016-05-10 16:03:39 -07:00
Joe Pettersson 2288c78395 chore_fix-quickstart-docs (#2946)
Fixes a small grammatical error in the `./docs/content/tutorials/quickstart.md` whereby a sentence didn't make sense
2016-05-10 09:52:24 -07:00
Walton Seymour 13a2b265a5 Adds init scripts (#2939) 2016-05-10 09:00:13 -07:00
Slim 45b2e65d75 [QTL] adding listDelimiter to lookup parser spec (#2941)
* adding listDelimiter to lookup parser spec

* cleaning code
2016-05-10 15:41:16 +05:30
Charles Allen 0c04650e69 Lookup Announcer eager starting (#2944) 2016-05-10 12:21:47 +05:30
Charles Allen 454bb034f1 Nicer toString on ListneingAnnouncerConfig (#2936)
* Helps with debugging
2016-05-10 12:21:06 +05:30
Charles Allen a31348450f Add toString for LookupConfig (#2935)
* Helps with operations and getting where the snapshot dir is
2016-05-09 18:20:00 -07:00
Gian Merlino b8af84d1fc Update tutorials to tranquility v0.8.0. (#2937) 2016-05-09 11:50:37 -07:00
Dave Li 79a54283d4 Optimize filter for timeseries, search, and select queries (#2931)
* Optimize filter for timeseries, search, and select queries

* exception at failed toolchest type check

* took out query type check

* java7 error fix and test improvement
2016-05-09 11:04:06 -07:00
David Lim 2cfd337378 Merge pull request #2933 from dclim/SQLMetadataSupervisorManagerTest-fix
add uuid to primary key for supervisor table
2016-05-09 10:41:32 -06:00
Charles Allen 90b0b0a4ad Make URIExtraction not require FileSystem impls for URIs it understands (#2929)
* Make URIExtraction not require FileSystem impls for URIs it understands
* Fixes #2928

* Preserve URI information

* Simply case for exact matching

* Move unused variable
2016-05-08 23:23:53 +05:30
Slim 8b570ab130 make it clear what LookupExtractorFactory start/stop methods return (#2925) 2016-05-05 10:38:40 -07:00