Commit Graph

7354 Commits

Author SHA1 Message Date
Himanshu ab4209c82a killDataSourceWhitelist in CoordinatorDynamicConfig accepts comma separated list of strings in addition to json array of strings so that coordinator console can do the updates correctly (#3095) 2016-06-07 15:39:41 -07:00
Jonathan Wei c5dbf364e3 Fix JSON flatten docs, add link to path expression tester (#3105) 2016-06-07 14:39:57 -07:00
Kirill Kozlov 4ab675e863 Fix command name in example (#3088) 2016-06-07 10:44:27 -07:00
Kirill Kozlov 9f93be448e Fix logical operator in example (#3093) 2016-06-07 10:44:18 -07:00
Gian Merlino 99ee3f4dc3 Fixups, clarifications to lookup docs. (#3060) 2016-06-07 10:43:35 -07:00
Charles Allen fa41a6466a Cleanup the base lookup cluster wide config docs (#3061)
* Cleanup the base lookup cluster wide config docs

* Add better examples in lookups-cached-global.md

* Use actual valid stock lookups

* Fixed maps with :

* Add mix of lookups

* Better examples in extension

* Remove unneeded namespace requirement

* Add extra line space

* Add link to lookup tiers

* Renamed header
2016-06-07 10:42:41 -07:00
Keuntae Park e6b32c24ae bug fix for getNewNodes() in ListenerDiscoverer (#3092) 2016-06-07 16:32:42 +05:30
David Lim 5a3db634ff add synchronization to SupervisorManager (#3077) 2016-06-07 00:29:23 -06:00
Charles Allen 8cac710546 Async lookups-cached-global by default (#3074)
* Async lookups-cached-global by default
* Also better lookup docs

* Fix test timeouts

* Fix timing of deserialized test

* Fix problem with 0 wait failing immediately
2016-06-03 15:58:10 -05:00
David Lim a2290a8f05 support seamless config changes (#3051) 2016-06-03 13:50:19 -07:00
David Lim 6d38dde2f8 exclude slf4j-log4j12 (#3075) 2016-06-03 11:39:23 -07:00
Gian Merlino 54139c6815 Fix NPE in registeredLookup extractionFn when "optimize" is not provided. (#3064) 2016-06-03 12:58:17 -05:00
Gian Merlino 2db5f49f35 Fix JavaScriptConfig. (#3062) 2016-06-02 23:59:00 -07:00
Charles Allen bbc5509078 Limit number of jetty threads that can be used by lookups (#3068) 2016-06-02 22:33:12 -07:00
Gian Merlino 6171e078c8 Improve NPE message in LookupDimensionSpec when lookup does not exist. (#3065)
The message used to be empty, which made things hard to debug.
2016-06-02 19:59:12 -07:00
Gian Merlino 603fbbcc20 Fix docs for "contains" search spec. (#3066) 2016-06-02 19:03:40 -07:00
Charles Allen 474286bbce Make TaskMaster giant lock fair (#3050) 2016-06-02 12:10:40 -07:00
Vadim Ogievetsky 13c267bfee Added new line for site formatting (#3059) 2016-06-02 11:36:45 -07:00
Parag Jain 44237e25d9 fix duration format and number format (#3057) 2016-06-02 10:09:21 -07:00
Erik Dubbelboer b4737336e5 Added info about Google Cloud Storage (#3056) 2016-06-02 10:06:07 -07:00
Charles Allen 447033985e Make S3DataSegmentMover not bother checking for items if they are the same (#3032)
* Make S3DataSegmentMover not bother checking for items if they are the same
2016-06-02 17:27:21 +01:00
sainath batthala d552a5c034 Documented getAuditTime, getPayload methods in AuditEntry.java (#3045)
* Documented getAuditTime, getPayload methods in AuditEntry.java

* author tag removed from documentation
2016-06-02 08:20:33 -07:00
Slim 545cdd63ab ensure the cleaning of overshadowed unloaded segments (#3048)
* ensure the cleaning of overshadowed unloaded segments

* add testing plus comments
2016-06-02 09:03:58 -05:00
Vadim Ogievetsky 767190d5db Clear up confusing wording (#3052)
There is no such thing as a "Java aggregator" in Druid from a user's point of view, there are just native aggregator that happen to be implemented in Java.
2016-06-01 15:41:50 -07:00
Gian Merlino 874a0a4bdd MetadataResource: Fix handling of includeDisabled. (#3042) 2016-06-01 11:56:37 -07:00
Gian Merlino 5e44ed7132 Fix redirect-making and version-replacing in deploy-docs. (#3041) 2016-05-31 16:36:10 -07:00
Gian Merlino cd5c5419bb Make docs deploying better. (#3040)
- Make redirects for old links based on _redirects.json
- Replace #{DRUIDVERSION} tokens in docs with current version
- Allow origins named something other than "origin"
- Can use either s3cmd or awscli, depending on availability
2016-05-31 15:34:58 -07:00
David Lim f6c39cc844 Kafka task minimum message time (#3035)
* add KafkaIndexTask support for minimumMessageTime

* add Kafka supervisor support for lateMessageRejectionPeriod
2016-05-31 11:37:00 -07:00
John Wang e662efa79f segment interface refactor for proposal 2965 (#2990) 2016-05-26 20:36:41 -07:00
John Wang a004f1e1c5 appenderator plumber work from gian's branch (#2913) 2016-05-26 14:46:32 -07:00
Charles Allen 847501a939 Add better messages around LookupCoordinatorManager failures (#3027)
* Add better messages around LookupCoordinatorManager failures
* Catches #3026

* A few more little tests

* Add more forceful shutdown
2016-05-26 14:32:35 -05:00
jaehong choi e2653a8cf4 handle a NPE in LookupCoordinatorManager.start() (#3026) 2016-05-26 09:55:33 -07:00
scusjs ebb6831770 rm , of jobProperties. jackson can not parse it (#3012) 2016-05-26 09:46:33 -07:00
David Lim 3ef24c03b3 Validate X-Druid-Task-Id header in request/response and support retrying on outdated TaskLocation information, add KafkaIndexTaskClient unit tests (#3006)
* validate X-Druid-Task-Id header in request and add header to response

* modify KafkaIndexTaskClient to take a TaskLocationProvider as the TaskLocation may not remain constant
2016-05-25 22:05:18 -07:00
Kurt Young b5bd406597 fix #2991: race condition in OnheapIncrementalIndex#addToFacts (#3002)
* fix #2991: race condition in OnheapIncrementalIndex#addToFacts

* add missing header

* handle parseExceptions when first doing first agg
2016-05-25 19:05:46 -07:00
Jonathan Wei b72c54c4f8 Add benchmark data generator, basic ingestion/persist/merge/query benchmarks (#2875) 2016-05-25 16:39:37 -07:00
Himanshu 7e67397b5a fix-3010: look through all versions to find the set with complete partitions (#3013) 2016-05-25 11:01:22 -07:00
Charles Allen 245077b47f Fix formatting in lookups-cached-global.md (#3009) 2016-05-24 17:28:39 -07:00
Charles Allen c738c0e1cd Silly Typo in docs 2016-05-24 13:31:58 -07:00
Charles Allen 8024b915e2 [QTL] Implement LookupExtractorFactory of namespaced lookup (#2926)
* support LookupReferencesManager registration of namespaced lookup and eliminate static configurations for lookup from namespecd lookup extensions

- druid-namespace-lookup and druid-kafka-extraction-namespace are modified
- However, druid-namespace-lookup still has configuration about ON/OFF
  HEAP cache manager selection, which is not namespace wide
  configuration but node wide configuration as multiple namespace shares
  the same cache manager

* update KafkaExtractionNamespaceTest to reflect argument signature changes

* Add more synchronization functionality to NamespaceLookupExtractorFactory

* Remove old way of using extraction namespaces

* resolve compile error by supporting LookupIntrospectHandler

* Remove kafka lookups

* Remove unused stuff

* Fix start and stop behavior to be consistent with new javadocs

* Remove unused strings

* Add timeout option

* Address comments on configurations and improve docs

* Add more options and update hash key and replaces

* Move monitoring to the overriding classes

* Add better start/stop logging

* Remove old docs about namespace names

* Fix bad comma

* Add `@JsonIgnore` to lookup factory

* Address code review comments

* Remove ExtractionNamespace from module json registration

* Fix problems with naming and initialization. Add tests

* Optimize imports / reformat

* Fix future not being properly cancelled on failed initial scheduling

* Fix delete returns

* Add more docs about whole introspection

* Add `/version` introspection point for lookups

* Add more tests and address comments

* Add StaticMap extraction namespace for testing. Also add a bunch of tests

* Move cache system property to `druid.lookup.namespace.cache.type`

* Make VERSION lower case

* Change poll period to 0ms  for StaticMap

* Move cache key to bytebuffer

* Change hashCode and equals on static map extraction fn

* Add more comments on StaticMap

* Address comments

* Make scheduleAndWait use a latch

* Sanity renames and fix imports

* Remove extra info in docs

* Fix review comments

* Strengthen failure on start from warn to error

* Address comments

* Rename namespace-lookup to lookups-cached-global

* Fix injective mis-naming
* Also add serde test
2016-05-24 10:56:40 -07:00
Nishant 0ac1b27d53 Allow manually setting of shutoffTime for EventReceiverFirehose (#2803)
* Allow dynamically setting of shutoffTime for EventReceiverFirehose

Allow dynamically setting shutoffTime for EventReceiverFirehose

review comments and tests

* shut down exec on close
2016-05-24 07:24:00 -07:00
Nishant dea4391a49 fix broken links (#3003) 2016-05-23 06:38:21 -07:00
Dave Li dcabd4b1ee Add lookup optimization for InDimFilter (#2938)
* Add lookup optimization for InDimFilter

* tests for in filter with lookup extraction fn

* refactor

* refactor2 and modified filter test

* make optimizeLookup private
2016-05-19 16:29:16 -07:00
Fangjin Yang 00de26c76a fix extensions docs (#2995)
* fix extensions docs

* fix mistakes
2016-05-19 14:01:06 -07:00
Charles Allen 99e8fb0216 Fix typo in CONTRIBUTING.md (#2988) 2016-05-18 13:46:53 -07:00
Nishant 778f97a80e attempt to fix-2906 (#2985)
* attempt to fix-2984

* review comments

* Add test
2016-05-18 15:12:38 -05:00
Charles Allen a8c2f07f1c Make formatting IntelliJ 2016 friendly (#2978) 2016-05-18 12:42:21 -07:00
Gian Merlino 970614875b Fix race where results from an IncrementalIndexSegment could be cached. (#2983) 2016-05-18 13:57:50 +05:30
Charles Allen 15ccf451f9 Move QueryGranularity static fields to QueryGranularities (#2980)
* Move QueryGranularity static fields to QueryGranularityUtil
* Fixes #2979

* Add test showing #2979

* change name to QueryGranularities
2016-05-17 16:23:48 -07:00
Charles Allen eaaad01de7 [QTL] Datasource as lookupTier (#2955)
* Datasource as lookup tier
* Adds an option to let indexing service tasks pull their lookup tier from the datasource they are working for.

* Fix bad docs for lookups lookupTier

* Add Datasource name holder

* Move task and datasource to be pulled from Task file

* Make LookupModule pull from bound dataSource

* Fix test

* Fix code style on imports

* Fix formatting

* Make naming better

* Address code comments about naming
2016-05-17 15:44:42 -07:00