Commit Graph

8289 Commits

Author SHA1 Message Date
Clint Wylie 491f8cca81 fix timewarp query results when using timezones and crossing DST transitions (#5157)
* timewarp and timezones
changes:
* `TimewarpOperator` will now compensate for daylight savings time shifts between date translation ranges for queries using a `PeriodGranularity` with a timezone defined
* introduces a new abstract query type `TimeBucketedQuery` for all queries which have a `Granularity` (100% not attached to this name). `GroupByQuery`, `SearchQuery`, `SelectQuery`, `TimeseriesQuery`, and `TopNQuery` all extend `TimeBucke
tedQuery`, cutting down on some duplicate code and providing a mechanism for `TimewarpOperator` (and anything else) that needs to be aware of granularity

* move precondition check to TimeBucketedQuery, add Granularities.nullToAll, add getTimezone to TimeBucketQuery

* formatting

* more formatting

* unused import

* changes:
* add 'getGranularity' and 'getTimezone' to 'Query' interface
* merge 'TimeBucketedQuery' into 'BaseQuery'
* fixup tests from resulting serialization changes

* dedupe

* fix after merge

* suppress warning
2018-01-11 12:39:33 -08:00
Shen Liu 3c69717202 Fix typo in configuration/index.md (#5249) (#5250)
* Fix #5212 - typo in auth.md.

* Fix typo in configuration (#5249)

* Add a backquote.

* Fix typo from HttpEmitterMonitor to HttpEmittingMonitor.
2018-01-11 18:29:12 +09:00
Roman Leventov 8877ce38d6
Enforce modifier order with Checkstyle (#5246) 2018-01-11 09:50:42 +01:00
Atul Mohan 3cc4a0ab19 Support for encryption of MySQL connections (#5122)
* Encrypting MySQL connections

* Update docs

* Make verifyServerCertificate a configurable parameter

* Change password parameter and doc update

* Make server certificate verification disabled by default

* Update tostring

* Update docs

* Add check for trust store passwords

* Add warning for null password
2018-01-10 11:33:54 -08:00
Jihoon Son 5d0619f5ce Support retrying for PrefetchableTextFilesFirehoseFactory when prefetch is disabled (#5162)
* Add RetryingInputStream

* unnecessary exception

* fix PrefetchableTextFilesFirehoseFactoryTest

* Fix retrying on connection reset

* fix start offset

* fix checkstyle

* fix check connection reset

* address comments

* fix compile

* address comments

* address comments
2018-01-10 17:37:19 +01:00
Gian Merlino 7b8b0a96d6 IndexTask: Add summary stats line at the end. (#5241) 2018-01-10 14:06:54 +09:00
Jihoon Son 972b4d189a Fix topN doc (#5240) 2018-01-09 20:10:13 -08:00
Parag Jain 83c6c48bed Fix state check bug in Kafka Index Task (#5204)
* fix state check for replacement task

* fix comments

* rebase with master
2018-01-08 18:01:36 -08:00
Jonathan Wei cdd374a417 Throw away rows with timestamps beyond long bounds in kafka indexing (#5215)
* Throw away rows with timestamps beyond long bounds in kafka indexing

* PR comments
2018-01-08 17:40:50 -06:00
Jonathan Wei 9186547689 Exclude sketches-core from druid-sql (#5223) 2018-01-05 17:12:20 -06:00
Jonathan Wei 02544f9197 Add missing auth doc links (#5224) 2018-01-05 16:23:13 -06:00
Roman Leventov 535ec437e9 Apply 'power of 2' optimization to BlockLayoutIndexedDoubleSupplier (#5176)
* Apply 'power of 2' optimization to BlockLayoutIndexedDoubleSupplier; slight optimization of buffer.get() in block layout indexed suppliers

* Fix byte order
2018-01-05 16:08:07 +09:00
Himanshu a46d34daa2 HTTP based task/worker management. (#5104)
* just renaming of SegmentChangeRequestHistory etc

* additional change history refactoring changes

* WorkerTaskManager a replica of WorkerTaskMonitor

* HttpServerInventoryView refactoring to extract sync code and robustification

* Introducing HttpRemoteTaskRunner

* Additional Worker side updates
2018-01-04 19:19:35 -08:00
Jonathan Wei 935ac646f4
Upgrade to Calcite 1.15.0 (#5210)
* Upgrade to Calcite 1.15.0

* Use Filtration.eternity()
2018-01-04 12:11:24 -08:00
Shen Liu 5a8ea5f8ab Fix #5212 - typo in auth.md. (#5213) 2018-01-04 12:09:42 -08:00
Nishant Bangarwa 4cc31e4e7a Update Zookeeper version (#5184) 2018-01-04 10:59:20 +08:00
Roman Leventov 579f9fbedf Add IndexedInts.debugToString() and AbstractIndex.toString(); Add Sequence.toList() and limit() (#5175)
* Add IndexedInts.debugToString() and AbstractIndex.toString()

* Fix AppenderatorTest
2018-01-04 09:56:47 +09:00
David Lim a7967ade4d Support replaceExisting parameter for segments pushers (#5187)
* support replaceExisting parameter for segments pushers

* code review changes

* code review changes
2018-01-03 16:13:21 -08:00
Nishant Bangarwa 59af4d3b14 Fix broken KafkaEmitterConfig parsing (#5201)
* Fix broken KafkaEmitterConfig parsing

This was a regression introduced in
https://github.com/druid-io/druid/pull/4722

KafkaEmitterConfig property names have dot(.) in the name of properties
and JsonConfigurator behavior was changed to not support that.
Added a test and fixed parsing of properties that have dot(.) in
property names

* Fix test failure
2018-01-03 12:08:40 -08:00
Charles Allen 0f773aff80 Fix lookup logging on node start (#5206)
* Add better logging messages in lookups startup on query nodes

* Make sure list is mutable

* Move list to be with other `final` variables
2018-01-03 13:13:55 -06:00
Shimi Kiviti 0ebfb3fbb0 fixed missing gz extension from wikiticker json sample (#5194) 2017-12-28 01:02:12 +09:00
Yuya Fujiwara 3d3b04e1b8 docs: fix broken link to ingestions and tasks on the Druid Concepts page (#5197)
* fix broken links

* add newline
2017-12-27 07:55:24 -08:00
Slim c3f7da2128 Remove extra logging by making it debug level (#5193)
Change-Id: Iaa255862389bdff7fa42b2c08c1e078448b5ee6c
2017-12-23 00:01:10 +03:00
Jihoon Son b31abd03ad
Fix timeout in RemoteTaskRunnerTest (#5191)
* Fix timeout in RemoteTaskRunnerTest

* add message for npe
2017-12-22 17:40:11 +09:00
Jonathan Wei ba873c614b Fix max connections in DruidAvaticaHandlerTest (#5188)
* Fix max connections in DruidAvaticaHandlerTest

* Fix additional tests

* Added comment
2017-12-21 15:32:14 -06:00
Nishant Bangarwa 6582b16a81 [Integration-tests] Allow specifying additional properties while connecting to kafka (#5185)
* [Integration-tests] Allow specifying additional properties while connecting to kafka

Use Case - For running integration tests with a secure kafka cluster we
need to set additional properties.

This PR adds a way to pass in additional properties to kafka
consumer/producer used in kafka integration tests.
Additionally it also adds a flag to skip creating/deleting kafka topic
from tests since not all secure kafka clusters allow all users to
manage topics.

* fix check style
2017-12-21 13:51:37 -06:00
Himanshu 0f5c7d1aec Add freeSpacePercent config in segment location to enforce free space while storing segments (#5137)
* Add freeSpacePercent config in segment location config to enforce free space while storing segments

* address review comments

* address review comments: more doc on freeSpacePercent and use Double for freeSpacePercent
2017-12-21 15:31:09 +03:00
Nishant Bangarwa e538aa227b use java pointed by JAVA_HOME if JAVA_HOME is explicitly set (#5186) 2017-12-20 21:38:09 -06:00
Himanshu f57496ed8b FilteredHttpServerInventoryViewProvider to start with always false predicate for each segment discovered (#5123)
* FilteredHttpServerInventoryViewProvider to start with always false predicate for each segment discovered

* update HttpServerInventoryViewTest to ensure that predicates are honored

* add docs for HttpServerInventoryView.defaultFilter

* change to javadoc style comment
2017-12-20 18:56:00 -08:00
Nishant Bangarwa 494e0b79ed Allow configuring header size for druid requests (#5174)
* Allow configuring header size for druid requests

* fix configuration name in doc.

* add more info to docs.

* Add info to kerberos doc.
2017-12-20 18:51:40 -08:00
Roman Leventov dc87e4fda1 Renamed IndexedFloats/Doubles/Longs to ColumnarFloats/..., IndexedMultivalue to ColumnarMultiInts, separate IndexedInts from ColumnarInts, many other renames for consistency in io.druid.segment.data package (#5171) 2017-12-20 18:50:07 -08:00
Jihoon Son 9199d61389 Automatic pendingSegments cleanup (#5149)
* PendingSegments cleanup

* fix build

* address comments

* address comments

* fix potential npe

* address comments

* fix build

* fix test

* fix test
2017-12-20 14:46:34 -08:00
Parag Jain c56a9807d4
prevent npe on mismatch between number of kafka partitions and task count (#5139) 2017-12-20 16:23:15 -06:00
Clint Wylie 1181411901 small optimization in timeseries if 'skipEmptyBuckets' is true and cursor completed (#5178) 2017-12-19 16:47:00 -06:00
Roman Leventov f18eba50ee Remove Aggregator.reset() (#5177) 2017-12-19 14:09:17 -08:00
Kirill Kozlov d337f49034 Remove pached TestNG classes cause current maven-failsafe-plugin version supports testrunfactory option (#5165) 2017-12-18 14:13:45 -06:00
Atul Mohan 0eecf2a805 Bump version of druid-basic-security (#5166) 2017-12-15 11:12:00 -08:00
Roman Leventov 5787d04fad Bump Druid version to 0.12.0 (#5138) 2017-12-15 07:37:01 -08:00
Jonathan Wei f48c9d7be1
Basic auth extension (#5099)
* Basic auth extension

* Add auth configuration integration test

* Fix missing authorizerName property

* PR comments

* Fix missing @JsonProperty annotation

* PR comments

* more PR comments
2017-12-14 10:36:04 -08:00
Jonathan Wei 3b4395a16e
Use deprecated-2017Q4 group for Travis (#5155)
* Use deprecated-2017Q4 group for Travis

* Add comment
2017-12-12 17:14:44 -08:00
Parag Jain 677e24b760 prevent NPE from supressing actual exception (#5146) 2017-12-12 11:42:30 -08:00
Roman Leventov 64848c7ebf DataSegment memory optimizations (#5094)
* Deduplicate DataSegments contents (loadSpec's keys, dimensions and metrics lists as a whole) more aggressively; use ArrayMap instead of default LinkedHashMap for DataSegment.loadSpec, because they have only 3 entries on average; prune DataSegment.loadSpec on brokers

* Fix DataSegmentTest

* Refinements

* Try to fix

* Fix the second DataSegmentTest

* Nullability

* Fix tests

* Fix tests, unify to use TestHelper.getJsonMapper()

* Revert TestUtil as ServerTestHelper, fix tests

* Add newline

* Fix indexing tests

* Fix s3 tests

* Try to fix tests, remove lazy caching of ObjectMapper in TestHelper, rename TestHelper.getJsonMapper() to makeJsonMapper()

* Fix HDFS tests

* Fix HdfsDataSegmentPusherTest

* Capitalize constant names
2017-12-12 11:41:40 -08:00
Gian Merlino 294bd3ffb3
DruidCoordinatorBalancer: Fix log message. (#5152) 2017-12-11 20:27:37 -08:00
Gian Merlino 4f5e2b4549 Fix some unemitted alerts. (#5141) 2017-12-06 18:37:01 -08:00
Charles Allen 4365390310 Remove duplicate fastutil dependency in processing pom.xml (#5142) 2017-12-07 08:54:48 +09:00
Jihoon Son 80f5e89a11
Fix DoublesSketchComplexMetricSerde.getSerializer() (#5140) 2017-12-06 15:10:19 +09:00
Alexander Saydakov 45f91a241e numeric quantiles sketch aggregator (#5002)
* numeric quantiles sketch aggregator

* it seems that we need to synchronize all methods, which modify the state

* Seems like a false positive with -Pstrict

* code style fix

* code style fix

* use sketches-core-0.10.3

* moved cache ids to the central place

* better class names

* support large columns

* explained autodetection, added exception

* added comments regarding sketches moving on heap

* support reindexing

* implemented suggestions from jihoonson

* style fix

* use max(k, other.k) for better accuracy

* check for NilColumnValueSelector instead of null

* throw exceptions instead of providing no-op comparators
2017-12-06 08:18:08 +09:00
Roman Leventov a7a6a0487e Replace IOPeon with SegmentWriteOutMedium; Improve buffer compression (#4762)
* Replace IOPeon with OutputMedium; Improve compression

* Fix test

* Cleanup CompressionStrategy

* Javadocs

* Add OutputBytesTest

* Address comments

* Random access in OutputBytes and GenericIndexedWriter

* Fix bugs

* Fixes

* Test OutputBytes.readFully()

* Address comments

* Rename OutputMedium to SegmentWriteOutMedium and OutputBytes to WriteOutBytes

* Add comments to ByteBufferInputStream

* Remove unused declarations
2017-12-04 18:04:27 -08:00
Parag Jain 7c01f77b04 Parse Batch support (#5081)
* add parseBatch and deprecate parse method in InputRowParser

add addAll method, skip max rows in memory check for it

remove parse method from implemetations

transform transformers

add string multiplier input row parser

fix withParseSpec

fix kafka batch indexing

fix isPersistRequired

comments

* add unit test

* make persist async

* review comments
2017-12-04 16:06:16 -06:00
mh2753 c3bb03dcec Make enabled final in JavaScriptConfig (#5130)
* Make enabled final in JavaScriptConfig as it is used in equals and hashCode

* Removing the cryptic boolean condition

Replacing the cryptic boolean condition with a simple assignment.
Changing the param value to boolean from Boolean.
2017-12-04 10:59:30 -08:00