Commit Graph

8542 Commits

Author SHA1 Message Date
Roman Leventov ad6cdf5d09
Reuse IndexedInts returned from DimensionSelector.getRow() implementations (#5172)
* Reuse IndexedInts in DimensionSelector implementations

* Remove BaseObjectColumnValueSelector.getObject() doc

* typo
2018-01-17 16:01:26 +01:00
Jihoon Son 241efafbb2
Automatic compaction by coordinators (#5102)
* Automatic compaction by coordinator

* add links

* skip compaction for very recent segments if they are small

* fix finding search interval

* fix finding search interval

* fix TimelineHolder iteration

* add test for newestSegmentFirstPolicy

* add CompactionSegmentIterator

* add numTargetCompactionSegments

* add missing config

* fix skipping huge shards

* fix handling large number of segments per shard

* fix test failure

* change recursive call to loop

* fix logging

* fix build

* fix test failure

* address comments

* change dataSources type

* check running pendingTasks at each run

* fix test

* address comments

* fix build

* fix test

* address comments

* address comments

* add doc for segment size optimization

* address comment
2018-01-13 13:52:37 +09:00
Gian Merlino a11049c82f Fix APPROX_QUANTILE on outer groupBys. (#5253) 2018-01-12 12:01:32 -08:00
Clint Wylie 491f8cca81 fix timewarp query results when using timezones and crossing DST transitions (#5157)
* timewarp and timezones
changes:
* `TimewarpOperator` will now compensate for daylight savings time shifts between date translation ranges for queries using a `PeriodGranularity` with a timezone defined
* introduces a new abstract query type `TimeBucketedQuery` for all queries which have a `Granularity` (100% not attached to this name). `GroupByQuery`, `SearchQuery`, `SelectQuery`, `TimeseriesQuery`, and `TopNQuery` all extend `TimeBucke
tedQuery`, cutting down on some duplicate code and providing a mechanism for `TimewarpOperator` (and anything else) that needs to be aware of granularity

* move precondition check to TimeBucketedQuery, add Granularities.nullToAll, add getTimezone to TimeBucketQuery

* formatting

* more formatting

* unused import

* changes:
* add 'getGranularity' and 'getTimezone' to 'Query' interface
* merge 'TimeBucketedQuery' into 'BaseQuery'
* fixup tests from resulting serialization changes

* dedupe

* fix after merge

* suppress warning
2018-01-11 12:39:33 -08:00
Shen Liu 3c69717202 Fix typo in configuration/index.md (#5249) (#5250)
* Fix #5212 - typo in auth.md.

* Fix typo in configuration (#5249)

* Add a backquote.

* Fix typo from HttpEmitterMonitor to HttpEmittingMonitor.
2018-01-11 18:29:12 +09:00
Roman Leventov 8877ce38d6
Enforce modifier order with Checkstyle (#5246) 2018-01-11 09:50:42 +01:00
Atul Mohan 3cc4a0ab19 Support for encryption of MySQL connections (#5122)
* Encrypting MySQL connections

* Update docs

* Make verifyServerCertificate a configurable parameter

* Change password parameter and doc update

* Make server certificate verification disabled by default

* Update tostring

* Update docs

* Add check for trust store passwords

* Add warning for null password
2018-01-10 11:33:54 -08:00
Jihoon Son 5d0619f5ce Support retrying for PrefetchableTextFilesFirehoseFactory when prefetch is disabled (#5162)
* Add RetryingInputStream

* unnecessary exception

* fix PrefetchableTextFilesFirehoseFactoryTest

* Fix retrying on connection reset

* fix start offset

* fix checkstyle

* fix check connection reset

* address comments

* fix compile

* address comments

* address comments
2018-01-10 17:37:19 +01:00
Gian Merlino 7b8b0a96d6 IndexTask: Add summary stats line at the end. (#5241) 2018-01-10 14:06:54 +09:00
Jihoon Son 972b4d189a Fix topN doc (#5240) 2018-01-09 20:10:13 -08:00
Parag Jain 83c6c48bed Fix state check bug in Kafka Index Task (#5204)
* fix state check for replacement task

* fix comments

* rebase with master
2018-01-08 18:01:36 -08:00
Jonathan Wei cdd374a417 Throw away rows with timestamps beyond long bounds in kafka indexing (#5215)
* Throw away rows with timestamps beyond long bounds in kafka indexing

* PR comments
2018-01-08 17:40:50 -06:00
Jonathan Wei 9186547689 Exclude sketches-core from druid-sql (#5223) 2018-01-05 17:12:20 -06:00
Jonathan Wei 02544f9197 Add missing auth doc links (#5224) 2018-01-05 16:23:13 -06:00
Roman Leventov 535ec437e9 Apply 'power of 2' optimization to BlockLayoutIndexedDoubleSupplier (#5176)
* Apply 'power of 2' optimization to BlockLayoutIndexedDoubleSupplier; slight optimization of buffer.get() in block layout indexed suppliers

* Fix byte order
2018-01-05 16:08:07 +09:00
Himanshu a46d34daa2 HTTP based task/worker management. (#5104)
* just renaming of SegmentChangeRequestHistory etc

* additional change history refactoring changes

* WorkerTaskManager a replica of WorkerTaskMonitor

* HttpServerInventoryView refactoring to extract sync code and robustification

* Introducing HttpRemoteTaskRunner

* Additional Worker side updates
2018-01-04 19:19:35 -08:00
Jonathan Wei 935ac646f4
Upgrade to Calcite 1.15.0 (#5210)
* Upgrade to Calcite 1.15.0

* Use Filtration.eternity()
2018-01-04 12:11:24 -08:00
Shen Liu 5a8ea5f8ab Fix #5212 - typo in auth.md. (#5213) 2018-01-04 12:09:42 -08:00
Nishant Bangarwa 4cc31e4e7a Update Zookeeper version (#5184) 2018-01-04 10:59:20 +08:00
Roman Leventov 579f9fbedf Add IndexedInts.debugToString() and AbstractIndex.toString(); Add Sequence.toList() and limit() (#5175)
* Add IndexedInts.debugToString() and AbstractIndex.toString()

* Fix AppenderatorTest
2018-01-04 09:56:47 +09:00
David Lim a7967ade4d Support replaceExisting parameter for segments pushers (#5187)
* support replaceExisting parameter for segments pushers

* code review changes

* code review changes
2018-01-03 16:13:21 -08:00
Nishant Bangarwa 59af4d3b14 Fix broken KafkaEmitterConfig parsing (#5201)
* Fix broken KafkaEmitterConfig parsing

This was a regression introduced in
https://github.com/druid-io/druid/pull/4722

KafkaEmitterConfig property names have dot(.) in the name of properties
and JsonConfigurator behavior was changed to not support that.
Added a test and fixed parsing of properties that have dot(.) in
property names

* Fix test failure
2018-01-03 12:08:40 -08:00
Charles Allen 0f773aff80 Fix lookup logging on node start (#5206)
* Add better logging messages in lookups startup on query nodes

* Make sure list is mutable

* Move list to be with other `final` variables
2018-01-03 13:13:55 -06:00
Shimi Kiviti 0ebfb3fbb0 fixed missing gz extension from wikiticker json sample (#5194) 2017-12-28 01:02:12 +09:00
Yuya Fujiwara 3d3b04e1b8 docs: fix broken link to ingestions and tasks on the Druid Concepts page (#5197)
* fix broken links

* add newline
2017-12-27 07:55:24 -08:00
Slim c3f7da2128 Remove extra logging by making it debug level (#5193)
Change-Id: Iaa255862389bdff7fa42b2c08c1e078448b5ee6c
2017-12-23 00:01:10 +03:00
Jihoon Son b31abd03ad
Fix timeout in RemoteTaskRunnerTest (#5191)
* Fix timeout in RemoteTaskRunnerTest

* add message for npe
2017-12-22 17:40:11 +09:00
Jonathan Wei ba873c614b Fix max connections in DruidAvaticaHandlerTest (#5188)
* Fix max connections in DruidAvaticaHandlerTest

* Fix additional tests

* Added comment
2017-12-21 15:32:14 -06:00
Nishant Bangarwa 6582b16a81 [Integration-tests] Allow specifying additional properties while connecting to kafka (#5185)
* [Integration-tests] Allow specifying additional properties while connecting to kafka

Use Case - For running integration tests with a secure kafka cluster we
need to set additional properties.

This PR adds a way to pass in additional properties to kafka
consumer/producer used in kafka integration tests.
Additionally it also adds a flag to skip creating/deleting kafka topic
from tests since not all secure kafka clusters allow all users to
manage topics.

* fix check style
2017-12-21 13:51:37 -06:00
Himanshu 0f5c7d1aec Add freeSpacePercent config in segment location to enforce free space while storing segments (#5137)
* Add freeSpacePercent config in segment location config to enforce free space while storing segments

* address review comments

* address review comments: more doc on freeSpacePercent and use Double for freeSpacePercent
2017-12-21 15:31:09 +03:00
Nishant Bangarwa e538aa227b use java pointed by JAVA_HOME if JAVA_HOME is explicitly set (#5186) 2017-12-20 21:38:09 -06:00
Himanshu f57496ed8b FilteredHttpServerInventoryViewProvider to start with always false predicate for each segment discovered (#5123)
* FilteredHttpServerInventoryViewProvider to start with always false predicate for each segment discovered

* update HttpServerInventoryViewTest to ensure that predicates are honored

* add docs for HttpServerInventoryView.defaultFilter

* change to javadoc style comment
2017-12-20 18:56:00 -08:00
Nishant Bangarwa 494e0b79ed Allow configuring header size for druid requests (#5174)
* Allow configuring header size for druid requests

* fix configuration name in doc.

* add more info to docs.

* Add info to kerberos doc.
2017-12-20 18:51:40 -08:00
Roman Leventov dc87e4fda1 Renamed IndexedFloats/Doubles/Longs to ColumnarFloats/..., IndexedMultivalue to ColumnarMultiInts, separate IndexedInts from ColumnarInts, many other renames for consistency in io.druid.segment.data package (#5171) 2017-12-20 18:50:07 -08:00
Jihoon Son 9199d61389 Automatic pendingSegments cleanup (#5149)
* PendingSegments cleanup

* fix build

* address comments

* address comments

* fix potential npe

* address comments

* fix build

* fix test

* fix test
2017-12-20 14:46:34 -08:00
Parag Jain c56a9807d4
prevent npe on mismatch between number of kafka partitions and task count (#5139) 2017-12-20 16:23:15 -06:00
Clint Wylie 1181411901 small optimization in timeseries if 'skipEmptyBuckets' is true and cursor completed (#5178) 2017-12-19 16:47:00 -06:00
Roman Leventov f18eba50ee Remove Aggregator.reset() (#5177) 2017-12-19 14:09:17 -08:00
Kirill Kozlov d337f49034 Remove pached TestNG classes cause current maven-failsafe-plugin version supports testrunfactory option (#5165) 2017-12-18 14:13:45 -06:00
Atul Mohan 0eecf2a805 Bump version of druid-basic-security (#5166) 2017-12-15 11:12:00 -08:00
Roman Leventov 5787d04fad Bump Druid version to 0.12.0 (#5138) 2017-12-15 07:37:01 -08:00
Jonathan Wei f48c9d7be1
Basic auth extension (#5099)
* Basic auth extension

* Add auth configuration integration test

* Fix missing authorizerName property

* PR comments

* Fix missing @JsonProperty annotation

* PR comments

* more PR comments
2017-12-14 10:36:04 -08:00
Jonathan Wei 3b4395a16e
Use deprecated-2017Q4 group for Travis (#5155)
* Use deprecated-2017Q4 group for Travis

* Add comment
2017-12-12 17:14:44 -08:00
Parag Jain 677e24b760 prevent NPE from supressing actual exception (#5146) 2017-12-12 11:42:30 -08:00
Roman Leventov 64848c7ebf DataSegment memory optimizations (#5094)
* Deduplicate DataSegments contents (loadSpec's keys, dimensions and metrics lists as a whole) more aggressively; use ArrayMap instead of default LinkedHashMap for DataSegment.loadSpec, because they have only 3 entries on average; prune DataSegment.loadSpec on brokers

* Fix DataSegmentTest

* Refinements

* Try to fix

* Fix the second DataSegmentTest

* Nullability

* Fix tests

* Fix tests, unify to use TestHelper.getJsonMapper()

* Revert TestUtil as ServerTestHelper, fix tests

* Add newline

* Fix indexing tests

* Fix s3 tests

* Try to fix tests, remove lazy caching of ObjectMapper in TestHelper, rename TestHelper.getJsonMapper() to makeJsonMapper()

* Fix HDFS tests

* Fix HdfsDataSegmentPusherTest

* Capitalize constant names
2017-12-12 11:41:40 -08:00
Gian Merlino 294bd3ffb3
DruidCoordinatorBalancer: Fix log message. (#5152) 2017-12-11 20:27:37 -08:00
Gian Merlino 4f5e2b4549 Fix some unemitted alerts. (#5141) 2017-12-06 18:37:01 -08:00
Charles Allen 4365390310 Remove duplicate fastutil dependency in processing pom.xml (#5142) 2017-12-07 08:54:48 +09:00
Jihoon Son 80f5e89a11
Fix DoublesSketchComplexMetricSerde.getSerializer() (#5140) 2017-12-06 15:10:19 +09:00
Alexander Saydakov 45f91a241e numeric quantiles sketch aggregator (#5002)
* numeric quantiles sketch aggregator

* it seems that we need to synchronize all methods, which modify the state

* Seems like a false positive with -Pstrict

* code style fix

* code style fix

* use sketches-core-0.10.3

* moved cache ids to the central place

* better class names

* support large columns

* explained autodetection, added exception

* added comments regarding sketches moving on heap

* support reindexing

* implemented suggestions from jihoonson

* style fix

* use max(k, other.k) for better accuracy

* check for NilColumnValueSelector instead of null

* throw exceptions instead of providing no-op comparators
2017-12-06 08:18:08 +09:00