Commit Graph

8346 Commits

Author SHA1 Message Date
Gian Merlino 4f5e2b4549 Fix some unemitted alerts. (#5141) 2017-12-06 18:37:01 -08:00
Charles Allen 4365390310 Remove duplicate fastutil dependency in processing pom.xml (#5142) 2017-12-07 08:54:48 +09:00
Jihoon Son 80f5e89a11
Fix DoublesSketchComplexMetricSerde.getSerializer() (#5140) 2017-12-06 15:10:19 +09:00
Alexander Saydakov 45f91a241e numeric quantiles sketch aggregator (#5002)
* numeric quantiles sketch aggregator

* it seems that we need to synchronize all methods, which modify the state

* Seems like a false positive with -Pstrict

* code style fix

* code style fix

* use sketches-core-0.10.3

* moved cache ids to the central place

* better class names

* support large columns

* explained autodetection, added exception

* added comments regarding sketches moving on heap

* support reindexing

* implemented suggestions from jihoonson

* style fix

* use max(k, other.k) for better accuracy

* check for NilColumnValueSelector instead of null

* throw exceptions instead of providing no-op comparators
2017-12-06 08:18:08 +09:00
Roman Leventov a7a6a0487e Replace IOPeon with SegmentWriteOutMedium; Improve buffer compression (#4762)
* Replace IOPeon with OutputMedium; Improve compression

* Fix test

* Cleanup CompressionStrategy

* Javadocs

* Add OutputBytesTest

* Address comments

* Random access in OutputBytes and GenericIndexedWriter

* Fix bugs

* Fixes

* Test OutputBytes.readFully()

* Address comments

* Rename OutputMedium to SegmentWriteOutMedium and OutputBytes to WriteOutBytes

* Add comments to ByteBufferInputStream

* Remove unused declarations
2017-12-04 18:04:27 -08:00
Parag Jain 7c01f77b04 Parse Batch support (#5081)
* add parseBatch and deprecate parse method in InputRowParser

add addAll method, skip max rows in memory check for it

remove parse method from implemetations

transform transformers

add string multiplier input row parser

fix withParseSpec

fix kafka batch indexing

fix isPersistRequired

comments

* add unit test

* make persist async

* review comments
2017-12-04 16:06:16 -06:00
mh2753 c3bb03dcec Make enabled final in JavaScriptConfig (#5130)
* Make enabled final in JavaScriptConfig as it is used in equals and hashCode

* Removing the cryptic boolean condition

Replacing the cryptic boolean condition with a simple assignment.
Changing the param value to boolean from Boolean.
2017-12-04 10:59:30 -08:00
Roman Leventov aacc57131b Don't use HistoricalXxxPrototypes in PooledTopNAlgorithm when Cursor's offset is FilteredOffset (#5133)
* Don't use HistoricalXxxPrototypes in PooledTopNAlgorithm when Cursor's offset is FilteredOffset, because it doens't support cloning

* Add test
2017-12-01 17:04:22 -08:00
Gleb Smirnov aafd0373be Fix query/node/time metric calculation (#5131) 2017-12-01 09:39:04 -08:00
Gian Merlino 5f6bdd940b SQL: Improve translation of time floor expressions. (#5107)
* SQL: Improve translation of time floor expressions.

The main change is to TimeFloorOperatorConversion.applyTimestampFloor.

- Prefer timestamp_floor expressions to timeFormat extractionFns, to
  avoid turning things into strings when it isn't necessary.
- Collapse CAST(FLOOR(X TO Y) AS DATE) to FLOOR(X TO Y) if appropriate.

* Fix tests.
2017-11-29 12:06:03 -08:00
Roman Leventov 55d894e104 Fix IndexMergeBenchmark (#5118) 2017-11-26 16:03:58 -08:00
Slim 790678f02c Fix typo in docs (#5074)
Small fix for the realtime pull docs fix issue #5072
2017-11-22 23:16:36 -03:00
Chuanlei Ni 368d03146b assign granularity.all to SelectQuery by default (#5091) 2017-11-21 17:10:19 -08:00
Daniel 22c49b0d33 docs: fix broken link to broker configuration (#5105) 2017-11-21 13:32:00 +09:00
Fokko Driesprong 2487152b59 Update Avro to 1.8.2 (#5075)
And add exclusions that are required to have a single version of
Apache Avro on the classpath.
2017-11-20 20:29:17 -08:00
Jihoon Son 322b7f6a27 Fix AnnouncerTest.testSanity() (#5077)
* Fix announcer test

* rename

* split server module test

* add name

* polling until /test1 is created

* add comment
2017-11-20 20:27:41 -08:00
Jihoon Son 645de02fb2 Remove Injector from IngestSegmentFirehoseFactory (#5045)
* Add lock check to segment list actions

* fix test

* remove lock check
2017-11-20 15:54:35 -08:00
Roman Leventov dbb37b727d Add useL2 and populateL2 configs to HybridCache (#5088)
* Add useL2 and populateL2 configs to HybridCache

* typo
2017-11-20 16:57:05 -06:00
Slim e115da39df Add relogin logic to renew the Kerberos TGT once it expire (#5096)
* Kerberos TGT will expire after some pre-determined time, this patch add relogin calls

Change-Id: I17ccb9b42aa3032de5d28c8c21e4ffbe8222b815

* exit if the first login passed

Change-Id: Ifefd5e9e0dd7d07b05cc493ab1f72415de557ec2
2017-11-20 17:33:39 +05:30
chaoqiang 50140ce820 StatsD Emitter Doc on blankHolder (#5101)
* fix equalDistribution worker select strategy

* replace anonymous Comparator

* keep previous version sorting comment

* fix code style

* update comment

* move JsonProperty

* fix statsD emitter with blank character

* Add blankHolder doc On statsD monitor
2017-11-18 12:00:47 -08:00
Parag Jain cb03efeb14 Kafka Index Task that supports Incremental handoffs (#4815)
* Kafka Index Task that supports Incremental handoffs
- Incrementally handoff segments when they hit maxRowsPerSegment limit
- Decouple segment partitioning from Kafka partitioning, all records from consumed partitions go to a single druid segment
- Support for restoring task on middle manager restarts by check pointing end offsets for segments

* take care of review comments

* make getCurrentOffsets call async, keep track of publishing sequence, review comments

* fix setEndoffset duplicate request handling, formatting

* fix unit test

* backward compatibility

* make AppenderatorDriverMetadata backwards compatible

* add unit test

* fix deadlock between persist and push executors in AppenderatorImpl

* fix formatting

* use persist dir instead of work dir

* review comments

* fix deadlock

* actually fix deadlock
2017-11-17 16:05:20 -06:00
zhangxinyu1 590633c595 fix bug in method reset of ByteBufferHashTable.java (#5100) 2017-11-17 13:19:16 -03:00
Jonathan Wei ec6774039e
Fix numLookupLoadingThreads default value (#5097) 2017-11-16 15:13:52 -08:00
Gian Merlino 486159ba8c SQL: Add TIMESTAMPADD. (#5079) 2017-11-16 12:00:34 -08:00
Jihoon Son 93459f748f Fix missing intervals after compacting intervals (#5092)
* Fix missing intervals after compacting intervals

* fix build
2017-11-16 11:42:38 -08:00
chaoqiang b34d471aa2 Fix StatsD Emitter with blank character (#5044)
* fix equalDistribution worker select strategy

* replace anonymous Comparator

* keep previous version sorting comment

* fix code style

* update comment

* move JsonProperty

* fix statsD emitter with blank character
2017-11-16 11:12:24 -08:00
Jonathan Wei af44d1142b Add unsecured /health endpoint, remove auth checks from isLeader (#5087)
* Add unsecured /health endpoint, remove auth checks from isLeader

* PR comments
2017-11-15 14:41:30 -06:00
Gian Merlino 7722401cb3 SQL: Add rule to prune unused aggregations. (#5049) 2017-11-13 20:24:45 -08:00
Gian Merlino 77df5e0673 ExpressionSelectors: Add optimized selectors. (#5048)
* ExpressionSelectors: Add caching selectors.

- SingleLongInputCaching selector for expressions on the __time column,
  using a similar optimization to SingleScanTimeDimSelector
- SingleStringInputDimensionSelector for expressions on string columns
  that return strings, using a similar optimization to ExtractionFn
  based DimensionSelectors.
- SingleStringInputCaching selector for expressions on string columns
  that return primitives.

Also, in the SQL planner, prefer expressions for time operations
rather than extractionFns.

* Code review comments.
2017-11-13 20:24:24 -08:00
Gian Merlino 4fd4444b42 SQL: Add "array" result format, and document result formats. (#5032)
* SQL: Add "array" result format, and document result formats.

* Code style.
2017-11-13 20:24:06 -08:00
Gian Merlino 6c0c858913 SQL: Support CASE-style filtered count distinct. (#5047)
i.e., aggregations like COUNT(DISTINCT CASE WHEN x THEN y END). This
patch also changes complex columns to report as nullable, which
is required for them to type-check properly when used in these kinds
of filtered aggregations.
2017-11-13 20:23:54 -08:00
Jonathan Wei 9ac150c23a
Split internal client escalation from Authenticator interface (#5073)
* Split internal client escalation from Authenticator interface

* PR comments
2017-11-13 19:29:08 -08:00
Akash Dwivedi c1538f29fc maxQueryTimeout property in runtime properties. (#4852)
* maxQueryTimeout property in runtime properties.

* extra line

* move withTimeoutAndMaxScatterGatherBytes method to QueryLifeCycle.

* Fix initialize method.

* remove unused import.

* doc update.

* some more details in doc about query failure..

* minor fix.

* decorating QueryRunner to set and verify context. Added by servers.

* remove whitespace.
2017-11-13 19:23:11 -06:00
Jonathan Wei 819700cbc5 Automatically insert authenticator/authorizer names into config properties (#5071) 2017-11-13 13:12:31 -08:00
Himanshu 1c64f02574 make AsyncHttpCLient instance timer thread daemon (#5076) 2017-11-10 19:54:29 -06:00
Egor Riashin 3ad5f51f7e Integration tests port fix, logs output, lib share changes (#5061)
* integration tests port fix, logs output, lib share changes

* Fix test compilation

* docker image fix

* Annotate DiscoverySelector.pick() as Nullable, use formatting instead of string concatenation for logging

* integration-tests logging check

* corrected integration-tests log location

* removed an experimental travis test exception
2017-11-10 14:07:37 -08:00
Gian Merlino 9444da5038 SQL: Improved behavior when implicitly casting strings to date/time literals. (#5023)
* SQL: Improved behavior when implicitly casting strings to date/time literals.

- Handle all flavors of ISO8601 and SQL literals.
- Throw errors on other literals instead of silently transforming them to 0.

* Respect timeZone when format is null.
2017-11-10 17:43:22 +09:00
Himanshu bbb678efd7 fix lookups endpoint collisions (#5058)
* fix lookups endpoint collissions

* fix errors
2017-11-09 17:39:53 -08:00
Goh Wei Xiang 81f249874b Use daemon thread pool for AsyncHttpClient in emitters (#5057)
* use daemon thread pool for AsyncHttpClient in emitters

* changed to use existing helper methods

* refactored creation of AsyncHttpClient
2017-11-09 17:38:19 -08:00
Himanshu 2ecebb3173 Fix coordinator/overlord redirects when TLS is enabled (#5037)
* Fix coordinator/overlord redirects when TLS is enabled

* address review comment

* fix UTs

* workaround to not ignore URL instance to fix the teamcity build

* update tls doc
2017-11-09 13:10:28 -08:00
Roman Leventov 95154f9ef6 Netty downgrade (#5059) 2017-11-09 11:27:50 -08:00
Jihoon Son c11c71ab3e Using ImmutableDruidDataSource as a key for map and set instead of DruidDataSource (#5054)
* use ImmutableDruidDataSource for map and set

* address comments

* unused import

* allow returning only ImmutableDruidDataSource in MetadataSegmentManager

* address comments

* remove TreeSet

* revert to use TreeSet
2017-11-09 16:07:58 -03:00
Roman Leventov 3541b7544b Prohibit and remove unused declarations in the processing module (#4930)
* Prohibit and remove unused declarations in the processing module

* Fix tests

* Fix integration tests

* Suppress unused

* Try to remove SuppressWarnings unused in VirtualColumn

* Remove reset 'false positives'

* Annotate CliCommandCreator as ExtensionPoint

* Unused import warning instead of error in IntelliJ

* Fixes

* Add comment

* Fix AzureBlob

* Fix CloudFilesBlob

* Address comments

* Add Project SDK section to INTELLIJ_SETUP.md

* Fix image
2017-11-09 09:27:27 -08:00
Jisoo Kim 1bf253f6e6 Fix Router race condition and use default broker service name for invalid priority (#5050)
* use default brokerServiceName when priority is not valid

* use AtomicInteger for NodesHolder.roundRobinIndex

* revert inspectionProfiles change

* adjust TieredBrokerHostSelectorTest

* combine if statements and ensure index does not become negative

* set next index with mod if overflows

* fix codestyle

* use nextIndex

* extract the while loop to a method
2017-11-09 09:31:18 -06:00
Roman Leventov a8dc056c09
Add retries for coordinator fetch and lookup start in LookupReferencesManager (#5029)
* Add retries for coordinator fetch and lookup start in LookupReferencesManager

* Fix LookupConfigTest

* Address comments

* Address more comments

* And address more comments

* Address comms

* Recognize 'not found' lookups in LookupReferencesManager.tryGetLookupListFromCoordinator(), by @egor-ryashin
2017-11-09 02:30:36 -03:00
Gian Merlino e6ec4310b1 IT: Switch to OpenJDK8 base image. (#5060)
* IT: Switch to OpenJDK8 base image.

Also split the Docker image into a base image and a child image, and
build the base image ahead of time for efficiency's sake. Also upgrade
ZK to 3.4.10.

* Additional comments about ZK upgrades.
2017-11-08 19:56:31 -08:00
Jihoon Son 5f3c863d5e Add compaction task (#4985)
* Add compaction task

* added doc

* use combining aggregators

* address comments

* add support for dimensionsSpec

* fix getUniqueDims and getUniqueMetics

* find unique dimensionsSpec

* fix compilation

* add unit test

* fix test

* fix test

* test for different dimension orderings and types, and doc for type and ordering

* add control for custom ordering and type

* update doc

* fix compile

* fix compile

* add segments param

* fix serde error

* fix build
2017-11-03 21:55:27 -06:00
Roman Leventov 5eb08c27cb Add Emitter monitoring (#4973)
* Add Emitter monitoring

* Fix typo

* Fixes

* testing new emitter

* Fix failed test (#71)

* testing new emitter

* fix on failed test

* Remove emitter's readTimeout from docs

* Update docs

* Add HttpEmittingMonitor

* Update java-util to 1.3.2
2017-11-03 21:27:57 -06:00
Jonathan Wei 13c0d88ffc Fix coordinator->overlord proxy auth failure (#5039)
* Fix coordinator->overlord proxy auth failure

* PR comment
2017-11-03 13:54:41 -07:00
Jiaqi Liu 7c8b14f18c Fix doc link (#5040) 2017-11-03 11:04:33 -07:00