Commit Graph

9013 Commits

Author SHA1 Message Date
QiuMM 601183b4c7 Add period drop before rule (#6415)
* Add period drop before rule

* add license header

* support period drop before rule in coordinator console

* address comments
2018-10-24 12:44:30 -07:00
Alexander Saydakov ec9d1827a0 updated to use the latest sketches-core-0.12.0 (#6381) 2018-10-23 11:20:19 -07:00
Roman Leventov 84ac18dc1b
Catch some incorrect method parameter or call argument formatting patterns with checkstyle (#6461)
* Catch some incorrect method parameter or call argument formatting patterns with checkstyle

* Fix DiscoveryModule

* Inline parameters_and_arguments.txt

* Fix a bug in PolyBind

* Fix formatting
2018-10-23 07:17:38 -03:00
Faxian Zhao c5bf4e7503 update insert pending segments logic to synchronous (#6336)
* 1. Mysql default transaction isolation is REPEATABLE_READ, treat it as READ_COMMITTED will reduce insert id conflict.
2. Add an index to 'dataSource used end' is work well for the most of scenarios(get recently segments), and it will speed up sync add pending segments in DB.
3. 'select and insert' is not need within transaction.

* Use TaskLockbox.doInCriticalSection instead of synchronized syntax to speed up insert pending segments.

* fix typo for NullPointerException
2018-10-22 19:48:20 -07:00
Samarth Jain 359576a80b Implement force push down for nested group by query (#5471)
* Force nested query push down

* Code review changes
2018-10-22 13:43:47 -07:00
elloooooo 1e82b6291e Remove consumer.listTopics() method in case when too many topics in kafka causes the FullGC in Overlord (#6455)
* remove consumer.listTopics() method

* add consumerLock and exception handling for consumer.partitionFor() and remove some useless checks

* add check in case consumer.partitionsFor() returns null

* fix CI failure

* fix failed UT

* Revert "fix CI failure"

This reverts commit f839d09e1e.

* revert unless commit and re-commit the useful part to fix failed UT
2018-10-22 10:46:31 -07:00
Clint Wylie e83cc22996 use a sha512 hash of bloom filter for cache key instead of filter bytes (#6488)
* use a sha512 hash of bloom filter for cache key instead of filter bytes

* make serde private, BloomDimFilter.toString and BloomDimFilter.equals use hash instead of bloomKFilter which has no tostring or equals of its own

* keep and use HashCode object instead of converting to bytes up front

* uneeded imports oops

* tweaks from review

* refactor dupe code

* refactor
2018-10-22 07:57:21 -07:00
David Lim 822e564f54 include mysql-metadata-storage extension in distribution, but without… (#6497)
* include mysql-metadata-storage extension in distribution, but without the GPL-licensed connector library

* Install mysql connector package

* use symlinks to avoid versioning issues

* add documentation for fetching the mysql connector
2018-10-20 18:18:58 -07:00
Jonathan Wei 9851921f43 Fix integration test service logging (#6479) 2018-10-20 14:55:14 -07:00
Joshua Sun bf90b2b183 fix shallow clone git-commit-id plugin unable to find commits until tag issue (#6495) 2018-10-20 14:45:09 -07:00
David Lim e1a53fd17a fix distribution to not include contrib extensions by default, don't pull the entire AWS SDK bundle (#6494) 2018-10-19 13:50:05 -07:00
QiuMM f5f4171a45 QueryCountStatsMonitor: emit query/count (#6473)
Let `QueryCountStatsMonitor` emit `query/count`, then I can monitor QPS of my services, or I have to count it by myself.
2018-10-19 10:15:02 -03:00
Joshua Sun b662fe84c5 fix TaskRunnerUtils String formatting issue (#6492)
* fix TaskRunnerUtils String formatting issue

* additional fixes
2018-10-18 19:16:46 -06:00
David Lim 73780536a6 Apache-ize POM (#6482)
* Apache-ize POM

* put revision information into MANIFEST.MF for binary release

* remove nightly profile

* fix flaky travis by overriding maven-remote-resources-plugin execution from the parent POM
2018-10-17 18:37:01 -07:00
patelh f0b977ea7f Upgrade lz4-java to 1.5.0 (#6478) 2018-10-17 18:34:20 -07:00
Clint Wylie c034ed8c43 fix a couple of missed changes that should been in #6443 (#6487) 2018-10-17 16:22:17 -07:00
David Lim f8710c8e20 add DISCLAIMER file and modify NOTICE file for ASF (#6475)
* add DISCLAIMER file and modify NOTICE file for ASF

* add line breaks
2018-10-17 08:36:13 -07:00
Jonathan Wei b9d01f5770 More portable expired cert gen for IT (#6481) 2018-10-17 08:24:15 -07:00
patelh c780aacc03 Add ability to specify dbcp properties file (#6419)
* Add ability to specify dbcp properties file

* Address PR comments, use mock config, remove setter

* Add documentation

* APRC, updated docs with example file contents

* APRC, add @Nullable, @VisibileForTesting, update doc

* APRC, remove error log, use props directly as jackson binding

* Remove unused files
2018-10-16 12:27:19 -07:00
Roman Leventov 789c9a1dc7 Prohibit using Object|Long|Float|DoubleColumnSelector in instanceof statements (#6470)
* Prohibit using Object|Long|Float|DoubleColumnSelector in instanceof statements

* Doc fixes
2018-10-15 15:41:43 -07:00
QiuMM 85a89e2703 make druid node bind address configurable (#6464)
* make druid node bind address configurable

* fix tests

* fix travis-ci
2018-10-15 14:19:40 -07:00
Gian Merlino f537c0069a SQL: Support for selecting multi-value dimensions. (#6462)
* SQL: Support for selecting multi-value dimensions.

Fixes #4637. Doesn't completely address everything mentioned in #4638,
but at least fixes one issue on the way there.

* Fix null cases in tests.
2018-10-15 14:01:21 -07:00
Roman Leventov b57d5df074
Emit emitter/buffers/allocated/delta and emitter/buffers/failed/delta (#6425)
@gianm gauge `emitter/buffers/failed` and `emitter/buffers/totalAllocated` metrics proved to be very inconvenient in practice: they are not additive across a fleet of nodes, are randomly reset when a JVM restarts (that may be completely independent from metrics emitting issues). Regarding [problem identification](https://github.com/apache/incubator-druid/pull/4973#issuecomment-338119125), `emitter/buffers/dropped` serves this well.

Currently in this PR I added `/delta` vs. `/total` metrics, but if there are no objections I would remove `/total` and so align `emitter/buffers/allocated` and `/failed` with `/dropped` and `/events/emitted` which are already delta-only.
2018-10-15 17:30:29 -03:00
QiuMM 6c71ee5ed5 fix type mismatch caused by #6377 (#6466) 2018-10-15 17:34:18 +09:00
Roman Leventov aa121da25f Use NodeType enum instead of Strings (#6377)
* Use NodeType enum instead of Strings

* Make NodeType constants uppercase

* Fix CommonCacheNotifier and NodeType/ServerType comments

* Reconsidering comment

* Fix import

* Add a comment to CommonCacheNotifier.NODE_TYPES
2018-10-14 20:49:38 -07:00
robertervin 95ab1ea737 Fix Empty InDimFilter Failure (#6330)
* fix empty InDimFilter failure (#6101)

* Add test case for empty values input

* Add documentation for empty values in InDimFilter
2018-10-14 20:43:16 -07:00
Clint Wylie 84598fba3b combine druid-api, druid-common, java-util into druid-core (#6443)
* combine druid-api, druid-common, java-util

* spacing
2018-10-14 20:37:37 -07:00
Diego Costa d5ccd582b5 Reducing the usage of Jmh Invocation level on setup (#6459)
* Reducing the usage of invocation level setup

* Removing LoadStatusBenchmark as it has no use in the codebase
2018-10-14 20:37:11 -07:00
Roman Leventov e3397ba00f Enforce Druid's exception class use (#6456) 2018-10-13 16:35:14 -07:00
Surekha e908fd6db7 Add check for nullable numRows (#6460)
* Add check for nullable numRows

* Make numRows long instead of Long type

* Add check for numRows in unit test
* small refactoring

* Modify test

PR comment from https://github.com/apache/incubator-druid/pull/6094#pullrequestreview-163937783

* Add a test for serverSegments table

* update tests
2018-10-13 15:08:42 -07:00
dongyifeng b06ac54a5e add PrefixFilteredDimensionSpec for multi-value dimensions (#6307)
* add PrefixFilteredDimensionSpec for multi-value dimensions

* add docs for PrefixFilteredDimensionSpec

* remove unnecessary null handling

* add null check to the result of NullHandling
2018-10-12 17:51:09 -07:00
Charles Allen 0f4f5f2877 Cleanup jackson dependency exclusions (#6438)
* Remove pulling in jackson from fasterxml in a lot of places

* Remove codehaus extensions
2018-10-12 17:25:39 -06:00
vishnu rao 6567fff9e7 Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header (#4033)
* o- Query Response format to be based on http 'accept' header & Query Payload contenty type to be based on 'content-type' header

* o- Query Response format to be based on http 'accept' header & Query Payload contenty type to be based on 'content-type' header
o- if Accept header is absent, it defaults to Content-Type header

* Feature: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type'  PR #4033
Minor change to a comment - restoring to previous wording

* Feature: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type'  PR #4033
o- minor change to check for empty string
2018-10-12 14:29:14 -07:00
Surekha 3be4a97150 Fix inconsistent segment size(#6448) (#6451)
* Fix inconsistent segment size(#6448)

* Fix the segment size for published segments
* Changes to get numReplicas
* Make coordinator segments API truly streaming

* Changes to store partial segment data

* Simplify SegmentMetadataHolder
* Store partial the columns from available segments

* Address comments
2018-10-12 12:55:20 -07:00
Clint Wylie 39d61b9ae5 update druid-console to 0.0.4 (#6450) 2018-10-11 22:37:08 -06:00
David Lim 20ab213ba6 change project versions to 0.13.0-incubating-SNAPSHOT (#6453) 2018-10-11 19:28:01 -07:00
David Lim 1e913a0416 update two license headers to ASF (#6449) 2018-10-11 16:04:23 -07:00
Charles Allen c55b37d7ec Add optional `name` to top level of FilteredAggregatorFactory (#6219)
* Add optional `name` to top level of FilteredAggregatorFactory

* Add compat constructor for tests

* Address comments

* Add equals and hash code updates

* Rename test

* Fix imports and code style
2018-10-11 11:56:53 -07:00
Atul Mohan ab7b4798cc Securing passwords used for SSL connections to Kafka (#6285)
* Secure credentials in consumer properties

* Merge master

* Refactor property population into separate method

* Fix property setter

* Fix tests
2018-10-11 10:03:01 -07:00
QiuMM f8f4526b16 Add suspend|resume|terminate all supervisors endpoints. (#6272)
* ability to showdown all supervisors

* add doc

* address comments

* fix code style

* address comments

* change ternary assignment to if statement

* better docs
2018-10-10 21:41:59 -07:00
Clint Wylie f7775d1db3 fixes for LookupReferencesManagerTest (#6444)
* some fixes for LookupReferencesManagerTest

* docs

* formatting

* more formatting fixes
2018-10-10 18:02:11 -07:00
Surekha 3a0a667fe0 Introduce SystemSchema tables (#5989) (#6094)
* Added SystemSchema with following tables (#5989)

* SEGMENTS table provides details on served and published segments
* SERVERS table provides details on data servers
* SERVERSEGMETS table is the JOIN of SEGMENTS and SERVERS
* TASKS table provides details on tasks

* Add documentation for system schema

* Fix static-analysis warnings

* Address PR comments

*Add unit tests

* Fix a test

* Try to fix a test

* Fix a bug around replica count

* rename io.druid to org.apache.druid

* Major change is to make tasks and segment queries streaming

* Made tasks/segments stream to calcite instead of storing it in memory
* Add num_rows to segments table
* Refactor JsonParserIterator
* Replace with closeable iterator

* Fix docs, make num_rows column nullable, some unit test changes

* make num_rows column type long, allow it to be null

fix a compile error after merge, add TrafficCop param to InputStreamResponseHandler

* Filter null rows for segments table from Linq4j enumerable

* change num_replicas datatype to long in segments table

* Fix some tests and address comments

* Doc updates, other PR comments

* Update tests

* Address comments

* Add auth check
* Update docs
* Refactoring

* Fix teamcity warning, change the getQueryableServer in TimelineServerView

* Fix compilation after rebase

* Use the stream API from AuthorizationUtils

* Added LeaderClient interface and NoopDruidLeaderClient class

* Revert "Added LeaderClient interface and NoopDruidLeaderClient class"

This reverts commit 100fa46e39.

* Make the naming consistent to server_segments for the join table

* Add ForbiddenException on auth check failure
* Remove static block from SystemSchema

* Try to fix a test in CalciteQueryTest due to rename of server_segments

* Fix the json output format in the coordinator API

* Add auth check in the segments API
* Add null check to avoid NPE

* Use annonymous class object instead of mock for DruidLeaderClient in SqlBenchmark

* Fix test failures, type long/BIGINT can be nullable

* Revert long nullability to fix tests

* Fix style for tests

* PR comments

* Address PR comments

* Add the missing BytesAccumulatingResponseHandler class

* Use Sequences.withBaggage in DruidPlanner

* Fix docs, add comments

* Close the iterator if hasNext returns false
2018-10-10 17:17:29 -07:00
Atul Mohan e69a2f217b Fix avro parser issue while deserializing sketches (#6440)
* Fix object transform for avro parser

* Remove unwanted space
2018-10-10 08:53:26 -07:00
Clint Wylie 1fa045862a exclude all redundant and unecessary hive-exec depends in orc-extensions pom.xml (#6441) 2018-10-10 08:24:28 -07:00
Roman Leventov 09126c021a
Remove Aggregator.clone() methods (#6437)
* Remove Aggregator.clone() methods

* Remove CardinalityAggregator.name
2018-10-10 10:07:56 -03:00
QiuMM d559dfecb2 replace deprecated druid.port by druid.plaintextPort in docs (#6427) 2018-10-09 10:57:01 -07:00
Jihoon Son 9343cbc63a Fix CompactionTask to consider only latest segments (#6429)
* CompactionTask should consider only latest segments

* fix test
2018-10-08 21:53:16 -07:00
Roman Leventov af9efdbedf
HttpPostEmitterMonitor: don't emit maxTime and minTime if no times were recorded (#6418)
* HttpPostEmitterMonitor: don't emit maxTime and minTime if no times were recorded

* Don't emit sum and count if none

* Remove outdated comments
2018-10-08 17:11:42 -03:00
Jihoon Son 2b76d57347 Fail compactionTask if it fails to run one of indexTaskSpecs (#6428)
* Fail compactionTask if it fails to run one of indexTaskSpecs

* add log
2018-10-08 08:53:32 -07:00
Jihoon Son 88d23b77b7 Add support keepSegmentGranularity for automatic compaction (#6407)
* Add support keepSegmentGranularity for automatic compaction

* skip unknown dataSource

* ignore single semgnet to compact

* add doc

* address comments

* address comment
2018-10-07 16:48:58 -07:00