Commit Graph

8947 Commits

Author SHA1 Message Date
David Lim f8710c8e20 add DISCLAIMER file and modify NOTICE file for ASF (#6475)
* add DISCLAIMER file and modify NOTICE file for ASF

* add line breaks
2018-10-17 08:36:13 -07:00
Jonathan Wei b9d01f5770 More portable expired cert gen for IT (#6481) 2018-10-17 08:24:15 -07:00
patelh c780aacc03 Add ability to specify dbcp properties file (#6419)
* Add ability to specify dbcp properties file

* Address PR comments, use mock config, remove setter

* Add documentation

* APRC, updated docs with example file contents

* APRC, add @Nullable, @VisibileForTesting, update doc

* APRC, remove error log, use props directly as jackson binding

* Remove unused files
2018-10-16 12:27:19 -07:00
Roman Leventov 789c9a1dc7 Prohibit using Object|Long|Float|DoubleColumnSelector in instanceof statements (#6470)
* Prohibit using Object|Long|Float|DoubleColumnSelector in instanceof statements

* Doc fixes
2018-10-15 15:41:43 -07:00
QiuMM 85a89e2703 make druid node bind address configurable (#6464)
* make druid node bind address configurable

* fix tests

* fix travis-ci
2018-10-15 14:19:40 -07:00
Gian Merlino f537c0069a SQL: Support for selecting multi-value dimensions. (#6462)
* SQL: Support for selecting multi-value dimensions.

Fixes #4637. Doesn't completely address everything mentioned in #4638,
but at least fixes one issue on the way there.

* Fix null cases in tests.
2018-10-15 14:01:21 -07:00
Roman Leventov b57d5df074
Emit emitter/buffers/allocated/delta and emitter/buffers/failed/delta (#6425)
@gianm gauge `emitter/buffers/failed` and `emitter/buffers/totalAllocated` metrics proved to be very inconvenient in practice: they are not additive across a fleet of nodes, are randomly reset when a JVM restarts (that may be completely independent from metrics emitting issues). Regarding [problem identification](https://github.com/apache/incubator-druid/pull/4973#issuecomment-338119125), `emitter/buffers/dropped` serves this well.

Currently in this PR I added `/delta` vs. `/total` metrics, but if there are no objections I would remove `/total` and so align `emitter/buffers/allocated` and `/failed` with `/dropped` and `/events/emitted` which are already delta-only.
2018-10-15 17:30:29 -03:00
QiuMM 6c71ee5ed5 fix type mismatch caused by #6377 (#6466) 2018-10-15 17:34:18 +09:00
Roman Leventov aa121da25f Use NodeType enum instead of Strings (#6377)
* Use NodeType enum instead of Strings

* Make NodeType constants uppercase

* Fix CommonCacheNotifier and NodeType/ServerType comments

* Reconsidering comment

* Fix import

* Add a comment to CommonCacheNotifier.NODE_TYPES
2018-10-14 20:49:38 -07:00
robertervin 95ab1ea737 Fix Empty InDimFilter Failure (#6330)
* fix empty InDimFilter failure (#6101)

* Add test case for empty values input

* Add documentation for empty values in InDimFilter
2018-10-14 20:43:16 -07:00
Clint Wylie 84598fba3b combine druid-api, druid-common, java-util into druid-core (#6443)
* combine druid-api, druid-common, java-util

* spacing
2018-10-14 20:37:37 -07:00
Diego Costa d5ccd582b5 Reducing the usage of Jmh Invocation level on setup (#6459)
* Reducing the usage of invocation level setup

* Removing LoadStatusBenchmark as it has no use in the codebase
2018-10-14 20:37:11 -07:00
Roman Leventov e3397ba00f Enforce Druid's exception class use (#6456) 2018-10-13 16:35:14 -07:00
Surekha e908fd6db7 Add check for nullable numRows (#6460)
* Add check for nullable numRows

* Make numRows long instead of Long type

* Add check for numRows in unit test
* small refactoring

* Modify test

PR comment from https://github.com/apache/incubator-druid/pull/6094#pullrequestreview-163937783

* Add a test for serverSegments table

* update tests
2018-10-13 15:08:42 -07:00
dongyifeng b06ac54a5e add PrefixFilteredDimensionSpec for multi-value dimensions (#6307)
* add PrefixFilteredDimensionSpec for multi-value dimensions

* add docs for PrefixFilteredDimensionSpec

* remove unnecessary null handling

* add null check to the result of NullHandling
2018-10-12 17:51:09 -07:00
Charles Allen 0f4f5f2877 Cleanup jackson dependency exclusions (#6438)
* Remove pulling in jackson from fasterxml in a lot of places

* Remove codehaus extensions
2018-10-12 17:25:39 -06:00
vishnu rao 6567fff9e7 Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type' header (#4033)
* o- Query Response format to be based on http 'accept' header & Query Payload contenty type to be based on 'content-type' header

* o- Query Response format to be based on http 'accept' header & Query Payload contenty type to be based on 'content-type' header
o- if Accept header is absent, it defaults to Content-Type header

* Feature: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type'  PR #4033
Minor change to a comment - restoring to previous wording

* Feature: Query Response format to be based on http 'accept' header & Query Payload content type to be based on 'content-type'  PR #4033
o- minor change to check for empty string
2018-10-12 14:29:14 -07:00
Surekha 3be4a97150 Fix inconsistent segment size(#6448) (#6451)
* Fix inconsistent segment size(#6448)

* Fix the segment size for published segments
* Changes to get numReplicas
* Make coordinator segments API truly streaming

* Changes to store partial segment data

* Simplify SegmentMetadataHolder
* Store partial the columns from available segments

* Address comments
2018-10-12 12:55:20 -07:00
Clint Wylie 39d61b9ae5 update druid-console to 0.0.4 (#6450) 2018-10-11 22:37:08 -06:00
David Lim 20ab213ba6 change project versions to 0.13.0-incubating-SNAPSHOT (#6453) 2018-10-11 19:28:01 -07:00
David Lim 1e913a0416 update two license headers to ASF (#6449) 2018-10-11 16:04:23 -07:00
Charles Allen c55b37d7ec Add optional `name` to top level of FilteredAggregatorFactory (#6219)
* Add optional `name` to top level of FilteredAggregatorFactory

* Add compat constructor for tests

* Address comments

* Add equals and hash code updates

* Rename test

* Fix imports and code style
2018-10-11 11:56:53 -07:00
Atul Mohan ab7b4798cc Securing passwords used for SSL connections to Kafka (#6285)
* Secure credentials in consumer properties

* Merge master

* Refactor property population into separate method

* Fix property setter

* Fix tests
2018-10-11 10:03:01 -07:00
QiuMM f8f4526b16 Add suspend|resume|terminate all supervisors endpoints. (#6272)
* ability to showdown all supervisors

* add doc

* address comments

* fix code style

* address comments

* change ternary assignment to if statement

* better docs
2018-10-10 21:41:59 -07:00
Clint Wylie f7775d1db3 fixes for LookupReferencesManagerTest (#6444)
* some fixes for LookupReferencesManagerTest

* docs

* formatting

* more formatting fixes
2018-10-10 18:02:11 -07:00
Surekha 3a0a667fe0 Introduce SystemSchema tables (#5989) (#6094)
* Added SystemSchema with following tables (#5989)

* SEGMENTS table provides details on served and published segments
* SERVERS table provides details on data servers
* SERVERSEGMETS table is the JOIN of SEGMENTS and SERVERS
* TASKS table provides details on tasks

* Add documentation for system schema

* Fix static-analysis warnings

* Address PR comments

*Add unit tests

* Fix a test

* Try to fix a test

* Fix a bug around replica count

* rename io.druid to org.apache.druid

* Major change is to make tasks and segment queries streaming

* Made tasks/segments stream to calcite instead of storing it in memory
* Add num_rows to segments table
* Refactor JsonParserIterator
* Replace with closeable iterator

* Fix docs, make num_rows column nullable, some unit test changes

* make num_rows column type long, allow it to be null

fix a compile error after merge, add TrafficCop param to InputStreamResponseHandler

* Filter null rows for segments table from Linq4j enumerable

* change num_replicas datatype to long in segments table

* Fix some tests and address comments

* Doc updates, other PR comments

* Update tests

* Address comments

* Add auth check
* Update docs
* Refactoring

* Fix teamcity warning, change the getQueryableServer in TimelineServerView

* Fix compilation after rebase

* Use the stream API from AuthorizationUtils

* Added LeaderClient interface and NoopDruidLeaderClient class

* Revert "Added LeaderClient interface and NoopDruidLeaderClient class"

This reverts commit 100fa46e39.

* Make the naming consistent to server_segments for the join table

* Add ForbiddenException on auth check failure
* Remove static block from SystemSchema

* Try to fix a test in CalciteQueryTest due to rename of server_segments

* Fix the json output format in the coordinator API

* Add auth check in the segments API
* Add null check to avoid NPE

* Use annonymous class object instead of mock for DruidLeaderClient in SqlBenchmark

* Fix test failures, type long/BIGINT can be nullable

* Revert long nullability to fix tests

* Fix style for tests

* PR comments

* Address PR comments

* Add the missing BytesAccumulatingResponseHandler class

* Use Sequences.withBaggage in DruidPlanner

* Fix docs, add comments

* Close the iterator if hasNext returns false
2018-10-10 17:17:29 -07:00
Atul Mohan e69a2f217b Fix avro parser issue while deserializing sketches (#6440)
* Fix object transform for avro parser

* Remove unwanted space
2018-10-10 08:53:26 -07:00
Clint Wylie 1fa045862a exclude all redundant and unecessary hive-exec depends in orc-extensions pom.xml (#6441) 2018-10-10 08:24:28 -07:00
Roman Leventov 09126c021a
Remove Aggregator.clone() methods (#6437)
* Remove Aggregator.clone() methods

* Remove CardinalityAggregator.name
2018-10-10 10:07:56 -03:00
QiuMM d559dfecb2 replace deprecated druid.port by druid.plaintextPort in docs (#6427) 2018-10-09 10:57:01 -07:00
Jihoon Son 9343cbc63a Fix CompactionTask to consider only latest segments (#6429)
* CompactionTask should consider only latest segments

* fix test
2018-10-08 21:53:16 -07:00
Roman Leventov af9efdbedf
HttpPostEmitterMonitor: don't emit maxTime and minTime if no times were recorded (#6418)
* HttpPostEmitterMonitor: don't emit maxTime and minTime if no times were recorded

* Don't emit sum and count if none

* Remove outdated comments
2018-10-08 17:11:42 -03:00
Jihoon Son 2b76d57347 Fail compactionTask if it fails to run one of indexTaskSpecs (#6428)
* Fail compactionTask if it fails to run one of indexTaskSpecs

* add log
2018-10-08 08:53:32 -07:00
Jihoon Son 88d23b77b7 Add support keepSegmentGranularity for automatic compaction (#6407)
* Add support keepSegmentGranularity for automatic compaction

* skip unknown dataSource

* ignore single semgnet to compact

* add doc

* address comments

* address comment
2018-10-07 16:48:58 -07:00
Jihoon Son 45aa51a00c Add support hash partitioning by a subset of dimensions to indexTask (#6326)
* Add support hash partitioning by a subset of dimensions to indexTask

* add doc

* fix style

* fix test

* fix doc

* fix build
2018-10-06 16:45:07 -07:00
Roman Leventov c5872bef41 Improve GC metrics documentation (#6423) 2018-10-05 14:57:01 -07:00
Charles Allen 1c4f787ed4
Upgrade Netty to 4.1.x (#6417)
* Update netty to 4.1.30.Final

* Fix compile time problems with new netty

* Remove netty-all from rocketmq extension
2018-10-05 12:30:00 -07:00
Atul Mohan 868ebfaca0 Handle case when ignoreInvalidRows is null (#6420) 2018-10-05 11:03:37 -07:00
Jonathan Wei c7ac8785a1 Prevent failed KafkaConsumer creation from blocking overlord startup (#6383)
* Prevent failed KafkaConsumer creation from blocking overlord startup

* PR comments

* Fix random task ID length

* Adjust test timer

* Use Integer.SIZE
2018-10-03 19:08:20 -07:00
QiuMM 0b8085aff7 Prohibit jackson ObjectMapper#reader methods which are deprecated (#6386)
* Prohibit jackson ObjectMapper#reader methods which are deprecated

* address comments
2018-10-03 17:55:20 -03:00
Roman Leventov 3ae563263a
Renamed 'Generic Column' -> 'Numeric Column'; Fixed a few resource leaks in processing; misc refinements (#5957)
This PR accumulates many refactorings and small improvements that I did while preparing the next change set of https://github.com/druid-io/druid/projects/2. I finally decided to make them a separate PR to minimize the volume of the main PR.

Some of the changes:
 - Renamed confusing "Generic Column" term to "Numeric Column" (what it actually implies) in many class names.
 - Generified `ComplexMetricExtractor`
2018-10-02 14:50:22 -03:00
Gian Merlino 244046fda5 SQL: Fix too-long headers in http responses. (#6411)
Fixes #6409 by moving column name info from HTTP headers into the
result body.
2018-10-01 18:13:08 -07:00
Surekha 42e5385e56 make 0.13 tasks API backwards compatible with 0.12 (#6333) (#6334)
* Replace statusCode with status (#6333)

Also changed runnerStatusCode to runnerStatus to keep things consistent

* Add unit test

* Add status param to TaskStatusPlus

Revert to statusCode and runnerStatusCode

* Add additional status member to TaskStatusPlus

* Change TaskResponseObject to match overlord's response object

* Address PR comments

* address comments

* Add runtime exception after logging error

* Remove (deprecated)status member variable from TaskStatusPlus

* Minor change
2018-10-01 15:33:24 -07:00
Jihoon Son cb14a43038 Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask (#6393)
* Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask

* update doc and remove auto conversion

* remove remaining doc

* fix teamcity
2018-10-01 12:03:35 -07:00
Gian Merlino 6e878249a0 ITTLSTest: Retry on "Connection reset by peer". (#6406)
* ITTLSTest: Retry on "Connection reset by peer".

This sometimes happens and causes tests to fail spuriously.

* Better error messages when retries are exhausted.
2018-09-30 12:30:10 -07:00
Gian Merlino 9fa4afdb8e URL encode datasources, task ids, authenticator names. (#5938)
* URL encode datasources, task ids, authenticator names.

* Fix URL encoding for router forwarding servlets.

* Fix log-with-offset API.

* Fix test.

* Test adjustments.

* Task client fixes.

* Remove unused import.
2018-09-30 12:29:51 -07:00
Gian Merlino 3548396a45 SQL: Update to Calcite 1.17.0. (#6404)
* SQL: Update to Calcite 1.17.0.

Other than keeping things fresh, another motivation is that
this fixes CALCITE-1436 (AggregateNode NPE for aggregators other
than SUM/COUNT), which affects aggregate functions on our system
tables.

Also sets shouldConvertRaggedUnionTypesToVarying = true, a new
type system parameter that prefers VARCHAR over CHAR. This is
better for Druid, because we don't really have support for a
true CHAR type.

* Remove unused import.
2018-09-29 18:33:29 -07:00
Shiv Toolsidass a56ffe6ab2 Added backpressure metric to docs and defaultMetricDimensions (#6405)
* Added backpressure metric to docs and defaultMetricDimensions.json

* Reworded description for backpressure metric in docs
2018-09-29 17:57:29 -07:00
Shiv Toolsidass 5a894f830b Added backpressure metric (#6335)
* Added backpressure metric

* Updated channelReadable to AtomicBoolean and fixed broken test

* Moved backpressure metric logic to NettyHttpClient

* Fix placement of calculating backPressureDuration
2018-09-29 14:24:04 -07:00
Jihoon Son f09e718c68 Implement MapVirtualColumn.makeDimensionSelector properly (#6396)
* Implement MapVirtualColumn.makeDimensionSelector properly

* address comments
2018-09-29 14:13:05 -07:00