Commit Graph

9297 Commits

Author SHA1 Message Date
Gian Merlino 7f3a0dae28
ParseSpec: Remove default setting. (#6310)
* ParseSpec: Remove default setting.

Having a default ParseSpec implementation is bad for users, because it masks
problems specifying the format. Two common problems masked by this are specifying
the "format" at the wrong level of the JSON, and specifying a format that
Druid doesn't support. In both cases, having a default implementation means that
users will get the delimited parser rather than an error, and then be confused
when, later on, their data failed to parse.

* Fix integration tests.
2018-09-11 19:16:19 -07:00
Gian Merlino d6cbdf86c2
Broker backpressure. (#6313)
* Broker backpressure.

Adds a new property "druid.broker.http.maxQueuedBytes" and a new context
parameter "maxQueuedBytes". Both represent a maximum number of bytes queued
per query before exerting backpressure on the channel to the data server.

Fixes #4933.

* Fix query context doc.
2018-09-10 09:33:29 -07:00
Gian Merlino 4669f0878f SQL: UNION ALL operator. (#6314)
* SQL: UNION ALL operator.

* Remove unused import.
2018-09-09 22:32:56 -07:00
Clint Wylie e6e068ce60 Add support for 'maxTotalRows' to incremental publishing kafka indexing task and appenderator based realtime task (#6129)
* resolves #5898 by adding maxTotalRows to incremental publishing kafka index task and appenderator based realtime indexing task, as available in IndexTask

* address review comments

* changes due to review

* merge fail
2018-09-07 13:17:49 -07:00
Clint Wylie e095f63e8e fix coordinator console loading (#6276) 2018-09-06 16:59:51 -07:00
Jonathan Wei 60cbc64472
Use PasswordProvider, fix info on initial passwords in basic security extension docs (#6303)
* Fix info on initial passwords in basic security extension docs

* Use PasswordProvider

* Compile fix
2018-09-05 17:07:16 -07:00
Himanshu d61f708ef5 make COMPLEX column optionally filterable in Druid code (#6223)
* make COMPLEX column filterable in Druid code

* Revert "make COMPLEX column filterable in Druid code"

This reverts commit 9fc6ec768c.

* complex columns can be optionally made filterable

* some types are always filterable

* add ColumnCapabilitiesImpl serde tests

* add SuppresedWarnings annotation
2018-09-05 12:28:49 -07:00
Gian Merlino be6c901114 Like filter: Fix escapes escaping themselves. (#6295)
Escapes should escape themselves.
2018-09-05 09:29:07 -07:00
Jonathan Wei 4caa61d8fa Fix tutorial sample data filename, fix logger classname in metrics docs (#6299) 2018-09-04 21:47:12 -07:00
QiuMM 84810f6358 correct metric name in emitter configuration files (#6290) 2018-09-04 14:23:04 -07:00
adursun 71ac3ada21 Fix link related to metadata storage (#6294) 2018-09-04 14:20:57 -07:00
Eyal Yurman 10ca290d64 Correct file name typo in Quickstart tutorial (#6297)
Correct name wikipedia-2015-09-12-sampled.json.gz to wikiticker-2015-09-12-sampled.json.gz
2018-09-04 14:20:17 -07:00
Jonathan Wei 180e3ccfad
Docs consistency cleanup (#6259) 2018-09-04 12:54:41 -07:00
Dayue Gao 743547fc3b Unauthorized sql request should return 403 (#6279) 2018-09-01 09:17:18 -07:00
Jonathan Wei d0fb83760e
Fix PostgreSQLConnectorConfig binding (#6273) 2018-08-31 14:18:29 -07:00
Dayue Gao 951b36e2bc BytesFullResponseHandler should only consume readableBytes of ChannelBuffer (#6270) 2018-08-30 20:22:08 -07:00
QiuMM 9b04846e6b correct metric name in doc file (#6271) 2018-08-30 10:57:35 -07:00
Gian Merlino 431d3d8497
Rename io.druid to org.apache.druid. (#6266)
* Rename io.druid to org.apache.druid.

* Fix META-INF files and remove some benchmark results.

* MonitorsConfig update for metrics package migration.

* Reorder some dimensions in inner queries for some reason.

* Fix protobuf tests.
2018-08-30 09:56:26 -07:00
Himanshu 1fae6513e1 add "subtotalsSpec" attribute to groupBy query (#5280)
* add subtotalsSpec attribute to groupBy query

* dont sent subtotalsSpec to downstream nodes from broker and other updates

* address review comment

* fix checkstyle issues after merge to master

* add docs for subtotalsSpec feature

* address doc review comments
2018-08-28 17:46:38 -07:00
Dayue Gao fcf8c8d53c RowBasedKeySerde should use empty dictionary in constructor (#6256) 2018-08-28 17:22:18 -07:00
Jonathan Wei c9a27e3e8e
Don't let catch/finally suppress main exception in IncrementalPublishingKafkaIndexTaskRunner (#6258) 2018-08-28 16:12:02 -07:00
Gian Merlino 80224df36a SQL: Fix post-aggregator naming logic for sort-project. (#6250)
The old code assumes that post-aggregator prefixes are one character
long followed by numbers. This isn't always true (we may pad with
underscores to avoid conflicts). Instead, the new code uses a different
base prefix for sort-project postaggregators ("s" instead of "p") and
uses the usual Calcites.findUnusedPrefix function to avoid conflicts.
2018-08-28 10:59:32 -07:00
Dayue Gao a879022bc8 fix AssertionError of semi join query (#6244) 2018-08-27 17:49:51 -07:00
Jim Slattery d957295b98 spelling: storage (#6248) 2018-08-27 16:35:31 -07:00
Dayue Gao 2325844a38 fix incorrect check of maxSemiJoinRowsInMemory (#6242) 2018-08-27 16:28:29 -07:00
Gian Merlino 4a8b09b6a9 Fix NPE on constant null numeric expressions. (#6232)
The bug was caused by makeExprEvalSelector returning a null object, which
it isn't supposed to do. Fixed this by renaming ConstantColumnValueSelector
to ConstantExprEvalSelector (it was only used for ExprEval anyway) and
putting logic in that class to make sure the selectors behave as expected.
2018-08-27 15:30:56 -07:00
Gian Merlino 71c1a70ff6 FilteredBufferAggregator: Fix missing relocate, isNull methods. (#6233) 2018-08-27 15:30:45 -07:00
Gian Merlino 157e75a1fe Minor followup to #6220. (#6231)
Adjustments to comments and usage of generics.
2018-08-27 12:01:44 -05:00
Jihoon Son bda5a8a95e Fix NPE in KafkaSupervisor.checkpointTaskGroup (#6206)
* Fix NPE in KafkaSupervisor.checkpointTaskGroup

* address comments

* address comment
2018-08-26 22:23:33 -07:00
Gian Merlino 0172326c62 SQL: Support more result formats, add columns header. (#6191)
* SQL: Support more result formats, add columns header.

- Add result formats for line-based JSON and CSV.
- Add X-Druid-Sql-Columns header with a list of all columns that
the response will contain.
- Add more comprehensive documentation on what callers should expect
when making Druid SQL queries.

* Fix some tests.

* Adjust tests.

* Adjust trailer, add types header.

* Fix trailers.
2018-08-26 23:00:14 -06:00
Jihoon Son 64d33eef7e Fix timeout in KafkaSupervisorTest.testCheckpointForInactiveTaskGroup (#6207)
* Fix timeout in KafkaSupervisorTest.testCheckpointForInactiveTaskGroup

* fix npe

* add taskRunner.getRunningTasks()
2018-08-26 19:59:01 -06:00
Gian Merlino cb40b6d369 Fix all inspection errors currently reported. (#6236)
* Fix all inspection errors currently reported.

TeamCity builds on master are reporting inspection errors, possibly
because there was a while where it was not running due to the Apache
migration, and there was some drift.

* Fix one more location.

* Fix tests.

* Another fix.
2018-08-26 18:36:01 -06:00
QiuMM ef91fdbf03 Zstandard decompression support (#6224) 2018-08-26 16:09:24 -07:00
Gian Merlino 23ba6f7ad7 Fix four bugs with numeric dimension output types. (#6220)
* Fix four bugs with numeric dimension output types.

This patch includes the following bug fixes:

- TopNColumnSelectorStrategyFactory: Cast dimension values to the output type
  during dimExtractionScanAndAggregate instead of updateDimExtractionResults.
  This fixes a bug where, for example, grouping on doubles-cast-to-longs would
  fail to merge two doubles that should have been combined into the same long value.
- TopNQueryEngine: Use DimExtractionTopNAlgorithm when treating string columns
  as numeric dimensions. This fixes a similar bug: grouping on string-cast-to-long
  would fail to merge two strings that should have been combined.
- GroupByQuery: Cast numeric types to the expected output type before comparing them
  in compareDimsForLimitPushDown. This fixes #6123.
- GroupByQueryQueryToolChest: Convert Jackson-deserialized dimension values into
  the proper output type. This fixes an inconsistency between results that came
  from cache vs. not-cache: for example, Jackson sometimes deserializes integers
  as Integers and sometimes as Longs.

And the following code-cleanup changes, related to the fixes above:

- DimensionHandlerUtils: Introduce convertObjectToType, compareObjectsAsType,
  and converterFromTypeToType to make it easier to handle casting operations.
- TopN in general: Rename various "dimName" variables to "dimValue" where they
  actually represent dimension values. The old names were confusing.

* Remove unused imports.
2018-08-25 14:31:46 -07:00
Himanshu c3aaf8122d fix TaskQueue-HRTR deadlock (#6212)
* fix TaskQueue-HRTR deadlock causing https://github.com/apache/incubator-druid/issues/6201

* address review comments
2018-08-25 14:15:57 -07:00
Gian Merlino 28e6ae3664
SQL: Finalize aggregations for inner queries when necessary. (#6221)
* SQL: Finalize aggregations for inner queries when necessary.

Fixes #5779.

* Fixed test method name.
2018-08-25 13:56:23 -07:00
QiuMM 9803ce954a fix port conflict for druid peon (#6202) 2018-08-23 19:05:13 -07:00
Himanshu ddb26f2696 do not ignore ms in ruby time (#6217) 2018-08-23 14:09:31 -07:00
Susie 6e73ad6231 Fix bound query keys for Filtering on numeric values (#5881)
It is currently showing the use of `lowerBound` and `upperBound` instead of `lower` and `upper` for the range.
2018-08-23 14:07:10 -07:00
QiuMM ceb8f8e625 remove unnecessary tlsPortFinder to avoid potential port conflicts (#6194) 2018-08-23 10:41:49 -07:00
Himanshu a76bf9ab2a add ability to do optional rollup in AggregationTestHelper (#6213) 2018-08-22 16:38:36 -07:00
Karol Woźniak 83c25ca3bd Fix running Overlord inside Coordinator (#6195) 2018-08-22 15:50:06 -07:00
Ryan Plessner 9c500fb69f Add PostgreSQLConnectorConfig to expose SSL configuration options (#6181)
* Add PostgreSQLConnectorConfig to expose SSL configuration options for the Postgres Metadata Storage module.

* Fix checkstyle violations and add license header

* Convert properties in the postgres docs to be the full property path and fix typo

* Fix grammar in sslFactory docs
2018-08-21 16:45:27 -07:00
Benedict Jin 3647d4c94a Make time-related variables more readable (#6158)
* Make time-related variables more readable

* Patch some improvements from the code reviewer

* Remove unnecessary boxing of Long type variables
2018-08-21 15:29:40 -07:00
QiuMM 266f3dfbcb remove duplicate link to operations/recommendations.html (#6193) 2018-08-21 12:02:43 -07:00
Benedict Jin 7d4b2d51e8 Fix assertionError at testCheckpointForInactiveTaskGroup in KafkaSupervisorTest (#6192) 2018-08-21 11:33:45 -07:00
Jihoon Son 2bfe1b6a5a Fix NPE for taskGroupId when rolling update (#6168)
* Fix NPE for taskGroupId

* missing changes

* fix wrong annotation

* fix potential race

* keep baseSequenceName

* make deprecated old param
2018-08-17 10:15:45 -07:00
kaijianding 78fc5b246c trigger clean dir when unzip fails with IOException (#6178) 2018-08-17 10:15:10 -07:00
QiuMM b0cf8d0252 'shutdownAllTasks' API for a dataSource (#6185)
* 'shutdownAllTasks' API for a dataSource

Change-Id: I30d14390457d39e0427d23a48f4f224223dc5777

* fix api path and return

Change-Id: Ib463f31ee2c4cb168cf2697f149be845b57c42e5

* optimize implementation

Change-Id: I50a8dcd44dd9d36c9ecbfa78e103eb9bff32eab9
2018-08-17 12:57:09 -04:00
Samarth Jain 1c8032f9f3 Composite request logger doesn't invoke @LifeCycleStart and @LifeCycleStop methods on its dependencies (#6173) 2018-08-17 12:34:25 -04:00