Commit Graph

8980 Commits

Author SHA1 Message Date
Gian Merlino 431d3d8497
Rename io.druid to org.apache.druid. (#6266)
* Rename io.druid to org.apache.druid.

* Fix META-INF files and remove some benchmark results.

* MonitorsConfig update for metrics package migration.

* Reorder some dimensions in inner queries for some reason.

* Fix protobuf tests.
2018-08-30 09:56:26 -07:00
Himanshu 1fae6513e1 add "subtotalsSpec" attribute to groupBy query (#5280)
* add subtotalsSpec attribute to groupBy query

* dont sent subtotalsSpec to downstream nodes from broker and other updates

* address review comment

* fix checkstyle issues after merge to master

* add docs for subtotalsSpec feature

* address doc review comments
2018-08-28 17:46:38 -07:00
Dayue Gao fcf8c8d53c RowBasedKeySerde should use empty dictionary in constructor (#6256) 2018-08-28 17:22:18 -07:00
Jonathan Wei c9a27e3e8e
Don't let catch/finally suppress main exception in IncrementalPublishingKafkaIndexTaskRunner (#6258) 2018-08-28 16:12:02 -07:00
Gian Merlino 80224df36a SQL: Fix post-aggregator naming logic for sort-project. (#6250)
The old code assumes that post-aggregator prefixes are one character
long followed by numbers. This isn't always true (we may pad with
underscores to avoid conflicts). Instead, the new code uses a different
base prefix for sort-project postaggregators ("s" instead of "p") and
uses the usual Calcites.findUnusedPrefix function to avoid conflicts.
2018-08-28 10:59:32 -07:00
Dayue Gao a879022bc8 fix AssertionError of semi join query (#6244) 2018-08-27 17:49:51 -07:00
Jim Slattery d957295b98 spelling: storage (#6248) 2018-08-27 16:35:31 -07:00
Dayue Gao 2325844a38 fix incorrect check of maxSemiJoinRowsInMemory (#6242) 2018-08-27 16:28:29 -07:00
Gian Merlino 4a8b09b6a9 Fix NPE on constant null numeric expressions. (#6232)
The bug was caused by makeExprEvalSelector returning a null object, which
it isn't supposed to do. Fixed this by renaming ConstantColumnValueSelector
to ConstantExprEvalSelector (it was only used for ExprEval anyway) and
putting logic in that class to make sure the selectors behave as expected.
2018-08-27 15:30:56 -07:00
Gian Merlino 71c1a70ff6 FilteredBufferAggregator: Fix missing relocate, isNull methods. (#6233) 2018-08-27 15:30:45 -07:00
Gian Merlino 157e75a1fe Minor followup to #6220. (#6231)
Adjustments to comments and usage of generics.
2018-08-27 12:01:44 -05:00
Jihoon Son bda5a8a95e Fix NPE in KafkaSupervisor.checkpointTaskGroup (#6206)
* Fix NPE in KafkaSupervisor.checkpointTaskGroup

* address comments

* address comment
2018-08-26 22:23:33 -07:00
Gian Merlino 0172326c62 SQL: Support more result formats, add columns header. (#6191)
* SQL: Support more result formats, add columns header.

- Add result formats for line-based JSON and CSV.
- Add X-Druid-Sql-Columns header with a list of all columns that
the response will contain.
- Add more comprehensive documentation on what callers should expect
when making Druid SQL queries.

* Fix some tests.

* Adjust tests.

* Adjust trailer, add types header.

* Fix trailers.
2018-08-26 23:00:14 -06:00
Jihoon Son 64d33eef7e Fix timeout in KafkaSupervisorTest.testCheckpointForInactiveTaskGroup (#6207)
* Fix timeout in KafkaSupervisorTest.testCheckpointForInactiveTaskGroup

* fix npe

* add taskRunner.getRunningTasks()
2018-08-26 19:59:01 -06:00
Gian Merlino cb40b6d369 Fix all inspection errors currently reported. (#6236)
* Fix all inspection errors currently reported.

TeamCity builds on master are reporting inspection errors, possibly
because there was a while where it was not running due to the Apache
migration, and there was some drift.

* Fix one more location.

* Fix tests.

* Another fix.
2018-08-26 18:36:01 -06:00
QiuMM ef91fdbf03 Zstandard decompression support (#6224) 2018-08-26 16:09:24 -07:00
Gian Merlino 23ba6f7ad7 Fix four bugs with numeric dimension output types. (#6220)
* Fix four bugs with numeric dimension output types.

This patch includes the following bug fixes:

- TopNColumnSelectorStrategyFactory: Cast dimension values to the output type
  during dimExtractionScanAndAggregate instead of updateDimExtractionResults.
  This fixes a bug where, for example, grouping on doubles-cast-to-longs would
  fail to merge two doubles that should have been combined into the same long value.
- TopNQueryEngine: Use DimExtractionTopNAlgorithm when treating string columns
  as numeric dimensions. This fixes a similar bug: grouping on string-cast-to-long
  would fail to merge two strings that should have been combined.
- GroupByQuery: Cast numeric types to the expected output type before comparing them
  in compareDimsForLimitPushDown. This fixes #6123.
- GroupByQueryQueryToolChest: Convert Jackson-deserialized dimension values into
  the proper output type. This fixes an inconsistency between results that came
  from cache vs. not-cache: for example, Jackson sometimes deserializes integers
  as Integers and sometimes as Longs.

And the following code-cleanup changes, related to the fixes above:

- DimensionHandlerUtils: Introduce convertObjectToType, compareObjectsAsType,
  and converterFromTypeToType to make it easier to handle casting operations.
- TopN in general: Rename various "dimName" variables to "dimValue" where they
  actually represent dimension values. The old names were confusing.

* Remove unused imports.
2018-08-25 14:31:46 -07:00
Himanshu c3aaf8122d fix TaskQueue-HRTR deadlock (#6212)
* fix TaskQueue-HRTR deadlock causing https://github.com/apache/incubator-druid/issues/6201

* address review comments
2018-08-25 14:15:57 -07:00
Gian Merlino 28e6ae3664
SQL: Finalize aggregations for inner queries when necessary. (#6221)
* SQL: Finalize aggregations for inner queries when necessary.

Fixes #5779.

* Fixed test method name.
2018-08-25 13:56:23 -07:00
QiuMM 9803ce954a fix port conflict for druid peon (#6202) 2018-08-23 19:05:13 -07:00
Himanshu ddb26f2696 do not ignore ms in ruby time (#6217) 2018-08-23 14:09:31 -07:00
Susie 6e73ad6231 Fix bound query keys for Filtering on numeric values (#5881)
It is currently showing the use of `lowerBound` and `upperBound` instead of `lower` and `upper` for the range.
2018-08-23 14:07:10 -07:00
QiuMM ceb8f8e625 remove unnecessary tlsPortFinder to avoid potential port conflicts (#6194) 2018-08-23 10:41:49 -07:00
Himanshu a76bf9ab2a add ability to do optional rollup in AggregationTestHelper (#6213) 2018-08-22 16:38:36 -07:00
Karol Woźniak 83c25ca3bd Fix running Overlord inside Coordinator (#6195) 2018-08-22 15:50:06 -07:00
Ryan Plessner 9c500fb69f Add PostgreSQLConnectorConfig to expose SSL configuration options (#6181)
* Add PostgreSQLConnectorConfig to expose SSL configuration options for the Postgres Metadata Storage module.

* Fix checkstyle violations and add license header

* Convert properties in the postgres docs to be the full property path and fix typo

* Fix grammar in sslFactory docs
2018-08-21 16:45:27 -07:00
Benedict Jin 3647d4c94a Make time-related variables more readable (#6158)
* Make time-related variables more readable

* Patch some improvements from the code reviewer

* Remove unnecessary boxing of Long type variables
2018-08-21 15:29:40 -07:00
QiuMM 266f3dfbcb remove duplicate link to operations/recommendations.html (#6193) 2018-08-21 12:02:43 -07:00
Benedict Jin 7d4b2d51e8 Fix assertionError at testCheckpointForInactiveTaskGroup in KafkaSupervisorTest (#6192) 2018-08-21 11:33:45 -07:00
Jihoon Son 2bfe1b6a5a Fix NPE for taskGroupId when rolling update (#6168)
* Fix NPE for taskGroupId

* missing changes

* fix wrong annotation

* fix potential race

* keep baseSequenceName

* make deprecated old param
2018-08-17 10:15:45 -07:00
kaijianding 78fc5b246c trigger clean dir when unzip fails with IOException (#6178) 2018-08-17 10:15:10 -07:00
QiuMM b0cf8d0252 'shutdownAllTasks' API for a dataSource (#6185)
* 'shutdownAllTasks' API for a dataSource

Change-Id: I30d14390457d39e0427d23a48f4f224223dc5777

* fix api path and return

Change-Id: Ib463f31ee2c4cb168cf2697f149be845b57c42e5

* optimize implementation

Change-Id: I50a8dcd44dd9d36c9ecbfa78e103eb9bff32eab9
2018-08-17 12:57:09 -04:00
Samarth Jain 1c8032f9f3 Composite request logger doesn't invoke @LifeCycleStart and @LifeCycleStop methods on its dependencies (#6173) 2018-08-17 12:34:25 -04:00
Jonathan Wei 0c3bb47558 Change hybrid cache default types in docs to caffeine (#6182) 2018-08-17 12:17:43 -04:00
Gian Merlino 4d2ff0f6c7
Serde test for JdbcExtractionNamespace. (#6186) 2018-08-17 11:54:06 -04:00
Kirill Kozlov 62e580050c Use JUnit TemporaryFolder rule instead of system temp folder (#6070)
* Use JUnit TemporaryFolder rule instead of system tmp folder

* Allow to forbid apis which present not in all mvn modules
2018-08-16 11:05:45 -07:00
es1220 5726692f8f Accept total rows over Integer.MAX_VALUE. (#6080) 2018-08-15 14:03:22 -07:00
Gian Merlino 5ce3185b9c Fix three bugs with segment publishing. (#6155)
* Fix three bugs with segment publishing.

1. In AppenderatorImpl: always use a unique path if requested, even if the segment
   was already pushed. This is important because if we don't do this, it causes
   the issue mentioned in #6124.
2. In IndexerSQLMetadataStorageCoordinator: Fix a bug that could cause it to return
   a "not published" result instead of throwing an exception, when there was one
   metadata update failure, followed by some random exception. This is done by
   resetting the AtomicBoolean that tracks what case we're in, each time the
   callback runs.
3. In BaseAppenderatorDriver: Only kill segments if we get an affirmative false
   publish result. Skip killing if we just got some exception. The reason for this
   is that we want to avoid killing segments if they are in an unknown state.

Two other changes to clarify the contracts a bit and hopefully prevent future bugs:

1. Return SegmentPublishResult from TransactionalSegmentPublisher, to make it
more similar to announceHistoricalSegments.
2. Make it explicit, at multiple levels of javadocs, that a "false" publish result
must indicate that the publish _definitely_ did not happen. Unknown states must be
exceptions. This helps BaseAppenderatorDriver do the right thing.

* Remove javadoc-only import.

* Updates.

* Fix test.

* Fix tests.
2018-08-15 13:55:53 -07:00
Caroline1000 f447b784de update sigar link (#6175) 2018-08-14 16:58:29 -07:00
kaijianding 8d886e59bf fix compress fail when file size is Integer.MAX_VALUE (#6159) 2018-08-14 15:13:47 -07:00
Alexander Saydakov c47032d566 Implemented makeAggregateCombiner() in ArrayOfDoublesSketchAggregatorFactory (#6093)
* implemented makeAggregateCombiner()

* test for makeAggregateCombiner()

* license, style fix
2018-08-13 14:19:11 -07:00
QiuMM 69f555019b convert all time-intervals in ISO 8601 format to uppercase in doc files (#6118)
Change-Id: I904fed4cfb600a8a42664335557f611133a5078d
2018-08-13 12:58:47 -07:00
Jonathan Wei 94a937b5e8
New doc fixes (#6156) 2018-08-13 11:11:32 -07:00
Jihoon Son a7ca4589dd Fix race in testCheckpointForUnknownTaskGroup() of KafkaSupervisorTest (#6153) 2018-08-11 08:26:46 -07:00
Karol Woźniak da3a1f61ac Fix appenderator_realtime creating shards bigger by 1 than maxRowsPerSegment (#6125) 2018-08-10 22:29:06 -07:00
Jihoon Son ecee3e0a24 Further optimize memory for Travis jobs (#6150)
* Further optimize memory for Travis jobs

* fix build

* sudo false
2018-08-10 22:03:36 -07:00
Benedict Jin bd95b426c9 Fix missing exception handling as part of `io.druid.java.util.http.client.netty.HttpClientPipelineFactory` (#6090)
* Fix missing exception handling as part of `io.druid.java.util.http.client.netty.HttpClientPipelineFactory`

* 1. Extends SimpleChannelUpstreamHandler; 2. Remove sendUpstream; 3. Using ExpectedException.

* Add more checks for channel

* Fix missing exception handler in NettyHttpClient and ChannelResourceFactory

* Rename the anonymous class of `SimpleChannelUpstreamHandler` as connectionErrorHandler & use `addLast` instead of `addFirst`

* Remove `removeHandlers()`

* Using expectedException.expect instead of Assert.assertNotNull in testHttpsEchoServer

* Using handshakeFuture.setFailure instead of logger

* Using handshakeFuture.setFailure instead of logger
2018-08-10 17:02:53 -07:00
Atul Mohan 064c22c937 Fix redirects (#6151) 2018-08-10 13:55:47 -07:00
Christoph Hösler 1a37dfdcd1 Fetch unhandled curator exceptions (#6131)
* fix: stop druid on unhandled curator exceptions

* catch exceptions when stopping lifecycle
2018-08-09 21:47:42 -07:00
Jonathan Wei b0805540af
Fix kafka tutorial typo (#6141) 2018-08-09 18:41:05 -07:00