Commit Graph

8671 Commits

Author SHA1 Message Date
Gian Merlino 71c1a70ff6 FilteredBufferAggregator: Fix missing relocate, isNull methods. (#6233) 2018-08-27 15:30:45 -07:00
Gian Merlino 157e75a1fe Minor followup to #6220. (#6231)
Adjustments to comments and usage of generics.
2018-08-27 12:01:44 -05:00
Jihoon Son bda5a8a95e Fix NPE in KafkaSupervisor.checkpointTaskGroup (#6206)
* Fix NPE in KafkaSupervisor.checkpointTaskGroup

* address comments

* address comment
2018-08-26 22:23:33 -07:00
Gian Merlino 0172326c62 SQL: Support more result formats, add columns header. (#6191)
* SQL: Support more result formats, add columns header.

- Add result formats for line-based JSON and CSV.
- Add X-Druid-Sql-Columns header with a list of all columns that
the response will contain.
- Add more comprehensive documentation on what callers should expect
when making Druid SQL queries.

* Fix some tests.

* Adjust tests.

* Adjust trailer, add types header.

* Fix trailers.
2018-08-26 23:00:14 -06:00
Jihoon Son 64d33eef7e Fix timeout in KafkaSupervisorTest.testCheckpointForInactiveTaskGroup (#6207)
* Fix timeout in KafkaSupervisorTest.testCheckpointForInactiveTaskGroup

* fix npe

* add taskRunner.getRunningTasks()
2018-08-26 19:59:01 -06:00
Gian Merlino cb40b6d369 Fix all inspection errors currently reported. (#6236)
* Fix all inspection errors currently reported.

TeamCity builds on master are reporting inspection errors, possibly
because there was a while where it was not running due to the Apache
migration, and there was some drift.

* Fix one more location.

* Fix tests.

* Another fix.
2018-08-26 18:36:01 -06:00
QiuMM ef91fdbf03 Zstandard decompression support (#6224) 2018-08-26 16:09:24 -07:00
Gian Merlino 23ba6f7ad7 Fix four bugs with numeric dimension output types. (#6220)
* Fix four bugs with numeric dimension output types.

This patch includes the following bug fixes:

- TopNColumnSelectorStrategyFactory: Cast dimension values to the output type
  during dimExtractionScanAndAggregate instead of updateDimExtractionResults.
  This fixes a bug where, for example, grouping on doubles-cast-to-longs would
  fail to merge two doubles that should have been combined into the same long value.
- TopNQueryEngine: Use DimExtractionTopNAlgorithm when treating string columns
  as numeric dimensions. This fixes a similar bug: grouping on string-cast-to-long
  would fail to merge two strings that should have been combined.
- GroupByQuery: Cast numeric types to the expected output type before comparing them
  in compareDimsForLimitPushDown. This fixes #6123.
- GroupByQueryQueryToolChest: Convert Jackson-deserialized dimension values into
  the proper output type. This fixes an inconsistency between results that came
  from cache vs. not-cache: for example, Jackson sometimes deserializes integers
  as Integers and sometimes as Longs.

And the following code-cleanup changes, related to the fixes above:

- DimensionHandlerUtils: Introduce convertObjectToType, compareObjectsAsType,
  and converterFromTypeToType to make it easier to handle casting operations.
- TopN in general: Rename various "dimName" variables to "dimValue" where they
  actually represent dimension values. The old names were confusing.

* Remove unused imports.
2018-08-25 14:31:46 -07:00
Himanshu c3aaf8122d fix TaskQueue-HRTR deadlock (#6212)
* fix TaskQueue-HRTR deadlock causing https://github.com/apache/incubator-druid/issues/6201

* address review comments
2018-08-25 14:15:57 -07:00
Gian Merlino 28e6ae3664
SQL: Finalize aggregations for inner queries when necessary. (#6221)
* SQL: Finalize aggregations for inner queries when necessary.

Fixes #5779.

* Fixed test method name.
2018-08-25 13:56:23 -07:00
QiuMM 9803ce954a fix port conflict for druid peon (#6202) 2018-08-23 19:05:13 -07:00
Himanshu ddb26f2696 do not ignore ms in ruby time (#6217) 2018-08-23 14:09:31 -07:00
Susie 6e73ad6231 Fix bound query keys for Filtering on numeric values (#5881)
It is currently showing the use of `lowerBound` and `upperBound` instead of `lower` and `upper` for the range.
2018-08-23 14:07:10 -07:00
QiuMM ceb8f8e625 remove unnecessary tlsPortFinder to avoid potential port conflicts (#6194) 2018-08-23 10:41:49 -07:00
Himanshu a76bf9ab2a add ability to do optional rollup in AggregationTestHelper (#6213) 2018-08-22 16:38:36 -07:00
Karol Woźniak 83c25ca3bd Fix running Overlord inside Coordinator (#6195) 2018-08-22 15:50:06 -07:00
Ryan Plessner 9c500fb69f Add PostgreSQLConnectorConfig to expose SSL configuration options (#6181)
* Add PostgreSQLConnectorConfig to expose SSL configuration options for the Postgres Metadata Storage module.

* Fix checkstyle violations and add license header

* Convert properties in the postgres docs to be the full property path and fix typo

* Fix grammar in sslFactory docs
2018-08-21 16:45:27 -07:00
Benedict Jin 3647d4c94a Make time-related variables more readable (#6158)
* Make time-related variables more readable

* Patch some improvements from the code reviewer

* Remove unnecessary boxing of Long type variables
2018-08-21 15:29:40 -07:00
QiuMM 266f3dfbcb remove duplicate link to operations/recommendations.html (#6193) 2018-08-21 12:02:43 -07:00
Benedict Jin 7d4b2d51e8 Fix assertionError at testCheckpointForInactiveTaskGroup in KafkaSupervisorTest (#6192) 2018-08-21 11:33:45 -07:00
Jihoon Son 2bfe1b6a5a Fix NPE for taskGroupId when rolling update (#6168)
* Fix NPE for taskGroupId

* missing changes

* fix wrong annotation

* fix potential race

* keep baseSequenceName

* make deprecated old param
2018-08-17 10:15:45 -07:00
kaijianding 78fc5b246c trigger clean dir when unzip fails with IOException (#6178) 2018-08-17 10:15:10 -07:00
QiuMM b0cf8d0252 'shutdownAllTasks' API for a dataSource (#6185)
* 'shutdownAllTasks' API for a dataSource

Change-Id: I30d14390457d39e0427d23a48f4f224223dc5777

* fix api path and return

Change-Id: Ib463f31ee2c4cb168cf2697f149be845b57c42e5

* optimize implementation

Change-Id: I50a8dcd44dd9d36c9ecbfa78e103eb9bff32eab9
2018-08-17 12:57:09 -04:00
Samarth Jain 1c8032f9f3 Composite request logger doesn't invoke @LifeCycleStart and @LifeCycleStop methods on its dependencies (#6173) 2018-08-17 12:34:25 -04:00
Jonathan Wei 0c3bb47558 Change hybrid cache default types in docs to caffeine (#6182) 2018-08-17 12:17:43 -04:00
Gian Merlino 4d2ff0f6c7
Serde test for JdbcExtractionNamespace. (#6186) 2018-08-17 11:54:06 -04:00
Kirill Kozlov 62e580050c Use JUnit TemporaryFolder rule instead of system temp folder (#6070)
* Use JUnit TemporaryFolder rule instead of system tmp folder

* Allow to forbid apis which present not in all mvn modules
2018-08-16 11:05:45 -07:00
es1220 5726692f8f Accept total rows over Integer.MAX_VALUE. (#6080) 2018-08-15 14:03:22 -07:00
Gian Merlino 5ce3185b9c Fix three bugs with segment publishing. (#6155)
* Fix three bugs with segment publishing.

1. In AppenderatorImpl: always use a unique path if requested, even if the segment
   was already pushed. This is important because if we don't do this, it causes
   the issue mentioned in #6124.
2. In IndexerSQLMetadataStorageCoordinator: Fix a bug that could cause it to return
   a "not published" result instead of throwing an exception, when there was one
   metadata update failure, followed by some random exception. This is done by
   resetting the AtomicBoolean that tracks what case we're in, each time the
   callback runs.
3. In BaseAppenderatorDriver: Only kill segments if we get an affirmative false
   publish result. Skip killing if we just got some exception. The reason for this
   is that we want to avoid killing segments if they are in an unknown state.

Two other changes to clarify the contracts a bit and hopefully prevent future bugs:

1. Return SegmentPublishResult from TransactionalSegmentPublisher, to make it
more similar to announceHistoricalSegments.
2. Make it explicit, at multiple levels of javadocs, that a "false" publish result
must indicate that the publish _definitely_ did not happen. Unknown states must be
exceptions. This helps BaseAppenderatorDriver do the right thing.

* Remove javadoc-only import.

* Updates.

* Fix test.

* Fix tests.
2018-08-15 13:55:53 -07:00
Caroline1000 f447b784de update sigar link (#6175) 2018-08-14 16:58:29 -07:00
kaijianding 8d886e59bf fix compress fail when file size is Integer.MAX_VALUE (#6159) 2018-08-14 15:13:47 -07:00
Alexander Saydakov c47032d566 Implemented makeAggregateCombiner() in ArrayOfDoublesSketchAggregatorFactory (#6093)
* implemented makeAggregateCombiner()

* test for makeAggregateCombiner()

* license, style fix
2018-08-13 14:19:11 -07:00
QiuMM 69f555019b convert all time-intervals in ISO 8601 format to uppercase in doc files (#6118)
Change-Id: I904fed4cfb600a8a42664335557f611133a5078d
2018-08-13 12:58:47 -07:00
Jonathan Wei 94a937b5e8
New doc fixes (#6156) 2018-08-13 11:11:32 -07:00
Jihoon Son a7ca4589dd Fix race in testCheckpointForUnknownTaskGroup() of KafkaSupervisorTest (#6153) 2018-08-11 08:26:46 -07:00
Karol Woźniak da3a1f61ac Fix appenderator_realtime creating shards bigger by 1 than maxRowsPerSegment (#6125) 2018-08-10 22:29:06 -07:00
Jihoon Son ecee3e0a24 Further optimize memory for Travis jobs (#6150)
* Further optimize memory for Travis jobs

* fix build

* sudo false
2018-08-10 22:03:36 -07:00
Benedict Jin bd95b426c9 Fix missing exception handling as part of `io.druid.java.util.http.client.netty.HttpClientPipelineFactory` (#6090)
* Fix missing exception handling as part of `io.druid.java.util.http.client.netty.HttpClientPipelineFactory`

* 1. Extends SimpleChannelUpstreamHandler; 2. Remove sendUpstream; 3. Using ExpectedException.

* Add more checks for channel

* Fix missing exception handler in NettyHttpClient and ChannelResourceFactory

* Rename the anonymous class of `SimpleChannelUpstreamHandler` as connectionErrorHandler & use `addLast` instead of `addFirst`

* Remove `removeHandlers()`

* Using expectedException.expect instead of Assert.assertNotNull in testHttpsEchoServer

* Using handshakeFuture.setFailure instead of logger

* Using handshakeFuture.setFailure instead of logger
2018-08-10 17:02:53 -07:00
Atul Mohan 064c22c937 Fix redirects (#6151) 2018-08-10 13:55:47 -07:00
Christoph Hösler 1a37dfdcd1 Fetch unhandled curator exceptions (#6131)
* fix: stop druid on unhandled curator exceptions

* catch exceptions when stopping lifecycle
2018-08-09 21:47:42 -07:00
Jonathan Wei b0805540af
Fix kafka tutorial typo (#6141) 2018-08-09 18:41:05 -07:00
Jonathan Wei af0557c1f7
Unified configuration doc page (#6127)
* Unified configuration doc page

* Rename to index.md, update redirects

* PR comments

* PR comments

* PR comment
2018-08-09 14:52:14 -07:00
Jonathan Wei fea2ab7094
New docs intro (#6122)
* New docs intro

* PR comments

* Fix arch diagram

* PR comment

* PR comment

* PR comment
2018-08-09 14:19:11 -07:00
Jihoon Son d6a02de5b5 Add support 'keepSegmentGranularity' for compactionTask (#6095)
* Add keepSegmentGranularity for compactionTask

* fix build

* createIoConfig method

* fix build

* fix build

* address comments

* fix build
2018-08-09 13:51:20 -07:00
pdeva c028d18d74 update redis-cache documentation (#6109)
* update redis-cache documentation

added clarifying info on setup and enablement

* added link
2018-08-09 13:44:59 -07:00
Jonathan Wei aa660b8751 Add docs for virtual columns and transform specs (#6119)
* Add docs for virtual columns and transform specs

* PR Comments

* PR comment
2018-08-09 14:42:52 -06:00
Jonathan Wei 2b64025eaf Separate hadoop and native batch docs more (#6120)
* Separate hadoop and native batch docs more

* Rebase with parallel batch

* PR comments
2018-08-09 14:40:20 -06:00
Jonathan Wei 24f2e8ba26 New quickstart and tutorials (#6126)
* New quickstart and tutorials

* PR comments

* Fix tranquility
2018-08-09 14:37:52 -06:00
Jonathan Wei 2b0f03acb9 Unified API doc page (#6128)
* Unified API doc page

* PR comments

* Fix metadata endpoint
2018-08-09 14:27:42 -06:00
chengchengpei c3b7704b50 validate non-empty String baseDataSource in MaterializedViewSupervisorSpec, added tests (#6075) 2018-08-07 17:23:10 -07:00