Commit Graph

9168 Commits

Author SHA1 Message Date
Jonathan Wei 5486c2abf8
Update LICENSE and NOTICE files (#7026)
* Update LICENSE and NOTICE files

* Update react-table version
2019-03-04 18:45:22 -08:00
Clint Wylie 3398d3982f
fix intellij UnusedInspectionsScope.xml (#7158) 2019-03-04 14:56:41 -08:00
Roman Leventov 10c9f6d708
Fix and document concurrency of EventReceiverFirehose and TimedShutoffFirehose; Refine concurrency specification of Firehose (#7038)
#### `EventReceiverFirehoseFactory`
Fixed several concurrency bugs in `EventReceiverFirehoseFactory`:
 - Race condition over putting an entry into `producerSequences` in `checkProducerSequence()`.
 - `Stopwatch` used to measure time across threads, but it's a non-thread-safe class.
 - Use `System.nanoTime()` instead of `System.currentTimeMillis()` because the latter are [not suitable](https://stackoverflow.com/a/351571/648955)  for measuring time intervals.
 - `close()` was not synchronized by could be called from multiple threads concurrently.

Removed unnecessary `readLock` (protecting `hasMore()` and `nextRow()` which are always called from a single thread). Removed unnecessary `volatile` modifiers.

Documented threading model and concurrent control flow of `EventReceiverFirehose` instances.

**Important:** please read the updated Javadoc for `EventReceiverFirehose.addAll()`. It allows events from different requests (batches) to be interleaved in the buffer. Is this OK?

#### `TimedShutoffFirehoseFactory`
- Fixed a race condition that was possible because `close()` that was not properly synchronized.

Documented threading model and concurrent control flow of `TimedShutoffFirehose` instances.

#### `Firehose`

Refined concurrency contract of `Firehose` based on `EventReceiverFirehose` implementation. Importantly, now it states that `close()` doesn't affect `hasMore()` and `nextRow()` and could be called concurrently with them. In other words, specified that `close()` is for "row supply" side rather than "row consume" side. However, I didn't check that other `Firehose` implementatations adhere to this contract.

<hr>

This issue is the result of reviewing `EventReceiverFirehose` and `TimedShutoffFirehose` using [this checklist](https://medium.com/@leventov/code-review-checklist-java-concurrency-49398c326154).
2019-03-04 18:50:03 -03:00
David Glasser 7bf1ee4dc0 ITIndexerTest: validate new data source after reindex (#7171)
Previously, the test validated that the data source that we ingested from still
had the same query responses that it did before the second ingestion. This is
less useful than validating queries against the newly created data source.

The new queries file differs from the old one in that its maxTime is earlier due
to the interval selected by the reindex, and in that it does not query for the
dropped metric "count".
2019-03-04 11:05:40 -08:00
Clint Wylie 050728b115 add license checker to web-console (#7028)
* add license checker to web-console to ensure npm dependencies are apache license compatible

* add generate licenses file

* update check to remove excludes due to blueprintjs downgrade
2019-03-02 12:22:54 -08:00
Jihoon Son ded03d9d4c Improve doc for auto compaction (#7117)
* Improve doc for auto compaction

* fix doc

* address comments
2019-03-02 12:21:50 -08:00
Gian Merlino fa218f5160 Fix two SeekableStream serde issues. (#7176)
* Fix two SeekableStream serde issues.

1) Fix backwards-compatibility serde for SeekableStreamPartitions. It is needed
   for split 0.13 / 0.14 clusters to work properly during a rolling update.
2) Abstract classes don't need JsonCreator constructors; remove them.

* Comment fixes.
2019-03-01 22:27:08 -08:00
Jihoon Son 06c8229c08
Kill all running tasks when the supervisor task is killed (#7041)
* Kill all running tasks when the supervisor task is killed

* add some docs and simplify

* address comment
2019-03-01 11:28:03 -08:00
Jihoon Son 45f12de9ad Fix supported file formats for Hadoop vs Native batch doc (#7069)
* Fix supported file formats

* address comment
2019-02-28 19:44:45 -08:00
Jonathan Wei 32c418fdd8 Reword 'node' to 'process' (#7172) 2019-02-28 18:10:39 -08:00
Vadim Ogievetsky 66e8d35ddf downgrade react-table (#7170) 2019-02-28 18:09:38 -08:00
Jihoon Son 9a62157a06
Make MapBasedRow immutable (#7130)
* Make MapBasedRow immutable

* add null check
2019-02-28 16:07:14 -08:00
Jonathan Wei a0afd7931d
Add web consoles doc page (#7123)
* Add web consoles doc page

* PR comments

* Remove 'unified'

* PR comments

* Fix TOC

* PR comments

* More revisions

* GUI -> UI

* Update router docs

* Reword router doc
2019-02-28 14:02:39 -08:00
Furkan KAMACI e432965c13 * Overview and Login precedure are added. (#7135)
* Typos are fixed.
2019-02-27 20:51:59 -08:00
Justin Borromeo c7082ba36e Added friendlier dsql error message for 405 (which occurs when druid.sql.enabled=false) (#7112)
* Added friendlier error message for dsql 405

* no extra char

* Changed error message

* fixed weird spacing
2019-02-27 20:40:30 -08:00
Jonathan Wei 0b4f771062 Exclude hadoop-lzo from thrift-extensions build (#7151) 2019-02-27 19:57:53 -08:00
Jonathan Wei 3d247498ef Update tutorials for 0.14.0-incubating (#7157) 2019-02-27 19:50:31 -08:00
Jihoon Son cacdc83cad Improve error message for integer overflow in compaction task (#7131)
* improve error message for integer overflow in compaction task

* fix build
2019-02-28 11:07:37 +08:00
Jihoon Son 6b232d8195 Improve compaction tutorial to demonstrate compaction with keepSegmentGranularity = true (#7079)
* Improve compaction tutorial to demonstrate compaction with keepSegmentGranularity = true

* typo

* add a warning
2019-02-27 16:02:51 -08:00
Clint Wylie 9fa649b3bd segment metadata fallback analysis if no bitmaps (#7116)
* segment metadata fallback analysis if no bitmaps

* remove accidental line

* remove nonsense size estimation

* less ternary

* fix it

* do the thing
2019-02-26 11:27:41 -08:00
Vadim Ogievetsky b8f762037a Downgrade blueprintjs version in the web console to one with a vanilla Apache 2.0 license (#7139)
* revert bp

* fix tests

* move @types/hjson to dev dep

* removed all the package upgrades
2019-02-25 20:54:56 -08:00
Mirko Jotic f6a8e030cc Select query failing if miliseconds used as time for indexing (#6937)
* [#1332] Fix - select failing if milis used for idx.

* Formating correction.

* Address comment: throw original exception.

* Using constant values in tests

- Try converting to Integer and then multiply by 1000L to achieve milis.
- If not successful try converting to Long or rethrow original
exception.

* DateTime#of has to support "2011-01-01T00:00:00"

- in addition to seconds and milisecs, this method currently supports
even a date string.

* Handle only milisec timestamps and ISO8601 strings
2019-02-25 14:36:01 -08:00
Jihoon Son 9a066558a4 Fix exception when the scheme is missing in endpointUrl for S3 (#7129)
* Fix exception when the scheme is missing in endpointUrl for S3

* add null check
2019-02-25 11:10:35 -08:00
Himanshu Pandey 8b803cbc22 Added checkstyle for "Methods starting with Capital Letters" (#7118)
* Added checkstyle for "Methods starting with Capital Letters" and changed the method names violating this.

* Un-abbreviate the method names in the calcite tests

* Fixed checkstyle errors

* Changed asserts position in the code
2019-02-23 20:10:31 -08:00
David Glasser 1c2753ab90 ParallelIndexSubTask: support ingestSegment in delegating factories (#7089)
IndexTask had special-cased code to properly send a TaskToolbox to a
IngestSegmentFirehoseFactory that's nested inside a CombiningFirehoseFactory,
but ParallelIndexSubTask didn't.

This change refactors IngestSegmentFirehoseFactory so that it doesn't need a
TaskToolbox; it instead gets a CoordinatorClient and a SegmentLoaderFactory
directly injected into it.

This also refactors SegmentLoaderFactory so it doesn't depend on
an injectable SegmentLoaderConfig, since its only method always
replaces the preconfigured SegmentLoaderConfig anyway.
This makes it possible to use SegmentLoaderFactory without setting
druid.segmentCaches.locations to some dummy value.

Another goal of this PR is to make it possible for IngestSegmentFirehoseFactory
to list data segments outside of connect() --- specifically, to make it a
FiniteFirehoseFactory which can query the coordinator in order to calculate its
splits. See #7048.

This also adds missing datasource name URL-encoding to an API used by
CoordinatorBasedSegmentHandoffNotifier.
2019-02-23 17:02:56 -08:00
Jonathan Wei 417b9f2fe1 Add bug report and feature request GitHub issue templates (#7105)
* Add bug report GitHub template

* PR comments

* Add feature request template

* Tweak

* Add [REQUEST] title

* Remove request title, add note
2019-02-21 08:44:41 -08:00
Jihoon Son 4e2b085201
Remove DataSegmentFinder, InsertSegmentToDb, and descriptor.json file in deep storage (#6911)
* Remove DataSegmentFinder, InsertSegmentToDb, and descriptor.json file

* delete descriptor.file when killing segments

* fix test

* Add doc for ha

* improve warning
2019-02-20 15:10:29 -08:00
Mingming Qiu dd34691004 Coordinator await initialization before finishing startup (#6847)
* Curator server inventory await initialization

* address comments

* print exception object in log

* remove throws ISE

* cachingCost awaitInitialization default to false
2019-02-20 11:56:23 -08:00
David Glasser a81b1b8c9c index_parallel: support !appendToExisting with no explicit intervals (#7046)
* index_parallel: support !appendToExisting with no explicit intervals

This enables ParallelIndexSupervisorTask to dynamically request locks at runtime
if it is run without explicit intervals in the granularity spec and with
appendToExisting set to false.  Previously, it behaved as if appendToExisting
was set to true, which was undocumented and inconsistent with IndexTask and
Hadoop indexing.

Also, when ParallelIndexSupervisorTask allocates segments in the explicit
interval case, fail if its locks on the interval have been revoked.

Also make a few other additions/clarifications to native ingestion docs.

Fixes #6989.

* Review feedback.

PR description on GitHub updated to match.

* Make native batch ingestion partitions start at 0

* Fix to previous commit

* Unit test. Verified to fail without the other commits on this branch.

* Another round of review

* Slightly scarier warning
2019-02-20 10:54:26 -08:00
Furkan KAMACI 9a521526c7 Since notify might not wake up the right thread, notifyAll should be used instead. (#6931)
* Since notify might not wake up the right thread, notifyAll should be used instead.

* Comment is added about why notifyAll() is not used.
2019-02-20 09:02:58 -08:00
Justin Borromeo 871b9d2f4c [Benchmarking] Call blackhole#consume() on collections instead of iterating through each element (#7002)
* Replaced iteration with blackhole#consume(the collection)

* Added javadoc on Sequence#toList()
2019-02-20 08:48:06 -08:00
Dylan Wylie 554b0142c3 Autoclose old PRs using stale bot. (#7031)
* Autoclose old PRs using stale bot.

* add apache license

* Excempt bug label
2019-02-19 14:26:54 -08:00
Fangyuan Deng 7d1e8f353e bugfix: when building materialized-view, if taskCount>1, may cause concurrentModificationException (#6690)
* bugfix: when building materialized-view, if taskCount >1, may cause ConcurrentModificationException

* remove entry after iteration instead of using ConcurrentMap, and add unit test

* small change

* modify unit test for coverage

* remove unused method
2019-02-19 13:10:55 -08:00
Jonathan Wei 258485a2fb
Exclude github issue templates from license check (#7070)
* Exclude github issue templates from license check

* Adjust capitalization
2019-02-19 12:38:52 -08:00
Surekha 2b04e6d0bc add note on consistency of results for sys.segments queries (#7034)
* add doc

* change docs

* PR comments

* few more changes
2019-02-19 10:52:37 -08:00
Clint Wylie cadb6c5280 Missing Overlord and MiddleManager api docs (#7042)
* document middle manager api

* re-arrange

* correction

* document more missing overlord api calls, minor re-arrange of some code i was referencing

* fix it

* this will fix it

* fixup

* link to other docs
2019-02-19 10:52:05 -08:00
Surekha 80a2ef7be4 Support kafka transactional topics (#5404) (#6496)
* Support kafka transactional topics

* update kafka to version 2.0.0
* Remove the skipOffsetGaps option since it's not used anymore
* Adjust kafka consumer to use transactional semantics
* Update tests

* Remove unused import from test

* Fix compilation

* Invoke transaction api to fix a unit test

* temporary modification of travis.yml for debugging

* another attempt to get travis tasklogs

* update kafka to 2.0.1 at all places

* Remove druid-kafka-eight dependency from integration-tests, remove the kafka firehose test and deprecate kafka-eight classes

* Add deprecated in docs for kafka-eight and kafka-simple extensions

* Remove skipOffsetGaps and code changes for transaction support

* Fix indentation

* remove skipOffsetGaps from kinesis

* Add transaction api to KafkaRecordSupplierTest

* Fix indent

* Fix test

* update kafka version to 2.1.0
2019-02-18 11:50:08 -08:00
Jonathan Wei 61272d6daa Update handlebars dep to patch vulnerability (#7083) 2019-02-18 18:06:47 +08:00
Justin Borromeo c7eeeabf45 2528 Replace Incremental Index Global Flags with Getters (#7043)
* Eliminated reportParseExceptions and deserializeComplexMetrics

* Removed more global flags

* Cleanup

* Addressed Surekha's recommendations
2019-02-15 13:36:46 -08:00
scrawfor 0fa9000849 Add Postgresql SqlFirehose (#6813)
* Add Postgresql SqlFirehose

* Fix Code Style.

* Fix style.

* Fix Import Order.

* Add Line Break before package.
2019-02-14 22:52:03 -08:00
awelsh93 ee91e27fe7 Update api-reference.md doc (#7065)
- moving description of coordinator isLeader endpoint
2019-02-14 14:38:09 +00:00
Jonathan Wei 1f29940811
Fix momentsketch build issues (#7074)
* Fix momentsketch build issues

* Remove unused section in pom

* Fix test

* Remove unused method

* Checkstyle
2019-02-13 21:32:43 -08:00
Edward Gan 90c1a54b86 Moments Sketch custom aggregator (#6581)
* Moments Sketch Integration with Druid

* updates, add documentation, fix warnings

* nits

* disallowed base64

* update to druid 0.14
2019-02-13 14:03:47 -08:00
Jonathan Wei 673396ae74 Add proposal template (#7062)
* Add proposal template

Adds a proposal template based on the discussion in https://lists.apache.org/thread.html/bb9c5e1f8ce9b3148a5c26f95059f9b6629fae3bf8c617121d671395@%3Cdev.druid.apache.org%3E

* Add license
2019-02-13 13:43:31 -08:00
Jihoon Son 970308463d
Add doc for Hadoop-based ingestion vs Native batch ingestion (#7044)
* Add doc for Hadoop-based ingestion vs Native batch ingestion

* add links

* add links
2019-02-13 11:23:08 -08:00
Jihoon Son 1701fbcad3
Improve error message for revoked locks (#7035)
* Improve error message for revoked locks

* fix test

* fix test

* fix test

* fix toString
2019-02-13 11:22:48 -08:00
Jihoon Son b1c4a5de0d
Fix and improve doc for partitioning of local index (#7064) 2019-02-13 11:20:52 -08:00
Mingming Qiu d0abf5c20a fix kafka index task doesn't resume when recieve duplicate request (#6990)
* fix kafka index task doesn't resume when recieve duplicate request

* add unit test
2019-02-12 13:24:28 -08:00
Jonathan Wei 8ba11591b6 Add router conf to assembly.xml (#7051) 2019-02-12 10:33:18 +08:00
Surekha 02ef14f262 Fix num_rows in sys.segments (#6888)
* Fix the bug with num_rows in sys.segments

* Fix segmentMetadataInfo update in DruidSchema
* Add numRows to SegmentMetadataHolder builder's constructor, so it's not overwritten
* Rename SegSegmentSignature to setSegmentMetadataHolder and fix it so nested map is appended instead of recreated
* Replace Map<String, Set<String>> segmentServerMap with Set<String> for num_replica

* Remove unnecessary code and update test

* Add unit test for num_rows

* PR comments

* change access modifier to default package level

* minor changes to comments

* PR comments
2019-02-11 16:21:19 -08:00