Commit Graph

12066 Commits

Author SHA1 Message Date
Paul Rogers 8ad8582dc8
Refactor DruidSchema & DruidTable (#12835)
Refactors the DruidSchema and DruidTable abstractions to prepare for the Druid Catalog.

As we add the catalog, we’ll want to combine physical segment metadata information with “hints” provided by the catalog. This is best done if we tidy up the existing code to more clearly separate responsibilities.

This PR is purely a refactoring move: no functionality changed. There is no difference to user functionality or external APIs. Functionality changes will come later as we add the catalog itself.

DruidSchema
In the present code, DruidSchema does three tasks:

Holds the segment metadata cache
Interfaces with an external schema manager
Acts as a schema to Calcite
This PR splits those responsibilities.

DruidSchema holds the Calcite schema for the druid namespace, combining information fro the segment metadata cache, from the external schema manager and (later) from the catalog.
SegmentMetadataCache holds the segment metadata cache formerly in DruidSchema.
DruidTable
The present DruidTable class is a bit of a kitchen sink: it holds all the various kinds of tables which Druid supports, and uses if-statements to handle behavior that differs between types. Yet, any given DruidTable will handle only one such table type. To more clearly model the actual table types, we split DruidTable into several classes:

DruidTable becomes an abstract base class to hold Druid-specific methods.
DatasourceTable represents a datasource.
ExternalTable represents an external table, such as from EXTERN or (later) from the catalog.
InlineTable represents the internal case in which we attach data directly to a table.
LookupTable represents Druid’s lookup table mechanism.
The new subclasses are more focused: they can be selective about the data they hold and the various predicates since they represent just one table type. This will be important as the catalog information will differ depending on table type and the new structure makes adding that logic cleaner.

DatasourceMetadata
Previously, the DruidSchema segment cache would work with DruidTable objects. With the catalog, we need a layer between the segment metadata and the table as presented to Calcite. To fix this, the new SegmentMetadataCache class uses a new DatasourceMetadata class as its cache entry to hold only the “physical” segment metadata information: it is up to the DruidTable to combine this with the catalog information in a later PR.

More Efficient Table Resolution
Calcite provides a convenient base class for schema objects: AbstractSchema. However, this class is a bit too convenient: all we have to do is provide a map of tables and Calcite does the rest. This means that, to resolve any single datasource, say, foo, we need to cache segment metadata, external schema information, and catalog information for all tables. Just so Calcite can do a map lookup.

There is nothing special about AbstractSchema. We can handle table lookups ourselves. The new AbstractTableSchema does this. In fact, all the rest of Calcite wants is to resolve individual tables by name, and, for commands we don’t use, to provide a list of table names.

DruidSchema now extends AbstractTableSchema. SegmentMetadataCache resolves individual tables (and provides table names.)

DruidSchemaManager
DruidSchemaManager provides a way to specify table schemas externally. In this sense, it is similar to the catalog, but only for datasources. It originally followed the AbstractSchema pattern: it implements provide a map of tables. This PR provides new optional methods for the table lookup and table names operations. The default implementations work the same way that AbstractSchema works: we get the entire map and pick out the information we need. Extensions that use this API should be revised to support the individual operations instead. Druid code no longer calls the original getTables() method.

The PR has one breaking change: since the DruidSchemaManager map is read-only to the rest of Druid, we should return a Map, not a ConcurrentMap.
2022-08-10 10:24:04 +05:30
Clint Wylie ee41cc770f
fix issue with SQL sum aggregator due to bug with DruidTypeSystem and AggregateRemoveRule (#12880)
* fix issue with SQL sum aggregator due to bug with DruidTypeSystem and AggregateRemoveRule

* fix style

* add comment about using custom sum function
2022-08-09 15:17:45 -07:00
David Palmer 2855fb6ff8
Change Kafka Lookup Extractor to not register consumer group (#12842)
* change kafka lookups module to not commit offsets

The current behaviour of the Kafka lookup extractor is to not commit
offsets by assigning a unique ID to the consumer group and setting
auto.offset.reset to earliest. This does the job but also pollutes the
Kafka broker with a bunch of "ghost" consumer groups that will never again be
used.

To fix this, we now set enable.auto.commit to false, which prevents the
ghost consumer groups being created in the first place.

* update docs to include new enable.auto.commit setting behaviour

* update kafka-lookup-extractor documentation

Provide some additional detail on functionality and configuration.
Hopefully this will make it clearer how the extractor works for
developers who aren't so familiar with Kafka.

* add comments better explaining the logic of the code

* add spelling exceptions for kafka lookup docs
2022-08-09 16:14:22 +05:30
絵空事スピリット ebe783dbdc
Correct minor format issue (#12882) 2022-08-09 18:15:41 +08:00
Hamish Ball abd7a9748d
Remove kafka lookup records when a record is tombstoned (#12819)
* remove kafka lookup records from factory when record tombstoned

* update kafka lookup docs to include tombstone behaviour

* change test wait time down to 10ms

Co-authored-by: David Palmer <david.palmer@adscale.co.nz>
2022-08-09 10:42:51 +05:30
Clint Wylie a7e89de610
fix JsonNode leaking from JSON flattener (#12873)
* fix JsonNode leaking from JSON flattener

* adjustments
2022-08-08 19:51:57 -07:00
David Hergenroeder 533c39f35a
Fix rollup docs bullet formatting (#12876) 2022-08-09 10:10:07 +08:00
Suneet Saldanha 267b32c2e2
Set druid.processing.fifo to true by default (#12571) 2022-08-08 10:18:24 -07:00
Gian Merlino 01d555e47b
Adjust "in" filter null behavior to match "selector". (#12863)
* Adjust "in" filter null behavior to match "selector".

Now, both of them match numeric nulls if constructed with a "null" value.

This is consistent as far as native execution goes, but doesn't match
the behavior of SQL = and IN. So, to address that, this patch also
updates the docs to clarify that the native filters do match nulls.

This patch also updates the SQL docs to describe how Boolean logic is
handled in addition to how NULL values are handled.

Fixes #12856.

* Fix test.
2022-08-08 09:08:36 -07:00
Karan Kumar 607b0b9310
Adding withName implementation to AggregatorFactory (#12862)
* Adding agg factory with name impl

* Adding test cases

* Fixing test case

* Fixing test case

* Updated java docs.
2022-08-08 18:31:56 +05:30
Jonathan Wei 2045a1345c
Fix NPE when applying a transform that outputs to __time (#12870) 2022-08-07 19:21:47 +05:30
Herb Brewer 9f8982a9a6
fix(druid-indexing): failed to get shardSpec for interval issue (#12573) 2022-08-05 17:57:36 -07:00
AmatyaAvadhanula d294404924
Kinesis ingestion with empty shards (#12792)
Kinesis ingestion requires all shards to have at least 1 record at the required position in druid.
Even if this is satisified initially, resharding the stream can lead to empty intermediate shards. A significant delay in writing to newly created shards was also problematic.

Kinesis shard sequence numbers are big integers. Introduce two more custom sequence tokens UNREAD_TRIM_HORIZON and UNREAD_LATEST to indicate that a shard has not been read from and that it needs to be read from the start or the end respectively.
These values can be used to avoid the need to read at least one record to obtain a sequence number for ingesting a newly discovered shard.

If a record cannot be obtained immediately, use a marker to obtain the relevant shardIterator and use this shardIterator to obtain a valid sequence number. As long as a valid sequence number is not obtained, continue storing the token as the offset.

These tokens (UNREAD_TRIM_HORIZON and UNREAD_LATEST) are logically ordered to be earlier than any valid sequence number.

However, the ordering requires a few subtle changes to the existing mechanism for record sequence validation:

The sequence availability check ensures that the current offset is before the earliest available sequence in the shard. However, current token being an UNREAD token indicates that any sequence number in the shard is valid (despite the ordering)

Kinesis sequence numbers are inclusive i.e if current sequence == end sequence, there are more records left to read.
However, the equality check is exclusive when dealing with UNREAD tokens.
2022-08-05 22:38:58 +05:30
Adarsh Sanjeev 24f8f9e1ab
Add check for eternity time segment to SqlSegmentsMetadataQuery (#12844)
* Add check for eternity time segment to SqlSegmentsMetadataQuery

* Add check for half eternities

* Add multiple segments test

* Add failing test to document known issue
2022-08-04 22:33:08 -07:00
Gian Merlino ca4e64aea3
Frame processing and channels. (#12848)
* Frame processing and channels.

Follow-up to #12745. This patch adds three new concepts:

1) Frame channels are interfaces for doing nonblocking reads and writes
   of frames.

2) Frame processors are interfaces for doing nonblocking processing of
   frames received from input channels and sent to output channels.

3) Cluster-by keys, which can be used for sorting or partitioning.

The patch also adds SuperSorter, a user of these concepts, both to
illustrate how they are used, and also because it is going to be useful
in future work.

Central classes:

- ReadableFrameChannel. Implementations include
  BlockingQueueFrameChannel (in-memory channel that implements both interfaces),
  ReadableFileFrameChannel (file-based channel),
  ReadableByteChunksFrameChannel (byte-stream-based channel), and others.

- WritableFrameChannel. Implementations include BlockingQueueFrameChannel
  and WritableStreamFrameChannel (byte-stream-based channel).

- ClusterBy, a sorting or partitioning key.

- FrameProcessor, nonblocking processor of frames. Implementations include
  FrameChannelBatcher, FrameChannelMerger, and FrameChannelMuxer.

- FrameProcessorExecutor, an executor service that runs FrameProcessors.

- SuperSorter, a class that uses frame channels and processors to
  do parallel external merge sort of any amount of data (as long as there
  is enough disk space).

* Additional tests, fixes.

* Changes from review.

* Better implementation for ReadableInputStreamFrameChannel.

* Rename getFrameFileReference -> newFrameFileReference.

* Add InterruptedException to runIncrementally; add more tests.

* Cancellation adjustments.

* Review adjustments.

* Refactor BlockingQueueFrameChannel, rename doneReading and doneWriting to close.

* Additional changes from review.

* Additional changes.

* Fix test.

* Adjustments.

* Adjustments.
2022-08-04 21:29:04 -07:00
Katya Macedo c6dd9dd4af
Fix typo in compaction.md (#12774) 2022-08-04 14:47:22 -07:00
Clint Wylie 73cfc4e5d0
fix expression plan type inference to correctly handle complex types (#12857) 2022-08-04 02:56:05 -07:00
Paul Rogers a618458bf0
Tidy up construction of the Guice Injectors (#12816)
* Refactor Guice initialization

Builders for various module collections
Revise the extensions loader
Injector builders for server startup
Move Hadoop init to indexer
Clean up server node role filtering
Calcite test injector builder

* Revisions from review comments

* Build fixes

* Revisions from review comments
2022-08-04 00:05:07 -07:00
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Clint Wylie 623b075d12
fix nested column sql operator return type inference (#12851)
* fix nested column sql operator return type inference

* oops, final
2022-08-03 15:39:08 -07:00
刘小辉 6f5c1434b8
fix get task may be null (#12100) 2022-08-03 09:23:48 -07:00
AmatyaAvadhanula fbd1a07e7e
Fix kinesis IT flakiness (#12821) 2022-08-03 17:16:16 +05:30
Peter Marshall 0a4ed3ba61
Readme - link fix to build guide (#12849) 2022-08-03 19:32:37 +08:00
Karan Kumar 3290b49754
Log4j bump to 2.18 due to [LOG4J2-3419] (#12847)
* Log4j bump to 2.18 due to [LOG4J2-3419]

* Fixing license issues
2022-08-02 23:25:40 -07:00
Gian Merlino 2912a36a20
Use nonzero default value of maxQueuedBytes. (#12840)
* Use nonzero default value of maxQueuedBytes.

The purpose of this parameter is to prevent the Broker from running out
of memory. The prior default is unlimited; this patch changes it to a
relatively conservative 25MB.

This may be too low for larger clusters. The risk is that throughput
can decrease for queries with large resultsets or large amounts of intermediate
data. However, I think this is better than the risk of the prior default, which
is that these queries can cause the Broker to go OOM.

* Alter calculation.
2022-08-02 17:57:27 -07:00
Gian Merlino 0ca37c20a6
Python 3 support for post-index-task. (#12841)
* Python 3 support for post-index-task.

Useful when running on macOS or any other system that
doesn't have Python 2.

* Encode JSON returned by read_task_file.

* Adjust.

* Skip needless loads.

* Add a decode.

* Additional decodes needed.
2022-08-02 17:53:34 -07:00
Clint Wylie 6981b1cc12
fix bugs with nested column jsonpath parser (#12831) 2022-08-02 11:38:25 -07:00
Rohan Garg eabce8a159
Fix flakiness in query-retry ITs (#12818) 2022-08-02 17:20:16 +05:30
Tejaswini Bandlamudi cceb2e849e
Perform lazy initialization of parquet extensions module (#12827)
Historicals and middle managers crash with an `UnknownHostException` on trying
to load `druid-parquet-extensions` with an ephemeral Hadoop cluster. This happens
because the `fs.defaultFS` URI value cannot be resolved at start up time as the
hadoop cluster may not exist at startup time.

This commit fixes the error by performing initialization of the filesystem in
`ParquetInputFormat.createReader()` whenever a new reader is requested.
2022-08-02 13:41:12 +05:30
Clint Wylie 6046a392b6
add DictionaryEncodedStringValueIndex implementation to NestedFieldLiteralColumnIndexSupplier (#12837) 2022-08-01 21:40:35 -07:00
Rohan Garg 7ae6cc6e60
Fix string first/last aggregator comparator (#12773) 2022-08-01 20:54:15 +05:30
317brian 553ff47616
fix: fix broken link to Class TTest (#12836) 2022-07-31 10:18:14 +08:00
Clint Wylie d96a9c1e6f
add missing selectors for explicit null columns (#12834) 2022-07-29 19:08:58 -07:00
Clint Wylie 189e8b9d18
add NumericRangeIndex interface and BoundFilter support (#12830)
add NumericRangeIndex interface and BoundFilter support
changes:
* NumericRangeIndex interface, like LexicographicalRangeIndex but for numbers
* BoundFilter now uses NumericRangeIndex if comparator is numeric and there is no extractionFn
* NestedFieldLiteralColumnIndexSupplier.java now supports supplying NumericRangeIndex for single typed numeric nested literal columns

* better faster stronger and (ever so slightly) more understandable

* more tests, fix bug

* fix style
2022-07-29 18:58:49 -07:00
Paul Rogers d52abe7b38
Today is that day - Single pass through Calcite planner (#12636)
* Druid planner now makes only one pass through Calcite planner

Resolves the issue that required two parse/plan cycles: one
for validate, another for plan. Creates a clone of the Calcite
planner and validator to resolve the conflict that prevented
the merger.
2022-07-29 18:53:21 -07:00
Charles Smith efbb58e90e
docs: remove maxRowsPerSegment where appropriate (#12071)
* remove maxRowsPerSegment where appropriate

* fix tutorial, accept suggestions

* Update docs/design/coordinator.md

* additional tutorial file

* fix initial index spec

* accept comments

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* add back comment on maxrows per segment

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* Update docs/tutorials/tutorial-compaction.md

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>

* rm duplicate entry

* Update native-batch-simple-task.md

remove ref to `maxrowspersegment`

* Update native-batch.md

remove ref to `maxrowspersegment`

* final tenticles

* Apply suggestions from code review

Co-authored-by: Victoria Lim <vtlim@users.noreply.github.com>
2022-07-28 16:52:13 +05:30
Maytas Monsereenusorn 24c345cdf0
Allow dictionary encoded column to use a more generic index interface (#12826) 2022-07-27 15:23:00 -07:00
Paul Rogers a8b155e9c6
Fixes for the Avatica JDBC driver (#12709)
* Fixes for the Avatica JDBC driver

Correctly implement regular and prepared statements
Correctly implement result sets
Fix race condition with contexts
Clarify when parameters are used
Prepare for single-pass through the planner

* Addressed review comments

* Addressed review comment
2022-07-27 15:22:40 -07:00
Atul Mohan 93a9a4b1c5
Add retention for file request logs (#12559)
* Add retention for file request logs

* Spelling
2022-07-27 08:17:02 -07:00
Rohan Garg bf0886a8ab
Fix hash calcuation in RendezvousHasher (#12817) 2022-07-27 12:16:27 +05:30
Jacques Arnoux 6b0b1d7af3
replaces hard-coded probe delays with helm values (#12805) 2022-07-26 14:04:06 +05:30
Laksh Singla 2e616e633a
Determine type of `__time` column by RowSignature in case of External Datasource (#12770)
Some queries like `REPLACE INTO ... SELECT TIME_PARSE("__time") AS __time FROM ...`
fail at the Calcite layer because any column with name `__time` is considered to be of
type `SqlTypeName.TIMESTAMP`.

Changes:
- Modify `RowSignatures.toRelDataType()` so that the type of `__time` column
  is determined by the RowSignature's type.
2022-07-26 12:09:40 +05:30
Charles Smith d7d4314367
remove ref to plywood repo (#12809) 2022-07-26 10:12:13 +08:00
PJ Fanning 188b5b0027
Upgrade to jetty 9.4.48.v20220622 due to CVEs (#12801)
* Upgrade to jetty 9.4.48.v20220622 due to CVEs

* Update licenses.yaml
2022-07-26 10:11:48 +08:00
Tejaswini Bandlamudi 5772dfd155
Peons should not report SysMonitor stats since MiddleManager reports them. (#12802)
Sysmonitor stats (mem, fs, disk, net, cpu, swap, sys, tcp) are reported by all Druid processes, including Peons that are ephemeral in nature. Since Peons always run on the same host as the MiddleManager that spawned them and is unlikely to change, the SyMonitor metrics emitted by Peon are merely duplicates. This is often not a problem except when machines are super-beefy. Imagine a 64-core machine and 32 workers running on this machine. now you will have each Peon reporting metrics for each core. that's an increase of (32 * 64)x in the number of metrics. This leads to a metric explosion.

This PR updates MetricsModule to check node role running while registering SysMonitor and not to load any existing SysMonitor$Stats.
2022-07-23 13:32:16 +05:30
Victoria Lim 6394ecfd21
update figure and reference (#12813) 2022-07-22 15:54:25 -07:00
Maytas Monsereenusorn 5417aa2055
Fix: ParseException swallow cause Exception (#12810)
* add impl

* add impl

* fix checkstyle
2022-07-22 13:46:28 -07:00
Kashif Faraz 6c96d09680
Suppress some false alarm CVEs (#12812)
This commit suppresses the following CVEs:
- CVE-2021-43138: false alarm for async-http-client
- CVE-2021-34538: applicable to Hive server
- CVE-2020-25638: requires hibernate update, which causes Hadoop ingestion failure
- CVE-2021-27568: false alarm for accessors-smart which is a dependency of json-smart (already suppressed)
2022-07-22 22:27:31 +05:30
Kashif Faraz 9e5f0109fd
Fix CVE-2022-2048 (jetty) and CVE-2022-31159 (aws-java-sdk-s3) (#12807)
Changes:
- Upgrade aws sdk version from `1.12.37` to `1.12.264`
- Upgrade jetty version from `9.4.41.v20210516` to `9.4.47.v20220610`
2022-07-21 13:08:18 +05:30
Katya Macedo a2be685824
Remove the time bit, fix headings (#12808)
* Remove the time bit, fix headings

* Adopt review suggestions

* Edits

* Update smoosh file description

* Adopt review suggestions

* Update spelling
2022-07-20 15:37:57 -07:00