d294404924
Kinesis ingestion requires all shards to have at least 1 record at the required position in druid. Even if this is satisified initially, resharding the stream can lead to empty intermediate shards. A significant delay in writing to newly created shards was also problematic. Kinesis shard sequence numbers are big integers. Introduce two more custom sequence tokens UNREAD_TRIM_HORIZON and UNREAD_LATEST to indicate that a shard has not been read from and that it needs to be read from the start or the end respectively. These values can be used to avoid the need to read at least one record to obtain a sequence number for ingesting a newly discovered shard. If a record cannot be obtained immediately, use a marker to obtain the relevant shardIterator and use this shardIterator to obtain a valid sequence number. As long as a valid sequence number is not obtained, continue storing the token as the offset. These tokens (UNREAD_TRIM_HORIZON and UNREAD_LATEST) are logically ordered to be earlier than any valid sequence number. However, the ordering requires a few subtle changes to the existing mechanism for record sequence validation: The sequence availability check ensures that the current offset is before the earliest available sequence in the shard. However, current token being an UNREAD token indicates that any sequence number in the shard is valid (despite the ordering) Kinesis sequence numbers are inclusive i.e if current sequence == end sequence, there are more records left to read. However, the equality check is exclusive when dealing with UNREAD tokens. |
||
|---|---|---|
| .github | ||
| .idea | ||
| benchmarks | ||
| cloud | ||
| codestyle | ||
| core | ||
| dev | ||
| distribution | ||
| docs | ||
| examples | ||
| extendedset | ||
| extensions-contrib | ||
| extensions-core | ||
| helm/druid | ||
| hll | ||
| hooks | ||
| indexing-hadoop | ||
| indexing-service | ||
| integration-tests | ||
| licenses | ||
| processing | ||
| publications | ||
| server | ||
| services | ||
| sql | ||
| web-console | ||
| website | ||
| .asf.yaml | ||
| .backportrc.json | ||
| .codecov.yml | ||
| .dockerignore | ||
| .gitignore | ||
| .lgtm.yml | ||
| .travis.yml | ||
| CONTRIBUTING.md | ||
| LABELS | ||
| LICENSE | ||
| NOTICE | ||
| README.md | ||
| README.template | ||
| check_test_suite.py | ||
| check_test_suite_test.py | ||
| licenses.yaml | ||
| owasp-dependency-check-suppressions.xml | ||
| pom.xml | ||
| upload.sh | ||
README.md
Website | Documentation | Developer Mailing List | User Mailing List | Slack | Twitter | Download
Apache Druid
Druid is a high performance real-time analytics database. Druid's main value add is to reduce time to insight and action.
Druid is designed for workflows where fast queries and ingest really matter. Druid excels at powering UIs, running operational (ad-hoc) queries, or handling high concurrency. Consider Druid as an open source alternative to data warehouses for a variety of use cases. The design documentation explains the key concepts.
Getting started
You can get started with Druid with our local or Docker quickstart.
Druid provides a rich set of APIs (via HTTP and JDBC) for loading, managing, and querying your data. You can also interact with Druid via the built-in console (shown below).
Load data
Load streaming and batch data using a point-and-click wizard to guide you through ingestion setup. Monitor one off tasks and ingestion supervisors.
Manage the cluster
Manage your cluster with ease. Get a view of your datasources, segments, ingestion tasks, and services from one convenient location. All powered by SQL systems tables, allowing you to see the underlying query for each view.
Issue queries
Use the built-in query workbench to prototype DruidSQL and native queries or connect one of the many tools that help you make the most out of Druid.
Documentation
You can find the documentation for the latest Druid release on the project website.
If you would like to contribute documentation, please do so under
/docs in this repository and submit a pull request.
Community
Community support is available on the druid-user mailing list, which is hosted at Google Groups.
Development discussions occur on dev@druid.apache.org, which you can subscribe to by emailing dev-subscribe@druid.apache.org.
Chat with Druid committers and users in real-time on the Apache Druid Slack channel. Please use this invitation link to join and invite others.
Building from source
Please note that JDK 8 or JDK 11 is required to build Druid.
See the latest build guide for instructions on building Apache Druid from source.
Contributing
Please follow the community guidelines for contributing.
For instructions on setting up IntelliJ dev/intellij-setup.md