Apache Druid: a high performance real-time analytics database.
Go to file
Samarth Jain afbcb9c07f Improve parallelism of zookeeper based segment change processing (#7088)
* V1 - improve parallelism of zookeeper based segment change processing

* Create zk nodes in batches. Address code review comments.
Introduce various configs.

* Add documentation for the newly added configs

* Fix test failures

* Fix more test failures

* Remove prinstacktrace statements

* Address code review comments

* Use a single queue

* Address code review comments

Since we have a separate load peon for every historical, just having a single SegmentChangeProcessor
task per historical is enough. This commit also gets rid of the associated config druid.coordinator.loadqueuepeon.curator.numCreateThreads

* Resolve merge conflict

* Fix compilation failure

* Remove batching since we already have a dynamic config maxSegmentsInNodeLoadingQueue that provides that control

* Fix NPE in test

* Remove documentation for configs that are no longer needed

* Address code review comments

* Address more code review comments

* Fix checkstyle issue

* Address code review comments

* Code review comments

* Add back monitor node remove executor

* Cleanup code to isolate null checks  and minor refactoring

* Change param name since it conflicts with member variable name
2019-05-03 15:58:42 +02:00
.github Remove [PROPOSAL] from Proposal issue template; Add 'Other' template for convenience (#7364) 2019-03-28 18:40:43 +08:00
.idea Make JavaScript and XML errors non-TeamCity errors; Update JavaScript language level to ES6 in IntelliJ settings (#7541) 2019-04-25 11:21:58 -07:00
benchmarks Update scan benchmark for time ordering (#7385) 2019-03-30 11:36:25 -07:00
ci Move dev-related files and instructions to dev/ directory; add committer's instructions (#7279) 2019-04-17 15:27:14 +02:00
cloud Move GCP to a core extension (#6953) 2019-03-27 09:00:43 -07:00
codestyle Prohibit Throwables.propagate() (#7121) 2019-03-14 18:28:33 -03:00
core Data loader (sampler component) (#7531) 2019-05-01 22:37:14 -07:00
dev Move dev-related files and instructions to dev/ directory; add committer's instructions (#7279) 2019-04-17 15:27:14 +02:00
distribution Contributing Moving-Average Query to open source. (#6430) 2019-04-26 17:07:48 -07:00
docs Improve parallelism of zookeeper based segment change processing (#7088) 2019-05-03 15:58:42 +02:00
examples Deprecate NoneShardSpec and drop support for automatic segment merge (#6883) 2019-03-15 23:29:25 -07:00
extendedset Set version to 0.15.0-incubating-SNAPSHOT (#7014) 2019-02-07 14:02:52 -08:00
extensions-contrib Contributing Moving-Average Query to open source. (#6430) 2019-04-26 17:07:48 -07:00
extensions-core handle empty sketches (#7526) 2019-04-25 14:28:41 -07:00
hll Remove direct references to unsafe for Java 9+ 2019-04-30 14:21:45 -07:00
indexing-hadoop Pass in segmentTable correctly (#7492) 2019-04-17 20:07:22 -07:00
indexing-service Data loader (sampler component) (#7531) 2019-05-01 22:37:14 -07:00
integration-tests Adjust required permissions for system schema (#7579) 2019-05-02 07:18:02 -07:00
licenses Fix some bundled license files (#7516) 2019-04-19 16:07:44 -07:00
processing AggregatorFactory: Clarify methods that return other AggregatorFactories. (#7293) 2019-04-29 19:27:30 +02:00
publications add missing license headers, in particular to MD files; clean up RAT … (#6563) 2018-11-13 09:38:37 -08:00
server Improve parallelism of zookeeper based segment change processing (#7088) 2019-05-03 15:58:42 +02:00
services Improve parallelism of zookeeper based segment change processing (#7088) 2019-05-03 15:58:42 +02:00
sql Adjust required permissions for system schema (#7579) 2019-05-02 07:18:02 -07:00
web-console Add is_overshadowed column to segment table (#7555) 2019-05-01 12:28:53 -07:00
.dockerignore Add docker container for druid (#6896) 2019-02-08 12:12:28 +00:00
.gitignore Update LICENSE and NOTICE files (#7026) 2019-03-04 18:45:22 -08:00
.travis.yml add travis check to make sure -Pdist works (#7418) 2019-04-07 20:12:23 -07:00
CONTRIBUTING.md Move dev-related files and instructions to dev/ directory; add committer's instructions (#7279) 2019-04-17 15:27:14 +02:00
DISCLAIMER add missing license headers, in particular to MD files; clean up RAT … (#6563) 2018-11-13 09:38:37 -08:00
LABELS.md Update LICENSE and NOTICE files (#7026) 2019-03-04 18:45:22 -08:00
LICENSE Java 9 compatible ByteBuffer unmap operation (#7466) 2019-04-15 09:03:18 -07:00
LICENSE.BINARY fix LICENSE.BINARY to have correct datasketches version (#7557) 2019-04-25 22:00:13 -07:00
NOTICE Fix 0.14.0 rc1 voting issues (#7284) 2019-03-18 14:16:21 -07:00
NOTICE.BINARY Fix 0.14.0 rc1 voting issues (#7284) 2019-03-18 14:16:21 -07:00
README.md Fix 0.14.0 rc1 voting issues (#7284) 2019-03-18 14:16:21 -07:00
pom.xml Contributing Moving-Average Query to open source. (#6430) 2019-04-26 17:07:48 -07:00
upload.sh Adding licenses and enable apache-rat-plugin. (#6215) 2018-09-18 08:39:26 -07:00

README.md

Build Status Inspections Status Coverage Status IRC#druid-dev

Apache Druid (incubating)

Apache Druid (incubating) is a high performance analytics data store for event-driven data.

Disclaimer: Apache Druid is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.

License

Apache License, Version 2.0

More Information

More information about Druid can be found on http://www.druid.io.

Documentation

You can find the documentation for the latest Druid release on the project website.

If you would like to contribute documentation, please do so under /docs/content in this repository and submit a pull request.

Getting Started

You can get started with Druid with our quickstart.

Reporting Issues

If you find any bugs, please file a GitHub issue.

Community

The Druid community is in the process of migrating to Apache by way of the Apache Incubator. Eventually, as we proceed along this path, our site will move from http://druid.io/ to https://druid.apache.org/.

Community support is available on the druid-user mailing list(druid-user@googlegroups.com), which is hosted at Google Groups.

Development discussions occur on dev@druid.apache.org, which you can subscribe to by emailing dev-subscribe@druid.apache.org.

We also have a couple people hanging out on IRC in #druid-dev on irc.freenode.net.

Building From Source

Please note that JDK 8 is required to build Druid.

For instructions on building Druid from source, see docs/content/development/build.md

Contributing

Please follow the guidelines listed here.