mirror of https://github.com/apache/druid.git
59bca0951a
During ingestion, incremental segments are created in memory for the different time chunks and persisted to disk when certain thresholds are reached (max number of rows, max memory, incremental persist period etc). In the case where there are a lot of dimension and metrics (1000+) it was observed that the creation/serialization of incremental segment file format for persistence and persisting the file took a while and it was blocking ingestion of new data. This affected the real-time ingestion. This serialization and persistence can be parallelized across the different time chunks. This update aims to do that. The patch adds a simple configuration parameter to the ingestion tuning configuration to specify number of persistence threads. The default value is 1 if it not specified which makes it the same as it is today. |
||
|---|---|---|
| .. | ||
| avro-extensions | ||
| azure-extensions | ||
| datasketches | ||
| druid-aws-rds-extensions | ||
| druid-basic-security | ||
| druid-bloom-filter | ||
| druid-catalog | ||
| druid-kerberos | ||
| druid-pac4j | ||
| druid-ranger-security | ||
| ec2-extensions | ||
| google-extensions | ||
| hdfs-storage | ||
| histogram | ||
| kafka-extraction-namespace | ||
| kafka-indexing-service | ||
| kinesis-indexing-service | ||
| kubernetes-extensions | ||
| lookups-cached-global | ||
| lookups-cached-single | ||
| multi-stage-query | ||
| mysql-metadata-storage | ||
| orc-extensions | ||
| parquet-extensions | ||
| postgresql-metadata-storage | ||
| protobuf-extensions | ||
| s3-extensions | ||
| simple-client-sslcontext | ||
| stats | ||
| testing-tools | ||