druid/docs
Pramod Immaneni 59bca0951a
Parallelize storage of incremental segments (#13982)
During ingestion, incremental segments are created in memory for the different time chunks and persisted to disk when certain thresholds are reached (max number of rows, max memory, incremental persist period etc). In the case where there are a lot of dimension and metrics (1000+) it was observed that the creation/serialization of incremental segment file format for persistence and persisting the file took a while and it was blocking ingestion of new data. This affected the real-time ingestion. This serialization and persistence can be parallelized across the different time chunks. This update aims to do that.

The patch adds a simple configuration parameter to the ingestion tuning configuration to specify number of persistence threads. The default value is 1 if it not specified which makes it the same as it is today.
2024-02-07 10:43:05 +05:30
..
api-reference Remove documentation for unused segments retrieval API (#15721) 2024-01-18 18:25:48 -08:00
assets Add SpectatorHistogram extension (#15340) 2024-01-14 09:52:30 -08:00
comparisons remove ref to plywood repo (#12809) 2022-07-26 10:12:13 +08:00
configuration Disable eager initialization for non-query connection requests (#15751) 2024-01-25 14:38:50 +05:30
data-management docs: revise concurrent append and replace (#15760) 2024-02-01 11:03:36 -08:00
design docs: Fix broken anchor links (#15621) 2024-01-03 15:28:27 -08:00
development Parallelize storage of incremental segments (#13982) 2024-02-07 10:43:05 +05:30
ingestion Add range filtering support for iceberg ingestion (#15782) 2024-02-01 23:32:30 -08:00
misc Update Ingestion section (#14023) 2023-05-19 09:42:27 -07:00
multi-stage-query Update the docs for EARLIEST_BY/LATEST_BY aggregators with the newly added numeric capabilities (#15670) 2024-02-01 10:24:43 +05:30
operations docs: clean up some rolling updates stuff (#15762) 2024-01-26 14:10:53 -08:00
querying docs: fix rendering (#15835) 2024-02-06 07:18:43 -08:00
release-info docs: Anchor link checker (#15624) 2024-01-08 15:19:05 -08:00
tutorials Doc fixes for query from deep storage and MSQ (#15313) 2023-11-03 10:52:20 +05:30