mirror of https://github.com/apache/druid.git
d6949b1b79
Follow up to #13520 Bytes processed are currently tracked for intermediate stages in MSQ ingestion. This patch adds the capability to track the bytes processed by an MSQ controller task while reading from an external input source or a segment source. Changes: - Track `processedBytes` for every `InputSource` read in `ExternalInputSliceReader` - Update `ChannelCounters` with the above obtained `processedBytes` when incrementing the input file count. - Update task report structure in docs The total input processed bytes can be obtained by summing the `processedBytes` as follows: totalBytes = 0 for every root stage (i.e. a stage which does not have another stage as an input): for every worker in that stage: for every input channel: (i.e. channels with prefix "input", e.g. "input0", "input1", etc.) totalBytes += processedBytes |
||
|---|---|---|
| .. | ||
| data-formats.md | ||
| data-model.md | ||
| faq.md | ||
| hadoop.md | ||
| index.md | ||
| ingestion-spec.md | ||
| migrate-from-firehose-ingestion.md | ||
| native-batch-firehose.md | ||
| native-batch-input-source.md | ||
| native-batch-simple-task.md | ||
| native-batch.md | ||
| partitioning.md | ||
| rollup.md | ||
| schema-design.md | ||
| standalone-realtime.md | ||
| tasks.md | ||
| tranquility.md | ||