Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update Kinesis resharding information about task failures (#9104)

This commit is contained in:
Jonathan Wei 2020-01-07 17:44:48 -06:00 committed by Clint Wylie
parent f540216931
commit d1500c1328
1 changed files with 4 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
docs/development/extensions-core/kinesis-ingestion.md
View File

@ -443,11 +443,12 @@ Then when submitting a supervisor-spec, set `deaggregate` to true.
## Resharding
When changing the shard count for a Kinesis stream, there will be a window of time around the resharding operation with early shutdown of Kinesis ingestion tasks.
This occurs because the supervisor will update the shard -> task group mappings as shards are closed and fully read, to ensure that tasks are not running
When changing the shard count for a Kinesis stream, there will be a window of time around the resharding operation with early shutdown of Kinesis ingestion tasks and possible task failures.
The early shutdowns and task failures are expected, and they occur because the supervisor will update the shard -> task group mappings as shards are closed and fully read, to ensure that tasks are not running
with an assignment of closed shards that have been fully read and to ensure a balanced distribution of active shards across tasks.
This window with early task shutdowns will conclude when:
This window with early task shutdowns and possible task failures will conclude when:
- All closed shards have been fully read and the Kinesis ingestion tasks have published the data from those shards, committing the "closed" state to metadata storage
- Any remaining tasks that had inactive shards in the assignment have been shutdown (these tasks would have been created before the closed shards were completely drained)