They cause services to create a brand-new startup injector and reload all
modules whenever /status is requested. Not sure how bad this is, but I think
we're only supposed to have one startup injector so there may be a better
approach out there.
- Workers are not added to zkWorkers until caches have been initialized.
- Worker status we haven't heard about will be added to runningTasks or
completeTasks as appropriate.
- TaskRunnerWorkItem now only needs a taskId, not the entire Task. This makes
it possible to create them from TaskStatus objects, if that's all we have.
- Also remove some dead code.
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.
Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).
Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
