Fixed WriteFlusher to distinguish between a flush that consumes all content and returns false, from one
that consumes all content and returns true.
If false is returned, the flusher needs to remain in pending so encrypted buffers can be flushed.
Made the SelectorManager use the CaS state machine for both locking and controlling
the mode of handling changes.
Replaced the concurrent change queue with a pair of array lists that are switched while the lock state is held
The state machine now simply tracks if the endpoint is selecting or has been selected.
The slight complexity is that any transition between these two states goes via a locked
state, where there is exclusive access to the interested ops and selection key.
It was possible that updateKey() was seeing a SELECTING state and
therefore attempt to call setKeyInterests(), while changeInterests()
was also seeing the SELECTING state, then moving to CHANGING so that
_interestOps was accessed concurrently.
Also made the update task to call updateKey() instead of calling
directly setKeyInterests(), in order to comply with the state
machine; this required to have onSelected() handle additional states
that are created by updateKey().
Finally, in updateKey() now setKeyInterests() is called before
updating the state to isolate the call into its own state.
Fixed IllegalStateException by handling NEED_UNWRAP for the CLOSED
state in fill().
The EOFException does not seem to be an issue with the client.
Also removed an unneeded catch block and an empty if statement.
WebSocket and multiplexed protocols are always read interested.
It may happen that while the application is writing, the write
blocks, resulting in a call to changeInterests().
At the same time, the selector may detect data to read and call
onSelected(), so there is a possibility that onSelected() runs
concurrently with changeInterests().
The fix adds an additional state (PROCESSING) that isolates the
changes that onSelected() makes to _interestOps, spin-waiting if
changeInterests() is running concurrently.
Likewise, changeInterests() spin-waits until onSelected() is running
concurrently.
The race was happening when updateKey() lost the race with
changeInterests() to update the interests and subsequently the key.
In that case, the selector thread that called updateKey() was returning
while changeInterests() was executing the CHANGING state.
Since updateKey() lost the race, the actual key interests was still
(typically) OP_READ so the selector thread would select() and call
onSelected() while changeInterests() was still executing, causing the
IllegalStateException.
Introduced parameter "dispatchIO" in the relevant factories so that
they can be configured by users and connections will be created
taking into account this parameter.
For less configurable connection factories, this parameter is
currently hardcoded to either true or false depending on the case.
For example, ALPN and NPN connections have it to false, since they
don't do any blocking operation in onFillable().
Previously, SCEP was skipping the update of the interestOps in case it
was either input or output shutdown.
Now it always set the interestOps as requested and leaves to the
connection decide what interestOps needs to be set.
Introduced a state machine to handle the various scenarios (ST = selector thread, Tx = pooled thread):
ST: call to SCEP.onSelected() moves from SELECTING -> PENDING.
ST: call to SCEP.updateKey() moves from PENDING -> UPDATING -> SELECTING
T1: call to SCEP.changeInterests() moves (SELECTING | PENDING) -> CHANGING -> SELECTING
The race between ST and T1 to move from PENDING to either UPDATING or CHANGING will be won
by one thread only, which will then perform the call to SelectionKey.setInterestOps().
Preferably, this will be done by ST during an updateKey() call. If updateKey() has already
been invoked, then changeInterests() will perform the call to SelectionKey.setInterestOps().
However, if T1 loses, it still has to perform the key update, so it will spin until ST
moves back to SELECTING.
