Currently, if a Response has an ArrayBuffer body and text() is called, Angular
attempts to convert the ArrayBuffer to a string. Doing this requires knowing
the encoding of the bytes in the buffer, which is context that we don't have.
Instead, we assume that the buffer is encoded in UTF-16, and attempt to process
it that way. Unfortunately the approach chosen (interpret buffer as Uint16Array and
create a Javascript string from each entry using String.fromCharCode) is incorrect
as it does not handle UTF-16 surrogate pairs. What Angular actually implements, then,
is UCS-2 decoding, which is equivalent to UTF-16 with characters restricted to the
base plane.
No standard way of decoding UTF-8 or UTF-16 exists in the browser today. APIs like
TextDecoder are only supported in a few browsers, and although hacks like using the
FileReader API with a Blob to force browsers to do content encoding detection and
decoding exist, they're slow and not compatible with the synchronous text() API.
Thus, this bug is fixed by introducing an encodingHint parameter to text(). The
default value of this parameter is 'legacy', indicating that the existing broken
behavior should be used - this prevents breaking existing apps. The only other
possible value of the hint is 'iso-8859' which interprets each byte of the buffer
with String.fromCharCode. UTF-8 and UTF-16 are not supported - it is up to the
consumer to get the ArrayBuffer and decode it themselves.
The parameter is a hint, as it's not always used (for example, if the conversion
to text doesn't involve an ArrayBuffer source). Additionally, this leaves the door
open for future implementations to perform more sophisticated encoding detection
and ignore the user-provided value if it can be proven to be incorrect.
Fixes#15932.
PR Close#16420
7ae7a8440c