In the past, only the starting index of an expression Token has been
recorded, so a parser could demarkate the span of a token only by the
start locations of two tokens. This may lead to trailing whitespace
being included in the token span:
```html
{{ token1 + token2 }}
^^^^^^^^^ recorded span of `token1`
```
It's also not enough for a parser to determine the end of a token by
adding the length of the token value to the token's start location,
because lexed expression values may not exactly reflect the source code.
For example, `"d\\"e"` is lexed as a string token whose value is `d"e`.
Instead, this commit adds a `end` field to expression tokens. `end`
is one past the last index of the token source code. This will enable a
parser to determine the span of a token just by looking at that token.
This is a breaking change because the contructor interface of `Token`
has changed.
Part of #33477.
PR Close#33549
Joey Perrott
Alex Rickabaugh
ayazhafiz
Misko Hevery
Jason Aden