This commit ensures that when leading whitespace is skipped by
the tokenizer, the original start location (before skipping) is captured
in the `fullStart` property of the token's source-span.
PR Close#39486
Let's say we have a code like
```html
<div<span>123</span>
```
Currently this gets parsed into a tree with the element tag `div<span`.
This has at least two downsides:
- An incorrect diagnostic that `</span>` doesn't close an element is
emitted.
- A consumer of the parse tree using it for editor services is unable to
provide correct completions for the opening `<span>` tag.
This patch attempts to fix both issues by instead parsing the code into
the same tree that would be parsed for `<div></div><span>123</span>`.
In particular, we do this by optimistically scanning an open tag as
usual, but if we do not notice a terminating '>', we mark the tag as
"incomplete". A parser then emits an error for the incomplete tag and
adds a synthetic (recovered) element node to the tree with the
incomplete open tag's name.
What's the downside of this? For one, a breaking change.
<ol>
<li>
The first breaking change is that `<` symbols that are ambiguously text
or opening tags will be parsed as opening tags instead of text in
element bodies. Take the code
```html
<p>a<b</p>
```
Clearly we cannot have the best of both worlds, and this patch chooses
to swap the parsing strategy to support the new feature. Of course, `<`
can still be inserted as text via the `<` entity.
</li>
</ol>
Part of #38596
PR Close#38681
Previously, the `sourceSpan` and `startSourceSpan` were the same
object, which meant that you had the following situation:
```
element = <div>some content</div>
sourceSpan = <div>
startSourceSpan = <div>
endSourceSpan = </div>
```
This made `sourceSpan` redundant and meant that if you
wanted a span for the whole element including its content
and closing tag, it had to be computed.
Now `sourceSpan` is separated from `startSourceSpan`
resulting in:
```
element = <div>some content</div>
sourceSpan = <div>some content</div>
startSourceSpan = <div>
endSourceSpan = </div>
```
PR Close#38581
Previously, the `startSourceSpan` property could be null
but in reality it is always well defined - except for a legacy
case in the old i18n extraction/merging code, where the
typings for source-spans are already being undermined.
Making this property non-null, simplifies code elsewhere
in the project.
PR Close#38581
Previously the lexer was responsible for deciding whether an "inline"
template should also have its line-endings normalized.
Now this decision is made higher up in the call stack to allow more
flexibility in the parser/lexer.
PR Close#38581
Within an angular template, when a character entity is unable to be parsed, previously a generic
unexpected character error was thrown. This does not properly express the issue that was discovered
as the issue is actually caused by the discovered character making the whole of the entity unparsable.
The compiler will now instead inform via the error message what string was attempted to be parsed
and what it was attempted to be parsed as.
Example, for this template:
```
<p>
ģp
</p>
```
Before this change:
`Unexpected character "p"`
After this change:
`Unable to parse entity "ģp" - hexadecimal character reference entities must end with ";"`
Fixes#26067
PR Close#38319
HTML is very lenient when it comes to closing elements, so Angular's parser has
rules that specify which elements are implicitly closed when closing a tag.
The parser keeps track of the nesting of tag names using a stack and parsing
a closing tag will pop as many elements off the stack as possible, provided
that the elements can be implicitly closed.
For example, consider the following templates:
- `<div><br></div>`, the `<br>` is implicitly closed when parsing `</div>`,
because `<br>` is a void element.
- `<div><p></div>`, the `<p>` is implicitly closed when parsing `</div>`,
as `<p>` is allowed to be closed by the closing of its parent element.
- `<ul><li>A <li>B</ul>`, the first `<li>` is implicitly closed when parsing
the second `<li>`, whereas the second `<li>` would be implicitly closed when
parsing the `</ul>`.
In all the cases above the parsed structure would be correct, however the source
span of the closing `</div>` would incorrectly be assigned to the element that
is implicitly closed. The problem was that closing an element would associate
the source span with the element at the top of the stack, however this may not
be the element that is actually being closed if some elements would be
implicitly closed.
This commit fixes the issue by assigning the end source span with the element
on the stack that is actually being closed. Any implicitly closed elements that
are popped off the stack will not be assigned an end source span, as the
implicit closing implies that no ending element is present.
Note that there is a difference between self-closed elements such as `<input/>`
and implicitly closed elements such as `<input>`. The former does have an end
source span (identical to its start source span) whereas the latter does not.
Fixes#36118
Resolves FW-2004
PR Close#38126
The html parser already normalizes line endings (converting `\r\n` to `\n`)
for most text in templates but it was missing the expressions of ICU expansions.
In ViewEngine backticked literal strings, used to define inline templates,
were already normalized by the TypeScript parser.
In Ivy we are parsing the raw text of the source file directly so the line
endings need to be manually normalized.
This change ensures that inline templates have the line endings of ICU
expression normalized correctly, which matches the ViewEngine.
In ViewEngine external templates, defined in HTML files, the behavior was
different, since TypeScript was not normalizing the line endings.
Specifically, ICU expansion "expressions" are not being normalized.
This is a problem because it means that i18n message ids can be different on
different machines that are setup with different line ending handling,
or if the developer moves a template from inline to external or vice versa.
The goal is always to normalize line endings, whether inline or external.
But this would be a breaking change since it would change i18n message ids
that have been previously computed. Therefore this commit aligns the ivy
template parsing to have the same "buggy" behavior for external templates.
There is now a compiler option `i18nNormalizeLineEndingsInICUs`, which
if set to `true` will ensure the correct non-buggy behavior. For the time
being this option defaults to `false` to ensure backward compatibility while
allowing opt-in to the desired behavior. This option's default will be
flipped in a future breaking change release.
Further, when this option is set to `false`, any ICU expression tokens,
which have not been normalized, are added to the `ParseResult` from the
`HtmlParser.parse()` method. In the future, this collection of tokens could
be used to diagnose and encourage developers to migrate their i18n message
ids. See FW-2106.
Closes#36725
PR Close#36741
Previously, an expansion case could only start with an alpha numeric character.
This commit fixes this by allowing an expansion case to start with any character
except `}`.
The [ICU spec](http://userguide.icu-project.org/formatparse/messages) is pretty vague:
> Use a "select" argument to select sub-messages via a fixed set of keywords.
It does not specify what can be a "keyword" but from looking at the surrounding syntax it
appears that it can indeed be any string that does not contain a `}` character.
Closes#31586
PR Close#36123
The major one that affects the angular repo is the removal of the bootstrap attribute in nodejs_binary, nodejs_test and jasmine_node_test in favor of using templated_args --node_options=--require=/path/to/script. The side-effect of this is that the bootstrap script does not get the require.resolve patches with explicitly loading the targets _loader.js file.
PR Close#34736
The major one that affects the angular repo is the removal of the bootstrap attribute in nodejs_binary, nodejs_test and jasmine_node_test in favor of using templated_args --node_options=--require=/path/to/script. The side-effect of this is that the bootstrap script does not get the require.resolve patches with explicitly loading the targets _loader.js file.
PR Close#34589
This is a breaking change in nodejs rules 0.40.0 as part of the API review & cleanup for the 1.0 release. Their APIs are identical as ts_web_test was just karma_web_test without the config_file attribute.
PR Close#33802
Similar to interpolation, we do not want to completely remove whitespace
nodes that are siblings of an expansion.
For example, the following template
```html
<div>
<strong>items left<strong> {count, plural, =1 {item} other {items}}
</div>
```
was being collapsed to
```html
<div><strong>items left<strong>{count, plural, =1 {item} other {items}}</div>
```
which results in the text looking like
```
items left4
```
instead it should be collapsed to
```html
<div><strong>items left<strong> {count, plural, =1 {item} other {items}}</div>
```
which results in the text looking like
```
items left 4
```
---
**Analysis of the code and manual testing has shown that this does not cause
the generated ids to change, so there is no breaking change here.**
PR Close#31962
Leading trivia, such as whitespace or comments, is
confusing for developers looking at source-mapped
templates, since they expect the source-map segment
to start after the trivia.
This commit adds skipping trivial characters to the lexer;
and then implements that in the template parser.
PR Close#30095
This PR alligns markup language lexer with the previous behaviour in version 7.x:
https://stackblitz.com/edit/angular-iancj2
While this behaviour is not perfect (we should be giving users an error message
here about invalid HTML instead of assuming text node) this is probably best we
can do without more substential re-write of lexing / parsing infrastructure.
This PR just fixes#29231 and restores VE behaviour - a more elaborate fix will
be done in a separate PR as it requries non-trivial rewrites.
PR Close#29328
This PR alligns markup language lexer with the previous behaviour in version 7.x:
https://stackblitz.com/edit/angular-iancj2
While this behaviour is not perfect (we should be giving users an error message
here about invalid HTML instead of assuming text node) this is probably best we
can do without more substential re-write of lexing / parsing infrastructure.
This PR just fixes#29231 and restores VE behaviour - a more elaborate fix will
be done in a separate PR as it requries non-trivial rewrites.
PR Close#29328
BREAKING CHANGE:
Certain elements (like `<tr>` or `<col>`) require parent elements to be of a certain type by the HTML specification
(ex. <tr> can only be inside <tbody> / <thead>). Before this change Angular template parser was auto-correcting
"invalid" HTML using the following rules:
- `<tr>` would be wrapped in `<tbody>` if not inside `<tbody>`, `<tfoot>` or `<thead>`;
- `<col>` would be wrapped in `<colgroup>` if not inside `<colgroup>`.
This meachanism of automatic wrapping / auto-correcting was problematic for several reasons:
- it is non-obvious and arbitrary (ex. there are more HTML elements that has rules for parent type);
- it is incorrect for cases where `<tr>` / `<col>` are at the root of a component's content, ex.:
```html
<projecting-tr-inside-tbody>
<tr>...</tr>
</projecting-tr-inside-tbody>
```
In the above example the `<projecting-tr-inside-tbody>` component culd be "surprised" to see additional
`<tbody>` elements inserted by Angular HTML parser.
PR Close#29219
Previously the start of a character indicated by an escape sequence
was being incorrectly computed by the lexer, which caused tokens
to include the start of the escaped character sequence in the
preceding token. In particular this affected the name extracted
from opening tags if the name was terminated by an escape sequence.
For example, `<t\n>` would have the name `t\` rather than `t`.
This fix refactors the lexer to use a "cursor" object to iterate over
the characters in the template source. There are two cursor implementations,
one expects a simple string, the other expects a string that contains
JavaScript escape sequences that need to be unescaped.
PR Close#28978
The parts of a token are supposed to be an array of not-null strings,
but we were using `null` for tags that had no prefix. This has been
fixed to use the empty string in such cases, which allows the `null !`
hack to be removed.
PR Close#28978
When tokenizing markup (e.g. HTML) element attributes
can have quoted or unquoted values (e.g. `a=b` or `a="b"`).
The `ATTR_VALUE` tokens were capturing the quotes, which
was inconsistent and also affected source-mapping.
Now the tokenizer captures additional `ATTR_QUOTE` tokens,
which the HTML related parsers understand and factor into their
token parsing.
PR Close#28055
In order to support source mapping of templates, we need
to be able to tokenize the template in its original context.
When the template is defined inline as a JavaScript string
in a TS/JS source file, the tokenizer must be able to handle
string escape sequences, such as `\n` and `\"` as they
appear in the original source file.
This commit teaches the lexer how to unescape these
sequences, but only when the `escapedString` option is
set to true. Otherwise there is no change to the tokenizing
behaviour.
PR Close#28055
The lexer that does the tokenizing can now process only a part the source
string, by passing a `range` property in the `options` argument. The
locations of the nodes that are tokenized will now take into account the
position of the span in the context of the original source string.
This `range` option is, in turn, exposed from the template parser as well.
Being able to process parts of files helps to enable SourceMap support
when compiling inline component templates.
PR Close#28055
This commit consolidates the options that can modify the
parsing of text (e.g. HTML, Angular templates, CSS, i18n)
into an AST for further processing into a single `options`
hash.
This makes the code cleaner and more readable, but also
enables us to support further options to parsing without
triggering wide ranging changes to code that should not
be affected by these new options. Specifically, it will let
us pass information about the placement of a template
that is being parsed in its containing file, which is essential
for accurate SourceMap processing.
PR Close#28055
Most of the specs in these tests are not relevant to Ivy:
//packages/compiler/test:test
//packages/compiler/test:test_web_chromium-local
However, a few test pieces of the compiler infrastructure that are used in
Ivy, and new BUILD.bazel files are created to separate them from the above
disabled targets:
//packages/compiler/test/css_parser:css_parser
//packages/compiler/test/css_parser:css_parser_web
//packages/compiler/test/expression_parser:expression_parser
//packages/compiler/test/expression_parser:expression_parser_web
//packages/compiler/test/ml_parser:ml_parser
//packages/compiler/test/ml_parser:ml_parser_web
//packages/compiler/test/selector:selector
//packages/compiler/test/selector:selector_web
PR Close#27301
BREAKING CHANGE:
The `<template>` tag was deprecated in Angular v4 to avoid collisions (i.e. when
using Web Components).
This commit removes support for `<template>`. `<ng-template>` should be used
instead.
BEFORE:
<!-- html template -->
<template>some template content</template>
# tsconfig.json
{
# ...
"angularCompilerOptions": {
# ...
# This option is no more supported and will have no effect
"enableLegacyTemplate": [true|false]
}
}
AFTER:
<!-- html template -->
<ng-template>some template content</ng-template>
PR Close#22783
Marcono1234
Pete Bacon Darwin
Ayaz Hafiz
Joey Perrott
JoostK
Alex Rickabaugh
Sonu Kapoor
Greg Magolan
atscott
Pawel Kozlowski
Matias Niemelä
Victor Berchet
Misko Hevery
Olivier Combe
Trotyl Yu
Dzmitry Shylovich
Jason Aden