Insertion modes in an HTML parser may include instructions like "process
the token in the IN HEAD insertion mode." The rules do not change the
insertion mode of the parser, but the errors are triggered outside of the
rules for the current insertion mode. These will be misleading when
bailing on these instructions, because it will point someone to the wrong
place in the code to find the source of the error.
In this patch all of the bail-points due to lacking insertion mode support
are hard-coded to better orient someone to the section of the code lacking
support for handling the input HTML.
Developed in https://github.com/wordpress/wordpress-develop/pull/7043
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58679].
Props: dmsnell, jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58781
git-svn-id: http://core.svn.wordpress.org/trunk@58183 1a063a9b-81f0-0310-95a4-ce76da25c4cd
In part of a larger review of CSS semantics and behaviors, this patch
takes the opportunity to remove leading whitespace in an updated class
attribute after the first class in the attribute has been removed.
Previously, if the first class name had been removed, the whitespace
that formerly followed it would remain in the class attribute. This
stood in contrast to removing other class names, which removed their
associated whitespace.
There should be no semantic or functional changes in this patch, only
a slightly-large diff for modified HTML documents that looks prettier
when removing the first class name in a class attribute.
Developed in https://github.com/WordPress/wordpress-develop/pull/6933
Discussed in https://core.trac.wordpress.org/ticket/61531
Props dmsnell, jonsurrell.
See #61531.
Built from https://develop.svn.wordpress.org/trunk@58740
git-svn-id: http://core.svn.wordpress.org/trunk@58142 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML Processor internally throws an exception when it reaches HTML
that it knows it cannot process, but this exception is not made
available to calling code. It can be useful to extract more knowledge
about why it gave up, especially for debugging purposes.
In this patch, more context is added to the WP_HTML_Unsupported_Exception
and the last exception is made available to calling code through a new
method, `get_unsupported_exception()`.
Developed in https://github.com/WordPress/wordpress-develop/pull/6985
Discussed in https://core.trac.wordpress.org/ticket/61646
Props bernhard-reiter, dmsnell, jonsurrell.
See #61646.
Built from https://develop.svn.wordpress.org/trunk@58714
git-svn-id: http://core.svn.wordpress.org/trunk@58116 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Since the HTML Processor started visiting all nodes in a document, both
real and virtual, the breadcrumb accounting became a bit complicated
and it's not entirely clear that it is fully reliable.
In this patch the breadcrumbs are rebuilt separately from the stack of
open elements in order to eliminate the problem of the stateful stack
interactions and the post-hoc event queue.
Breadcrumbs are greatly simplified as a result, and more verifiably
correct, in this construction.
Developed in https://github.com/WordPress/wordpress-develop/pull/6981
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58590].
Props bernhard-reiter, dmsnell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58713
git-svn-id: http://core.svn.wordpress.org/trunk@58115 1a063a9b-81f0-0310-95a4-ce76da25c4cd
In order to add support for the SELECT and TABLE tags in the HTML Processor, it
needs to implement the HTML algorithm named "reset the insertion mode
appropriately".
This patch implements that algorithm to unblock the additional tag support. The
algorithm resets the parsing mode after specific state changes in complicated
situations where alternative rules are in effect (such as rules governing how
the parser handles tags found within a TABLE element).
Developed in https://github.com/WordPress/wordpress-develop/pull/6020
Discussed in https://core.trac.wordpress.org/ticket/61549
Props dmsnell, jonsurrell.
Fixes#61549.
Built from https://develop.svn.wordpress.org/trunk@58656
git-svn-id: http://core.svn.wordpress.org/trunk@58070 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When [58304] introduced the abililty to visit virtual nodes in the HTML document,
those being the nodes which are implied by the HTML but no explicitly present in
the raw text, a bug was introduced in the `get_breadcrumbs()` method because it
wasn't updated to be aware of the virtual nodes. Therefore it would report the
wrong breadcrumbs for virtual nodes. Since the new `get_depth()` method is based
on the same logic it was also broken for virtual nodes.
In this patch, the breadcrumbs have been updated to account for the virtual nodes
and the depth method has been updated to rely on the fixed breadcrumb logic.
Developed in https://github.com/WordPress/wordpress-develop/pull/6914
Discussed in https://core.trac.wordpress.org/ticket/61348
Follow-up to [58304].
Props dmsnell, jonsurrell, zieladam.
See #61348.
Built from https://develop.svn.wordpress.org/trunk@58588
git-svn-id: http://core.svn.wordpress.org/trunk@58035 1a063a9b-81f0-0310-95a4-ce76da25c4cd
This patch introduces two related changes:
- It adds missing subclass methods on the HTML Processor which needed
to be implemented since it started visiting virtual nodes. These
methods need to account for the fact that not all tokens truly exist.
- It adds a new concept and internal method, `is_virtual()`, indicating
if the currently-matched token comes from the raw text in the input
HTML document or if it was the byproduct of semantic parsing rules.
This internal method and new vocabulary around token provenance
considerably simplifies the logic spread throughout the rest of the
class and its subclass methods.
Developed in https://github.com/WordPress/wordpress-develop/pull/6860
Discussed in https://core.trac.wordpress.org/ticket/61348
Follow-up to [58304].
Props dmsnell, jonsurrell, gziolo.
See #61348.
Built from https://develop.svn.wordpress.org/trunk@58558
git-svn-id: http://core.svn.wordpress.org/trunk@58006 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When the `WP_HTML_Processor` was introduced with its `::create_fragment()`
static creator method, that method has been returning a `new self(...)`.
Unfortunately, this means that subclasses cannot use that method since it
will return the `WP_HTML_Processor` instead of the subclass.
With this patch, the static creator method returns `new static(...)` to preserve
the intended behavior. A new test asserts this behavior for future changes.
Developed in https://github.com/WordPress/wordpress-develop/pull/6729
Discussed in https://core.trac.wordpress.org/ticket/61374
Props dmsnell, jonsurrell.
Follow-up to [56274].
Fixes#61374.
Built from https://develop.svn.wordpress.org/trunk@58363
git-svn-id: http://core.svn.wordpress.org/trunk@57812 1a063a9b-81f0-0310-95a4-ce76da25c4cd
HTML is a kind of short-hand for a DOM structure. This means that there are
many cases in HTML where an element's opening tag or closing tag is missing (or
both). This is because many of the parsing rules imply creating elements in the
DOM which may not exist in the text of the HTML.
The HTML Processor, being the higher-level counterpart to the Tag Processor, is
already aware of these nodes, but since it's inception has not paused on them
when scanning through a document. Instead, these are visible when pausing on a
child of such an element, but otherwise not seen.
In this patch the HTML Processor starts exposing those implicitly-created nodes,
including opening tags, and closing tags, that aren't foudn in the text content
of the HTML input document.
Previously, the sequence of matched tokens when scanning with
`WP_HTML_Processor::next_token()` would depend on how the HTML document was written,
but with this patch, all semantically equal HTML documents will parse and scan in
the same exact manner, presenting an idealized or "perfect" view of the document
the same way as would occur when traversing a DOM in a browser.
Developed in https://github.com/WordPress/wordpress-develop/pull/6348
Discussed in https://core.trac.wordpress.org/ticket/61348
Props audrasjb, dmsnell, gziolo, jonsurrell.
Fixes#61348.
Built from https://develop.svn.wordpress.org/trunk@58304
git-svn-id: http://core.svn.wordpress.org/trunk@57761 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Provides a custom decoder for strings coming from HTML attributes and
markup. This custom decoder is necessary because of deficiencies in
PHP's `html_entity_decode()` function:
- It isn't aware of 720 of the possible named character references in
HTML, leaving many out that should be translated.
- It isn't aware of the ambiguous ampersand rule, which allows
conversion of character references in certain contexts when they
are missing their closing `;`.
- It doesn't draw a distinction for the ambiguous ampersand rule
when decoding attribute values instead of markup values.
- Use of `html_entity_decode()` requires manually passing non-default
paramter values to ensure it decodes properly.
This decoder also provides some conveniences, such as making a
single-pass and interruptable decode operation possible. This will
provide a number of opportunities to optimize detection and decoding
of things like value prefixes, and whether a value contains a given
substring.
Developed in https://github.com/WordPress/wordpress-develop/pull/6387
Discussed in https://core.trac.wordpress.org/ticket/61072
Props dmsnell, gziolo, jonsurrell, jorbin, westonruter, zieladam.
Fixes#61072.
Built from https://develop.svn.wordpress.org/trunk@58281
git-svn-id: http://core.svn.wordpress.org/trunk@57741 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The Tag Processor stores the byte-offsets into its HTML document where
the current token starts and ends, and also for every bookmark. In some
cases for tags, the end offset has been off by one.
In this patch the offset is fixed so that a bookmark always properly
refers to the full span of the token it's bookmarking. Also the current
token byte offsets are properly recorded.
While this is a defect in the Tag Processor, it hasn't been exposed
through the public interface and has not affected any of the working
of the processor. Only subclasses which rely on the length of a bookmark
have been potentially affected, and these are not supported environments
in the ongoing work.
This fix is important for future work and for ensuring that subclasses
performing custom behaviors remain as reliable as the public interface.
Developed in https://github.com/WordPress/wordpress-develop/pull/6625
Discussed in https://core.trac.wordpress.org/ticket/61301
Props dmsnell, gziolo, jonsurrell, westonruter.
Fixes#61301.
Built from https://develop.svn.wordpress.org/trunk@58233
git-svn-id: http://core.svn.wordpress.org/trunk@57696 1a063a9b-81f0-0310-95a4-ce76da25c4cd
This patch adds a new method, `WP_HTML_Processor->expects_closer()` to indicate
if the currently-matched node expects to find a closing token. For example, a
`DIV` element expects a closing `</div>` tag, but an `<img>` expects none, because
it's a void element. Similarly, `#text` nodes and HTML comments only appear as
unitary nodes on the stack of open elements. Once proceeding further in the
document they are immediately removed without any closing tag.
This new method serves as a helper to indicate whether or not to expect the
closer, as this can be more complicated than it seems, and calling code
shouldn't have to build custom interpretations and implementations. Instead,
the HTML Processor ought to export its internal knowledge to make it easy for
consuming code and projects.
Developed in https://github.com/WordPress/wordpress-develop/pull/6600
Discussed in https://core.trac.wordpress.org/ticket/61257Fixes#61257.
Props dmsnell, jonsurrell.
Built from https://develop.svn.wordpress.org/trunk@58192
git-svn-id: http://core.svn.wordpress.org/trunk@57655 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML Processor maintains a stack of open elements, where every element,
every `#text` node, every HTML comment, and other node is pushed and popped while
traversing the document. The "depth" of each of these nodes represents how deep
that stack is where the node appears. Unfortunately this information isn't
exposed to calling code, which has led different projects to attempt to
calculate this value externally. This isn't always trivial, but the HTML
Processor could make it so by exposing the internal knowledge in a new method.
In this patch the `get_current_depth()` method returns just that. Since the
processor always exists within a context, the depth includes nesting from the
always-present html element and also the body, since currently the HTML
Processor only supports parsing in the IN BODY context.
This means that the depth reported for the `DIV` in `<div>` is 3, not 1, because
its breadcrumbs path is `HTML > BODY > DIV`.
Developed in https://github.com/WordPress/wordpress-develop/pull/6589
Discussed in https://core.trac.wordpress.org/ticket/61255Fixes#61255.
Props dmsnell, jonsurrell.
Built from https://develop.svn.wordpress.org/trunk@58191
git-svn-id: http://core.svn.wordpress.org/trunk@57654 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Since [60428] the Tag Processor has been misidentifying single-character
funky comments. It has been asserting that the full token-length for a
funky comment must be at least three characters after the opening (e.g.
`</1>`), but it has been starting to look for the closing `>` after
those same three characters. This means that it has been skipping the
actual close of these funky comments and swallowing up the next syntax
until it finds a `>`, often consuming the next tag in the process.
This patch fixes the detector and restores finding the following token.
Developed in https://github.com/WordPress/wordpress-develop/pull/6412
Discussed in https://core.trac.wordpress.org/ticket/60170
Follow-up to [60428].
Fixes#60170.
Props dmsnell, gziolo, jonsurrell.
Built from https://develop.svn.wordpress.org/trunk@58040
git-svn-id: http://core.svn.wordpress.org/trunk@57506 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When encountering text nodes in an HTML document, the HTML parser needs
to run the active format reconstruction algorithm, even if it doesn't
stop to visit those text nodes. This is because the formats, which might
need reconstructing, will impact the breadcrumbs of all downstream nodes
from the text node.
In this patch, this process is triggered, which properly triggers the
active format reconstruction. It also enables the visiting of other token
types as is possible in the Tag Processor.
Developed in https://github.com/WordPress/wordpress-develop/pull/6054
Discussed in https://core.trac.wordpress.org/ticket/60170
Props: dmsnell, jonsurrell, westonruter.
Fixes: #60455.
Follow-up to: [57348].
Built from https://develop.svn.wordpress.org/trunk@57806
git-svn-id: http://core.svn.wordpress.org/trunk@57307 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When making repeated updates to a document, the Tag Processor will end
up copying the entire document once for every update. This can lead to
catastrophic behavior in the worse case.
However, when batch-applying updates it's able to copy chunks of the
document in one thread and only end up copying the entire document once
for the entire batch.
Previously the Tag Processor has been eagerly applying udpates, but in
this patch it defers applying those updates as long as is possible.
Developed in https://github.com/WordPress/wordpress-develop/pull/6120
Discussed in https://core.trac.wordpress.org/ticket/60697
Props: dmsnell, bernhard-reiter, jonsurrell, westonruter.
Fixes#60697.
Follow-up to [55706], [56941], [57348].
Built from https://develop.svn.wordpress.org/trunk@57805
git-svn-id: http://core.svn.wordpress.org/trunk@57306 1a063a9b-81f0-0310-95a4-ce76da25c4cd
In some cases, it's possible to seek back into a location found inside
an element which has been closed before the point in the document where
the `seek()` was made. In these cases the breadcrumb stack is lost, and
calling `get_breadcrumbs()` after the seek will return the wrong information.
In this patch, the HTML Processor takes a conservative approach and
moves to the front of the document, then reparses the document until
it reaches the sought-after location. This ensures consistency on
the stack of open elements and active formats, and preserves
breadcrumbs.
Developed in https://github.com/WordPress/wordpress-develop/pull/6185
Discussed in https://core.trac.wordpress.org/ticket/60687
Props jonsurrell.
Follow-up to [60687].
See #58517.
Fixes#60687.
Built from https://develop.svn.wordpress.org/trunk@57768
git-svn-id: http://core.svn.wordpress.org/trunk@57269 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When returning modifiable text in the HTML API, if the text segment
coerces to `false` inside `empty()`, then an empty string has been
returned instead of the string itself. For example, the text node in the
following HTML snippet:
{{{
<div>0</div>
}}}
In this patch the `empty()` check is removed. The purpose of the original
check was to skip further processing if the text content is empty, but
the check is not needed and the additioanl processing is minimal.
Removing the code removes the defect and leaves a cleaner method in its
absence.
Developed in https://github.com/WordPress/wordpress-develop/pull/6199
Follow-up to [57348]
Follow-up to #60170
Built from https://develop.svn.wordpress.org/trunk@57738
git-svn-id: http://core.svn.wordpress.org/trunk@57239 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The WP_HTML_Processor::has_bookmark() method has not correctly reported bookmarks
which have been set, because it wraps the given bookmark names when setting them.
Additionally, WP_HTML_Processor::seek() does not seek to correct location if HTML
has been updated because it wasn't flushing enqueued updates to the document.
In this patch both problems are resolved and added tests guard these behaviors
against future regressions.
Developed in https://github.com/WordPress/wordpress-develop/pull/6039
Discussed in https://core.trac.wordpress.org/ticket/60474
Follow-up to [56274].
Props dmsnell, jonsurrell.
Fixes#60474.
Built from https://develop.svn.wordpress.org/trunk@57582
git-svn-id: http://core.svn.wordpress.org/trunk@57083 1a063a9b-81f0-0310-95a4-ce76da25c4cd
A fix was introduced to the Tag Processor to ensure that contiguous text
in an HTML document emerges as a single text node spanning the full
sequence. Unfortunately, that patch was marginally over-zealous in
checking if a "<" started a syntax token or not. It used the following:
{{{
<?php
if ( 'A' <= $c && 'z' >= $c ) { ... }
}}}
This was based on the assumption that the A-Z and a-z letters are
contiguous in the ASCII range; they aren't, and there's a gap of
several characters in between. The result of this is that in some
cases the parser created a text boundary when it didn't need to.
Text boundaries can be surprising and can be created when reaching
invalid syntax, HTML comments, and more hidden elements, so
semantically this wasn't a major bug, but it was an aesthetic
challenge.
In this patch the check is properly compared for both upper- and
lower-case variants that could potentially form tag names.
{{{
<?php
if ( ( 'A' <= $c && 'Z' >= $c ) || ( 'a' <= $c && 'z' >= $c ) ) { ... }
}}}
This solves the problem and ensures that contiguous text appears
as a single text node when scanning tokens.
Developed in https://github.com/WordPress/wordpress-develop/pull/6041
Discussed in https://core.trac.wordpress.org/ticket/60385
Follow-up to [57489]
Props dmsnell, jonsurrell
Fixes#60385
Built from https://develop.svn.wordpress.org/trunk@57542
git-svn-id: http://core.svn.wordpress.org/trunk@57043 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML Processor tracks whether a token was found with the self-closing flag.
Depending on the context, this flag may or may not indicate that the element is
self closing. Unfortunately it's been tracking the wrong flag: it's been tracking
the end-tag flag, which indicates that a token is an end tag.
In this patch the right flag is set in the HTML Processor. This hasn't been an
issue because the HTML Processor doesn't yet read that stored flag, but it's an
important fix to make before adding support for foreign content (SVG and MathML)
since that behavior depends on reading the correct flag.
Follow-up to [56274].
Props dmsnell.
Built from https://develop.svn.wordpress.org/trunk@57528
git-svn-id: http://core.svn.wordpress.org/trunk@57029 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When `next_token()` was introduced, it introduced a regression in the HTML
Processor whereby void tags remain on the stack of open elements when they
shouldn't. This led to invalid values returned from `get_breadcrumbs()`.
The reason was that calling `next_token()` works through a different code path
than the HTML Processor runs everything else. To solve this, its sub-classed
`next_token()` called `step( self::REPROCESS_CURRENT_TOKEN )` so that the proper
HTML accounting takes place.
Unfortunately that same reprocessing code path skipped the step whereby void
and self-closing elements are popped from the stack of open elements.
In this patch, that step is run with a third mode for `step()`, which is the
new `self::PROCESS_CURRENT_TOKEN`. This mode acts as if `self::PROCESS_NEXT_NODE`
were called, except it doesn't advance the parser.
Developed in https://github.com/WordPress/wordpress-develop/pull/5975
Discussed in https://core.trac.wordpress.org/ticket/60382
Follow-up to [57348]
Props dmsnell, jonsurrell
Fixes#60382
Built from https://develop.svn.wordpress.org/trunk@57507
git-svn-id: http://core.svn.wordpress.org/trunk@57008 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When `next_token()` was introduced to the HTML Tag Processor, it started
classifying comments that look like they were intended to be CDATA sections.
In one of the changes made during development, however, a typo slipped
through code review that treated comments as CDATA even if they only
ended in `]>` and not the required `]]>`.
The consequences of this defect were minor because in all cases these are
treated as HTML comments from invalid syntax, but this patch adds the
missing check to ensure the proper reporting of CDATA-lookalikes.
Follow-up to [57348]
Props jonsurrell
Fixes#60406
Built from https://develop.svn.wordpress.org/trunk@57506
git-svn-id: http://core.svn.wordpress.org/trunk@57007 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When `next_token()` was introduced, it brought a subtle bug. When encountering a `<` in the HTML stream which did not lead to a tag or comment or other token, it was treating the full text span to that point as one text node, and the following span another text node.
The entire span should be one text node.
In this patch the Tag Processor properly detects this scenario and combines the spans into one text node.
Follow-up to [57348]
Props jonsurrell
Fixes#60385
Built from https://develop.svn.wordpress.org/trunk@57489
git-svn-id: http://core.svn.wordpress.org/trunk@56990 1a063a9b-81f0-0310-95a4-ce76da25c4cd
dmsnell
audrasjb
Peter Wilson
Pascal Birchler