Commit Graph

71 Commits

Author SHA1 Message Date
dmsnell 011d45abbc HTML API: Add sentinels for unreachable code.
There are places in the HTML API code where some tools get confused and flag invalid types for the return of a function because they are unable to detect that the end of the function is unreachable.

Since PHP doesn't provide a way to encode total matching in the source code, this patch adds a few extra lines in those unreachable locations to satisfy any tooling which isn't able to fully analyze the code.

Additionally this serves as extra guarding in case someone changes these functions in a way which would break them and the existing test suite doesn't catch those breakages.

Developed in https://github.com/WordPress/wordpress-develop/pull/7315
Discussed in https://core.trac.wordpress.org/ticket/62018

Props dlh, dmsnell.
Fixes #62018.

Built from https://develop.svn.wordpress.org/trunk@59001


git-svn-id: http://core.svn.wordpress.org/trunk@58397 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-09 22:12:15 +00:00
dmsnell 72bc81c048 HTML API: Only examine HTML nodes in `pop_until()` instack of open elements.
The `pop_until( $tag_name )` method in the stack of open elements should only be examining HTML elements, but it has only been checking the tag name. This has led to closing the wrong tags when run from inside foreign content. A very specific situation where this may arise is when a `TEMPLATE` closer is found inside foreign content, inside another template.

{{{
HTML:template   SVG:template                 HTML:/template
<template><svg><template><foreignObject><div></template><div>
╰──< this outer TEMPLATE is closed by this one >───╯
}}}

This patch constains the method to checking for elements matching the tag name which are in the HTML namespace so that the proper detection occurs.

Developed in https://github.com/WordPress/wordpress-develop/pull/7286
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58867].

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58992


git-svn-id: http://core.svn.wordpress.org/trunk@58388 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-04 19:25:14 +00:00
dmsnell d0d8ba24d8 HTML API: Respect document compat mode when handling CSS class names.
The HTML API has been behaving as if CSS class name selectors matched class names in an ASCII case-insensitive manner. This is only true if the document in question is set to quirks mode. Unfortunately most documents processed will be set to no-quirks mode, meaning that some CSS behaviors have been matching incorrectly when provided with case variants of class names.

In this patch, the CSS methods have been audited and updated to adhere to the rules governing ASCII case sensitivity when matching classes. This includes `add_class()`, `remove_class()`, `has_class()`, and `class_list()`. Now, it is assumed that a document is in no-quirks mode unless a full HTML parser infers quirks mode, and these methods will treat class names in a byte-for-byte manner. Otherwise, when a document is in quirks mode, the methods will compare the provided class names against existing class names for the tag in an ASCII case insensitive way, while `class_list()` will return a lower-cased version of the existing class names.

The lower-casing in `class_list()` is performed for consistency, since it's possible that multiple case variants of the same comparable class name exists on a tag in the input HTML.

Developed in https://github.com/WordPress/wordpress-develop/pull/7169
Discussed in https://core.trac.wordpress.org/ticket/61531

Props dmsnell, jonsurrell.
See #61531.

Built from https://develop.svn.wordpress.org/trunk@58985


git-svn-id: http://core.svn.wordpress.org/trunk@58381 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-04 04:34:15 +00:00
dmsnell 2ce04ceab9 HTML API: Ensure that NULL and whitespace-only CDATA sections don't forbid FRAMESET.
When CDATA sections (which can only occur inside SVG and MathML content) consist only of NULL bytes or whitespace characters they should not clear the "frameset ok" flag. Previously they have always been clearing this flag, but in this patch the logic is updated to detect these sequences properly.

Developed in https://github.com/WordPress/wordpress-develop/pull/7230
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58867].

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58977


git-svn-id: http://core.svn.wordpress.org/trunk@58373 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-03 19:50:14 +00:00
dmsnell 8d2a10b300 HTML API: Allow subdividing text nodes by meaningful prefixes.
HTML parsing rules at times differentiate character tokens that are all null bytes, all whitespace, or other content. This patch introduces a new function which may be used to classify text node sub-regions and lead to more efficient application of these parsing rules.

Further, when classified in this way, application code may skip some rules and decoding entirely, improving performance. For example, this can be used to ease the implementation of skipping inter-element whitespace, which is usually not rendered.

Developed in https://github.com/WordPress/wordpress-develop/pull/7236
Discussed in https://core.trac.wordpress.org/ticket/61974

Props dmsnell, jonsurrell.
Fixes #61974.

Built from https://develop.svn.wordpress.org/trunk@58970


git-svn-id: http://core.svn.wordpress.org/trunk@58366 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-02 23:21:18 +00:00
dmsnell eac35f6ae6 HTML API: Add missing NOBR end tag handling to HTML Processor.
When the HTML Processor was introduced, an oversight led to a missing case for handling a closing `NOBR` tag. The `NOBR` element is a deprecated tag and should not be used.

This patch adds the missing case so that the deprecated `NOBR` end tag is appropriately handled.

Developed in https://github.com/WordPress/wordpress-develop/pull/7282
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [56274].

Props jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58967


git-svn-id: http://core.svn.wordpress.org/trunk@58363 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-02 22:03:15 +00:00
dmsnell d7f9d30df8 HTML API: Fix logic bug in HTML Processor when opening A element.
A mistake in the original code handling opening A elements in the HTML Processor led to mistakes in parsing where the Processor would bail in situations when it could have proceeded. While this was errant behavior, it didn't violate the public contract since it would bail in these situations.

This patch fixes the mistake, which was to only break out of the innermost loop instead of breaking from the containing loop, which resolves the issue.

Developed in https://github.com/WordPress/wordpress-develop/pull/7281
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [56274].

Props jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58966


git-svn-id: http://core.svn.wordpress.org/trunk@58362 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-02 21:54:15 +00:00
dmsnell 813d784931 HTML API: Fix a bug where the namespace was forced to 'html'
While working on other reviews and audits, a bug was discovered in the HTML API where the wrong namespace was being assigned to a token because the default value of 'html' was used. This patch fixes the bug by calling the `parent::get_namespace()` method instead of assuming 'html'.

Developed in https://github.com/wordpress/wordpress-develop/7232
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58868].

Props jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58940


git-svn-id: http://core.svn.wordpress.org/trunk@58336 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-27 03:37:12 +00:00
dmsnell 01d2199622 HTML API: Add support for missing FRAMESET and "after" insertion modes.
As part of work to add more spec support to the HTML API, this patch adds support for the FRAMESET-related insertion modes, as well as the set of missing after insertion modes. These modes run at the end of parsing a document, closing it and taking care of any lingering tags.

Developed in https://github.com/wordpress/wordpress-develop/7165
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58926


git-svn-id: http://core.svn.wordpress.org/trunk@58322 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-23 15:42:17 +00:00
dmsnell e3a3c13497 HTML API: Parse DOCTYPE tokens and set HTML parser mode accordingly.
This patch adds until-now missing code to parse the structure of HTML DOCTYPE declarations. The DOCTYPE is mostly unused but can dictate the document compatability mode, which governs whether CSS class names match in a ASCII-case-insensitive way or not, and whether TABLE elements close an open P element.

The DOCTYPE information is made available through a new method on the Tag Processor, `get_doctype_info()`.

Developed in https://github.com/wordpress/wordpress-develop/pull/7195
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58925


git-svn-id: http://core.svn.wordpress.org/trunk@58321 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-23 14:55:15 +00:00
dmsnell 5cd8048fc2 HTML API: Make "reset insertion mode appropriately" algorithm private.
The algorithm is described and referred to in the specification as
"reset the insertion mode appropriately." This patch renames it to
more accurately reflect the specification, and it turns the public
method private. As an internal algorithm from the HTML specification
it should not be open to extension.

Developed in https://github.com/wordpress/wordpress-develop/pull/7194
Discussed in https://core.trac.wordpress.org/ticket/61549

Follow-up to [28656].
Props jonsurrell.
See #61549.

Built from https://develop.svn.wordpress.org/trunk@58898


git-svn-id: http://core.svn.wordpress.org/trunk@58294 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-14 21:19:16 +00:00
dmsnell 98e51518b3 HTML API: Remove completed TODO comments.
This patch removes TODO comments indicating the need to verify certain behaviors and algorithms. Those verifications have taken place and the comments are no longer needed.

Developed in https://github.com/wordpress/wordpress-develop/pull/7174
Discussed in https://core.trac.wordpress.org/ticket/61646

Follow-up to [58867], [58870].

Props jonsurrell.
See #64646.

Built from https://develop.svn.wordpress.org/trunk@58877


git-svn-id: http://core.svn.wordpress.org/trunk@58273 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-10 05:00:14 +00:00
dmsnell c834d20762 HTML API: Test and fix SVG script handling.
When support was added for foreign content, an ambiguity in the HTML specification led to code that followed the wrong path when encountering a self-closing SCRIPT element in the SVG namespace. Further, a fallthrough was discovered during manual testing.

This patch adds a new test to assert the proper behaviors and fixes these issues. In the case of the SCRIPT element, the outcome was the same with the wrong code path, making the defect benign. In the case of the fallthrough, the wrong behavior would occur.

The updates in this patch also resolves a todo relating to the spec ambiguity.

Developed in https://github.com/wordpress/wordpress-develop/pull/7164
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58868].

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58871


git-svn-id: http://core.svn.wordpress.org/trunk@58267 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-08 17:04:19 +00:00
dmsnell 7611ec9415 HTML API: `expect_closer()` should report false for self-closing foreign elements.
Previously, `WP_HTML_Processor::expects_closer()` would report `true` for self-closing foreign elements when called without supplying a node in question, but it should have been reporting `true` just as it does for HTML elements.

This patch adds a test case demonstrating the issue and a bugfix.

The `html5lib` test runner was relying on the incorrect behavior, accidentally working. This is also corrected and the `html5lib` test now relies on the correct behavior of `expects_closer()`.

Developed in https://github.com/wordpress/wordpress-develop/pull/7162
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58868].

Props: dmsnell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58870


git-svn-id: http://core.svn.wordpress.org/trunk@58266 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-08 16:15:20 +00:00
dmsnell 1ffe7a456c HTML API: Add support for SVG and MathML (Foreign content)
As part of work to add more spec support to the HTML API, this patch adds
support for SVG and MathML elements, or more generally, "foreign content."

The rules in foreign content are a mix of XML and HTML parsing rules and
introduce additional complexity into the processor, but is important in
order to avoid getting lost when inside these elements.

Developed in https://github.com/wordpress/wordpress-develop/pull/6006
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell, westonruter.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58867


git-svn-id: http://core.svn.wordpress.org/trunk@58263 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-08 07:25:15 +00:00
dmsnell 90a66c7ad1 HTML API: Add support for IN SELECT IN TABLE parsing.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN SELECT IN TABLE insertion mode. This small section of the
spec handles rules for the `<select>` element and its children when found
inside of a `<table>`.

Developed in https://github.com/wordpress/wordpress-develop/pull/7044
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58841


git-svn-id: http://core.svn.wordpress.org/trunk@58237 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-01 22:53:16 +00:00
dmsnell 40114e8ed5 HTML API: Add support for IN CAPTION parsing.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN CAPTION insertion mode. This small section of the
spec handles rules for the `<caption>` element.

Developed in https://github.com/wordpress/wordpress-develop/pull/7041
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58840


git-svn-id: http://core.svn.wordpress.org/trunk@58236 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-01 22:36:13 +00:00
dmsnell 5cfd2dee00 HTML API: Add support for IN COLUMN GROUP parsing.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN COLUMN GROUP insertion mode. This small section of the
spec handles rules for the `<colgroup>` element.

Developed in https://github.com/wordpress/wordpress-develop/pull/7042
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58839


git-svn-id: http://core.svn.wordpress.org/trunk@58235 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-01 22:06:12 +00:00
dmsnell af6e4904af HTML API: Introduce full parsing mode in HTML Processor.
The HTML Processor has only supported a specific kind of parsing mode
called _the fragment parsing mode_, where it behaves in the same way
that `node.innerHTML = html` does in the DOM. This mode assumes a
context node and doesn't support parsing an entire document.

As part of work to add more spec support to the HTML API, this patch
introduces a full parsing mode, which can parse a full HTML document
from start to end, including the doctype declaration and head tags.

Developed in https://github.com/wordpress/wordpress-develop/pull/6977
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58836


git-svn-id: http://core.svn.wordpress.org/trunk@58232 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-31 16:56:15 +00:00
dmsnell f483d6dc4e HTML API: Add TEMPLATE and related support in HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN TEMPLATE and IN HEAD insertion modes. These changes are
primarily about adding support for TEMPLATE elements in the HTML Processor,
but include support for other tags commonly found in the document head, such
as LINK, META, SCRIPT, STYLE, and TITLE.

Developed in https://github.com/wordpress/wordpress-develop/pull/7046
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell, westonruter.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58833


git-svn-id: http://core.svn.wordpress.org/trunk@58229 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-30 18:46:35 +00:00
dmsnell d8ab99f84b HTML API: Close all elements at the end of a document.
When the model of breadcrumb generation in the HTML Processor and node
traversal was simplified, the change introduced a bug whereby unclosed
nodes at the end of a document would remain unvisited and unclosed.

In this patch, a fix is applied to ensure that all open elements close
while traversing a document. A couple of minor documentation typos are
fixed in the patch as well.

Developed in https://github.com/wordpress/wordpress-develop/pull/7085
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58713].

Props: dmsnell, gziolo, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58828


git-svn-id: http://core.svn.wordpress.org/trunk@58224 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-29 17:39:17 +00:00
dmsnell 539ee9221b HTML API: Add TABLE support in HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
support for various table-related insertion modes. This includes support
for tables, table rows, table cells, table column groups, etc...

Developed in https://github.com/wordpress/wordpress-develop/pull/6040
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58806


git-svn-id: http://core.svn.wordpress.org/trunk@58202 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-24 18:41:16 +00:00
dmsnell 8529edd888 HTML API: Fix unsupported insertion mode messages.
Insertion modes in an HTML parser may include instructions like "process
the token in the IN HEAD insertion mode." The rules do not change the
insertion mode of the parser, but the errors are triggered outside of the
rules for the current insertion mode. These will be misleading when
bailing on these instructions, because it will point someone to the wrong
place in the code to find the source of the error.

In this patch all of the bail-points due to lacking insertion mode support
are hard-coded to better orient someone to the section of the code lacking
support for handling the input HTML.

Developed in https://github.com/wordpress/wordpress-develop/pull/7043
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58679].

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58781


git-svn-id: http://core.svn.wordpress.org/trunk@58183 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-22 23:21:14 +00:00
dmsnell 471b482bec HTML API: Add missing tags in IN BODY insertion mode to HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
support for the remaining missing tags in the IN BODY insertion mode. Not
all of the added tags are supported, because in some cases they reset the
insertion mode and are reprocessed where they will be rejected.

This patch also improves the support of `get_modifiable_text()`, removing
a leading newline inside a LISTING, PRE, or TEXTAREA element.

Developed in https://github.com/WordPress/wordpress-develop/pull/6972
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell, westonruter.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58779


git-svn-id: http://core.svn.wordpress.org/trunk@58181 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-22 22:24:15 +00:00
dmsnell 51afccc65f HTML API: Add PHP type annotations.
This patch adds type annotations to internal and private methods of the HTML
API and the supporting WP_Token_Map. Annotations have not been added to the
public interfaces where it would likely crash a site if called wrong.

These annotations should help avoid unnecessary type-related bugs (as have
been uncovered in earlier work adding such annotations) and provide additional
guidance to developers when interacting with these classes in an IDE.

Developed in https://github.com/WordPress/wordpress-develop/pull/6753
Discussed in https://core.trac.wordpress.org/ticket/61399

Props dmsnell, jonsurrell.
See #61399.

Built from https://develop.svn.wordpress.org/trunk@58769


git-svn-id: http://core.svn.wordpress.org/trunk@58171 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-19 23:44:16 +00:00
dmsnell b7b504dc0d Fix phpdoc nullable types in some files.
It was found that in several places in the HTML API and its supporting files,
the wrong form of type annotation was used for optional parameters.

Instead of using `?type`, this patch uses `type|type-of-default-value` instead,
noting where important if the parameter is optional, and if so, what its default
value is.

Developed in https://github.com/WordPress/wordpress-develop/pull/7031
Discussed in https://core.trac.wordpress.org/ticket/61399

Props dmsnell, jonsurrell.
See #61399.

Built from https://develop.svn.wordpress.org/trunk@58742


git-svn-id: http://core.svn.wordpress.org/trunk@58144 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-17 00:53:17 +00:00
dmsnell 4f85cc258c HTML API: Add context to Unsupported_Exception class for improved debugging.
The HTML Processor internally throws an exception when it reaches HTML
that it knows it cannot process, but this exception is not made
available to calling code. It can be useful to extract more knowledge
about why it gave up, especially for debugging purposes.

In this patch, more context is added to the WP_HTML_Unsupported_Exception
and the last exception is made available to calling code through a new
method, `get_unsupported_exception()`.

Developed in https://github.com/WordPress/wordpress-develop/pull/6985
Discussed in https://core.trac.wordpress.org/ticket/61646

Props bernhard-reiter, dmsnell, jonsurrell.
See #61646.

Built from https://develop.svn.wordpress.org/trunk@58714


git-svn-id: http://core.svn.wordpress.org/trunk@58116 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-12 22:29:13 +00:00
dmsnell 519a8f6bbd HTML API: Simplify breadcrumb accounting.
Since the HTML Processor started visiting all nodes in a document, both
real and virtual, the breadcrumb accounting became a bit complicated
and it's not entirely clear that it is fully reliable.

In this patch the breadcrumbs are rebuilt separately from the stack of
open elements in order to eliminate the problem of the stateful stack
interactions and the post-hoc event queue.

Breadcrumbs are greatly simplified as a result, and more verifiably
correct, in this construction.

Developed in https://github.com/WordPress/wordpress-develop/pull/6981
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58590].

Props bernhard-reiter, dmsnell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58713


git-svn-id: http://core.svn.wordpress.org/trunk@58115 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-12 22:20:15 +00:00
dmsnell 85092d6493 HTML API: Correct node name in `generate_implied_end_tags()`.
The `generate_implied_end_tags()` algorithm has been comparing the
current node to a list of node names, which means that it won't ever
pop any elements from the stack of open elements.

This patch corrects the mistake by comparing node name against the
list, thus fixing the algorithm. This was noted in development work
for the 6.7 release.

Developed in https://github.com/WordPress/wordpress-develop/pull/6988
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58702


git-svn-id: http://core.svn.wordpress.org/trunk@58104 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-09 21:45:17 +00:00
dmsnell bccb3e46e1 HTML API: Respect `tag_name` query arg in `HTML_Processor::next_tag()`
Previously the HTML Processor was ignoring the `tag_name` argument in
the `next_tag()` query if it existed. This was wrong adn would lead to
calling code finding the very next tag, regardless of tag name, instead
of the requested taag.

This patch adds the tag name detection code into `next_tag()` to fix
the bug and ensure that `next_tag()` always returns only when finding
a tag of the given name.

Developed in https://github.com/WordPress/wordpress-develop/pull/6980
Discussed in https://core.trac.wordpress.org/ticket/61581

Follow-up to [56274].

Fixes #61581.

Built from https://develop.svn.wordpress.org/trunk@58681


git-svn-id: http://core.svn.wordpress.org/trunk@58083 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-06 10:20:15 +00:00
dmsnell e1c3eab4ce HTML API: Stub out remaining insertion modes in the HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
stubs for all of the remaining parser insertion modes in the HTML Processor.
These modes are not all supported yet, but they will be necessary to continue
adding support for other tags and markup.

Developed in https://github.com/WordPress/wordpress-develop/pull/6973
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58679


git-svn-id: http://core.svn.wordpress.org/trunk@58081 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-05 16:39:17 +00:00
dmsnell 411205cf87 HTML API: Support SELECT insertion mode.
As part of work to add more spec support to the HTML API, this patch adds
support for the SELECT, OPTION, and OPTGROUP elements, including the
requisite support for the IN SELECT insertion mode.

Developed in https://github.com/WordPress/wordpress-develop/pull/5908
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58677


git-svn-id: http://core.svn.wordpress.org/trunk@58079 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-05 00:52:18 +00:00
dmsnell bb1da1f82e HTML API: Add `current_node_is()` helper method to stack of open elements.
As part of work to add more spec support to the HTML API, this new method
will make it easier to implement the logic when in the SELECT and TABLE
insertion modes.

Developed in https://github.com/WordPress/wordpress-develop/pull/6968
Discussed in https://core.trac.wordpress.org/ticket/51576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58676


git-svn-id: http://core.svn.wordpress.org/trunk@58078 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-04 23:16:16 +00:00
dmsnell eda0d41f46 HTML API: Implement the _reset insertion mode appropriately_ algorithm.
In order to add support for the SELECT and TABLE tags in the HTML Processor, it
needs to implement the HTML algorithm named "reset the insertion mode
appropriately".

This patch implements that algorithm to unblock the additional tag support. The
algorithm resets the parsing mode after specific state changes in complicated
situations where alternative rules are in effect (such as rules governing how
the parser handles tags found within a TABLE element).

Developed in https://github.com/WordPress/wordpress-develop/pull/6020
Discussed in https://core.trac.wordpress.org/ticket/61549

Props dmsnell, jonsurrell.
Fixes #61549.

Built from https://develop.svn.wordpress.org/trunk@58656


git-svn-id: http://core.svn.wordpress.org/trunk@58070 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-03 17:07:15 +00:00
dmsnell ede77aca35 HTML API: Report breadcrumbs properly when visiting virtual nodes.
When [58304] introduced the abililty to visit virtual nodes in the HTML document,
those being the nodes which are implied by the HTML but no explicitly present in
the raw text, a bug was introduced in the `get_breadcrumbs()` method because it
wasn't updated to be aware of the virtual nodes. Therefore it would report the
wrong breadcrumbs for virtual nodes. Since the new `get_depth()` method is based
on the same logic it was also broken for virtual nodes.

In this patch, the breadcrumbs have been updated to account for the virtual nodes
and the depth method has been updated to rely on the fixed breadcrumb logic.

Developed in https://github.com/WordPress/wordpress-develop/pull/6914
Discussed in https://core.trac.wordpress.org/ticket/61348

Follow-up to [58304].

Props dmsnell, jonsurrell, zieladam.
See #61348.

Built from https://develop.svn.wordpress.org/trunk@58588


git-svn-id: http://core.svn.wordpress.org/trunk@58035 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-27 20:49:19 +00:00
dmsnell a636cd1c42 HTML API: Add missing subclass methods to HTML Processor and add token provenance.
This patch introduces two related changes:

 - It adds missing subclass methods on the HTML Processor which needed
   to be implemented since it started visiting virtual nodes. These
   methods need to account for the fact that not all tokens truly exist.

 - It adds a new concept and internal method, `is_virtual()`, indicating
   if the currently-matched token comes from the raw text in the input
   HTML document or if it was the byproduct of semantic parsing rules.
   This internal method and new vocabulary around token provenance
   considerably simplifies the logic spread throughout the rest of the
   class and its subclass methods.

Developed in https://github.com/WordPress/wordpress-develop/pull/6860
Discussed in https://core.trac.wordpress.org/ticket/61348

Follow-up to [58304].

Props dmsnell, jonsurrell, gziolo.
See #61348.

Built from https://develop.svn.wordpress.org/trunk@58558


git-svn-id: http://core.svn.wordpress.org/trunk@58006 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-25 03:11:19 +00:00
dmsnell 8683a502a0 HTML API: Call `$this->is_tag_closer()` in HTML Processor.
The HTML Processor had been calling the parent class `is_tag_closer()`
method, but since visiting virtual nodes was introduced, it's important
that all of the methods are called on the subclass.

This patch fixes one issue identified where the parent method was called
instead, and it fixes another case where the change from calling the
parent method to the `$this` method was done improperly.

Developed in https://github.com/WordPress/wordpress-develop/pull/6726
Discussed in https://core.trac.wordpress.org/ticket/61348

Props jonsurrell.
See #61348.
Follow-up to [58304].

Built from https://develop.svn.wordpress.org/trunk@58365


git-svn-id: http://core.svn.wordpress.org/trunk@57814 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-08 12:12:08 +00:00
dmsnell cf2a88a397 HTML API: Fix three small type-related bugs.
This applies three fixes that appeared as part of
adding type annotations to the HTML API.

Developed in https://github.com/WordPress/wordpress-develop/pull/6754
Discussed in https://core.trac.wordpress.org/ticket/61399

Props jonsurrell.
See #61399.
Follow-up to [55203], [56274].

Built from https://develop.svn.wordpress.org/trunk@58364


git-svn-id: http://core.svn.wordpress.org/trunk@57813 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-08 11:12:11 +00:00
dmsnell 4b8fa1bc8b HTML API: Return subclass from `::create_fragment`
When the `WP_HTML_Processor` was introduced with its `::create_fragment()`
static creator method, that method has been returning a `new self(...)`.
Unfortunately, this means that subclasses cannot use that method since it
will return the `WP_HTML_Processor` instead of the subclass.

With this patch, the static creator method returns `new static(...)` to preserve
the intended behavior. A new test asserts this behavior for future changes.

Developed in https://github.com/WordPress/wordpress-develop/pull/6729
Discussed in https://core.trac.wordpress.org/ticket/61374

Props dmsnell, jonsurrell.
Follow-up to [56274].
Fixes #61374.

Built from https://develop.svn.wordpress.org/trunk@58363


git-svn-id: http://core.svn.wordpress.org/trunk@57812 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-08 10:57:14 +00:00
dmsnell 2107b186fe HTML API: Report real and virtual nodes in the HTML Processor.
HTML is a kind of short-hand for a DOM structure. This means that there are
many cases in HTML where an element's opening tag or closing tag is missing (or
both). This is because many of the parsing rules imply creating elements in the
DOM which may not exist in the text of the HTML.

The HTML Processor, being the higher-level counterpart to the Tag Processor, is
already aware of these nodes, but since it's inception has not paused on them
when scanning through a document. Instead, these are visible when pausing on a
child of such an element, but otherwise not seen.

In this patch the HTML Processor starts exposing those implicitly-created nodes,
including opening tags, and closing tags, that aren't foudn in the text content
of the HTML input document.

Previously, the sequence of matched tokens when scanning with 
`WP_HTML_Processor::next_token()` would depend on how the HTML document was written,
but with this patch, all semantically equal HTML documents will parse and scan in
the same exact manner, presenting an idealized or "perfect" view of the document
the same way as would occur when traversing a DOM in a browser.

Developed in https://github.com/WordPress/wordpress-develop/pull/6348
Discussed in https://core.trac.wordpress.org/ticket/61348

Props audrasjb, dmsnell, gziolo, jonsurrell.
Fixes #61348.

Built from https://develop.svn.wordpress.org/trunk@58304


git-svn-id: http://core.svn.wordpress.org/trunk@57761 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-03 19:47:15 +00:00
dmsnell ec7c61e320 HTML API: Add `expects_closer()` method to HTML Processor
This patch adds a new method, `WP_HTML_Processor->expects_closer()` to indicate
if the currently-matched node expects to find a closing token. For example, a
`DIV` element expects a closing `</div>` tag, but an `<img>` expects none, because
it's a void element. Similarly, `#text` nodes and HTML comments only appear as
unitary nodes on the stack of open elements. Once proceeding further in the
document they are immediately removed without any closing tag.

This new method serves as a helper to indicate whether or not to expect the
closer, as this can be more complicated than it seems, and calling code
shouldn't have to build custom interpretations and implementations. Instead,
the HTML Processor ought to export its internal knowledge to make it easy for
consuming code and projects.

Developed in https://github.com/WordPress/wordpress-develop/pull/6600
Discussed in https://core.trac.wordpress.org/ticket/61257

Fixes #61257.
Props dmsnell, jonsurrell.


Built from https://develop.svn.wordpress.org/trunk@58192


git-svn-id: http://core.svn.wordpress.org/trunk@57655 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-05-24 01:21:10 +00:00
dmsnell 07a35d5d41 HTML API: Add method to report depth of currently-matched node.
The HTML Processor maintains a stack of open elements, where every element,
every `#text` node, every HTML comment, and other node is pushed and popped while
traversing the document. The "depth" of each of these nodes represents how deep
that stack is where the node appears. Unfortunately this information isn't
exposed to calling code, which has led different projects to attempt to
calculate this value externally. This isn't always trivial, but the HTML
Processor could make it so by exposing the internal knowledge in a new method.

In this patch the `get_current_depth()` method returns just that. Since the
processor always exists within a context, the depth includes nesting from the
always-present html element and also the body, since currently the HTML
Processor only supports parsing in the IN BODY context.

This means that the depth reported for the `DIV` in `<div>` is 3, not 1, because
its breadcrumbs path is `HTML > BODY > DIV`.

Developed in https://github.com/WordPress/wordpress-develop/pull/6589
Discussed in https://core.trac.wordpress.org/ticket/61255

Fixes #61255.
Props dmsnell, jonsurrell.

Built from https://develop.svn.wordpress.org/trunk@58191


git-svn-id: http://core.svn.wordpress.org/trunk@57654 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-05-23 23:37:12 +00:00
dmsnell c3b76697cd HTML API: Respect `class_name` query arg in `HTML_Processor::next_tag()`
Previously the HTML Process was ignoring the `class_name` argument in
the `next_tag()` query if it existed. This was wrong and would lead to
calling code finding the wrong tag if provided.

This patch adds the class detection code back into `next_tag()` to fix
the bug.

Developed in https://github.com/WordPress/wordpress-develop/pull/6618
Discussed in https://core.trac.wordpress.org/ticket/58517

See #58517.
Follow-up to [56274].
Props: dmsnell, jonsurrell.

Built from https://develop.svn.wordpress.org/trunk@58190


git-svn-id: http://core.svn.wordpress.org/trunk@57653 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-05-23 22:45:13 +00:00
dmsnell 6badd91e7b HTML API: Trigger active format reconstruction when reaching text nodes.
When encountering text nodes in an HTML document, the HTML parser needs
to run the active format reconstruction algorithm, even if it doesn't
stop to visit those text nodes. This is because the formats, which might
need reconstructing, will impact the breadcrumbs of all downstream nodes
from the text node.

In this patch, this process is triggered, which properly triggers the
active format reconstruction. It also enables the visiting of other token
types as is possible in the Tag Processor.

Developed in https://github.com/WordPress/wordpress-develop/pull/6054
Discussed in https://core.trac.wordpress.org/ticket/60170

Props: dmsnell, jonsurrell, westonruter.
Fixes: #60455.
Follow-up to: [57348].


Built from https://develop.svn.wordpress.org/trunk@57806


git-svn-id: http://core.svn.wordpress.org/trunk@57307 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-03-12 00:24:09 +00:00
dmsnell 9e0a116dbf HTML API: Ensure that breadcrumbs are properly retained after seeking.
In some cases, it's possible to seek back into a location found inside
an element which has been closed before the point in the document where
the `seek()` was made. In these cases the breadcrumb stack is lost, and
calling `get_breadcrumbs()` after the seek will return the wrong information.

In this patch, the HTML Processor takes a conservative approach and
moves to the front of the document, then reparses the document until
it reaches the sought-after location. This ensures consistency on
the stack of open elements and active formats, and preserves
breadcrumbs.

Developed in https://github.com/WordPress/wordpress-develop/pull/6185
Discussed in https://core.trac.wordpress.org/ticket/60687

Props jonsurrell.
Follow-up to [60687].
See #58517.
Fixes #60687.


Built from https://develop.svn.wordpress.org/trunk@57768


git-svn-id: http://core.svn.wordpress.org/trunk@57269 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-03-05 05:34:11 +00:00
dmsnell fd1cbd8980 HTML API: Add subclassed has_bookmark() and fix seek()
The WP_HTML_Processor::has_bookmark() method has not correctly reported bookmarks
which have been set, because it wraps the given bookmark names when setting them.
Additionally, WP_HTML_Processor::seek() does not seek to correct location if HTML
has been updated because it wasn't flushing enqueued updates to the document.

In this patch both problems are resolved and added tests guard these behaviors
against future regressions.

Developed in https://github.com/WordPress/wordpress-develop/pull/6039
Discussed in https://core.trac.wordpress.org/ticket/60474

Follow-up to [56274].
Props dmsnell, jonsurrell.
Fixes #60474.


Built from https://develop.svn.wordpress.org/trunk@57582


git-svn-id: http://core.svn.wordpress.org/trunk@57083 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-02-10 22:54:13 +00:00
dmsnell 9fb7267542 HTML API: Fix typo setting the wrong self-closing flag.
The HTML Processor tracks whether a token was found with the self-closing flag.
Depending on the context, this flag may or may not indicate that the element is
self closing. Unfortunately it's been tracking the wrong flag: it's been tracking
the end-tag flag, which indicates that a token is an end tag.

In this patch the right flag is set in the HTML Processor. This hasn't been an
issue because the HTML Processor doesn't yet read that stored flag, but it's an
important fix to make before adding support for foreign content (SVG and MathML)
since that behavior depends on reading the correct flag.

Follow-up to [56274].

Props dmsnell.


Built from https://develop.svn.wordpress.org/trunk@57528


git-svn-id: http://core.svn.wordpress.org/trunk@57029 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-02-02 23:27:14 +00:00
dmsnell 0dc136bcb9 HTML API: Fix void tag nesting with next_token
When `next_token()` was introduced, it introduced a regression in the HTML
Processor whereby void tags remain on the stack of open elements when they
shouldn't. This led to invalid values returned from `get_breadcrumbs()`.

The reason was that calling `next_token()` works through a different code path
than the HTML Processor runs everything else. To solve this, its sub-classed
`next_token()` called `step( self::REPROCESS_CURRENT_TOKEN )` so that the proper
HTML accounting takes place.

Unfortunately that same reprocessing code path skipped the step whereby void
and self-closing elements are popped from the stack of open elements.

In this patch, that step is run with a third mode for `step()`, which is the
new `self::PROCESS_CURRENT_TOKEN`. This mode acts as if `self::PROCESS_NEXT_NODE`
were called, except it doesn't advance the parser.

Developed in https://github.com/WordPress/wordpress-develop/pull/5975
Discussed in https://core.trac.wordpress.org/ticket/60382

Follow-up to [57348]

Props dmsnell, jonsurrell
Fixes #60382


Built from https://develop.svn.wordpress.org/trunk@57507


git-svn-id: http://core.svn.wordpress.org/trunk@57008 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-02-01 00:43:15 +00:00
dmsnell 62e0ef411b HTML API: Scan all syntax tokens in a document, read modifiable text.
Since its introduction in WordPress 6.2 the HTML Tag Processor has
provided a way to scan through all of the HTML tags in a document and
then read and modify their attributes. In order to reliably do this, it
also needed to be aware of other kinds of HTML syntax, but it didn't
expose those syntax tokens to consumers of the API.

In this patch the Tag Processor introduces a new scanning method and a
few helper methods to read information about or from each token. Most
significantly, this introduces the ability to read `#text` nodes in the
document.

What's new in the Tag Processor?
================================

 - `next_token()` visits every distinct syntax token in a document.
 - `get_token_type()` indicates what kind of token it is.
 - `get_token_name()` returns something akin to `DOMNode.nodeName`.
 - `get_modifiable_text()` returns the text associated with a token.
 - `get_comment_type()` indicates why a token represents an HTML comment.

Example usage.
==============

{{{
<?php
function strip_all_tags( $html ) {
        $text_content = '';
        $processor    = new WP_HTML_Tag_Processor( $html );

        while ( $processor->next_token() ) {
                if ( '#text' !== $processor->get_token_type() ) {
                        continue;
                }

                $text_content .= $processor->get_modifiable_text();
        }

        return $text_content;
}
}}}

What changes in the Tag Processor?
==================================

Previously, the Tag Processor would scan the opening and closing tag of
every HTML element separately. Now, however, there are special tags
which it only visits once, as if those elements were void tags without
a closer.

These are special tags because their content contains no other HTML or
markup, only non-HTML content.

 - SCRIPT elements contain raw text which is isolated from the rest of
   the HTML document and fed separately into a JavaScript engine. There
   are complicated rules to avoid escaping the script context in the HTML.
   The contents are left verbatim, and character references are not decoded.

 - TEXTARA and TITLE elements contain plain text which is decoded
   before display, e.g. transforming `&amp;` into `&`. Any markup which
   resembles tags is treated as verbatim text and not a tag.

 - IFRAME, NOEMBED, NOFRAMES, STYLE, and XMP elements are similar to the
   textarea and title elements, but no character references are decoded.
   For example, `&amp;` inside a STYLE element is passed to the CSS engine
   as the literal string `&amp;` and _not_ as `&`.

Because it's important not treat this inner content separately from the
elements containing it, the Tag Processor combines them when scanning
into a single match and makes their content available as modifiable
text (see below).

This means that the Tag Processor will no longer visit a closing tag for
any of these elements unless that tag is unexpected.

{{{
    <title>There is only a single token in this line</title>
    <title>There are two tokens in this line></title></title>
    </title><title>There are still two tokens in this line></title>
}}}

What are tokens?
================

The term "token" here is a parsing term, which means a primitive unit in
HTML. There are only a few kinds of tokens in HTML:

 - a tag has a name, attributes, and a closing or self-closing flag.
 - a text node, or `#text` node contains plain text which is displayed
   in a browser and which is decoded before display.
 - a DOCTYPE declaration indicates how to parse the document.
 - a comment is hidden from the display on a page but present in the HTML.

There are a few more kinds of tokens that the HTML Tag Processor will
recognize, some of which don't exist as concepts in HTML. These mostly
comprise XML syntax elements that aren't part of HTML (such as CDATA and
processing instructions) and invalid HTML syntax that transforms into
comments.

What is a funky comment?
========================

This patch treats a specific kind of invalid comment in a special way.
A closing tag with an invalid name is considered a "funky comment." In
the browser these become HTML comments just like any other, but their
syntax is convenient for representing a variety of bits of information
in a well-defined way and which cannot be nested or recursive, given
the parsing rules handling this invalid syntax.

 - `</1>`
 - `</%avatar_url>`
 - `</{"wp_bit": {"type": "post-author"}}>`
 - `</[post-author]>`
 - `</__( 'Save Post' );>`

All of these examples become HTML comments in the browser. The content
inside the funky content is easily parsable, whereby the only rule is
that it starts at the `<` and continues until the nearest `>`. There
can be no funky comment inside another, because that would imply having
a `>` inside of one, which would actually terminate the first one.

What is modifiable text?
========================

Modifiable text is similar to the `innerText` property of a DOM node.
It represents the span of text for a given token which may be modified
without changing the structure of the HTML document or the token.

There is currently no mechanism to change the modifiable text, but this
is planned to arrive in a later patch.

Tags
====

Most tags have no modifiable text because they have child nodes where
text nodes are found. Only the special tags mentioned above have
modifiable text.

{{{
    <div class="post">Another day in HTML</div>
    └─ tag ──────────┘└─ text node ─────┘└────┴─ tag
}}}

{{{
    <title>Is <img> &gt; <image>?</title>
    │      └ modifiable text ───┘       │ "Is <img> > <image>?"
    └─ tag ─────────────────────────────┘
}}}

Text nodes
==========

Text nodes are entirely modifiable text.

{{{
    This HTML document has no tags.
    └─ modifiable text ───────────┘
}}}

Comments
========

The modifiable text inside a comment is the portion of the comment that
doesn't form its syntax. This applies for a number of invalid comments.

{{{
    <!-- this is inside a comment -->
    │   └─ modifiable text ──────┘  │
    └─ comment token ───────────────┘
}}}

{{{
    <!-->
    This invalid comment has no modifiable text.
}}}

{{{
    <? this is an invalid comment -->
    │ └─ modifiable text ────────┘  │
    └─ comment token ───────────────┘
}}}

{{{
    <[CDATA[this is an invalid comment]]>
    │       └─ modifiable text ───────┘ │
    └─ comment token ───────────────────┘
}}}

Other token types also have modifiable text. Consult the code or tests
for further information.

Developed in https://github.com/WordPress/wordpress-develop/pull/5683
Discussed in https://core.trac.wordpress.org/ticket/60170

Follows [57575]

Props bernhard-reiter, dlh, dmsnell, jonsurrell, zieladam
Fixes #60170


Built from https://develop.svn.wordpress.org/trunk@57348


git-svn-id: http://core.svn.wordpress.org/trunk@56854 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-01-24 23:37:16 +00:00
dmsnell 7f22c4d4ea HTML API: Support INPUT tags.
Adds support for the following HTML elements to the HTML Processor:

 - INPUT

Previously this element was not supported and the HTML Processor would bail when encountering one. Now, with this patch applied, it will proceed to parse the HTML document.

Developed in https://github.com/WordPress/wordpress-develop/pull/5907
Discussed in https://core.trac.wordpress.org/ticket/60283

Props jonsurrell
See #60283


Built from https://develop.svn.wordpress.org/trunk@57343


git-svn-id: http://core.svn.wordpress.org/trunk@56849 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-01-24 03:30:14 +00:00