Commit Graph

76 Commits

Author SHA1 Message Date
dmsnell ed654177cd HTML API: Switch to HTML namespace when entering Integration Points.
When encountering inline SVG and MathML content in an HTML document, there are certain "integration points" which transition back into the HTML parsing ruleset. Previously, the HTML API was incorrectly switching into the namespace of the element transitioning into that ruleset.

In this patch, the correct transition is made, where all integration points refer to HTML rules, while non-integration points refer to the rules of the namespace corresponding to the token itself.

Developed in https://github.com/wordpress/wordpress-develop/pull/7425
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@59099


git-svn-id: http://core.svn.wordpress.org/trunk@58495 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-27 00:44:19 +00:00
dmsnell 5b3b3f7df2 HTML API: Add `normalize()` to give us the HTML we always wanted.
HTML often appears in ways that are unexpected. It may be missing implicit tags, may have unquoted, single-quoted, or double-quoted attributes, may contain duplicate attributes, may contain unescaped text content, or any number of other possible invalid constructions. The HTML API understands all fo these inputs, but downline parsers may not, and HTML snippets which are safe on their own may introduce problems when joined with other HTML snippets.

This patch introduces the `serialize()` method on the HTML Processor, which prints a fully-normative HTML output, eliminating invalid markup along the way. It produces a string which contains every missing tag, double-quoted attributes, and no duplicates. A `normalize()` static method on the HTML Processor provides a convenient wrapper for constructing a fragment parser and immediately serializing.

Subclasses relying on the `serialize_token()` method may perform structural HTML modifications with as much security as the upcoming `\Dom\HTMLDocument()` parser will, though these are not
able to provide the full safety that will eventually appear with `set_inner_html()`.

Further work may explore serializing to XML (which involves a number of other important transformations) and adding constraints to serialization (such as only allowing inline/flow/formatting elements and text).

Developed in https://github.com/wordpress/wordpress-develop/pull/7331
Discussed in https://core.trac.wordpress.org/ticket/62036

Props dmsnell, jonsurrell, westonruter.
Fixes #62036.

Built from https://develop.svn.wordpress.org/trunk@59076


git-svn-id: http://core.svn.wordpress.org/trunk@58472 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-20 22:32:17 +00:00
hellofromTonya 37e081e8dc Code Modernization: Fix implicitly nullable parameter in WP_HTML_Processor.
PHP 8.4 deprecates implicitly nullable parameters, i.e. typed parameters with a `null` default value, which are not explicitly declared as nullable.

This commit the one instance of this in the `WP_HTML_Processor` class.

Fixed by adding the nullability operator to the type, which is supported since PHP 7.1, so we can use it now the minimum supported PHP version is PHP 7.2.

As this deprecation is thrown at compile time, it can be seen at the top of the test output when running on PHP 8.4 (which will be gone once this change has been committed). It is not possible to write a test to cover this.

Ref: https://wiki.php.net/rfc/deprecate-implicitly-nullable-types

Follow-up to [58867], [58769], [58304], [58192].

Props jrf.
See #62061.
Built from https://develop.svn.wordpress.org/trunk@59053


git-svn-id: http://core.svn.wordpress.org/trunk@58449 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-18 15:04:14 +00:00
dmsnell 3aeb2980f0 HTML API: Prevent infinite loop in foreign content reprocessing step.
An infinite loop was discovered in specific situations within foreign content inside the HTML Processor when a given node inside foreign content must be handled in the rules for the current insertion mode.

This patch resolves the loop by handling those nodes directly instead of reprocessing the node, which previously was redirecting control flow back to where the loop started.

Developed in https://github.com/wordpress/wordpress-develop/7347
Discussed in https://core.trac.wordpress.org/ticket/61656

Follow-up to [58868].

Props jonsurrell.
See #61576.


Built from https://develop.svn.wordpress.org/trunk@59024


git-svn-id: http://core.svn.wordpress.org/trunk@58420 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-16 17:57:13 +00:00
dmsnell 7d9bb7efd3 HTML API: Make WP_HTML_Processor::get_tag() namespace aware.
The HTML specification indicates that an HTML tag with the name "IMAGE"
should be renamed as "IMG" and handled as if it were an "IMG", but this
only applies to elements in the HTML namespace.

In this patch the HTML Processor is updated to ensure that it doesn't
remap the tag name when processing foreign content, such as SVG and
MathML markup.

Developed in https://github.com/wordpress/wordpress-develop/7330
Discussed in https://core.trac.wordpress.org/ticket/61656

Props dmsnell, jonsurrell.
See #61576.


Built from https://develop.svn.wordpress.org/trunk@59014


git-svn-id: http://core.svn.wordpress.org/trunk@58410 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-11 16:13:17 +00:00
dmsnell 011d45abbc HTML API: Add sentinels for unreachable code.
There are places in the HTML API code where some tools get confused and flag invalid types for the return of a function because they are unable to detect that the end of the function is unreachable.

Since PHP doesn't provide a way to encode total matching in the source code, this patch adds a few extra lines in those unreachable locations to satisfy any tooling which isn't able to fully analyze the code.

Additionally this serves as extra guarding in case someone changes these functions in a way which would break them and the existing test suite doesn't catch those breakages.

Developed in https://github.com/WordPress/wordpress-develop/pull/7315
Discussed in https://core.trac.wordpress.org/ticket/62018

Props dlh, dmsnell.
Fixes #62018.

Built from https://develop.svn.wordpress.org/trunk@59001


git-svn-id: http://core.svn.wordpress.org/trunk@58397 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-09 22:12:15 +00:00
dmsnell 72bc81c048 HTML API: Only examine HTML nodes in `pop_until()` instack of open elements.
The `pop_until( $tag_name )` method in the stack of open elements should only be examining HTML elements, but it has only been checking the tag name. This has led to closing the wrong tags when run from inside foreign content. A very specific situation where this may arise is when a `TEMPLATE` closer is found inside foreign content, inside another template.

{{{
HTML:template   SVG:template                 HTML:/template
<template><svg><template><foreignObject><div></template><div>
╰──< this outer TEMPLATE is closed by this one >───╯
}}}

This patch constains the method to checking for elements matching the tag name which are in the HTML namespace so that the proper detection occurs.

Developed in https://github.com/WordPress/wordpress-develop/pull/7286
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58867].

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58992


git-svn-id: http://core.svn.wordpress.org/trunk@58388 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-04 19:25:14 +00:00
dmsnell d0d8ba24d8 HTML API: Respect document compat mode when handling CSS class names.
The HTML API has been behaving as if CSS class name selectors matched class names in an ASCII case-insensitive manner. This is only true if the document in question is set to quirks mode. Unfortunately most documents processed will be set to no-quirks mode, meaning that some CSS behaviors have been matching incorrectly when provided with case variants of class names.

In this patch, the CSS methods have been audited and updated to adhere to the rules governing ASCII case sensitivity when matching classes. This includes `add_class()`, `remove_class()`, `has_class()`, and `class_list()`. Now, it is assumed that a document is in no-quirks mode unless a full HTML parser infers quirks mode, and these methods will treat class names in a byte-for-byte manner. Otherwise, when a document is in quirks mode, the methods will compare the provided class names against existing class names for the tag in an ASCII case insensitive way, while `class_list()` will return a lower-cased version of the existing class names.

The lower-casing in `class_list()` is performed for consistency, since it's possible that multiple case variants of the same comparable class name exists on a tag in the input HTML.

Developed in https://github.com/WordPress/wordpress-develop/pull/7169
Discussed in https://core.trac.wordpress.org/ticket/61531

Props dmsnell, jonsurrell.
See #61531.

Built from https://develop.svn.wordpress.org/trunk@58985


git-svn-id: http://core.svn.wordpress.org/trunk@58381 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-04 04:34:15 +00:00
dmsnell 2ce04ceab9 HTML API: Ensure that NULL and whitespace-only CDATA sections don't forbid FRAMESET.
When CDATA sections (which can only occur inside SVG and MathML content) consist only of NULL bytes or whitespace characters they should not clear the "frameset ok" flag. Previously they have always been clearing this flag, but in this patch the logic is updated to detect these sequences properly.

Developed in https://github.com/WordPress/wordpress-develop/pull/7230
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58867].

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58977


git-svn-id: http://core.svn.wordpress.org/trunk@58373 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-03 19:50:14 +00:00
dmsnell 8d2a10b300 HTML API: Allow subdividing text nodes by meaningful prefixes.
HTML parsing rules at times differentiate character tokens that are all null bytes, all whitespace, or other content. This patch introduces a new function which may be used to classify text node sub-regions and lead to more efficient application of these parsing rules.

Further, when classified in this way, application code may skip some rules and decoding entirely, improving performance. For example, this can be used to ease the implementation of skipping inter-element whitespace, which is usually not rendered.

Developed in https://github.com/WordPress/wordpress-develop/pull/7236
Discussed in https://core.trac.wordpress.org/ticket/61974

Props dmsnell, jonsurrell.
Fixes #61974.

Built from https://develop.svn.wordpress.org/trunk@58970


git-svn-id: http://core.svn.wordpress.org/trunk@58366 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-02 23:21:18 +00:00
dmsnell eac35f6ae6 HTML API: Add missing NOBR end tag handling to HTML Processor.
When the HTML Processor was introduced, an oversight led to a missing case for handling a closing `NOBR` tag. The `NOBR` element is a deprecated tag and should not be used.

This patch adds the missing case so that the deprecated `NOBR` end tag is appropriately handled.

Developed in https://github.com/WordPress/wordpress-develop/pull/7282
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [56274].

Props jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58967


git-svn-id: http://core.svn.wordpress.org/trunk@58363 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-02 22:03:15 +00:00
dmsnell d7f9d30df8 HTML API: Fix logic bug in HTML Processor when opening A element.
A mistake in the original code handling opening A elements in the HTML Processor led to mistakes in parsing where the Processor would bail in situations when it could have proceeded. While this was errant behavior, it didn't violate the public contract since it would bail in these situations.

This patch fixes the mistake, which was to only break out of the innermost loop instead of breaking from the containing loop, which resolves the issue.

Developed in https://github.com/WordPress/wordpress-develop/pull/7281
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [56274].

Props jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58966


git-svn-id: http://core.svn.wordpress.org/trunk@58362 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-09-02 21:54:15 +00:00
dmsnell 813d784931 HTML API: Fix a bug where the namespace was forced to 'html'
While working on other reviews and audits, a bug was discovered in the HTML API where the wrong namespace was being assigned to a token because the default value of 'html' was used. This patch fixes the bug by calling the `parent::get_namespace()` method instead of assuming 'html'.

Developed in https://github.com/wordpress/wordpress-develop/7232
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58868].

Props jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58940


git-svn-id: http://core.svn.wordpress.org/trunk@58336 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-27 03:37:12 +00:00
dmsnell 01d2199622 HTML API: Add support for missing FRAMESET and "after" insertion modes.
As part of work to add more spec support to the HTML API, this patch adds support for the FRAMESET-related insertion modes, as well as the set of missing after insertion modes. These modes run at the end of parsing a document, closing it and taking care of any lingering tags.

Developed in https://github.com/wordpress/wordpress-develop/7165
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58926


git-svn-id: http://core.svn.wordpress.org/trunk@58322 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-23 15:42:17 +00:00
dmsnell e3a3c13497 HTML API: Parse DOCTYPE tokens and set HTML parser mode accordingly.
This patch adds until-now missing code to parse the structure of HTML DOCTYPE declarations. The DOCTYPE is mostly unused but can dictate the document compatability mode, which governs whether CSS class names match in a ASCII-case-insensitive way or not, and whether TABLE elements close an open P element.

The DOCTYPE information is made available through a new method on the Tag Processor, `get_doctype_info()`.

Developed in https://github.com/wordpress/wordpress-develop/pull/7195
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58925


git-svn-id: http://core.svn.wordpress.org/trunk@58321 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-23 14:55:15 +00:00
dmsnell 5cd8048fc2 HTML API: Make "reset insertion mode appropriately" algorithm private.
The algorithm is described and referred to in the specification as
"reset the insertion mode appropriately." This patch renames it to
more accurately reflect the specification, and it turns the public
method private. As an internal algorithm from the HTML specification
it should not be open to extension.

Developed in https://github.com/wordpress/wordpress-develop/pull/7194
Discussed in https://core.trac.wordpress.org/ticket/61549

Follow-up to [28656].
Props jonsurrell.
See #61549.

Built from https://develop.svn.wordpress.org/trunk@58898


git-svn-id: http://core.svn.wordpress.org/trunk@58294 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-14 21:19:16 +00:00
dmsnell 98e51518b3 HTML API: Remove completed TODO comments.
This patch removes TODO comments indicating the need to verify certain behaviors and algorithms. Those verifications have taken place and the comments are no longer needed.

Developed in https://github.com/wordpress/wordpress-develop/pull/7174
Discussed in https://core.trac.wordpress.org/ticket/61646

Follow-up to [58867], [58870].

Props jonsurrell.
See #64646.

Built from https://develop.svn.wordpress.org/trunk@58877


git-svn-id: http://core.svn.wordpress.org/trunk@58273 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-10 05:00:14 +00:00
dmsnell c834d20762 HTML API: Test and fix SVG script handling.
When support was added for foreign content, an ambiguity in the HTML specification led to code that followed the wrong path when encountering a self-closing SCRIPT element in the SVG namespace. Further, a fallthrough was discovered during manual testing.

This patch adds a new test to assert the proper behaviors and fixes these issues. In the case of the SCRIPT element, the outcome was the same with the wrong code path, making the defect benign. In the case of the fallthrough, the wrong behavior would occur.

The updates in this patch also resolves a todo relating to the spec ambiguity.

Developed in https://github.com/wordpress/wordpress-develop/pull/7164
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58868].

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58871


git-svn-id: http://core.svn.wordpress.org/trunk@58267 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-08 17:04:19 +00:00
dmsnell 7611ec9415 HTML API: `expect_closer()` should report false for self-closing foreign elements.
Previously, `WP_HTML_Processor::expects_closer()` would report `true` for self-closing foreign elements when called without supplying a node in question, but it should have been reporting `true` just as it does for HTML elements.

This patch adds a test case demonstrating the issue and a bugfix.

The `html5lib` test runner was relying on the incorrect behavior, accidentally working. This is also corrected and the `html5lib` test now relies on the correct behavior of `expects_closer()`.

Developed in https://github.com/wordpress/wordpress-develop/pull/7162
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58868].

Props: dmsnell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58870


git-svn-id: http://core.svn.wordpress.org/trunk@58266 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-08 16:15:20 +00:00
dmsnell 1ffe7a456c HTML API: Add support for SVG and MathML (Foreign content)
As part of work to add more spec support to the HTML API, this patch adds
support for SVG and MathML elements, or more generally, "foreign content."

The rules in foreign content are a mix of XML and HTML parsing rules and
introduce additional complexity into the processor, but is important in
order to avoid getting lost when inside these elements.

Developed in https://github.com/wordpress/wordpress-develop/pull/6006
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell, westonruter.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58867


git-svn-id: http://core.svn.wordpress.org/trunk@58263 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-08 07:25:15 +00:00
dmsnell 90a66c7ad1 HTML API: Add support for IN SELECT IN TABLE parsing.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN SELECT IN TABLE insertion mode. This small section of the
spec handles rules for the `<select>` element and its children when found
inside of a `<table>`.

Developed in https://github.com/wordpress/wordpress-develop/pull/7044
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58841


git-svn-id: http://core.svn.wordpress.org/trunk@58237 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-01 22:53:16 +00:00
dmsnell 40114e8ed5 HTML API: Add support for IN CAPTION parsing.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN CAPTION insertion mode. This small section of the
spec handles rules for the `<caption>` element.

Developed in https://github.com/wordpress/wordpress-develop/pull/7041
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58840


git-svn-id: http://core.svn.wordpress.org/trunk@58236 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-01 22:36:13 +00:00
dmsnell 5cfd2dee00 HTML API: Add support for IN COLUMN GROUP parsing.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN COLUMN GROUP insertion mode. This small section of the
spec handles rules for the `<colgroup>` element.

Developed in https://github.com/wordpress/wordpress-develop/pull/7042
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58839


git-svn-id: http://core.svn.wordpress.org/trunk@58235 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-08-01 22:06:12 +00:00
dmsnell af6e4904af HTML API: Introduce full parsing mode in HTML Processor.
The HTML Processor has only supported a specific kind of parsing mode
called _the fragment parsing mode_, where it behaves in the same way
that `node.innerHTML = html` does in the DOM. This mode assumes a
context node and doesn't support parsing an entire document.

As part of work to add more spec support to the HTML API, this patch
introduces a full parsing mode, which can parse a full HTML document
from start to end, including the doctype declaration and head tags.

Developed in https://github.com/wordpress/wordpress-develop/pull/6977
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58836


git-svn-id: http://core.svn.wordpress.org/trunk@58232 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-31 16:56:15 +00:00
dmsnell f483d6dc4e HTML API: Add TEMPLATE and related support in HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
support for the IN TEMPLATE and IN HEAD insertion modes. These changes are
primarily about adding support for TEMPLATE elements in the HTML Processor,
but include support for other tags commonly found in the document head, such
as LINK, META, SCRIPT, STYLE, and TITLE.

Developed in https://github.com/wordpress/wordpress-develop/pull/7046
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell, westonruter.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58833


git-svn-id: http://core.svn.wordpress.org/trunk@58229 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-30 18:46:35 +00:00
dmsnell d8ab99f84b HTML API: Close all elements at the end of a document.
When the model of breadcrumb generation in the HTML Processor and node
traversal was simplified, the change introduced a bug whereby unclosed
nodes at the end of a document would remain unvisited and unclosed.

In this patch, a fix is applied to ensure that all open elements close
while traversing a document. A couple of minor documentation typos are
fixed in the patch as well.

Developed in https://github.com/wordpress/wordpress-develop/pull/7085
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58713].

Props: dmsnell, gziolo, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58828


git-svn-id: http://core.svn.wordpress.org/trunk@58224 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-29 17:39:17 +00:00
dmsnell 539ee9221b HTML API: Add TABLE support in HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
support for various table-related insertion modes. This includes support
for tables, table rows, table cells, table column groups, etc...

Developed in https://github.com/wordpress/wordpress-develop/pull/6040
Discussed in https://core.trac.wordpress.org/ticket/61576

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58806


git-svn-id: http://core.svn.wordpress.org/trunk@58202 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-24 18:41:16 +00:00
dmsnell 8529edd888 HTML API: Fix unsupported insertion mode messages.
Insertion modes in an HTML parser may include instructions like "process
the token in the IN HEAD insertion mode." The rules do not change the
insertion mode of the parser, but the errors are triggered outside of the
rules for the current insertion mode. These will be misleading when
bailing on these instructions, because it will point someone to the wrong
place in the code to find the source of the error.

In this patch all of the bail-points due to lacking insertion mode support
are hard-coded to better orient someone to the section of the code lacking
support for handling the input HTML.

Developed in https://github.com/wordpress/wordpress-develop/pull/7043
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58679].

Props: dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58781


git-svn-id: http://core.svn.wordpress.org/trunk@58183 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-22 23:21:14 +00:00
dmsnell 471b482bec HTML API: Add missing tags in IN BODY insertion mode to HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
support for the remaining missing tags in the IN BODY insertion mode. Not
all of the added tags are supported, because in some cases they reset the
insertion mode and are reprocessed where they will be rejected.

This patch also improves the support of `get_modifiable_text()`, removing
a leading newline inside a LISTING, PRE, or TEXTAREA element.

Developed in https://github.com/WordPress/wordpress-develop/pull/6972
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell, westonruter.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58779


git-svn-id: http://core.svn.wordpress.org/trunk@58181 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-22 22:24:15 +00:00
dmsnell 51afccc65f HTML API: Add PHP type annotations.
This patch adds type annotations to internal and private methods of the HTML
API and the supporting WP_Token_Map. Annotations have not been added to the
public interfaces where it would likely crash a site if called wrong.

These annotations should help avoid unnecessary type-related bugs (as have
been uncovered in earlier work adding such annotations) and provide additional
guidance to developers when interacting with these classes in an IDE.

Developed in https://github.com/WordPress/wordpress-develop/pull/6753
Discussed in https://core.trac.wordpress.org/ticket/61399

Props dmsnell, jonsurrell.
See #61399.

Built from https://develop.svn.wordpress.org/trunk@58769


git-svn-id: http://core.svn.wordpress.org/trunk@58171 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-19 23:44:16 +00:00
dmsnell b7b504dc0d Fix phpdoc nullable types in some files.
It was found that in several places in the HTML API and its supporting files,
the wrong form of type annotation was used for optional parameters.

Instead of using `?type`, this patch uses `type|type-of-default-value` instead,
noting where important if the parameter is optional, and if so, what its default
value is.

Developed in https://github.com/WordPress/wordpress-develop/pull/7031
Discussed in https://core.trac.wordpress.org/ticket/61399

Props dmsnell, jonsurrell.
See #61399.

Built from https://develop.svn.wordpress.org/trunk@58742


git-svn-id: http://core.svn.wordpress.org/trunk@58144 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-17 00:53:17 +00:00
dmsnell 4f85cc258c HTML API: Add context to Unsupported_Exception class for improved debugging.
The HTML Processor internally throws an exception when it reaches HTML
that it knows it cannot process, but this exception is not made
available to calling code. It can be useful to extract more knowledge
about why it gave up, especially for debugging purposes.

In this patch, more context is added to the WP_HTML_Unsupported_Exception
and the last exception is made available to calling code through a new
method, `get_unsupported_exception()`.

Developed in https://github.com/WordPress/wordpress-develop/pull/6985
Discussed in https://core.trac.wordpress.org/ticket/61646

Props bernhard-reiter, dmsnell, jonsurrell.
See #61646.

Built from https://develop.svn.wordpress.org/trunk@58714


git-svn-id: http://core.svn.wordpress.org/trunk@58116 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-12 22:29:13 +00:00
dmsnell 519a8f6bbd HTML API: Simplify breadcrumb accounting.
Since the HTML Processor started visiting all nodes in a document, both
real and virtual, the breadcrumb accounting became a bit complicated
and it's not entirely clear that it is fully reliable.

In this patch the breadcrumbs are rebuilt separately from the stack of
open elements in order to eliminate the problem of the stateful stack
interactions and the post-hoc event queue.

Breadcrumbs are greatly simplified as a result, and more verifiably
correct, in this construction.

Developed in https://github.com/WordPress/wordpress-develop/pull/6981
Discussed in https://core.trac.wordpress.org/ticket/61576

Follow-up to [58590].

Props bernhard-reiter, dmsnell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58713


git-svn-id: http://core.svn.wordpress.org/trunk@58115 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-12 22:20:15 +00:00
dmsnell 85092d6493 HTML API: Correct node name in `generate_implied_end_tags()`.
The `generate_implied_end_tags()` algorithm has been comparing the
current node to a list of node names, which means that it won't ever
pop any elements from the stack of open elements.

This patch corrects the mistake by comparing node name against the
list, thus fixing the algorithm. This was noted in development work
for the 6.7 release.

Developed in https://github.com/WordPress/wordpress-develop/pull/6988
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58702


git-svn-id: http://core.svn.wordpress.org/trunk@58104 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-09 21:45:17 +00:00
dmsnell bccb3e46e1 HTML API: Respect `tag_name` query arg in `HTML_Processor::next_tag()`
Previously the HTML Processor was ignoring the `tag_name` argument in
the `next_tag()` query if it existed. This was wrong adn would lead to
calling code finding the very next tag, regardless of tag name, instead
of the requested taag.

This patch adds the tag name detection code into `next_tag()` to fix
the bug and ensure that `next_tag()` always returns only when finding
a tag of the given name.

Developed in https://github.com/WordPress/wordpress-develop/pull/6980
Discussed in https://core.trac.wordpress.org/ticket/61581

Follow-up to [56274].

Fixes #61581.

Built from https://develop.svn.wordpress.org/trunk@58681


git-svn-id: http://core.svn.wordpress.org/trunk@58083 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-06 10:20:15 +00:00
dmsnell e1c3eab4ce HTML API: Stub out remaining insertion modes in the HTML Processor.
As part of work to add more spec support to the HTML API, this patch adds
stubs for all of the remaining parser insertion modes in the HTML Processor.
These modes are not all supported yet, but they will be necessary to continue
adding support for other tags and markup.

Developed in https://github.com/WordPress/wordpress-develop/pull/6973
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58679


git-svn-id: http://core.svn.wordpress.org/trunk@58081 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-05 16:39:17 +00:00
dmsnell 411205cf87 HTML API: Support SELECT insertion mode.
As part of work to add more spec support to the HTML API, this patch adds
support for the SELECT, OPTION, and OPTGROUP elements, including the
requisite support for the IN SELECT insertion mode.

Developed in https://github.com/WordPress/wordpress-develop/pull/5908
Discussed in https://core.trac.wordpress.org/ticket/61576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58677


git-svn-id: http://core.svn.wordpress.org/trunk@58079 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-05 00:52:18 +00:00
dmsnell bb1da1f82e HTML API: Add `current_node_is()` helper method to stack of open elements.
As part of work to add more spec support to the HTML API, this new method
will make it easier to implement the logic when in the SELECT and TABLE
insertion modes.

Developed in https://github.com/WordPress/wordpress-develop/pull/6968
Discussed in https://core.trac.wordpress.org/ticket/51576

Props dmsnell, jonsurrell.
See #61576.

Built from https://develop.svn.wordpress.org/trunk@58676


git-svn-id: http://core.svn.wordpress.org/trunk@58078 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-04 23:16:16 +00:00
dmsnell eda0d41f46 HTML API: Implement the _reset insertion mode appropriately_ algorithm.
In order to add support for the SELECT and TABLE tags in the HTML Processor, it
needs to implement the HTML algorithm named "reset the insertion mode
appropriately".

This patch implements that algorithm to unblock the additional tag support. The
algorithm resets the parsing mode after specific state changes in complicated
situations where alternative rules are in effect (such as rules governing how
the parser handles tags found within a TABLE element).

Developed in https://github.com/WordPress/wordpress-develop/pull/6020
Discussed in https://core.trac.wordpress.org/ticket/61549

Props dmsnell, jonsurrell.
Fixes #61549.

Built from https://develop.svn.wordpress.org/trunk@58656


git-svn-id: http://core.svn.wordpress.org/trunk@58070 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-07-03 17:07:15 +00:00
dmsnell ede77aca35 HTML API: Report breadcrumbs properly when visiting virtual nodes.
When [58304] introduced the abililty to visit virtual nodes in the HTML document,
those being the nodes which are implied by the HTML but no explicitly present in
the raw text, a bug was introduced in the `get_breadcrumbs()` method because it
wasn't updated to be aware of the virtual nodes. Therefore it would report the
wrong breadcrumbs for virtual nodes. Since the new `get_depth()` method is based
on the same logic it was also broken for virtual nodes.

In this patch, the breadcrumbs have been updated to account for the virtual nodes
and the depth method has been updated to rely on the fixed breadcrumb logic.

Developed in https://github.com/WordPress/wordpress-develop/pull/6914
Discussed in https://core.trac.wordpress.org/ticket/61348

Follow-up to [58304].

Props dmsnell, jonsurrell, zieladam.
See #61348.

Built from https://develop.svn.wordpress.org/trunk@58588


git-svn-id: http://core.svn.wordpress.org/trunk@58035 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-27 20:49:19 +00:00
dmsnell a636cd1c42 HTML API: Add missing subclass methods to HTML Processor and add token provenance.
This patch introduces two related changes:

 - It adds missing subclass methods on the HTML Processor which needed
   to be implemented since it started visiting virtual nodes. These
   methods need to account for the fact that not all tokens truly exist.

 - It adds a new concept and internal method, `is_virtual()`, indicating
   if the currently-matched token comes from the raw text in the input
   HTML document or if it was the byproduct of semantic parsing rules.
   This internal method and new vocabulary around token provenance
   considerably simplifies the logic spread throughout the rest of the
   class and its subclass methods.

Developed in https://github.com/WordPress/wordpress-develop/pull/6860
Discussed in https://core.trac.wordpress.org/ticket/61348

Follow-up to [58304].

Props dmsnell, jonsurrell, gziolo.
See #61348.

Built from https://develop.svn.wordpress.org/trunk@58558


git-svn-id: http://core.svn.wordpress.org/trunk@58006 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-25 03:11:19 +00:00
dmsnell 8683a502a0 HTML API: Call `$this->is_tag_closer()` in HTML Processor.
The HTML Processor had been calling the parent class `is_tag_closer()`
method, but since visiting virtual nodes was introduced, it's important
that all of the methods are called on the subclass.

This patch fixes one issue identified where the parent method was called
instead, and it fixes another case where the change from calling the
parent method to the `$this` method was done improperly.

Developed in https://github.com/WordPress/wordpress-develop/pull/6726
Discussed in https://core.trac.wordpress.org/ticket/61348

Props jonsurrell.
See #61348.
Follow-up to [58304].

Built from https://develop.svn.wordpress.org/trunk@58365


git-svn-id: http://core.svn.wordpress.org/trunk@57814 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-08 12:12:08 +00:00
dmsnell cf2a88a397 HTML API: Fix three small type-related bugs.
This applies three fixes that appeared as part of
adding type annotations to the HTML API.

Developed in https://github.com/WordPress/wordpress-develop/pull/6754
Discussed in https://core.trac.wordpress.org/ticket/61399

Props jonsurrell.
See #61399.
Follow-up to [55203], [56274].

Built from https://develop.svn.wordpress.org/trunk@58364


git-svn-id: http://core.svn.wordpress.org/trunk@57813 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-08 11:12:11 +00:00
dmsnell 4b8fa1bc8b HTML API: Return subclass from `::create_fragment`
When the `WP_HTML_Processor` was introduced with its `::create_fragment()`
static creator method, that method has been returning a `new self(...)`.
Unfortunately, this means that subclasses cannot use that method since it
will return the `WP_HTML_Processor` instead of the subclass.

With this patch, the static creator method returns `new static(...)` to preserve
the intended behavior. A new test asserts this behavior for future changes.

Developed in https://github.com/WordPress/wordpress-develop/pull/6729
Discussed in https://core.trac.wordpress.org/ticket/61374

Props dmsnell, jonsurrell.
Follow-up to [56274].
Fixes #61374.

Built from https://develop.svn.wordpress.org/trunk@58363


git-svn-id: http://core.svn.wordpress.org/trunk@57812 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-08 10:57:14 +00:00
dmsnell 2107b186fe HTML API: Report real and virtual nodes in the HTML Processor.
HTML is a kind of short-hand for a DOM structure. This means that there are
many cases in HTML where an element's opening tag or closing tag is missing (or
both). This is because many of the parsing rules imply creating elements in the
DOM which may not exist in the text of the HTML.

The HTML Processor, being the higher-level counterpart to the Tag Processor, is
already aware of these nodes, but since it's inception has not paused on them
when scanning through a document. Instead, these are visible when pausing on a
child of such an element, but otherwise not seen.

In this patch the HTML Processor starts exposing those implicitly-created nodes,
including opening tags, and closing tags, that aren't foudn in the text content
of the HTML input document.

Previously, the sequence of matched tokens when scanning with 
`WP_HTML_Processor::next_token()` would depend on how the HTML document was written,
but with this patch, all semantically equal HTML documents will parse and scan in
the same exact manner, presenting an idealized or "perfect" view of the document
the same way as would occur when traversing a DOM in a browser.

Developed in https://github.com/WordPress/wordpress-develop/pull/6348
Discussed in https://core.trac.wordpress.org/ticket/61348

Props audrasjb, dmsnell, gziolo, jonsurrell.
Fixes #61348.

Built from https://develop.svn.wordpress.org/trunk@58304


git-svn-id: http://core.svn.wordpress.org/trunk@57761 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-06-03 19:47:15 +00:00
dmsnell ec7c61e320 HTML API: Add `expects_closer()` method to HTML Processor
This patch adds a new method, `WP_HTML_Processor->expects_closer()` to indicate
if the currently-matched node expects to find a closing token. For example, a
`DIV` element expects a closing `</div>` tag, but an `<img>` expects none, because
it's a void element. Similarly, `#text` nodes and HTML comments only appear as
unitary nodes on the stack of open elements. Once proceeding further in the
document they are immediately removed without any closing tag.

This new method serves as a helper to indicate whether or not to expect the
closer, as this can be more complicated than it seems, and calling code
shouldn't have to build custom interpretations and implementations. Instead,
the HTML Processor ought to export its internal knowledge to make it easy for
consuming code and projects.

Developed in https://github.com/WordPress/wordpress-develop/pull/6600
Discussed in https://core.trac.wordpress.org/ticket/61257

Fixes #61257.
Props dmsnell, jonsurrell.


Built from https://develop.svn.wordpress.org/trunk@58192


git-svn-id: http://core.svn.wordpress.org/trunk@57655 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-05-24 01:21:10 +00:00
dmsnell 07a35d5d41 HTML API: Add method to report depth of currently-matched node.
The HTML Processor maintains a stack of open elements, where every element,
every `#text` node, every HTML comment, and other node is pushed and popped while
traversing the document. The "depth" of each of these nodes represents how deep
that stack is where the node appears. Unfortunately this information isn't
exposed to calling code, which has led different projects to attempt to
calculate this value externally. This isn't always trivial, but the HTML
Processor could make it so by exposing the internal knowledge in a new method.

In this patch the `get_current_depth()` method returns just that. Since the
processor always exists within a context, the depth includes nesting from the
always-present html element and also the body, since currently the HTML
Processor only supports parsing in the IN BODY context.

This means that the depth reported for the `DIV` in `<div>` is 3, not 1, because
its breadcrumbs path is `HTML > BODY > DIV`.

Developed in https://github.com/WordPress/wordpress-develop/pull/6589
Discussed in https://core.trac.wordpress.org/ticket/61255

Fixes #61255.
Props dmsnell, jonsurrell.

Built from https://develop.svn.wordpress.org/trunk@58191


git-svn-id: http://core.svn.wordpress.org/trunk@57654 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-05-23 23:37:12 +00:00
dmsnell c3b76697cd HTML API: Respect `class_name` query arg in `HTML_Processor::next_tag()`
Previously the HTML Process was ignoring the `class_name` argument in
the `next_tag()` query if it existed. This was wrong and would lead to
calling code finding the wrong tag if provided.

This patch adds the class detection code back into `next_tag()` to fix
the bug.

Developed in https://github.com/WordPress/wordpress-develop/pull/6618
Discussed in https://core.trac.wordpress.org/ticket/58517

See #58517.
Follow-up to [56274].
Props: dmsnell, jonsurrell.

Built from https://develop.svn.wordpress.org/trunk@58190


git-svn-id: http://core.svn.wordpress.org/trunk@57653 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-05-23 22:45:13 +00:00
dmsnell 6badd91e7b HTML API: Trigger active format reconstruction when reaching text nodes.
When encountering text nodes in an HTML document, the HTML parser needs
to run the active format reconstruction algorithm, even if it doesn't
stop to visit those text nodes. This is because the formats, which might
need reconstructing, will impact the breadcrumbs of all downstream nodes
from the text node.

In this patch, this process is triggered, which properly triggers the
active format reconstruction. It also enables the visiting of other token
types as is possible in the Tag Processor.

Developed in https://github.com/WordPress/wordpress-develop/pull/6054
Discussed in https://core.trac.wordpress.org/ticket/60170

Props: dmsnell, jonsurrell, westonruter.
Fixes: #60455.
Follow-up to: [57348].


Built from https://develop.svn.wordpress.org/trunk@57806


git-svn-id: http://core.svn.wordpress.org/trunk@57307 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-03-12 00:24:09 +00:00
dmsnell 9e0a116dbf HTML API: Ensure that breadcrumbs are properly retained after seeking.
In some cases, it's possible to seek back into a location found inside
an element which has been closed before the point in the document where
the `seek()` was made. In these cases the breadcrumb stack is lost, and
calling `get_breadcrumbs()` after the seek will return the wrong information.

In this patch, the HTML Processor takes a conservative approach and
moves to the front of the document, then reparses the document until
it reaches the sought-after location. This ensures consistency on
the stack of open elements and active formats, and preserves
breadcrumbs.

Developed in https://github.com/WordPress/wordpress-develop/pull/6185
Discussed in https://core.trac.wordpress.org/ticket/60687

Props jonsurrell.
Follow-up to [60687].
See #58517.
Fixes #60687.


Built from https://develop.svn.wordpress.org/trunk@57768


git-svn-id: http://core.svn.wordpress.org/trunk@57269 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-03-05 05:34:11 +00:00