Fixes the issue when an HTML_Processor bookmark was set at a virtual token (a node in the resulting document that does not correspond to an HTML token present in the input string), seek behavior was unreliable.
Props jonsurrell, gziolo.
Fixes#62521.
Built from https://develop.svn.wordpress.org/trunk@59502
git-svn-id: http://core.svn.wordpress.org/trunk@58888 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML specification does not close HTML or BODY tags (pop them off the stack of open elements) when their tag closers are encountered. The HTML processor correctly handled this behavior, however it incorrectly "paused" by returning true from the step functions.
Props jonsurrell, dmsnell, gziolo.
Fixes#62583.
Built from https://develop.svn.wordpress.org/trunk@59500
git-svn-id: http://core.svn.wordpress.org/trunk@58886 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The current implementation of `create_fragment` (and the underlying `create_fragment_at_current_node`) allows passing in a context that might result in a tree that cannot be represented by HTML. For example, a user might use `<p>` as context, and attempt to create a fragment that also consists of a paragraph element, `<p>like this`. This would result in a paragraph node nested inside another -- something that can never result from parsing HTML.
To prevent this, this changeset makes `create_fragment_at_current_node` private and limits `create_fragment` to only `<body>` as context, while a comprehensive solution to allow other contexts is being worked on.
Follow-up to [59444], [59467].
Props jonsurrell, dmsnell, bernhard-reiter.
Fixes#62584.
Built from https://develop.svn.wordpress.org/trunk@59469
git-svn-id: http://core.svn.wordpress.org/trunk@58855 1a063a9b-81f0-0310-95a4-ce76da25c4cd
This changeset modifies `WP_HTML_Processor::create_fragment( $html, $context )` to use a full processor and `create_fragment_at_node` instead of the other way around. This makes more sense and makes the main factory methods more clear, where the state required for fragments is set up in `create_fragment_at_node` instead of in both `create_fragment` and `create_fragment_at_current_node`.
This allows for more HTML contexts to be provided to the basic `create_fragment` where the provided context HTML is appended to `<!DOCTYPE html>`, a full processor is created, the last tag opener is found, and a fragment parser is created at that node via `create_fragment_at_current_node`.
The HTML5lib tests are updated accordingly to use this new method to create fragments.
Props jonsurrell, dmsnell, bernhard-reiter.
Fixes#62584.
Built from https://develop.svn.wordpress.org/trunk@59467
git-svn-id: http://core.svn.wordpress.org/trunk@58853 1a063a9b-81f0-0310-95a4-ce76da25c4cd
HTML Fragment parsing always happens with a context node, which may impact how a fragment of HTML is parsed. HTML Fragment Processors can be instantiated with a `BODY` context node via `WP_HTML_Processor::create_fragment( $html )`.
This changeset adds a static method called `create_fragment_at_current_node( string $html_fragment )`. It can only be called when the processor is paused at a `#tag`, with some additional constraints:
- The opening and closing tags must appear in the HTML input (no virtual tokens).
- No "self-contained" elements are allowed ( `IFRAME`, `SCRIPT`, `TITLE`, etc.).
If successful, the method will return a `WP_HTML_Processor` instance whose context is inherited from the node that the method was called from.
Props jonsurrell, bernhard-reiter, gziolo.
Fixes#62357.
Built from https://develop.svn.wordpress.org/trunk@59444
git-svn-id: http://core.svn.wordpress.org/trunk@58830 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Changeset [59399] fixed missing DOCTYPEs in normalized HTML output. It missed an edge case where public and system identifiers may contain double quotes, in which case they must be quoted with single quotes.
This commit addresses that issue and adds tests.
Follow-up to [59399].
Props jonsurrell, luisherranz, apermo.
Fixes#62396.
Built from https://develop.svn.wordpress.org/trunk@59410
git-svn-id: http://core.svn.wordpress.org/trunk@58796 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Output DOCTYPE when calling `WP_HTML_Processor::serialize` on a full document that includes a DOCTYPE.
The DOCTYPE should be included in the serialized/normalized HTML output as it has an impact in how the document is handled, in particular whether the document should be handled in quirks or no-quirks mode.
This only affects the serialization of full parsers at this time because DOCTYPE tokens are currently ignored in all possible fragments. The omission of the DOCTYPE is subtle but can change the serialized document's quirks/no-quirks mode.
Props jonsurrell.
Fixes#62396.
Built from https://develop.svn.wordpress.org/trunk@59399
git-svn-id: http://core.svn.wordpress.org/trunk@58785 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When the HTML Processor seeks to an earlier place, it returns the the beginning of the document and proceeds forward until it reaches the appropriate location. This requires resetting internal state so that the processor can correctly proceed from the beginning of the document.
The seeking reset logic was not adapted to account for the full processor (i.e. when created via `WP_HTML_Processor::create_full_parser()`). This change updates the seek logic to account for the full and fragment parsers as well as other state that has been introduced in the interim and should be reset.
Props jonsurrell, dmsnell, westonruter, mi5t4n.
Fixes#62290.
Built from https://develop.svn.wordpress.org/trunk@59391
git-svn-id: http://core.svn.wordpress.org/trunk@58777 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Break out logic from the `next_token()` method into a private method which may call itself recursively. This allows for subclasses to override the `next_token()` method and be assured that each call to `next_token()` corresponds with the consumption of one single token. This also parallels how `WP_HTML_Tag_Processor::next_token()` wraps a private `base_class_next_token()` method.
Props westonruter, jonsurrell.
Fixes#62269.
Built from https://develop.svn.wordpress.org/trunk@59285
git-svn-id: http://core.svn.wordpress.org/trunk@58677 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When encountering inline SVG and MathML content in an HTML document, there are certain "integration points" which transition back into the HTML parsing ruleset. Previously, the HTML API was incorrectly switching into the namespace of the element transitioning into that ruleset.
In this patch, the correct transition is made, where all integration points refer to HTML rules, while non-integration points refer to the rules of the namespace corresponding to the token itself.
Developed in https://github.com/wordpress/wordpress-develop/pull/7425
Discussed in https://core.trac.wordpress.org/ticket/61576
Props dmsnell, jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@59099
git-svn-id: http://core.svn.wordpress.org/trunk@58495 1a063a9b-81f0-0310-95a4-ce76da25c4cd
HTML often appears in ways that are unexpected. It may be missing implicit tags, may have unquoted, single-quoted, or double-quoted attributes, may contain duplicate attributes, may contain unescaped text content, or any number of other possible invalid constructions. The HTML API understands all fo these inputs, but downline parsers may not, and HTML snippets which are safe on their own may introduce problems when joined with other HTML snippets.
This patch introduces the `serialize()` method on the HTML Processor, which prints a fully-normative HTML output, eliminating invalid markup along the way. It produces a string which contains every missing tag, double-quoted attributes, and no duplicates. A `normalize()` static method on the HTML Processor provides a convenient wrapper for constructing a fragment parser and immediately serializing.
Subclasses relying on the `serialize_token()` method may perform structural HTML modifications with as much security as the upcoming `\Dom\HTMLDocument()` parser will, though these are not
able to provide the full safety that will eventually appear with `set_inner_html()`.
Further work may explore serializing to XML (which involves a number of other important transformations) and adding constraints to serialization (such as only allowing inline/flow/formatting elements and text).
Developed in https://github.com/wordpress/wordpress-develop/pull/7331
Discussed in https://core.trac.wordpress.org/ticket/62036
Props dmsnell, jonsurrell, westonruter.
Fixes#62036.
Built from https://develop.svn.wordpress.org/trunk@59076
git-svn-id: http://core.svn.wordpress.org/trunk@58472 1a063a9b-81f0-0310-95a4-ce76da25c4cd
PHP 8.4 deprecates implicitly nullable parameters, i.e. typed parameters with a `null` default value, which are not explicitly declared as nullable.
This commit the one instance of this in the `WP_HTML_Processor` class.
Fixed by adding the nullability operator to the type, which is supported since PHP 7.1, so we can use it now the minimum supported PHP version is PHP 7.2.
As this deprecation is thrown at compile time, it can be seen at the top of the test output when running on PHP 8.4 (which will be gone once this change has been committed). It is not possible to write a test to cover this.
Ref: https://wiki.php.net/rfc/deprecate-implicitly-nullable-types
Follow-up to [58867], [58769], [58304], [58192].
Props jrf.
See #62061.
Built from https://develop.svn.wordpress.org/trunk@59053
git-svn-id: http://core.svn.wordpress.org/trunk@58449 1a063a9b-81f0-0310-95a4-ce76da25c4cd
There are places in the HTML API code where some tools get confused and flag invalid types for the return of a function because they are unable to detect that the end of the function is unreachable.
Since PHP doesn't provide a way to encode total matching in the source code, this patch adds a few extra lines in those unreachable locations to satisfy any tooling which isn't able to fully analyze the code.
Additionally this serves as extra guarding in case someone changes these functions in a way which would break them and the existing test suite doesn't catch those breakages.
Developed in https://github.com/WordPress/wordpress-develop/pull/7315
Discussed in https://core.trac.wordpress.org/ticket/62018
Props dlh, dmsnell.
Fixes#62018.
Built from https://develop.svn.wordpress.org/trunk@59001
git-svn-id: http://core.svn.wordpress.org/trunk@58397 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The `pop_until( $tag_name )` method in the stack of open elements should only be examining HTML elements, but it has only been checking the tag name. This has led to closing the wrong tags when run from inside foreign content. A very specific situation where this may arise is when a `TEMPLATE` closer is found inside foreign content, inside another template.
{{{
HTML:template SVG:template HTML:/template
<template><svg><template><foreignObject><div></template><div>
╰──< this outer TEMPLATE is closed by this one >───╯
}}}
This patch constains the method to checking for elements matching the tag name which are in the HTML namespace so that the proper detection occurs.
Developed in https://github.com/WordPress/wordpress-develop/pull/7286
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58867].
Props dmsnell, jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58992
git-svn-id: http://core.svn.wordpress.org/trunk@58388 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML API has been behaving as if CSS class name selectors matched class names in an ASCII case-insensitive manner. This is only true if the document in question is set to quirks mode. Unfortunately most documents processed will be set to no-quirks mode, meaning that some CSS behaviors have been matching incorrectly when provided with case variants of class names.
In this patch, the CSS methods have been audited and updated to adhere to the rules governing ASCII case sensitivity when matching classes. This includes `add_class()`, `remove_class()`, `has_class()`, and `class_list()`. Now, it is assumed that a document is in no-quirks mode unless a full HTML parser infers quirks mode, and these methods will treat class names in a byte-for-byte manner. Otherwise, when a document is in quirks mode, the methods will compare the provided class names against existing class names for the tag in an ASCII case insensitive way, while `class_list()` will return a lower-cased version of the existing class names.
The lower-casing in `class_list()` is performed for consistency, since it's possible that multiple case variants of the same comparable class name exists on a tag in the input HTML.
Developed in https://github.com/WordPress/wordpress-develop/pull/7169
Discussed in https://core.trac.wordpress.org/ticket/61531
Props dmsnell, jonsurrell.
See #61531.
Built from https://develop.svn.wordpress.org/trunk@58985
git-svn-id: http://core.svn.wordpress.org/trunk@58381 1a063a9b-81f0-0310-95a4-ce76da25c4cd
HTML parsing rules at times differentiate character tokens that are all null bytes, all whitespace, or other content. This patch introduces a new function which may be used to classify text node sub-regions and lead to more efficient application of these parsing rules.
Further, when classified in this way, application code may skip some rules and decoding entirely, improving performance. For example, this can be used to ease the implementation of skipping inter-element whitespace, which is usually not rendered.
Developed in https://github.com/WordPress/wordpress-develop/pull/7236
Discussed in https://core.trac.wordpress.org/ticket/61974
Props dmsnell, jonsurrell.
Fixes#61974.
Built from https://develop.svn.wordpress.org/trunk@58970
git-svn-id: http://core.svn.wordpress.org/trunk@58366 1a063a9b-81f0-0310-95a4-ce76da25c4cd
A mistake in the original code handling opening A elements in the HTML Processor led to mistakes in parsing where the Processor would bail in situations when it could have proceeded. While this was errant behavior, it didn't violate the public contract since it would bail in these situations.
This patch fixes the mistake, which was to only break out of the innermost loop instead of breaking from the containing loop, which resolves the issue.
Developed in https://github.com/WordPress/wordpress-develop/pull/7281
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [56274].
Props jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58966
git-svn-id: http://core.svn.wordpress.org/trunk@58362 1a063a9b-81f0-0310-95a4-ce76da25c4cd
When support was added for foreign content, an ambiguity in the HTML specification led to code that followed the wrong path when encountering a self-closing SCRIPT element in the SVG namespace. Further, a fallthrough was discovered during manual testing.
This patch adds a new test to assert the proper behaviors and fixes these issues. In the case of the SCRIPT element, the outcome was the same with the wrong code path, making the defect benign. In the case of the fallthrough, the wrong behavior would occur.
The updates in this patch also resolves a todo relating to the spec ambiguity.
Developed in https://github.com/wordpress/wordpress-develop/pull/7164
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58868].
Props: dmsnell, jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58871
git-svn-id: http://core.svn.wordpress.org/trunk@58267 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Previously, `WP_HTML_Processor::expects_closer()` would report `true` for self-closing foreign elements when called without supplying a node in question, but it should have been reporting `true` just as it does for HTML elements.
This patch adds a test case demonstrating the issue and a bugfix.
The `html5lib` test runner was relying on the incorrect behavior, accidentally working. This is also corrected and the `html5lib` test now relies on the correct behavior of `expects_closer()`.
Developed in https://github.com/wordpress/wordpress-develop/pull/7162
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58868].
Props: dmsnell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58870
git-svn-id: http://core.svn.wordpress.org/trunk@58266 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML Processor has only supported a specific kind of parsing mode
called _the fragment parsing mode_, where it behaves in the same way
that `node.innerHTML = html` does in the DOM. This mode assumes a
context node and doesn't support parsing an entire document.
As part of work to add more spec support to the HTML API, this patch
introduces a full parsing mode, which can parse a full HTML document
from start to end, including the doctype declaration and head tags.
Developed in https://github.com/wordpress/wordpress-develop/pull/6977
Discussed in https://core.trac.wordpress.org/ticket/61576
Props: dmsnell, jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58836
git-svn-id: http://core.svn.wordpress.org/trunk@58232 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Insertion modes in an HTML parser may include instructions like "process
the token in the IN HEAD insertion mode." The rules do not change the
insertion mode of the parser, but the errors are triggered outside of the
rules for the current insertion mode. These will be misleading when
bailing on these instructions, because it will point someone to the wrong
place in the code to find the source of the error.
In this patch all of the bail-points due to lacking insertion mode support
are hard-coded to better orient someone to the section of the code lacking
support for handling the input HTML.
Developed in https://github.com/wordpress/wordpress-develop/pull/7043
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58679].
Props: dmsnell, jonsurrell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58781
git-svn-id: http://core.svn.wordpress.org/trunk@58183 1a063a9b-81f0-0310-95a4-ce76da25c4cd
The HTML Processor internally throws an exception when it reaches HTML
that it knows it cannot process, but this exception is not made
available to calling code. It can be useful to extract more knowledge
about why it gave up, especially for debugging purposes.
In this patch, more context is added to the WP_HTML_Unsupported_Exception
and the last exception is made available to calling code through a new
method, `get_unsupported_exception()`.
Developed in https://github.com/WordPress/wordpress-develop/pull/6985
Discussed in https://core.trac.wordpress.org/ticket/61646
Props bernhard-reiter, dmsnell, jonsurrell.
See #61646.
Built from https://develop.svn.wordpress.org/trunk@58714
git-svn-id: http://core.svn.wordpress.org/trunk@58116 1a063a9b-81f0-0310-95a4-ce76da25c4cd
Since the HTML Processor started visiting all nodes in a document, both
real and virtual, the breadcrumb accounting became a bit complicated
and it's not entirely clear that it is fully reliable.
In this patch the breadcrumbs are rebuilt separately from the stack of
open elements in order to eliminate the problem of the stateful stack
interactions and the post-hoc event queue.
Breadcrumbs are greatly simplified as a result, and more verifiably
correct, in this construction.
Developed in https://github.com/WordPress/wordpress-develop/pull/6981
Discussed in https://core.trac.wordpress.org/ticket/61576
Follow-up to [58590].
Props bernhard-reiter, dmsnell.
See #61576.
Built from https://develop.svn.wordpress.org/trunk@58713
git-svn-id: http://core.svn.wordpress.org/trunk@58115 1a063a9b-81f0-0310-95a4-ce76da25c4cd
gziolo
Bernhard Reiter
cbravobernal
czapla
Weston Ruter
dmsnell
hellofromTonya