Commit Graph

152 Commits

Author SHA1 Message Date
dmsnell 141ba4ff59 HTML API: Add support for HR element.
Adds support for the following HTML elements to the HTML Processor:

 - HR

Previously, this element was not supported and the HTML Processor would bail when encountering
it. Now, with this patch, it will proceed to parse an HTML document when encountering one.

Developed in WordPress/wordpress-develop#5897

Props jonsurrell, dmsnell
Fixes #60283


Built from https://develop.svn.wordpress.org/trunk@57314


git-svn-id: http://core.svn.wordpress.org/trunk@56820 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-01-19 19:03:09 +00:00
dmsnell 29dd837333 HTML API: Add support for list elements.
Adds support for the following HTML elements to the HTML Processor:

 - LI, OL, UL.
 - DD, DL, DT.

Previously, these elements were not supported and the HTML Processor would bail when encountering them.
With this patch it will proceed to parse an HTML document when encountering those tags as long as other normal conditions don't cause it to bail (such as complicated format reconstruction).

Props audrasjb, jonsurrell, bernhard-reiter.
Fixes #60215.


Built from https://develop.svn.wordpress.org/trunk@57264


git-svn-id: http://core.svn.wordpress.org/trunk@56770 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-01-10 14:05:17 +00:00
Bernhard Reiter 372e3c4b97 HTML API: Add explicit handling or failure for all tags.
The HTML API HTML processor does not yet support all tags. Many tags (e.g. list elements) have some complicated rules in the [https://html.spec.whatwg.org/#parsing-main-inbody "in body" insertion mode].

Implementing these special rules is blocking the implementation for a catch-all rule for "any other tag" because we need to prevent special rules from being handled by the catch-all.

  Any other start tag
  Reconstruct the active formatting elements, if any.

  Insert an HTML element for the token.

  …

This change ensures the HTML Processor fails when handling special tags. This is the same as existing behavior, but will allow us to implement the catch-all "any other tag" handling without unintentionally handling special elements.

Additionally, we add tests that assert the special elements are unhandled. As these tags are implemented, this should help to ensure they're removed from the unsupported tag list.

Props jonsurrell, dmsnell.
Fixes #60092.
Built from https://develop.svn.wordpress.org/trunk@57248


git-svn-id: http://core.svn.wordpress.org/trunk@56754 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2024-01-08 14:05:24 +00:00
Sergey Biryukov 13f7ee3063 Coding Standards: Correct alignment in `WP_HTML_Tag_Processor::apply_attributes_updates()`.
This fixes an `Equals sign not aligned correctly` WPCS warning.

Follow-up to [57179].

Props antonvlasenko, dmsnell, ironprogrammer.
Fixes #60078.
Built from https://develop.svn.wordpress.org/trunk@57227


git-svn-id: http://core.svn.wordpress.org/trunk@56733 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-12-25 14:03:19 +00:00
Bernhard Reiter 8e5db640de HTML API: Avoid processing incomplete tokens.
Currently the Tag Processor assumes that an input document is a ''full'' HTML document. Because of this, if there's lingering content after the last tag match it will treat that content as plaintext and skip over it. This is fine for the Tag Processor because if there is lingering content that isn't a valid tag then there's nothing for `next_tag()` to match.

However, in order to support a number of feature expansions it is important to recognize that the remaining content ''may'' involve partial syntax elements, such as incomplete tags, attributes, or comments.

In this patch we're adding a mode inside the Tag Processor which will flip when we start parsing HTML syntax but the document finishes before the token does. This will provide the ability to:

- extend the input document,
- avoid misinterpreting syntax as text, and
- guess if we have a complete document, know if we have an incomplete document.

In the process of building this patch a few fixes were identified and fixed in the Tag Processor, namely in the handling of incomplete syntax elements.

Props dmsnell, jonsurrell.
Fixes #60122, #60108.
Built from https://develop.svn.wordpress.org/trunk@57211


git-svn-id: http://core.svn.wordpress.org/trunk@56717 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-12-20 17:52:30 +00:00
Bernhard Reiter a586b85d8e HTML API: Apply linting changes to `@TODO` comments.
Lowercase `@TODO` comments introduced by [57186], and remove spurious colons after them.

Props dmsnell, TobiasBg, mukesh27, sergeybiryukov, jonsurrell.
Fixes #60060.
Built from https://develop.svn.wordpress.org/trunk@57209


git-svn-id: http://core.svn.wordpress.org/trunk@56715 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-12-20 12:36:31 +00:00
Bernhard Reiter 511dcb7520 HTML API: Add support for H1-H6 elements in the HTML Processor.
Previously these have been unsupported, but in this patch, support is added for the tags so that the HTML Processor can process documents containing them.

There was a design discussion about introducing a constant to communicate "any of the H1 - H6 elements" but this posed a number of challenges that don't need to be answered in this patch. For the time being, because the HTML specification treats H1 - H6 specially as a single kind of element, the HTML Processor uses an internal hard-coded string to indicate this. By using a hard-coded string it's possible to avoid introducing a class constant which cannot be made private due to PHP's class design. In the future, this will probably appear as a special constant in a new constant-containing class.

Props dmsnell, jonsurrell.
Fixes #60060.
Built from https://develop.svn.wordpress.org/trunk@57186


git-svn-id: http://core.svn.wordpress.org/trunk@56697 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-12-13 17:53:19 +00:00
zieladam 760fae6c43 HTML API: Track spans of text with (offset, length) instead of (start, end).
Updates the internal representation of the text span coordinates. The mixture of (offset, length) and (start, end) coordinates becomes confusing, this commit replaces it with a (offset, length) pair. There should be no functional or behavioral changes in this patch. For the internal helper classes this patch introduces breaking changes, but those classes are marked private and should not be used outside of the HTML API itself.

Props dmsnell.
Fixes #59993.


Built from https://develop.svn.wordpress.org/trunk@57179


git-svn-id: http://core.svn.wordpress.org/trunk@56690 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-12-10 13:19:28 +00:00
Joe McGill 46406afcc4 HTML-API: Prevent unintended behavior when WP_HTML_Token is unserialized.
Props dmsnell, peterwilsoncc, dd32, xknown, rawrly, johnbillion, barry, jeffpaul, vortfu, isabel_brison, mikeschroder, jorbin.

Built from https://develop.svn.wordpress.org/trunk@57163


git-svn-id: http://core.svn.wordpress.org/trunk@56674 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-12-06 16:05:19 +00:00
Bernhard Reiter b5ac702f60 HTML API: Fix typo in documentation example.
The example code in the PHPDoc comment for the HTML Tag Processor class
previously showed calling `next_tag()` with an array containing a `class`
key, which should have been `class_name`. This patch fixes this by using
the appropriate `class_name` key.

Props dmsnell, gaambo, crstauf, atachibana, audrasjb, krupalpanchal.
Fixes #59891.
Built from https://develop.svn.wordpress.org/trunk@57116


git-svn-id: http://core.svn.wordpress.org/trunk@56627 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-11-17 06:40:24 +00:00
Bernhard Reiter b86334b447 HTML API: Add support for containers elements, including ARTICLE.
There are a handful of elements which behave similarly and are generically container elements. These are the following elements:

    ADDRESS, ARTICLE, ASIDE, BLOCKQUOTE, CENTER, DETAILS, DIALOG, DIR,
    DL, DIV, FIELDSET, FIGCAPTION, FIGURE, FOOTER, HEADER, HGROUP, MAIN,
    MENU, NAV, SEARCH, SECTION, SUMMARY

This patch adds support to the HTML Processor for handling these elements. They do not require any additional logic in the rest of the class, and carry no specific semantic rules for parsing beyond what is listed in their group in the IN BODY section of the HTML5 specification.

Props dmsnell.
Fixes #59914.
Built from https://develop.svn.wordpress.org/trunk@57115


git-svn-id: http://core.svn.wordpress.org/trunk@56626 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-11-17 06:13:25 +00:00
Sergey Biryukov 7da5644617 Docs: Use proper case for `@todo` tags.
The correct tag is `@todo`, not `@TODO` or `@todo:` (note the trailing colon).

Reference: [https://developer.wordpress.org/coding-standards/inline-documentation-standards/php/#phpdoc-tags PHP Documentation Standards: PHPDoc tags].

Follow-up to [55203], [56274], [56565], [56698].

Props jrf.
See #59651.
Built from https://develop.svn.wordpress.org/trunk@57077


git-svn-id: http://core.svn.wordpress.org/trunk@56588 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-11-07 00:14:23 +00:00
Sergey Biryukov fe3efe9810 HTML API: Scan to end of tag when getting updated HTML output.
When applying updates to HTML, one step was left out in [56941] which updated the position of the end of the current tag. This made it possible to create bookmarks with null or earlier end positions than their start position. This in turn broke the Directive Processor in Gutenberg during the backport of changes from Core into Gutenberg.

In this commit, after applying updates, the HTML document is now scanned fully to the end of the current tag, updating the internal pointer to its end, so that nothing else will be broken or misaligned.

Follow-up to [56941].

Props dmsnell.
Fixes #59643.
Built from https://develop.svn.wordpress.org/trunk@56953


git-svn-id: http://core.svn.wordpress.org/trunk@56464 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-10-17 10:39:27 +00:00
Sergey Biryukov 89a5bbb997 HTML API: Avoid calling subclass method while internally scanning in Tag Processor.
After modifying tags in the HTML API, the Tag Processor backs up to before the tag being modified and then re-parses its attributes. This saves on the code complexity involved in applying updates, which have already been transformed to “lexical updates” by the time they are applied.

In order to do that, `::get_updated_html()` called `::next_tag()` to reuse its logic. However, as a public method, subclasses may change the behavior of that method, and the HTML Processor does just this. It maintains an HTML stack of open elements and when the Tag Processor calls this method to re-scan a tag and its attributes, it leads to a broken stack.

This commit replaces the call to `::next_tag()` with a more appropriate reapplication of its internal parsing logic to rescan the tag name and its attributes. Given the limited nature of what's occurring in `::get_updated_html()`, this should bring with it certain guarantees that no HTML structure is being changed (that structure will only be changed by subclasses like the HTML Processor).

Follow-up to [56274], [56702].

Props dmsnell, zieladam, nicolefurlan.
Fixes #59607.
Built from https://develop.svn.wordpress.org/trunk@56941


git-svn-id: http://core.svn.wordpress.org/trunk@56452 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-10-16 14:01:27 +00:00
Sergey Biryukov 4fd39c6620 HTML API: Rename `WP_HTML_Processor::createFragment()` to follow WPCS.
`WP_HTML_Processor::create_fragment()` is the correct method name as per the WordPress PHP coding standards.

Follow-up to [56274].

Props dmsnell, jrf, hellofromTonya, SergeyBiryukov.
Fixes #59547.
Built from https://develop.svn.wordpress.org/trunk@56790


git-svn-id: http://core.svn.wordpress.org/trunk@56302 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-10-05 22:42:26 +00:00
Sergey Biryukov ffd72aac28 Coding Standards: Remove redundant ignore annotations, take 5.
The `VariableAnalysis` standard is not used by WP Core.

Follow-up to [50958], [51003], [52049], [52051], [52069], [53072], [54132], [55132], [56363], [56738], [56743], [56751], [56752].

Props jrf.
See #59161.
Built from https://develop.svn.wordpress.org/trunk@56753


git-svn-id: http://core.svn.wordpress.org/trunk@56265 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-10-02 11:27:24 +00:00
Bernhard Reiter 2d9b5cb098 HTML API: Add class name utilities `has_class()` and `class_list()`.
This patch adds two new public methods to the HTML Tag Processor:
 - `has_class()` indicates if a matched tag contains a given CSS class name.
 - `class_list()` returns a generator to iterate over all the class names in a matched tag.

Included in this patch is a refactoring of the internal logic when matching
a tag to reuse the new `has_class()` function. Previously it was relying on
optimized code in the `matches()` function which performed byte-for-byte
class name comparison. With the change in this patch it will perform class
name matching on the decoded value, which might differ if a class attribute
contains character references.

These methods may be useful for running more complicated queries based
on the presence or absence of CSS class names. The use of these methods
avoids the need to manually decode the class attribute as reported by
`$process->get_attribute( 'class' )`.

Props dmsnell.
Fixes #59209.
Built from https://develop.svn.wordpress.org/trunk@56703


git-svn-id: http://core.svn.wordpress.org/trunk@56215 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-26 09:17:18 +00:00
Bernhard Reiter 374bcd9b3d HTML API: Add `matches_breadcrumbs()` method for better querying.
Inside a `next_tag()` loop it can be challenging to use breadcrumbs because they are only exposed inside the call to `next_tag()` via the `$query` arg.

In this patch a new method, `matches_breadcrumbs()`, is exposed which allows for querying within the `next_tag()` loop for more complicated queries.

This method exposes a wildcard `*` operator to allow matching ''any HTML tag'' that the currently-matched tag is a child or descendant of.

Props dmsnell, westonruter, mukesh27.
Fixes #59400.
Built from https://develop.svn.wordpress.org/trunk@56702


git-svn-id: http://core.svn.wordpress.org/trunk@56214 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-26 08:20:17 +00:00
Bernhard Reiter 061c32d574 HTML API: Remove all duplicate copies of an attribute when removing.
When encountering an HTML tag with duplicate copies of an attribute the tag processor ignores the duplicate values, according to the specification. However, when removing an attribute it must remove all copies of that attribute lest one of the duplicates becomes the primary and it appears as if no attributes were removed.

In this patch we're adding tests that will be used to ensure that all attribute copies are removed from a tag when one is request to be removed.

**Before**

{{{#!php
<?php
$p = new WP_HTML_Tag_Processor( '<br id=one id="two" id='three' id>' );
$p->next_tag();
$p->remove_attribute( 'id' );
$p->get_updated_html();
// <br id="two" id='three' id>
}}}

**After**

{{{#!php
<?php
$p = new WP_HTML_Tag_Processor( '<br id=one id="two" id='three' id>' );
$p->next_tag();
$p->remove_attribute( 'id' );
$p->get_updated_html();
// <br>
}}}

Previously we have been overlooking duplicate attributes since they don't have an impact on what parses into the DOM. However, as one unit test affirmed (asserting the presence of the bug in the tag processor) when removing an attribute where duplicates exist this meant we ended up changing the value of an attribute instead of removing it.

In this patch we're tracking the text spans of the parsed duplicate attributes so that ''if'' we attempt to remove them then we'll have the appropriate information necessary to do so. When an attribute isn't removed we'll simply forget about the tracked duplicates. This involves some overhead for normal operation ''when'' in fact there are duplicate attributes on a tag, but that overhead is minimal in the form of integer pairs of indices for each duplicated attribute.

Props dmsnell, zieladam.
Fixes #58119.
Built from https://develop.svn.wordpress.org/trunk@56684


git-svn-id: http://core.svn.wordpress.org/trunk@56196 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-25 19:04:19 +00:00
Bernhard Reiter 15c3f036fd HTML API: Update documentation and rename internal variable on HTML Processor
This patch updates documentation and an internal variable name within the
HTML Processor class so that they are more helpful and complete to a reader.

There should be no functional or visual changes in this patch.

Props dmsnell, mukesh27.
Fixes #59267.
Built from https://develop.svn.wordpress.org/trunk@56565


git-svn-id: http://core.svn.wordpress.org/trunk@56077 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-13 13:02:17 +00:00
Bernhard Reiter deaa431a90 HTML API: Skip over contents of RAWTEXT elements such as STYLE.
When encountering elements that imply switching into the RAWTEXT parsing state,
the Tag Processor should skip processing until exiting the RAWTEXT state.

In this patch the Tag Processor does just that, except for the case of the
deprecated XMP element which implies further and more complicated rules.

There's an implicit assumption that the SCRIPT ENABLED flag in HTML parsing
is enabled so that the contents of NOSCRIPT can be skipped. Otherwise, it would
be required to parse the contents of that tag.

Props dmsnell.
Fixes #59292.
Built from https://develop.svn.wordpress.org/trunk@56563


git-svn-id: http://core.svn.wordpress.org/trunk@56075 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-13 12:49:16 +00:00
Bernhard Reiter 251a9c7653 HTML API: Store current token reference in HTML Processor state.
The `$current_token` reference has been stored in the HTML Processor itself, but I suggested to move it into the externalized state so that it can be stored and replaced.

In this patch the reference is moved to that state variable and it should become more possible to save and load state, to resume execution after pausing.

Props dmsnell.
Fixes #59268.
Built from https://develop.svn.wordpress.org/trunk@56558


git-svn-id: http://core.svn.wordpress.org/trunk@56070 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-12 15:12:17 +00:00
Sergey Biryukov b80ce60f70 Coding Standards: Use pre-increment/decrement for stand-alone statements.
Note: This is enforced by WPCS 3.0.0:

1. There should be no space between an increment/decrement operator and the variable it applies to.
2. Pre-increment/decrement should be favoured over post-increment/decrement for stand-alone statements. “Pre” will in/decrement and then return, “post” will return and then in/decrement. Using the “pre” version is slightly more performant and can prevent future bugs when code gets moved around.

References:
* [https://developer.wordpress.org/coding-standards/wordpress-coding-standards/php/#increment-decrement-operators WordPress PHP Coding Standards: Increment/decrement operators]
* [https://github.com/WordPress/WordPress-Coding-Standards/pull/2130 WPCS: PR #2130 Core: add sniffs to check formatting of increment/decrement operators]

Props jrf.
See #59161, #58831.
Built from https://develop.svn.wordpress.org/trunk@56549


git-svn-id: http://core.svn.wordpress.org/trunk@56061 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-09-09 09:28:26 +00:00
Bernhard Reiter c67fe4b6e5 HTML API: Stop processing HTML when encountering unsupported markup.
It was a design goal of the HTML Processor to abort processing its input document when encountering unsupported markup. Unfortunately there was no test for this and so-far, the HTML Processor has paused, but continued processing in these situations.

In this patch a new test ensures that the HTML Processor stops and refuses to move forward after encountering any unsupported markup. It also ensures that it doesn't report any current tag names since unsupported markup could imply that the read tag name is different than the parsed tag name.

Props dmsnell.
Fixes #59167.
Built from https://develop.svn.wordpress.org/trunk@56493


git-svn-id: http://core.svn.wordpress.org/trunk@56005 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-08-30 15:39:16 +00:00
Bernhard Reiter 12884f0361 HTML API: Add support for BUTTON element.
This patch adds support to process the BUTTON element. This requires adding some additional semantic rules to handle situations where a BUTTON element is already in scope.

Also included is a fixup to enforce that `WP_HTML_Processor::next_tag()` never returns for a tag closer. This is useful with the Tag Processor, but not for the HTML Processor. There were tests relying on this behavior to assert that internal processes were working as they should, but those tests have been updated to use the semi-private `step()` function, which does stop on tag closers.

This patch is one in a series of changes to expand support within the HTML API, moving gradually to allow for more focused changes that are easier to review and test. The HTML Processor is a work in progress with a certain set of features slated to be ready and tested by 6.4.0, but it will only contain partial support of the HTML5 specification even after that. Whenever it cannot positively recognize and process its input it bails, and certain function stubs and logical stubs exist to structure future expansions of support.

Props dmsnell.
Fixes #58961.
Built from https://develop.svn.wordpress.org/trunk@56380


git-svn-id: http://core.svn.wordpress.org/trunk@55892 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-08-10 08:37:20 +00:00
Aaron Jorbin 5341f9b212 HTML API: Fix missing * for docblock.
Follow up to [56363].

Props dmsnell.
See #58918. Fixes #59010.

Built from https://develop.svn.wordpress.org/trunk@56376


git-svn-id: http://core.svn.wordpress.org/trunk@55888 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-08-08 23:01:26 +00:00
Bernhard Reiter fa83c8e1cd HTML API: Adjust code styling to Gutenberg's linter's preferences.
Adjust the code style according to the rules that the linting process in Gutenberg requires.

There are only a couple code changes that should have no effect on the runtime:
 - A missing check to verify that only `UTF-8` is supported has been added (brought up because it was identified as an undefined variable).
 - A few `return false;` statements have been added to avoid having the linter complain that functions don't return a value despite indicating they return `bool`. The functions are stubs for coming support and currently `throw`, so the `return` statements are unreachable.

Props dmsnell, costdev, davidbaumwald, peterwilsoncc, SergeyBiryukov.
Fixes #58918.
Built from https://develop.svn.wordpress.org/trunk@56363


git-svn-id: http://core.svn.wordpress.org/trunk@55875 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-08-07 13:50:27 +00:00
Bernhard Reiter 8fa9aad5e6 HTML API: Add support for SPAN element.
In this patch we're introducing support for the SPAN element, which is the first
in the class of "any other tag" in the "in body" insertion mode.

This patch introduces the mechanisms required to handle that class of tags but
only introduces SPAN to keep the change focused. With the tests and mechanisms
in place it will be possible to follow-up and add another limited set of tags.

It's important that this not use the default catch-all in the switch handling
`step_in_body` because that would catch tags that have specific rules in previous
case statements that aren't yet added. For example, we don't want to treat the
`TABLE` element as "any other tag".

Props dmsnell.
Fixes #58907.
Built from https://develop.svn.wordpress.org/trunk@56331


git-svn-id: http://core.svn.wordpress.org/trunk@55843 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-08-01 07:56:23 +00:00
Sergey Biryukov 80e5ebb0cc Coding Standards: Always declare visibility for class methods.
This adds a missing `public` keyword for `WP_HTML_Tag_Processor::get_attribute_names_with_prefix()`.

Follow-up to [55203].

Props jrf.
See #58831.
Built from https://develop.svn.wordpress.org/trunk@56301


git-svn-id: http://core.svn.wordpress.org/trunk@55813 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-07-25 13:16:21 +00:00
Bernhard Reiter 44629e6286 HTML-API: Introduce minimal HTML Processor.
This patch introduces the //first// of //many// iterations on the evolution of the HTML API, the HTML Processor, which is built in order to understand HTML structure including nesting, misnesting, and complicated semantic rules.

In the first iteration, the HTML Processor is arbitrarily limited to a minimal subset of functionality so that we can review it, ship it, test it, and collect feedback before moving forward. This means that this patch is more or less an extension to the Tag Processor query language, providing the ability not only to scan for a tag of a given name, but also to find an HTML element in a specific nesting path.

The HTML Processor also aborts any time it encounters:
 - a tag that isn't a `P`, `DIV`, `FIGURE`, `FIGCAPTION`, `IMG`, `STRONG`, `B`, `EM`, `I`, `A`, `BIG`, `CODE`, `FONT`, `SMALL`, `STRIKE`, `TT`, or `U` tag. this limit exists because many HTML elements require specific rules and we are trying to limit the number of rules introduced at once. this work is targeted at existing work in places like the image block.
 - certain misnesting constructs that evoke complicated resolution inside the HTML spec. where possible and where simple to do reliably, certain parse errors are handled. in most cases the HTML Processor aborts.

The structure of the HTML Processor is established in this patch. Further spec-compliance comes through filling out //more of the same// kind and nature of code as is found in this patch. Certain critical HTML algorithms are partially supported, and where support requires more than is present, the HTML Processor acknowledges this and refuses to operate.

In this patch are explorations for how to verify that new HTML support is fully added (instead of allowing for partial updates that leave some code paths non-compliant). Performance is hard to measure since support is so limited at the current time, but it should generally follow the performance of the Tag Processor somewhat close as the overhead is minimized as much as practical.

Props dmsnell, zieladam, costdev.
Fixes #58517.
Built from https://develop.svn.wordpress.org/trunk@56274


git-svn-id: http://core.svn.wordpress.org/trunk@55786 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-07-20 13:43:25 +00:00
Andrew Ozz 817c701f29 HTML API: Fix a fatal error when processing malformed document with unclosed attribute.
Props: dlh, costdev, dmsnell.
Fixes: #58637.
Built from https://develop.svn.wordpress.org/trunk@56133


git-svn-id: http://core.svn.wordpress.org/trunk@55645 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-07-04 20:45:23 +00:00
Sergey Biryukov 5e0592d8f4 Docs: Improve HTML API file and class headers per the documentation standards.
Follow-up to [55203], [55304], [55718], [55724], [55727].

See #57840.
Built from https://develop.svn.wordpress.org/trunk@55734


git-svn-id: http://core.svn.wordpress.org/trunk@55246 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-09 11:19:21 +00:00
Sergey Biryukov 2ffb7ef083 Docs: Update code examples formatting in `WP_HTML_Tag_Processor` documentation.
Per the [https://developer.wordpress.org/coding-standards/inline-documentation-standards/php/#description documentation standards], code samples should be created by indenting every line of the code by 4 spaces, with a blank line before and after. This matches the format used by the rest of core.

Follow-up to [55203], [55304], [55718], [55724].

Props juanmaguitar, coffee2code, azaozz, costdev, dmsnell, johnbillion, SergeyBiryukov.
Fixes #58028.
Built from https://develop.svn.wordpress.org/trunk@55727


git-svn-id: http://core.svn.wordpress.org/trunk@55239 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-08 17:32:24 +00:00
Sergey Biryukov 8b53b81183 Docs: Remove `@return void` from various DocBlocks.
Per the documentation standards, it should not be used outside of the default bundled themes.

Follow-up to [52049], [52051], [53331], [54156], [54214], [55203], [55719].

See #57840.
Built from https://develop.svn.wordpress.org/trunk@55725


git-svn-id: http://core.svn.wordpress.org/trunk@55237 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-06 11:40:24 +00:00
Sergey Biryukov 24eb807cbd Docs: Correct the placement of `@see` tags in `WP_HTML_Tag_Processor` class.
This moves a reference link in `::get_attribute_names_with_prefix()` below the code example, so that it is correctly displayed in the Developer Resources.

Includes updating some other `@see` tags for consistency as per the documentation standards.

Additionally, the example code for `WP_HTML_Tag_Processor::get_tag()` is updated to show lowercase tag names in the input HTML, so that it does not convey the wrong impression that the uppercase output from `::get_tag()` depends on the case of the input HTML.

Follow-up to [55203].

Props dmsnell, johnbillion, audrasjb, SergeyBiryukov.
Fixes #58254.
Built from https://develop.svn.wordpress.org/trunk@55724


git-svn-id: http://core.svn.wordpress.org/trunk@55236 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-05 12:43:21 +00:00
audrasjb 3911398ec5 HTML API: Restore mistakenly-removed content in documentation.
In [55718] the Unicode replacement character was mistakenly removed. The purpose of including the character was to communicate what it looks like and why the Tag Processor won't insert it into the document.

This changeset brings the character back and adds a small clue to fix the confusion that may lead to its removal.

Follow-up to [55718].

Props dmsnell.
Fixes #58256
See #57840.

Built from https://develop.svn.wordpress.org/trunk@55723


git-svn-id: http://core.svn.wordpress.org/trunk@55235 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-05 10:58:23 +00:00
Bernhard Reiter f2d315036b HTML API: Adjust coding style to pass Gutenberg linter.
This patch adjusts some minor neutral whitespace that the Gutenberg linting rejects.
There are no code changes otherwise.

Props dmsnell.
Fixes #58250.
Built from https://develop.svn.wordpress.org/trunk@55721


git-svn-id: http://core.svn.wordpress.org/trunk@55233 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-04 08:42:22 +00:00
John Blackbourn 89cbdce5c3 Docs: Improve formatting of markup in the docs for `WP_HTML_Tag_Processor`.
Code blocks wrapped inside backtacks don't need to be indented.

See #57840

Built from https://develop.svn.wordpress.org/trunk@55718


git-svn-id: http://core.svn.wordpress.org/trunk@55230 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-03 23:55:16 +00:00
Bernhard Reiter 5a32396d8f HTML API: Accumulate shift for internal parsing pointer.
A bug was discovered where where the parser wasn't returning to the
start of the affected tag after making some updates.

In few words, the Tag Processor has not been treating its own internal
pointer `bytes_already_parsed` the same way it treats its bookmarks.
That is, when updates are applied to the input document and then
`get_updated_html()` is called, the internal pointer transfers to
the newly-updated content as if no updates had been applied since
the previous call to `get_updated_html()`.

In this patch we're creating a new "shift accumulator" to account for
all of the updates that accrue before calling `get_updated_html()`.
This accumulated shift will be applied when swapping the input document
with the output buffer, which should result in the pointer pointing to
the same logical spot in the document it did before the udpate.

In effect this patch adds a single workaround for treating the
internal pointer like a bookmark, plus a temporary pointer which points
to the beginning of the current tag when calling `get_updated_html()`.
This will preserve the assumption that updating a document doesn't
move that pointer, or shift which tag is currently matched.

Props dmsnell, zieladam.
Fixes #58179.
Built from https://develop.svn.wordpress.org/trunk@55706


git-svn-id: http://core.svn.wordpress.org/trunk@55218 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-05-03 11:31:18 +00:00
zieladam 8659101491 HTML API: Fix a case where updates are overlooked when seeking to earlier locations.
This retains the WP_HTML_Tag_Processor attribute updates applied before calling seek() – they were erroneously erased in some cases.

Props dmsnell.
Fixes #58160.




Built from https://develop.svn.wordpress.org/trunk@55675


git-svn-id: http://core.svn.wordpress.org/trunk@55187 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-04-21 13:32:25 +00:00
zieladam 1919350606 HTML API: Update code style so it passes when backported into Gutenberg.
This changes the indentation of a variable in class-wp-html-tag-processor.php 
to satisfy both WordPress and Gutenberg linters.

Props dmsnell, ntsekouras.
Fixes #58170.


Built from https://develop.svn.wordpress.org/trunk@55674


git-svn-id: http://core.svn.wordpress.org/trunk@55186 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-04-21 12:33:23 +00:00
Bernhard Reiter b116fcdb27 HTML API: Add support for a few invalid HTML comment forms.
- Comments created by means of a tag closer with an invalid tag name, e.g. `</3>`.
 - Comments closed with the invalid `--!>` closer. (Comments should be closed by `-->` but if the `!` appears it will also close it, in error.)
 - Empty tag name elements, which are technically skipped over and aren't comments, e.g. `</>`.

Props dmsnell, costdev.
Fixes #58007.
Built from https://develop.svn.wordpress.org/trunk@55667


git-svn-id: http://core.svn.wordpress.org/trunk@55179 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-04-20 17:10:20 +00:00
Bernhard Reiter 3b58785908 HTML API: Ensure attribute updates happen only once for case variants.
When setting a new value for an attribute multiple times and providing
multiple case variations of the attribute name the Tag Processor has
been appending multiple copies of the attribute into the updated HTML.

This means that only the first attribute set determines the value in
the final output, plus the output will //appear// wrong.

In this patch we're adding a test to catch the situation and resolving it
by using the appropriate comparable attribute name as a key for storing
the updates as we go. Previously we stored updates to the attribute by
its given `$name`, but when a new update of the same name with a
case variant was queued, it would not override the previously-enqueued
value as it out to have.

Props dmsnell, zieladam.
Fixes #58146.
Built from https://develop.svn.wordpress.org/trunk@55659


git-svn-id: http://core.svn.wordpress.org/trunk@55171 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-04-19 09:27:22 +00:00
Bernhard Reiter fbc006e2b2 HTML API: Add `has_self_closing_flag()` to Tag Processor.
In this patch we're adding `has_self_closing_flag()` to the HTML Tag Processor.
This exposes whether a currently-matched tag contains the self-closing flag `/`.

This information is critical for the evolution of the HTML API in order
to track and parse HTML structure, specifically, knowing whether an
HTML foreign element is self-closing or not.

Props dmsnell, zieladam.
Fixes #58009.
Built from https://develop.svn.wordpress.org/trunk@55619


git-svn-id: http://core.svn.wordpress.org/trunk@55131 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-04-04 10:06:27 +00:00
hellofromTonya 95f3aceea7 HTML API: Add bookmark invalidation logic.
While `WP_HTML_Tag_Processor` currently only supports changing a given tag's attributes, the plan is to provide methods to make broader changes (possibly through a subclass of `WP_HTML_Tag_Processor`). The API will have the potential of replacing a tag that a bookmark points to. To prepare, this changeset makes sure that all bookmarks affected by a HTML replacement are invalidated (i.e. released).

Changes:
* Extends the existing loop in `WP_HTML_Tag_Processor::apply_attributes_updates()` that adjusts bookmarks' start and end positions upon HTML changes to check if the entire bookmark is within a portion of the HTML that has been replaced.
* Adds `WP_HTML_Tag_Processor::has_bookmark() to check whether the given bookmark name exists.

References:
* [https://github.com/WordPress/gutenberg/pull/47559 Gutenberg PR 47559]
* [https://github.com/WordPress/gutenberg/releases/tag/v15.3.0 Released in Gutenberg 15.3.0]

Follow-up to [55203].

Props bernhard-reiter, dmsnell, zieladam.
Fixes #57788.
Built from https://develop.svn.wordpress.org/trunk@55555


git-svn-id: http://core.svn.wordpress.org/trunk@55067 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-03-16 13:11:24 +00:00
hellofromTonya 2257e9b451 HTML API: Document shorthand usage of the next_tag().
Documents the shorthand usage, i.e. `$this->next_tag( 'img' )`, of `WP_HTML_Tag_Processor::next_tag()`.

Also includes table alignments and formatting adjustments in the class docs.

Follow-up to [55203], [55206].

Props zieladam, poena, dmsnell, costdev, hellofromTonya.
Fixes #57863.
See #57575.
Built from https://develop.svn.wordpress.org/trunk@55477


git-svn-id: http://core.svn.wordpress.org/trunk@55010 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-03-07 16:48:21 +00:00
hellofromTonya 39308664bf HTML API: Fix finding RCData and Script tag closers.
Fixes finding the following tag closers `</script>`, `</textarea>`, and `</title>` in `WP_HTML_Tag_Processor`.

Follow-up to [55407], [55203].

Props zieladam, dmsnell, hellofromTonya.
Fixes #57852.
See #57575.
Built from https://develop.svn.wordpress.org/trunk@55469


git-svn-id: http://core.svn.wordpress.org/trunk@55002 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-03-06 18:55:21 +00:00
hellofromTonya 1579f32cc6 HTML API: Fix finding bookmarks set on closing tag WP_HTML_Tag_Processor.
Setting a bookmark on a tag should set its "start" position before the opening "<", e.g.:
{{{
<div> Testing a <b>Bookmark</b>
----------------^
}}}

The previous calculation assumed this is always one byte to the left from `$tag_name_starts_at`.

However, in a closing tag that index points to a solidus symbol "/":
{{{
<div> Testing a <b>Bookmark</b>
----------------------------^
}}}

The bookmark should therefore start two bytes before the tag name:
{{{
<div> Testing a <b>Bookmark</b>
---------------------------^
}}}

This changeset achieves this by:
* Using the correct starting index for closing tag bookmarks.
* Adding `array( 'tag_closers' => 'visit' )` in `WP_HTML_Tag_Processor::seek()`.

Follow-up to [55203].

Props zieladam, dmsnell, flixos90.
Fixes #57787.
See #57575.
Built from https://develop.svn.wordpress.org/trunk@55407


git-svn-id: http://core.svn.wordpress.org/trunk@54940 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-02-22 20:55:23 +00:00
gziolo 582feb3ffb HTML API: Set $this->html to protected to support subclassing
When the HTML API was introduced a number of fields were switched from private visibility to protected so that Gutenberg and other systems could more easily enhance the behaviors through subclassing. The $this->html property was overlooked but important for systems using the Tag Processor to stich HTML, specifically performing operations on innerHTML and innerText.

Follow-up [55203].
Props dmsnell.
See #57575.


Built from https://develop.svn.wordpress.org/trunk@55402


git-svn-id: http://core.svn.wordpress.org/trunk@54935 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-02-22 06:24:22 +00:00
Sergey Biryukov 09cc873d3a Docs: Replace short array syntax in `WP_HTML_Tag_Processor` documentation.
Per [https://developer.wordpress.org/coding-standards/wordpress-coding-standards/php/#declaring-arrays WordPress PHP Coding Standards]:
> Using long array syntax ( `array( 1, 2, 3 )` ) for declaring arrays is generally more readable than short array syntax ( `[ 1, 2, 3 ]` ), particularly for those with vision difficulties. Additionally, it’s much more descriptive for beginners.
> 
> Arrays must be declared using long array syntax.

Original PR from Gutenberg repository:
* [https://github.com/WordPress/gutenberg/pull/47958 #47958 Docs: Don't recommend using short array syntax in WP_HTML_Tag_Processor]

Follow-up to [55203], [55206].

Props aristath, poena.
Fixes #57691.
Built from https://develop.svn.wordpress.org/trunk@55304


git-svn-id: http://core.svn.wordpress.org/trunk@54837 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-02-10 10:59:25 +00:00
Andrew Ozz 7bc792bb73 Fix couple of typos in inline docs.
Props: ironprogrammer.
See #57575.
Built from https://develop.svn.wordpress.org/trunk@55206


git-svn-id: http://core.svn.wordpress.org/trunk@54739 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-02-03 02:14:19 +00:00
Andrew Ozz be73904dc7 Introduce HTML API with HTML Tag Processor
This commit pulls in the HTML Tag Processor from the Gutenbeg repository.

The Tag Processor attempts to be an HTML5-spec-compliant parser that provides the ability in PHP to find specific HTML tags and then add, remove, or update attributes on that tag. It provides a safe and reliable way to modify the attribute on HTML tags.

More information: https://github.com/WordPress/wordpress-develop/pull/3920.

Props: antonvlasenko, bernhard-reiter, costdev, dmsnell, felixarntz, gziolo, hellofromtonya, zieladam, flixos90, ntsekouras, peterwilsoncc, swissspidy, andrewserong, onemaggie, get_dave, aristath, scruffian, justlevine, andraganescu, noisysocks, dlh, soean, cbirdsong, revgeorge, azaozz.
Fixes #57575.
Built from https://develop.svn.wordpress.org/trunk@55203


git-svn-id: http://core.svn.wordpress.org/trunk@54736 1a063a9b-81f0-0310-95a4-ce76da25c4cd
2023-02-03 01:05:17 +00:00