Commit Graph

108 Commits

Author SHA1 Message Date
David Taylor 2e00d4d024
DEV: Fix flaky twitter onebox behavior (#18141)
The order in which Onebox engines are loaded is not guaranteed. Occasionally during tests, the twitter engine would be loaded before the instagram engine, and cause the Instagram Onebox spec to fail due to the lack of `Onebox.options.twitter_client`.

This commit makes the load order of Onebox engines consistent, and fixes the issue in the twitter_status_onebox.
2022-08-31 08:42:55 +08:00
Bianca Nenciu 626d50c15c
FIX: Disable Twitter onebox without API support (#17519)
Twitter removed OpenGraph tags from their pages. We can no longer
extract all the information (for example, the quoted tweet) we need
to render Oneboxes without using their API.
2022-08-17 18:32:48 +03:00
Bianca Nenciu e7f04a8674
FIX: Use URI#merge to merge base and relative URLs (#17454)
The old implementation did not handle all cases, such as the case when
`src` is a relative URL that starts with `..`.
2022-07-18 14:17:54 +03:00
Ghassan Maslamani d0a4bc636f
FIX: Vimeo regex pattern (#17277)
Vimeo has two url structure:

- Normal /video_id

- Private/Unlisted /video_id/hash_string

This changes change the regex pattern thus it would be able to

catch both. Also it tolerate trailing slash.

This shall fixes:

https://meta.discourse.org/t/vimeo-embed-urls-parsed-incorrectly-in-email/231042
2022-06-30 13:13:25 -03:00
Rafael dos Santos Silva f130ec35d9
FEATURE: Use full post width for Vimeo embeds (#17289) 2022-06-30 13:08:24 -03:00
jbrw 9874fe3fb3
FIX: Improve mixcloud oneboxing (#17237)
- Sets `https://www.mixcloud.com` as a `requires_iframe_origins` to allow the iframe content to be displayed
- Attempts to render something approximating the Mixcloud content in the preview pane of the Composer, rather than just displaying a large version of the artwork associated with the link
2022-06-27 08:32:24 +10:00
Penar Musaraj 3baefa25b5
FIX: Use first supported type item when JSON-LD returns array (#17217) 2022-06-23 13:02:01 -04:00
Kris dfba188bd6
UX: remove extra whitespace in github onebox (#17111) 2022-06-17 11:09:45 +10:00
Jarek Radosz f723b4c322
FIX: Handle sites with more than 1 JSON-LD element (#17095)
A followup to #17007
2022-06-15 02:55:55 +02:00
sansnumero f0c6dd5682
Add support for JSON LD in Onebox (#17007)
* FIX: Fix a bug that is accessing the values in a hash wrongly and write tests

I decided to write tests in order to be confident in my refactor that's in the next commit.
Meanwhile I have discovered a potential bug. The `title_attr` key was accessed as a string,
but all the keys are actually symbols so it was never evaluated to be true.

irb(main):025:0> d = {key: 'value'}
=> {:key=>"value"}
irb(main):026:0> d['key']
=> nil
irb(main):027:0> d[:key]
=> "value"

* DEV: Extract methods for readability

I will be adding a new method following the conventions in place for adding a new normalizer. And this will make the readability of the `raw` block even more difficult; so I am extracting self contained private methods beforehand.

* FEATURE: Parse JSON-LD and introduce Movie object

JSON LD data is very easily transferable to Ruby objects because they contain types. If these types are mapped to Ruby objects, it is also better to make all the parsed data very explicit and easily extendable.

JSON-LD has many more standardized item types, with a full list here: https://schema.org/docs/full.html
However in order to decrease the scope, I only adapted the movie type.

* DEV: Change inheritance between normalizers

Normalizers are not supposed to have an inheritance relationships amongst each other. They are all normalizers, but all normalizing separate protocols. This is why I chose to extract a parent class and relieve Open Graph off that responsibility. Removing the parent class altogether could also a possibility, but I am keeping the scope limited to having a more accurate representation of the normalizers while making it easier to add a new one.

* Lint changes

* Bring back the Oembed OpenGraph inheritance

There is one test that caught that this inheritance was necessary. I still think modelling wise this inheritance shouldn't exist, but this can be tackled separately.

* Return empty hash if the json received is invalid

Before this change if there was a parsing error with JSON it would throw an exception. The goal of this commit is to rescue that exception and then log a warning. I chose to use Discourse's logger wrapper `warn_exception` to have the backtrace and not just used Rails logger. I considered raising an `InvalidParameters` error however if the JSON here is invalid it should not block showing of the Onebox, so logging is enough.

* Prep to support more JSONLD schema types with case

* Extract mustache template object created from JSONLD
2022-06-13 17:32:34 +02:00
Mayfield 99b0578b4c
FIX: escape youtube title when constructing onebox preview html (#16999) 2022-06-08 13:42:37 +08:00
David Taylor 8fe3934856
UX: Make YouTube playlist onebox full width to match video onebox (#16936) 2022-05-27 10:39:12 +01:00
Loïc Guitaut 46176b7dd7 DEV: Don’t patch Sanitize::Config
Currently we’re reopening the `Sanitize::Config` class (which is part of
the `sanitize` gem) to put our custom config for Onebox in it. This is
unnecessary as we can simply create a dedicated module to hold our
custom configuration.
2022-04-06 17:10:51 +02:00
David Taylor ff93833fdf
UX: Use committed date for GitHub oneboxes (#16318)
Our copy says 'committed {date}`, but we were previously using the commit's authored date
2022-03-30 09:16:28 +08:00
jbrw 528c3e311a
FIX: Only display the first listed price (#16138)
Multiple prices may be returned by Amazon (e.g. for new, and also for used). We should only display the first price.
2022-03-08 15:24:45 -05:00
jbrw fc30669db2
FIX: Support new layout on Amazon product pages (#16091)
Some product pages on Amazon are using a new HTML structure, meaning the previous Onebox engine was unable to gather the price and/or description. This change should allow these pages to be Oneboxed.
2022-03-04 18:31:53 -05:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Alan Guo Xiang Tan 7afe768d60
DEV: Add tests for wistia onebox. (#15860)
Follow-up to 4ef56b0ca4
2022-02-08 13:04:32 +08:00
jbrw 4ef56b0ca4
FIX: Explicitly set `allowfullscreen` on Wistia Oneboxes (#15828) 2022-02-08 13:02:32 +11:00
Rafael dos Santos Silva 5b5cbbfe5c
FEATURE: Onebox for news.ycombinator.com (#15781) 2022-02-03 13:39:21 -03:00
Natalie Tay aac9f43038
Only block domains at the final destination (#15689)
In an earlier PR, we decided that we only want to block a domain if 
the blocked domain in the SiteSetting is the final destination (/t/59305). That 
PR used `FinalDestination#get`. `resolve` however is used several places
 but blocks domains along the redirect chain when certain options are provided.

This commit changes the default options for `resolve` to not do that. Existing
users of `FinalDestination#resolve` are
- `Oneboxer#external_onebox`
- our onebox helper `fetch_html_doc`, which is used in amazon, standard embed 
and youtube
  - these folks already go through `Oneboxer#external_onebox` which already
  blocks correctly
2022-01-31 15:35:12 +08:00
Bianca Nenciu 847c77de65
FIX: Add another method to check binary file (#15648)
This method looks for a NULL byte that is not usually contained in text
files. Follow up to 376799b1a4.
2022-01-20 23:47:18 +02:00
Bianca Nenciu 376799b1a4
FIX: Hide excerpt of binary files in GitHub onebox (#15639)
Oneboxer did not know if a file is binary or not and always tried to
show an excerpt of the file.
2022-01-19 14:45:36 +02:00
jbrw 2909b8b820
FIX: origins_to_regexes should always return an array (#15589)
If the SiteSetting `allowed_onebox_iframes` contains a value of `*`, it will use the values of `all_iframe_origins` during the Oneboxing process. If `all_iframe_origins` itself contains a value of `*`, `origins_to_regexes` will try to return a "catch-all" regex.

Other code assumes `origins_to_regexes`will return an array, so this change ensures the `*` case will return an array containing only the catch-all regex.
2022-01-17 12:48:41 -05:00
Jarek Radosz 31b27b3712
FIX: Broken GitHub folder onebox logic (#15612)
1. `html_doc.css('.Box.md')` always returns a truthy value (e.g. `[]`) so the second branch of the if-elsif never ran
2. `node&.css('text()')` was invalid code that would raise an error
3. Matching on h3 elements is no longer correct with the current html structure returned by GitHub
2022-01-17 18:32:07 +01:00
Joffrey JAFFEUX f56eff2303
FIX: limits pre-line impact to tweet text (#15583) 2022-01-14 10:44:21 +01:00
jbrw 6e925fee6f
FIX: Use basic meta description if other description tags are missing (#15356)
When attempting to Onebox a page if there is no `meta property="og:description"` tag but there is a  `meta name="description"` tag, Onebox should try to use that value.
2021-12-17 19:36:54 -05:00
Natalie Tay 4c46c7e334
DEV: Remove xlink hrefs (#15059) 2021-11-25 15:22:43 +11:00
jbrw aec125b617
FIX: Display Instagram Oneboxes in an iframe (#14789)
We are no longer able to display the image returned by Instagram directly within a Discourse site (either in the composer, or within a cooked post within a topic), so:

- Display an image placeholder in the composer preview
- A cooked post should use an iframe to display the Instagram 'embed' content
2021-11-02 14:34:51 -04:00
Dan Ungureanu 69f0f48dc0
DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
Faris Masad 3fbfec06fc Update replit onebox to accept .com 2021-10-19 16:37:33 -04:00
Bianca Nenciu ba81d1853b
FIX: Disable previews if diffhtml is enabled (#14537)
diffhtml should not rerender video and audio elements so there is no
point in having these.
2021-10-08 15:57:08 +03:00
Arpit Jalan fbe9cd49b6
FIX: Vimeo private video oneboxes were broken (#14510) 2021-10-05 15:46:58 +05:30
Jarek Radosz 02a6b991fe
FIX: Correct the play icon position (#14295) 2021-09-09 15:10:32 +02:00
Arpit Jalan 11a07b37e1
FIX: ignore canonical link for medium.com oneboxes (#14278)
https://meta.discourse.org/t/bug-in-onebox-link-being-rendered-as-a-gist-when-it-isnt/202463
2021-09-08 20:19:57 +05:30
Chema Balsas d27d7c8cca FIX: Unescapes hash section with present to account for url-encoded chars
Sections with unreserverd characters will appear url-encoded and need to
be unescaped before using it.

Wikipedia generates 2 different spans in this case in the same page, one
with an id resulting of replacing the % symbols with . and the other with
the decoded version of the string. For example, for /wiki/foo#A%C3%A1A it
will generate:

<span id="A.C3.A1A"></span>
<span id="AáA">AáA</span>

Unescaping the `m_url_hash_name` should work in all cases to target the
proper section span.
2021-08-12 10:43:50 -04:00
Arpit Jalan bb2c48b065
FIX: update iframe url for simplecast onebox (#13957)
https://meta.discourse.org/t/onebox-regression-simplecast-com/187911
2021-08-05 18:29:04 +05:30
Arpit Jalan a341dba5d9
FIX: update oEmbed URL for simplecast onebox (#13956) 2021-08-05 17:42:38 +05:30
jbrw 2f28ba318c
FEATURE: Onebox can match engines based on the content_type (#13876)
* FEATURE: Onebox can match engines based on the content_type

`FinalDestination` now returns the `content_type` of a resolved URL.

`Oneboxer` passes this value to `Onebox` itself. Onebox engines can now specify a `matches_content_type` regex of content_types that the engine can handle, regardless of the URL.

`ImageOnebox` will match URLs with a content type of `image/png`, `jpg`, `gif`, `bmp`, `tif`, etc.

This will allow images that exist at a URL without a file type extension to be correctly rendered, assuming a valid `content_type` is returned.
2021-07-30 13:36:30 -04:00
Alan Guo Xiang Tan 02b84dbff2 DEV: Fix flaky instagram onebox spec by not mutating constant. 2021-07-27 13:54:14 +08:00
David Taylor 8b89787426
SECURITY: Sanitize YouTube Onebox data (#13748)
CVE-2021-32764
2021-07-15 19:31:50 +01:00
jbrw a64aea38b7
FIX: Don’t use `user_generated` images as avatar images in Oneboxed Twitter content (#13712)
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Arpit Jalan 05bdbd9f97
SECURITY: Onebox canonical links bypassing FinalDestination checks (#13605) 2021-07-01 20:09:29 +05:30
Arpit Jalan b63c9febe8
FIX: ignore canonical link to localhost (#13577) 2021-06-30 13:55:17 +05:30
Jarek Radosz 04baca593b
UX: Tweak the timestamp line in Twitter onebox (#13551)
Fixed alignment and made the color less intrusive to make the actual content pop out more.
2021-06-28 15:04:33 +02:00
Roman Rizzi fa4e5e8dad
FEATURE: Render emojis on GitHub labels when oneboxing an issue. (#13531) 2021-06-25 14:48:36 -03:00
Joffrey JAFFEUX e50b7e9111
SECURITY: ensures timeouts are correctly used on connect (#13455) 2021-06-21 17:34:01 +02:00
jbrw 09bc95d46b
FIX: Quoting Oneboxed content should exclude formatting (#13296)
* FIX: Quoting Oneboxed content should exclude formatting

When a post is quoted that includes Oneboxed content, we should not include the formatting generated by the Onebox. Rather, we should attempt to collapse the link referenced by the Onebox to a single line text link.

* DEV: fix tests
2021-06-07 13:03:53 -04:00
Arpit Jalan 2e4f07678e
FIX: IMDb links were being oneboxed as posters (#13310)
IMDb movie links were being rendered as posters. This was because
IMDb was sending `og:type` as `image` randomly in some cases. To
fix this we'll now default all IMDb links as article type. This will
ensure that the IMDb onebox link includes all the information instead
of showing just a poster without any context.
2021-06-07 18:45:59 +05:30
jbrw 461a2c334b
FIX: return an empty result if response from Amazon is missing expected attributes (#13173)
* FIX: return an empty result if response from Amazon is missing attributes

Check we have the basic attributes requires to construct a Onebox for Amazon.

This is an attempt to handle scenarios where we receive a valid 200-status response from an Amazon request that does not include the data we’re expecting.

* Update lib/onebox/engine/amazon_onebox.rb

Co-authored-by: Régis Hanol <regis@hanol.fr>

Co-authored-by: Régis Hanol <regis@hanol.fr>
2021-06-01 16:23:18 -04:00