Commit Graph

78 Commits

Author SHA1 Message Date
David Taylor 68b4fe4cf8
SECURITY: Expand and improve SSRF Protections ()
See https://github.com/discourse/discourse/security/advisories/GHSA-rcc5-28r3-23rr

Co-authored-by: OsamaSayegh <asooomaasoooma90@gmail.com>
Co-authored-by: Daniel Waterworth <me@danielwaterworth.com>
2022-11-01 16:33:17 +00:00
Bianca Nenciu 266e165885
FIX: Use only first line from commit message ()
Linking a commit from a GitHub pull request included the complete commit
message, instead of just the first line. The rest of the commit message
will be added to the body of the Onebox.
2022-10-24 22:26:48 +03:00
Bianca Nenciu 73e9875a1d
FEATURE: Handle oneboxes for complex GitHub URLs ()
GitHub PR URLs can link to a commit of the PR, a comment or a review
discussion.
2022-10-06 20:26:04 +03:00
David Taylor 2e00d4d024
DEV: Fix flaky twitter onebox behavior ()
The order in which Onebox engines are loaded is not guaranteed. Occasionally during tests, the twitter engine would be loaded before the instagram engine, and cause the Instagram Onebox spec to fail due to the lack of `Onebox.options.twitter_client`.

This commit makes the load order of Onebox engines consistent, and fixes the issue in the twitter_status_onebox.
2022-08-31 08:42:55 +08:00
Bianca Nenciu 626d50c15c
FIX: Disable Twitter onebox without API support ()
Twitter removed OpenGraph tags from their pages. We can no longer
extract all the information (for example, the quoted tweet) we need
to render Oneboxes without using their API.
2022-08-17 18:32:48 +03:00
Ghassan Maslamani d0a4bc636f
FIX: Vimeo regex pattern ()
Vimeo has two url structure:

- Normal /video_id

- Private/Unlisted /video_id/hash_string

This changes change the regex pattern thus it would be able to

catch both. Also it tolerate trailing slash.

This shall fixes:

https://meta.discourse.org/t/vimeo-embed-urls-parsed-incorrectly-in-email/231042
2022-06-30 13:13:25 -03:00
Rafael dos Santos Silva f130ec35d9
FEATURE: Use full post width for Vimeo embeds () 2022-06-30 13:08:24 -03:00
jbrw 9874fe3fb3
FIX: Improve mixcloud oneboxing ()
- Sets `https://www.mixcloud.com` as a `requires_iframe_origins` to allow the iframe content to be displayed
- Attempts to render something approximating the Mixcloud content in the preview pane of the Composer, rather than just displaying a large version of the artwork associated with the link
2022-06-27 08:32:24 +10:00
sansnumero f0c6dd5682
Add support for JSON LD in Onebox ()
* FIX: Fix a bug that is accessing the values in a hash wrongly and write tests

I decided to write tests in order to be confident in my refactor that's in the next commit.
Meanwhile I have discovered a potential bug. The `title_attr` key was accessed as a string,
but all the keys are actually symbols so it was never evaluated to be true.

irb(main):025:0> d = {key: 'value'}
=> {:key=>"value"}
irb(main):026:0> d['key']
=> nil
irb(main):027:0> d[:key]
=> "value"

* DEV: Extract methods for readability

I will be adding a new method following the conventions in place for adding a new normalizer. And this will make the readability of the `raw` block even more difficult; so I am extracting self contained private methods beforehand.

* FEATURE: Parse JSON-LD and introduce Movie object

JSON LD data is very easily transferable to Ruby objects because they contain types. If these types are mapped to Ruby objects, it is also better to make all the parsed data very explicit and easily extendable.

JSON-LD has many more standardized item types, with a full list here: https://schema.org/docs/full.html
However in order to decrease the scope, I only adapted the movie type.

* DEV: Change inheritance between normalizers

Normalizers are not supposed to have an inheritance relationships amongst each other. They are all normalizers, but all normalizing separate protocols. This is why I chose to extract a parent class and relieve Open Graph off that responsibility. Removing the parent class altogether could also a possibility, but I am keeping the scope limited to having a more accurate representation of the normalizers while making it easier to add a new one.

* Lint changes

* Bring back the Oembed OpenGraph inheritance

There is one test that caught that this inheritance was necessary. I still think modelling wise this inheritance shouldn't exist, but this can be tackled separately.

* Return empty hash if the json received is invalid

Before this change if there was a parsing error with JSON it would throw an exception. The goal of this commit is to rescue that exception and then log a warning. I chose to use Discourse's logger wrapper `warn_exception` to have the backtrace and not just used Rails logger. I considered raising an `InvalidParameters` error however if the JSON here is invalid it should not block showing of the Onebox, so logging is enough.

* Prep to support more JSONLD schema types with case

* Extract mustache template object created from JSONLD
2022-06-13 17:32:34 +02:00
Mayfield 99b0578b4c
FIX: escape youtube title when constructing onebox preview html () 2022-06-08 13:42:37 +08:00
David Taylor 8fe3934856
UX: Make YouTube playlist onebox full width to match video onebox () 2022-05-27 10:39:12 +01:00
David Taylor ff93833fdf
UX: Use committed date for GitHub oneboxes ()
Our copy says 'committed {date}`, but we were previously using the commit's authored date
2022-03-30 09:16:28 +08:00
jbrw 528c3e311a
FIX: Only display the first listed price ()
Multiple prices may be returned by Amazon (e.g. for new, and also for used). We should only display the first price.
2022-03-08 15:24:45 -05:00
jbrw fc30669db2
FIX: Support new layout on Amazon product pages ()
Some product pages on Amazon are using a new HTML structure, meaning the previous Onebox engine was unable to gather the price and/or description. This change should allow these pages to be Oneboxed.
2022-03-04 18:31:53 -05:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs ()
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Alan Guo Xiang Tan 7afe768d60
DEV: Add tests for wistia onebox. ()
Follow-up to 4ef56b0ca4
2022-02-08 13:04:32 +08:00
jbrw 4ef56b0ca4
FIX: Explicitly set `allowfullscreen` on Wistia Oneboxes () 2022-02-08 13:02:32 +11:00
Rafael dos Santos Silva 5b5cbbfe5c
FEATURE: Onebox for news.ycombinator.com () 2022-02-03 13:39:21 -03:00
Jarek Radosz 31b27b3712
FIX: Broken GitHub folder onebox logic ()
1. `html_doc.css('.Box.md')` always returns a truthy value (e.g. `[]`) so the second branch of the if-elsif never ran
2. `node&.css('text()')` was invalid code that would raise an error
3. Matching on h3 elements is no longer correct with the current html structure returned by GitHub
2022-01-17 18:32:07 +01:00
jbrw 6e925fee6f
FIX: Use basic meta description if other description tags are missing ()
When attempting to Onebox a page if there is no `meta property="og:description"` tag but there is a  `meta name="description"` tag, Onebox should try to use that value.
2021-12-17 19:36:54 -05:00
jbrw aec125b617
FIX: Display Instagram Oneboxes in an iframe ()
We are no longer able to display the image returned by Instagram directly within a Discourse site (either in the composer, or within a cooked post within a topic), so:

- Display an image placeholder in the composer preview
- A cooked post should use an iframe to display the Instagram 'embed' content
2021-11-02 14:34:51 -04:00
Dan Ungureanu 69f0f48dc0
DEV: Fix rubocop issues () 2021-10-27 11:39:28 +03:00
Faris Masad 3fbfec06fc Update replit onebox to accept .com 2021-10-19 16:37:33 -04:00
Bianca Nenciu ba81d1853b
FIX: Disable previews if diffhtml is enabled ()
diffhtml should not rerender video and audio elements so there is no
point in having these.
2021-10-08 15:57:08 +03:00
Arpit Jalan fbe9cd49b6
FIX: Vimeo private video oneboxes were broken () 2021-10-05 15:46:58 +05:30
Jarek Radosz 02a6b991fe
FIX: Correct the play icon position () 2021-09-09 15:10:32 +02:00
Chema Balsas d27d7c8cca FIX: Unescapes hash section with present to account for url-encoded chars
Sections with unreserverd characters will appear url-encoded and need to
be unescaped before using it.

Wikipedia generates 2 different spans in this case in the same page, one
with an id resulting of replacing the % symbols with . and the other with
the decoded version of the string. For example, for /wiki/foo#A%C3%A1A it
will generate:

<span id="A.C3.A1A"></span>
<span id="AáA">AáA</span>

Unescaping the `m_url_hash_name` should work in all cases to target the
proper section span.
2021-08-12 10:43:50 -04:00
Arpit Jalan bb2c48b065
FIX: update iframe url for simplecast onebox ()
https://meta.discourse.org/t/onebox-regression-simplecast-com/187911
2021-08-05 18:29:04 +05:30
Arpit Jalan a341dba5d9
FIX: update oEmbed URL for simplecast onebox () 2021-08-05 17:42:38 +05:30
jbrw 2f28ba318c
FEATURE: Onebox can match engines based on the content_type ()
* FEATURE: Onebox can match engines based on the content_type

`FinalDestination` now returns the `content_type` of a resolved URL.

`Oneboxer` passes this value to `Onebox` itself. Onebox engines can now specify a `matches_content_type` regex of content_types that the engine can handle, regardless of the URL.

`ImageOnebox` will match URLs with a content type of `image/png`, `jpg`, `gif`, `bmp`, `tif`, etc.

This will allow images that exist at a URL without a file type extension to be correctly rendered, assuming a valid `content_type` is returned.
2021-07-30 13:36:30 -04:00
David Taylor 8b89787426
SECURITY: Sanitize YouTube Onebox data ()
CVE-2021-32764
2021-07-15 19:31:50 +01:00
jbrw a64aea38b7
FIX: Don’t use `user_generated` images as avatar images in Oneboxed Twitter content ()
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Roman Rizzi fa4e5e8dad
FEATURE: Render emojis on GitHub labels when oneboxing an issue. () 2021-06-25 14:48:36 -03:00
Arpit Jalan 2e4f07678e
FIX: IMDb links were being oneboxed as posters ()
IMDb movie links were being rendered as posters. This was because
IMDb was sending `og:type` as `image` randomly in some cases. To
fix this we'll now default all IMDb links as article type. This will
ensure that the IMDb onebox link includes all the information instead
of showing just a poster without any context.
2021-06-07 18:45:59 +05:30
jbrw 461a2c334b
FIX: return an empty result if response from Amazon is missing expected attributes ()
* FIX: return an empty result if response from Amazon is missing attributes

Check we have the basic attributes requires to construct a Onebox for Amazon.

This is an attempt to handle scenarios where we receive a valid 200-status response from an Amazon request that does not include the data we’re expecting.

* Update lib/onebox/engine/amazon_onebox.rb

Co-authored-by: Régis Hanol <regis@hanol.fr>

Co-authored-by: Régis Hanol <regis@hanol.fr>
2021-06-01 16:23:18 -04:00
Penar Musaraj 06e1af2b1d
FIX: Giphy oneboxing when the response is an image () 2021-05-28 15:10:32 -04:00
Penar Musaraj 47e09700fe
FIX: Support pausing GIFs for giphy/tenor oneboxes () 2021-05-28 08:40:30 -04:00
Dan Ungureanu 723d7de18c
Various GitHub Onebox improvements ()
* FIX: Improve GitHub folder regexp in Onebox

It used to match any GitHub URL that was not matched by the other GitHub
Oneboxes and it did not do a good job at handling those. With this
change, the generic Onebox will handle the remaining URLs.

* FEATURE: Add Onebox for GitHub Actions

* FEATURE: Add Onebox for PR check runs

* FIX: Remove image from GitHub folder Oneboxes

It is a generic, auto-generated image which does not provide any value.

* DEV: Add tests

* FIX: Strip HTML comments from PR body
2021-05-27 12:38:42 +03:00
Penar Musaraj 1270c7ad15
UX: Twitter onebox layout adjustments () 2021-05-27 15:35:32 +10:00
Arpit Jalan 283b08d45f
DEV: Absorb onebox gem into core ()
* Move onebox gem in core library

* Update template file path

* Remove warning for onebox gem caching

* Remove onebox version file

* Remove onebox gem

* Add sanitize gem

* Require onebox library in lazy-yt plugin

* Remove onebox web specific code

This code was used in standalone onebox Sinatra application

* Merge Discourse specific AllowlistedGenericOnebox engine in core

* Fix onebox engine filenames to match class name casing

* Move onebox specs from gem into core

* DEV: Rename `response` helper to `onebox_response`

Fixes a naming collision.

* Require rails_helper

* Don't use `before/after(:all)`

* Whitespace

* Remove fakeweb

* Remove poor unit tests

* DEV: Re-add fakeweb, plugins are using it

* Move onebox helpers

* Stub Instagram API

* FIX: Follow additional redirect status codes ()

Don’t throw errors if we encounter 303, 307 or 308 HTTP status codes in responses

* Remove an empty file

* DEV: Update the license file

Using the copy from https://choosealicense.com/licenses/gpl-2.0/#

Hopefully this will enable GitHub to show the license UI?

* DEV: Update embedded copyrights

* DEV: Add Onebox copyright notice

* DEV: Add MIT license, convert COPYRIGHT.txt to md

* DEV: Remove an incorrect copyright claim

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
Co-authored-by: jbrw <jamie@goatforce5.org>
2021-05-26 15:11:35 +05:30
David Taylor 8fd46c04ea
Drop flash video onebox ()
Flash was discontinued by Adobe at the end of 2020. There is no need to continue OneBox support for it
2021-03-02 17:11:14 +00:00
David Taylor a3577435f7
FEATURE: Additional control of iframes in oneboxes ()
This commit adds a new site setting "allowed_onebox_iframes". By default, all onebox iframes are allowed. When the list of domains is restricted, Onebox will automatically skip engines which require those domains, and use a fallback engine.
2020-08-27 20:12:13 +01:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology ()
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo ()
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan ad5082d969 Make rubocop happy again. 2018-06-07 13:28:18 +08:00
Sam f028ffaf29 SECURITY: correct local onebox category checks
Also removes ugly "source_topic_id" from cooked posts

Patch was authored by @zogstrip

Signed-off-by: Sam <sam.saffron@gmail.com>
2018-02-14 10:40:46 +11:00
Maja Komel 018cb7f36b add a custom user onebox ()
* add custom user onebox

* add specs
2018-01-30 11:03:08 +01:00
Régis Hanol d6b22e6cc1 FIX: whitelist oneboxed iframes 2017-12-23 01:56:33 +01:00
Vinoth Kannan 230fec68ca FIX: Topic links onebox differently if end in / 2017-11-06 01:41:36 +05:30