Commit Graph

8837 Commits

Author SHA1 Message Date
jbrw 9874fe3fb3
FIX: Improve mixcloud oneboxing (#17237)
- Sets `https://www.mixcloud.com` as a `requires_iframe_origins` to allow the iframe content to be displayed
- Attempts to render something approximating the Mixcloud content in the preview pane of the Composer, rather than just displaying a large version of the artwork associated with the link
2022-06-27 08:32:24 +10:00
Jarek Radosz d1d6868325
DEV: Update xss.js (#17216)
* DEV: Update xss.js

* Fix our hack
2022-06-24 10:28:05 +08:00
Penar Musaraj 3baefa25b5
FIX: Use first supported type item when JSON-LD returns array (#17217) 2022-06-23 13:02:01 -04:00
Jarek Radosz 624c684d51
DEV: Transpile all plugin js by default (#17175)
Goodbye `# transpile_js: true`? 🙂
2022-06-21 22:07:10 +02:00
Roman Rizzi e0ba35350e
FEATURE: Custom unsubscribe options (#17090)
With this change, plugins can create custom unsubscribe keys, extend the unsubscribe view with custom preferences, and decide how they are updated.
2022-06-21 15:49:47 -03:00
Alan Guo Xiang Tan 3562c598ef
DEV: Upgrade Sidekiq to 6.5 (#17142) 2022-06-21 09:23:36 +08:00
Jarek Radosz c9d3c45ba9
DEV: Remove obsolete parts of our custom loader (#17157) 2022-06-20 18:40:25 +02:00
Jarek Radosz a1170970db
FIX: Add wizard.js to ember-cli assets (#17153) 2022-06-20 17:18:35 +02:00
Jarek Radosz 2c1fc28d00
DEV: Remove ember-cli flags from the backend (#17147)
…and other auxiliary code

* Restore `QUNIT_EMBER_CLI` flag warning
* Add `ALLOW_EMBER_CLI_PROXY_BYPASS`
2022-06-20 16:33:05 +02:00
David Taylor 1b4692039e DEV: Remove legacy JS manifests and vendored scripts
Now that we've switched to Ember CLI, these things are no longer used.

- These sprockets manifests are superceded by the assets generated by ember cli
- These vendored scripts are now fetched by ember-auto-import at compile time
2022-06-20 15:01:06 +01:00
Osama Sayegh 5176c689e9
UX: Change wording for 'regular' categories to 'normal' (#17134)
At some point in the past we decided to rename the 'regular' notification state of topics/categories to 'normal'. However, some UI copy was missed when the initial renaming was done so this commit changes the spots that were missed to the new name.
2022-06-20 06:49:33 +03:00
Discourse Translator Bot 7ac485fb9f
FEATURE: Add Croatian language (#17130)
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2022-06-18 00:18:22 +02:00
Gerhard Schlager 1022d8872c
FIX: Add missing pluralization rules for Bosnian (#17131) 2022-06-18 00:17:15 +02:00
Jarek Radosz 2eeb50dfc1
FIX: Extra-locale merging didn't account for fallbacks (#17128)
Regressed in #17027
2022-06-17 19:34:08 +02:00
David Taylor 12980418ae
DEV: Disable the use of 'legacy' Ember assets (#17127)
Anyone still using `EMBER_CLI_PROD_ASSETS=0` in development or production will be gracefully switched to Ember CLI. In development, a repeated message will be logged to STDERR.

Similarly, passing `QUNIT_EMBER_CLI=0` to the qunit rake task will now do nothing. A warning will be printed, and ember-cli mode will be used. Note that we've chosen not to fail the task, so that existing plugin/theme CI jobs don't immediately start failing. We may switch to a hard fail in the coming days/weeks.
2022-06-17 16:51:28 +01:00
Jarek Radosz 91bc4442d1
DEV: Remove wizard step from docker tests (#17125)
Wizard tests are now in core.
2022-06-17 15:35:18 +02:00
Jarek Radosz fcb4e5a1a1
DEV: Make wizard an ember addon (#17027)
Co-authored-by: David Taylor <david@taylorhq.com>
2022-06-17 14:50:21 +02:00
David Taylor e0ff367b49
DEV: Stop logging error response body in FileHelper (#17120)
This doesn't cope well with gzipped, binary, or large responses. Ideally we would teach FinalDestination to safely retrieve and decode some of the response body. But for now, let's remove the broken implementation.
2022-06-17 11:50:30 +01:00
Kris dfba188bd6
UX: remove extra whitespace in github onebox (#17111) 2022-06-17 11:09:45 +10:00
Gerhard Schlager 888f50543d
DEV: Fix `javascript:update` rake task (#17098)
* The `javascript:update` rake task failed because recent versions of chart.js use a lowercase filename (`chart.min.js` instead of `Chart.min.js`)

* Changed `loadScript()` to use lowercase keys to lookup scripts

* `svg-arrow.css` seems to have changed slightly (linebreak at the end of file)
2022-06-15 19:49:04 +02:00
Jarek Radosz f723b4c322
FIX: Handle sites with more than 1 JSON-LD element (#17095)
A followup to #17007
2022-06-15 02:55:55 +02:00
Martin Brennan 4d3c1ceb44
FEATURE: Log the SMTP response in EmailLog (#17056)
When sending emails with delivery_method_options -> return_response
set to true, the SMTP sending code inside Mail will return the SMTP
response when calling deliver! for mail within the app. This commit
ensures that Email::Sender captures this response if it is returned
and stores it against the EmailLog created for the sent email.

A follow up PR will make this visible within the admin email UI.
2022-06-15 10:28:30 +10:00
Gerhard Schlager 96e87af605
FIX: Rake tasks related to uploads were broken (#17085) 2022-06-14 11:05:03 +02:00
Blake Erickson f313276966
Version bump to v2.9.0.beta5 (#17077) 2022-06-13 15:27:06 -06:00
sansnumero f0c6dd5682
Add support for JSON LD in Onebox (#17007)
* FIX: Fix a bug that is accessing the values in a hash wrongly and write tests

I decided to write tests in order to be confident in my refactor that's in the next commit.
Meanwhile I have discovered a potential bug. The `title_attr` key was accessed as a string,
but all the keys are actually symbols so it was never evaluated to be true.

irb(main):025:0> d = {key: 'value'}
=> {:key=>"value"}
irb(main):026:0> d['key']
=> nil
irb(main):027:0> d[:key]
=> "value"

* DEV: Extract methods for readability

I will be adding a new method following the conventions in place for adding a new normalizer. And this will make the readability of the `raw` block even more difficult; so I am extracting self contained private methods beforehand.

* FEATURE: Parse JSON-LD and introduce Movie object

JSON LD data is very easily transferable to Ruby objects because they contain types. If these types are mapped to Ruby objects, it is also better to make all the parsed data very explicit and easily extendable.

JSON-LD has many more standardized item types, with a full list here: https://schema.org/docs/full.html
However in order to decrease the scope, I only adapted the movie type.

* DEV: Change inheritance between normalizers

Normalizers are not supposed to have an inheritance relationships amongst each other. They are all normalizers, but all normalizing separate protocols. This is why I chose to extract a parent class and relieve Open Graph off that responsibility. Removing the parent class altogether could also a possibility, but I am keeping the scope limited to having a more accurate representation of the normalizers while making it easier to add a new one.

* Lint changes

* Bring back the Oembed OpenGraph inheritance

There is one test that caught that this inheritance was necessary. I still think modelling wise this inheritance shouldn't exist, but this can be tackled separately.

* Return empty hash if the json received is invalid

Before this change if there was a parsing error with JSON it would throw an exception. The goal of this commit is to rescue that exception and then log a warning. I chose to use Discourse's logger wrapper `warn_exception` to have the backtrace and not just used Rails logger. I considered raising an `InvalidParameters` error however if the JSON here is invalid it should not block showing of the Onebox, so logging is enough.

* Prep to support more JSONLD schema types with case

* Extract mustache template object created from JSONLD
2022-06-13 17:32:34 +02:00
Gabe Pacuilla c04cb223f3
FIX: twitter onebox keeps whitespace for expanded links (#17055) 2022-06-10 01:20:26 -04:00
Sérgio Saquetim 300f835703
DEV: Supress logs when RetrieveTitle.crawl fails with Net::ReadTimeout errors (#16971)
This PR changes the rescue block to rescue only Net::TimeoutError exceptions and removes the log line to prevent clutter the logs with errors that are ignored. Other errors can bubble up because they're errors we probably want to know about
2022-06-09 16:30:22 -03:00
Bianca Nenciu 9db8f00b3d
FEATURE: Create upload_references table (#16146)
This table holds associations between uploads and other models. This can be used to prevent removing uploads that are still in use.

* DEV: Create upload_references
* DEV: Use UploadReference instead of PostUpload
* DEV: Use UploadReference for SiteSetting
* DEV: Use UploadReference for Badge
* DEV: Use UploadReference for Category
* DEV: Use UploadReference for CustomEmoji
* DEV: Use UploadReference for Group
* DEV: Use UploadReference for ThemeField
* DEV: Use UploadReference for ThemeSetting
* DEV: Use UploadReference for User
* DEV: Use UploadReference for UserAvatar
* DEV: Use UploadReference for UserExport
* DEV: Use UploadReference for UserProfile
* DEV: Add method to extract uploads from raw text
* DEV: Use UploadReference for Draft
* DEV: Use UploadReference for ReviewableQueuedPost
* DEV: Use UploadReference for UserProfile's bio_raw
* DEV: Do not copy user uploads to upload references
* DEV: Copy post uploads again after deploy
* DEV: Use created_at and updated_at from uploads table
* FIX: Check if upload site setting is empty
* DEV: Copy user uploads to upload references
* DEV: Make upload extraction less strict
2022-06-09 09:24:30 +10:00
Loïc Guitaut 03732049ed DEV: Use nested form for `FreedomPatches` module
It seems that not using the nested form for the `FreedomPatches`
namespace is breaking stuff in some dev env.
2022-06-08 15:10:26 +02:00
Loïc Guitaut fff3ff11c0 FIX: Make disabling TLS in mail possible again
Following the Rails 7 upgrade, the `DISCOURSE_SMTP_ENABLE_START_TLS`
setting doesn’t work anymore. This is because Rails upgraded the
`net-smtp` gem to the 0.3.1 version which enables `starttls` by default.
The `mail` gem doesn’t support this new behavior yet and doesn’t know
how to disable TLS. This should be fixed in an upcoming release.

Meanwhile applying this patch allows us to get back the previous
behavior which is expected by many.
2022-06-08 14:04:05 +02:00
Mayfield 99b0578b4c
FIX: escape youtube title when constructing onebox preview html (#16999) 2022-06-08 13:42:37 +08:00
Alan Guo Xiang Tan 82ac698d4f
FIX: Missing tracked sub category topics from tracked topic list (#17034)
Follow-up to 7ae647d092
2022-06-08 10:45:59 +08:00
David Taylor 5238f6788c
FEATURE: Allow hotlinked media to be blocked (#16940)
This commit introduces a new site setting: `block_hotlinked_media`. When enabled, all attempts to hotlink media (images, videos, and audio) will fail, and be replaced with a linked placeholder. Exceptions to the rule can be added via `block_hotlinked_media_exceptions`.

`download_remote_image_to_local` can be used alongside this feature. In that case, hotlinked images will be blocked immediately when the post is created, but will then be replaced with the downloaded version a few seconds later.

This implementation is purely server-side, and does not impact the composer preview.

Technically, there are two stages to this feature:

1. `PrettyText.sanitize_hotlinked_media` is called during `PrettyText.cook`, and whenever new images are introduced by Onebox. It will iterate over all src/srcset attributes in the post HTML and check if they're allowed. If not, the attributes will be removed and replaced with a `data-blocked-hotlinked-src(set)` attribute

2. In the `CookedPostProcessor`, we iterate over all `data-blocked-hotlinked-src(set)` attributes and check whether we have a downloaded version of the media. If yes, we update the src to use the downloaded version. If not, the entire media element is replaced with a placeholder. The placeholder is labelled 'external media', and is a link to the offsite media.
2022-06-07 15:23:04 +01:00
Jarek Radosz 1a5dbbf430
FIX: Correctly handle invalid auth cookies (#16995)
Previously it would blow up on invalid utf byte sequences. This was a source of spec flakiness.
2022-06-07 13:00:25 +02:00
Daniel Waterworth d7bc395182
FIX: Seed multisite dbs after migrating in development (#17017)
Dumping the schema cache reset the current_db and we only need to do
this once.
2022-06-06 14:43:43 -05:00
Martin Brennan f94682e2c4
FIX: Do not use SVGs for twitter:image metadata (#16973)
Twitter does not allow SVGs to be used for twitter:image
metadata (see https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup)
so we should fall back to the site logo if the image option
provided to `crawlable_meta_data` or SiteSetting.site_twitter_summary_large_image_url
is an SVG, and do not add the meta tag for twitter:image at all
if the site logo is an SVG.
2022-06-03 09:02:57 +10:00
tshenry 08a8c27f16
DEV: Mark discourse-group-tracker as official (#16574) 2022-06-01 15:07:51 +01:00
Alan Guo Xiang Tan 1e9f132b15
FIX: Topic list nav items count not respecting tracked filter. (#16935)
This commit seeks to only handle the `f=tracked` and `filter=tracked`
query params for a topic list. There are other "hidden" filters for a
topic list which can be activated by passing the right query param to
the request. However, they are hidden because there is no way to
activate those filters via the UI. We are handling the `f=tracked`
filter because we will soon be adding a link that allows a user to
quickly view their tracked topics.
2022-06-01 14:54:42 +08:00
Alan Guo Xiang Tan 7ae647d092
FIX: tracked filter did not account for max_category_nesting of 3 (#16963) 2022-06-01 12:09:58 +08:00
Sam 020c77440e
FEATURE: allow for overlapping DiscourseConnect secrets per domain (#16915)
Previously we limited Discourse Connect provider to 1 secret per domain.

This made it pretty awkward to cycle secrets in environments where config
takes time to propagate

This change allows for the same domain to have multiple secrets

Also fixes internal implementation on DiscourseConnectProvider which was
not thread safe as it leaned on class variables to ferry data around

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
Co-authored-by: David Taylor <david@taylorhq.com>
2022-05-31 15:24:04 +10:00
David Taylor 38324da6db
DEV: Add event for plugins to customize system messages (#16953) 2022-05-30 14:01:21 +01:00
David Taylor 8fe3934856
UX: Make YouTube playlist onebox full width to match video onebox (#16936) 2022-05-27 10:39:12 +01:00
Martin Brennan 61b9e3ee30
FIX: InlineOneboxer watched word censor error (#16921)
In 7328a2bfb0 we changed the
InlineOneboxer#onebox_for method to run the title of the
onebox through WatchedWord#censor_text. However, it is
allowable for the title to be nil, which was causing this
error in production:

> NoMethodError : undefined method gsub for nil:NilClass

We just need to check whether the title is nil before trying
to censor it.
2022-05-26 14:01:44 +10:00
Martin Brennan 641c4e0b7a
FEATURE: Make S3 presigned GET URL expiry configurable (#16912)
Previously we hardcoded the DOWNLOAD_URL_EXPIRES_AFTER_SECONDS const
inside S3Helper to be 5 minutes (300 seconds). For various reasons,
some hosted sites may need this to be longer for other integrations.

The maximum expiry time for presigned URLs is 1 week (which is
604800 seconds), so that has been added as a validation on the
setting as well. The setting is hidden because 99% of the time
it should not be changed.
2022-05-26 09:53:01 +10:00
Bianca Nenciu 7328a2bfb0
FIX: Apply censored words to inline onebox (#16873)
Censored watched words were not censored inside the title of an inline
oneboxes. Malicious users could exploit this behaviour to insert bad
words. The same issue has been fixed for regular Oneboxes in commit
d184fe59ca.
2022-05-25 14:51:47 +03:00
Alan Guo Xiang Tan 8471eabf62 DEV: Add env to skip wizard tests 2022-05-25 13:48:09 +08:00
Penar Musaraj 8222810099
FIX: Limits for PM and group header search (#16887)
When searching for PMs or PMs in a group inbox, results in the header search were not being limited to 5 with a "More" link to the full page search. This PR fixes that.

It also simplifies the logic and updates the search API docs to include recently added `in:messages` and `group_messages:groupname` options.
2022-05-24 11:31:24 -04:00
Martin Brennan 1ee973e6e2
DEV: Add save_user_preferences option to BookmarkManager (#16894)
When saving / creating bookmarks, we have code to save
the user's preference of bookmark_auto_delete_preference
to their user_options.

Unfortunately this can cause weirdness when plugins
have code using BookmarkManager to set the auto delete preference for
only a specific bookmark.

This commit introduces a save_user_preferences option (false
by default) so that this user preference is not saved unless
specified by the consumer of BookmarkManager, so plugins will
not have to worry about it.
2022-05-24 11:13:21 +10:00
Bianca Nenciu 6c8f491dc3
DEV: Allow plugins to register Onebox handlers (#16870)
This targets only the local Oneboxes and allows plugins to customize
regular or inline Oneboxes for routes inside the site.
2022-05-23 20:02:02 +03:00
David Taylor bf6f8299a7 FEATURE: Pull hotlinked images immediately after posting
Previously, with the default `editing_grace_period`, hotlinked images were pulled 5 minutes after a post is created. This delay was added to reduce the chance of automated edits clashing with user edits.

This commit refactors things so that we can pull hotlinked images immediately. URLs are immediately updated in the post's `cooked` HTML. The post's raw markdown is updated later, after the `editing_grace_period`.

This involves a number of behind-the-scenes changes including:

- Schedule Jobs::PullHotlinkedImages immediately after Jobs::ProcessPost. Move scheduling to after the `update_column` call to avoid race conditions

- Move raw changes into a separate job, which is delayed until after the ninja-edit window

- Move disable_if_low_on_disk_space logic into the `pull_hotlinked_images` job

- Move raw-parsing/replacing logic into `InlineUpload` so it can be easily be shared between `UpdateHotlinkedRaw` and `PullUserProfileHotlinkedImages`
2022-05-23 14:28:02 +01:00