Commit Graph

10072 Commits

Author SHA1 Message Date
Mark VanLandingham d42a1c8885
DEV: Pass recipient email address to message_builder modifiers (#27308) 2024-06-04 08:00:30 -05:00
Alan Guo Xiang Tan e97ef7e9af
FEATURE: Allow site admin to mark a user's password as expired (#27314)
This commit adds the ability for site administrators to mark users'
passwords as expired. Note that this commit does not add any client side
interface to mark a user's password as expired.

The following changes are introduced in this commit:

1. Adds a `user_passwords` table and `UserPassword` model. While the
   `user_passwords` table is currently used to only store expired
   passwords, it will be used in the future to store a user's current
   password as well.

2. Adds a `UserPasswordExpirer.expire_user_password` method which can
   be used from the Rails console to mark a user's password as expired.

3. Updates `SessionsController#create` to check that the user's current
   password has not been marked as expired after confirming the
   password. If the password is determined to be expired based on the
   existence of a `UserPassword` record with the `password_expired_at`
   column set, we will not log the user in and will display a password
   expired notice. A forgot password email is automatically send out to
   the user as well.
2024-06-04 15:42:53 +08:00
Alan Guo Xiang Tan aec892339e
DEV: Add rspec tests for `SignalTrapLogger` (#27302)
Follow-up to 23c38cbf11
2024-06-03 13:40:21 +08:00
Alan Guo Xiang Tan 23c38cbf11
DEV: Log Unicorn worker timeout backtraces to `Rails.logger` (#27257)
This commit introduces the following changes:

1. Introduce the `SignalTrapLogger` singleton which starts a single
   thread that polls a queue to log messages with the specified logger.
   This thread is necessary becasue most loggers cannot be used inside
   the `Signal.trap` context as they rely on mutexes which are not
   allowed within the context.

2. Moves the monkey patch in `freedom_patches/unicorn_http_server_patch.rb` to
   `config/unicorn.config.rb` which is already monkey patching
   `Unicorn::HttpServer`.

3. `Unicorn::HttpServer` will now automatically send a `USR2` signal to
   a unicorn worker 2 seconds before the worker is timed out by the
   Unicorn master.

4. When a Unicorn worker receives a `USR2` signal, it will now log only
   the main thread's backtraces to `Rails.logger`. Previously, it was
   `put`ing the backtraces to `STDOUT` which most people wouldn't read.
   Logging it via `Rails.logger` will make the backtraces easily
   accessible via `/logs`.
2024-06-03 12:51:12 +08:00
Alan Guo Xiang Tan dc55b645b2
DEV: Allow site administrators to mark S3 uploads with a missing status (#27222)
This commit introduces the following changes which allows a site
administrator to mark `Upload` records with the `s3_file_missing`
verification status which will result in the `Upload` record being ignored when
`Discourse.store.list_missing_uploads` is ran on a site where S3 uploads
are enabled and `SiteSetting.enable_s3_inventory` is set to `true`.

1. Introduce `s3_file_missing` to `Upload.verification_statuses`
2. Introduce `Upload.mark_invalid_s3_uploads_as_missing` which updates
   `Upload#verification_status` of all `Upload` records from `invalid_etag` to `s3_file_missing`.
3. Introduce `rake uploads:mark_invalid_s3_uploads_as_missing` Rake task
   which allows a site administrator to change `Upload` records with
`invalid_etag` verification status to the `s3_file_missing`
verificaton_status.
4. Update `S3Inventory` to ignore `Upload` records with the
   `s3_file_missing` verification status.
2024-05-30 08:37:38 +08:00
Blake Erickson f292e645b9
FEATURE: Show video thumbnail in composer (#27233)
When uploading a video, the composer will now show a thumbnail image in
the composer preview instead of just the video placeholder image.

If `enable_diffhtml_preview` is enabled the video will be rendered in
the composer preview and is playable.
2024-05-29 08:24:29 -06:00
Alan Guo Xiang Tan f76d143919
DEV: Log backtrace along with the error (#27242)
Just logging the error message is useless if we don't know where the
error message is raised from.
2024-05-29 16:13:23 +08:00
Krzysztof Kotlarek 963b9fd157
FEATURE: admin can disable flags (#27171)
UI for admins to disable system flags.
2024-05-29 14:39:58 +10:00
Alan Guo Xiang Tan e9c8e182d3
DEV: Use Unicorn logger to log Sidekiq signal handling events (#27239)
This commit updates all Sidekiq signal handling event logs to go through
Unicorn's logger instead of logging to STDOUT. Going through a proper logger
means the log messages are logged in the format which the logger has configured.
This means we get proper timestamp for the log messages.
2024-05-29 11:15:20 +08:00
Martin Brennan 9c85ea5945
DEV: Remove old TODOs for message-id formats (#27196)
Introduced back in 2022 in
e3d495850d,
our new more specific message-id format for inbound and
outbound emails has now been in use for a very long time,
we can remove the support for the old formats:

`topic/:topic_id/:post_id.:random@:host`
`topic/:topic_id@:host`
`topic/:topic_id.:random@:host`
2024-05-28 13:57:09 +10:00
Krzysztof Kotlarek 39902c148f
FIX: move ServiceRunner from Chat to Core (#27219)
In this PR service objects were moved to Core https://github.com/discourse/discourse/pull/26506

However, ServiceRunner should be moved as well. Mostly for CI to run effortlessly without loading plugins.
2024-05-28 13:55:46 +10:00
Loïc Guitaut 2a28cda15c DEV: Update to lastest rubocop-discourse 2024-05-27 18:06:14 +02:00
Osama Sayegh 361992bb74
FIX: Apply crawler rate limits to cached requests (#27174)
This commit moves the logic for crawler rate limits out of the application controller and into the request tracker middleware. The reason for this move is to apply rate limits to all crawler requests instead of just the requests that make it to the application controller. Some requests are served early from the middleware stack without reaching the Rails app for performance reasons (e.g. `AnonymousCache`) which results in crawlers getting 200 responses even though they've reached their limits and should be getting 429 responses.

Internal topic: t/128810.
2024-05-27 16:26:35 +03:00
Juan David Martínez Cubillos 7992d7a65a
DEV: Refactor to add Custom emoji sanitization (#24368)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-05-27 15:24:55 +02:00
Discourse Translator Bot 9d8044a2ee
FEATURE: Add Uyghur language (#27183)
Co-authored-by: Gerhard Schlager <gerhard.schlager@discourse.org>
2024-05-27 09:58:18 +02:00
Gerhard Schlager 9aede9c8d8
FIX: "Replace Text" didn't list "Welcome Topic" when "General" category is missing (#27182)
Replacing the text of seeded topics doesn't require categories to exist, so this change removes that requirement for updates.
2024-05-27 09:57:30 +02:00
Alan Guo Xiang Tan 6cafe59c76
DEV: Add `DISCOURSE_DUMP_BACKTRACES_ON_UNICORN_WORKER_TIMEOUT` env (#27199)
This commit adds a `DISCOURSE_DUMP_BACKTRACES_ON_UNICORN_WORKER_TIMEOUT`
environment that will allow us to dump all backtraces for all threads of
a Unicorn worker 2 seconds before it times out. In development,
backtraces are dumped to `STDOUT` and in production we will dump it to
`unicorn.stdout.log`.

We want to dump all the backtraces to make it easier to identify the
cause of a Unicorn worker timing out.
2024-05-27 12:20:38 +08:00
Ted Johansson 69205cb1e5
DEV: Catch missing translations during test runs (#26258)
This configuration makes it so that a missing translation will raise an error during test execution. Better discover there than after deploy.
2024-05-24 22:15:53 +08:00
Alan Guo Xiang Tan df16ab0758
FIX: `S3Inventory` to ignore files older than last backup restore date (#27166)
This commit updates `S3Inventory#files` to ignore S3 inventory files
which have a `last_modified` timestamp which are not at least 2 days
older than `BackupMetadata.last_restore_date` timestamp.

This check was previously only in `Jobs::EnsureS3UploadsExistence` but
`S3Inventory` can also be used via Rake tasks so this protection needs
to be in `S3Inventory` and not in the scheduled job.
2024-05-24 10:54:06 +08:00
Penar Musaraj f9192835a7
FIX: Do not reset sidebar defaults in wizard (#27156)
We removed the option to enable/disable the sidebar in the wizard in
https://github.com/discourse/discourse/pull/26926, but these lines of
code were resetting it quietly to `header_dropdown`.
2024-05-23 16:45:47 -04:00
Jan Cernik 1ed1a1f96a
FIX: Add onebox support for x.com (#27140) 2024-05-23 10:25:42 -03:00
Krzysztof Kotlarek cfbbfd177c
DEV: move post flags into database (#27125)
This is preparation for a feature that will allow admins to define their custom flags. Current behaviour should stay untouched.
2024-05-23 12:19:07 +10:00
marstall d75339af76
DEV: let reply_by_email, visit_link_to_respond email strings be modified by plugins (#27133)
* DEV: allow reply_by_email, visit_link_to_respond strings to be modified by plugins

* DEV: separate visit_link_to_respond and reply_by_email modifiers out
2024-05-22 15:33:06 -04:00
Amanda Alves Branquinho b0d95c8c78
FEATURE: Add bulk action to bookmark (#26856)
This PR aims to add bulk actions to the user's bookmarks.

After this feature, all users should be able to select multiple bookmarks and perform the actions of "deleting" or "clear reminders"
2024-05-22 12:50:21 -03:00
Régis Hanol 8f7a3e5b29 FIX: subfolder absolute links in summaries
This fixes the `PrettyText.make_all_links_absolute` to better handle subfolder.

In subfolder, when given the cooked version of a post, links to mentions includes the `Discourse.base_path` prefix. Adding the `Discourse.base_url` was doubling the `Discourse.base_path`.

The issue was hidden behind the specs which was stubbing `Discourse.base_url` instead of relying on `Discourse.base_path`.

This fixes both the "algorithm" used in `PrettyText.make_all_links_absolute` to better handle this case and correct the specs to properly handle subfolder cases.

There are lots of changes in the specs due to a refactoring to use squiggly heredoc strings for easier reading and less escaping.
2024-05-22 15:38:18 +02:00
Krzysztof Kotlarek 40d65dddf8
Revert "DEV: move post flags into database (#26951)" (#27102)
This reverts commit 7aff9806eb.
2024-05-21 16:21:07 +10:00
Krzysztof Kotlarek 7aff9806eb
DEV: move post flags into database (#26951)
This is preparation for a feature that will allow admins to define their custom flags. Current behaviour should stay untouched.
2024-05-21 13:15:32 +10:00
Alan Guo Xiang Tan 34c527d694
DEV: Pull compatible themes in tests workflow (#27093)
This commit adds a step in our tests workflow on Github actions to update the themes to
use the compatible version when not running aginast the `main` branch.
This is to ensure that we are not running
the tests for themes against an incompatible version of Discourse.
2024-05-21 10:38:41 +08:00
Ted Johansson 32aaf2e8d3
DEV: Remove deprecated AuthProvider#enabled_setting= (#27081)
AuthProvider#enabled_setting=, used primarily by plugins, has been deprecated since version 2.9, in favour of Authenticator#enabled?. This PR confirms we are seeing no more usage and removes the method.
2024-05-20 18:10:15 +08:00
Régis Hanol b908abe35a
FIX: keep topic.word_count in sync (#27065)
Whenever one creates, updates, or deletes a post, we should keep the `topic.word_count` counter in sync.

Context - https://meta.discourse.org/t/-/308062
2024-05-17 17:05:49 +02:00
Jan Cernik af6759f5e2
DEV: Use the correct open command in the version bump rake task (#27046) 2024-05-16 10:18:33 -03:00
Martin Brennan d964709644
DEV: Add more _map extensions for list site settings (#27045)
Following on from eea74e0e32,
this commit adds the automatic _map splitting shorthand
for emoji_list, tag_list site settings.
2024-05-16 13:43:10 +10:00
Alan Guo Xiang Tan e31cf66f11
FIX: `FinalDestination#get` forwarding `Authorization` header on redirects (#27043)
This commits updates `FinalDestination#get` to not forward
`Authorization` header on redirects since most HTTP clients I tested like
curl and wget does not it.

This also fixes a recent problem in `DiscourseIpInfo.mmdb_download`
where we will fail to download the databases when both `GlobalSetting.maxmind_account_id` and
`GlobalSetting.maxmind_license_key` has been set. The failure is due to
the bug above where the redirected URL given by MaxMind does not accept
an `Authorization` header.
2024-05-16 08:37:34 +08:00
Jan Cernik fb63ddd7d2
Bump version to v3.3.0.beta3-dev 2024-05-15 12:24:11 -03:00
Jan Cernik c723c126f2
Bump version to v3.3.0.beta2 2024-05-15 12:24:11 -03:00
Alan Guo Xiang Tan 2134ca9031
PERF: Optimise query for excluding topics in certain categories in TopicsFilter (#27027)
This commit optimises the database query generated by
`TopicsFilter#filter_categories` when the `-category:*` filter is used.
Previously, the method will add the `topics.category_id NOT IN
(<category ids to be excluded>)` filter to the resulting query. However,
we noticed that the performance of the query degrades as the number of
rows in the `topics` table grow and when the number of category ids to be
excluded is large.

Sample of query we ran on a large database in production to demonstrate
the improvement:

Before:

```
SELECT topics.id FROM topics WHERE topics.category_id NOT IN (83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87) ORDER BY topics.id DESC LIMIT 5;

                                                                                                       QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.317..30.165 rows=5 loops=1)
   ->  Sort  (cost=27795.34..27795.34 rows=1 width=4) (actual time=29.316..30.163 rows=5 loops=1)
         Sort Key: id DESC
         Sort Method: top-N heapsort  Memory: 25kB
         ->  Gather  (cost=1000.10..27795.33 rows=1 width=4) (actual time=0.187..26.132 rows=73478 loops=1)
               Workers Planned: 2
               Workers Launched: 2
               ->  Parallel Seq Scan on topics  (cost=0.10..26795.23 rows=1 width=4) (actual time=0.013..22.252 rows=24493 loops=3)
                     Filter: (category_id <> ALL ('{83,136,149,143,153,165,161,123,155,163,144,134,69,135,158,141,151,160,131,133,89,104,150,147,132,145,108,146,122,100,128,154,95,102,140,139,88,91,87}'::integer[]))
                     Rows Removed by Filter: 77276
 Planning Time: 0.140 ms
 Execution Time: 30.181 ms
```

After:

```
SELECT topics.id FROM topics WHERE NOT EXISTS (
  SELECT 1
  FROM unnest(array[83, 136, 149, 143, 153, 165, 161, 123, 155, 163, 144, 134, 69, 135, 158, 141, 151, 160, 131, 133, 89, 104, 150, 147, 132, 145, 108, 146, 122, 100, 128, 154, 95, 102, 140, 139, 88, 91, 87]) AS excluded_categories(category_id)
  WHERE topics.category_id IS NULL OR excluded_categories.category_id = topics.category_id
) ORDER BY topics.id DESC LIMIT 5 ;

                                                                        QUERY PLAN
-----------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=0.42..13.52 rows=5 width=4) (actual time=0.028..0.110 rows=5 loops=1)
   ->  Nested Loop Anti Join  (cost=0.42..179929.62 rows=68715 width=4) (actual time=0.027..0.109 rows=5 loops=1)
         Join Filter: ((topics.category_id IS NULL) OR (excluded_categories.category_id = topics.category_id))
         Rows Removed by Join Filter: 239
         ->  Index Scan Backward using forum_threads_pkey on topics  (cost=0.42..108925.71 rows=305301 width=8) (actual time=0.012..0.062 rows=44 loops=1)
         ->  Function Scan on unnest excluded_categories  (cost=0.00..0.39 rows=39 width=4) (actual time=0.000..0.001 rows=6 loops=44)
 Planning Time: 0.126 ms
 Execution Time: 0.124 ms
(8 rows)
```
2024-05-15 11:03:07 +08:00
dsims e6e3eaf472
FIX: avoid error from missing meta tags (#26927) 2024-05-14 11:41:53 -04:00
marstall 6df2f94bbc
DEV add modifiers to message_builder so plugins can customize subject/body/html (#26867) 2024-05-13 14:59:15 -04:00
Bianca Nenciu ebc1763aa5
FIX: Change request method for categories/search (#26976)
This commit changes request method for "categories/search" from GET to
POST to make sure that long filters can be passed to the server. For
example, category selectors with many categories are setting the full
list of selected category IDs to ensure these are filtered out from the
list of choices. This can result in a long URL that exceeds the maximum
length.
2024-05-13 14:37:17 +03:00
David Taylor 97847f6cd8
Revert "DEV: `@babel/plugin-proposal-decorators` -> `decorator-transforms` (#25290)" (#26971)
This reverts commit 0f4520867b.

This has led to two problems:

1. An incompatibility with Cloudflare's "auto minify" feature. They've deprecated this feature because of incompatibility with modern JS syntax. But unfortunately it will remain enabled on existing properties until 2024-08-05.

2. Discourse fails to boot in Safari 15. This is strange, because Safari does support all the required features in our production JS bundles. Even more strangely, things start working as soon as you open the developer tools. That suggests the cause could be a Safari bug rather than a simple incompatibility.

Reverting while we work out a path forward on both those issues.
2024-05-10 12:48:16 +01:00
Alan Guo Xiang Tan 7079698cdf
FIX: Use MaxMind supplied permalinks to download MaxMind databases (#26847)
This commit switches `DiscourseIpInfo.mmdb_download` to use the
permalinks supplied by MaxMind to download the MaxMind databases as
specified in
https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases
which states:

```
To directly download databases, follow these steps:

1. In the "Download Links" column, click "Get Permalink(s)" for the desired database.
2. Copy the permalink(s) provided in the modal window.
3. Provide your account ID and your license key using Basic Authentication to authenticate.
```

Previously we are downloading from `https://download.maxmind.com/app/geoip_download` but this is not
documented anyway on MaxMind's docs so this URL can in theory break
in the future without warning. Therefore, we are taking a proactive
approach to download the databases from MaxMind the recommended way
instead of relying on a hidden URL. This old way of downloading the
databases with only a license key will be deprecated in 3.3 and be
removed in 3.4.
2024-05-09 15:11:56 +08:00
Alan Guo Xiang Tan c8da2a33e8
FIX: Attempt to onebox even if response body exceeds `max_download_kb` (#26929)
In 95a82d608d, we lowered the default for
`Onebox.options.max_download_kb` from 10mb to 2mb for security hardening
purposes. However, this resulted in multiple bug reports where seemingly
nomral URLs stopped being oneboxed. It turns out that lowering
`Onebox.options.max_download_kb` resulted in `Onebox::Helpers::DownloadTooLarge` being raised
more often for more URLs  in `Onebox::Helpers.fetch_response` which
`Onebox::Helpers.fetch_html_doc` relies on. When
`Onebox::Helpers::DownloadTooLarge` is raised in
`Onebox::Helpers.fetch_response`, we throw away whatever response body
which we have already downloaded at that point. This is not ideal
because Nokogiri can parse incomplete HTML documents and there is a
really high chance that the incomplete HTML document still contains the
information which we need for oneboxing.

Therefore, this commit updates `Onebox::Helpers.fetch_html_doc` to not
throw away the response body when the size of the response body exceeds
`Onebox.options.max_download_size`. Instead, we just take whatever
response which we have and get Nokogiri to parse it.
2024-05-09 07:00:34 +08:00
David Taylor ece0150cb7
FIX: Ensure RequestTracker handles bubbled exceptions correctly (#26940)
This can happen for various reasons including rate limiting and middleware bugs. This should resolve the warning we're seeing in the logs

```
RequestTracker.get_data failed : NoMethodError : undefined method `[]' for nil:NilClass
```
2024-05-08 16:08:39 +01:00
David Taylor 0f4520867b
DEV: `@babel/plugin-proposal-decorators` -> `decorator-transforms` (#25290)
decorator-transforms (https://github.com/ef4/decorator-transforms) is a modern replacement for babel's plugin-proposal-decorators. It provides a decorator implementation using modern browser features, without needing to enable babel's full suite of class feature transformations. This improves the developer experience and performance.

In local testing with Google's 'tachometer' tool, this reduces Discourse's 'init-to-render' time by around 3-4% (230ms -> 222ms).

It reduces our initial gzip'd JS payloads by 3.2% (2.43MB -> 2.35MB), or 7.5% (14.5MB -> 13.4MB) uncompressed.
2024-05-08 10:40:51 +01:00
Martin Brennan ce2388e40b
FEATURE: Remove "Enable Sidebar" step from setup wizard (#26926)
This keeps coming up in user testing as something
we want to get rid of. The `navigation_menu` setting
has been set to sidebar by default for some time now,
and we are rolling out admin sidebar widely. It just
doesn't make sense to let people turn this off in
the first step of the wizard -- we _want_ people to
use the sidebar.
2024-05-08 10:00:40 +10:00
Régis Hanol 12cba2ce24 PERF: bail out of expensive post validations
Whenever a post already failed "lightweight" validations, we skip all the expensive validations (that cooks the post or run SQL queries) so that we reply as soon as possible.

Also skip validating polls when there's no "[/poll]" in the raw.

Internal ref - t/115890
2024-05-07 18:56:16 +02:00
misaka4e21 ba357dd6cc FIX: ignore SVGs when regenerating missing optimized images.
When running `rake uploads:regenerate_missing_optimized`,
a `Discourse::InvalidAccess` will be raised if an SVG
file is being processed as `OptimizedImage.prepend_decoder!`
doesn't support the svg extension. This commit simply copies
the original SVG file as the thumbnail, just like currently
`OptimizedImage.create_for` does.
2024-05-07 14:39:04 +02:00
Alan Guo Xiang Tan 0b947b6aab
DEV: Improve code comment about when ignored columns can be removed (#26894)
Ignored columns can only be dropped when its associated post-deploy
migration has been promoted to a regular migration. This is so because
Discourse doesn't rely on a schema file system to setup a brand new
database and thus the column information will be loaded by the
application first before the post-deploy migration runs.
2024-05-07 11:06:31 +08:00
Gerhard Schlager 1872047053
DEV: Uploads import script can download files (#26816)
Uploads import script can download files
2024-05-04 22:48:16 +02:00
David Taylor f230767722
FIX: Serialization of staff_writes_only (#26866) 2024-05-03 14:36:13 -04:00
Daniel Waterworth a9ca35b671
DEV: Use safer SQL functions for string queries when looking for tags (#26838) 2024-05-02 10:13:45 -05:00
David Taylor 9db5eafb15
PERF: Improve production JS build in low-memory environments (#26849)
- Use 'cheap-source-map' webpack config on low-memory machines

  This results in worse quality sourcemaps in browser dev tools, but it significantly reduces memory use in our webpack build. In approximate local testing it drops from 1100mb to 590mb. This should make the rebuild process on low-memory machines much faster and less likely to trigger OOM errors.

  In development, and on higher-memory machines, the higher-quality 'source-map' option is maintained.

- Disable Webpack's built-in `minimize` feature. Embroider already applies Terser after the webpack build is complete. There is no need to double-minimize the output.

- Update ember-cli-progress-ci to print to stderr instead of stdout. For some reason, pups (used by discourse_docker) buffers the stdout of commands and only prints when they are finished. stderr does not have this same limitation, so switching will mean sysadmins can see the progress of the ember build in real-time.

Given the number of variables it's hard to promise exact numbers. But, in my tests on a DO droplet with 1GB RAM (+2GB swap), this reduced the `ember build` portion of a `./launcher rebuild app` from ~50 minutes to ~15 minutes.
2024-05-02 11:43:59 +01:00
Daniel Waterworth 9f9c7f0a23
FIX: Handle tags with underscores correctly (#26839) 2024-05-01 20:01:39 -05:00
Alan Guo Xiang Tan a6624af66e
DEV: Add `isValidUrl` helper function to theme migrations (#26817)
This commit adds a `isValidUrl` helper function to the context in
which theme migrations are ran in. This helper function is to make it
easier for theme developers to check if a string is a valid URL or path
when writing theme migrations. This can be helpful in cases when
migrating a string based setting to `type: objects` which contain `type:
string` properties with URL validations enabled.

This commit also introduces the `UrlHelper.is_valid_url?` method
which actually checks that the URL string is of the valid format instead of
only checking if the URL string is parseable which is what `UrlHelper.relaxed_parse` does
and is not sufficient for our needs.
2024-04-30 16:45:07 +08:00
Régis Hanol bfc0f3f4cd FIX: prevent duplicate attachments in incoming emails - take 2
This is a follow up of 5fcb7c262d

It was missing the case where secure uploads is enabled, which creates a copy of the upload no matter what.

So this checks for the original_sha1 of the uploads as well when checking for duplicates.
2024-04-30 08:15:07 +02:00
Jan Cernik 9fb888923d
FIX: Do not show hidden posts in search results (#26800) 2024-04-29 12:32:02 -03:00
Régis Hanol f7a1272fa4 DEV: cleanup custom filters to prevent leaks
Ensures we clean up any custom filters added in the specs to prevent any leaks when running the specs.

Follow up to https://github.com/discourse/discourse/pull/26770#discussion_r1582464760
2024-04-29 16:11:12 +02:00
David Taylor 620f76cec1
DEV: Log original exception/backtrace for RequestTracker errors (#26802) 2024-04-29 09:05:32 +01:00
Martin Brennan edec941a87
FIX: Better tracking of topic visibility changes (#26709)
This commit introduces a few changes as a result of
customer issues with finding why a topic was relisted.
In one case, if a user edited the OP of a topic that was
unlisted and hidden because of too many flags, the topic
would get relisted by directly changing topic.visible,
instead of going via TopicStatusUpdater.

To improve tracking we:

* Introduce a visibility_reason_id to topic which functions
  in a similar way to hidden_reason_id on post, this column is
  set from the various places we change topic visibility
* Fix Post#unhide! which was directly modifying topic.visible,
  instead we use TopicStatusUpdater which sets visibility_reason_id
  and also makes a small action post
* Show the reason topic visibility changed when hovering the
  unlisted icon in topic status on topic titles
2024-04-29 10:34:46 +10:00
Régis Hanol 803c275bd7 DEV: add support for adding custom status filter
Those can be used in the /filter routes.
2024-04-26 14:04:03 +02:00
Osama Sayegh 2215fa0c8e
FIX: Pass values of objects typed settings to theme migrations (#26751)
This commit fixes a bug in theme settings migrations where values of `objects` typed theme settings aren't passed to migrations even when there are overriding values for those settings. What causes this bug is that, when creating the hash that contains all the overridden settings and will be passed to migrations, the values of `objects` typed settings are incorrectly retrieved from the `value` column (which is always nil for `objects` type) instead of `json_value`. `objects` settings are different from all other types in that they store their values in the `json_value` column and they need to be special-cased when retrieving their values.
2024-04-25 16:39:22 +03:00
David Taylor 2f2da72747
FEATURE: Add experimental tracking of 'real browser' pageviews (#26647)
Our 'page_view_crawler' / 'page_view_anon' metrics are based purely on the User Agent sent by clients. This means that 'badly behaved' bots which are imitating real user agents are counted towards 'anon' page views.

This commit introduces a new method of tracking visitors. When an initial HTML request is made, we assume it is a 'non-browser' request (i.e. a bot). Then, once the JS application has booted, we notify the server to count it as a 'browser' request. This reliance on a JavaScript-capable browser matches up more closely to dedicated analytics systems like Google Analytics.

Existing data collection and graphs are unchanged. Data collected via the new technique is available in a new 'experimental' report.
2024-04-25 11:00:01 +01:00
Alan Guo Xiang Tan 35bc27a36d
FIX: `themes:update` rake task not rolling back transaction on error (#26750)
This commit fixes a bug in the `themes:update` rake task which resulted
in the ActiveRecord transaction not being rolled back when an error was
encountered. The transaction was first introduced in
7f0682f4f2 which changed a `begin..rescue`
block to `transaction do..rescue`. The problem with that change
prevented the transaction from ever rolling back as the code block
looks something like this:

```
transaction do
  begin
    update_theme
  rescue => e
    # surpress error
  end
end
```

From the transaction's point of view now, it will never rollback even if
an error was encountered when updating the remote theme because it will
never see the error.

Instead we should have done something like this if we wanted to surpress
the errors encountered while still ensuring that the transaction is
rolled back.

```
begin
  transaction do
    update_theme
  end
rescue => e
  # surpress error
end
```
2024-04-25 15:19:23 +08:00
Alan Guo Xiang Tan 9b829216b2
DEV: Add site's DB name in `themes:update` rake task when printing error (#26749)
This is essential for us to determine which site is encountering an
error while updating remote themes. We are also including the theme's id
because themes can have the same name.
2024-04-25 12:41:14 +08:00
David Taylor dcd994a9f1
DEV: Drop workbox dependency (#26735)
This service-worker caching functionality was disabled by default in 1c58395bca, and the setting to re-enable was marked as experimental. Now we are dropping all the related logic.
2024-04-24 10:19:12 +01:00
David Taylor bca855f239
FIX: Improve handling of 'PublicExceptions' when bootstrap_error_pages enabled (#26700)
- Run the CSP-nonce-related middlewares on the generated response

- Fix the readonly mode checking to avoid empty strings being passed (the `check_readonly_mode` before_action will not execute in the case of these re-dispatched exceptions)

- Move the BlockRequestsMiddleware cookie-setting to the middleware, so that it is included even for unusual HTML responses like these exceptions
2024-04-24 09:40:13 +01:00
Penar Musaraj 98d400f7b5
DEV: Refactor discover setting reporting (#26706) 2024-04-23 09:52:01 -04:00
Michael Brown a5ef7b1999 FIX: in EmailSettingsValidator, unset smtp authentication when there's no user and password
net-smtp 0.5.0 bails when authentication is set without username/password

followup-to: 7b8d60dc, 897be759
2024-04-19 14:02:22 -04:00
Ted Johansson 9e31135eca
FEATURE: Allow users to sign in using LinkedIn OpenID Connect (#26281)
LinkedIn has grandfathered its old OAuth2 provider. This can only be used by existing apps. New apps have to use the new OIDC provider.

This PR adds a linkedin_oidc provider to core. This will exist alongside the discourse-linkedin-auth plugin, which will be kept for those still using the deprecated provider.
2024-04-19 18:47:30 +08:00
Jarek Radosz 285acf0b86
DEV: Add missing svg icons to the svg_sprite list (#26674) 2024-04-18 13:01:48 +02:00
Ted Johansson f3cad5f3a2
FIX: Correctly re-attach allowed images in activity summary e-mail (#26642)
For e-mails, secure uploads redacts all secure images, and later uses the access control post to re-attached allowed ones. We pass the ID of this post through the X-Discourse-Post-Id header. As the name suggests, this assumes there's only ever one access control post. This is not true for activity summary e-mails, as they summarize across posts.

This adds a new header, X-Discourse-Post-Ids, which is used the same way as the old header, but also works for the case where an e-mail is associated with multiple posts.
2024-04-18 10:27:46 +08:00
Krzysztof Kotlarek 98fc614162
FEATURE: mandatory fields for group site setting (#26612)
Automatically add `moderators` and `admins` auto groups to specific site settings.

In the new group-based permissions systems, we just want to check the user’s groups since it more accurately reflects reality

Affected settings:
- tag_topic_allowed_groups
- create_tag_allowed_groups
- send_email_messages_allowed_groups
- personal_message_enabled_groups
- here_mention_allowed_groups
- approve_unless_allowed_groups
- approve_new_topics_unless_allowed_groups
- skip_review_media_groups
- email_in_allowed_groups
- create_topic_allowed_groups
- edit_wiki_post_allowed_groups
- edit_post_allowed_groups
- self_wiki_allowed_groups
- flag_post_allowed_groups
- post_links_allowed_groups
- embedded_media_post_allowed_groups
- profile_background_allowed_groups
- user_card_background_allowed_groups
- invite_allowed_groups
- ignore_allowed_groups
- user_api_key_allowed_groups
2024-04-18 08:53:52 +10:00
Martin Brennan 380e5ca6cb
DEV: Move more service code to core (#26613)
This is to enable :array type attributes for Contract
attributes in services, this is a followup to the move
of services from chat to core here:

cab178a405

Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2024-04-12 13:14:19 +02:00
Gerhard Schlager 4d045bfc61
DEV: Import script should insert more data into `user_stats` table (#26551)
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.

This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
2024-04-11 14:05:21 +02:00
Martin Brennan b7a2d29b7b
DEV: Introduce post_should_secure_uploads? plugin modifier (#26508)
This modifier allows plugins to alter the outcome of
`should_secure_uploads?` on a Post record, for cases when
plugins need post-attached uploads to always be secure (or
not secure) in specific scenarios.
2024-04-10 12:02:44 +10:00
Alan Guo Xiang Tan e2ced85757
DEV: Allow enum typed theme objects property to be optional (#26571)
This commit changes enum typed theme objects property to be optional.
Previously, an enum typed property is always required but we have found
that this might not be ideal so we want to change it.
2024-04-09 11:26:24 +08:00
Martin Brennan 0d0dbd391a
DEV: Rename with_secure_uploads? to should_secure_uploads? on Post (#26549)
This method name is a bit confusing; with_secure_uploads implies
it may return a block or something with the uploads of the post,
and has_secure_uploads implies that it's checking whether the post
is linked to any secure uploads.

should_secure_uploads? communicates the true intent of this method --
which is to say whether uploads attached to this post should be
secure or not.
2024-04-09 13:23:11 +10:00
Alan Guo Xiang Tan 6edadeab27
FIX: `DiscourseIpInfo.mmdb_download` incorrectly joining URLs (#26545)
This commit changes `DiscourseIpInfo.mmdb_download` to use `File.join`
instead of `URI.join` when `GlobalSetting.maxmind_mirror_url` has been
configured. This is necessary because `URI.join` does not work the way I
expect it to work when I implemented it previously.

`URI.join("http://www.example.com/mirror", "test.tar.gz") results in
`http://www.example.com/test.tar.gz` instead of our expected
`http://www.exmaple.com/mirror/test.tar.gz`. For our simple use case
here, `File.join` is sufficient.
2024-04-08 10:27:40 +08:00
Blake Erickson 8da49b5664
FEATURE: Add message to bulk close topics (#26535)
This will allow you to type a single message when bulk closing topics
that will be applied to every topic, that way they all have the same
message.
2024-04-05 14:56:52 -06:00
Rafael dos Santos Silva dd83a07550
FEATURE: Hide summarization in PMs (#26532) 2024-04-05 12:12:59 -03:00
Jan Cernik cab178a405
DEV: Move chat service objects into core (#26506) 2024-04-04 10:57:41 -03:00
Gerhard Schlager 82c62fe44f
DEV: Correctly pluralize error messages (#26469) 2024-04-04 15:02:09 +02:00
David Taylor 0ad4034c58
DEV: Add a note about the version bump rake tasks (#26512) 2024-04-04 11:59:29 +01:00
Gerhard Schlager 689d2754f6
DEV: Bump the version to 3.3.0.beta2-dev (#26499)
This was accidentally omitted during the beta1 release process
2024-04-03 22:08:04 +01:00
Vinoth Kannan 9dc6325821
DEV: add logo URL and locale details to the Discover stats. (#26320)
We will be collecting the logo URL and the site's default locale values along with existing basic details to display the site on the Discourse Discover listing page. It will be included only if the site is opted-in by enabling the "`include_in_discourse_discover`" site setting.

Also, we no longer going to use `about.json` and `site/statistics.json` endpoints retrieve these data. We will be using only the `site/basic-info.json` endpoint.
2024-04-04 00:22:28 +05:30
Osama Sayegh 3d4faf3272
FEATURE: Merge discourse-automation (#26432)
Automation (previously known as discourse-automation) is now a core plugin.
2024-04-03 18:20:43 +03:00
David Taylor 929b4f89d6
DEV: Send proper 'stop' notification in turbo_rspec (#26488)
Doesn't actually seem to be used by any of our formatters, but let's send the proper data anyway for future-proofing. Followup to ff6cb1bc05 and 8098876bfa
2024-04-03 14:00:47 +01:00
David Taylor ff6cb1bc05
DEV: Fix turbo_rspec formatters by sending real 'Start' notification (#26487) 2024-04-03 13:27:39 +01:00
David Taylor 8098876bfa
DEV: Restore RSpec 'documentation' output, but collapse in CI (#26485)
This reverts commit ef895f1c32 and 57df0d526e, but adds GitHub actions line grouping to the verbose output to reduce scrolling.
2024-04-03 10:54:55 +01:00
David Taylor 1e42e86601
DEV: Restore qunit 'tap' reporter, but collapse verbose output in CI (#26484)
This reverts commit 5adfb299ac, but uses GitHub actions grouping to achieve the same result: less scrolling to see failures.
2024-04-03 10:22:20 +01:00
Krzysztof Kotlarek ba04fc6a01
FEATURE: ignore manually deactivated users when purging (#26478)
When a user is manually deactivated, they should not be deleted by our background job that purges inactive users.

In addition, site settings keywords should accept an array of keywords.
2024-04-03 14:06:31 +11:00
Penar Musaraj 1eb70973a2
DEV: allow themes to render their own custom homepage (#26291)
This PR adds a theme modifier and route so that custom themes can opt to show their own homepage. See PR description for example usage.
2024-04-02 11:05:08 -04:00
Vinoth Kannan cd43985de0
UX: rename the word "Upgrade" to "Update" in the UI. (#26396)
In this PR, all references in the UI to the word "`upgrade`" are changed to "`update`". This is to differentiate the update process in self-hosted sites from the plan "upgrade" process in hosted sites.

Follow-up to the PR: https://github.com/discourse/docker_manager/pull/208
2024-04-02 20:04:37 +05:30
Alan Guo Xiang Tan 9182501366
DEV: Introduce `maxmind_mirror_url` GlobalSetting (#26458)
Why this change?

This allows downloading the MaxMind databases from a mirror in cases
where downloading directly from MaxMind's API endpoint is problematic
due to API limits.
2024-04-02 14:53:53 +08:00
Alan Guo Xiang Tan 972db687fe
FIX: `maxmind:refresh` does not respect `refresh_maxmind_db_during_precompile_days` (#26457)
Why this change?

We currently support `GlobalSetting.refresh_maxmind_db_during_precompile_days` which
should cache the maxmind databases on disk for the configured number of
days before it downloads the databases from maxmind again via the API.
This was previously added to help us avoid hitting the API rate limit from maxmind.
However, there was a bug in the `copy_maxmind` when we copied the latest
downloaded database to the cache directory. In particular, `FileUtils.cp` was called with
`preserve: true` which would preserve the modified time of the file
being copied. This is problematic because download the database from
maxmind on 2 April 2024 can give us a file with an mtime of 29 March
2024. If `GlobalSetting.refresh_maxmind_db_during_precompile_days` is
set to `2` for example, the cache will never be used since we will
think that the file has been downloaded for more than 2 days in our
checks.

What is the fix here?

While we want to preserve the owner and group of the file, we do not
want to preserve the modified time and hence we will call
`FileUtils.touch` when copying the file.
2024-04-02 14:25:05 +08:00
Alan Guo Xiang Tan a84757fd91
FIX: Error not being raised for required typed categories property (#26443)
Why this change?

For a schema like this:

```
schema = {
  name: "section",
  properties: {
    category_property: {
      type: "categories",
      required: true,
    },
  },
}
```

When the value of the property is set to an empty array, we are not
raising an error which we should because the property is marked as
required.
2024-04-01 10:11:40 +08:00
carehabit 11877f3b9c
DEV: remove repetitive words (#26439) 2024-04-01 06:23:21 +08:00
Alan Guo Xiang Tan a670d6d4af
DEV: Change group type to groups type for theme object schema (#26417)
Why this change?

This is a follow-up to 86b2e3a.

Basically, we want to allow people to select more than 1 group as well.

What does this change do?

1. Change `type: group` to `type: groups` and support `min` and `max`
   validations for `type: groups`.

2. Fix the `<SchemaThemeSetting::Types::Groups>` component to support the
   `min` and `max` validations and switch it to use the `<GroupChooser>` component
   instead of the `<ComboBoxComponent>` component which previously only supported
   selecting a single group.
2024-03-28 22:05:48 +08:00
Ted Johansson 0c875cb4d5
DEV: Make problem check registration more explicit (#26413)
Previously the problem check registry simply looked at the subclasses of ProblemCheck. This was causing some confusion in environments where eager loading is not enabled, as the registry would appear empty as a result of the classes never being referenced (and thus never loaded.)

This PR changes the approach to a more explicit one. I followed other implementations (bookmarkable and hashtag autocomplete.) As a bonus, this now has a neat plugin entry point as well.
2024-03-28 14:00:47 +08:00
Angus McLeod 7dc552c9cc
DEV: Add `import_embed_unlisted` site setting (#26222) 2024-03-27 08:57:43 -04:00