Commit Graph

5978 Commits

Author SHA1 Message Date
Ted Johansson f08c6d2756
DEV: Switch over category settings to new table - Part 3 (#20657)
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.

This PR moves the require_topic_approval and require_reply_approval from custom fields to the new CategorySetting model.

This PR is nearly identical to #20580, which migrated num_auto_bump_daily, but since these are slightly more sensitive, they are moved after the previous one is verified.
2023-09-12 09:51:49 +08:00
Alan Guo Xiang Tan d2e4b32c87
DEV: Add support for uploading a theme from a directory in system tests (#23402)
Why this change?

Currently, we do not have an easy way to test themes and theme components
using Rails system tests. While we support QUnit acceptance tests for
themes and theme components, QUnit acceptance tests stubs out the server
and setting up the fixtures for server responses is difficult and can lead to a
frustrating experience. System tests on the other hand allow authors to
set up the test fixtures using our fabricator system which is much
easier to use.

What does this change do?

In order for us to allow authors to run system tests with their themes
installed, we are adding a `upload_theme` helper that is made available
when writing system tests. The `upload_theme` helper requires a single
`directory` parameter where `directory` is the directory of the theme
locally and returns a `Theme` record.
2023-09-12 07:38:47 +08:00
Bianca Nenciu 4db5310135
DEV: Remove unused topic_create_allowed_category_ids (#23463) 2023-09-08 12:03:22 +03:00
Martin Brennan c532f6eb3d
FEATURE: Secure uploads in PMs only (#23398)
This adds a new secure_uploads_pm_only site setting. When secure_uploads
is true with this setting, only uploads created in PMs will be marked
secure; no uploads in secure categories will be marked as secure, and
the login_required site setting has no bearing on upload security
either.

This is meant to be a stopgap solution to prevent secure uploads
in a single place (private messages) for sensitive admin data exports.
Ideally we would want a more comprehensive way of saying that certain
upload types get secured which is a hybrid/mixed mode secure uploads,
but for now this will do the trick.
2023-09-06 09:39:09 +10:00
Arpit Jalan e5f3c26d20
FEATURE: add group filter for admin reports (#23381)
FEATURE: add group filter for admin reports

DEV: add plugin outlet for admin dashboard tabs
2023-09-05 11:17:18 +05:30
Joffrey JAFFEUX f1d8cd529e
Revert "Revert "PERF: Cache each theme field value once (#23192)" (#23354)" (#23356)
This reverts commit 9821ca9413.
2023-08-31 14:12:03 -05:00
Joffrey JAFFEUX 9821ca9413
Revert "PERF: Cache each theme field value once (#23192)" (#23354)
This reverts commit 82a56334a3.
2023-08-31 19:04:43 +02:00
Daniel Waterworth 82a56334a3
PERF: Cache each theme field value once (#23192)
Previously, theme fields from components would be cached for each of
their parent themes.
2023-08-31 11:24:02 -05:00
Penar Musaraj 006a5166e5
DEV: Refactor rp_id and rp_name (#23339)
They're both constant per-instance values, there is no need to store them
in the session. This also makes the code a bit more readable by moving
the `session_challenge_key` method up to the `DiscourseWebauthn` module.
2023-08-31 09:11:23 -04:00
Renato Atilio 58b49bce41
FEATURE: support to initial values for form templates through /new-topic (#23313)
* FEATURE: adds support for initial values through /new-topic to form templates
2023-08-29 18:41:33 -03:00
Ted Johansson 4b52269827
DEV: Move option to delete user under reviewable reject menu (#23257)
Follow-up to #23199 in which we moved the "delete user" options under the relevant action menu for flagged post. This change does the same, but to queued posts.
2023-08-27 10:05:05 +08:00
Ted Johansson 12cdd7db1a
DEV: Move option to delete user under reviewable agree menu (#23199)
As per discussion, we want to move the options to delete the user under the "Yes" menu of the review queue, since these options are often the most recommended, but also frequently missed because they are tucked away under their own menu.

In addition to the move, I removed the title case of the options so that it matches the other options in the "Yes" menu.
2023-08-25 10:53:56 +08:00
Jarek Radosz 70f1cc5552
DEV: Use esbuild to make DiscourseJsProcessor (#23223)
Reverts e2705df and re-lands #23187 and #23219.

The issue was incorrect order of execution of Rails' `assets:precompile` task in our own precompilation stack.

Co-authored-by: David Taylor <david@taylorhq.com>
2023-08-24 16:36:22 +02:00
David Taylor e2705df0f4
Revert "DEV: Use esbuild to make DiscourseJsProcessor (#23187)" (#23221)
This reverts commit 4dfe25d062 and 4fdeb6281e. We are investigating an issue related to asset compilation and S3 assets
2023-08-24 13:25:44 +01:00
Jarek Radosz 4dfe25d062
DEV: Use esbuild to make DiscourseJsProcessor (#23187)
Co-authored-by: David Taylor <david@taylorhq.com>
2023-08-24 12:43:59 +02:00
Mark VanLandingham 730f652255
DEV: Add plugin modifier locations for user search locations (#23169) 2023-08-21 12:23:42 -05:00
Juan David Martínez Cubillos 477a5dd371
FEATURE: Digest suppression by tags (#23089)
* FEATURE: Digest suppression by tags

* fixed stree issues

* fixed code so untagged topics are not suppressed when suppressing certain tags
2023-08-18 14:28:20 -05:00
Joffrey JAFFEUX b2b84cc957
FEATURE: implements user based sidebar mode (#23078) 2023-08-18 20:33:07 +02:00
David Taylor 82b16f4f47
DEV: Do not manipulate theme module paths at build-time (#23148)
Manipulating theme module paths means that the paths you author are not the ones used at runtime. This can lead to some very unexpected behavior and potential module name clashes. It also meant that the refactor in 16c6ab8661 was unable to correctly match up theme connector js/templates.

While this could technically be a breaking change, I think it is reasonably safe because:

1. Themes are already forced to use relative paths when referencing their own modules (since they're namespaced based on the site-specific id). The only time this might be problematic is when theme tests reference modules in the theme's main `javascripts` directory

2. For things like components/services/controllers/etc. our custom Ember resolver works backwards from the end of the path, so adding `discourse/` in the middle will not affect resolution.
2023-08-18 18:15:23 +01:00
marstall 0dd1ee2e09
FIX: correct bulk invite expire time for DST (#23073)
This is a bug that happens only when the current date is less than 90 days from a date on which the time zone transitions into or out of Daylight Savings Time.

In these conditions, bulk invites show the time of day of their expiration as being 1 hour later than the current time.

Whereas it should match the time of day the invite was generated.

This is because the server has not been using the user's timezone in calculating the expiration time of day. This PR fixes issue by considering the user's timezone when doing the date math.

https://meta.discourse.org/t/bulk-invite-logic-to-generate-expire-date-bug/274689
2023-08-18 12:33:40 -04:00
Daniel Waterworth 91f560a521
DEV: Make every DistributedCache lazily instantiated (#23147) 2023-08-18 10:59:11 -05:00
Selase Krakani 87ebbec9b2
FIX: Pending post deletion by creator (#23130)
`ReviewableQueuedPost` got refactored a while back to use the more
appropriate `target_created_by` for the user of the post being queued
instead of `created_by`. The change was not extended to the `DELETE
/review/:id` endpoint leading to error responses for a user attempting
to deleting their own queued post.

This fix extends the `Reviewable` lookup implementation in
`ReviewablesController#destroy` and Guardian implementation to account
for this change.
2023-08-18 15:30:59 +00:00
Penar Musaraj 10c6b2a0c2
WIP: Rename Webauthn to DiscourseWebauthn (#23077) 2023-08-18 08:39:10 -04:00
David Taylor 16c6ab8661
DEV: Allow plugin outlets to be defined using gjs (#23142)
Previously we were discovering plugin outlets by checking first for dedicated template files, and then looking for classes to match them. This doesn't work for components which are entirely defined in JS (e.g. those authored with gjs, or those which are re-exports of a colocated component).

This commit refactors our detection logic to look for both class and template modules in a single pass. It also refactors things so that the modules themselves are required lazily when needd, rather than all being loaded during app boot.
2023-08-18 12:07:10 +01:00
Ted Johansson 79e3d4e2bd
FIX: Don't run post validations when hiding post (#23139)
When hiding a post (essentially updating hidden, hidden_at, and hidden_reason_id) our callbacks are running the whole battery of post validations. This can cause the hiding to fail in a number of edge cases. The issue is similar to the one fixed in #11680, but applies to all post validations, none of which should apply when hiding a post.

After some code reading and discussion, none of the validations in PostValidator seem to be relevant when hiding posts, so instead of just skipping unique check, we skip all post validator checks.
2023-08-18 10:55:17 +08:00
Daniel Waterworth 80a0d88e4c
FIX: Ensure javascript caches are unique per theme/theme_field (#23126)
We were assuming that this was the case before, but not enforcing it.
2023-08-17 13:57:04 -05:00
Krzysztof Kotlarek f8cd1da92a
FIX: increase sidebar URL limit to 1000 (#23120)
Before this change, sidebar URL had a limit of 200 characters. In some cases it is not enough, therefore it was increased to 1000.
2023-08-17 14:46:24 +10:00
Andreas Klöckner 9e568df316
FIX: reference to non-existent `groups#remove_members` in API key scope (#23042) 2023-08-17 07:20:55 +10:00
marstall 77626c088e
UX: support links in tag descriptions (#22994)
* scrub non-a html tags from tag descriptions on create, strips all tags from tag description when displayed in tag hover

* test for tag description links

* UX: basic render-tag test

* UX: fix linting

* UX: fix linting

* fix broken tests

* Update spec/models/tag_spec.rb

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>

* UX: use has_sanitizable_fields instead of has_scrubbable_fields to ensafen tag.description

---------

Co-authored-by: Penar Musaraj <pmusaraj@gmail.com>
2023-08-16 11:43:54 -04:00
Renato Atilio 840bea3c51
FEATURE: add topic voting webhook event type (#23072)
* FEATURE: add topic upvote webhook event type

* DEV: use a generic event type name for other actions in the same plugin
2023-08-11 13:42:28 -03:00
Ted Johansson 8053cb0c21
DEV: Add a HasDeprecatedColumns concern for better deprecation messages (#22930)
Currently when we decide we're going to drop a column in the future we just mark it with a TODO comment and add it to ignored_columns. This makes it instantly unavailable, and we mostly forget about the TODO in the end. 😬

This change adds a HasDeprecatedColumns concern which offers a little bit more flexibility. We can still simulate the old behaviour by setting drop_from to the current version, but we can also set it to a future version, causing it to raise a deprecation warning until then if used.
2023-08-11 15:25:44 +08:00
Jarek Radosz 94649565ce
DEV: Correct `Style/RedundantReturn` rubocop issues (#23052) 2023-08-10 02:03:38 +02:00
Gerhard Schlager b2fee68b3f DEV: Add rake task for generating avatars from SSO 2023-08-09 20:56:14 +02:00
Martin Brennan 09223e5ae7
DEV: Remove enable_experimental_hashtag_autocomplete logic (#22820)
This commit removes any logic in the app and in specs around
enable_experimental_hashtag_autocomplete and deletes some
old category hashtag code that is no longer necessary.

It also adds a `slug_ref` category instance method, which
will generate a reference like `parent:child` for a category,
with an optional depth, which hashtags use. Also refactors
PostRevisor which was using CategoryHashtagDataSource directly
which is a no-no.

Deletes the old hashtag markdown rule as well.
2023-08-08 11:18:55 +10:00
David Taylor e76e0ad592
DEV: In development, refresh client when theme changes are made (#22978)
This brings the theme development experience (via the discourse_theme cli) closer to the experience of making javascript changes in Discourse core/plugins via Ember CLI. Whenever a change is made to a non-css theme field, all clients will be instructed to immediately refresh via message-bus.
2023-08-04 11:02:26 +01:00
Ted Johansson 1f7e5e8e75
DEV: Switch over category settings to new table - Part 2 (#20580)
In #20135 we prevented invalid inputs from being accepted in category setting form fields on the front-end. We didn't do anything on the back-end at that time, because we were still discussing which path we wanted to take. Eventually we decided we want to move this to a new CategorySetting model.

This PR moves the num_auto_bump_daily from custom fields to the new CategorySetting model.

In addition it sets the default value to 0, which exhibits the same behaviour as when the value is NULL.
2023-08-04 10:53:22 +08:00
Renato Atilio 701ae8764e
FIX: keep first post edit history when moving/merging (#22966) 2023-08-03 22:04:35 -03:00
TheJammiestDodger e680437030
FIX: Add 'Ignored' flags to Moderator Activity report (#22041)
* FIX Add 'Ignored' flags to Moderator Activity report

The Moderator Activity query didn’t include the number of deferred flags in the Flags Reviewed totals. As this number is designed to reflect how many flags a moderator has seen, reviewed, and made a judgement on, the Ignored ones should also be included.

* Apply suggestions from code review

---------

Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2023-08-02 12:27:31 +01:00
Daniel Waterworth 50d527b80c
DEV: Remove db_timeout setting (#22912)
It doesn't actually do anything.
2023-08-01 14:17:43 -05:00
Martin Brennan 6286e790b2
DEV: Remove unread_private_messages and deprecation (#22893)
This was added all the way back in 2020 in b79ea986ac,
enough time has passed, we can delete this now.
2023-08-01 14:44:39 +10:00
Kelv 5f0bc4557f
FEATURE: Count only approved flagged posts in user pages (#22799)
FEATURE: Only approved flags for post counters

* Why was this change necessary?
The counters for flagged posts in the user's profile and user index from
the admin view include flags that were rejected, ignored or pending
review. This introduces unnecessary noise. Also the flagged posts
counter in the user's profile includes custom flags which add further
noise to this signal.

* How does it address the problem?

* Modifying User#flags_received_count to return posts with only approved
  standard flags
* Refactoring User#number_of_flagged_posts to alias to
  User#flags_received_count
* Updating the flagged post staff counter hyperlink to navigate to a
  filtered view of that user's approved flagged posts to maintain
  consistency with the counter
* Adding system tests for the profile page to cover the flagged posts
  staff counter
2023-07-31 13:33:10 +08:00
Alan Guo Xiang Tan 2f5e66b6f8
PERF: Optimise `TopicTrackingState.report` query to speed up query (#22871)
In the query generated by `TopicTrackingState.report`, there are two
subqueies being executed. The first subquery fetches all the topics
that are new for a given user while the second subquery fetches all the topics with
unread posts for a given user. For the second subquery, there is a
filter `topics.updated_at >= user_stats.first_unread_at` which is used
as a performance optimisation to reduce the number of rows that PG has
to scan through the `topics` table.

However, we started to notice in production that the PG planner doesn't
always execute the filter first to reduce the number of rows that it has
to scan through. Running the following query in one of our production
instance,

```
EXPLAIN ANALYZE
SELECT
           DISTINCT topics.id as topic_id,
           u.id as user_id,
           topics.created_at,
           topics.updated_at,
           topics.highest_staff_post_number AS highest_post_number,
           last_read_post_number,
           c.id as category_id,
           c.topic_id AS category_topic_id,
           tu.notification_level,
           us.first_unread_at,
           GREATEST(
              CASE
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                u.previous_visit_at,u.created_at
              )
              ELSE ('2023-07-31 03:29:45.737630'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
              END, u.created_at, '2023-07-25 15:06:44'
           ) AS treat_as_new_topic_start_date
FROM topics
JOIN users u on u.id = 13455
JOIN user_stats AS us ON us.user_id = u.id
JOIN user_options AS uo ON uo.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id

WHERE u.id = 13455 AND
       topics.updated_at >= us.first_unread_at AND
      topics.archetype <> 'private_message' AND
      (("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_staff_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND

      NOT (
  COALESCE((select array_agg(tag_id) from topic_tags where topic_tags.topic_id = topics.id), ARRAY[]::int[]) && ARRAY[451,452,453]
) AND

      topics.deleted_at IS NULL AND

      NOT (
        last_read_post_number IS NULL AND
        (
          topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 13455 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 13455 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
          AND tu.notification_level <= 1
        )
      )
```

we get the following

```
                                                                                                                                                                                                                                                                                                                          QUERY PLAN
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=201606.06..201608.15 rows=76 width=60) (actual time=91.279..91.294 rows=14 loops=1)
   ->  Sort  (cost=201606.06..201606.25 rows=76 width=60) (actual time=91.278..91.284 rows=14 loops=1)
         Sort Key: topics.id, topics.created_at, topics.updated_at, topics.highest_staff_post_number, tu.last_read_post_number, c.id, c.topic_id, tu.notification_level, us.first_unread_at, (GREATEST(CASE WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-1'::integer) THEN u.created_at WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-2'::integer) THEN COALESCE(u.previous_visit_at, u.created_at) ELSE ('2023-07-31 03:29:45.73763'::timestamp without time zone - ('00:01:00'::interval * (COALESCE(uo.new_topic_duration_minutes, 2880))::double precision)) END, u.created_at, '2023-07-25 15:06:44'::timestamp without time zone))
         Sort Method: quicksort  Memory: 26kB
         ->  Hash Join  (cost=97519.51..201603.69 rows=76 width=60) (actual time=87.662..91.268 rows=14 loops=1)
               Hash Cond: (topics.id = tu.topic_id)
               Join Filter: ((tu.last_read_post_number < topics.highest_staff_post_number) AND ((tu.last_read_post_number IS NOT NULL) OR (NOT (hashed SubPlan 2)) OR (tu.notification_level > 1)))
               Rows Removed by Join Filter: 10
               ->  Nested Loop  (cost=1.54..104075.36 rows=3511 width=68) (actual time=0.055..3.609 rows=548 loops=1)
                     ->  Nested Loop  (cost=1.13..25.20 rows=1 width=32) (actual time=0.027..0.033 rows=1 loops=1)
                           ->  Nested Loop  (cost=0.71..16.76 rows=1 width=28) (actual time=0.020..0.023 rows=1 loops=1)
                                 ->  Index Scan using users_pkey on users u  (cost=0.42..8.44 rows=1 width=20) (actual time=0.010..0.012 rows=1 loops=1)
                                       Index Cond: (id = 13455)
                                 ->  Index Scan using user_stats_pkey on user_stats us  (cost=0.29..8.31 rows=1 width=12) (actual time=0.008..0.010 rows=1 loops=1)
                                       Index Cond: (user_id = 13455)
                           ->  Index Scan using index_user_options_on_user_id_and_default_calendar on user_options uo  (cost=0.42..8.44 rows=1 width=8) (actual time=0.007..0.008 rows=1 loops=1)
                                 Index Cond: (user_id = 13455)
                     ->  Nested Loop  (cost=0.41..104015.12 rows=3504 width=36) (actual time=0.026..3.503 rows=548 loops=1)
                           ->  Seq Scan on categories c  (cost=0.00..13.73 rows=73 width=8) (actual time=0.003..0.039 rows=73 loops=1)
                           ->  Index Only Scan using index_topics_on_updated_at_public on topics  (cost=0.41..1424.20 rows=48 width=28) (actual time=0.012..0.046 rows=8 loops=73)
                                 Index Cond: ((updated_at >= us.first_unread_at) AND (category_id = c.id))
                                 Filter: (NOT (COALESCE((SubPlan 1), '{}'::integer[]) && '{451,452,453}'::integer[]))
                                 Heap Fetches: 553
                                 SubPlan 1
                                   ->  Aggregate  (cost=4.31..4.32 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=548)
                                         ->  Index Only Scan using index_topic_tags_on_topic_id_and_tag_id on topic_tags  (cost=0.29..4.31 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=548)
                                               Index Cond: (topic_id = topics.id)
                                               Heap Fetches: 178
               ->  Hash  (cost=97222.14..97222.14 rows=19914 width=16) (actual time=87.545..87.546 rows=42884 loops=1)
                     Buckets: 65536 (originally 32768)  Batches: 1 (originally 1)  Memory Usage: 2387kB
                     ->  Bitmap Heap Scan on topic_users tu  (cost=1217.47..97222.14 rows=19914 width=16) (actual time=14.419..78.286 rows=42884 loops=1)
                           Recheck Cond: (user_id = 13455)
                           Filter: (COALESCE(notification_level, 1) >= 2)
                           Rows Removed by Filter: 15839
                           Heap Blocks: exact=45285
                           ->  Bitmap Index Scan on index_topic_users_on_user_id_and_topic_id  (cost=0.00..1212.49 rows=59741 width=0) (actual time=6.448..6.448 rows=58723 loops=1)
                                 Index Cond: (user_id = 13455)
               SubPlan 2
                 ->  Nested Loop Left Join  (cost=0.74..46.90 rows=1 width=4) (never executed)
                       Join Filter: (category_users2.category_id = categories2.id)
                       Filter: (((category_users.id IS NULL) AND (COALESCE(category_users2.notification_level, 1) = 0)) OR (COALESCE(category_users.notification_level, 1) = 0))
                       ->  Nested Loop Left Join  (cost=0.45..32.31 rows=73 width=16) (never executed)
                             Join Filter: (category_users.category_id = categories.id)
                             ->  Nested Loop Left Join  (cost=0.15..18.45 rows=73 width=8) (never executed)
                                   ->  Seq Scan on categories  (cost=0.00..13.73 rows=73 width=8) (never executed)
                                   ->  Memoize  (cost=0.15..0.28 rows=1 width=4) (never executed)
                                         Cache Key: categories.parent_category_id
                                         Cache Mode: logical
                                         ->  Index Only Scan using categories_pkey on categories categories2  (cost=0.14..0.27 rows=1 width=4) (never executed)
                                               Index Cond: (id = categories.parent_category_id)
                                               Heap Fetches: 0
                             ->  Materialize  (cost=0.29..11.69 rows=2 width=12) (never executed)
                                   ->  Index Scan using idx_category_users_user_id_category_id on category_users  (cost=0.29..11.68 rows=2 width=12) (never executed)
                                         Index Cond: (user_id = 13455)
                       ->  Materialize  (cost=0.29..11.69 rows=2 width=8) (never executed)
                             ->  Index Scan using idx_category_users_user_id_category_id on category_users category_users2  (cost=0.29..11.68 rows=2 width=8) (never executed)
                                   Index Cond: (user_id = 13455)
 Planning Time: 1.740 ms
 Execution Time: 91.414 ms
(59 rows)
```

From the execution plan, we can see the most of the time is spent
joining about 42888 rows in the `topics` table to the `topic_users` table.
However, we know that we only have to scan through a
subset of the `topics` table because the user's last unread at is '2023-07-20 11:33:05'.
If we filter the `topics` table with `topics.updated_at >= '2023-07-20 11:33:05'`, this would only
return about 1500 rows.

From our testing in production, the PG planner is able to execute a
better query plan when we avoid the unnecessary joins on `user_stats` just to be
able to get the user's `UserStat#first_unread_at`. Instead, we can just
pass the value of `UserStat#first_unread_at` directly as a query
parameter.

```
EXPLAIN ANALYZE
SELECT
           DISTINCT topics.id as topic_id,
           u.id as user_id,
           topics.created_at,
           topics.updated_at,
           topics.highest_staff_post_number AS highest_post_number,
           last_read_post_number,
           c.id as category_id,
           c.topic_id AS category_topic_id,
           tu.notification_level,
           GREATEST(
              CASE
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -1 THEN u.created_at
              WHEN COALESCE(uo.new_topic_duration_minutes, 2880) = -2 THEN COALESCE(
                u.previous_visit_at,u.created_at
              )
              ELSE ('2023-07-31 03:29:45.737630'::timestamp - INTERVAL '1 MINUTE' * COALESCE(uo.new_topic_duration_minutes, 2880))
              END, u.created_at, '2023-07-25 15:06:44'
           ) AS treat_as_new_topic_start_date
FROM topics
JOIN users u on u.id = 13455
JOIN user_options AS uo ON uo.user_id = u.id
JOIN categories c ON c.id = topics.category_id
LEFT JOIN topic_users tu ON tu.topic_id = topics.id AND tu.user_id = u.id

WHERE u.id = 13455 AND
       topics.updated_at >= '2023-07-20 11:33:05' AND
      topics.archetype <> 'private_message' AND
      (("topics"."deleted_at" IS NULL AND (tu.last_read_post_number < topics.highest_staff_post_number) AND (COALESCE(tu.notification_level, 1) >= 2)) OR (1=0)) AND

      NOT (
  COALESCE((select array_agg(tag_id) from topic_tags where topic_tags.topic_id = topics.id), ARRAY[]::int[]) && ARRAY[451,452,453]
) AND

      topics.deleted_at IS NULL AND

      NOT (
        last_read_post_number IS NULL AND
        (
          topics.category_id IN (SELECT "categories"."id" FROM "categories" LEFT JOIN categories categories2 ON categories2.id = categories.parent_category_id LEFT JOIN category_users ON category_users.category_id = categories.id AND category_users.user_id = 13455 LEFT JOIN category_users category_users2 ON category_users2.category_id = categories2.id AND category_users2.user_id = 13455 WHERE ((category_users.id IS NULL AND COALESCE(category_users2.notification_level, 1) = 0) OR COALESCE(category_users.notification_level, 1) = 0))
          AND tu.notification_level <= 1
        )
      );
```

Note how the filter is now `topics.updated_at >= '2023-07-20 11:33:05'`
instead of `topics.updated_at >= us.first_unread_at`. The modified query
above generates the following execution plan.

```
                                                                                                                                                                                                                                                                                                                QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Unique  (cost=5189.86..5189.88 rows=1 width=52) (actual time=4.991..5.002 rows=14 loops=1)
   ->  Sort  (cost=5189.86..5189.86 rows=1 width=52) (actual time=4.990..4.994 rows=14 loops=1)
         Sort Key: topics.id, topics.created_at, topics.updated_at, topics.highest_staff_post_number, tu.last_read_post_number, c.id, c.topic_id, tu.notification_level, (GREATEST(CASE WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-1'::integer) THEN u.created_at WHEN (COALESCE(uo.new_topic_duration_minutes, 2880) = '-2'::integer) THEN COALESCE(u.previous_visit_at, u.created_at) ELSE ('2023-07-31 03:29:45.73763'::timestamp without time zone - ('00:01:00'::interval * (COALESCE(uo.new_topic_duration_minutes, 2880))::double precision)) END, u.created_at, '2023-07-25 15:06:44'::timestamp without time zone))
         Sort Method: quicksort  Memory: 26kB
         ->  Nested Loop  (cost=52.11..5189.85 rows=1 width=52) (actual time=0.093..4.974 rows=14 loops=1)
               ->  Nested Loop  (cost=51.70..5181.39 rows=1 width=60) (actual time=0.084..4.931 rows=14 loops=1)
                     ->  Nested Loop  (cost=51.28..5172.94 rows=1 width=44) (actual time=0.076..4.887 rows=14 loops=1)
                           ->  Nested Loop  (cost=0.41..1698.46 rows=59 width=36) (actual time=0.029..3.537 rows=548 loops=1)
                                 ->  Seq Scan on categories c  (cost=0.00..13.73 rows=73 width=8) (actual time=0.005..0.039 rows=73 loops=1)
                                 ->  Index Only Scan using index_topics_on_updated_at_public on topics  (cost=0.41..23.07 rows=1 width=28) (actual time=0.012..0.047 rows=8 loops=73)
                                       Index Cond: ((updated_at >= '2023-07-20 11:33:05'::timestamp without time zone) AND (category_id = c.id))
                                       Filter: (NOT (COALESCE((SubPlan 1), '{}'::integer[]) && '{451,452,453}'::integer[]))
                                       Heap Fetches: 552
                                       SubPlan 1
                                         ->  Aggregate  (cost=4.31..4.32 rows=1 width=32) (actual time=0.002..0.002 rows=1 loops=548)
                                               ->  Index Only Scan using index_topic_tags_on_topic_id_and_tag_id on topic_tags  (cost=0.29..4.31 rows=1 width=4) (actual time=0.002..0.002 rows=1 loops=548)
                                                     Index Cond: (topic_id = topics.id)
                                                     Heap Fetches: 178
                           ->  Index Scan using index_topic_users_on_user_id_and_topic_id on topic_users tu  (cost=50.86..58.88 rows=1 width=16) (actual time=0.002..0.002 rows=0 loops=548)
                                 Index Cond: ((user_id = 13455) AND (topic_id = topics.id))
                                 Filter: ((COALESCE(notification_level, 1) >= 2) AND (last_read_post_number < topics.highest_staff_post_number) AND ((last_read_post_number IS NOT NULL) OR (NOT (hashed SubPlan 2)) OR (notification_level > 1)))
                                 Rows Removed by Filter: 0
                                 SubPlan 2
                                   ->  Nested Loop Left Join  (cost=0.74..50.43 rows=1 width=4) (never executed)
                                         Join Filter: (category_users2.category_id = categories2.id)
                                         Filter: (((category_users.id IS NULL) AND (COALESCE(category_users2.notification_level, 1) = 0)) OR (COALESCE(category_users.notification_level, 1) = 0))
                                         ->  Nested Loop Left Join  (cost=0.45..35.84 rows=73 width=16) (never executed)
                                               Join Filter: (category_users.category_id = categories.id)
                                               ->  Nested Loop Left Join  (cost=0.15..21.97 rows=73 width=8) (never executed)
                                                     ->  Seq Scan on categories  (cost=0.00..13.73 rows=73 width=8) (never executed)
                                                     ->  Memoize  (cost=0.15..0.61 rows=1 width=4) (never executed)
                                                           Cache Key: categories.parent_category_id
                                                           Cache Mode: logical
                                                           ->  Index Only Scan using categories_pkey on categories categories2  (cost=0.14..0.60 rows=1 width=4) (never executed)
                                                                 Index Cond: (id = categories.parent_category_id)
                                                                 Heap Fetches: 0
                                               ->  Materialize  (cost=0.29..11.69 rows=2 width=12) (never executed)
                                                     ->  Index Scan using idx_category_users_user_id_category_id on category_users  (cost=0.29..11.68 rows=2 width=12) (never executed)
                                                           Index Cond: (user_id = 13455)
                                         ->  Materialize  (cost=0.29..11.69 rows=2 width=8) (never executed)
                                               ->  Index Scan using idx_category_users_user_id_category_id on category_users category_users2  (cost=0.29..11.68 rows=2 width=8) (never executed)
                                                     Index Cond: (user_id = 13455)
                     ->  Index Scan using users_pkey on users u  (cost=0.42..8.44 rows=1 width=20) (actual time=0.003..0.003 rows=1 loops=14)
                           Index Cond: (id = 13455)
               ->  Index Scan using index_user_options_on_user_id_and_default_calendar on user_options uo  (cost=0.42..8.44 rows=1 width=8) (actual time=0.002..0.002 rows=1 loops=14)
                     Index Cond: (user_id = 13455)
 Planning Time: 1.281 ms
 Execution Time: 5.092 ms
(48 rows)
```

With the new query, PG first does an index scan using the `index_topics_on_updated_at_public` index to filter away most of the topics making the subsequent joins much cheaper. Total query time has been reduced from ~90ms to ~5ms.

This optimisation will mostly affect users with very few/recent unread topics since a large `UserStat#firsts_unread_at` value will still mean scanning through a large portion of the `topics` table.
2023-07-31 12:21:41 +08:00
Ted Johansson c4d0bbce62
DEV: Delete upload references upon deleting draft (#22851)
We currently are accumulating orphaned upload references whenever drafts are deleted.

This change deals with future cases by adding a dependent strategy of delete_all on the Draft#upload_references association. (We don't really need destroy strategy here, since UploadReference is a simple data bag and there are no validations or callbacks on the model.)

It deals with existing cases through a migration that deletes all existing, orphaned draft upload references.
2023-07-31 10:16:23 +08:00
Selase Krakani 81cf481b16
FIX: Missing pending queued posts from topic view (#22838)
A previous change updated `ReviewableQueuedPost`'s `created_by`
to be consistent with other reviewable types. It assigns
the the creator of the post being queued to `target_created_by` and sets
the `created_by` to the creator of the reviewable itself.

This fix updates some of the `created_by` references missed during the
intial fix.
2023-07-28 16:16:23 +00:00
Bianca Nenciu 0736611423
SECURITY: Hide restricted tags in noscript view
The hidden tags are usually filtered out by the serializer, but the
noscript view uses the topic objects instead of the serialized objects.
2023-07-28 12:53:50 +01:00
Penar Musaraj dcc825bda5
SECURITY: Limit length of edit reason column 2023-07-28 12:53:49 +01:00
Blake Erickson 62a609ea2d
SECURITY: Handle concurrent invite accepts
Raise an error on concurrent invite accept attempts.
2023-07-28 12:53:48 +01:00
Alan Guo Xiang Tan bfc3132bb2
SECURITY: Impose a upper bound on limit params in various controllers
What is the problem here?

In multiple controllers, we are accepting a `limit` params but do not
impose any upper bound on the values being accepted. Without an upper
bound, we may be allowing arbituary users from generating DB queries
which may end up exhausing the resources on the server.

What is the fix here?

A new `fetch_limit_from_params` helper method is introduced in
`ApplicationController` that can be used by controller actions to safely
get the limit from the params as a default limit and maximum limit has
to be set. When an invalid limit params is encountered, the server will
respond with the 400 response code.
2023-07-28 12:53:46 +01:00
Natalie Tay 173de8afe6
DEV: Add logging and rescue when user already exists and connecting via DiscourseConnect (#22833)
This is happening because despite the user already existing in the forum, the `SingleSignOnRecord` doesn't exist and "require_activation" is set on the provider, causing us to skip looking for the email, and resulting in us creating a new User then seeing Validation failed: Primary email has already been taken when DiscourseConnect is attempting to make a new account.
2023-07-28 02:53:33 +08:00
Joffrey JAFFEUX db0aef1192
DEV: removes unused group_manager model (#22827)
The associated table has been removed in 2015: 6dd4bc7d57 (diff-53b8234c51c429b92eb91d0212e15bbab16fcd5d1cbd3db64509977c9e1a060d)
2023-07-27 17:10:58 +02:00