Commit Graph

8580 Commits

Author SHA1 Message Date
Penar Musaraj e9b1d29d8b
UX: Revamp quick search (#14499)
Co-authored-by: Robin Ward <robin.ward@gmail.com>
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-10-06 11:42:52 -04:00
Osama Sayegh d9d877fee7
DEV: Pass kwargs to the redis gem when calling methods/commands that we don't wrap (#14530)
This commit fixes the `eval` and `evalsha` commands/methods and any other methods that don't have a wrapper in `DiscourseRedis` and expect keyword arguments. I noticed this problem in Logster when I was trying to fetch some log messages in JSON format using the rails console and saw the messages were missing the `env` field. Logster uses the `eval` command to fetch messages `env`s:

dc351fd00f/lib/logster/redis_store.rb (L250-L253)

and that code was not fetching anything because `DiscourseRedis` didn't pass the `keys` keyword arg to the redis gem.
2021-10-06 17:42:04 +03:00
Bianca Nenciu f58ab2283d
FIX: Parse address lists in embedded emails (#14514)
Same fix is applied to emails immediately after being parsed because
long headers are sometimes in an invalid format.
2021-10-06 15:07:29 +03:00
Yasuo Honda dbbfad7ed0 FIX: Support Ruby 3 keyword arguments 2021-10-05 11:25:00 -04:00
Arpit Jalan fbe9cd49b6
FIX: Vimeo private video oneboxes were broken (#14510) 2021-10-05 15:46:58 +05:30
Dan Ungureanu 74a9c0509b
FIX: Use addresses to compare email header (#14509)
Usually, when an email is received a user lookup is performed using the
email address found in the `From` header. When an email has an
`X-Original-From` header, if it is equal to `Reply-To` then it uses that
one instead. The comparison was sensitive to whitespaces and other
insignificant characters such as quotes because it reconstructed the
`From` header.

For the fixture added in this commit, it compared the reconstructed
`From` header `John Doe <johndoe@example.com>` with the `Reply-To`
header `"John Doe"    <johndoe@example.com>`.
2021-10-05 12:42:19 +03:00
Roman Rizzi 90a3fbc07b
DEV: Remove HTML setting type and sanitization logic. (#14440)
* DEV: Remove HTML setting type and sanitization logic.

We concluded that we don't want settings to contain HTML, so I'm removing the setting type and sanitization logic. Additionally, we no longer allow the global-notice text to contain HTML.

I searched for usages of this setting type in the `all-the-plugins` repo and found none, so I haven't added a migration for existing settings.

* Mark Global notices containing links as HTML Safe.
2021-10-04 15:40:35 -03:00
Andrei Prigorshnev 149e869c22
FEATURE: make username suggester suggest `user1`, `user2` etc. for input that contains invalid characters only (#14179)
We were suggesting 111, 1111, 1112 before.

See the discussion on Meta – https://meta.discourse.org/t/curious-account-creation-behaviour/199970/14.
2021-10-04 16:47:55 +04:00
Alan Guo Xiang Tan 34cebfd867
FIX: Exclude PMs that user sent to themselves. (#14496)
Regression from 016efeadf6

Follow-up to 016efeadf6
2021-10-04 11:55:35 +08:00
Roman Rizzi 4c2d5158c5
FIX: Follow the canonical URL when importing a remote topic. (#14489)
FinalDestination now supports the `follow_canonical` option, which will perform an initial GET request, parse the canonical link if present, and perform a HEAD request to it.

We use this mode during embeds to avoid treating URLs with different query parameters as different topics.
2021-10-01 12:48:21 -03:00
Alan Guo Xiang Tan 0359adc0b8
PERF: Avoid running ignored users DB query for anon users. (#14487) 2021-10-01 15:01:27 +08:00
Vinoth Kannan c8d5c049eb
DEV: skip S3 CDN urls with different path in prefix. (#14488)
Previously, while retrieving each upload urls in a post S3 CDN urls with different path in prefix (external urls technically) are considered as uploaded url. It created issue while checking missing uploads.
2021-10-01 12:25:17 +05:30
Dan Ungureanu 76a7b75d8a
DEV: Reuse can_invite_to_forum? in can_invite_to? (#14392)
This commit resolves refactors can_invite_to? to use
can_invite_to_forum? for checking the site-wide permissions and then
perform topic specific checkups.

Similarly, can_invite_to? is always used with a topic object and this is
now enforced.

There was another problem before when `must_approve_users` site setting
was not checked when inviting users to forum, but was checked when
inviting to a topic.

Another minor security issue was that group owners could invite to
group topics even if they did not have the minimum trust level to do
it.
2021-09-29 17:40:16 +03:00
Andrei Prigorshnev b609f6c11c
FIX: restrict other user's notification routes (#14442)
It was possible to see notifications of other users using routes:
- notifications/responses
- notifications/likes-received
- notifications/mentions
- notifications/edits

We weren't showing anything private (like notifications about private messages), only things that're publicly available in other places. But anyway, it feels strange that it's possible to look at notifications of someone else. Additionally, there is a risk that we can unintentionally leak something on these pages in the future.

This commit restricts these routes.
2021-09-29 16:24:28 +04:00
Alan Guo Xiang Tan 9d5da2b383
PERF: Revert all inboxes from messages route. (#14445)
The all inboxes was introduced in
016efeadf6 but we decided to roll it back
for performance reasons. The main performance challenge here is that PG
has to basically loop through all the PMs that a user is allowed to view
before being able to order by `Topic#bumped_at`. The all inboxes was not
planned as part of the new/unread filter so we've decided not to tackle
the performance issue for the upcoming release.

Follow-up to 016efeadf6
2021-09-28 11:58:04 +08:00
Yasuo Honda d06d09f479
Fix: Support Ruby 3 keyword arguments for DiscourseRedis (#14444) 2021-09-28 10:11:18 +08:00
Alan Guo Xiang Tan cd64e88711
PERF: Improve database query perf when loading topics for a category. (#14416)
* PERF: Improve database query perf when loading topics for a category.

Instead of left joining the `topics` table against `categories` by filtering with `categories.id`,
we can improve the query plan by filtering against `topics.category_id`
first before joining which helps to reduce the number of rows in the
topics table that has to be joined against the other tables and also
make better use of our existing index.

The following is a before and after of the query plan for a category
with many subcategories.

Before:

```
                                                                                                       QUERY PLAN

-------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------
 Limit  (cost=1.28..747.09 rows=30 width=12) (actual time=85.502..2453.727 rows=30 loops=1)
   ->  Nested Loop Left Join  (cost=1.28..566518.36 rows=22788 width=12) (actual time=85.501..2453.722 rows=30 loops=1)
         Join Filter: (category_users.category_id = topics.category_id)
         Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
         ->  Nested Loop Left Join  (cost=1.00..566001.58 rows=22866 width=20) (actual time=85.494..2453.702 rows=30 loops=1)
               Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((t
opics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
               Rows Removed by Filter: 1
               ->  Nested Loop  (cost=0.57..528561.75 rows=68606 width=24) (actual time=85.472..2453.562 rows=31 loops=1)
                     Join Filter: ((topics.category_id = categories.id) AND ((categories.topic_id <> topics.id) OR (categories.id = 1
1)))
                     Rows Removed by Join Filter: 13938306
                     ->  Index Scan using index_topics_on_bumped_at on topics  (cost=0.42..100480.05 rows=715549 width=24) (actual ti
me=0.010..633.015 rows=464623 loops=1)
                           Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text))
                           Rows Removed by Filter: 105321
                     ->  Materialize  (cost=0.14..36.04 rows=30 width=8) (actual time=0.000..0.002 rows=30 loops=464623)
                           ->  Index Scan using categories_pkey on categories  (cost=0.14..35.89 rows=30 width=8) (actual time=0.006.
.0.040 rows=30 loops=1)
                                 Index Cond: (id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,1
13,104,98,100,96,108,109,110,111}'::integer[]))
               ->  Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu  (cost=0.43..0.53 rows=1 width=16) (a
ctual time=0.004..0.004 rows=0 loops=31)
                     Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
         ->  Materialize  (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=30)
               ->  Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users  (cost=0.28..2.29 rows=1 width
=8) (actual time=0.004..0.004 rows=0 loops=1)
                     Index Cond: (user_id = 1103877)
 Planning Time: 1.359 ms
 Execution Time: 2453.765 ms
(23 rows)
```

After:

```
                                                                                                                            QUERY PLAN
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Limit  (cost=1.28..438.55 rows=30 width=12) (actual time=38.297..657.215 rows=30 loops=1)
   ->  Nested Loop Left Join  (cost=1.28..195944.68 rows=13443 width=12) (actual time=38.296..657.211 rows=30 loops=1)
         Filter: ((categories.topic_id <> topics.id) OR (topics.category_id = 11))
         Rows Removed by Filter: 29
         ->  Nested Loop Left Join  (cost=1.13..193462.59 rows=13443 width=16) (actual time=38.289..657.092 rows=59 loops=1)
               Join Filter: (category_users.category_id = topics.category_id)
               Filter: ((topics.category_id = 11) OR (COALESCE(category_users.notification_level, 1) <> 0) OR (tu.notification_level > 1))
               ->  Nested Loop Left Join  (cost=0.85..193156.79 rows=13489 width=20) (actual time=38.282..657.059 rows=59 loops=1)
                     Filter: ((COALESCE(tu.notification_level, 1) > 0) AND ((topics.category_id <> 11) OR (topics.pinned_at IS NULL) OR ((topics.pinned_at <= tu.cleared_pinned_at) AND (tu.cleared_pinned_at IS NOT NULL))))
                     Rows Removed by Filter: 1
                     ->  Index Scan using index_topics_on_bumped_at on topics  (cost=0.42..134521.06 rows=40470 width=24) (actual time=38.267..656.850 rows=60 loops=1)
                           Filter: ((deleted_at IS NULL) AND ((archetype)::text <> 'private_message'::text) AND (category_id = ANY ('{11,53,57,55,54,56,112,94,107,115,116,117,97,95,102,103,101,105,99,114,106,113,104,98,100,96,108,109,110,111}'::integer[])))
                           Rows Removed by Filter: 569895
                     ->  Index Scan using index_topic_users_on_topic_id_and_user_id on topic_users tu  (cost=0.43..1.43 rows=1 width=16) (actual time=0.003..0.003 rows=0 loops=60)
                           Index Cond: ((topic_id = topics.id) AND (user_id = 1103877))
               ->  Materialize  (cost=0.28..2.30 rows=1 width=8) (actual time=0.000..0.000 rows=0 loops=59)
                     ->  Index Scan using index_category_users_on_user_id_and_last_seen_at on category_users  (cost=0.28..2.29 rows=1 width=8) (actual time=0.004..0.004 rows=0 loops=1)
                           Index Cond: (user_id = 1103877)
         ->  Index Scan using categories_pkey on categories  (cost=0.14..0.17 rows=1 width=8) (actual time=0.001..0.001 rows=1 loops=59)
               Index Cond: (id = topics.category_id)
 Planning Time: 1.633 ms
 Execution Time: 657.255 ms
(22 rows)
```

* PERF: Optimize index on topics bumped_at.

Replace `index_topics_on_bumped_at` index with a partial index on `Topic#bumped_at` filtered by archetype since there is already another index that covers private topics.
2021-09-28 10:05:00 +08:00
tshenry ba17d9106e
DEV: Make discourse-reactions official and sort existing plugin list (#14452) 2021-09-28 09:44:10 +08:00
Bianca Nenciu 1f57b29147
SECURITY: Escape watched word in error message (#14434) 2021-09-24 11:55:15 +03:00
Martin Brennan dba6a5eabf
FEATURE: Humanize file size error messages (#14398)
The file size error messages for max_image_size_kb and
max_attachment_size_kb are shown to the user in the KB
format, regardless of how large the limit is. Since we
are going to support uploading much larger files soon,
this KB-based limit soon becomes unfriendly to the end
user.

For example, if the max attachment size is set to 512000
KB, this is what the user sees:

> Sorry, the file you are trying to upload is too big (maximum
size is 512000KB)

This makes the user do math. In almost all file explorers that
a regular user would be familiar width, the file size is shown
in a format based on the maximum increment (e.g. KB, MB, GB).

This commit changes the behaviour to output a humanized file size
instead of the raw KB. For the above example, it would now say:

> Sorry, the file you are trying to upload is too big (maximum
size is 512 MB)

This humanization also handles decimals, e.g. 1536KB = 1.5 MB
2021-09-22 07:59:45 +10:00
David Taylor 89994cff40 DEV: Allow Ember CLI for `rake qunit:test` and `rake plugin:qunit`
To use Ember CLI, set QUNIT_EMBER_CLI=1
2021-09-21 18:10:04 +01:00
Martin Brennan 27699648ef
FEATURE: Go to last unread for topic-level bookmark links (#14396)
Instead of going to the OP of the topic for topic-level bookmarks
(which are bookmarks where for_topic is true) when clicking on the
bookmark in the quick access menu or on the user bookmark list,
this commit takes the user to the last unread post in
the topic instead. This should be generally more useful than landing
on the unchanging OP.

To make this work nicely, I needed to add the last_read_post_number to
the BookmarkQuery based on the TopicUser association. It should not add
too much extra weight to the query, because it is limited to the user
that we are fetching bookmarks for.

Also fixed an issue where the bookmark serializer highest_post_number was
not taking into account whether the user was staff, which is when we
should use highest_staff_post_number instead.
2021-09-21 13:49:56 +10:00
Martin Brennan 0c42a1e5f3
FEATURE: Topic-level bookmarks (#14353)
Allows creating a bookmark with the `for_topic` flag introduced in d1d2298a4c set to true. This happens when clicking on the Bookmark button in the topic footer when no other posts are bookmarked. In a later PR, when clicking on these topic-level bookmarks the user will be taken to the last unread post in the topic, not the OP. Only the OP can have a topic level bookmark, and users can also make a post-level bookmark on the OP of the topic.

I had to do some pretty heavy refactors because most of the bookmark code in the JS topics controller was centred around instances of Post JS models, but the topic level bookmark is not centred around a post. Some refactors were just for readability as well.

Also removes some missed reminderType code from the purge in 41e19adb0d
2021-09-21 08:45:47 +10:00
Martin Brennan 4fb7d045a0
FIX: Handle forwarded email quotes around Reply-To display name (#14384)
The display name can have quotes around it, which does not work
with our current comparison of a from field (in this case Reply-To)
and another header (X-Original-From), because we are not comparing
the two values in the same way. This causes an issue where the
commit here: b88d8c8 will not
work properly; the forwarded email gets the From address instead
of the Reply-To address as intended.
2021-09-20 16:26:18 +10:00
Gerhard Schlager 962ccf0ab5
FIX: Hoisting linebreaks shouldn't fail for HTML5 elements (#14364) 2021-09-17 10:41:34 +02:00
Vinoth Kannan 2c7cc40be3
DEV: only use the records that are auto populated by the task. (#14360)
Previously, it was using existing user and topic records to generate random posts.
2021-09-17 09:47:32 +05:30
Penar Musaraj ab9c63e4ee
UX: Optionally show a "Summarize" button in topic timeline (#13533) 2021-09-16 15:15:00 -04:00
Vinoth Kannan a6de4a5ce9
DEV: use upload id to save in theme setting instead of URL. (#14341)
When we use URL instead it creates the problem while changing the CDN hostname.
2021-09-16 07:58:53 +05:30
Martin Brennan 41e19adb0d
DEV: Ignore reminder_type for bookmarks (#14349)
We don't actually use the reminder_type for bookmarks anywhere;
we are just storing it. It has no bearing on the UI. It used
to be relevant with the at_desktop bookmark reminders (see
fa572d3a7a)

This commit marks the column as readonly, ignores it, and removes
the index, and it will be dropped in a later PR. Some plugins
are relying on reminder_type partially so some stubs have been
left in place to avoid errors.
2021-09-16 09:56:54 +10:00
Yasuo Honda eed3773b97 FIX: Address ArgumentError to support Ruby 3 keyword arguments 2021-09-15 12:56:54 -04:00
Alan Guo Xiang Tan 27bad28c53
Partially revert "PERF: Improve query performance all inbox private messages. (#14304)" (#14344)
This partially reverts commit ddb458343d.

Seeing performance degrade on larger sites so back to drawing board on
this one. Instead of the DISTINCT LEFT JOIN, we switch back to
IN(subquery).
2021-09-15 11:32:10 +08:00
Alan Guo Xiang Tan ddb458343d
PERF: Improve query performance all inbox private messages. (#14304)
First reported in https://meta.discourse.org/t/-/202482/19

There are two optimizations being applied here:

1. Fetch a user's group ids in a seperate query instead of including it
   as a sub-query. When I tried a subquery, the query plan becomes very
inefficient.

1. Join against the `topic_allowed_users` and `topic_allowed_groups`
   table instead of doing an IN against a subquery where we UNION the
`topic_id`s from the two tables. From my profiling, this enables PG to
do a backwards index scan on the `index_topics_on_timestamps_private`
index.

This commit fixes a bug where listing all messages was incorrectly
excluding topics if a topic has been archived by a group even if the
user did not belong to the group.

This commit also fixes another bug where dismissing private messages
selectively was subjected to the default limit of 30.
2021-09-15 10:29:42 +08:00
Martin Brennan 22208836c5
DEV: Ignore bookmarks.topic_id column and remove references to it in code (#14289)
We don't need no stinkin' denormalization! This commit ignores
the topic_id column on bookmarks, to be deleted at a later date.
We don't really need this column and it's better to rely on the
post.topic_id as the canonical topic_id for bookmarks, then we
don't need to remember to update both columns if the bookmarked
post moves to another topic.
2021-09-15 10:16:54 +10:00
Martin Brennan 0d809197aa
FIX: Make sure S3 object headers are preserved on copy (#14302)
When copying an existing upload stub temporary object
on S3 to its final destination we were not copying across
its additional headers such as content-disposition and
cache-control, which led to issues like attachments not
downloading with their original filename when clicking
the download links in posts.

This is because the metadata_directive = REPLACE option
was not being passed to object.copy_from(), so only the
source object's headers were being used. Added an option
for apply_metadata_to_destination to apply this option
conditionally, because we may not always want to replace
this metadata, but we definitely do when copying a temporary
upload.
2021-09-10 12:59:51 +10:00
Alan Guo Xiang Tan bc23dcd30b
FIX: Don't publish PM archive events to acting user. (#14291)
When a user archives a personal message, they are redirected back to the
inbox and will refresh the list of the topics for the given filter.
Publishing an event to the user results in an incorrect incoming message
because the list of topics has already been refreshed.

This does mean that if a user has two tabs opened, the non-active tab
will not receive the incoming message but at this point we do not think
the technical trade-offs are worth it to support this feature. We
basically have to somehow exclude a client from an incoming message
which is not easy to do.

Follow-up to fc1fd1b416
2021-09-10 09:20:50 +08:00
Daniel Waterworth 05c356f7c6
DEV: Add rake task to check that the DB can be accessed (#14300) 2021-09-09 14:42:10 -05:00
Daniel Waterworth 6262396d8a
FIX: Don't attempt to migrate multisite test db while holding the mutex (#14298)
Since you'll have to wait for the mutex to timeout before it can
continue.
2021-09-09 13:31:52 -05:00
Jarek Radosz 02a6b991fe
FIX: Correct the play icon position (#14295) 2021-09-09 15:10:32 +02:00
Arpit Jalan 11a07b37e1
FIX: ignore canonical link for medium.com oneboxes (#14278)
https://meta.discourse.org/t/bug-in-onebox-link-being-rendered-as-a-gist-when-it-isnt/202463
2021-09-08 20:19:57 +05:30
Joffrey JAFFEUX 95b15acb1e
DEV: uses forked Mousetrap to avoid leaking listeners (#14198) 2021-09-08 14:48:13 +02:00
Krzysztof Kotlarek e3793e6d7c
FIX: better filter for groups search (#14262)
Follow up of https://github.com/discourse/discourse/pull/14216

Allow plugins to register custom filter with block
2021-09-08 09:38:45 +10:00
Alan Guo Xiang Tan fc1fd1b416
FEATURE: Display new/unread count in browse more messages for PMs. (#14188)
In order to include the new/unread count in the browse more message
under suggested topics, a couple of technical changes have to be made.

1. `PrivateMessageTopicTrackingState` is now auto-injected which is
   similar to how it is done for `TopicTrackingState`. This is done so
we don't have to attempt to pass the `PrivateMessageTopicTrackingState`
object multiple levels down into the suggested-topics component. While
the object is auto-injected, we only fetch the initial state and start
tracking when the relevant private messages routes has been hit and only
when a private message's suggested topics is loaded. This is
done as we do not want to add the extra overhead of fetching the inital
state to all page loads but instead wait till the private messages
routes are hit.

2. Previously, we would stop tracking once the `user-private-messages`
   route has been deactivated. However, that is not ideal since
navigating out of the route and back means we send an API call to the
server each time. Since `PrivateMessageTopicTrackingState` is kept in
sync cheaply via messageBus, we can just continue to track the state
even if the user has navigated away from the relevant stages.
2021-09-07 12:30:40 +08:00
Martin Brennan 9f36d8ad43
FIX: Capture CC addresses for forwarded emails (#14254)
When forwarding emails into the group inbox, we now use the
original sender email as the from_address since
2ac9fd9dff. However, we have not
been saving the original CC addresses of the forwarded email,
which are needed to include those recipients in on the conversation
when replying via the group inbox.

This commit captures the CC addresses on the incoming email, and
makes sure the emails are created as staged users and added to the
list of topic allowed users so they are included on CC's sent by
the GroupSmtpEmail and other jobs.
2021-09-07 08:46:28 +10:00
Jean 34ff7bfeeb
FEATURE: Hide suspended users from site-wide search to regular users (#14245) 2021-09-06 09:59:35 -04:00
Martin Brennan 7b392cee50
FIX: Correct the forwarded by user small post for group inbox (#14252)
When 2ac9fd9dff was done, this
affected the small post that is created when forwarding an email
into the group inbox. Instead of using the name and the email of
the user who forwarded the email, it used the original from email
and name to create the small post. So instead of something like
"Discourse Team forwarded the above email" we ended up with
"John Smith forwarded the above email" which is incorrect.

This fixes the issue by creating a staged user for the forwarding
email address (if such a user does not yet exist) and uses that
for the "forwarded" small post instead.
2021-09-06 15:02:13 +10:00
Martin Brennan dd4b8c2afa
FIX: Use random file name for temporary uploads (#14250)
Other locale characters in file names (e.g. é, ä) as well
as special characters can cause issues on S3, notably the S3
copy object operation does not support these special characters.
Instead of storing the original file name in the key, which is
unnecessary, we now generate a random file name with the original
extension for the temporary file and use that for all external
upload stub operations.
2021-09-06 10:21:20 +10:00
Krzysztof Kotlarek f859fd6bde
FEATURE: allow plugins to extend Groups (#14216)
* add_permitted_group_param API for plugins
* add groups-interaction-custom-options outlet
* custom search can use custom group scope
2021-09-06 10:18:51 +10:00
Daniel Waterworth d7873dd823
FIX: Don't attempt to migrate concurrently with other migrations (#14231) 2021-09-03 10:22:25 -05:00
Arpit Jalan 763f48abc7
FIX: increase chunk size to fetch title tag correctly (#14144) 2021-09-03 13:15:58 +05:30
Penar Musaraj 90a23c6fc8
FEATURE: Enable auto dark mode on new instances (#14208) 2021-09-02 14:55:38 -04:00
Neil Lalonde f0d2b0f2f0
Version bump to v2.8.0.beta6 (#14228)
v2.8.0.beta5 can cause segfaults due to the oj gem v3.13.3.
2021-09-02 14:25:33 -04:00
Joshua Rosenfeld 823354d177
DEV: Update Plugin Release Notes heading format (#14218) 2021-09-02 11:08:31 +08:00
Neil Lalonde c6f1818b85
Version bump to v2.8.0.beta5 (#14210) 2021-09-01 13:29:36 -04:00
David Taylor ed688bec8c
FIX: Ensure id sequences are not reset during db:migrate (#14184)
The seed-fu gem resets the sequence on all the tables it touches. In some situations, this can cause primary keys to be re-used. This commit introduces a freedom patch which ensures seed-fu only touches the sequence when it is less than the id of one of the seeded records.
2021-08-30 12:31:22 +01:00
David Taylor b47f288b13
FIX: Update PresenceChannel#present to work for redis 6.0 (#14180)
The `LT` parameter was only added in redis 6.2. We can make do with `NX` instead
2021-08-27 19:44:28 +01:00
David Taylor 01e2836071
FIX: Allow PresenceChannel to work on Redis 6.0 (#14178)
The `ZRANGE ... BYSCORE` syntax was only added in 6.2. We can use the older `ZRANGEBYSCORE` instead
2021-08-27 18:55:54 +01:00
David Taylor 31db83527b DEV: Introduce PresenceChannel API for core and plugin use
PresenceChannel aims to be a generic system for allow the server, and end-users, to track the number and identity of users performing a specific task on the site. For example, it might be used to track who is currently 'replying' to a specific topic, editing a specific wiki post, etc.

A few key pieces of information about the system:
- PresenceChannels are identified by a name of the format `/prefix/blah`, where `prefix` has been configured by some core/plugin implementation, and `blah` can be any string the implementation wants to use.
- Presence is a boolean thing - each user is either present, or not present. If a user has multiple clients 'present' in a channel, they will be deduplicated so that the user is only counted once
- Developers can configure the existence and configuration of channels 'just in time' using a callback. The result of this is cached for 2 minutes.
- Configuration of a channel can specify permissions in a similar way to MessageBus (public boolean, a list of allowed_user_ids, and a list of allowed_group_ids). A channel can also be placed in 'count_only' mode, where the identity of present users is not revealed to end-users.
- The backend implementation uses redis lua scripts, and is designed to scale well. In the future, hard limits may be introduced on the maximum number of users that can be present in a channel.
- Clients can enter/leave at will. If a client has not marked itself 'present' in the last 60 seconds, they will automatically 'leave' the channel. The JS implementation takes care of this regular check-in.
- On the client-side, PresenceChannel instances can be fetched from the `presence` ember service. Each PresenceChannel can be used entered/left/subscribed/unsubscribed, and the service will automatically deduplicate information before interacting with the server.
- When a client joins a PresenceChannel, the JS implementation will automatically make a GET request for the current channel state. To avoid this, the channel state can be serialized into one of your existing endpoints, and then passed to the `subscribe` method on the channel.
- The PresenceChannel JS object is an ember object. The `users` and `count` property can be used directly in ember templates, and in computed properties.
- It is important to make sure that you `unsubscribe()` and `leave()` any PresenceChannel objects after use

An example implementation may look something like this. On the server:

```ruby
register_presence_channel_prefix("site") do |channel|
  next nil unless channel == "/site/online"
  PresenceChannel::Config.new(public: true)
end
```

And on the client, a component could be implemented like this:

```javascript
import Component from "@ember/component";
import { inject as service } from "@ember/service";

export default Component.extend({
  presence: service(),
  init() {
    this._super(...arguments);
    this.set("presenceChannel", this.presence.getChannel("/site/online"));
  },
  didInsertElement() {
    this.presenceChannel.enter();
    this.presenceChannel.subscribe();
  },
  willDestroyElement() {
    this.presenceChannel.leave();
    this.presenceChannel.unsubscribe();
  },
});
```

With this template:

```handlebars
Online: {{presenceChannel.count}}
<ul>
  {{#each presenceChannel.users as |user|}} 
    <li>{{avatar user imageSize="tiny"}} {{user.username}}</li>
  {{/each}}
</ul>
```
2021-08-27 16:26:06 +01:00
Martin Brennan 99ec8eb6df
FIX: Capture S3 metadata when calling create_multipart (#14161)
The generate_presigned_put endpoint for direct external uploads
(such as the one for the uppy-image-uploader) records allowed
S3 metadata values on the uploaded object. We use this to store
the sha1-checksum generated by the UppyChecksum plugin, for later
comparison in ExternalUploadManager.

However, we were not doing this for the create_multipart endpoint,
so the checksum was never captured and compared correctly.

Also includes a fix to make sure UppyChecksum is the last preprocessor to run.
It is important that the UppyChecksum preprocessor is the last one to
be added; the preprocessors are run in order and since other preprocessors
may modify the file (e.g. the UppyMediaOptimization one), we need to
checksum once we are sure the file data has "settled".
2021-08-27 09:50:23 +10:00
David Taylor 0aa8798046
DEV: Remove `yarn install` during `assets:precompile` (#14166)
Instead, we'll do it alongside `bundle install` in `web.template.yml`: https://github.com/discourse/discourse_docker/pull/565
2021-08-26 19:14:50 +01:00
Bianca Nenciu f0005025c4
FIX: Correctly display GitHub code oneboxes (#14149)
Code fragments without the code lines were not displayed correctly.

Follow-up-to 721a946d77.
2021-08-26 12:55:46 +03:00
Martin Brennan e43a8af3bd
FIX: Do not send emails to mailing_list_mode subscribers for PMs (#14159)
This bug was introduced by f66007ec83.

In PostJobsEnqueuer we previously did not fire the after_post_create
event and after_topic_create event for private message topics. This was
changed in the above commit in order to publish message bus messages
for topic tracking state updates. Unfortunately this caused the
NotifyMailingListSubscribers job to be enqueued for all posts including
private messages, and admins and the users involved in the PMs got
emailed the contents of the PMs if they had mailing list mode enabled.

Luckily the impact of this was mitigated by a Guardian#can_see? check
for each mailing list mode user in the NotifyMailingListSubscribers job.
We never want to notify mailing list mode subscribers for private messages
so an early return has been added there, plus the logic in PostJobsEnqueuer
has been fixed, and tests have been added to that class where there were
none before.
2021-08-26 15:16:35 +10:00
Martin Brennan 841e054907
FIX: Do not prefix temp/ S3 keys with s3_bucket_folder_path in S3Helper (#14145)
This is unnecessary, as when the temporary key is created
in S3Store we already include the s3_bucket_folder_path, and
the key will always start with temp/ to assist with lifecycle
rules for multipart uploads.

This was affecting Discourse.store.object_from_path,
Discourse.store.signed_url_for_path, and possibly others.

See also: e0102a5
2021-08-26 08:50:49 +10:00
Penar Musaraj 85b8fea262
UX: Add Styling step to wizard (#14132)
Refactors three wizard steps (colors, fonts, homepage style) into one new step called Styling.
2021-08-25 17:10:12 -04:00
Bianca Nenciu 721a946d77
FIX: Remove spacing from GitHub oneboxes (#14147)
Each code line was surrounded by a lot of white space. The new styles
are similar to what the web interface is using.
2021-08-25 16:34:01 +03:00
Bianca Nenciu 197532dc31
FIX: Add plugin event to topic list user lookup (#14116)
This can be used to change the list of topic posters. For example,
discourse-solved can use this to move the user who posted the solution
after the original poster.
2021-08-25 13:16:08 +03:00
Arpit Jalan 419d71abcb
FEATURE: allow admin to delete all posts by a user irrespectively (#14128)
This commit allows admin to delete all posts by a user irrespective of
site settings `delete_user_max_post_age` and `delete_all_posts_max`.
2021-08-25 10:14:22 +05:30
Alan Guo Xiang Tan f66007ec83
FEATURE: Display unread and new counts for messages. (#14059)
There are certain design decisions that were made in this commit.

Private messages implements its own version of topic tracking state because there are significant differences between regular and private_message topics. Regular topics have to track categories and tags while private messages do not. It is much easier to design the new topic tracking state if we maintain two different classes, instead of trying to mash this two worlds together.

One MessageBus channel per user and one MessageBus channel per group. This allows each user and each group to have their own channel backlog instead of having one global channel which requires the client to filter away unrelated messages.
2021-08-25 11:17:56 +08:00
Penar Musaraj 1167b16913
FIX: Order outputted theme stylesheets (#14133) 2021-08-25 09:37:07 +08:00
Martin Brennan e0102a533a
FIX: Restructure temp/ folders for direct S3 uploads (#14137)
Previously we had temp/ in the middle of the S3 key path like so

* /uploads/default/temp/randomstring/test.png (normal site)
* /sitename/uploads/default/temp/randomstring/test.png (s3 folder path site)
* /standard10/uploads/sitename/temp/randomstring/test.png (multisite site)

However this necessitates making a lifecycle rule to clean up incomplete
S3 multipart uploads for every site, something which we cannot do. It makes
much more sense to have a structure with /temp at the start of the key,
which is what this commit does:

* /temp/uploads/default/randomstring/test.png (normal site)
* /temp/sitename/uploads/default/randomstring/test.png (s3 folder path site)
* /temp/standard10/uploads/sitename/randomstring/test.png (multisite site)
2021-08-25 09:22:36 +10:00
Martin Brennan d295a16dab
FEATURE: Uppy direct S3 multipart uploads in composer (#14051)
This pull request introduces the endpoints required, and the JavaScript functionality in the `ComposerUppyUpload` mixin, for direct S3 multipart uploads. There are four new endpoints in the uploads controller:

* `create-multipart.json` - Creates the multipart upload in S3 along with an `ExternalUploadStub` record, storing information about the file in the same way as `generate-presigned-put.json` does for regular direct S3 uploads
* `batch-presign-multipart-parts.json` - Takes a list of part numbers and the unique identifier for an `ExternalUploadStub` record, and generates the presigned URLs for those parts if the multipart upload still exists and if the user has permission to access that upload
* `complete-multipart.json` - Completes the multipart upload in S3. Needs the full list of part numbers and their associated ETags which are returned when the part is uploaded to the presigned URL above. Only works if the user has permission to access the associated `ExternalUploadStub` record and the multipart upload still exists.

  After we confirm the upload is complete in S3, we go through the regular `UploadCreator` flow, the same as `complete-external-upload.json`, and promote the temporary upload S3 into a full `Upload` record, moving it to its final destination.
* `abort-multipart.json` - Aborts the multipart upload on S3 and destroys the `ExternalUploadStub` record if the user has permission to access that upload.

Also added are a few new columns to `ExternalUploadStub`:

* multipart - Whether or not this is a multipart upload
* external_upload_identifier - The "upload ID" for an S3 multipart upload
* filesize - The size of the file when the `create-multipart.json` or `generate-presigned-put.json` is called. This is used for validation.

When the user completes a direct S3 upload, either regular or multipart, we take the `filesize` that was captured when the `ExternalUploadStub` was first created and compare it with the final `Content-Length` size of the file where it is stored in S3. Then, if the two do not match, we throw an error, delete the file on S3, and ban the user from uploading files for N (default 5) minutes. This would only happen if the user uploads a different file than what they first specified, or in the case of multipart uploads uploaded larger chunks than needed. This is done to prevent abuse of S3 storage by bad actors.

Also included in this PR is an update to vendor/uppy.js. This has been built locally from the latest uppy source at d613b849a6. This must be done so that I can get my multipart upload changes into Discourse. When the Uppy team cuts a proper release, we can bump the package.json versions instead.
2021-08-25 08:46:54 +10:00
Bianca Nenciu ff367e22fb
FEATURE: Make allow_uploaded_avatars accept TL (#14091)
This gives admins more control over who can upload custom profile
pictures.
2021-08-24 10:46:28 +03:00
Martin Brennan 2ac9fd9dff
FIX: Use original from address when forwarding to group inbox (#14114)
When emails were forwarded to a group inbox by the email address
of the group, for example when an email ends up in spam and must
be manually forwarded to the group+site@discoursemail.com address,
the OP of the topic ended up being the group's email address instead
of the sender who originally sent the email to the group inbox.

This commit detects that an email has been forwarded using existing
tools, and if the from address matches one of the group incoming
email addresses, then we look at the forwarded email's from address
and use that instead for the incoming email from address as well as
the staged/regular user used for the Topic.user.

This will make it much cleaner to forward emails into a group inbox,
and will prevent issues with PostAlerter where the OP is double-notified
for these emails.
2021-08-24 08:57:28 +10:00
Kris bde6f7e9b0
UX: Update "get a room" composer message (#14104) 2021-08-23 18:34:23 -04:00
Penar Musaraj 3d6cbb5ab3
DEV: Watch for changes in wizard.scss (#14119)
This ensures the stylesheet is recompiled. Devs still need to refresh
the page manually, since we don't include the message bus in the wizard
app.
2021-08-23 13:39:53 -04:00
Joffrey JAFFEUX efbc2481d8
DEV: removes plugin generator (#14101)
I don't have the time to maintain this and https://github.com/discourse/discourse-plugin-skeleton is doing 80% of the job very easily.
2021-08-20 11:29:06 +02:00
Osama Sayegh ec352a1969
FEATURE: Order pinned topics by their `pinned_at` column (#14090)
Currently, pinned topics are ordered by the `bumped_at` column. This behavior is not desired because it gives admins no control over the order of pinned topics. This PR makes pinned topics ordered by the `pinned_at` column. A topic that is pinned last appears first in topic lists. If an admin wants an already pinned topic to appear first in the list of pinned topics, they'll have to unpin that topic and pin it again.

Meta topic: https://meta.discourse.org/t/how-do-i-set-the-order-of-pinned-topics/16935/23?u=osama.
2021-08-19 14:43:58 +03:00
Dan Ungureanu c0ec1e931e
FIX: Make themes:update work with multisites (#14082)
Running this Rake task in a multisite cluster updated only the default
site.
2021-08-19 09:41:58 +03:00
Arpit Jalan c481f2ce16
UX: do not show selected composer education messages on whisper post (#14078)
This commit disables the "sequential_replies" and "duplicate_link"
education message on composer when creating a whipser post.
2021-08-19 09:32:32 +05:30
Bianca Nenciu 5c9abab0cd
FIX: Remove markers from all translated languages (#14081)
Emails can include the marker in a different language, depending on
site and user settings. The email receiver always looked for the marker
in default language.
2021-08-18 19:42:04 +03:00
Arpit Jalan 7db3888f17
FIX: log proper error message when SSO nonce verification fails (#14077) 2021-08-18 18:44:12 +05:30
Bianca Nenciu 4380ba34d5
FIX: Make site tasks work with duplicated uploads (#13972)
Uploads can be reused between site settings. This change allows the same
upload to be exported only once and then the same file is reused. The
same applies to import.
2021-08-18 15:57:31 +03:00
Alan Guo Xiang Tan d13716286c
FIX: Unread group PMs should use `GroupUser#first_unread_pm_at`. (#14075)
This bug was causing unread PMs for groups to appear inaccurate.
2021-08-18 11:23:28 +08:00
Rafael dos Santos Silva b136375582
FEATURE: Rate limit exceptions via ENV (#14033)
Allow admins to configure exceptions to our Rails rate limiter.

Configuration happens in the environment variables, and work with both
IPs and CIDR blocks.

Example:

```
env:
  DISCOURSE_MAX_REQS_PER_IP_EXCEPTIONS: >-
    14.15.16.32/27
    216.148.1.2
```
2021-08-13 12:00:23 -03:00
Rafael dos Santos Silva b2e4c91818
DEV: Allow removing local assets after s3 sync (#14013) 2021-08-12 14:20:05 -03:00
Chema Balsas d27d7c8cca FIX: Unescapes hash section with present to account for url-encoded chars
Sections with unreserverd characters will appear url-encoded and need to
be unescaped before using it.

Wikipedia generates 2 different spans in this case in the same page, one
with an id resulting of replacing the % symbols with . and the other with
the decoded version of the string. For example, for /wiki/foo#A%C3%A1A it
will generate:

<span id="A.C3.A1A"></span>
<span id="AáA">AáA</span>

Unescaping the `m_url_hash_name` should work in all cases to target the
proper section span.
2021-08-12 10:43:50 -04:00
jbrw 20a6bad87e
FIX: Category group moderators can read flagged post meta_topics (#14014)
When a post is flagged with the reason of 'Something Else' a brief message can be added by the user which subsequently creates a `meta_topic` private message. The group `moderators` is automatically added to this topic.

If category group moderation is enabled, and the post belongs to a category with a reviewable group, that group should also be added to the meta_topic.

Note: This extends the `notify_moderators` logic, and will add the reviewable group to the meta_topic, regardless of the settings of that group.
2021-08-11 18:11:22 -04:00
Rafael dos Santos Silva 8e45fdfbb1
DEV: Log duration of ember-cli asset build (#13980) 2021-08-10 23:43:08 -03:00
Krzysztof Kotlarek 195f7346ba
FIX: check if BasicBadge is enabled for TL1 welcome message (#13983)
In 2018 check was added that TL1 welcome message is sent unless user already has BasicBadge granted.

I think we should also check if BasicBadge is even enabled. Otherwise, each time group is assigned to a user and trust level is recalculated, they will receive a welcome message.
2021-08-11 08:39:25 +10:00
Andrei Prigorshnev 622859dbe6
FEATURE: add Unseen view (#13977)
This view is the same as Latest except it hides the topics you have fully read. Based on this plugin of @davidtaylorhq https://meta.discourse.org/t/simple-unread-list-plugin-discourse-simple-unread/70013.
2021-08-10 18:30:34 +04:00
Dmitry Klimensky d54b339809 Escape values of HTML attributes 2021-08-10 10:25:15 -04:00
Joffrey JAFFEUX 644441852e
FIX: prevents exception on malformatted messages (#13997)
The following example message would generate an exception:

```
Return-Path: <discourse@bar.com>
From: Foo Bar <discourse@bar.com>
To: reply+4f97315cc828096c9cb34c6f1a0d6fe8@bar.com
Date: Fri, 15 Jan 2016 00:12:43 +0100
Message-ID: <21@foo.bar.mail>
Mime-Version: 1.0
Content-Type: text/html; charset=UTF-8

</div>

```

Exception:

```
NoMethodError:
       undefined method `split' for nil:NilClass
```
2021-08-10 15:49:32 +02:00
Arpit Jalan 3006de39d1
REVERT "FIX: do not show private group flair on user avatars" (#13991)
This reverts commit fe3e18f981 and 0d8fd9ace6
2021-08-10 17:25:11 +05:30
Laurent Arnoud 03896a9c43
FIX: Errno::EXDEV when across filesystem boundaries (#13976)
ref https://bugs.ruby-lang.org/issues/12615

```
        W: rake aborted!
        W: Errno::EXDEV: Invalid cross-device link @ rb_file_s_rename - (/app/tmp/adeeee62504de67238341871bda1aaf0, /app/app/assets/javascripts/plugins/discourse-cakeday.js.e
rb)
        W: lib/discourse.rb:65:in `rename'
        W: lib/discourse.rb:65:in `atomic_write_file'
        W: /app/lib/plugin/instance.rb:683:in `activate!'
        W: lib/discourse.rb:246:in `block in activate_plugins!'
        W: lib/discourse.rb:243:in `each'
        W: lib/discourse.rb:243:in `activate_plugins!'
```
2021-08-09 12:20:26 +02:00
Robin Ward 3fdb64184b FIX: Don't swallow an error if we can't run `yarn ember build` 2021-08-06 15:13:18 -04:00
Andrei Prigorshnev 09ad3ed41d
FEATURE: revert disallowing putting URLs in titles for TL0 users (#13970)
This reverts a part of changes introduced by https://github.com/discourse/discourse/pull/13947

In that PR I:
1. Disallowed topic feature links for TL-0 users
2. Additionally, disallowed just putting any URL in topic titles for TL-0 users

Actually, we don't need the second part. It introduced unnecessary complexity for no good reason. In fact, it tries to do the job that anti-spam plugins (like Akismet plugin) should be doing.

This PR reverts this second change.
2021-08-06 20:07:42 +04:00
Arpit Jalan bb2c48b065
FIX: update iframe url for simplecast onebox (#13957)
https://meta.discourse.org/t/onebox-regression-simplecast-com/187911
2021-08-05 18:29:04 +05:30
Robin Ward 18c5e9338f DEV: Allow us to use Ember CLI assets in production
This adds an optional ENV variable, `EMBER_CLI_PROD_ASSETS`. If truthy,
compiling production assets will be done via Ember CLI and will replace
the assets Rails would otherwise use.
2021-08-05 08:32:33 -04:00
Arpit Jalan a341dba5d9
FIX: update oEmbed URL for simplecast onebox (#13956) 2021-08-05 17:42:38 +05:30
Andrei Prigorshnev 0c0a11b66a
FEATURE: Disallow putting urls in the title for TL-0 users (#13947)
This disallows putting URLs in topic titles for TL0 users, which means that:

If a TL-0 user puts a link into the title, a topic featured link won't be generated (as if it was disabled in the site settings)
Server methods for creating and updating topics will be refusing featured links when they are called by TL-0 users
TL-0 users won't be able to put any link into the topic title. For example, the title "Hey, take a look at https://my-site.com" will be rejected.

Also, it improves a bit server behavior when creating or updating feature links on topics in the categories with disabled featured links. Before the server just silently ignored a featured link field that was passed to him, now it will be returning 422 response.
2021-08-05 13:38:39 +04:00
Alan Guo Xiang Tan 0bf27242ec FIX: Group inbox new filter not accounting for dismissed topics.
Follow-up to 2c046cc670
2021-08-05 16:53:12 +08:00
Alan Guo Xiang Tan 2c046cc670 FEATURE: Dismiss new and unread for PM inboxes. 2021-08-05 12:56:15 +08:00
Krzysztof Kotlarek 354c939656
FIX: remove Nokogumbo references (#13951)
Specs broken after f4720205c0
2021-08-05 11:46:25 +10:00
Martin Brennan d8a0d2262c
DEV: Update pretender and fake-xml-http-request (#13937)
We are still on a version of pretender since 2017
https://github.com/pretenderjs/pretender/releases/tag/v1.6.1

Since then many changes have been made, including adding support
for xhr.upload. Upgrading will let us write proper acceptance
tests for uppy, which uses XmlHTTPRequest internally including
xhr.upload.

Updates pretender to 3.4.7 and fake-xml-http-request to 2.1.2.

Note: There have been no breaking changes in the releases that would
affect us, mainly dropping support for old node versions.
2021-08-05 08:23:01 +10:00
Bianca Nenciu d9843d757a
FIX: Update draft count when sequence is increased (#13940)
* FIX: Update draft count when sequence is increased

Sometimes users ended up having a draft count higher than the actual
number of drafts.

* FIX: Do not update draft count twice

The call to DraftSequence.next! above already does it.
2021-08-04 13:30:37 +03:00
Bianca Nenciu e2af2a2219
FIX: Make rake site:export_structure export uploads (#13938)
It failed because of an undefined variable.
2021-08-04 11:14:30 +03:00
Bianca Nenciu 54e2b95539
FIX: Remove 'reply above line' marker (#13927)
This marker is automatically added by Discourse's group SMTP mailer.
When user responded to emails sent by it, Discourse did not trim this
marker.
2021-08-03 20:08:19 +03:00
Bianca Nenciu e2c415457c
FEATURE: Attach backup log as upload (#13849)
Discourse automatically sends a private message after backup or
restore finished. The private message used to contain the log inline
even when it was very long. A very long log can create issues because
the length of the post will be over the maximum allowed length of a
post. When that happens, Discourse will try to create an upload with
the logs. If that fails, it will trim the log and inline it.
2021-08-03 20:06:50 +03:00
Bianca Nenciu 52520638ca
FIX: Inline secure images with duplicated names (#13926)
Inlining secure images with the same name was not possible because they
were indexed by filename. If an email contained two files with the same
name, only the first image was used for both of them. The other file
was still attached to the email.
2021-08-03 18:58:34 +03:00
Martin Brennan b88d8c8894
FIX: Use reply-to address for incoming emails if present (#13896)
When the Reply-To header is present for incoming emails we
want to use it instead of the from address. This is usually the
case when forwarding an email via a mailing list into Discourse.

For now we are only using the Reply-To header if the email has
been forwarded via Google Groups, which is why we are checking the
X-Original-From header too. In future we may want to use the Reply-To
header in more cases.
2021-08-03 08:01:17 +10:00
Jean e7b8e75583
FEATURE: Add post edits count to user activity (#13495) 2021-08-02 10:15:53 -04:00
Bianca Nenciu fbf7627c8e
FIX: Make search work with sub-sub-categories (#13901)
Searching in a category looked only one level down, ignoring the site
setting max_category_nesting. The user interface did not support the
third level of categories and did not display them in the "Categorized"
input of the advanced search options.
2021-08-02 14:04:13 +03:00
Alan Guo Xiang Tan 016efeadf6
FEATURE: New and Unread messages for user personal messages. (#13603)
* FEATURE: New and Unread messages for user personal messages.

Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
2021-08-02 12:41:41 +08:00
Arpit Jalan fe3e18f981
FIX: do not show private group flair on user avatars (#13872)
Meta ref: https://meta.discourse.org/t/visible-flair-for-invisible-groups-is-that-on-purpose/167674
2021-08-02 06:21:00 +05:30
jbrw 2f28ba318c
FEATURE: Onebox can match engines based on the content_type (#13876)
* FEATURE: Onebox can match engines based on the content_type

`FinalDestination` now returns the `content_type` of a resolved URL.

`Oneboxer` passes this value to `Onebox` itself. Onebox engines can now specify a `matches_content_type` regex of content_types that the engine can handle, regardless of the URL.

`ImageOnebox` will match URLs with a content type of `image/png`, `jpg`, `gif`, `bmp`, `tif`, etc.

This will allow images that exist at a URL without a file type extension to be correctly rendered, assuming a valid `content_type` is returned.
2021-07-30 13:36:30 -04:00
Gerhard Schlager a37b5b177b
FIX: Remapping of uploads could fail during restore of backup (#13897)
Sidekiq should be paused until the end of the restore, otherwise it might interfere (deadlock) with the remap of upload URLs in posts.
2021-07-30 11:25:28 +02:00
Alan Guo Xiang Tan 849827841f Revert "DEV: Move private message topic tracking state publish into sidekiq."
This reverts commit c51c80466c.

There is a conditional in post jobs enqueuer that only enqueues the job
for non-PM posts.
2021-07-30 11:36:01 +08:00
Bianca Nenciu 300db3d3fa
FIX: Update draft count after creating a post (#13884)
When a post is created, the draft sequence is increased and then older
drafts are automatically executing a raw SQL query. This skipped the
Draft model callbacks and did not update user's draft count.

I fixed another problem related to a raw SQL query from Draft.cleanup!
method.
2021-07-29 17:06:11 +03:00
Alan Guo Xiang Tan c51c80466c DEV: Move private message topic tracking state publish into sidekiq.
Same intend as b1f32f2f57.
2021-07-29 14:57:19 +08:00
Andrei Prigorshnev 5a2ad7e386
DEV: remove calls to guardian from GroupActionLogger (#13835)
We shouldn't be checking if a user is allowed to do an action in the logger. We should be checking it just before we perform the action. In fact, guardians in the logger can make things even worse in case of a security bug. Let's say we forgot to check user's permissions before performing some action, but we still have a call to the guardian in the logger. In this case, a user would perform the action anyway, and this action wouldn't even be logged!

I've checked all cases and I confirm that we're safe to delete this calls from the logger.

I've added two calls to guardians in admin/user_controller. We didn't have security bugs there, because regular users can't access admin/... routes at all. But it's good to have calls to guardian in these methods anyway, neighboring methods have them.
2021-07-28 15:04:04 +04:00
Martin Brennan b500949ef6
FEATURE: Initial implementation of direct S3 uploads with uppy and stubs (#13787)
This adds a few different things to allow for direct S3 uploads using uppy. **These changes are still not the default.** There are hidden `enable_experimental_image_uploader` and `enable_direct_s3_uploads`  settings that must be turned on for any of this code to be used, and even if they are turned on only the User Card Background for the user profile actually uses uppy-image-uploader.

A new `ExternalUploadStub` model and database table is introduced in this pull request. This is used to keep track of uploads that are uploaded to a temporary location in S3 with the direct to S3 code, and they are eventually deleted a) when the direct upload is completed and b) after a certain time period of not being used. 

### Starting a direct S3 upload

When an S3 direct upload is initiated with uppy, we first request a presigned PUT URL from the new `generate-presigned-put` endpoint in `UploadsController`. This generates an S3 key in the `temp` folder inside the correct bucket path, along with any metadata from the clientside (e.g. the SHA1 checksum described below). This will also create an `ExternalUploadStub` and store the details of the temp object key and the file being uploaded.

Once the clientside has this URL, uppy will upload the file direct to S3 using the presigned URL. Once the upload is complete we go to the next stage.

### Completing a direct S3 upload

Once the upload to S3 is done we call the new `complete-external-upload` route with the unique identifier of the `ExternalUploadStub` created earlier. Only the user who made the stub can complete the external upload. One of two paths is followed via the `ExternalUploadManager`.

1. If the object in S3 is too large (currently 100mb defined by `ExternalUploadManager::DOWNLOAD_LIMIT`) we do not download and generate the SHA1 for that file. Instead we create the `Upload` record via `UploadCreator` and simply copy it to its final destination on S3 then delete the initial temp file. Several modifications to `UploadCreator` have been made to accommodate this.

2. If the object in S3 is small enough, we download it. When the temporary S3 file is downloaded, we compare the SHA1 checksum generated by the browser with the actual SHA1 checksum of the file generated by ruby. The browser SHA1 checksum is stored on the object in S3 with metadata, and is generated via the `UppyChecksum` plugin. Keep in mind that some browsers will not generate this due to compatibility or other issues.

    We then follow the normal `UploadCreator` path with one exception. To cut down on having to re-upload the file again, if there are no changes (such as resizing etc) to the file in `UploadCreator` we follow the same copy + delete temp path that we do for files that are too large.

3. Finally we return the serialized upload record back to the client

There are several errors that could happen that are handled by `UploadsController` as well.

Also in this PR is some refactoring of `displayErrorForUpload` to handle both uppy and jquery file uploader errors.
2021-07-28 08:42:25 +10:00
wbhouston efe38efb0a
FIX: Don't grant sharing badges to users who don't exist (#13851)
In badge queries for 'First Share' and 'Nice/Good/Great Share' badges,
check that the user exists.

For 'Nice+ Share' badges, also grant badges if the number of shares is
equal to the threshhold count to better match the descriptions.
2021-07-27 16:32:59 +10:00
Alan Guo Xiang Tan 02b84dbff2 DEV: Fix flaky instagram onebox spec by not mutating constant. 2021-07-27 13:54:14 +08:00
Martin Brennan e1d2b67178
DEV: Fix stylesheet manager flaky spec (#13846)
The cache_fullpath for the Stylesheet::Manager was the same for
every test runner in a parallel test environment, so when other
specs or other places e.g. the stylesheets_controller_spec ran
rm -rf Stylesheet::Manager.cache_fullpath this caused errors
for other specs running that went through the
Stylesheet::Manager::Builder#compile path, causing the error

```
Errno::ENOENT:
       No such file or directory @ rb_sysopen
```


Also fixed the stylesheet_controller which was interpolating Rails.root + CACHE_PATH
itself instead of just using Stylesheet::Manager.cache_fullpath
2021-07-26 14:22:54 +10:00
Joffrey JAFFEUX ed9e63b00e DEV: update doc as current would raise 2021-07-26 08:54:34 +08:00
Robin Ward 7b45a5ce55 FIX: Better and more secure validation of periods for TopicQuery
Co-authored-by: Martin Brennan <mjrbrennan@gmail.com>
2021-07-23 14:24:44 -04:00
Alan Guo Xiang Tan 2923abdec7 SECURITY: Don't leak user of previous whisper post when deleting a topic.
A topic's last poster can be incorrectly set to a user of a whisper post
if the whisper post is before the last post and the last post is
deleted.
2021-07-23 15:42:23 +08:00
Alan Guo Xiang Tan 303e9e42b6 SECURITY: Do not reveal post whisperer in personal messages.
Prior to this fix, post whisperer in personal messages are revealed in
the topic's participants list even though non-staff users are unable to
see the whisper.
2021-07-23 15:42:23 +08:00
Neil Lalonde cb858af8c7
Version bump to v2.8.0.beta4 (#13824) 2021-07-22 12:23:50 -04:00
Bianca Nenciu 18c32a809b
FIX: Validate email_accent_bg_color color (#13778)
Using an invalid value was allowed. This commit tries to automatically
fix the color by adding missing # symbol or will show an error to the
user if it is not possible and it is not a CSS color either.
2021-07-22 17:42:47 +03:00
Martin Brennan f41908ad5b
SECURITY: Validate period param for top topic routes (#13818)
Fixes a possible SQL injection vector
2021-07-22 16:31:53 +10:00
jbrw 09d23a37a5
DEV: Add default `Accept-Language` to FinalDestination requests (#13817)
Not specifying an `Accept-Language` should be equivalent to specifying an `Accept-Language` of `*`, however some webservers seem to prefer it if we are explicit about being able to handle a response of content in any language.
2021-07-22 15:49:59 +10:00
Penar Musaraj 8a470e508e
UX: Improve quick search suggestions (#13813) 2021-07-21 14:00:27 -04:00
Bianca Nenciu 4da0a33524
FIX: Allow only groups with flairs to be selected (#13744)
It used the same permission check as for primary groups which is
wrong because not all groups that can be primary have a flair.
2021-07-21 14:41:04 +03:00
Jarek Radosz 4223541bc5
DEV: Fix release_note task warnings (#13794)
There was a bunch of warnings repeated over and over during spec runs:

```
/var/www/discourse/lib/tasks/release_note.rake:3: warning: already initialized constant DATE_REGEX
/var/www/discourse/lib/tasks/release_note.rake:3: warning: previous definition of DATE_REGEX was here
/var/www/discourse/lib/tasks/release_note.rake:5: warning: already initialized constant CHANGE_TYPES
/var/www/discourse/lib/tasks/release_note.rake:5: warning: previous definition of CHANGE_TYPES was here
```
2021-07-20 20:25:48 +02:00
Kris 5f6b9e36ed
UX: New text and style for dominating topic message (#13789) 2021-07-20 13:58:38 -04:00
Andrei Prigorshnev 1a8c949900
UX: suspend forever time period messages (#13776)
When the Forever option is selected for suspending a user, the user is suspended for 1000 years. Without customizing the site’s text, this time period is displayed to the user in the suspension email that is sent to the user, and if the user attempts to log back into the site. Telling someone that they have been suspended for 1000 years seems likely to come across as a bad attempt at humour.

This PR special case messages when a user suspended or silenced forever.
2021-07-20 14:42:08 +04:00
Vinoth Kannan 08c6a7679d
DEV: allow `dev:populate` rake task in production environment. (#13777)
To use this rake task in production `ALLOW_DEV_POPULATE` environment variable should be set to "1".
2021-07-20 12:25:59 +05:30
Michael Brown 784e929499 upload_creator: force ImageMagick to use internal SVG code to determine size
This change largely targets dev users, but it could potentially change
behaviour in production.

Jamie Wilson & I debugged a problem where "should not be larger than the
maximum thumbnail size" would fail due to timeouts.

On our systems, on ImageMagick 7.1.0-2, with inkscape installed, IM would
attempt to rasterise the svg then check the resulting filesize, causing the
test to timeout.

As of now, we haven't found a way to cause this to behave better, but have a
workaround in that forcing IM to use the internal renderer (`MSVG:`) seems to
make it perform the same on development workstations as it does in our docker
container.
2021-07-19 22:03:57 -04:00
Michael Brown 5f7e60d9dc discourse/discourse change from 'master' to 'main' 2021-07-19 11:46:15 -04:00
Alan Guo Xiang Tan a1047f5ef4
FEATURE: Add new plugin API to allow plugins to extend `Site#categories` (#13773) 2021-07-19 13:54:19 +08:00
Neil Lalonde b0f06b8ed0
FIX: don't allow category and tag tracking settings on staged users (#13688)
Configuring staged users to watch categories and tags is a way to sign
them up to get many emails. These emails may be unwanted and get marked
as spam, hurting the site's email deliverability.
Users can opt-in to email notifications by logging on to their
account and configuring their own preferences.

If staff need to be able to configure these preferences on behalf of
staged users, the "allow changing staged user tracking" site setting
can be enabled. Default is to not allow it.

Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2021-07-16 14:50:40 -04:00
Neil Lalonde 5f8fa976d4
Version bump to v2.8.0.beta3 (#13703) 2021-07-15 14:54:02 -04:00
David Taylor 8b89787426
SECURITY: Sanitize YouTube Onebox data (#13748)
CVE-2021-32764
2021-07-15 19:31:50 +01:00
Penar Musaraj a23153fdca
FIX: Add order to outputted stylesheet link tags (#13735)
See PR for details. (Disabled by default in this commit.)
2021-07-15 12:51:46 -04:00
Vinoth Kannan 74b3730143
DEV: return populated data at the end of the method. (#13739)
And some minor refactoring.
2021-07-15 17:45:32 +05:30
David Taylor 4ce58fbf0b
DEV: Improve rake `release_note:generate` date handling (#13726)
* DEV: Improve rake `release_note:generate` date handling

A commit-ish value like HEAD@{2021-01-01} is based on the **local state** of HEAD on that date. It does not use dates attached to commits.

Instead, the rake task now detects date-like strings and supplies them to `git log` via the `--after` and `--before` flags

* Skip printing plugin when there are no changes found

A list of skipped plugins is printed on a single line at the end of the output
2021-07-15 16:15:57 +10:00
Vinoth Kannan 76fe3a16a9
DEV: trigger new discourse events `after_create_dev_record` & `after_populate_dev_records`. (#13733)
After every new random record created using the `dev:populate` rake task a new Discourse event will be triggered. So the plugins can modify the records if needed.
2021-07-15 11:11:41 +05:30
Penar Musaraj f7ab852e12
FIX: Issues with custom icons in themes (#13732)
Fixes two issues:
- ignores invalid XML in custom icon sprite SVG file (and outputs an error if sprite was uploaded via admin UI)
- clears SVG sprite cache when deleting an `icons-sprite` upload in a theme
2021-07-14 15:18:29 -04:00
Jarek Radosz c750bfb4af
FIX: A memoization bug in UserLookup and refactor (#13692)
`@group_lookup` memoization bug was introduced in #13587.
2021-07-14 13:51:33 +02:00
Gerhard Schlager 2a95f892af
DEV: Update `i18n:check` rake task to detect invalid Markdown links (#13728)
In addition to that it fixes a problem where the check failed on empty locale files and allows calling the rake task with multiple locales.
2021-07-14 13:26:12 +02:00
Kim Lindberger 7e52eada20
FIX: Use Terser for minification even if uglify-js is not available (#13683) 2021-07-14 14:20:57 +08:00
jbrw a64aea38b7
FIX: Don’t use `user_generated` images as avatar images in Oneboxed Twitter content (#13712)
By default, Twitter will return the URL for the avatar image of the tweet poster as the `og:image` value.

However, if the `user_generated` attribute is true, we should not use this as the avatar URL as this will be an URL of an image in the tweet itself (e.g., an image belonging to a tweeted news story).
2021-07-13 14:54:28 -04:00
Guhyoun Nam 0dc96ce817
FEATURE: Setting to allow moderators to change post ownership (#13708) 2021-07-13 09:40:11 -05:00
Dan Ungureanu 49090c3524
FIX: Suggest current username for staged users (#13706)
If user had a staged account and logged in using a third party service
a different username was suggested. This change will try to use the
username given by the authentication provider first, then the current
staged username and last suggest a new one.
2021-07-13 09:15:06 +10:00
David Taylor 76f279dfb4
DEV: Update release notes output format, and add plugin task (#13709) 2021-07-12 21:23:50 +01:00
Bianca Nenciu f74640ac58
FIX: Show correct dates in topic timelines (#13670)
The date shown in topic timeline was one day later if the post at that
position was made near midnight. This happened because the days number
was rounded down.
2021-07-12 19:35:24 +03:00
Kane York 81d1a61d17
DEV: Report message-id, skip reason in rake email:test (#13674)
meta/t/185109

Additionally, print an extra warning if the `disable_emails` site setting is set.
2021-07-09 11:14:18 -07:00
Bianca Nenciu 87c1e98571
FEATURE: Let users select flair (#13587)
User flair was given by user's primary group. This PR separates the
two, adds a new field to the user model for flair group ID and users
can select their flair from user preferences now.
2021-07-08 10:46:21 +03:00
Blake Erickson 448a564845
DEV: Remove super old code comment TODO (#13663)
This TODO comment has existed for 8 years. Sort must be working just
fine or we would have prioritized fixing it.

Removing this comment as a tiny step toward keeping our codebase nice
and tidy.
2021-07-07 14:34:05 -06:00
Arpit Jalan dec7e19da3
FIX: fix error message for fix_missing_s3 rake task (#13661) 2021-07-07 19:59:03 +05:30
Penar Musaraj 35110f6681
FIX: Set CSP base-uri to `self` (#13654) 2021-07-07 09:43:48 -04:00
Arpit Jalan 236d6d91b2
FIX: `fix_missing_s3` task fails on failed upload (take 2) (#13660)
ref: 935aadbfdd
2021-07-07 18:53:43 +05:30
Arpit Jalan 935aadbfdd
FIX: do not stop `fix_missing_s3` task if saving an upload failed (#13658)
This commit logs an error and moves to next upload when saving a single
upload record fails when running `uploads:fix_missing_s3` task.
2021-07-07 16:57:24 +05:30
Martin Brennan 35f6441938
DEV: Add uppy.js to build and project (#13645)
This PR adds uppy to the project with a custom JS build and the shims needed to import it into our JS code. We need a custom build of Uppy because we do not use webpack for our JS modules/build. The only way to get what you want from Uppy is to use the webpack modules or to include the entire Uppy project including all plugins in a single JS file. This way we can just use the plugins we actually want. Future PRs will actually use Uppy!
2021-07-07 10:39:33 +10:00
dependabot[bot] dda41cf253
Build(deps): Bump rubocop from 1.18.2 to 1.18.3 (#13653)
* Build(deps): Bump rubocop from 1.18.2 to 1.18.3

Bumps [rubocop](https://github.com/rubocop/rubocop) from 1.18.2 to 1.18.3.
- [Release notes](https://github.com/rubocop/rubocop/releases)
- [Changelog](https://github.com/rubocop/rubocop/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rubocop/rubocop/compare/v1.18.2...v1.18.3)

---
updated-dependencies:
- dependency-name: rubocop
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix offences

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-07-07 01:51:43 +02:00
Penar Musaraj 95b5794331
DEV: Compile core and plugin stylesheets independently of themes (#13638)
Take 2 of https://github.com/discourse/discourse/pull/13466. 

Fixes a few issues with the original PR: 

- color definition stylesheet target now includes the theme id, to avoid themes set to use the default color scheme loading the same stylesheet 
- changes the internal cache key for color definition stylesheet to reset the pre-existing cache
2021-07-06 13:11:10 -04:00
David Taylor 8c370c3fe3 DEV: Add `annotate` rake tasks, and enforce via GitHub actions
`bin/rake annotate` is an alias of `bin/annotate --models`
`bin/rake annotate:clean` generates annotations by using a temporary, freshly migrated database. This should help us to produce more consistent annotations, even if development databases have been polluted by plugin migrations.

A GitHub actions task is also added which generates annotations on a clean database, and raises an error if they differ from the committed annotations.
2021-07-06 10:11:06 +01:00
David Taylor 2955d64703
DEV: Allow annotations to work in symlinked plugins, add binstub (#13635) 2021-07-05 15:43:10 +01:00
Alan Guo Xiang Tan 37b8ce79c9
FEATURE: Add last visit indication to topic view page. (#13471)
This PR also removes grey old unread bubble from the topic badges by
dropping `TopicUser#highest_seen_post_number`.
2021-07-05 14:17:31 +08:00
Penar Musaraj 02999f5eb2
Skip compressing theme test files (#13619) 2021-07-02 12:16:16 -04:00
Roman Rizzi 6be956339c
DEV: Tests can reset a single registry (#13618)
Plugins can use this method to reset their specific registries without touching the ones defined in core.
2021-07-02 12:40:50 -03:00
Dan Ungureanu 6ea4bbd2ec
DEV: Prefer .pluck_first over .pluck.first (#13607) 2021-07-02 10:03:54 +08:00
Arpit Jalan 05bdbd9f97
SECURITY: Onebox canonical links bypassing FinalDestination checks (#13605) 2021-07-01 20:09:29 +05:30
Alan Guo Xiang Tan 3312e09d02
DEV: Incorrect name in precompile output. (#13602)
Follow-up to c54d58e28f
2021-07-01 10:52:21 +08:00
Alan Guo Xiang Tan c54d58e28f
FIX: Child themes being precompiled multiple times take 2. (#13599)
This fix was reverted in 128fdf9d9c but
fix is still relevant.
2021-07-01 10:09:39 +08:00
Martin Brennan 59582102d8
FIX: Rename ninja edit terminology to grace period (#13598)
We renamed the site setting for this long ago, but there
were a few places left in the code base where "ninja edit"
needed to be turned into "grace period". Doing this here
to avoid combatative language.
2021-07-01 11:27:11 +10:00
Penar Musaraj 128fdf9d9c
DEV: Revert stylesheet refactors (#13584)
* Revert "FIX: Clear appropriate cache when updating font settings (#13582)"

This reverts commit de6cc7a924.

* Revert "DEV: Improve output of `Stylesheet::Mananger.precompile_theme_css`."

This reverts commit 95038856c9.

* Revert "FIX: Child themes being precompiled multiple times."

This reverts commit 6986b36985.

* Revert "Update spec/components/stylesheet/manager_spec.rb"

This reverts commit ddaa7cc7ea.

* Revert "Refactor scss live refreshing"

This reverts commit a838293aaf.

* Revert "Precompile core stylesheets independently of themes"

This reverts commit 99d259d39b.

* Revert "DEV: Add simple digest for core stylesheets"

This reverts commit d82c58e6cc.
2021-06-30 21:33:15 +08:00
Bianca Nenciu 715ecabd85
FIX: TL4 users cannot delete others posts (#13554) 2021-06-30 15:51:35 +03:00
Alan Guo Xiang Tan 95038856c9 DEV: Improve output of `Stylesheet::Mananger.precompile_theme_css`. 2021-06-30 16:58:12 +08:00
Arpit Jalan b63c9febe8
FIX: ignore canonical link to localhost (#13577) 2021-06-30 13:55:17 +05:30
Alan Guo Xiang Tan 6986b36985 FIX: Child themes being precompiled multiple times. 2021-06-30 16:09:45 +08:00
Penar Musaraj a838293aaf Refactor scss live refreshing 2021-06-30 13:42:40 +08:00
Penar Musaraj 99d259d39b Precompile core stylesheets independently of themes 2021-06-30 13:42:40 +08:00
Penar Musaraj d82c58e6cc DEV: Add simple digest for core stylesheets
And move fonts + category_backgrounds to color definitions stylesheet.
This will let us use the same core + plugin stylesheets in multisite.
2021-06-30 13:42:40 +08:00
mintsaxon 7d0d13c32e FEATURE: add slow_mode_prevents_editing setting..
..as per https://meta.discourse.org/t/slow-mode-for-a-category/179574/16
2021-06-28 13:08:10 -04:00
Jarek Radosz 04baca593b
UX: Tweak the timestamp line in Twitter onebox (#13551)
Fixed alignment and made the color less intrusive to make the actual content pop out more.
2021-06-28 15:04:33 +02:00
Andrei Prigorshnev 6be4699954
FIX: topic level bookmark button (#13530)
We changed (https://github.com/discourse/discourse/pull/13407) behaviour of the topic level bookmark button recently. That PR made the button be opening the edit bookmark modal when there is only one bookmark on the topic instead of just removing that bookmark as it was before.

This PR fixes the next problems that weren't taken into account in the previous PR:

1. Everything should work fine even on very big topics when a bookmarked post is unloaded from the post stream. I've added code that loads the post we need and makes everything work as expected
2. When at least one bookmark on the topic has a reminder, we should always be showing the icon with a clock on the topic level bookmark button
3. We should show correct tooltips for the topic level bookmark button
2021-06-28 12:24:23 +04:00
Alan Guo Xiang Tan 7719453fb7 DEV: Don't eager load tags when tagging is not enabled. 2021-06-28 16:00:10 +08:00
Alan Guo Xiang Tan ff1c53dd6f FIX: Missing category edit icon.
Follow-up to 0e4b8c5318
2021-06-28 10:54:23 +08:00
Martin Brennan d3e27cabf6
FIX: Improve participant display in group SMTP emails (#13539)
This PR makes several changes to the group SMTP email contents to make it look more like a support inbox message.

* Remove the context posts, they only add clutter to the email and replies
* Display email addresses of staged users instead of odd generated usernames
* Add a "please reply above this line" message to sent emails
2021-06-28 10:42:06 +10:00
Martin Brennan 87684f7c5e
FEATURE: Use group SMTP job and mailer instead of UserNotifications change (#13489)
This PR backtracks a fair bit on this one https://github.com/discourse/discourse/pull/13220/files.

Instead of sending the group SMTP email for each user via `UserNotifications`, we are changing to send only one email with the existing `Jobs::GroupSmtpEmail` job and `GroupSmtpMailer`. We are changing this job and mailer along with `PostAlerter` to make the first topic allowed user the `to_address` for the email and any other `topic_allowed_users` to be the CC address on the email. This is to cut down on emails sent via SMTP, which is subject to daily limits from providers such as Gmail. We log these details in the `EmailLog` table now.

In addition to this, we have changed `PostAlerter` to no longer rely on incoming email email addresses for sending the `GroupSmtpEmail` job. This was unreliable as a user's email could have changed in the meantime. Also it was a little overcomplicated to use the incoming email records -- it is far simpler to reason about to just use topic allowed users.

This also adds a fix to include cc_addresses in the EmailLog.addressed_to_user scope.
2021-06-28 08:55:13 +10:00
Jarek Radosz 203d56719d
UX: Improve blockquote styling in emails (#13527) 2021-06-25 20:13:46 +02:00
Roman Rizzi fa4e5e8dad
FEATURE: Render emojis on GitHub labels when oneboxing an issue. (#13531) 2021-06-25 14:48:36 -03:00
Jarek Radosz 61472d6aaa
DEV: Rename `hilight` to `highlight` (#13526) 2021-06-25 18:05:50 +02:00
Martin Brennan c3394ed9bb
DEV: Update aws-sdk-s3 gem for S3 multipart uploads (#13523)
We are a few versions behind on this gem. We need to update it
for S3 multipart uploads. In the current version we are using, we
cannot do this:

```ruby
Discourse.store.s3_helper.object(key).presigned_url(:upload_part, part_number: 1, upload_id: multipart_upload_id)
```

The S3 client raises an error, saying the operation is undefined. Once
I updated the gem this operation works as expected and returns a
presigned URL for the upload_part operation.

Also remove use of Aws::S3::FileUploader::FIFTEEN_MEGABYTES.
This was part of a private API and should not have been used.
2021-06-25 14:22:31 +10:00
Arpit Jalan 0adeddde61
FIX: follow redirects for inline/mini onebox (#13512) 2021-06-24 19:53:39 +05:30
Alan Guo Xiang Tan 0e4b8c5318 PERF: Cache categories in Site model take 3.
Previous attempt resulted in custom fields going missing in the
serialized output.

This reverts commit 83a6ad32ff.
2021-06-24 13:30:51 +08:00
Jarek Radosz 046a875222
DEV: Improve `script/downsize_uploads.rb` (#13508)
* Only shrink images that are used in Posts and no other models
* Don't save the upload if the size is the same
2021-06-24 00:09:40 +02:00
David Taylor d2c5165052 FIX: Check all migrations for dropped columns/tables during restore
Previously only post-deploy migrations were being checked for DROPPED_(COLUMNS|TABLES) constants
2021-06-23 17:43:38 +01:00
Rafael dos Santos Silva fa4a462517
FEATURE: Optimize images before upload (#13432)
Integrates [mozJPEG](https://github.com/mozilla/mozjpeg) and [Resize](https://github.com/PistonDevelopers/resize) using WebAssembly to optimize user uploads in the composer on the client-side.

NPM libraries are sourced from our [Squoosh fork](https://github.com/discourse/squoosh/tree/discourse), which was needed because we have an older asset pipeline.
2021-06-23 12:31:12 -03:00
Penar Musaraj cfc60f41f0
DEV: Rename emoji icon (#13499) 2021-06-23 11:12:48 -04:00
Osama Sayegh fa62b5e83b
DEV: Add a way to exclude ENV vars from getting unset in `themes:isolated_test` (#13494) 2021-06-23 14:50:54 +03:00
Osama Sayegh d3a3d1b94c
DEV: Introduce `TemporaryRedis` and unset `DISCOURSE_*` env vars in the `themes:isolated_test` rake task (#13401)
The `themes:isolated_test` rake task will now unset all `DISCOURSE_*` env variables if `UNSET_DISCOURSE_ENV_VARS` env var is set and will also spin up a temporary redis server so the unicorn web server that's spun up for the tests doesn't leak into the "main" redis server.
2021-06-23 07:38:43 +03:00
Jarek Radosz 425ead8f35 DEV: Remove a perf fix that's no longer needed
It has been added to Rails in cc2d614e63 (v6.0.0.beta2)
2021-06-23 11:21:12 +08:00
Penar Musaraj fc0da499f8
DEV: Refactor custom svg icon caching (#13483)
Previously, we were storing custom svg sprite paths in the cache. This is a problem because sprites in themes get stored as uploads, and the returned paths were files in the temporary download cache which could sometimes be cleaned up, resulting in a broken cache.

I previously tried to fix this by skipping the missing files and clearing the cache, but that didn't work out well with CDNs. This PR stores the contents of the files in the custom_svg_sprites cache to avoid the problem of missing temp files.

Also, plugin custom icons are only included if the plugin is enabled.
2021-06-22 14:07:23 -04:00
Mark VanLandingham 7fc3d7bdde
DEV: Plugin API to add directory columns (#13440) 2021-06-22 13:00:04 -05:00
Roman Rizzi e0e1e24c14
FIX: Ignore posts needing approval when calculating reviewable counts. (#13464)
In #12841, we started setting the ReviewableQueuedPost's target and topic after approving it instead of storing them in the payload. As a result, the reviewable_counts query started to include queued posts.

When a category is set to require approval, every post has an associated reviewable. Pointing that each post has an associated queued post is not necessary in this case, so I added a WHERE clause to skip them.
2021-06-22 12:12:39 -03:00
David Taylor e76c583b91
DEV: Promote old post-deploy migrations to pre-deploy migrations (#13477)
Having a large number of post-deploy migrations running out-of-numerical-sequence with pre-deploy migrations can be problematic. For example, if we have the sequence

- db/migrate/2017... - add column
- db/post_migrate/2018... - drop the column
- db/migrate/2021... - add the same column again

It will work fine in numerical order. But if you run the pre-deploy migrations **followed by** the post-deploy migrations, you will not get the same result.

Our post-deploy system is designed to allow for seamless upgrades of Discourse. However, it is reasonable for us to only support this totally seamless experience for a limited period of time. This commit moves all post_deploy migrations which are more than 1 year old (i.e. more than 2 major Discourse versions ago) into the regular pre-deploy migrations directory. This limits the impact of any edge cases caused by out-of-numerical-sequence migrations.
2021-06-22 16:02:24 +01:00
Gerhard Schlager 820068ddaf
FIX: `fix_missing_s3` rake task could fail due to missing upload (#13479) 2021-06-22 17:00:55 +02:00
Penar Musaraj f343cfd92e
DEV: Remove IntersectionObserver polyfill (#13445) 2021-06-22 09:30:44 -04:00
Jarek Radosz cd8a82a82f
FIX: ShrinkUploadedImage was broken since cb12a721 (#13474) 2021-06-22 14:34:38 +02:00
Alan Guo Xiang Tan 83012b6473 FIX: Previewing theme components broken when component has no SCSS. 2021-06-22 09:48:48 +08:00
Martin Brennan 5222247746
FEATURE: Add more columns to outbound EmailLog (#13449)
This adds the following columns to EmailLog:

* cc_addresses
* cc_user_ids
* topic_id
* raw

This is to bring the EmailLog table closer in parity to
IncomingEmail so it can be better utilized for Group SMTP
and IMAP mailing.

The raw column contains the full content of the outbound email,
but _only_ if the new hidden site setting
enable_raw_outbound_email_logging is enabled. Most sites do not
need it, and it's mostly required for IMAP and SMTP sending.

In the next pull request, there will be a migration to backfill
topic_id on the EmailLog table, at which point we can remove the
topic fallback method on EmailLog.
2021-06-22 08:32:01 +10:00
Kane York 83a6ad32ff Revert "PERF: Cache categories in Site model take 2."
This reverts commit 06fa1efd3d.

Breakage in solved plugin
2021-06-21 12:25:04 -07:00
Bianca Nenciu e70e8d8d8f
FIX: TL4 users can see their deleted posts (#13364) 2021-06-21 19:10:02 +03:00
Joffrey JAFFEUX e50b7e9111
SECURITY: ensures timeouts are correctly used on connect (#13455) 2021-06-21 17:34:01 +02:00
Penar Musaraj 515fd8a4c3
DEV: Change method used to reference custom font assets (#13446)
The `asset-url` helper has been deprecated.
2021-06-21 09:33:12 -04:00
Dan Ungureanu c69b58e22e
FIX: Downsize image last when saving uploads (#13458)
The other processing operations, such as fixing orientation or cropping,
can in rare cases increase the size of the uploaded image. Running the
downsize step after all these operations should create the best image
possible.
2021-06-21 14:13:24 +03:00
Alan Guo Xiang Tan e64f9c87fb DEV: Fix broken stylesheet watcher.
Follow-up to 8e3691d537
2021-06-21 16:43:07 +08:00
Alan Guo Xiang Tan 7901f73597 FIX: Multiple links being generated for non-theme targets.
Follow-up to 8e3691d537
2021-06-21 16:26:49 +08:00
Alan Guo Xiang Tan be655762d8 FIX: Color scheme stylesheet not loaded when no default theme is set.
Follow-up to 8e3691d537
2021-06-21 12:50:45 +08:00
Martin Brennan 7b31d8a11b
DEV: Move chrome binary check into a shared lib (#13451)
We had checks for the chrome binary in 3 different places
for tests and only one of them checked for google-chrome-stable,
which is problematic for Arch linux users (there are dozens of us!)

This PR moves all the code to one place and references it instead
of copying and pasting.
2021-06-21 13:28:48 +10:00
Alan Guo Xiang Tan 44aa46ca05 Code review comments. 2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan 8e3691d537 PERF: Eager load Theme associations in Stylesheet Manager.
Before this change, calling `StyleSheet::Manager.stylesheet_details`
for the first time resulted in multiple queries to the database. This is
because the code was modelled in a way where each `Theme` was loaded
from the database one at a time.

This PR restructures the code such that it allows us to load all the
theme records in a single query. It also allows us to eager load the
required associations upfront. In order to achieve this, I removed the
support of loading multiple themes per request. It was initially added
to support user selectable theme components but the feature was never
completed and abandoned because it wasn't a feature that we thought was
worth building.
2021-06-21 11:06:58 +08:00
Alan Guo Xiang Tan 06fa1efd3d PERF: Cache categories in Site model take 2.
Follow-up to aa4f0aee67.

Fixed the security problem in the previous attempt.
2021-06-21 09:47:05 +08:00
Martin Brennan 22b96c9ce1
FIX: Prevent resurrecting old topics via email reply for group inboxes with SMTP enabled (#13382)
We already reject email replies to public topics via `SiteSetting.disallow_reply_by_email_after_days` and raising the `OldDestinationError`. This PR introduces similar behaviour for group inboxes, but without the rejection, and **only when SMTP is enabled for the group**.

If a reply is sent via email and the post is older than `SiteSetting.disallow_reply_by_email_after_days` days ago, then we create a new topic instead of making a reply in the old one and link back to the original topic. This is done to prevent long running group inbox discussions.
2021-06-21 11:45:00 +10:00
Bianca Nenciu 1e992d9193
FIX: Do not check for duplicate links in Onebox (#13345)
If a user posted a URL that appeared inside a Onebox, then the user
got a duplicate link notice. This was fixed by skipping those links in
Ruby.

If a user posted a URL that was Oneboxes and contained other links that
appeared in previous posts, then the user got a duplicate link notice.
This was fixed by skipping those links in JavaScript.
2021-06-18 18:55:24 +03:00
Joffrey JAFFEUX 0fd55acf84 Update lib/site_settings/type_supervisor.rb
Co-authored-by: David Taylor <david@taylorhq.com>
2021-06-18 16:36:23 +08:00
jjaffeux dc63613c96 DEV: improve error message for invalid setting’s value
Before this fix we would display this exception:

```
Discourse::InvalidParameters:
  value
```

After this fix we will display:

```
Discourse::InvalidParameters:
  Invalid `x` value for `s3_region`
```
2021-06-18 16:36:23 +08:00
Martin Brennan ff6114d83f
FIX: Do not add mailing list headers to group SMTP emails (#13431)
When we are emailing people from a group inbox, we are having
a PM conversation with them, as a support account would. In this
case mailing list headers do not make sense. It is not like a forum
topic where you may have tens or hundreds of participants -- it is a
conversation between the group and a small handful of people
directly contacting the group, often just one person.

The only header left in tact was List-Unsubsribe which is important
for letting people opt out to notifications.
2021-06-18 14:36:17 +10:00
jbrw fbfd1fd80b
FIX: Allow SVG uploads if dimensions are a fraction of a unit (#13409)
* FIX: Allow SVG uploads if dimensions are a fraction of a unit

`UploadCreator` counts the number of pixels in an file to determine if it is valid. `pixels` is calculated by multiplying the width and height of the image, as determined by FastImage.

SVG files can have their width/height expressed in a variety of different units of measurement. For example, ‘px’, ‘in’, ‘cm’, ‘mm’, ‘pt’, ‘pc’, etc are all valid within SVG files. If an image has a width of `0.5in`, FastImage may interpret this as being a width of `0`, meaning it will report the `size` as being `0`.

However, we don’t need to concern ourselves with the number of ‘pixels’ in a SVG files, as that is irrelevant for this file format, so we can skip over the check for `pixels == 0` when processing this file type.

* DEV: Speed up getting SVG dimensions

The `-ping` flag prevents the entire image from being rasterized before a result is returned. See:

https://imagemagick.org/script/command-line-options.php#ping
2021-06-17 15:56:11 -04:00
Mark VanLandingham 95b51669ad
DEV: Revert 3 commits for plugin API to add directory columns (#13423) 2021-06-17 12:37:37 -05:00
Osama Sayegh c47f55253f
DEV: Add optional theme test step to the `smoke:test` rake task (#13418)
The purpose of this is to allow us to catch regressions for a feature we've built recently that allows theme tests to run in production. We recently had a regression that we didn't notice for days, so to prevent that from happening again we'll use this in our internal CI pipelines.
2021-06-17 20:09:29 +03:00
Mark VanLandingham 0c42a29dc4
DEV: Plugin API to allow creation of directory columns with item query (#13402)
The first thing we needed here was an enum rather than a boolean to determine how a directory_column was created. Now we have `automatic`, `user_field` and `plugin` directory columns.

This plugin API is assuming that the plugin has added a migration to a column to the `directory_items` table.

This was created to be initially used by discourse-solved. PR with API usage - https://github.com/discourse/discourse-solved/pull/137/
2021-06-17 09:06:18 -05:00
Alan Guo Xiang Tan aa4f0aee67 Revert "PERF: Cache categories in Site model."
This reverts commit 7dc0f88acd.
2021-06-17 15:20:35 +08:00
Alan Guo Xiang Tan 7dc0f88acd PERF: Cache categories in Site model.
Profiling showed that we were roughly 10% of a request time creating all
the ActiveRecord objects for categories in the `Site` model on a site with 61 categories.
Instead of querying for the categories each time based on which categories the user can see,
we can just preload all of the categories upfront and filter out the
categories that the user can not see.
2021-06-17 13:17:42 +08:00
Martin Brennan 6fe78cd542
FIX: Make sure reset-new for tracked is not limited by per_page count (#13395)
When dismissing new topics for the Tracked filter, the dismiss was
limited to 30 topics which is the default per page count for TopicQuery.
This happened even if you specified which topic IDs you were
selectively dismissing. This PR fixes that bug, and also moves
the per_page_count into a DEFAULT_PER_PAGE_COUNT for the TopicQuery
so it can be stubbed in tests.

Also moves the unused stub_const method into the spec helpers
for cases like this; it is much better to handle this in one place
with an ensure. In a follow up PR I will clean up other specs that
do the same thing and make them use stub_const.
2021-06-17 08:20:09 +10:00
Brandon Gastelo 20dbcbf022
FIX: Sort filelists to ensure consistant asset precompilation hash (#13393)
Dir.glob does not guarantee file order and can change when ran on different machines.
This means that running asset precompilation on the exact same codebase will output
different content hashes.
2021-06-16 11:04:21 -04:00
David Taylor b0416cb1c1
FEATURE: Upload to s3 in parallel to speed up backup restores (#13391)
Uploading lots of small files can be made significantly faster by parallelizing the `s3.put_object` calls. In testing, an UPLOAD_CONCURRENCY of 10 made a large restore 10x faster. An UPLOAD_CONCURRENCY of 20 made the same restore 18x faster.

This commit is careful to parallelize as little as possible, to reduce the chance of concurrency issues. In the worker threads, no database transactions are performed. All modification of shared objects is controlled with a mutex.

Unfortunately we do not have any existing tests for the `ToS3Migration` class. This change has been tested with a large site backup (120k uploads totalling 45GB)
2021-06-16 10:34:39 +01:00
Martin Brennan c659e3e95b
FIX: Make sure topic_user.bookmarked is synced in more places (#13383)
When we call Bookmark.cleanup! we want to make sure that
topic_user.bookmarked is updated for topics linked to the
bookmarks that were deleted. Also when PostDestroyer calls
destroy and recover. We have a job for this already --
SyncTopicUserBookmarked -- so we just utilize that.
2021-06-16 08:30:40 +10:00
Osama Sayegh 503017474c
DEV: Skip CSS watcher when running QUnit tests and expose more Chrome logs (#13390)
There are 2 changes in this PR:

1) Add a new environment variable called `DISCOURSE_SKIP_CSS_WATCHER` to disable our stylesheet watcher, and make the `qunit:test` rake task set this variable on the Unicorn/Rails server it spins up to disable our stylesheet watcher when running the tests because it doesn't really need it.

2) Print more Chrome logs (such as network/security errors) to the console.
2021-06-15 18:27:15 +03:00
Roman Rizzi fa57316a4e
FIX: Validate upload is still valid after calling the "before_upload_creation" event (#13091)
Since we use the event to perform additional validations on the file, we should check if it added any errors to the upload before saving it. This change makes the UploadCreator more consistent since we no longer have to rely on exceptions.
2021-06-15 10:10:03 -03:00
Martin Brennan 7fca7fb7ff
DEV: Add SMTP group ID to EmailLog (#13381)
Adds a new `smtp_group_id` column to `EmailLog` which is filled in if the mail `from_address` matches a group's `email_username`. This is for easier debugging, so we know which emails have been sent via group SMTP.
2021-06-15 11:29:46 +10:00
Alan Guo Xiang Tan 5f8f139ac9 PERF: Remove extra PG query.
In `Theme.list_baked_fields`, common is always included as a target.
2021-06-15 08:51:00 +08:00
Kane York c780ae9d25
FEATURE: Add a messages view for all official warnings of a user (#12659)
Moderators are allowed to see the warnings list, with an access warning.

https://meta.discourse.org/t/why-arent-warnings-easily-accessible-like-suspensions-are/164043
2021-06-14 14:01:17 -07:00
Jarek Radosz 091beaf4a2
DEV: Work around a Docker issue (#13368)
Fixes our backend spec suite in GitHub Actions CI. For more information about the Docker issue see: https://github.com/docker/for-linux/issues/1015

(It's possible that error could also happen in dev/production, though thankfully that hasn't happened yet afaik)
2021-06-14 17:38:36 +02:00
Vinoth Kannan 6abc45e57b
DEV: move `discourse_dev` gem to the core. (#13360)
And get avatar images from `discourse_dev_assets` gem.
2021-06-14 20:34:44 +05:30
Martin Brennan e9dc88a7b6
FIX: Link up reply to post correctly when emailing group (#13339)
When replying to a user_private_message email originating from
a group PM that does _not_ have a reply key (e.g. when replying
directly to the group's SMTP address), we were mistakenly linking
the new post created from the reply to the OP and the user who
created the topic, based on the first IncomingEmail message ID in
the topic, rather than using the correct reply to user and post number
that the user actually replied to.

We now use the In-Reply-To header to look up the corresponding EmailLog
record when the user who replied was sent a user_private_message email,
and use the post from that as the reply_to_user/post.

This also removes superfluous filtering of incoming_email records. After
already filtering by message_id and then addressed_to_user (which only
returns incoming emails where the to, from, or cc address includes any
of the user's emails), we were filtering again but in the ruby code for
the exact same conditions. After removing this all existing tests still
pass.
2021-06-10 15:28:50 +10:00
Penar Musaraj 6f76479054
FEATURE: Add upgrade-insecure-requests to CSP when force_https is enabled (#13348)
If force_https is enabled all resource (including markdown preview and so on) will be accessed using HTTPS

If for any reason you attempt to link to non HTTPS reachable content content may appear broken
2021-06-10 10:53:10 +10:00
Alan Guo Xiang Tan 2a4a20ad67
PERF: Avoid running a pointless PG query when theme has no variables. (#13342)
When `Theme#all_theme_variables` returns an empty array, we were running
a pointless query in `StyleSheet::Manager#uploads_digest`.

`SELECT "sha1" FROM "theme_fields" INNER JOIN "uploads" ON
"uploads"."id" = "theme_fields"."upload_id" WHERE 1=0`
2021-06-10 10:50:17 +10:00