Commit Graph

3129 Commits

Author SHA1 Message Date
Guo Xiang Tan b1cc7825c5
DEV: Remove unused fabrication. 2020-07-24 09:54:34 +08:00
jbrw 2aec92d0b4
FEATURE - allow Group Moderators to edit category description (#10292)
Co-authored-by: Alan Guo Xiang Tan <gxtan1990@gmail.com>
2020-07-23 09:50:00 -04:00
Guo Xiang Tan 609ba50fe8
DEV: Add more granularity to `SearchIndexer` versions.
Sometimes, we just want to reindex a specific model and not all the
things.
2020-07-23 14:24:06 +08:00
Sam Saffron 88dcdf776b
FEATURE: add tracked filter to topic lists
This adds a special filter to topic lists that will filter to tracked and
watched categories.

To use it you can visit:

`https://sitename/?filter=tracked`
`https://sitename/unread?filter=tracked`

and so on

Note, we do not include explicitly tracked and watched topics **outside** of
the tracked categories and tags.

We can consider a `filter=all_tracked` to cover this edge case.
2020-07-23 10:30:23 +10:00
David Taylor c09b5807f3
FIX: Include resolved locale in anonymous cache key (#10289)
This only applies when set_locale_from_accept_language_header is enabled
2020-07-22 18:00:07 +01:00
Roman Rizzi 9fc7bd5797
FIX: Reviews that are auto-hidden by a trusted spam flagger should always have enough weight. (#10284) 2020-07-22 11:42:15 -03:00
Martin Brennan 93a8e34f47
FIX: Allow user to recover/delete post if they can review the topic (#10273)
To reproduce the initial issue here:

1. A user makes a post, which discourse-akismet marks as spam (I cheated and called `DiscourseAkismet::PostsBouncer.new.send(:mark_as_spam, post)` for this)
2. The post lands in the review queue
3. The category the topic is in has a `reviewable_by_group_id`
4. A user in that group goes and looks at the Review queue, decides the post is not spam, and clicks Not Spam
5. Weird stuff happens because the `PostDestroyer#recover` method didn't handle this (the user who clicked Not Spam was not the owner of the post and was not a staff member, so the post didn't get un-destroyed and post counts didn't get updated)

Now users who belong to a group who can review a category now have the ability to recover/delete posts fully.
2020-07-22 11:57:16 +10:00
David Taylor ec4024fe6d
FIX: Keep by_users check in S3 inventory
Partial revert of 8515d8fa - the by_users check is ensuring we don't raise errors for fixtures
2020-07-21 17:19:56 +01:00
Vinoth Kannan ef37460c93 FIX: delete synonyms in topics if target tag is already added.
Currently, while adding a synonym tag if both target and synonym tags are already available in a topic then it's returning an error.
2020-07-21 21:02:01 +05:30
Joffrey JAFFEUX 936a40adcf
linting (#10276) 2020-07-21 08:39:14 +02:00
Guo Xiang Tan 94fced2133
FIX: Handle PG readonly mode in `Auth::DefaultCurrentUserProvider`.
Avoid writing to the DB when PG is in readonly mode.
2020-07-21 13:44:05 +08:00
Martin Brennan 41b43a2a25
FEATURE: Add "delete on owner reply" bookmark functionality (#10231)
This adds an option to "delete on owner reply" to bookmarks. If you select this option in the modal, then reply to the topic the bookmark is in, the bookmark will be deleted on reply.

This PR also changes the checkboxes for these additional bookmark options to an Integer column in the DB with a combobox to select the option you want.

The use cases are:

* Sometimes I will bookmark the topics to read it later. In this case we definitely don’t need to keep the bookmark after I replied to it.
* Sometimes I will read the topic in mobile and I will prefer to reply in PC later. Or I may have to do some research before reply. So I will bookmark it for reply later.
2020-07-21 10:00:39 +10:00
Blake Erickson 690f17bcbe
FEATURE: Allow List for PMs (#10270)
* FEATURE: Allow List for PMs

This feature adds a new user setting that is disabled by default that
allows them to specify a list of users that are allowed to send them
private messages. This way they don't have to maintain a large list of
users they don't want to here from and instead just list the people they
know they do want. Staff will still always be able to send messages to
the user.

* Update PR based on feedback
2020-07-20 15:23:49 -06:00
jbrw 7ab5658462
FEATURE: Allow group moderators to add/remove staff notes (#10252)
* FEATURE: Allow group moderators to add/remove staff notes
2020-07-20 15:53:47 -04:00
David Taylor 5f3dfce4eb
FIX: Listing topics with muted mixed-case tags (#10268)
When visiting a tag page directly, we should display all topics, even if that tag is muted. This was not working for mixed-case tags.
2020-07-20 11:01:29 +01:00
Daniel Waterworth 9313706649 DEV: Enable preserve_email_structure_when_styling by default
In 1bd8a075, a hidden site setting was added that causes Email::Styles
to treat its input as a complete document in all cases.

This commit enables that setting by default.

Some tests were removed that were broken by this change. They tested the
behaviour of applying email styles to empty strings. They weren't useful
because:

 * Sending empty email is not something we ever intend to do,
 * They were testing incidental behaviour - there are lots of
   valid ways to process the empty string,
 * Their intent wasn't clear from their descriptions,
2020-07-20 10:21:32 +01:00
Robin Ward 8e3f667d7c FIX: Show background images for both slug formats
It seems there was a discrepancy in that background images were attached
to the full slug category class: `category-:slug-:id` and our body class
only had `category-:slug`.

This fix adds support for both formats.
2020-07-17 13:42:30 -04:00
David Taylor fab8b8649e
PERF: Combine avatar_lookup and primary_group_lookup into user_lookup (#10253)
These two classes were running very similar queries, which could be expensive on large topics
2020-07-17 10:48:08 +01:00
Guo Xiang Tan ff7678e210
FIX: Reindex posts when `Topic#title` or `Category#name` changes. 2020-07-17 11:12:31 +08:00
Mark VanLandingham 62d5a9690f
FIX: Remove user_deleted when staff recovers post (#10245) 2020-07-16 09:15:01 -05:00
Vinoth Kannan 3252cb847c FIX: : trigger `user_updated` event only if email changed after user creation.
Follow-up to 1460d7957c
2020-07-16 18:21:30 +05:30
Guo Xiang Tan af87911178
FIX: `in:title` search should only search through topic first posts. 2020-07-16 12:21:19 +08:00
Guo Xiang Tan 8ceb7f490f
DEV: Ignore order of categories in search spec. 2020-07-16 09:29:23 +08:00
Guo Xiang Tan 6385fbbfbf
FIX: Ignore document length in search when ranking by relevance.
Considering document length in search introduced too much variance in
our search results such that it makes certain searches better but at the
same time made certain searches worst. Instead, we want to have a more
determistic way of ranking search so that it is easier to reason about
why a post is rank higher in search than another.

The long term plan to tackle repeated terms is to restrict the number of
positions for a given lexeme in our search index.
2020-07-15 13:43:14 +08:00
Guo Xiang Tan 5bf0a0893b
FIX: Search by relevance may return incorrect post number.
Follow up to d8c796bc4.

Note that his change increases query time by around 40% in the following
benchmark against `dev.discourse.org` but this is a tradeoff that has to be taken so that relevance
search is accurate.

```
require 'benchmark/ips'

Benchmark.ips do |x|
  x.config(time: 10, warmup: 2)

  x.report("current aggregate search query") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT topics.id, min(posts.post_number) post_number FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted)) GROUP BY topics.id ORDER BY MAX((
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
    ) DESC, topics.bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.report("current aggregate search query with proper ranking") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT subquery.topic_id id, (ARRAY_AGG(subquery.post_number ORDER BY rank DESC, bumped_at DESC))[1] post_number, MAX(subquery.rank) rank, MAX(subquery.bumped_at) bumped_at FROM (SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id", (
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
     rank, topics.bumped_at bumped_at FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted))) subquery GROUP BY subquery.topic_id ORDER BY rank DESC, bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.compare!
end
```

```
Warming up --------------------------------------
current aggregate search query
                         1.000  i/100ms
current aggregate search query with proper ranking
                         1.000  i/100ms
Calculating -------------------------------------
current aggregate search query
                         18.040  (± 0.0%) i/s -    181.000  in  10.035241s
current aggregate search query with proper ranking
                         12.992  (± 0.0%) i/s -    130.000  in  10.007214s

Comparison:
current aggregate search query:       18.0 i/s
current aggregate search query with proper ranking:       13.0 i/s - 1.39x  (± 0.00) slower
```
2020-07-15 11:45:56 +08:00
jbrw 06073fe8c6
FEATURE: Allow group moderators to close/archive topics
* FEATURE: Allow group moderators to close/archive topics
2020-07-14 12:36:19 -04:00
Guo Xiang Tan 94a2a70462
DEV: Use a longer TTL for pg readonly mode. 2020-07-14 16:15:58 +08:00
Guo Xiang Tan 5c230266d3
FIX: Inject extra lexemes for host lexeme.
```
discourse_development=# SELECT alias, lexemes FROM TS_DEBUG('www.discourse.org');
 alias |       lexemes
-------+---------------------
 host  | {www.discourse.org}

discourse_development=# SELECT TO_TSVECTOR('www.discourse.org');
      to_tsvector
-----------------------
 'www.discourse.org':1
```

Given the above lexeme, we will inject additional lexeme by splitting
the host on `.`. The actual tsvector stored will look something like

```
               tsvector
---------------------------------------
 'discourse':1 'discourse.org':1 'org':1 'www':1 'www.discourse.org':1
```
2020-07-14 15:32:40 +08:00
Guo Xiang Tan 5c31216aea
FIX: Search for whole URLs wasn't working. 2020-07-14 15:31:48 +08:00
Guo Xiang Tan d8c796bc44
FIX: Ensure that aggregating search shows the post with the higest rank.
Previously, we would only take either the `MIN` or `MAX` for
`post_number` during aggregation meaning that the ranking is not
considered.

```
require 'benchmark/ips'

Benchmark.ips do |x|
  x.config(time: 10, warmup: 2)

  x.report("current aggregate search query") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT topics.id, min(posts.post_number) post_number FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted)) GROUP BY topics.id ORDER BY MAX((
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
    ) DESC, topics.bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.report("current aggregate search query with proper ranking") do
    DB.exec <<~SQL
    SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id" FROM "posts" JOIN (SELECT *, row_number() over() row_number FROM (SELECT subquery.topic_id id, (ARRAY_AGG(subquery.post_number))[1] post_number, MAX(subquery.rank) rank, MAX(subquery.bumped_at) bumped_at FROM (SELECT "posts"."id", "posts"."user_id", "posts"."topic_id", "posts"."post_number", "posts"."raw", "posts"."cooked", "posts"."created_at", "posts"."updated_at", "posts"."reply_to_post_number", "posts"."reply_count", "posts"."quote_count", "posts"."deleted_at", "posts"."off_topic_count", "posts"."like_count", "posts"."incoming_link_count", "posts"."bookmark_count", "posts"."score", "posts"."reads", "posts"."post_type", "posts"."sort_order", "posts"."last_editor_id", "posts"."hidden", "posts"."hidden_reason_id", "posts"."notify_moderators_count", "posts"."spam_count", "posts"."illegal_count", "posts"."inappropriate_count", "posts"."last_version_at", "posts"."user_deleted", "posts"."reply_to_user_id", "posts"."percent_rank", "posts"."notify_user_count", "posts"."like_score", "posts"."deleted_by_id", "posts"."edit_reason", "posts"."word_count", "posts"."version", "posts"."cook_method", "posts"."wiki", "posts"."baked_at", "posts"."baked_version", "posts"."hidden_at", "posts"."self_edits", "posts"."reply_quoted", "posts"."via_email", "posts"."raw_email", "posts"."public_version", "posts"."action_code", "posts"."locked_by_id", "posts"."image_upload_id", (
      TS_RANK_CD(
        post_search_data.search_data,
        TO_TSQUERY('english', '''postgres'':*ABCD'),
        1|32
      ) *
      (
        CASE categories.search_priority
        WHEN 2
        THEN 0.6
        WHEN 3
        THEN 0.8
        WHEN 4
        THEN 1.2
        WHEN 5
        THEN 1.4
        ELSE
          CASE WHEN topics.closed
          THEN 0.9
          ELSE 1
          END
        END
      )
    )
     rank, topics.bumped_at bumped_at FROM "posts" INNER JOIN "post_search_data" ON "post_search_data"."post_id" = "posts"."id" INNER JOIN "topics" ON "topics"."id" = "posts"."topic_id" AND ("topics"."deleted_at" IS NULL) LEFT JOIN categories ON categories.id = topics.category_id WHERE ("posts"."deleted_at" IS NULL) AND "posts"."post_type" IN (1, 2, 3, 4) AND (topics.visible) AND (topics.archetype <> 'private_message') AND (post_search_data.search_data @@ TO_TSQUERY('english', '''postgres'':*ABCD')) AND (categories.id NOT IN (
      SELECT categories.id WHERE categories.search_priority = 1
    )
    ) AND ((categories.id IS NULL) OR (NOT categories.read_restricted))) subquery GROUP BY subquery.topic_id ORDER BY rank DESC, bumped_at DESC LIMIT 51 OFFSET 0) xxx) x ON x.id = posts.topic_id AND x.post_number = posts.post_number WHERE ("posts"."deleted_at" IS NULL) ORDER BY row_number;
    SQL
  end

  x.compare!
end
```

```
Warming up --------------------------------------
current aggregate search query
                         1.000  i/100ms
current aggregate search query with proper ranking
                         1.000  i/100ms
Calculating -------------------------------------
current aggregate search query
                         17.726  (± 0.0%) i/s -    178.000  in  10.045107s
current aggregate search query with proper ranking
                         17.802  (± 0.0%) i/s -    178.000  in  10.002230s

Comparison:
current aggregate search query with proper ranking:       17.8 i/s
current aggregate search query:       17.7 i/s - 1.00x  (± 0.00) slower
```
2020-07-14 13:39:13 +08:00
Guo Xiang Tan 4009c9f711
DEV: Fix search specs to take note of order in assertions.
`contain_exactly` does not care about the order which isn't what we
want.
2020-07-14 13:37:44 +08:00
Guo Xiang Tan ce39733b1a
FIX: Incorrect search blurb when advanced search filters are used take2
Also remove include_blurbs attribute which isn't used.
2020-07-14 11:50:40 +08:00
Kane York 8ddd45d524
PERF: topic_view participant post count: don't send back ID list (#10210)
On large topics, the cost of sending the entire post ID list back over to the database is signficant. Just have the DB recalculate the list of visible posts instead.
2020-07-13 18:42:09 -07:00
Robin Ward 7045a2a87c FIX: Don't strip `noopener` from oneboxes 2020-07-13 16:54:42 -04:00
Dan Ungureanu c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Robin Ward b1c6ff9e1c FIX: Test output related to `Discourse::VERSION`
It's a little awkward to test constants by re-assigning them so
I've added a new parameter to `Discourse.find_compatible_resource`
which can be used by tests.
2020-07-09 14:57:27 -04:00
Martin Brennan e0713455ca
PERF: Load topic bookmarks for the user in user_post_bookmarks (#10197)
Instead of loading all of the user bookmarks using all the post IDs in a topic, load all the bookmarks for a user using the topic ID. This eliminates a costly WHERE ID IN query.
2020-07-09 15:46:52 +10:00
Bianca Nenciu bd842cd2b0
FEATURE: Parse images in email signatures (#10137)
* FEATURE: Parse images in email signatures

* DEV: Fix tests

* Code review
2020-07-08 15:50:30 +10:00
Guo Xiang Tan 0c742dd022
DEV: Simple formatting fix. 2020-07-07 15:46:14 +08:00
Vinoth Kannan f3f30edf3f SPEC: use post number to create canoncial path in mega topics.
6d17765924
2020-07-07 12:20:31 +05:30
Jeff Wong 339549d14a
Support plugin and Theme compatibility version manifests (#9995)
Adds a new rake task `plugin:checkout_compatible_all` and
`plugin:checkout_compatible[plugin-name]` that check out compatible plugin
versions.

Supports a .discourse-compatibility file in the root of plugins and themes that
list out a plugin's compatibility with certain discourse versions:

eg: .discourse-compatibility
```
2.5.0.beta6: some-git-hash
2.4.4.beta4: some-git-tag
2.2.0: git-reference
```

This ensures older Discourse installs are able to find and install older
versions of plugins without intervention, through the manifest only.

It iterates through the versions in descending order. If the current Discourse
version matches an item in the manifest, it checks out the listed plugin target.
If the Discourse version is greater than an item in the manifest, it checks out
the next highest version listed in the manifest.

If no versions match, it makes no change.
2020-07-06 14:48:00 -07:00
David Taylor da0fc0a9d3
DEV: Cleanup PostActionType and ReviewableScore changes in tests
Followup to 2df388ffd7
2020-07-06 17:12:23 +01:00
David Taylor 7f2b5a446a
PERF: Remove post_upload recovery in daily EnsureS3UploadsExistence job (#10173)
This is a very expensive process, and it should only be required in exceptional circumstances. It is possible to run a similar recovery using `rake uploads:recover` (5284d41a8e/lib/upload_recovery.rb (L135-L184))
2020-07-06 16:26:40 +01:00
David Taylor 977766e7a8
FEATURE: sso_overrides_(email|username|name) for all auth methods
These settings previously applied only to discourse-sso. Now they work for all external authentication methods.
2020-07-06 10:18:45 +01:00
Vinoth Kannan 06d426bd87 FIX: skip hidden posts while generating canonical url.
Previously, while generating the topic page's canoncial url we used the current post number. It will create invalid canonical path if the topic has whsiper posts. Now we only taking the visible posts for current page index calculation.
2020-07-05 14:04:31 +05:30
Roman Rizzi 2df388ffd7
DEV: Plugins can extend ReviewableScore types. (#10156) 2020-07-02 11:47:43 -03:00
David Taylor 0edffcc47d
FIX: Correct version comparison logic when comparing stable to beta (#10135)
* FIX: Correct version comparison logic when comparing stable to beta

For example, version 1.3.0 should be considered higher than 1.3.0.beta3. So `Discourse.has_needed_version?('1.3.0', '1.3.0.beta3')` should return true

* Switch to use Gem::Version to compare versions
2020-06-29 17:52:33 +10:00
Régis Hanol 7109d94ee7 FIX: properly invalidate inline oneboxes when rebaking
When rebaking a post we were invalidating _regular_ oneboxes but not inline oneboxes.

DEV: also renamed 'InlineOneboxer.purge' to 'InlineOneboxer.invalidate' to keep
the API consistent with 'Oneboxer.invalidate'
2020-06-24 11:54:54 +02:00
Régis Hanol 91c89df68a FIX: onebox local topic when using slug-less URL
When linking to a topic in the same Discourse, we try to onebox the link to show the title
and other various information depending on whether it's a "standard" or "inline" onebox.

However, we were not properly detecting links to topics that had no slugs (eg. https://meta.discourse.org/t/1234).
2020-06-23 17:18:38 +02:00
Martin Brennan e92909aa77
FIX: Use ActionDispatch::Http::ContentDisposition for uploads content-disposition (#10108)
See https://meta.discourse.org/t/broken-pipe-error-when-uploading-to-a-s3-clone-a-pdf-with-a-name-containing-e-i-etc/155414

When setting content-disposition for attachment, use the ContentDisposition class to format it. This handles filenames with weird characters and localization (accented characters) correctly.
2020-06-23 17:10:56 +10:00
Guo Xiang Tan add2a9411e
DEV: Remove specs that are no longer relevant. 2020-06-23 12:09:04 +08:00
Guo Xiang Tan 25db91e351
Revert "FIX: These tests are broken"
This reverts commit b1114b9a20.
2020-06-23 08:41:10 +08:00
Robin Ward b1114b9a20 FIX: These tests are broken 2020-06-22 16:30:46 -04:00
Bianca Nenciu 68f767a557
FEATURE: Check if selectable avatars exist before enabling them (#10032) 2020-06-22 16:58:26 +03:00
Daniel Waterworth 9cf77372a2 FIX: Guardian#can_remove_allowed_users? shouldn't break for ownerless topics
A topic can outlive its original author. TopicGuardian should still work
in this situation.
2020-06-19 10:35:52 +01:00
Bianca Nenciu db1bebddce
FIX: Hide the post history for TL4 (#10065) 2020-06-18 13:27:51 +03:00
Dan Ungureanu d21a08c284
DEV: Deprecate Category#url_with_id in favor of Category#url (#9972) 2020-06-18 11:32:14 +03:00
Dan Ungureanu 7ed7b1ef64
DEV: Add test (#10064)
Follow-up-to 84dfaad137
2020-06-17 21:41:16 +03:00
Robin Ward e8756e1a95 FIX: Muted/Ignore should prevent PMs regardless of case sensitivity 2020-06-17 14:26:14 -04:00
Guo Xiang Tan 0ff86b00cb
DEV: Upgrade Redis to 4.2.1. 2020-06-15 10:05:22 +08:00
Joffrey JAFFEUX 4b793a1072
FIX: allows PM owner to remove any user if >= TL2 (#10036) 2020-06-12 12:54:28 +02:00
Guo Xiang Tan 3aab98781a
Fix tests. 2020-06-12 10:22:22 +08:00
Dan Ungureanu b7e70850e4
FIX: Allow users to add emails which were deleted before 2020-06-11 14:54:11 +03:00
Dan Ungureanu 5bfe1ee4f1
FEATURE: Improve UX support for multiple email addresses (#9691) 2020-06-10 19:11:49 +03:00
Guo Xiang Tan a3dfd553a1
Revert "Bump redis to 4.2.0."
This reverts commit 98bc28cea2.
2020-06-10 14:52:05 +08:00
Guo Xiang Tan 98bc28cea2
Bump redis to 4.2.0. 2020-06-10 14:28:56 +08:00
Jeff Wong 70a88111dd
FIX: prevent re-flagging when we have reviewed flags before (#10010)
FIX: prevent re-flagging when we have reviewed flags before

Fixes an edge case where a review can be reflagged when:
User flags as inappropriate.
Moderator rejects the flag.
Another user re-flags the post as spam.

Before, anyone was able to re-flag as inappropriate despite it being flagged
previously. With this, users are unable to re-flag for the same reason
regardless of reviewable status.
2020-06-09 15:26:10 -07:00
Penar Musaraj 2d880b42a3
UX: Add simple-list setting type (#9970) 2020-06-04 10:44:54 -04:00
Guo Xiang Tan 2188ccccd5 DEV: Remove `initiating_user` keyword arg from `EmailUpdater`.
The guardian contains the acting user.
2020-06-04 13:21:56 +08:00
Blake Erickson a89574ccb9 FIX: Inline error when converting html to markdown
Looks like some html elements like `aside` and `section` will throw an error
when checking if they are inline or not. The commit simply handles

```
Job exception: undefined method `inline?' for nil:NilClass
```

and adds a test for it.
2020-06-03 15:59:19 -06:00
Guo Xiang Tan 04a291ceea
DEV: Fix race conditions due to directory removal for uploads spec. 2020-06-03 12:28:39 +08:00
Sam Saffron 57a3d4e0d2
FEATURE: whitelist theme repo mode (experimental)
In some restricted setups all JS payloads need tight control.

This setting bans admins from making changes to JS on the site and
requires all themes be whitelisted to be used.

There are edge cases we still need to work through in this mode
hence this is still not supported in production and experimental.

Use an example like this to enable:

`DISCOURSE_WHITELISTED_THEME_REPOS="https://repo.com/repo.git,https://repo.com/repo2.git"`

By default this feature is not enabled and no changes are made.

One exception is that default theme id was missing a security check
this was added for correctness.
2020-06-03 13:19:57 +10:00
Guo Xiang Tan 389bdcf5ab
DEV: Fix implementation for `DiscourseRedis#exists`. 2020-06-01 13:11:32 +08:00
Guo Xiang Tan df62407f35
DEV: Implement multiple keys support for `DiscourseRedis#exists`. 2020-06-01 11:20:26 +08:00
David Taylor e159fb06df
FEATURE: Download remote images even for old posts (#9925)
When a post is rebaked, the admins expect it to work the same regardless of how old the post is.
2020-05-29 17:13:55 +01:00
David Taylor 28f46c171c
FIX: Pull hotlinked images even when edited by system users (#9890)
Previously the pull hotlinked images job was skipped after system edits. This ensured that we never had an infinite loop of system-edit/pull-hotlinked/system-edit/pull-hotlinked etc.

A side effect was that edits made by system for any other reason (e.g. API, removing full quotes) would prevent pulling hotlinked images. This commit removes the system edit check, and replaces it with another method to avoid an infinite job scheduling loop.
2020-05-29 13:07:47 +01:00
Vinoth Kannan ce1491e830
UX: remove `in:unpinned` filter from advanced search page. (#9911) 2020-05-29 00:47:28 +05:30
Joffrey JAFFEUX 77801aa9be
FIX: allows to have custom emoji translation without static file (#9893) 2020-05-27 20:11:52 +02:00
Penar Musaraj b1c726be0d
Remove support for FontAwesome 4.7 icon names (#9871) 2020-05-26 14:53:32 -04:00
Gerhard Schlager 69ee94b526 FIX: XML files could be detected as SVG files 2020-05-26 18:18:20 +02:00
Vinoth Kannan 8e56197728
UX: use "icon-picker" & "image-uploader" fields to set group flair. (#9779) 2020-05-25 11:08:47 +05:30
Michael Brown d9a02d1336
Revert "Revert "Merge branch 'master' of https://github.com/discourse/discourse""
This reverts commit 20780a1eee.

* SECURITY: re-adds accidentally reverted commit:
  03d26cd6: ensure embed_url contains valid http(s) uri
* when the merge commit e62a85cf was reverted, git chose the 2660c2e2 parent to land on
  instead of the 03d26cd6 parent (which contains security fixes)
2020-05-23 00:56:13 -04:00
Jeff Atwood 20780a1eee Revert "Merge branch 'master' of https://github.com/discourse/discourse"
This reverts commit e62a85cf6f, reversing
changes made to 2660c2e21d.
2020-05-22 20:25:56 -07:00
Guo Xiang Tan f4b82f1dc0
DEV: Fix randomly failing spec.
If a user is created with an id of 999, a `upload.user_id ==
user_avatar.user_id` will return true. This fix increases the id of the
upload to something that we will not hit in the foreseeable future.
2020-05-21 11:41:07 +08:00
Martin Brennan df68d11c38
FEATURE: Add topic excerpt max length site setting (#9847)
Adds a new topic_excerpt_maxlength site setting.

* When topic excerpt is requested for a post, use the new topic_excerpt_maxlength site setting to limit the size of the excerpt
* Remove code for getting/setting Post.excerpt_size as it is not used anywhere
2020-05-21 13:19:48 +10:00
Osama Sayegh 02f44def56
FIX: Don't blow up when trying to parse invalid or non-ASCII URLs (#9838)
* FIX: Don't blow up when trying to parseinvalid or non-ASCII URLs

Follow-up to 72f139191e
2020-05-20 12:46:27 +03:00
Martin Brennan 72f139191e
FIX: S3 store has_been_uploaded? was not taking into account s3 bucket path (#9810)
In some cases, between Discourse forums the hostname of a URL could match if they are hosting S3 files on the same bucket but the S3 bucket path might not. So e.g. https://testbucket.somesite.com/testpath/some/file/url.png vs https://testbucket.somesite.com/prodpath/some/file/url.png. So has_been_uploaded? was returning true for the second URL, even though it may have been uploaded on a different Discourse forum.

This is a very rare case but must be accounted for, because this impacts UrlHelper.is_local which mistakenly thinks the file has already been downloaded and thus allows the URL to be cooked, where we want to return the full URL to be downloaded using PullHotlinkedImages.
2020-05-20 10:40:38 +10:00
Guo Xiang Tan 96c02caba7
DEV: Change use of Redis `flushall` to `flushdb`.
FLUSHALL removes all keys from all databases. Instead we only want to
remove keys from the current Redis database.
2020-05-19 10:20:00 +08:00
Guo Xiang Tan d01c336899
DEV: Clean up some Redis leaks in test env. 2020-05-18 17:27:37 +08:00
David Taylor 461b4e5cc6
DEV: Add framework for filtered plugin registers (#9763)
* DEV: Add framework for filtered plugin registers

Plugins often need to add values to a list, and we need to filter those lists at runtime to ignore values from disabled plugins. This commit provides a re-usable way to do that, which should make it easier to add new registers in future, and also reduce repeated code.

Follow-up commits will migrate existing registers to use this new system

* DEV: Migrate user and group custom field APIs to plugin registry

This gives us a consistent system for checking plugin enabled state, so we are repeating less logic. API changes are backwards compatible
2020-05-15 14:04:38 +01:00
David Taylor 0495a748d0
FIX: Use CDN URLs for topic thumbnails 2020-05-15 13:35:20 +01:00
Guo Xiang Tan 9ab5801a1b
FIX: Allow post migrations using `#change` to carry out unsafe migration 2020-05-15 14:23:27 +08:00
Blake Erickson 1a2b9435b0
DEV: Standardize table sorting verbiage (#9757)
* DEV: Standardize table sorting verbiage

This commit creates a common component that tables can use to make their
headers sortable. This commit also standardizes on using `desc` as the
default and passing in the `asc=true` flag to adjust the sorting
direction.

* Add deprecation warnings

Adds deprecation warnings if using previous params and maintains
backwards compatibility. Set the default sort value for group members to
be asc.

* switch group requests to use common table-header-toggle

* update fixture
2020-05-14 20:10:59 -06:00
Kane York 869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Mark VanLandingham a047004c9a
FIX: Specific email error for replies to digest emails (#9770) 2020-05-14 09:04:58 -05:00
Dan Ungureanu 3ed6a0e904
FIX: Detect Wayback Machine using user agent (#9777) 2020-05-14 21:10:07 +10:00
David Taylor d4717f5d1e
DEV: Reduce repeated code in discourse_plugin_registry (#9705)
Use a helper method to simplify creating a new register. Previously this would require creating lots of different methods manually, and adding every register to the clear/reset functions
2020-05-13 12:25:34 +01:00
David Taylor 6230f5c554
FEATURE: Allow parameter authentication for UserApiKeys (#9742)
This refactors default_current_user_provider in a few ways:
- Introduce a generic `api_parameter_allowed?` method which checks for whitelisted routes/formats
- Only read the api_key parameter on allowed routes. It is now completely ignored on other routes (previously it would raise a 403)
- Start reading user_api_key parameter on allowed routes
- Refactor tests as end-end integration tests

A plugin API for PARAMETER_API_PATTERNS will be added soon
2020-05-12 13:35:36 +01:00
Guo Xiang Tan 3c423faafd
DEV: Avoid repeating draft key logic in multiple places. 2020-05-12 11:14:03 +08:00
David Taylor 5fc51ed49c
DEV: Remove unused DiscoursePlugin class (#9715) 2020-05-11 15:46:54 +01:00