Commit Graph

4038 Commits

Author SHA1 Message Date
Guo Xiang Tan 6525613b89 PERF: Use joins for `Post.for_mailing_list` instead of `NOT IN`.
Joining on the topics table and then filtering the topics against a large set of
ids is so much slower than doing a join on a sub query.
2019-04-06 07:58:55 +08:00
Guo Xiang Tan a7a5f90e20 Annotate models. 2019-04-05 17:13:12 +08:00
Sam Saffron 81a1bafe54 PERF: Speed up home page unread+new query
restructure query so it avoids ORs

It appears postgres is picking suboptimal indexes if too many ORs exist
despite how trivial the condition is.

This bypasses conditional in the query and evals them upfront.

On meta for my user this made a 10x perf difference.

This boils down to either having `OR u.admin` or not having `OR u.admin` in
the query.
2019-04-05 15:25:19 +11:00
Sam Saffron 5b837f620b correct unread resetting to handle nulls
Note, to avoid race conditions we are setting last_unread to 10 minutes ago
if there is nothing unread.

This is safer in case of in progress transactions
we don't want to lose unread for any window of time.
2019-04-05 13:42:41 +11:00
Sam Saffron 5f896ae8f7 PERF: Keep track of when a users first unread is
This optimisation avoids large scans joining the topics table with the
topic_users table.

Previously when a user carried a lot of read state we would have to join
the entire read state with the topics table. This operation would slow down
home page and every topic page. The more read state you accumulated the
larger the impact.

The optimisation helps people who clean up unread, however if you carry
unread from years ago it will only have minimal impact.
2019-04-05 12:44:45 +11:00
Robin Ward 8af7237092 FIX: Only perform logic on `create` commits 2019-04-04 17:01:47 -04:00
Robin Ward df48c657fd FIX: Reviewable counts were not updating properly
Sometimes sidekiq is so fast that it starts jobs before transactions
have comitted. This patch moves the message bus stuff until after things
have comitted.
2019-04-04 16:07:34 -04:00
Robin Ward 2055804e95 FIX: The option to delete replies was missing from the new review queue 2019-04-04 15:51:36 -04:00
Guo Xiang Tan 6815f777f9 DEV: Remove unused method. 2019-04-04 14:19:39 +08:00
Vinoth Kannan b3fb0a7039 FEATURE: ensure consistency of post uploads in cooked content 2019-04-04 02:23:28 +05:30
Robin Ward 111a502231 FIX: Deleting Users should work nicely with Reviewable Users
"Rejecting" a user in the queue is equivalent to deleting them, which
would then making it impossible to review rejected users. Now we store
information about the user in the payload so if they are deleted things
still display in the Rejected view.

Secondly, if a user is destroyed outside of the review queue, it will
now automatically "Reject" that queue item.
2019-04-03 16:42:39 -04:00
Robin Ward 828cdf3fea FIX: Protected method called, I'll fire myself now 2019-04-03 16:10:36 -04:00
Robin Ward 82bddcbe51 FIX: Don't create two reviewable scores for a user 2019-04-03 16:03:32 -04:00
Robin Ward c1ea63bdc1 FIX: Reviewables should not be created for users until they are active
Conversely, if a user is deactivated the reviewable should automatically
be rejected.

Before this fix, if a user was not active they'd still show in the
review queue but without an "Approve" button which was confusing.
2019-04-03 15:25:00 -04:00
Arpit Jalan 7b194743d7
FIX: clean up user export csv upload records in scheduled job (#7309) 2019-04-03 13:31:19 +05:30
Sam Saffron b2dc65f953 FIX: on rebakes should not recreate quoted_posts records
Previously every rebake would remove and recreate records in this table
This caused created_at and updated_at to keep changing

Yes, I know the SQL is somewhat complex, but this makes quote extraction
more efficient cause we do everything in 2 round trips.

This also removes some concurrency protection we should no longer need
2019-04-03 16:17:15 +11:00
Guo Xiang Tan 52c4711e80 DEV: Fix method that was incorrectly made private. 2019-04-03 12:37:50 +08:00
Guo Xiang Tan 4037a2fd0b FIX: Restrict scope of old scheme upload migration.
Some sites have external URLs that don't even match `%/uploads/%' and
some sites surprise me with URLs that contains the default path when it
is a site in a multisite cluster. We can't do anything about those.
2019-04-03 11:56:41 +08:00
Guo Xiang Tan feb731bffd FIX: Regenerate optimized images instead of migrating from old scheme.
`OptimizedImage.migrate_to_new_scheme` was optimizing optimized images
which we don't need to do. Regnerating the optimized image is way easier.
2019-04-03 09:45:02 +08:00
Guo Xiang Tan d85240335b Annotate models. 2019-04-02 13:19:14 +08:00
Robin Ward d5a61ab167 FIX: Don't log a second pending action
If two people flagged the same post you'd get a "Pending" history event
logged which looked bad.
2019-04-01 17:07:47 -04:00
Penar Musaraj fdf4145d4b
FEATURE: Delegated authentication via user api keys (#7272) 2019-04-01 13:18:53 -04:00
David Taylor b375dcb14a DEV: Introduce parallel rspec testing
Adds the parallel_tests gem, and redis/postgres configuration for running rspec tests in parallel. To use:

```
rake parallel:rake[db:create]
rake parallel:rake[db:migrate]
rake parallel:spec
```

This brings the test suite from 12m20s to 3m11s on my macOS machine
2019-04-01 11:06:47 -04:00
Régis Hanol a6596662dc FIX: don't enqueue TruncateUserFlagStats job when not needed 2019-04-01 15:53:55 +02:00
Joffrey JAFFEUX 6e0e36f790
fix spec (#7298) 2019-04-01 14:35:09 +02:00
Joffrey JAFFEUX e986e96227
DEV: removes old dashboard (#7295) 2019-04-01 12:39:49 +02:00
Guo Xiang Tan 8794d940d3 Revert `update_columns` -> `update!` when rebaking/post-processing post.
`update!` goes through validation which means old posts that doesn't
adhere to the existing validations will raise an error.
2019-04-01 16:29:00 +08:00
Maja Komel 88128f1ced UX: show which groups are missing permissions for parent category (#7252) 2019-04-01 15:34:52 +11:00
Guo Xiang Tan cfd507822f
PERF: Improve quality of `PostSearchData#raw_data`. (#7275)
This commit fixes the follow quality issue with `PostSearchData#raw_data`:

1. URLs are being tokenized and links with similar href and characters
are being duplicated in the raw data.

`Post#cooked`:

```
<p><a href=\"https://meta.discourse.org/some.png\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://meta.discourse.org/some.png</a></p>
```

`PostSearchData#raw_data` Before:

```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png discourse org/some png https://meta.discourse.org/some.png discourse org/some png
```

`PostSearchData#raw_data` After:

```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png meta discourse org
```

2. Ligthbox being included in search pollutes the
`PostSearchData#raw_data` unncessarily.

From 28 March 2018 to 28 March 2019, searches for the term `image` on
`meta.discourse.org` had a click through rate of 2.1%. Non-lightboxed images are not included in indexing for search yet we were indexing content within a lightbox. Also, search for terms like `image` was affected we were using `Pasted image` as the filename for
uploads that were pasted.

`Post#cooked`

```
<p>Let me see how I can fix this image<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://meta.discourse.org/some.png\" title=\"some.png\" rel=\"nofollow noopener\"><img src=\"https://meta.discourse.org/some.png\" width=\"275\" height=\"299\"><div class=\"meta\">\n<svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#far-image\"></use></svg><span class=\"filename\">some.png</span><span class=\"informations\">1750×2000</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#discourse-expand\"></use></svg>\n</div></a></div></p>
```

`PostSearchData#raw_data` Before:

```
This is a test topic 0 Uncategorized Let me see how I can fix this image some.png png https://meta.discourse.org/some.png discourse org/some png some.png png 1750×2000
```

`PostSearchData#raw_data` After:

```
This is a test topic 0 Uncategorized Let me see how I can fix this image
```

In terms of indexing performance, we now have to parse the given HTML
through nokogiri twice. However performance is not a huge worry here since a string length of 194170 takes only 30ms
to scrub plus the indexing takes place in a background job.
2019-04-01 10:14:29 +08:00
Guo Xiang Tan 7ac76fe935 DEV: Remove warning.
Library has already been loaded in application.rb.
2019-04-01 10:11:08 +08:00
Robin Ward 370355d754 FIX: Allow users with posts to be rejected 2019-03-29 13:53:46 -04:00
Maja Komel 4a3daacb1b FIX: reset embedding settings when no embeddable host, log host changes (#7264) 2019-03-29 17:05:51 +01:00
Maja Komel 9b00ca30ed FIX: check if user is already allowed before adding to topic allowed users (#7242) 2019-03-29 17:03:33 +01:00
Joffrey JAFFEUX 961fb2c70e
DEV: moves each report in its own file (#7277) 2019-03-29 10:54:56 +01:00
Guo Xiang Tan 1b3165d217 Make rubocop happy. 2019-03-29 12:15:39 +08:00
Guo Xiang Tan f458cba4cb FIX: Admin search logs should filter by date instead of timestamp.
The client side filters by date so it is confusion when the data changes as each second passes.
2019-03-29 11:50:25 +08:00
Guo Xiang Tan d8faf5f79e FIX: `SearcLog.term_details` generating incorrect data because of case.
Also match on equality rather than "LIKE ?" which is quite strange.
2019-03-29 11:31:01 +08:00
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Guo Xiang Tan d0a1922ad9 FIX: `Upload.migrate_to_new_scheme` undefined error when external image fails to download. 2019-03-28 16:00:13 +08:00
Tim Lange 12181599db FIX: Staff action records now also accepts action_name as filter (#7256) 2019-03-27 21:29:15 +01:00
Bianca Nenciu a9798f0c47
FEATURE: Add page for all group membership requests. (#6909) 2019-03-27 13:30:59 +02:00
Guo Xiang Tan 161a3c3870 DEV: Don't optimize image when migrating to new scheme.
The image has already been uploaded, the migrate to new scheme job's is
just to correct the scheme and not the content of the upload.
2019-03-26 15:12:10 +08:00
Guo Xiang Tan 0ac6a58b09 FIX: Skip validation when saving in `Upload#migrate_to_new_scheme`. 2019-03-26 14:28:39 +08:00
Bianca Nenciu f072da1bfe
FEATURE: Add every month / 6 months options to email digest. (#7247) 2019-03-25 12:45:01 +02:00
Guo Xiang Tan ac661e856a
FEATURE: Allow categories to be prioritized/deprioritized in search. (#7209) 2019-03-25 10:59:55 +08:00
Guo Xiang Tan 839a54b97b FIX: Destroy `OptimizedImage` record even if `Upload` record is invalid. 2019-03-22 16:47:06 +08:00
Maja Komel 34730a0b16 UX: show if webhook is disabled (#7217)
+ show in staff logs when webhook is created/updated/destroyed
2019-03-21 16:13:09 +01:00
Tarek Khalil 605530a77f FEATURE: Include muted users count within the ignored users report (#7230) 2019-03-21 14:31:45 +01:00
Tim Lange f7b156ffbd UX: Better emoji escaping for topic title (#7218)
* FIX: Fixed failing discourse-prometheus-alert-receiver plugin specs
2019-03-21 09:11:33 +01:00
Rishabh ad6ad3f679 DEV: Remove SiteSetting.s3_force_path_style (#7210)
- s3_force_path_style was added as a Minio specific url scheme but it has never been well supported in our code base.
- Our new migrate_to_s3 rake task does not work reliably with path style urls too
- Minio has also added support for virtual style requests i.e the same scheme as AWS S3/DO Spaces so we can rely on that instead of using path style requests.
- Add migration to drop s3_force_path_style from the site_settings table
2019-03-20 14:58:20 +01:00