Commit Graph

6033 Commits

Author SHA1 Message Date
Sam Saffron 84f12ac3f7 DEV: No need to carry two brotli implementations around
old version of brotli binary is long gone, require brotli 1.0.0 and up
2019-04-11 16:53:29 +10:00
Arpit Jalan a20f58554b IMPORT: create category definitions in `import:ensure_consistency` task 2019-04-11 12:06:37 +05:30
Sam Saffron baa7a9836c FEATURE: remove "COMPRESS_BROTLI" optional behavior
The compress brotli functionality is no longer optional, this has worked
well for years. The name of the ENV var is also confusing cause it does
not have a `DISCOURSE_` prefix which caused issues with the web upgrader

Brotli support is now unconditionally on
2019-04-11 12:36:18 +10:00
Robin Ward 87de5d9e8f Record the reason that a post was put into the queue 2019-04-10 17:42:49 -04:00
Robin Ward abf4ca9774 FIX: Should be able to recover a user deleted post if raw didn't change 2019-04-10 13:10:40 -04:00
Penar Musaraj eae22548de
Footer navigation for iOS PWAs and DiscourseHub app (#7347) 2019-04-10 12:23:18 -04:00
Vinoth Kannan 8d5c900142 DEV: add unique missing uploads index in post custom fields
https://review.discourse.org/t/feature-send-missing-post-uploads-stat-to-prometheus/2609/6?u=vinothkannans
2019-04-10 18:09:35 +05:30
Bianca Nenciu c523a12187 FIX: Fix assets:precompile rake task. 2019-04-10 14:03:26 +03:00
Tarek Khalil 442fb2facb FEATURE: Remove ignore feature SiteSetting and enable ignore by default (#7349) 2019-04-10 12:54:59 +02:00
Bianca Nenciu a723699519 FIX: Ensure TarReader is closed. 2019-04-10 13:37:04 +03:00
Bianca Nenciu 4555d0c598 FEATURE: Refresh MaxmindDb during assets:precompile. (#7340) 2019-04-10 11:37:29 +02:00
Bianca Nenciu b5008586c5 DEV: Remove HTML parser from Tautologistics. (#7344) 2019-04-10 11:21:22 +02:00
Vinoth Kannan d0fe42e2ef FIX: should look through posts for image markdown
Downloaded onebox images only included in the cooked HTML content. So we have to check 'post.cooked' instead of 'raw'. bfdd0fe64c
2019-04-10 13:52:35 +05:30
Vinoth Kannan 4878ee95e5 DEV: optimize sql query to narrow down the filtering of post with uploads 2019-04-10 12:58:10 +05:30
Guo Xiang Tan bac1bcc79f PERF: Add `index_reply_id_on_post_replies`.
Speeds up the reference posts query in `Email::Sender#send`.
2019-04-10 13:58:29 +08:00
Vinoth Kannan bfdd0fe64c FIX: empty the missing list on each post loop 2019-04-10 11:20:26 +05:30
Vinoth Kannan f5ac01253a FIX: use new changed method name and remove whitespaces 2019-04-09 02:25:26 +05:30
Vinoth Kannan 914ada1c74 DEV: convert scheduled job EnsurePostUploadsExistence into a rake task 2019-04-09 02:07:35 +05:30
Roman Rizzi aec457e09a
DEV: Expose a way to extend a method that returns a list 2019-04-08 14:42:36 -03:00
Robin Ward df1ab9259b FIX: Take action should agree with all pending flags 2019-04-08 12:39:18 -04:00
David Taylor 20cda5d689 DEV: Raise exceptions when jobs fail in test mode
Previously jobs would fail silently in test mode. Now they will raise the exception and cause the relevant test to fail. This identified a few broken tests, which I will fix in a followup commit
2019-04-08 15:59:04 +01:00
David Taylor f524f8f811
Remove Yahoo login support from core and deprecate OpenID2.0 (#7310)
- Plugin developers using OpenID2.0 should migrate to OAuth2 or OIDC. OpenID2.0 APIs will be removed in v2.4.0

- For sites requiring Yahoo login, it can be implemented using the OpenID Connect plugin: https://meta.discourse.org/t/103632

For more information, see https://meta.discourse.org/t/113249
2019-04-08 10:38:25 +01:00
David Taylor 6a05f190c6
PERF: Do not create staged users for most rejected incoming emails (#7301)
Previously we would create users, then destroy them at the end of the job if the post was rejected. Now we do not create users unless required.
2019-04-08 10:36:39 +01:00
Guo Xiang Tan d53cc07de1 Version bump to v2.3.0.beta8. 2019-04-08 12:28:34 +08:00
Robin Ward 8181ad646b FIX: Queued Posts were not saving tags properly 2019-04-05 14:07:47 -04:00
Neil Lalonde 347e3e5a46 Version bump to v2.3.0.beta7 2019-04-05 12:24:14 -04:00
Erick Guan 9dd325f805 FIX: skip some checks for CJK locale in TextSentinel (#7322) 2019-04-05 15:07:49 +02:00
Sam Saffron 0744d2ef73 PERF: speed up suggested unread
Previously we tried using a sub-query which has some terrible performance
implications, by adding 1 extra query we can eliminate the PG issue

A join to user_stats also is prone to the same slowdown
2019-04-05 15:55:29 +11:00
Sam Saffron 5f896ae8f7 PERF: Keep track of when a users first unread is
This optimisation avoids large scans joining the topics table with the
topic_users table.

Previously when a user carried a lot of read state we would have to join
the entire read state with the topics table. This operation would slow down
home page and every topic page. The more read state you accumulated the
larger the impact.

The optimisation helps people who clean up unread, however if you carry
unread from years ago it will only have minimal impact.
2019-04-05 12:44:45 +11:00
Neil Lalonde 83996fc8ea FEATURE: ability to restrict some tags to a category while allowing all others
A new checkbox has been added to the Tags tab of the category settings modal
which is used when some tags and/or tag groups are restricted to the category,
and all other unrestricted tags should also be allowed.
Default is the same as the previous behaviour: only allow the specified set of
tags and tag groups in the category.
2019-04-04 16:40:15 -04:00
Vinoth Kannan 35431a8ddb FIX: set missing count in redis even if zero 2019-04-04 20:05:57 +05:30
Vinoth Kannan df6ef856e6 DEV: save missing s3 uploads count in redis 2019-04-04 19:05:57 +05:30
Maja Komel b0053f3a1c FEATURE: bump onebox version, add styling for new reddit image onebox 2019-04-04 11:24:30 +02:00
Dax74 c5b76cd138
Add Yearly review plugin
Add Yearly review plugin in the official list
2019-04-03 11:30:49 +02:00
Guo Xiang Tan e8a4d72281 FIX: Avoid penalizing long documents too much in search.
This is a follow up to e87ca59401.
2019-04-03 14:09:57 +08:00
Sam Saffron 1c57ae6657 FIX: grant first quote at the date post was created
Previously due to #b2dc65f9534ea date on the quoted_posts table could not
be trusted.

This changes it so we use the date on the actual post as the grant date.

Note: there is an edge case where you create a post and only add a quote
a week later. In this case the badge will not be awarded at the correct
time (it will display it was granted a week ago).
That said this is far more rare than the current situation.
2019-04-03 16:41:52 +11:00
Robin Ward 123c05cb07 FIX: category id wasn't stored when enqueueing a topic 2019-04-02 11:12:17 -04:00
David Taylor ef3dd83367 DEV: Remove trailing blank line 2019-04-02 15:49:23 +01:00
David Taylor 02ed5e78e1 DEV: Use parallel-compatible formatter for logging autospec failures
The old method would cause the parallel processes to overwrite each other. The parallel formatter allows multiple processes to write to the same file.
2019-04-02 15:34:30 +01:00
Robin Ward 6ebadaed2c FIX: Do not allow `invite_only` and `enable_sso` at the same time
This functionality was never supported but before the new review queue
it didn't have any errors. Now the combination of settings is prevented
and existing sites with sso enabled will be migrated to remove invite
only.
2019-04-02 10:26:27 -04:00
Arpit Jalan d68d29f37a FIX: only staff can banner topics 2019-04-02 12:46:55 +05:30
Guo Xiang Tan 6a95d3fded FIX: `UploadRecovery` should look through posts for img src and bbcode. 2019-04-02 11:41:00 +08:00
Guo Xiang Tan 1f9799c979 FIX: Handle more cases in `UploadRecovery`. 2019-04-02 11:29:26 +08:00
Guo Xiang Tan 4b0ac91bfb DEV: Remove duplicated scope.
`.joins(:topic)` will automatically add the `deleted_at IS NULL` scope.
2019-04-02 10:48:17 +08:00
Guo Xiang Tan d8704c11ca PERF: Better use of index when queueing a topci for search reindex.
Also move `Search::INDEX_VERSION` to `SearchIndexer` which is where the
version is actually being used.
2019-04-02 09:53:37 +08:00
David Taylor 25feb287b8 DEV: Add parallel spec support to autospec
Set PARALLEL_SPEC=1 to use this functionality
2019-04-01 11:06:47 -04:00
Régis Hanol 2b721dff57 FIX: whitelist 'feGaussianBlur' and 'filter' svg elements 2019-04-01 12:56:47 +02:00
Guo Xiang Tan e87ca59401 FIX: Relevance search will now consider document length in ranking.
The default ranking options ranks by the number of matches which is
highly problematic when posts are stuffed with a keyword. The ranking
will now be divided by the document length which is a much fairer way to
rank.
2019-04-01 14:37:45 +08:00
Guo Xiang Tan cfd507822f
PERF: Improve quality of `PostSearchData#raw_data`. (#7275)
This commit fixes the follow quality issue with `PostSearchData#raw_data`:

1. URLs are being tokenized and links with similar href and characters
are being duplicated in the raw data.

`Post#cooked`:

```
<p><a href=\"https://meta.discourse.org/some.png\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://meta.discourse.org/some.png</a></p>
```

`PostSearchData#raw_data` Before:

```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png discourse org/some png https://meta.discourse.org/some.png discourse org/some png
```

`PostSearchData#raw_data` After:

```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png meta discourse org
```

2. Ligthbox being included in search pollutes the
`PostSearchData#raw_data` unncessarily.

From 28 March 2018 to 28 March 2019, searches for the term `image` on
`meta.discourse.org` had a click through rate of 2.1%. Non-lightboxed images are not included in indexing for search yet we were indexing content within a lightbox. Also, search for terms like `image` was affected we were using `Pasted image` as the filename for
uploads that were pasted.

`Post#cooked`

```
<p>Let me see how I can fix this image<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://meta.discourse.org/some.png\" title=\"some.png\" rel=\"nofollow noopener\"><img src=\"https://meta.discourse.org/some.png\" width=\"275\" height=\"299\"><div class=\"meta\">\n<svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#far-image\"></use></svg><span class=\"filename\">some.png</span><span class=\"informations\">1750×2000</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#discourse-expand\"></use></svg>\n</div></a></div></p>
```

`PostSearchData#raw_data` Before:

```
This is a test topic 0 Uncategorized Let me see how I can fix this image some.png png https://meta.discourse.org/some.png discourse org/some png some.png png 1750×2000
```

`PostSearchData#raw_data` After:

```
This is a test topic 0 Uncategorized Let me see how I can fix this image
```

In terms of indexing performance, we now have to parse the given HTML
through nokogiri twice. However performance is not a huge worry here since a string length of 194170 takes only 30ms
to scrub plus the indexing takes place in a background job.
2019-04-01 10:14:29 +08:00
Bianca Nenciu 034b8a7ecc FIX: Let users delete topics.
Follow-up to 31053f30de.
2019-03-29 22:00:36 +02:00