Doing .pluck(:column).first is a very common pattern in Discourse and in
most cases, a limit cause isn't being added. Instead of adding a limit
clause to all these callsites, this commit adds two new methods to
ActiveRecord::Relation:
pluck_first, equivalent to limit(1).pluck(*columns).first
and pluck_first! which, like other finder methods, raises an exception
when no record is found
- Increase size of the reviewable's conversation excerpt to prevent truncation of the new copy
- Remove the `domain` parameter from the `flag_linked_posts_as_spam` method in the user model since it is no longer needed
- Remove the `domain` interpolation variable from all translation files
- Add "All posts from this user that include links should be reviewed." to server.en.yml for added clarity on why the posts entered the queue
Trying to truncate encoded slugs will mean that we have to keep the URL
valid, which can be tricky as you have to be aware of multibyte
characters.
Since we already have upper bounds for the title, the slug won't grow
for more than title*6 in the worst case. The slug column in the topic
table can store that just fine.
Added a test to ensure that a generated slug is a valid URL too, so we
don't introduce regressions in the future.
Under exceptional situations the automatic draft feature can fail.
This new **hidden, default off** site setting
`backup_drafts_to_pm_length` will automatically backup any draft that is
saved by the system to a dedicated PM (originating from self)
The body of that PM will contain the text of the reply.
We can enable this feature strategically on sites exhibiting issues to
diagnose issues with the draft system and offer a recourse to users who
appear to lose drafts. We automatically checkpoint these drafts every 5
minutes forcing a new revision each 5 minutes so you can revert to old
content.
Longer term we are considering automatically enabling this kind of feature
for extremely long drafts where the risk is really high one could lose
days of writing.
This reverts commit ab74a50d85.
We really want to upgrade redis, but discovered some edge cases
around failover we need to test.
Holding off on the upgrade till a bit more testing happens
This feature amends it so instead of using one challenge and honeypot
statically per site we have a rotating honeypot and challenge value which
changes every hour.
This means you must grab a fresh copy of honeypot and challenge value once
an hour or account registration will be rejected.
We also now cycle the value of the challenge when after successful account
registration forcing an extra call to hp.json between account registrations
Client has been made aware of these changes.
Additionally this contains a JavaScript workaround for:
https://bugs.chromium.org/p/chromium/issues/detail?id=987293
This is client side code that is specific to Chrome user agent and swaps
a PASSWORD type honeypot with a TEXT type honeypot.
When an admin changes the site setting slug_generation_method to
encoded, we weren't really encoding the slug, but just allowing non-ascii
characters in the slug (unicode).
That brings problems when a user posts a link to topic without the slug, as
our topic controller tries to redirect the user to the correct URL that contains
the slug with unicode characters. Having unicode in the Location header in a
response is a RFC violation and some browsers end up in a redirection loop.
Bug report: https://meta.discourse.org/t/-/125371?u=falco
This commit also checks if a site uses encoded slugs and clear all saved slugs
in the db so they can be regenerated using an onceoff job.
Our instance used for template rendering needs a lock to ensure there is
no race condition where rendering happens on 2 threads at the same time.
This can lead to local poisoning which can cause unexpected results in
emails
* [WIP] - default turbo spec env to test
* FEATURE: support for --fast-fail in bin/turbo_rspec
* fast-fail -> fail_fast to match rspec
* Moved thread killing outside of fail-fast check
* Removed failure_count incrementation from fast_fail_met
If the setting is turned on, then the user will receive information
about the subject: if it was deleted or requires some special access to
a group (only if the group is public). Otherwise, the user will receive
a generic #404 error message. For now, this change affects only the
topics and categories controller.
This commit also tries to refactor some of the code related to error
handling. To make error pages more consistent (design-wise), the actual
error page will be rendered server-side.
Using popups is becoming increasingly rare. Full page redirects are already used on mobile, and for some providers. This commit removes all logic related to popup authentication, leaving only the full page redirect method.
For more info, see https://meta.discourse.org/t/do-we-need-popups-for-login/127988
Currently, if you try to run `./bin/turbo_rspec` you will got that error `There are pending migrations, run rake parallel:migrate`
Reason for that is that command is running in `development` mode which includes plugins migration files in ActiveRecord::Migrator.migrations_paths:
```
["db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-details/db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-details/db/post_migrate",
"/home/lis2/projects/discourse/plugins/discourse-local-dates/db/migrate",
"/home/lis2/projects/discourse/plugins/discourse-local-dates/db/post_migrate",
...
]
```
A workaround solution would be to run the command with the TEST environment like `RAILS_ENV=test ./bin/turbo_rspec`
I want to propose in this PR to override migration_paths to check only Discourse core migrations.
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)
This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes
1. We attempt to localize a string on each site, ensuring we warmup
the i18n
2. We preload all our templates (compiling .erb to class)
3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
Previous to this fix we were leaking methods on the internal action view
template class per render.
This caused email generation to be very low and a steady memory leak in the
application in sidekiq when sending out emails
The behavior change is new to Rails 6 so this fix does not need to be
backported into stable.
It was possible to add a category to more than one default group, e.g. "default categories muted" and "default categories watching first post".
The bug was caused by category validations inadvertently comparing strings and numbers.
Overwriting the same file with 'convert' is not always working as expected.
Adding a temporary file as the destination of the downsize makes this operation much more reliable.
Also switched to using (the more aggressive) 50% resize instead of halving the number of pixels.
* FEATURE: Adds an extra protection layer when decompressing files.
* Rename exporter/importer to zip importer. Update old locale
* Added a new composite class to decompress a file with multiple strategies
* Set max file size inside a site setting
* Ensure that file is deleted after compression
* Sanitize path and files before compressing/decompressing
Threadsafety
Since we use the same redis connection in multiple threads, a rogue
transaction in another thread can trample the connection state
(watched keys) that we need to acquire and release the lock properly.
This is fixed by preventing other threads from using the connection
when we are performing these actions.
Off-by-one error
A distributed mutex is now consistently determined to be expired if
the current time is strictly greater than the expire time.
Unwatch before transaction
Since the redis connection is used by so much of the code, it is
difficult to ensure that any watched keys have been cleared. In order
to defend against this rogue connection state, an unwatch has been
added before locking and unlocking.
Logging
Hopefully this log message is more clear.
I introduced DemonBase because I had got some conflict between `demon/base.rb` and `jobs/base.rb`, however, to not rename base class, it is possible to use regex on absolute path in Zeitwerk custom inflector.
Zeitwerk simplifies working with dependencies in dev and makes it easier reloading class chains.
We no longer need to use Rails "require_dependency" anywhere and instead can just use standard
Ruby patterns to require files.
This is a far reaching change and we expect some followups here.
Adds 2 factor authentication method via second factor security keys over [web authn](https://developer.mozilla.org/en-US/docs/Web/API/Web_Authentication_API).
Allows a user to authenticate a second factor on login, login-via-email, admin-login, and change password routes. Adds registration area within existing user second factor preferences to register multiple security keys. Supports both external (yubikey) and built-in (macOS/android fingerprint readers).
Currently, the topic is only validated for censored words and should be validated for blocked words as well.
Blocked word validation is now used by both Post and Topic. To avoid code duplication, I extracted blocked words validation code into separate Validator, and use it in both places.
The only downside is that even if the topic contains blocked words validation message is saying "Your post contains a word that's not allowed: tomato" but I think this is descriptive enough.
Forums without previously calculated scores would return the same values
for low/medium/high sensitivity. Now those are scaled based on the
default value.
The default value has also been changed from 10.0 to 12.5 based on
observing data from live discourse forums.
This means that TL0 users can message groups with "Who can message this
group?" set to "Everyone".
It also means that members of a group with "Who can message this
group?" set to "members, moderators and admins" can also message the
group, even when their trust level is below min_trust_to_send_messages.
In Rails 6 due to internal changes, the following sequence no longer works:
```
RAILS_ENV=test bin/rake db:migrate
RAILS_ENV=test bin/rake db:schema:dump
dropdb discourse_test
createdb discourse_test
RAILS_ENV=test bin/rake db:schema:load
RAILS_ENV=test bin/rake db:migrate
```
What appears to be happening is that our tracking of plugin migrations is
being missed on schema:dump or load.
A more comprehensive fix restoring schema:dump / load support will be
investigated.
Prior to the new review queue there were a couple special cases where
posts would be auto hidden:
* If a TL3 or above flagged a TL0 post as spam
* If a TL4 or above flagged a non-staff, non-TL4 post as spam, inappropriate or off
topic.
These cases are now removed in favour of the scoring system.
Prior to this change plugin migrations were not working and multisite
migrations not working.
Rails internals changed so we need to account for it.
Specifically semantics of `db:migrate` in rails changed so it is sort of
a "multisite:migrate".
This is a temporary workaround for the issue in https://github.com/rails/rails/pull/36949
Discussing a proper fix in Rails with the Rails team.
Prior to this fix we were spinning up a thread every time we closed a connection
to the db.
* Adjustments to pass specs on Rails 6.0.0
* Use classic autoloader instead of Zeitwerk
* Update Rails 6.0.0 deprecated methods
* Rails 6.0.0 not allowing column with integer name
* Drop freedom_patches/rails6.rb
* Default value for trigger_transactional_callbacks? is true
* Bump rspec-rails version to 4.0.0.beta2
* FIX: inline_uploads and subfolder
* if subfolder, also look for images with a path containing
cdn_url + relative_url_root
* FIX: migrate_to_s3 task and subfolder
* Extract QuickAccessPanel from UserNotifications.
* FEATURE: Quick access panels in user menu.
This feature adds quick access panels for bookmarks and personal
messages. It allows uses to browse recent items directly in the user
menu, without being redirected to the full pages.
* REFACTOR: Use QuickAccessItem for messages.
Reusing `DefaultNotificationItem` feels nice but it actually requires a
lot of extra work that is not needed for a quick access item.
Also, `DefaultNotificationItem` shows an incorrect tooptip ("unread
private message"), and it is not trivial to remove / override that.
* Use a plain JS object instead.
An Ember object was required when `DefaultNotificationItem` was used.
* Prefix instead suffix `_` for private helpers.
* Set to null instead of deleting object keys.
JavaScript engines can optimize object property access based on the
object’s shape. https://mathiasbynens.be/notes/shapes-ics
* Change trivial try/catch to one-liners.
* Return the promise in case needs to be waited on.
* Refactor showAll to a link with href
* Store `emptyStatePlaceholderItemText` in state.
* Store items in Session singleton instead.
We can drop `staleItems` (and `findStaleItems`) altogether. Because
`(old) items === staleItems` when switching back to a quick access
panel.
* Add `limit` parameter to the `user_actions` API.
* Explicitly import Session instead.
* FEATURE: Add tl2 threshold for editing new posts
* Adds a new setting and for tl2 editing posts (30 days same as old value)
* Sets the tl0/tl1 editing period as 1 day
* FIX: Spec uses wrong setting
* Fix site setting on guardian spec
* FIX: post editing period specs
* Avoid shared examples
* Use update_columns to avoid callbacks on user during tests
This commit introduces 2 features:
1. DISCOURSE_COMPRESS_ANON_CACHE (true|false, default false): this allows
you to optionally compress the anon cache body entries in Redis, can be
useful for high load sites with Redis that lives on a separate server to
to webs
2. DISCOURSE_ANON_CACHE_STORE_THRESHOLD (default 2), only pop entries into
redis if we observe them more than N times. This avoids situations where
a crawler can walk a big pile of topics and store them all in Redis never
to be used. Our default anon cache time for topics is only 60 seconds. Anon
cache is in place to avoid the "slashdot" effect where a single topic is
hit by 100s of people in one minute.
Start tracking the date an api key was last used. This has already been
the case for user_api_keys.
This information can provide us with the ability to automatically expire
unused api keys after N days.
This allows custom plugins such as prometheus exporter to log how many
requests are stored in the anon cache vs used by the anon cache.
This metric allows us to fine tune cache behaviors
* FIX: User should get notified when a post is deleted
* FEATURE: Notify posters when restoring flagged posts
* Fix typo
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Improve tests
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
* The read indicator now shows up when no member has read the last post of the topic (written by a non-member)
* The read indicator works on mobile and receives live updates from message bus
* The icon we display in the topic list was changed
* Added a title to the indicator to indicate its purpose when hovering over it
In some very specific cases (large sites) shared drafts can introduce a
performance hit due to the mechanism used to filter out topics
This avoids the entire process when shared drafts are not enabled
* Revert "Revert "FEATURE: Publish read state on group messages. (#7989) [Undo revert] (#8024)""
This reverts commit 36425eb9f0.
* Fix: Show who read only if the attribute is enabled
* PERF: Precalculate the last post readed by a group member
* Use book-reader icon instear of far-eye
* FIX: update topic groups correctly
* DEV: Tidy up read indicator update on write
This is a very long standing bug we had, if a plugin attempted to amend a
serializer core was not "correcting" the situation for all descendant classes
this often only showed up in production cause production eager loads serializers
prior to plugins amending them.
This is a critical fix for various plugins
* FIX: Heartbeat check per sidekiq process
* Rename method
* Remove heartbeat queues of previous bootups
* Regis feedback
* Refactor before_start
* Update lib/demon/sidekiq.rb
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Update lib/demon/sidekiq.rb
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* Expire redis keys after 3600 seconds
* Don't use redis to store the list of queues
This reverts commit 39c31a3d76.
Sorry about this, we have decided againse supporting 0-RTT directly in
core, this can be supported with similar hacks to this commit in a
plugin.
That said, we recommend against using a 0-RTT proxy for the Discourse
app due to inherit risk of replay attacks.
This change allows themes and components access to theme assets.
This means that inside theme js you can now get the URL for an asset with:
```
settings.theme_uploads.name
```
* Reenable: "FEATURE: Publish read state on group messages. (#7989)"
This reverts commit 67f5cc1ce8.
* FIX: Read indicator only appears when the group setting is enabled
* Enable or disable read state based on group attribute
* When read state needs to be published, the minimum unread count is calculated in the topic query. This way, we can know if someone reads the last post
* The option can be enabled/disabled from the UI
* The read indicator will live-updated using message bus
* Show read indicator on every post
* The read indicator now shows read count and can be expanded to see user avatars
* Read count gets updated everytime someone reads a message
* Simplify topic-list read indicator logic
* Unsubscribe from message bus on willDestroyElement, removed unnecesarry values from post-menu, and added a comment to explain where does minimum_unread_count comes from
Previously we relied on side effects to set tracking state correctly
when inviting groups to messages
Also has a minor optimisation in that we use pluck instead of pulling in
full record
This adds support for a `<d-topics-list>` tag you can embed in your site
that will be rendered as a list of discourse topics. Any attributes on
the tag will be passed as filters. For example:
`<d-topics-list discourse-url="URL" category="1234">` will filter to category 1234.
To use this feature, enable the `embed topics list` site setting. Then
on the site you want to embed, include the following javascript:
`<script
src="http://URL/javascripts/embed-topics.js"></script>`
Where `URL` is your discourse forum's URL.
Then include the `<d-topics-list discourse-url="URL">` tag in your HTML document and it will
be replaced with the list of topics.
In some very rare cases CssParser could be loaded but CssParser::Parser not
this ensures we check for the actual constant we plan to call for concurrent
digest generations
There are 5 visibility levels (similar to group visibility)
public (default)
logged-in users
members only
staff
owners
Admins & group owners always have visibility to group members.
Running this inline makes more sense otherwise there is extreme risk in
saturating sidekiq queue.
This also reworks ordering and selection so we double check if a post needs
rebaking prior to rebaking, this unlocks the ability to run this rake task
from multiple consoles.
This column often breaks remaps because of some weird, long URLs. The data isn't that important, so truncating a couple of URLs doesn't hurt that much.
When a user liked, unliked and liked again the same post, the poster
would receive a notification such as "X and X liked ...". This happened
because PostActionNotifier.post_action_created was called twice.
This displays more useful messages for the most common issues we see:
- CSRF (when the user switches browser)
- Invalid IAT (when the server clock is wrong)
- OAuth::Unauthorized for OAuth1 providers, when the credentials are incorrect
This commit also stops earlier for disabled authenticators. Now we stop at the request phase, rather than the callback phase.
This feature (when enabled) will allow for invite_only sites to require
external authentication before they can redeem an invite.
- Created hidden site setting to toggle this
- Enables sending invites with local logins disabled
- OAuth button added to invite form
- Requires OAuth email address to match invite email address
- Prevents redeeming invite if OAuth authentication fails
All posts created by the user are counted unless they are deleted,
belong to a PM sent between a non-human user and the user or belong
to a PM created by the user which doesn't have any other recipients.
It also makes the guardian prevent self-deletes when SSO is enabled.
Temporarily recreate already dropped functions in the discourse_functions schema in order to allow restoring of backups which still reference dropped functions.
The message_bus performs a fair amount of work prior to hijacking requests
this change ensures that if there is a situation where the server is flooded
message_bus will inform client to back off for 30 seconds + random(120 secs)
This back-off is ultra cheap and happens very early in the middleware.
It corrects a situation where a flood to message bus could cause the app
to become unresponsive
MessageBus update is here to ensure message_bus gem properly respects
Retry-After header and status 429.
Under normal state this code should never trigger, to disable raise the
value of DISCOURSE_REJECT_MESSAGE_BUS_QUEUE_SECONDS, default is to tell
message bus to go away if we are queueing for 100ms or longer
Previously we would only hold the post process mutex for 1 minute, that is
not enough when processing a post with lots of images. This raises the bar
to 10 minutes.
It also cleans up error reporting around distributed mutexes expiring. We
used to double report.
- Client-side censoring fixed for non-chrome browsers. (Regular expression rewritten to avoid lookback)
- Regex generation is now done on the server, to reduce repeated logic, and make it easier to extend in plugins
- Censor tests are moved to ruby, to ensure everything works end-to-end
- If "watched words regular expressions" is enabled, warn the admin when the generated regex is invalid
* REFACTOR: Rename SiteSetting.disable_edit_notifications to disable_system_edit_notifications
- The older name could cause some confusion because the setting does not disable all edit notifications, only system ones.
* FIX: Add frozen_string_literal: true in the migration
* DEV: Deprecate 'disable_edit_notifications'
This feature adds the ability to customize the HTML part of all emails using a custom HTML template and optionally some CSS to style it. The CSS will be parsed and converted into inline styles because CSS is poorly supported by email clients. When writing the custom HTML and CSS, be aware of what email clients support. Keep customizations very simple.
Customizations can be added and edited in Admin > Customize > Email Style.
Since the summary email is already heavily styled, there is a setting to disable custom styles for summary emails called "apply custom styles to digest" found in Admin > Settings > Email.
As part of this work, RTL locales are now rendered correctly for all emails.
Fixes bugs, simplifies code, more default files. General idea, more is more here as it's easier to just delete things than reading and passing all the options.
* FIX: Better error when SSO fails due to blank secret
* Update spec/requests/session_controller_spec.rb
Co-Authored-By: Robin Ward <robin.ward@gmail.com>
This feature is off by default and can can be configured with the `email_total_attachment_size_limit_kb` site setting.
Co-authored-by: Maja Komel <maja.komel@gmail.com>
* FEATURE: Add search operator to see all direct messages from a user
* Only show message if related messages >= 5
* Make "all messages" the hyperlink
* Review
This commit contains 3 features:
- FEATURE: Allow downloading watched words
This introduces a button that allows admins to download watched words per action in a `.txt` file.
- FEATURE: Allow clearing watched words in bulk
This adds a "Clear All" button that clears all deleted words per action (e.g. block, flag etc.)
- FEATURE: List all blocked words contained in the post when it's blocked
When a post is rejected because it contains one or more blocked words, the error message now lists all the blocked words contained in the post.
-------
This also changes the format of the file for importing watched words from `.csv` to `.txt` so it becomes inconsistent with the extension of the file when watched words are exported.
* DEV: group_list site settings should store IDs instead of group names
* Ship site setting to know when we should migrate group_list settings
* Migrate existing group_list site settings
* Bump migration timestamp and don't set null when migrating is not possible.
If a post arrives via email but must be reviewed, we now show an
icon that can be clicked to view the raw contents of the email.
This is useful if Discourse's email parser is acting odd and the user
reviewing the post wants to know what the original contents were before
approving/rejecting the post.
Follow up to: [FEATURE: Create a rake task for destroying categories][1]
- `Discourse.system_user` is my friend
- Remove puts statements from rake tasks that don't return anything
- `for_each` is also my friend
- Use `human_users` to also exclude discobot
- Sort/format categories:list
[1]: 092eeb5ca3
Follow up to [FIX: Empty backup names with unicode site titles][1]
- Use .presence - "It's cleaner"
- Update spec to use System.system_user so it is more readable
[1]: c8661674d4
* Revert "Revert "FEATURE: admin/user exports are compressed using the zip format (#7784)""
This reverts commit f89bd55576.
* Replace .tar.zip with .zip
Created a rake task for destroying multiple categories along with any
subcategories and topics the belong to those categories.
Also created a rake task for listing all of your categories.
Refactored existing destroy rake tasks to use new logging method, that
allows for puts output in the console but prevents it from showing in
the specs.
And don't load javascript assets if plugin is disabled.
* precompile auto generated plugin js assets
* SPEC: remove spec test functions
* remove plugin js from test_helper
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* DEV: using equality is slightly easier to read than inequality
Co-Authored-By: Régis Hanol <regis@hanol.fr>
* DEV: use `select` method instead of `find_all` for readability
Co-Authored-By: Régis Hanol <regis@hanol.fr>
This is a low severity security fix because it requires a logged in
admin user to update a site setting via the API directly to an invalid
value.
The fix adds validation for the affected site settings, as well as a
secondary fix to prevent injection in the event of bad data somehow
already exists.
If `SiteSetting.log_search_queries` is enabled 500 errors will occur
when searching if the master db is down. This fix allows searching to
still work under these conditions.
* FEATURE: admin/user exports are compressed using the zip format
* Update translations. Theme exporter now exports .zip file. Theme importer supports .zip and .gz files
* Fix controller test, updated locale and skip saving the csv export to disk
In order for this to work the Backuper stores a couple of site settings
in the new backup_metadata table, because the old setting values might
not be available on restore anymore.
It's hard to see which columns have been remapped when remapping prints
lots of "0 rows affected" lines. This changes it to output the row count
only for affected columns.
Groups can now be marked as visible to "logged on users". All automatic groups (except `everyone`) are now visible to "logged on users", previously they were marked as public but suppressed in the group page for non-staff.
The global setting disable_search_queue_threshold
(DISCOURSE_DISABLE_SEARCH_QUEUE_THRESHOLD) which default to 1 second was
added.
This protection ensures that when the application is unable to keep up with
requests it will simply turn off search till it is not backed up.
To disable this protection set this to 0.
The behaviour of #TERM in search has been amended
1. We try category or subcategory slugs
2. We try tags
3. We try tag-groups
The term `hello #my-group` will search for all posts tagged with any of
the tags in the tag group `My Group`
Future work may be introducing a slug cache here or caching it in the table
but the assumption is that the number of tag groups will not be huge
This can cause unbound CPU usage in some cases, and excessive logging in other cases. This commit moves redis readonly information into the local process, but maintains the DistributedCache for postgres readonly state.
* DEV: Add a new way to run specs in parallel with better output
This commit:
1. adds a new executable, `bin/interleaved_rspec` which works much like
`rspec`, but runs the tests in parallel.
2. adds a rake task, `rake interleaved:spec` which runs the whole test
suite.
3. makes autospec use this new wrapper by default. You can disable this
by running `PARALLEL_SPEC=0 rake autospec`.
It works much like the `parallel_tests` gem (and relies on it), but
makes each subprocess use a machine-readable formatter and parses this
output in order to provide a better overall summary.
(It's called interleaved, because parallel was taken and naming is
hard).
* Make popen3 invocation safer
* Use FileUtils instead of shelling out
* DRY up reporter
* Moved summary logic into Reporter
* s/interleaved/turbo/g
* Move Reporter into its own file
* Moved run into its own class
* Moved Runner into its own file
* Move JsonRowsFormatter under TurboTests
* Join on threads at the end
* Acted on feedback from eviltrout
This fixes a condition where an intermittent db connection could cause
invalid site settings to be stored
It also removes a catch all we had.
Somewhere around Rails 5 `db:create` started wanting full environment
this is a problem for Discourse since it needs to boot up data from the
db.
This removes the catch all and surgically adds a db / redis bypass to
db:create task.
In sidekiq, jobs are run in multiple threads within the same process. `cd` affects the entire process, so can cause unexpected issues in other running jobs.
Sometimes we would like to create a base image without any DB access, this
assists in creating custom base images with custom plugins that already
includes `public/assets`
Following this change set you can run:
```
SPROCKETS_CONCURRENT=1 DONT_PRECOMPILE_CSS=1 SKIP_DB_AND_REDIS=1 RAILS_ENV=production bin/rake assets:precompile
```
Then it is straight forward to create a base image without needing a DB or
Redis.
This previously was a hot path in topic view. Avoids an expensive active
record operation and instead perform SQL directly which is far more
targeted and efficient
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
* Expose a new plugin outlet. Pass group model to the group-member-dropdown so it can be accessed by plugins
* Added controller tests for group custom fields. update custom fields when updating a group
We now show if a queued or flagged post is a reply to another when in
the review queue. It's especially helpful for queued posts where
normally they are linked to the topic where they are created, and you
have no context about the reply.
Note that this will only apply to new queued posts going forward.
Previously queued posts will not show the "in reply to"
During profiling looking up topic users popped up as a hot path, this
change more than halved the amount of work it does
It reduces object allocations and method calls and avoids repeate translation
of common terms
This adds support for DISCOURSE_ENABLE_PERFORMANCE_HTTP_HEADERS
when set to `true` this will turn on performance related headers
```text
X-Redis-Calls: 10 # number of redis calls
X-Redis-Time: 1.02 # redis time in seconds
X-Sql-Commands: 102 # number of SQL commands
X-Sql-Time: 1.02 # duration in SQL in seconds
X-Queue-Time: 1.01 # time the request sat in queue (depends on NGINX)
```
To get queue time NGINX must provide: HTTP_X_REQUEST_START
We do not recommend you enable this without thinking, it exposes information
about what your page is doing, usually you would only enable this if you
intend to strip off the headers further down the stream in a proxy
Previously as soon as any override was defined we would regress to the slow
path for locale lookups. Additionally if `raise: true` was specified which
rails likes to add in views we would bypass the cache
The new design manages to use the fast path for many more cases
Before the locking here was added, replying to a post and liking a post
at roughly the same time could cause a deadlock.
Liking a post grabs an update lock on the post and then on the topic (to
update like counts).
We now lock the replied to post before getting the topic lock so that we
can update the replied to post later without causing a deadlock.
This was a fairly serious regression on sites with large (mega) topics,
however it was limited to staff.
The issue here is the query was using filtered_post_ids which I'd
assumed was already windowed to the current page, when in fact it was
all the ids in the topic. This fix corrects it by using the correct
windowed collection.
You can now add javascript files under `/javascripts/*` in a theme, and they will be loaded as if they were included in core, or a plugin. If you give something the same name as a core/plugin file, it will be overridden. Support file extensions are `.js.es6`, `.hbs` and `.raw.hbs`.
Previously theme setting descriptions were defined in the `settings.yml` file like this:
```
setting_name:
default: "My Default Value"
description:
en: "English description"
fr: "French description"
```
This commit allows developers to store the localised descriptions in the theme locale files instead:
```
en:
theme_metadata:
description: Theme Description
settings:
setting_name: "The localised description for setting_name"
```
SSO uses a special param to username suggester that whitelists a username
due to previous work we amended our lookup logic and started ignoring this
whitelist.
The fix ensures we always respect it, and also improves on the original
implementation that forgot to normalize the username.
This also corrects FileHelper.download so it supports "follow_redirect"
correctly (it used to always follow 1 redirect) and adds a `validate_url`
param that will bypass all uri validation if set to false (default is true)
This new `DISCOURSE_MAXMIND_BACKUP_PATH` can be used a secondary location
for maxmind db. That way a build machine, for example can cache it on the
host and reuse between builds.
Also per 5bfeef77 added proper error raising for download fails from
dedicated rake task
This also moves "refresh_maxmind_db_during_precompile_days" to a global
setting, it did not make sense in a site setting
* FEATURE: detect theme errors and catch them
* Bump COMPILER_VERSION
* Feedback
* Override eslint no console for one line
* Can't use our ajax method
* remove emoji from translation file
`rake posts:recover_uploads_from_index`
Searches through all missing uploads in the cluster, if it finds one it
tries to find it in the "upload index file" and creates a new upload for
it.
Previously we were only catching one type of data export, the new job will
catch every csv export we have
Job is pretty safe as it filters on system user id / pm with a particular
slug
Historically we would keep the user data export posts around but delete
the uploads.
This leaves a lot of broken uploads in the system.
This rake task allows us to clean up old mess.
Filename on disk may mismatch sha of file in some old 1X setups. This will
attempt to recover file even if sha1 mismatches. We had an old bug that
caused this.
This also adds `uploads:fix_relative_upload_links` which attempts to replace
urls of the format `/upload/default/...` with `upload://`
Rebaking posts can be expensive instead of blocking here simply mark posts
for rebake.
We can then work through them faster in other jobs, plus this should not
hold of a datacenter migration.
Previously this rake job would only run on a single site which is a bit
misleading
This also adds `VERBOSE=1 rake posts:missing_uploads` that will provide a
full report of missing uploads
This allows you to wait up to N seconds for the smoke test url to come up
in some cases you want to kick off the smoke test prior to having the smoke
test env ready to accept connections
Net::HTTP always returns ASCII-8BIT encoding. File.read auto-detects the encoding. This leads to an encoding inconsistency between a fresh download, and a cached download. This commit ensures all downloaded files are treated equally, by always returning the cached version from the filesystem, even during initial download.
One symptom of this problem is during theme exports: https://meta.discourse.org/t/116907
Related ruby ticket: https://bugs.ruby-lang.org/issues/2567
Having different behavior for staff and regular users can make it confusing for admins to understand how their configuration changes affect regular users
Previously username suggester would give up after 100 attempts at getting
a username and fallback to random string.
This amends the logic so we do all the work of figuring out a good username
in SQL and avoids a large amount of queries in cases where a lot of usernames
were used up.
This corrects an issue on sites with large numbers of anon users
Use the cooked version of the post and the quote to compare their content in
order to take into account the "typographer" option of the markdown pipeline.
We were blocking user registrations with same username and password,
but allowing usernames to be changed to be same as password later.
Also disallow names to be the same as password.
* English shouldn't fallback to any other locale
* Calculate fallback for default locale if it isn't English (useful for en_US)
* Reuse the fallback locale list when outputting translations to JavaScript
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
The instagram onebox sometimes surrounds the image with an `<a>` tag, which was breaking the aspect ratio logic, and therefore causing posts to change height on load.
This is a feature that used to be present in discourse-assign but is
much easier to implement in core. It also allows a topic to be assigned
without it claiming for review and vice versa and allows it to work with
category group reviewers.
If creating a topic via the api as an admin and the category you specify
cannot be found an error will now be returned instead of just creating
the topic with no category. This will prevent accidental public topic
creation originally intended for a private category.
This commit is follow up to 535c594891 and
still allows for the creation of topics where the category param is
blank.
`#find` raises an error if the id given to it is invalid. As a result,
the conditional to check whether a `group` or `badge` is `present?` will
not be executed if any of the ids are invalid.
Follow up to
6ba914033c.
If creating a topic via the api as an admin and the category you specify
cannot be found an error will now be returned instead of just creating
the topic with no category. This will prevent accidental public topic
creation originally intended for a private category.
Since 5bfe051e, Discourse user agents are marked as non-crawlers (to avoid accidental blacklisting). This makes sure pageviews for these agents are tracked as crawler hits.
#b9d82818 makes enormous improvements to our bootstrap time, however going
to still keep compress for now despite the cost and watch it for a few weeks
* Do not brotli all locales in precompile
* Try without gzip
* uglify without compressing, always gzip
* skip uglify for unused locales
* FIX: Uglifier needs harmony for ES6 compatibility
* Use node uglifier if available
* Minor refactor
We found score hard to understand. It is still there behind the scenes
for sorting purposes, but it is no longer shown.
You can now filter by minimum priority (low, med, high) instead of
score.
Daniel Waterworth
Joffrey JAFFEUX
Krzysztof Kotlarek
tshenry
Rafael dos Santos Silva
David Taylor
Sam Saffron
Neil Lalonde
Vinoth Kannan
Roman Rizzi
Régis Hanol
Nacho Caballero
Mark VanLandingham
Dan Ungureanu
Jarek Radosz
Robin Ward
Justin DiRose
Michael Brown
Martin Brennan
Gerhard Schlager
Penar Musaraj
Bianca Nenciu
Kyle Zhao
Rimian Perkins
OsamaSayegh
Arpit Jalan
Blake Erickson
Vinoth Kannan
Leo McArdle
Guo Xiang Tan
Rishabh
Angus McLeod
Michael Brown
Ralph Rooding
Josh Moore
Maja Komel
Andrew Schleifer
Bianca Nenciu
Michael Brown