While using `OpenStruct` is nice, it’s generally not a very good idea as
it usually leads to performance problems.
The `OpenStruct` source code even says basically to avoid it.
Since the context object is crucial in our services, this patch replaces
`OpenStruct` with a custom implementation instead.
When a post has some replies, and the user click on the button to show them, we would load ALL the replies. This could lead to DoS if there were a very large number of replies.
This adds support for pagination to these post replies.
Internal ref t/129773
FIX: Duplicated parent posts
DEV: Query refactor
XHR requests are handled differently by the application and the
responses do not have any preloaded data so the cache key needs to
differntiate between those requests.
Remove emoji.clear cache calls as data.js.es6.erb hasn't existed in a while.
Emoji data is now compiled separately via javascript rake tasks.
Skip db and redis precompilation when no db is present
Currently, when the MessageFormat compiler fails on some translations,
we just have the raw output from the compiler in the logs and that’s not
always very helpful.
Now, when there is an error, we iterate over the translation keys and
try to compile them one by one. When we detect one that is failing, it’s
added to a list that is now outputted in the logs. That way, it’s easier
to know which keys are not properly translated, and the problems can be
addressed quicker.
My theory is that there were nil entries (that we were filtering out) that then changed and we weren't resetting them properly.
(the failure no longer repro'd in 30 CI runs in this PR)
Currently in services, the `contract` step is only used to define where
the contract will be called in the execution flow. Then, a `Contract`
class has to be defined with validations in it.
This patch allows the `contract` step to take a block containing
validations, attributes, etc. directly. No need to then open a
`Contract` class later in the service.
It also has a nice side effect, as it’s now easy to define multiples
contracts inside the same service. Before, we had the `class_name:`
option, but it wasn’t really useful as you had to redefine a complete
new contract class.
Now, when using a name for the contract other than `default`, a new
contract will be created automatically using the provided name.
Example:
```ruby
contract(:user) do
attribute :user_id, :integer
validates :user_id, presence: true
end
```
This will create a `UserContract` class and use it, also putting the
resulting contract in `context[:user_contract]`.
Previously admins could still click on topics when `suppress_secured_categories_from_admin` was set
This change improves the block so admins without permission will not be allowed to click through till they add themselves to appropriate groups
Keep in mind this setting is a quality of life setting and not a SECURITY
setting, admins have an infinite way of bypassing visiblity limits
This commit introduces a feature that allows an admin to delete a user's
associated account. After deletion, a log will be recorded in staff
actions.
ref=t/136675
With the current implementation, a service step can be written as:
```ruby
def my_step(a_default_value: 2)
…
end
```
That’s a pattern we want to avoid as default values (if needed) should
be probably defined in a contract.
This patch makes a service raise an exception if a default value is
encountered.
Currently, when certain search terms are provided, this can lead to
`Search.need_segmenting?` raising an error because it makes `URI#path`
to return `nil` instead of a string.
This patch forces a cast to string so it won’t raise anymore.
At a point we split FileStore#download into two methods, one which returns nil (#download) and one which raises an exception (#download!) if there's a download error.
When we did this we added a deprecation warning to the #download method.
It has now been over a year and I have checked all relevant internal cases, so I think it's about time we remove this warning.
This will help to enforce a consistent pattern for creating service
actions.
This patch also namespaces actions and policies, making everything
related to a service available directly in
`app/services/<concept-name>`, making things more consistent at that
level too.
In TopicController, in addition to ensure_can_move_posts!, we also
checked if the topic is private message in this line:
```ruby
raise Discourse::InvalidAccess if params[:archetype] == "private_message" && !guardian.is_staff?
```
However, this was not present in `guardian.can_move_posts?`. As a result,
the frontend topic view got an incorrect serialized result, thinking
that TL4 could move the private message post. In fact, once they tried
to move it, they got the `InvalidAccess` error message.
This commit fixes that TL4 will no longer sees the "move to" option in
the "select post" panel for a private message.
The user directory (`/u`) excludes inactive and silenced users from the list, so for the sake parity, it makes sense to also exclude those users from the /about page stats.
Internal topic: t/70928.
- fetch models inside services
- validate `user_id` in contracts
- use policy objects
- extract more logic to actions
- write specs for services and action
The user option 'hide_profile_and_presence' is necessary to figure out
if the user status has to be displayed or not. In order to avoid N+1s
generated by `include_status?` method, both `user_status` and
`user_option` relations have to be included.
`track_sql_queries` only returned queries that were executed by
ActiveRecord. All queries executed through DB.exec, DB.query and others
were not returned.
We were running into errors running `ember build` on machines with high
CPU counts. It was then noted that `thread-loader`, which embroider uses, defaults to spinning
up x workers where x is number of physical CPU cores - 1. That is
probably too much so we set out to find out an optimial count to set for
the `JOBS` env which embroider will use to set the number of
`thread-loader` workers.
I first built an image using the following Dockerfile.
```
FROM discourse/base:release
RUN cd /var/www/discourse && sudo -EH -u discourse bundle exec rake plugin:install_all_official
RUN cd /var/www/discourse && sudo -EH -u discourse bundle exec rake assets:precompile:prereqs
```
I then ran the following command on my M3 Max Macbook Pro that has 14
phyisal CPU cores.
```
for j in 1 2 4 8 14; do echo "JOBS=$j"; time docker run --rm -it -e JOBS=$j test:latest /bin/bash -c "su discourse -c 'cd /var/www/discourse && bundle exec rake assets:precompile:build'"; done
```
These are the results I got:
```
JOBS=1 0.04s user 0.03s system 0% cpu 1:01.92 total
JOBS=2 0.04s user 0.02s system 0% cpu 42.605 total
JOBS=4 0.04s user 0.02s system 0% cpu 37.012 total
JOBS=8 0.04s user 0.02s system 0% cpu 35.199 total
JOBs=14 0.04s user 0.02s system 0% cpu 37.941 total
```
We think JOBS=2 is a good default when the `JOBS` env has not been set.
Anything above just consumes more resources for little benefit.
Previously we were opening the file handles, then putting them in a queue for upload. If that queue grows too large, we can hit a maximum open files limit.
This commit opens the file handle 'just in time', so the maximum number of open handles is equal to the upload concurrency (20).
This upgrade is designed to be fully backwards-compatible. Any icon names which have changed will be automatically remapped to the new name. For now, this will happen silently. In future, once core & official themes/plugins have been updated, we will start raising deprecation errors to help theme/plugin authors update their code.
Extracted from https://github.com/discourse/discourse/pull/28715
Announcement at https://meta.discourse.org/t/were-upgrading-our-icons-to-font-awesome-6/325349
Co-authored-by: awesomerobot <kris.aubuchon@discourse.org>
This was added 10 years ago, but currently there's not a single use in our public and private plugins and no reference in third-party plugins on github
* FEATURE: Add user to topic_tags_changed event
Add user to topic_tags_changed event context
Update automation plugin with new arguments in event
Update tests for new arguments
relates to https://github.com/discourse/discourse-chat-integration/pull/214
* DEV: change variable name for better readability
changed `tags` to be payload and used `values_at` to get the values of the keys
We currently have some occurrences of ____FaradayFormatter for OAuth logs. This commit creates a generic formatter so that any new authenticators can use it.
This patch removes the `with_service` helper from the code base.
Instead, we can pass a block with actions directly to the `.call` method
of a service.
This simplifies how to use services:
- use `.call` without a block to run the service and get its result
object.
- use `.call` with a block of actions to run the service and execute
arbitrary code depending on the service outcome.
It also means a service is now “self-contained” and can be used anywhere
without having to include a helper or whatever.
We need to start printing deprecation notices when the `show_in_ui` argument is used because it works only for the old about page which will be removed soon. For the new about page, we've introduced a new API `addAboutPageActivity` which is more flexible than a true/false argument on the server side.
Internal topic: t/136551.
Currently, categories support designating only 1 group as a moderation group on the category. This commit removes the one group limitation and makes it possible to designate multiple groups as mods on a category.
Internal topic: t/124648.
The recorded runtime log was specified when running the plugin specs,
but it failed to match the entries from the runtime log and the passed
as command line arguments because the runtime log entries were not
prefixed with `./`.
This commit introduces a new hidden site setting: `group_pm_user_limit`, default to `1000` which will raise an error when attempting to create a PM target a large group.
This will bring significant improvements to install speed & storage requirements. For information on how it may affect you, see https://meta.discourse.org/t/324521
This commit:
- removes the `yarn.lock` and replaces with `pnpm-lock.yaml`
- updates workspaces to pnpm format
- adjusts package dependencies to work with pnpm's stricter resolution strategy
- updates Rails app to load modules from more specific node_modules directories
- adds a `.pnpmfile` which automatically cleans up old yarn-managed `node_modules` directories
- updates various scripts to call `pnpm` instead of `yarn`
- updates patches to use pnpm's native patch system instead of patch-package
- adds a patch for licensee to support pnpm
Currently, when the default locale is Japanese, the search for a topic
using its URL, path or ID doesn’t work as expected. It will either
return wrong results or no result at all.
The problem lies with how we process the provided terms in Japanese
mode. For example, if `http://localhost/t/-/55` is provided, currently
this will result in `http localhost t 5 5` to be searched for.
This patch addresses the issue by checking whether the provided term
needs segmenting. If the provided term is a number, or a path or a full
URL, then it doesn’t need segmenting. When that happens we skip the
processing we normally apply for Japanese, making the search return the
expected results.
A new setting attribute is used to define the areas (separated by `|`).
In addition, endpoint `/admin/config/site_settings.json` accepts new `filter_area` data.
This commit will allow plugin developers to enable/disable the custom homepage.
Usage:
```ruby
register_modifier(:custom_homepage_enabled) do |enabled, args|
true
end
```
Args might contain request and/or current_user.
* Do not delete created external upload stubs for 2 days
instead of 1 hour if enable_upload_debug_mode is true,
this aids with server-side debugging.
* If using an API call, return the detailed error message
if enable_upload_debug_mode is true. In this case the user
is not using the UI, so a more detailed message is appropriate.
* Add a prefix to log messages in ExternalUploadHelpers, to
make it easier to find these in logster.
`ember exam` points all browsers at a single ember-cli server process. With very high parallelism, we've started seeing `ChunkLoadError` exceptions, which may indicate the server being overloaded. Capping to 8 browsers to avoid that.
Followup 76c56c8284
The change introduced above made it so the expired
bookmark reminders were cleared when using the bulk
action menu for bookmarks. However this also affected
clearing reminders for bookmarks when sending notifications.
When clearing bookmark reminders after sending notifications,
we take into account the auto delete preference:
* never - The bookmark `reminder_at` date should not be cleared,
and the bookmark is kept.
* clear_reminder - The bookmark `reminder_at` date is cleared and
the bookmark is kept
The `never` option made it so "expired" bookmark reminder show
on the user's bookmark list.
This commit fixes the change from the other commit and only
forces clearing of `reminder_at` if using the bookmark bulk
action service.
This commit improves `TopicsController#show` to not load suggested and
related topics unless it is the last page of the topic's view.
Previously, we avoided loading suggested and related topics by the use
of conditionals in the `TopicViewSerializer` to avoid calling
`TopicView#suggested_topics` and `TopicView#related_topics`. However,
this pattern is not reliable as the methods can still be called from
other spots in the code base. Instead, we ensure that
`TopicView#include_suggested` and `TopicView#include_related` is set
correctly on the instance of `TopicView` which ensures that for the
given instance, `TopicView#suggested_topics` and
`TopicView#related_topics` will be a noop.
What did this fix?
===============
Previously, we only triggered this event in the `user.logged_out` method.
This resulted in the event being triggered only when the user was logged
out by the administrator or the site had strict logout mode enabled.
This bug affected customers who managed user status via webhooks.
meta topic: https://meta.discourse.org/t/user-log-out-event-not-triggered-in-webhooks/249464
Previously, we were silently failing when a theme hit SSRF protection, or the `git clone` command failed for some reason. This commit updates them to be exceptions, so they provide more useful error messages
This commit implements 2 new metrics/stats in the /about page for the _estimated_ numbers of unique visitors from the EU and the rest of the world. This new feature is currently off by default, but it can be enabled by turning on the hidden `display_eu_visitor_stats` site settings via the rails console.
There are a number of assumptions that we're making here in order to estimate the number of unique visitors, specifically:
1. we're assuming that the average of page views per anonymous visitor is similar to the average number of page views that a logged-in visitor makes, and
2. we're assuming that the ratio of logged in visitors from the EU is similar to the ratio of anonymous visitors from the EU
Discourse keeps track of the number of both logged-in and anonymous page views, and also the number of unique logged-in visitors and where they're from. So with those numbers and the assumptions above, we can estimate the number of unique anonymous visitors from the EU and the rest of the world.
Internal topic: t/128480.
This patch allows using an AR relation as a model in services without
fetching associated records. It will just check if the relation is empty
or not. In the former case, the execution will stop at that point, as
expected.
This commit introduces a new frontend API to add custom items to the "Site activity" section in the new /about page. The new API is called `addAboutPageActivity` and it works along side the `register_stat` serve-side API which serializes the data that the frontend API consumes. More details of how the two APIs work together is in the JSDoc comment above the API function definition.
Internal topic: t/128545/9.
This commit fixes two codepaths which where incorrectly working with capitalized usernames as we were doing a mix of username_lower and non lower username.
Also adds two specs for these cases.
This patch removes two freedom patches:
- `mail_disable_starttls.rb`: this has been fixed in the 2.8 release of
the mail gem, so we don’t need it anymore.
- `rails4.rb`: those methods have been deprecated for a while now and
should have been dropped with Discourse v3.2.
In development mode and when a developer's email is configured as part
of `Rails.configuration.developer_emails`, the user can be trusted and
should not be required to be an admin user.
* System user attachment size WIP
* spec check
* controller update
* add max to system_user_max_attachment_size_kb
* DEV: update to use static method for `max_attachment_size_for_user`
add test to use large image.
add check for failure.
* DEV: update `system_user_max_attachment_size_kb` default value to 0
remove unecessary test.
update tests to reflect the new default value of `system_user_max_attachment_size_kb`
* DEV: update maximum_file_size to check when is an attachment made by a system user
Add tests for when `system_user_max_attachment_size_kb` is over and under the limit
Add test for checking interaction with `max_attachment_size_kb`
* DEV: move `max_attachment_size_for_user` to private methods
* DEV: turn `max_attachment_size_for_user` into a static method
* DEV: typo in test case
* DEV: move max_attachment_size_for_user to private class method
* Revert "DEV: move max_attachment_size_for_user to private class method"
This reverts commit 5d5ae0b715.
---------
Co-authored-by: Gabriel Grubba <gabriel@discourse.org>
* DEV: Removal of create_post_for_category_and_tag_changes setting
reverting commit: #65f35e1
and adding a migration to remove the setting
ref: t/132320
* DEV: change checks for zeros to check for nils
* DEV: remove create_post_for_category_and_tag_changes migration file
If anything goes wrong, we can always revert back to the previous state.
This commit updates `Demon::Base#start` to call `Discourse.before_fork`
before forking. According to the docs in `mini_racer`, we need to
"Dispose manually of all MiniRacer::Context objects prior to forking".
This commit is motivated by a segmentation fault which we are seeing in
production when killing a daemon process. Backtrace of the core dump
includes traces of `mini_racer` so we think this is the cause. Note that
we are not 100% sure if this will fix the issue.
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems.
### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons.
At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.
To test backward compatibility you can add this code to any plugin
```ruby
replace_flags do |flag_settings|
flag_settings.add(
4,
:inappropriate,
topic_type: true,
notify_type: true,
auto_action_type: true,
)
flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
end
```
### Why?
Before, all flags were static. Therefore, they were stored in class variables and serialized by SiteSerializer. Recently, we added an option for admins to add their own flags or disable existing flags. Therefore, the class variable had to be dropped because it was unsafe for a multisite environment. However, it started causing performance problems.
### Solution
When a new Flag system is used, instead of using PostActionType, we can serialize Flags and use fragment cache for performance reasons.
At the same time, we are still supporting deprecated `replace_flags` API call. When it is used, we fall back to the old solution and the admin cannot add custom flags. In a couple of months, we will be able to drop that API function and clean that code properly. However, because it may still be used, redis cache was introduced to improve performance.
To test backward compatibility you can add this code to any plugin
```ruby
replace_flags do |flag_settings|
flag_settings.add(
4,
:inappropriate,
topic_type: true,
notify_type: true,
auto_action_type: true,
)
flag_settings.add(1001, :trolling, topic_type: true, notify_type: true, auto_action_type: true)
end
```
Adds a new statistics (hidden from the UI, but available via the API) that tracks daily participating users.
A user is considered as "participating" if they have
- Reacted to a post
- Replied to a topic
- Created a new topic
- Created a new PM
- Sent a chat message
- Reacted to a chat message
Internal ref - t/131013
This commit adds a `MiniSchedulerLongRunningJobLogger` class which will
poll every 60 seconds for mini_scheduler jobs which are stuck. When it
detects that a job is stuck, it will log a warning message with the
current backtrace of the thread that is executing the job.
Note that for scheduled jobs which are executed at a frequency of less
than 30 minutes, we will log when the job has been executing for 30
minutes.
For scheduled jobs executed at a frequency of less than 2 hours, we will
log when the job has been executing for a duration greater than its
specified frequency.
For scheduled jobs executed at a frequency greater than 2 hours, we will
log as long as the job has been executing for more than 2 hours.
This commit continues on work laid out by 6039b513fe to redesign the /about page. In this commit, we add the site age and a section on the right hand side to show site activities/statistics such as topics, posts, sign-ups, likes etc.
Admin can create up to 50 custom flags. It is limited for performance reasons.
When the limit is reached "Add button" is disabled and backend is protected by guardian.
This commit patches `Net::HTTP` to reduce the default timeouts of 60
seconds when we are processing a request. There are certain routes in
Discourse which makes external requests and if the proper timeouts are
not set, we risk having the Unicorn master process force restarting the
Unicorn workers once the `30` seconds timeout is reached. This can
potentially become a vector for DoS attacks and this commit is aimed at
reducing the risk here.
The Safari 15 bugfix has been rolled into @babel/preset-env in the most recent version, so we no longer need to carry our vendored copy.
This commit updates @babel/preset-env, runs npx yarn-deduplicate yarn.lock, and removes the vendored transform.
This commit also refactors our theme transpiler to use @babel/preset-env, with the same list of target browsers as our ember-cli build uses. This means we no longer need to maintain a separate list of babel transforms for themes.
We were writing theme-transpiler JS files to the filesystem on a per-process basis, and then immediately reading them back in. Plus, there was no cleanup mechanism, so the tmp directory would grow indefinitely.
This commit refactors things so that the `build.js` script outputs the theme-transpiler source to stdout. That way, we can read it directly into the process, and then into mini-racer, without needing to go via the filesystem. No cleanup required!
In production, the theme-transpiler is still cached in a file during `assets:precompile`
When `SiteSetting.review_every_post` is true and the category `require_topic_approval` system creates two reviewable items.
1. Firstly, because the category needs approval, the `ReviewableQueuePost` record` is created - at this stage, no topic is created.
2. Admin is approving the review. The topic and first post are created.
3. Because `review_every_post` is true `queue_for_review_if_possible` callback is evaluated and `ReviewablePost` is created.
4. Then `ReviewableQueuePost` is linked to the newly generated topic and post.
At the beginning, we were thinking about hooking to those guards:
```
def self.queue_for_review_if_possible(post, created_or_edited_by)
return unless SiteSetting.review_every_post
return if post.post_type != Post.types[:regular] || post.topic.private_message?
return if Reviewable.pending.where(target: post).exists?
...
```
And add something like
```
return if Reviewable.approved.where(target: post).exists?
```
However, because the callback happens in point 3. before the `ReviewableQueuePost` is linked to the `Topic`, it was not possible.
Therefore, when `ReviewableQueuePost` is creating a `Topic`, a new option called `:reviewed_queued_post` is passed to `PostCreator` to avoid creating a second `Reviewable`.
We have been seeing `ZLib::BufError` when running the `assets:precompile` rake
task.
```
I, [2024-07-30T05:19:58.807019 #1059] INFO -- : Writing /var/www/discourse/public/assets/scripts/discourse-test-listen-boot-9b14a0fc65c689577e6a428dcfd680205516fe211700a71c7adb5cbcf4df2cc5.js
rake aborted!
Zlib::BufError: buffer error (Zlib::BufError)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb💯in `<<'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb💯in `set'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache.rb:212:in `set'
```
The hypothesis here is that some thread unsafe issue is causing the
problem since we download the Maxmind databases in a thread and run
decompression operations once the gzip file is downloaded.
In the near term, we plan to move downloading of Maxmind databases out
of the Rake task into a scheduled job so this patch should be considered
a temporary solution.
The trade-off here is that build time will slightly increase since we
are not longer downloading Maxmind databases while precompiling assets
at the same time.
When using `Discourse.cache.fetch` with an expiry, there's a potential for a race condition due to how we read the data from redis.
The code used to be
```ruby
raw = redis.get(key) if !force
entry = read_entry(key) if raw
return entry if raw && !(entry == :__corrupt_cache__)
```
with `read_entry` defined as follow
```ruby
def read_entry(key)
if data = redis.get(key)
Marshal.load(data)
end
rescue => e
:__corrupt_cache__
end
```
If the value at "key" expired in redis between `raw = redis.get` and `entry = read_entry`, the `entry` variable would be `nil` despite `raw` having a value.
We would then proceed to return `entry` (which is `nil`) thinking it had a value, when it didn't.
The first `redis.get` can be skipped altogether and we can rely only on `read_entry` to read the data from redis. Thus avoiding the race condition and removing the double read operations.
Internal ref - t/132507
Loïc Guitaut
Alan Guo Xiang Tan
Penar Musaraj
OsamaSayegh
Régis Hanol
Bianca Nenciu
Jeff Wong
Jarek Radosz
Keegan George
Sam
Daniel Waterworth
Linca
Ted Johansson
Kris
Bianca Nenciu
David Taylor
Jordan Vidrine
Sérgio Saquetim
Gabriel Grubba
Natalie Tay
Joffrey JAFFEUX
Krzysztof Kotlarek
Martin Brennan
Guhyoun Nam
Isaac Janzen
Jay Pfaffman
Blake Erickson