A scheduled job runs `Post.rebake_old` with a limit of 80 which does a
look up for the latest posts that have not been baked to the latest
version. Before this commit, the query would run using the primary key's
index and does a reverse scan. However the query performance quicky
becomes bad as more and more of the latest posts have been baked to the
latest version.
This optimisation avoids large scans joining the topics table with the
topic_users table.
Previously when a user carried a lot of read state we would have to join
the entire read state with the topics table. This operation would slow down
home page and every topic page. The more read state you accumulated the
larger the impact.
The optimisation helps people who clean up unread, however if you carry
unread from years ago it will only have minimal impact.
A new checkbox has been added to the Tags tab of the category settings modal
which is used when some tags and/or tag groups are restricted to the category,
and all other unrestricted tags should also be allowed.
Default is the same as the previous behaviour: only allow the specified set of
tags and tag groups in the category.
"Rejecting" a user in the queue is equivalent to deleting them, which
would then making it impossible to review rejected users. Now we store
information about the user in the payload so if they are deleted things
still display in the Rejected view.
Secondly, if a user is destroyed outside of the review queue, it will
now automatically "Reject" that queue item.
Conversely, if a user is deactivated the reviewable should automatically
be rejected.
Before this fix, if a user was not active they'd still show in the
review queue but without an "Approve" button which was confusing.
This functionality was never supported but before the new review queue
it didn't have any errors. Now the combination of settings is prevented
and existing sites with sso enabled will be migrated to remove invite
only.
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.
Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
- s3_force_path_style was added as a Minio specific url scheme but it has never been well supported in our code base.
- Our new migrate_to_s3 rake task does not work reliably with path style urls too
- Minio has also added support for virtual style requests i.e the same scheme as AWS S3/DO Spaces so we can rely on that instead of using path style requests.
- Add migration to drop s3_force_path_style from the site_settings table
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by
* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
Previous instagram records didn’t have unique index, as a result it's seems like in some cases we have duplicate ids, which is causing issues with the associated_accounts which does have unique index.
It's sorted by `updated_at DESC` on the thought that the most recent one is more likely to be the one we want to keep.
For some reason, setting a text site setting to empty in the past can
result in either an empty string or null as the value. This migration
is a follow up to 1981add261.
If the depercated SiteSetting.logo_url was set to "" previously, it will
not have been migrated by the onceoff job which we added. As such, we
need a migration to migrate the blank override so that the default for
certain upload site settings will not be used.
* FEATURE: Add `IgnoredUsersSummary` daily job
## Why?
This is part of the [Ability to ignore a user feature](https://meta.discourse.org/t/ability-to-ignore-a-user/110254/8).
We want to:
1. Send an automatic group PM that goes out to moderators
2. When {x} users have Ignored the same user, threshold defined by a site setting, default of 5
3. Only send this message every X days which is defined by another site setting
* Adds warnings to the "Edit Category" dialog
* Doesn't hide the "Security" tab on the "Edit Category" dialog anymore. Instead, it shows an explanation why permissions can't be changed.
* Makes the category name translatable
* Hides the category name from the edit dialog (it can be customized by overriding the translation)
* Creates a translation override if the category has been renamed in the past
Generally we should never be touching AR objects in migrations, this is
super risky as we may end up with invalid schema cache.
This code from 2013 did it unconditionally. This change amends it so:
1. We only load up schema if we have no choice
2. We flush the cache before and after
This makes this migration far less risky.
This commit copies `login_required.welcome_message` translation override to `login_required.welcome_message_invite_only` if the `login_required.welcome_message_invite_only` override is blank. This is done so as to ensure that login page customization remains same after [this commit](93eb0a0690).
New `about.json` fields (all optional):
- `authors`: An arbitrary string describing the theme authors
- `theme_version`: An arbitrary string describing the theme version
- `minimum_discourse_version`: Theme will be auto-disabled for lower versions. Must be a valid version descriptor.
- `maximum_discourse_version`: Theme will be auto-disabled for lower versions. Must be a valid version descriptor.
A localized description for a theme can be provided in the language files under the `theme_metadata.description` key
The admin UI has been re-arranged to display this new information, and give more prominence to the remote theme options.
Some badges always appeared in the "Other" group (the default group) and some badges were always moved back into the original group during seeding.
Now badges are either in the correct, seeded group or stay in a custom group if the admin moved the badge into a custom group.
- Themes can supply translation files in a format like `/locales/{locale}.yml`. These files should be valid YAML, with a single top level key equal to the locale being defined. For now these can only be defined using the `discourse_theme` CLI, importing a `.tar.gz`, or from a GIT repository.
- Fallback is handled on a global level (if the locale is not defined in the theme), as well as on individual keys (if some keys are missing from the selected interface language).
- Administrators can override individual keys on a per-theme basis in the /admin/customize/themes user interface.
- Theme developers should access defined translations using the new theme prefix variables:
JavaScript: `I18n.t(themePrefix("my_translation_key"))`
Handlebars: `{{theme-i18n "my_translation_key"}}` or `{{i18n (theme-prefix "my_translation_key")}}`
- To design for backwards compatibility, theme developers can check for the presence of the `themePrefix` variable in JavaScript
- As part of this, the old `{{themeSetting.setting_name}}` syntax is deprecated in favour of `{{theme-setting "setting_name"}}`
- adds migration to enable CSP for new sites
- removes "EXPERIMENTAL" labels from setting names
- sets CSP violation report to default off
- adds CSP-related note to GTM setting
We query this table when getting composer messages with the queries
such as:
```
SELECT 1 AS one FROM "user_histories"
WHERE "user_histories"."target_user_id" = 1 AND
"user_histories"."action" = 9 AND
"user_histories"."topic_id" = 105794
LIMIT 1
```
This index ensures this query remains very quick, regardless of user
history size.
This was done to pick up 3 changes
1. New pngquant which will result in much smaller images
2. Placeholder images which are missing from old posts
3. Retina images missing from old posts
Also picks up on Image Magick upgrade which slightly alters resize algorithm.
Rebake trickles per: `rebake_old_posts_count` site setting. (100 per 15 minutes)
Previously we had no idea what algorithm generated thumbnails, this starts tracking the version.
We also bumped up the version to force all optimized images to be generated. This is important cause we recently introduced pngquant which results in much smaller images.
Do not send an activation email to users invited via email. They
already confirmed their email address by clicking the invite link.
Users invited via link will need to confirm their email address before
they can login.
Changes to functionality
- Removed syncing of user metadata including gender, location etc.
These are no longer available to standard Facebook applications.
- Removed the remote 'revoke' functionality. No other providers have
it, and it does not appear to be standard practice in other apps.
- The 'facebook_no_email' event is no longer logged. The system can
cope fine with a missing email address.
Data is migrated to the new user_associated_accounts table.
facebook_user_infos can be dropped once we are confident the data has
been migrated successfully.
A generic implementation of Auth::Authenticator which stores data in the
new UserAssociatedAccount model. This should help significantly reduce the duplicated
logic across different auth providers.
* Doing it in a post migration was a bad idea
because the migration will fail if the site
is down while trying to download uploads
which points to the instance. This mainly
affects self-hosters using `discourse_docker`
where `./launcher rebuild` will take the
existing container down.
This splits off the logic between SSO keys used incoming vs outgoing, it allows to far better restrict who is allowed to log in using a site.
This allows for better auditing of the SSO provider feature
* FEATURE: add branch option to remote theme import
* FIX: Add missing variable in params
* FIX: Add missing param for import_theme method
* SPEC: Add test methods for branch support in git import
* FIX: Add missing space to scss style
* Do not assume default branch as master
* Change branch field placeholder
* FIX: add missing div start tag
This moves us away from the delayed drops pattern which
was problematic on two counts. First, it uses a hardcoded "delay for"
duration which may be too short for certain deployment strategies.
Second, delayed drop doesn't ensure that it only runs after
the latest application code has been deployed. If the migration runs
and the application code fails to deploy, running the migration after
"delay for" has been met will cause the application to blow up.
The new strategy allows post deployment migrations to be skipped if the
env `SKIP_POST_DEPLOYMENT_MIGRATIONS` is provided.
```
SKIP_POST_DEPLOYMENT_MIGRATIONS=1 rake db:migrate
-> deploy app servers
SKIP_POST_DEPLOYMENT_MIGRATIONS=0 rake db:migrate
```
To aid with the generation of a post deployment migration, a generator
has been added. Simply run `rails generate post_migration`.
- By default, behaviour is not changed: tags are made lowercase upon creation and edit.
- If force_lowercase_tags is disabled, then mixed case tags are allowed.
- Tags must remain case-insensitively unique. This is enforced by ActiveRecord and Postgres.
- A migration is added to provide a `UNIQUE` index on `lower(name)`. Migration includes a safety to correct any current tags that do not meet the criteria.
- A `where_name` scope is added to `models/tag.rb`, to allow easy case-insensitive lookups. This is used instead of `Tag.where(name: "blah")`.
- URLs remain lowercase. Mixed case URLs are functional, but have the lowercase equivalent as the canonical.
Introduces a hidden setting (default is 0.1) that erodes bounce score
every time we send an email. This means that erratic failures are less
painful cause system auto corrects
Previously we used width and height for thumbnails, new code ensures
1. We auto correct width and height
2. We added extra columns for thumbnail_width and height, this is determined
by actual upload and no longer passed in as a side effect
3. Optimized Image now stores filesize which can be used for analysis, decisions
Also
- fixes Android image manifest as a side effect
- fixes issue where a thumbnail generated that is smaller than the upload is no longer used
It was a dropdown to provide choices of color schemes,
and only one scheme could be shown.
With this commit, multiple color scheme previews can be displayed on
one page at the same time, making admins choose color schemes more
easily.
Theme preview windows are shrinked.
Imported default color schemes.
Co-Authored-By: Misaka 0x4e21 <misaka4e21@gmail.com>
```
change_column :topic_views, :ip_address, :inet, null: true
```
translates to
```
"ALTER TABLE \"topic_views\" ALTER COLUMN \"ip_address\" TYPE inet"
```
which locks the whole table while the migration is taking place.
### navigate_to_first_post_after_read setting for categories
When enabled on categories logged on users will return to OP after
reading the entire category. (useful for documentation categories)
### num_auto_bump_daily
Set a number of topics that will automatically bump daily on a category.
- Every 15 minutes we will check if any category has this setting
- Categories with the setting are shuffled
- We exclude pinned, closed, category description and archived topics
- Maximum of 1 topic for the list of categories is bumped till limit reached per category
- We always try to bump oldest first
- Limit is elastic using a RateLimiter that ensures that we only bump N per day
Also some minor organisation on category settings
Froze strings on category.rb
* Phase 0 for user-selectable theme components
- Drops `key` column from the `themes` table
- Drops `theme_key` column from the `user_options` table
- Adds `theme_ids` (array of ints default []) column to the `user_options` table and migrates data from `theme_key` to the new column.
- Removes the `default_theme_key` site setting and adds `default_theme_id` instead.
- Replaces `theme_key` cookie with a new one called `theme_ids`
- no longer need Theme.settings_for_client
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
* Feature: Push notifications for Android
Notification config for desktop and mobile are merged.
Desktop notifications stay as they are for desktop views.
If mobile mode, push notifications are enabled.
Added push notification subscriptions in their own table, rather than through
custom fields.
Notification banner prompts appear for both mobile and desktop when enabled.
Note, current practice if for plugins to submit PRs to core for any
migrations required for plugins, so we can better control schema.
Especially if core tables are being touched.
In this case index has close to zero cost unless akismet is installed
This reduces the akismet admin query from 20ms on every new page load to 0.5ms
Often we need to amend our schema, it is tempting to use
drop_table, rename_column and drop_column to amned schema
trouble though is that existing code that is running in production
can depend on the existance of previous schema leading to application
breaking until new code base is deployed.
The commit enforces new rules to ensure we can never drop tables or
columns in migrations and instead use Migration::ColumnDropper and
Migration::TableDropper to defer drop the db objects
This feature can be enabled by choosing a destination for the
`shared drafts category` site setting.
* Staff members can create shared drafts, choosing a destination
category for the topic when it is published.
* Shared Drafts can be viewed in their category, or above the
topic list for the destination category where it will end up.
* When the shared draft is ready, it can be published to the
appropriate category by clicking a button on the topic view.
* When published, Drafts change their timestamps to the current
time, and any edits to the original post are removed.
In the past we used suppress_from_homepage, it had mixed semantics
it would remove from category list if category list was on home and
unconditionally remove from latest.
New setting explicitly only removes from latest list but leaves the
category list alond
implemented review items.
Blocking previous codes - valid 2-factor auth tokens can only be authenticated once/30 seconds.
I played with updating the “last used” any time the token was attempted but that seemed to be overkill, and frustrating as to why a token would fail.
Translatable texts.
Move second factor logic to a helper class.
Move second factor specific controller endpoints to its own controller.
Move serialization logic for 2-factor details in admin user views.
Add a login ember component for de-duplication
Fix up code formatting
Change verbiage of google authenticator
add controller tests:
second factor controller tests
change email tests
change password tests
admin login tests
add qunit tests - password reset, preferences
fix: check for 2factor on change email controller
fix: email controller - only show second factor errors on attempt
fix: check against 'true' to enable second factor.
Add modal for explaining what 2fa with links to Google Authenticator/FreeOTP
add two factor to email signin link
rate limit if second factor token present
add rate limiter test for second factor attempts
Locking a post prevents it from being edited. This is useful if the user
has posted something which has been edited out, and the staff members don't
want them to be able to edit it back in again.
https://meta.discourse.org/t/search-logs-page/73281/11?u=techapj
This commit adds following features:
- support for tracking click through to user, tag and category
- new filter for search type (header, full page)
This commit also removes "most viewed topic" field from search logs page because we are now tracking multiple click through entities, so topic is not a special entity anymore. This also improves query perf. The query now takes `20.5ms` to runs, as opposed to `655.9ms` previously.
* Add user_home configuration option
* Use the new user_home preference to actually show the right home page
* Fix trailing whitespace
* Update user_option_serializer.rb
* Fix JavaScript default homepage tests
* Use an object instead of a giant switch
* Remove trailing whitespace
* Make the default `user_home` set to `null` instead of `0`
* Rename user_home to homepage_id
* Split alias levels in mentionable and messageable levels.
* Fixed some tests.
* Set messageable level to everyone by default.
* By defaults, groups are not mentionable or messageable.
* Made staff groups messageable by the system.
This change-set allows setting different defaults for different locales.
It also:
- Adds extensive testing around site setting validation
- raises deprecation error if site setting has the default property based on env
- relocated site settings for dev and tests in the initializer
- deprecated client_setting in the site setting's loading process
- ensure it raises when a enum site setting being set
- default_locale is promoted to `required` category.
- fixes incorrect default setting and validation
- fixes ensure type check for site settings
- creates a benchmark for site setting
- sets reasonable defaults for Chinese
There are 4 visibility levels
- public (default)
- members only
- staff
- owners
Note, admins and group owners ALWAYS have visibility to groups
Migration treated old "non public" as "members only"
Figuring out what unread topics a user has is a very expensive
operation over time.
Users can easily accumulate 10s of thousands of tracking state rows
(1 for every topic they ever visit)
When figuring out what a user has that is unread we need to join
the tracking state records to the topic table. This can very quickly
lead to cases where you need to scan through the entire topic table.
This commit optimises it so we always keep track of the "first" date
a user has unread topics. Then we can easily filter out all earlier
topics from the join.
We use pg functions, instead of nested queries here to assist the
planner.
This feature introduces the concept of themes. Themes are an evolution
of site customizations.
Themes introduce two very big conceptual changes:
- A theme may include other "child themes", children can include grand
children and so on.
- A theme may specify a color scheme
The change does away with the idea of "enabled" color schemes.
It also adds a bunch of big niceties like
- You can source a theme from a git repo
- History for themes is much improved
- You can only have a single enabled theme. Themes can be selected by
users, if you opt for it.
On a technical level this change comes with a whole bunch of goodies
- All CSS is now compiled using a custom pipeline that uses libsass
see /lib/stylesheet
- There is a single pipeline for css compilation (in the past we used
one for customizations and another one for the rest of the app
- The stylesheet pipeline is now divorced of sprockets, there is no
reliance on sprockets for CSS bundling
- CSS is generated with source maps everywhere (including themes) this
makes debugging much easier
- Our "live reloader" is smarter and avoid a flash of unstyled content
we run a file watcher in "puma" in dev so you no longer need to run
rake autospec to watch for CSS changes
- avoid access denied on bad cookie, instead just nuke it
- avoid marking a token unseen for first minute post rotation
- log path in user auth token logs
Revamped system for managing authentication tokens.
- Every user has 1 token per client (web browser)
- Tokens are rotated every 10 minutes
New system migrates the old tokens to "legacy" tokens,
so users still remain logged on.
Also introduces weekly job to expire old auth tokens.
- Regular users are not notified of whispers
- Regular users no longer have "stuck" topics in unread
- Additional tracking for staff highest post number
- Remove a bunch of unused columns in topics table
previously we supported blanket read and write for user API, this
change amends it so we can define more limited scopes. A scope only
covers a few routes. You can not grant access to part of the site and
leave a large amount of the information hidden to API consumer.
FIX: warning about popup dimensions when using facebook login
Rules are:
- On account creation we always import
- If you already have an avatar uploaded, nothing is changed
- If you have no avatar uploaded, we upload from facebook on login
- If you have no avatar uploaded, we select facebook unless gravatar already selected
This also fixes SSO issues where on account creation accounts had missing avatar uploads
This feature ensures session cookie lifespan is extended
when user is online.
Also decreases session timeout from 90 to 60 days.
Ensures all users (including logged on ones) get expiring sessions.
Users can no longer opt-in for "public" edit history
if site owner disables it.
This feature adds cost and complexity to post rendering since
user options need to be premeptively loaded for every user in the
stream. It is also confusing to explain to communities with private edit
history.
- All unsubscribes go to the exact same page
- You may unsubscribe from watching a category on that page
- You no longer need to be logged in to unsubscribe from a topic
- Simplified footer on emails
* Rearrange frontend to account for mailing list mode
* Allow update of user preference for mailing list frequency
* Add mailing list frequency estimate
* Simplify frequency estimate; disable activity summary for mailing list mode
* Remove combined updates
* Add specs for enqueue mailing list mode job
* Write mailing list method for mailer
* Fix linting error
* Account for stale topics
* Add translations for default mailing list setting
* One query for mailing list topics
* Fix failing spec
* WIP
* Flesh out html template
* First pass at text-based mailing list summary
* Add user avatar
* Properly format posts for mailing list
* Move make_all_links_absolute into Email::Styles
* Apply first_seen_at to user
* Send mailing list email summary hourly based on first_seen_at
* Branch and test cleanup
* Use existing mailing list mode estimate
* Fix failing specs
We now optionally add a Variable Email Return Path to every email we send.
This allows us to cleanly handle email bounces, which in turn will improve
deliverability.
We no longer include previous replies as "context", instead
we include and excerpt of the post being replied to at the bottom
of notifications, this information was previously missing.
Users may opt in to emailing previous replies if they wish
or opt out of "in-reply-to" which makes sense in some email clients that
are smarter about displaying a tree of replies.
- always means we always send previous replies with every email
- never means we do not
- "unless previously sent" ... is the default, in which we only email you each reply once
The default_email_previous_replies site setting can control this toggle
This commit adds a new tracking table that lets us know
- When a migration ran
- What version Discourse was at
- How long it took
- What version Rails was at
The built in tracking in Rails is very limited, does not track this info
As it stands we load up user records quite frequently on the topic pages,
this in turn pulls all the columns for the users being selected, just to
discard them after they are loaded
New structure keeps all options in a discrete table, this is better organised
and allows us to easily add more column without worrying about bloating the
user table
UX: improve messaging so notifications list is far more stable
PERF: improve performance of notifcation lookup queries
- Add feature "SetTransientHeader" that allows shipping info to server
in the next Ajax request
- remove local storage hack used for notifications
- amend lookupStale to return hydrated objects, move logic into store
- stop magically clearing various notifications (likes, invitee accepted, group_summary, granted badge)
Messages are now in 3 buckets
- Inbox for all new messages
- Sent for all sent messages
- Archive for all messages you are done with
You can select messages from your Inbox or Sent and move them to your Archive,
you can move messages out of your Archive similarly
Similar concept applied to group messages, except that archiving and unarchiving
will apply to all group members
Re-organise user page so it is easier to find interesting info
split it into tabs
- Introduce notifications and messages tabs
- Stop couting stuff for the user page to speed up rendering
- Suppress more information when viewing your own profile
Since I18n has a DB backend now, I've introduced a helper we can use to
skip overrides in certain situations. Otherwise migration from empty
databases was broken.
Group owners are regular users that can add or remove users to a group
The Admin UX allows admins to appoint group owners
The public group UX will display group owners first and unlock UI to
add and remove members
Group owners can only be appointed on non automatic groups
Group owners may not appoint another group owner
- how long were people typing?
- how long was composer open?
- how many drafts were created?
- correct, draft saved to go away after you continue typing
store in Post.find(xyz).post_stat
- Increase size of email column to varchar(513)
- Give error message on signup when email is too large
Overall impact: Low, allows signups from blocked domains. Main risk is increased spam.
- add missing index to user actions for fast retrieval by post
- add missing indexes to users for fast retrieval of staff
- only refresh topic_users liked/bookmarked cache for affected users
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar
PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
David Taylor
Robin Ward
Vinoth Kannan
Tarek Khalil
Guo Xiang Tan
Sam Saffron
Neil Lalonde
Joffrey JAFFEUX
Bianca Nenciu
Gerhard Schlager
Penar Musaraj
Rishabh
Guo Xiang Tan
Arpit Jalan
Saurabh Patel
Maja Komel
Kyle E. Mitchell
Régis Hanol
Bianca Nenciu
Kyle Zhao
Erin Kosewic
OsamaSayegh
Kris
Leo McArdle
riking
Misaka 0x4e21
Jeff Wong
OsamaSayegh
typeoneerror
Michael Howell
Erick Guan
Jakub Macina
Pat David
Claas Augner
Joe Buhlig
cpradio
James Kiesel
Thorben Egberts
Ubuntu
Jason W. May