Related to https://meta.discourse.org/t/host-is-invalid-error-when-tld-is-longer-than-7-characters/46081.
Using Discourse `v2.4.0.beta2 +119`, I can't add an host (when embedding, cf. `/admin/customize/embedding`) ending with `.engineering`.
Turns out current regex limits to 10 characters.
Fix is dumb: it only allows for up to 24 chars, which is the **current** max TLD length, see https://stackoverflow.com/a/22038535/1907212.
---
Maybe a better (and longer-term) fix would be to allow for up to 64 chars, which I understand comes from the RFC.
I'm not at ease with regexes, so can't be sure about it, but [this suggestion](https://meta.discourse.org/t/host-is-invalid-error-when-tld-is-longer-than-7-characters/46081/8?u=julienma) seems pretty good:
> rules of DNS labels are:
>
> - All labels are 1 to 63 characters, case insensitive A to Z, 0 to 9 and - (hyphen), all from ASCII.
> - No labels may start with a hyphen.
> - No top level domain label may start with a number.
>
>That means a regexp for a valid domain name would look like:
>
>`/^([a-z0-9][a-z0-9-]{0,62}\.)+[a-z][a-z0-9-]{0,62}\.?$/`
>
>Domains that are just a TLD are sufficiently bizarre as to be worth ignoring.
When `SiteSetting.embed_truncate` is enabled (by default), the truncated
string is mutatable and does not raise an error.
However, when the setting is disabled, the `contents` string is frozen
and immutable, and will raise a `FrozenError`.
No need to let notifications stay around when users can't access
a topic after it was converted into a PM or posts were moved
into a restricted topic.
Also makes sure that moving to a new topic correctly uses the
guardian for the first post by enqueuing jobs outside of a
transaction.
* FEATURE: Allow choice of category when making a PM public
Previously it would default to uncategorized, which was not ideal on
some forums. This gives the staff member more choice about what they'd
like to do.
* Make the optional category more explicit
* Joffrey's feedback
Previously, users who had any penalties (were silenced or suspended)
were not allowed to promote to Trust Level 3.
There is also a more subtle change here: if users were silenced or
suspended and then the operation was reverted (user was un-silenced
or un-suspended), then it would have been like the user was never
penalized in the first place. This is no longer the case. To forgive a
user earlier, administrators can use "Clear Penalty History" feature.
Lastly, Jobs::UnsilenceUsers will automatically unsilence any users who
should no longer be silenced (silenced_till < now()). This made it so
silence_count - unsilence_count == 0 for any user who is not silenced,
which defeated the purpose of this TL3 requirement.
DEV: deprecate `invite.via_email` in favor of `invite.emailed_status`
This commit adds a new column `emailed_status` in `invites` table for
tracking email sending status.
0 - not required
1 - pending
2 - bulk pending
3 - sending
4 - sent
For normal email invites, invite record is created with emailed_status
set to 'pending'.
When bulk invites are sent invite record is created with emailed_status
set to 'bulk pending'.
For invites that generates link, invite record is created with
emailed_status set to 'not required'.
When invite email is in queue emailed_status is updated to 'sending'
Once the email is sent via `InviteEmail` job the invite emailed_status
is updated to 'sent'.
This is a low severity security fix because it requires a logged in
admin user to update a site setting via the API directly to an invalid
value.
The fix adds validation for the affected site settings, as well as a
secondary fix to prevent injection in the event of bad data somehow
already exists.
In order for this to work the Backuper stores a couple of site settings
in the new backup_metadata table, because the old setting values might
not be available on restore anymore.
Context: https://meta.discourse.org/t/121589
This new setting option lets group owners message/mention large groups
without granting that privilege to all members.
Groups can now be marked as visible to "logged on users". All automatic groups (except `everyone`) are now visible to "logged on users", previously they were marked as public but suppressed in the group page for non-staff.
If a database exception is raised ActiveRecord will always rollback
even if caught.
Instead we build the query in manual SQL and DO NOTHING when there's a
conflict. If we detect nothing was done, perform an update.
This allows you to temporarily disable components without having to remove them from a theme.
This feature is very handy when doing quick fix engineering.
The behaviour of #TERM in search has been amended
1. We try category or subcategory slugs
2. We try tags
3. We try tag-groups
The term `hello #my-group` will search for all posts tagged with any of
the tags in the tag group `My Group`
Future work may be introducing a slug cache here or caching it in the table
but the assumption is that the number of tag groups will not be huge
Adds a second factor landing page that centralizes a user's second factor configuration.
This contains both TOTP and Backup, and also allows multiple TOTP tokens to be registered and organized by a name. Access to this page is authenticated via password, and cached for 30 minutes via a secure session.
Previously users were still allowed to create topic via API even if
uncategorized was disabled.
Not 100% happy with all this special casing, but I guess we have to do
something.
This also splits up a mega spec now that we have fab! into a more easy to
understand structure (I hope)
followup to #bc03c509
There were 2 problems
1. VALUES was not properly getting multiple results ... we need (1),(2),(3)
not (1,2,3)
2. changes was mistakenly lazy evaluated eg `changed ||=` which meant some
queries were not running
* Remove unused method
* Prefabricate user in category_user_spec.rb
* FIX: Remove notification_level from category_users unique indexes
* FIX: CategoryUser#batch_set wasn't updating pre-existing records
* Improve tests for CategoryUser#batch_set
* FIX: changed was being reported incorrectly
* DEV: Rewrote query to do a bulk insert
* DEV: remove unnecessary parentheses
Sometimes we would like to create a base image without any DB access, this
assists in creating custom base images with custom plugins that already
includes `public/assets`
Following this change set you can run:
```
SPROCKETS_CONCURRENT=1 DONT_PRECOMPILE_CSS=1 SKIP_DB_AND_REDIS=1 RAILS_ENV=production bin/rake assets:precompile
```
Then it is straight forward to create a base image without needing a DB or
Redis.
We now treat any external_id of blank string (" " or " " or "", etc) or a
invalid word (none, nil, blank, null) - case insensitive - as invalid.
In this case the client will see "please contact admin" the logs will explain
the reason clearly.
- Correct create_category_definition to skip validations and use a
transaction, no longer able to create corrupt topics
- ensure_consistency now clears topic_id if pointing at deleted or missing
topic_id
- Stop creating category definition topics for uncategorized
* Support private uploads in S3
* Use localStore for local avatars
* Add job to update private upload ACL on S3
* Test multisite paths
* update ACL for private uploads in migrate_to_s3 task
* Expose a new plugin outlet. Pass group model to the group-member-dropdown so it can be accessed by plugins
* Added controller tests for group custom fields. update custom fields when updating a group
During profiling looking up topic users popped up as a hot path, this
change more than halved the amount of work it does
It reduces object allocations and method calls and avoids repeate translation
of common terms
You can now add javascript files under `/javascripts/*` in a theme, and they will be loaded as if they were included in core, or a plugin. If you give something the same name as a core/plugin file, it will be overridden. Support file extensions are `.js.es6`, `.hbs` and `.raw.hbs`.
Previously theme setting descriptions were defined in the `settings.yml` file like this:
```
setting_name:
default: "My Default Value"
description:
en: "English description"
fr: "French description"
```
This commit allows developers to store the localised descriptions in the theme locale files instead:
```
en:
theme_metadata:
description: Theme Description
settings:
setting_name: "The localised description for setting_name"
```
This feature allows end users to "defer" topics by marking them unread
The functionality is default disabled. This also introduces the new site
setting default_other_enable_defer: to enable this by default on new user
accounts.
Previously we used custom fields to denote a user was anonymous, this was
risky in that custom fields are prone to race conditions and are not
properly dedicated, missing constraints and so on.
The new table `anonymous_users` is properly protected. There is only one
possible shadow account per user, which is enforced using a constraint.
Every anonymous user will have a unique row in the new table.
Adds `DISCOURSE_MESSAGE_BUS_REDIS_ENABLED` env var, that when set
to true, will allow Discourse to connect to a different redis
instance for MessageBus needs.
When enabled you can configure the same env vars user for redis,
but prefixed by `MESSAGE_BUS`, eg:
`DISCOURSE_MESSAGE_BUS_REDIS_HOST`
The `AutoQueueHandler` will ignore really old flags. In that case, don't
notify the user that the moderator is looking into it. They probably
never saw it because it didn't meet the reviewable minimum priority.
* FEATURE: detect theme errors and catch them
* Bump COMPILER_VERSION
* Feedback
* Override eslint no console for one line
* Can't use our ajax method
* remove emoji from translation file
User.publish_notifications_state is called every time a notification is
created, this can become a very critical code path.
On some heavy notification related sites this can be a major CPU user on PG
This index makes it much cheaper to publish notification state, cause a
simple index lookup does the trick.
We were blocking user registrations with same username and password,
but allowing usernames to be changed to be same as password later.
Also disallow names to be the same as password.
There was a race condition when 2 invites existed for 1 user where in some
cases data from both invites would be used for the redeem. Depending on DB
ordering.
Fix is to delete duplicate invites earlier in the process prior to
`redeem_from_email` being called.
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
Before: 6:05
After: 5:42
Featuring topics for `list/categories` is a very expensive operation that
happened each time we created a topic. This introduces a test only bypass
This is a feature that used to be present in discourse-assign but is
much easier to implement in core. It also allows a topic to be assigned
without it claiming for review and vice versa and allows it to work with
category group reviewers.
We found score hard to understand. It is still there behind the scenes
for sorting purposes, but it is no longer shown.
You can now filter by minimum priority (low, med, high) instead of
score.
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup
This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class
It also removes a couple of risky stubs from the auth provider test
This change shows a notification number besides the flag icon in the
post menu if there is reviewable content associated with the post.
Additionally, if there is pending stuff to review, the icon has a red
background.
We have also removed the list of links below a post with the flag
status. A reviewer is meant to click the number beside the flag icon to
view the flags. As a consequence of losing those links, we've removed
the ability to undo or ignore flags below a post.
Hidden (staff-only) post actions are whisper posts with no content, that
are later transformed by the client into post actions (discourse-assign
uses this).
After careful analysis of large data-sets it became apparent that avg_time
had no impact whatsoever on "best of" topic scoring. Calculating avg_time
was a very costly operation especially on large databases.
We have some longer term plans of introducing other weighting that is read
time based into our scoring for "best of" and "top" topics, but in the
interim to stop a large amount of work that is not achieving any value we
are removing the jobs.
Column removal will follow once we decide on a new replacement metric.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
Minor fixes to add Rails 6 support to Discourse, we now will boot
with RAILS_MASTER=1, all specs pass
Only one tiny deprecation left
Largest change was the way ActiveModel:Errors changed interface a
bit but there is a simple backwards compat way of working it
This change automatically resizes icons for various purposes. Admins can now upload `logo` and `logo_small`, and everything else will be auto-generated. Specific icons can still be uploaded separately if required.
## Core
- Adds an SiteIconManager module which manages automatic resizing and fallback
- Icons are looked up in the OptimizedImage table at runtime, and then cached in Redis. If the resized version is missing for some reason, then most icons will fall back to the original files. Some icons (e.g. PWA Manifest) will return `nil` (because an incorrectly sized icon is worse than a missing icon).
- `SiteSetting.site_large_icon_url` will return the optimized version, including any fallback. `SiteSetting.large_icon` continues to return the upload object. This means that (almost) no changes are required in core/plugins to support this new system.
- Icons are resized whenever a relevant site setting is changed, and during post-deploy migrations
## Wizard
- Allows `requiresRefresh` wizard steps to reload data via AJAX instead of a full page reload
- Add placeholders to the **icons** step of the wizard, which automatically update from the "Square Logo"
- Various copy updates to support the changes
- Remove the "upload-time" resizing for `large_icon`. This is no longer required.
## Site Settings UX
- Move logo/icon settings under a new "Branding" tab
- Various copy changes to support the changes
- Adds placeholder support to the `image-uploader` component
- Automatically reloads site settings after saving. This allows setting placeholders to change based on changes to other settings
- Upload site settings will be assigned a placeholder if SiteIconManager `responds_to?` an icon of the same name
## Dashboard Warnings
- Remove PWA icon and PWA title warnings. Both are now handled automatically.
## Bonus
- Updated the sketch logos to use @awesomerobot's new high-res designs
On busy sites, concurrent requests to insert into post_timings can
occur, which was dealt with using Ruby exceptions.
This moves the handling to PostgreSQL which makes it a bit faster,
and prevents a spam of ERROR in the database logs.
If a tag group is set to only be visible to staff, and is restricted
to a category that is visible by everyone, the tags in the group were
being shown on the /tags page. They weren't visible anywhere else.
This commit fixes it so they don't show on the /tags page.
This is for backwards compatibility purposes. Even if `Upload#url` has a
format that we don't recognize, we should still return the upload object
as long as the upload record is present.
If you turn it on now, default all users to approved since they were
previously. Also support approving a user that doesn't have a reviewable
record (it will be created first.)
This also includes a refactor to move class method calls to
`DiscourseEvent` into an initializer. Otherwise the load order of
classes makes a difference in the test environment and some settings
might be triggered and others not, randomly.
Theme developers can include any number of scss files within the /scss/ directory of a theme. These can then be imported from the main common/desktop/mobile scss.
restructure query so it avoids ORs
It appears postgres is picking suboptimal indexes if too many ORs exist
despite how trivial the condition is.
This bypasses conditional in the query and evals them upfront.
On meta for my user this made a 10x perf difference.
This boils down to either having `OR u.admin` or not having `OR u.admin` in
the query.
Note, to avoid race conditions we are setting last_unread to 10 minutes ago
if there is nothing unread.
This is safer in case of in progress transactions
we don't want to lose unread for any window of time.
This optimisation avoids large scans joining the topics table with the
topic_users table.
Previously when a user carried a lot of read state we would have to join
the entire read state with the topics table. This operation would slow down
home page and every topic page. The more read state you accumulated the
larger the impact.
The optimisation helps people who clean up unread, however if you carry
unread from years ago it will only have minimal impact.
Sometimes sidekiq is so fast that it starts jobs before transactions
have comitted. This patch moves the message bus stuff until after things
have comitted.
"Rejecting" a user in the queue is equivalent to deleting them, which
would then making it impossible to review rejected users. Now we store
information about the user in the payload so if they are deleted things
still display in the Rejected view.
Secondly, if a user is destroyed outside of the review queue, it will
now automatically "Reject" that queue item.
Conversely, if a user is deactivated the reviewable should automatically
be rejected.
Before this fix, if a user was not active they'd still show in the
review queue but without an "Approve" button which was confusing.
Previously every rebake would remove and recreate records in this table
This caused created_at and updated_at to keep changing
Yes, I know the SQL is somewhat complex, but this makes quote extraction
more efficient cause we do everything in 2 round trips.
This also removes some concurrency protection we should no longer need
Some sites have external URLs that don't even match `%/uploads/%' and
some sites surprise me with URLs that contains the default path when it
is a site in a multisite cluster. We can't do anything about those.
Adds the parallel_tests gem, and redis/postgres configuration for running rspec tests in parallel. To use:
```
rake parallel:rake[db:create]
rake parallel:rake[db:migrate]
rake parallel:spec
```
This brings the test suite from 12m20s to 3m11s on my macOS machine
This commit fixes the follow quality issue with `PostSearchData#raw_data`:
1. URLs are being tokenized and links with similar href and characters
are being duplicated in the raw data.
`Post#cooked`:
```
<p><a href=\"https://meta.discourse.org/some.png\" class=\"onebox\" target=\"_blank\" rel=\"nofollow noopener\">https://meta.discourse.org/some.png</a></p>
```
`PostSearchData#raw_data` Before:
```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png discourse org/some png https://meta.discourse.org/some.png discourse org/some png
```
`PostSearchData#raw_data` After:
```
This is a test topic 0 Uncategorized https://meta.discourse.org/some.png meta discourse org
```
2. Ligthbox being included in search pollutes the
`PostSearchData#raw_data` unncessarily.
From 28 March 2018 to 28 March 2019, searches for the term `image` on
`meta.discourse.org` had a click through rate of 2.1%. Non-lightboxed images are not included in indexing for search yet we were indexing content within a lightbox. Also, search for terms like `image` was affected we were using `Pasted image` as the filename for
uploads that were pasted.
`Post#cooked`
```
<p>Let me see how I can fix this image<br>\n<div class=\"lightbox-wrapper\"><a class=\"lightbox\" href=\"https://meta.discourse.org/some.png\" title=\"some.png\" rel=\"nofollow noopener\"><img src=\"https://meta.discourse.org/some.png\" width=\"275\" height=\"299\"><div class=\"meta\">\n<svg class=\"fa d-icon d-icon-far-image svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#far-image\"></use></svg><span class=\"filename\">some.png</span><span class=\"informations\">1750×2000</span><svg class=\"fa d-icon d-icon-discourse-expand svg-icon\" aria-hidden=\"true\"><use xlink:href=\"#discourse-expand\"></use></svg>\n</div></a></div></p>
```
`PostSearchData#raw_data` Before:
```
This is a test topic 0 Uncategorized Let me see how I can fix this image some.png png https://meta.discourse.org/some.png discourse org/some png some.png png 1750×2000
```
`PostSearchData#raw_data` After:
```
This is a test topic 0 Uncategorized Let me see how I can fix this image
```
In terms of indexing performance, we now have to parse the given HTML
through nokogiri twice. However performance is not a huge worry here since a string length of 194170 takes only 30ms
to scrub plus the indexing takes place in a background job.
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.
Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
- s3_force_path_style was added as a Minio specific url scheme but it has never been well supported in our code base.
- Our new migrate_to_s3 rake task does not work reliably with path style urls too
- Minio has also added support for virtual style requests i.e the same scheme as AWS S3/DO Spaces so we can rely on that instead of using path style requests.
- Add migration to drop s3_force_path_style from the site_settings table
* improved emoji support
- always optimize images as part of the task
- use the unicode standard ordering/naming for sections
* UX: more height for when there are recently used
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by
* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
* FEATURE: Exposing a way to add a generic report filter
## Why do we need this change?
Part of the work discussed [here](https://meta.discourse.org/t/gain-understanding-of-file-uploads-usage/104994), and implemented a first spike [here](https://github.com/discourse/discourse/pull/6809), I am trying to expose a single generic filter selector per report.
## How does this work?
We basically expose a simple, single generic filter that is computed and displayed based on backend values passed into the report.
This would be a simple contract between the frontend and the backend.
**Backend changes:** we simply need to return a list of dropdown / select options, and enable the report's newly introduced `custom_filtering` property.
For example, for our [Top Uploads](https://github.com/discourse/discourse/pull/6809/files#diff-3f97cbb8726f3310e0b0c386dbe89e22R1423) report, it can look like this on the backend:
```ruby
report.custom_filtering = true
report.custom_filter_options = [{ id: "any", name: "Any" }, { id: "jpg", name: "JPEG" } ]
```
In our javascript report HTTP call, it will look like:
```js
{
"custom_filtering": true,
"custom_filter_options": [
{
"id": "any",
"name": "Any"
},
{
"id": "jpg",
"name": "JPG"
}
]
}
```
**Frontend changes:** We introduced a generic `filter` param and a `combo-box` which hooks up into the existing framework for fetching a report.
This works alright, with the limitation of being a single custom filter per report. If we wanted to add, for an instance a `filesize filter`, this will not work for us. _I went through with this approach because it is hard to predict and build abstractions for requirements or problems we don't have yet, or might not have._
## How does it look like?
## More on the bigger picture
The major concern here I have is the solution I introduced might serve the `think small` version of the reporting work, but I don't think it serves the `think big`, I will try to shed some light into why.
Within the current design, It is hard to maintain QueryParams for dynamically generated params (based on the idea of introducing more than one custom filter per report).
To allow ourselves to have more than one generic filter, we will need to:
a. Use the Route's model to retrieve the report's payload (we are now dependent on changes of the QueryParams via computed properties)
b. After retrieving the payload, we can use the `setupController` to define our dynamic QueryParams based on the custom filters definitions we received from the backend
c. Load a custom filter specific Ember component based on the definitions we received from the backend
* First take
* Add support for sprites in themes
Automatically register any custom icons added via themes or plugins
* Fix theme sprite caching
* Simplify test
* Update lib/svg_sprite/svg_sprite.rb
Co-Authored-By: pmusaraj <pmusaraj@gmail.com>
* Fix /svg-sprite/search request
Julien Ma
Arpit Jalan
Vinoth Kannan
Kyle Zhao
Gerhard Schlager
Robin Ward
Dan Ungureanu
David Taylor
Joffrey JAFFEUX
Bianca Nenciu
Penar Musaraj
Ralph Rooding
OsamaSayegh
Sam Saffron
Jeff Wong
Daniel Waterworth
Guo Xiang Tan
Maja Komel
Bianca Nenciu
Roman Rizzi
Régis Hanol
Rafael dos Santos Silva
Neil Lalonde
Tarek Khalil
James Haggerty
Guo Xiang Tan
Tim Lange
Rishabh
Saurabh Patel
