The current implementation adds a "note" for chat messages with empty
messages, however chat messages with only uploads are allowed. This change
allows such messages to be imported.
* DEV: WIP generic bulk chat import support
This first iteration implements bulk import for:
* direct_messages
* chat_channels
* user_chat_channel_memberships
* chat_threads
* user_chat_thread_memberships
* chat_messages
* chat_reactions
* chat_mentions
* DEV: Refactor raw placeholder interpolation to support chat messages
This change adds support for chat message placeholder interpolation
and switches to using `Chat::Message.cook` for cooking in the interim
instead of hand-cooking chat messages like we currently do for posts
* DEV: Extend upload references import to support chat message uploads
* DEV: Explicity set chat retention
- Set both channel and dm chats to 0
- Add temporary workaround for testing only chat imports
* DEV: Compute channel and thread membership metadata
Compute and set various user channel/thread membership stats and
remove hardcoded test index seed data
* FIX: Fix chat reactions import
Allow multiple reactions on a message by a user
Currently, quotes imported via generic bulk import script do not include
references to the quoted post. This change includes both topic and post attributes
in a quote if the placeholder metadata includes a `post_id`
Adds the “s3_uploads” option (default: false) to the discourse backup script, which temporarily forces the inclusion of S3 uploads in the backup.
Also removed all unnecessary “require” statements as loading Rails is sufficient.
The rest of the changes were made by the linter.
* user_profiles - `location`
* users - `date_of_birth`
* topics - `pinned_at`, `pinned_until`, `pinned_globally`
This also include changes to correctly import PMs. Currently PM topics
are skipped because of a check in `import_users` step which requires `category_id`
to be present.
This commit updates `script/bench.rb` to only support Unicorn as the web
server. We don't intend to run Puma in production anytime soon so it is
pointless for us to maintain Puma related code.
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.
This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
* FEATURE: Import into `category_users` table
* FIX: Failed to import `user_options` unless `timezone` was set
* FIX: Prevent reusing original `id` from intermediate DB in `user_fields`
* FEATURE: Order posts by `post_nuber` if available
* FEATURE: Allow `[mention]` placeholder to reference users by"id" or "name" (username)
* FEATURE: Support `[quote]` placeholders in posts
* FEATURE: Support `[link]` placeholders in posts
* FEATURE: Support all kinds of permalinks and remove support for `old_relative_url`
* PERF: Speed up pre-cooking by removing DB lookups
* Print instructions when the `sqlite3` gem can't be loaded
* Use `display_filename` instead of `filename` if available
* Support uploading for a multisite
Why this change?
In https://www.postgresql.org/docs/current/non-durability.html, it is
recommended to turn off `synchronous_commit` in environments where
durability is not important. The `start_test_db.rb` script is mainly
used in the CI environment where durability is not important at all.
Previously only Sidekiq was allowed to generate more than one optimized image at the same time per machine. This adds an easy mechanism to allow the same in rake tasks and other tools.
The regen_ember_5_lockfile script was actually just duplicating the ember3 lockfile without changes 🤦♂️. This commit fixes that, and updates the ember-version-enforcement workflow to detect lockfile issues in future.
Why this change?
`github.job` returns the `job_id` per the docs but it doesn't actually
return the id of the job but instead returns the job's name strangely.
Per https://github.com/orgs/community/discussions/8945, there is no way
to get the `job_id` from the existing contexts in the actions run.
Therefore, we have to hit Github's API to fetch it. Not ideal but no
way around this.
Before this commit, this output is possible:
```
Rewriting all occurrences of STRING1 to STRING2
THIS TASK WILL REWRITE DATA, ARE YOU SURE (type YES)
WILL RUN ON ALL 1 DBS
```
Which, when run from a script, might lead one to believe that YES was
automatically inserted into STDIN and the script is continuing.
Turns out this isn't the case so the obvious expectation is broken.
This commit swaps the order of those last lines to make it clear that
the script is blocked on input.
Notable changes:
* Imports a lot more tables from core and plugins
* site settings
* uploads with necessary upload references
* groups and group members
* user profiles
* user options
* user fields & values
* muted users
* user notes (plugin)
* user followers (plugin)
* user avatars
* tag groups and tags
* tag users (notification settings for tags / user)
* category permissions
* polls with options and votes
* post votes (plugin)
* solutions (plugin)
* gamification scores (plugin)
* events (plugin)
* badges and badge groupings
* user badges
* optimized images
* topic users (notification settings for topics)
* post custom fields
* permalinks and permalink normalizations
* It creates the `migration_mappings` table which is used to store the mapping for a handful of imported tables
* Detects duplicate group names and renames them
* Pre-cooking for attachments, images and mentions
* Outputs instructions when gems are missing
* Supports importing uploads from a DB generated by `uploads_importer.rb`
* Checks that all required plugins exists and enables them if needed
* A couple of optimizations and additions in `import.rake`
This script preprocesses all uploads within a intermediate DB (output of converters) and uploads those files to S3. It does the same for optimized images. This speeds up migrations when you have to run them multiple times, because you only have to preprocess and upload the files once.
This script is very hacky and mostly undocumented for now. That will change in the future.
This commit introduces the scaffolding for us to easily switch between Ember 3.28 and Ember 5 on the `main` branch of Discourse. Unfortunately, there is no built-in system to apply this kind of flagging within yarn / ember-cli. There are projects like `ember-try` which are designed for running against multiple version of a dependency, but they do not allow us to 'lock' dependency/sub-dependency versions, and are therefore unsuitable for our use in production.
Instead, we will be maintaining two root `package.json` files, and two `yarn.lock` files. For ember-3, they remain as-is. For ember5, we use a yarn 'resolution' to override the version for ember-source across the entire yarn workspace.
To allow for easy switching with minimal diff against the repository, `package.json` and `yarn.lock` are symlinks which point to `package-ember3.json` and `yarn-ember3.lock` by default. To switch to Ember 5, we can run `script/switch ember version 5` to update the symlinks to point to `package-ember5.json` and `package-ember3.json` respectively. In production, and when using `bin/ember-cli` for development, the ember version can also be upgraded using the `EMBER_VERSION=5` environment variable.
When making changes to dependencies, these should be made against the default `ember3` versions, and then `script/regen_ember_5_lockfile` should be used to regenerate `yarn-ember5.lock` accordingly. A new 'Ember Version Lockfiles' GitHub workflow will automate this process on Dependabot PRs.
When running a local environment against Ember 5, the two symlink changes will show up as git diffs. To avoid us accidentally committing/pushing that change, another GitHub workflow is introduced which checks the default Ember version and raises an error if it is greater than v3.
Supporting two ember versions simultaneously obviously carries significant overhead, so our aim will be to get themes/plugins updated as quickly as possible, and then drop this flag.
The priority field in an SRV RR indicates a preferential order at which
the underlying targets should be utilised. We need to prefer healthy
services in order of priority, where 0 is highest.
Prior to this commit, we relied on whatever order the
dnsclient.getresources method returned. As it turns out, this assumption
is incorrect. The order returned is likely whatever order the system
resolver received DNS responses in, which may not be ordered according
to the spec.
This introduces a ResolvedAddress type which holds the priority value
for SRV targets, or a stand-in priority of zero for A/AAAA RRs. This
type is used as a return value from the underlying name resolution
routines in Name and SRVName.
In this manner, all ordering by priority and resolved time can be
performed directly within the ResolverCache class and calling code can
continue to be none-the-wiser.
Before sorting, we still ensure that we only consider targets with a
priority within the given threshold as previously implemented.
See t/115911.
These updates significantly improve IDE tooling for imports across the Discourse core codebase, and also for framework packages. The `@types/ember-*` packages are a temporary solution until we get onto Ember 5, which ships its types in the main package.
The previous approach of having jsconfig files in each package directory did work, but once you start adding all the possible interlinks between them, we hit the file count limit of VSCode's tooling (because it counts every file for every jsconfig its referenced in). Having one file at the root means that a single file can apply to all core packages and plugins.
Long-term, to get the same functionality for all themes/plugins, we may need to look at building/publishing a Discourse types package which can be added to theme/plugin package.json files for development purposes.
Discourse stopped using PostUpload in 9db8f00b3d. Since then, these importers have been writing to the table, but any data was totally unused. This commit updates the easy cases to use UploadReference, and adds an error to the discourse_merger import script, which needs more significant work.
Selase Krakani
Gerhard Schlager
Neil Lalonde
Michael Sandler
Daniel Waterworth
Régis Hanol
Loïc Guitaut
Ítalo Alves
Alan Guo Xiang Tan
Osama Sayegh
carehabit
David Taylor
Natalie Tay
Ted Johansson
Sebastian Wagner
Jarek Radosz
Kelv
Leonardo Mosquera
Keegan George
Michael Fitz-Payne
Constanza