Constants should always be only assigned once. The logical OR assignment
of a constant is a relic of the past before we used zeitwerk for
autoloading and had bugs where a file could be loaded twice resulting in
constant redefinition warnings.
The current implementation adds a "note" for chat messages with empty
messages, however chat messages with only uploads are allowed. This change
allows such messages to be imported.
* DEV: WIP generic bulk chat import support
This first iteration implements bulk import for:
* direct_messages
* chat_channels
* user_chat_channel_memberships
* chat_threads
* user_chat_thread_memberships
* chat_messages
* chat_reactions
* chat_mentions
* DEV: Refactor raw placeholder interpolation to support chat messages
This change adds support for chat message placeholder interpolation
and switches to using `Chat::Message.cook` for cooking in the interim
instead of hand-cooking chat messages like we currently do for posts
* DEV: Extend upload references import to support chat message uploads
* DEV: Explicity set chat retention
- Set both channel and dm chats to 0
- Add temporary workaround for testing only chat imports
* DEV: Compute channel and thread membership metadata
Compute and set various user channel/thread membership stats and
remove hardcoded test index seed data
* FIX: Fix chat reactions import
Allow multiple reactions on a message by a user
Currently, quotes imported via generic bulk import script do not include
references to the quoted post. This change includes both topic and post attributes
in a quote if the placeholder metadata includes a `post_id`
* user_profiles - `location`
* users - `date_of_birth`
* topics - `pinned_at`, `pinned_until`, `pinned_globally`
This also include changes to correctly import PMs. Currently PM topics
are skipped because of a check in `import_users` step which requires `category_id`
to be present.
This SQL tries to insert as much data as possible into the `user_stats` table by either calculating or by approximating stats based on existing. It also fixes an error in the calculation of `reply_count`which mistakenly contained all posts, not just replies.
This change also disables some steps in the `import:ensure_consistency` rake task by setting the `SKIP_USER_STATS` env variable. Otherwise, the rake task will overwrite the calculated data in the `user_stats` table with inaccurate data. I'm not changing or removing the logic from the rake task yet because other bulk import scripts seem to depend on it.
* FEATURE: Import into `category_users` table
* FIX: Failed to import `user_options` unless `timezone` was set
* FIX: Prevent reusing original `id` from intermediate DB in `user_fields`
* FEATURE: Order posts by `post_nuber` if available
* FEATURE: Allow `[mention]` placeholder to reference users by"id" or "name" (username)
* FEATURE: Support `[quote]` placeholders in posts
* FEATURE: Support `[link]` placeholders in posts
* FEATURE: Support all kinds of permalinks and remove support for `old_relative_url`
* PERF: Speed up pre-cooking by removing DB lookups
* Print instructions when the `sqlite3` gem can't be loaded
* Use `display_filename` instead of `filename` if available
* Support uploading for a multisite
Previously only Sidekiq was allowed to generate more than one optimized image at the same time per machine. This adds an easy mechanism to allow the same in rake tasks and other tools.
Notable changes:
* Imports a lot more tables from core and plugins
* site settings
* uploads with necessary upload references
* groups and group members
* user profiles
* user options
* user fields & values
* muted users
* user notes (plugin)
* user followers (plugin)
* user avatars
* tag groups and tags
* tag users (notification settings for tags / user)
* category permissions
* polls with options and votes
* post votes (plugin)
* solutions (plugin)
* gamification scores (plugin)
* events (plugin)
* badges and badge groupings
* user badges
* optimized images
* topic users (notification settings for topics)
* post custom fields
* permalinks and permalink normalizations
* It creates the `migration_mappings` table which is used to store the mapping for a handful of imported tables
* Detects duplicate group names and renames them
* Pre-cooking for attachments, images and mentions
* Outputs instructions when gems are missing
* Supports importing uploads from a DB generated by `uploads_importer.rb`
* Checks that all required plugins exists and enables them if needed
* A couple of optimizations and additions in `import.rake`
This script preprocesses all uploads within a intermediate DB (output of converters) and uploads those files to S3. It does the same for optimized images. This speeds up migrations when you have to run them multiple times, because you only have to preprocess and upload the files once.
This script is very hacky and mostly undocumented for now. That will change in the future.
Discourse stopped using PostUpload in 9db8f00b3d. Since then, these importers have been writing to the table, but any data was totally unused. This commit updates the easy cases to use UploadReference, and adds an error to the discourse_merger import script, which needs more significant work.
* Allow taking table prefix from env var
* FIX: remove unused column references
The columns `filedata` and `extension` are not present in a v4.2.4
database, and they aren't used in the method anyways.
* FIX: report progress for tables without imported_id
* FIX: effectively check for AR validation errors
NOTE: other migration scripts also have this problem; see /t/58202
* FIX: properly count Posts when importing attachments
* FIX: improve logging
* Remove leftover comment
* FIX: show progress when exporting Permalink file
* PERF: stream Permalink file
The current way results in tons of memory usage; write once per line instead
* Document fixes needed
* WIP - deduplicate category names
* Ignore non alphanumeric chars for grouping
* FIX: properly deduplicate user emails by merging accounts
* FIX: don't merge empty UserEmails
* Improve logging
* Merge users AFTER fixing primary key sequences
* Parallelize user merging
* Save duplicated users structure for debugging purposes
* Add progress logging for the (multiple hour) user merging step
`ActiveRecord::Base.connection_config` has been deprecated since Rails
6.1 and was completely removed from Rails 7.
Instead we need to use
`ActiveRecord::Base.connection_db_config.configuration_hash`.
Import scripts were forgotten when we did the Rails 7 upgrade, this
patch fixes them.
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.
Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.
Deprecated the old location in case someone is using it in a plugin.
No functionality change is in this commit.
Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
cocococosti
Gerhard Schlager
Alan Guo Xiang Tan
Selase Krakani
Loïc Guitaut
Ítalo Alves
Jarek Radosz
David Taylor
Leonardo Mosquera
Michael Brown
Peter Zhu