Commit Graph

46 Commits

Author SHA1 Message Date
Loïc Guitaut f7c57fbc19 DEV: Enable `unless` cops
We discussed the use of `unless` internally and decided to enforce
available rules from rubocop to restrict its most problematic uses.
2023-02-21 10:30:48 +01:00
David Taylor 436b3b392b
DEV: Apply syntax_tree formatting to `script/*` 2023-01-09 11:13:22 +00:00
Alan Guo Xiang Tan 7c321d3aad
PERF: Update `Group#user_count` counter cache outside DB transaction ()
While load testing our user creation code path in production, we
identified that executing the DB statement to update the `Group#user_count` column within a
transaction is creating a bottleneck for us. This is because the
creation of a user and addition of the user to the relevant groups are
done in a transaction. When we execute the DB statement to update
`Group#user_count` for the relevant group, a row level lock is held
until the transaction completes. This row level lock acts like a global
lock when the server is creating users that will be added to the same
group in quick succession.

Instead of updating the counter cache within a transaction which the
default ActiveRecord `counter_cache` option does, we simply update the
counter cache outside of the committing transaction.

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>

Co-authored-by: Rafael dos Santos Silva <xfalcox@gmail.com>
2022-11-30 11:52:08 -03:00
Leonardo Mosquera bfecbde837
Fixes for vBulletin bulk importer ()
* Allow taking table prefix from env var

* FIX: remove unused column references

The columns `filedata` and `extension` are not present in a v4.2.4
database, and they aren't used in the method anyways.

* FIX: report progress for tables without imported_id

* FIX: effectively check for AR validation errors

NOTE: other migration scripts also have this problem; see /t/58202

* FIX: properly count Posts when importing attachments

* FIX: improve logging

* Remove leftover comment

* FIX: show progress when exporting Permalink file

* PERF: stream Permalink file

The current way results in tons of memory usage; write once per line instead

* Document fixes needed

* WIP - deduplicate category names

* Ignore non alphanumeric chars for grouping

* FIX: properly deduplicate user emails by merging accounts

* FIX: don't merge empty UserEmails

* Improve logging

* Merge users AFTER fixing primary key sequences

* Parallelize user merging

* Save duplicated users structure for debugging purposes

* Add progress logging for the (multiple hour) user merging step
2022-11-28 16:30:19 -03:00
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs ()
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Michael Brown 3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 ()
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Michael Maroszek 5bec0e5763
fix vbulletin importer to import unreferenced attachments () 2021-04-19 21:05:16 +02:00
Michael Maroszek 144584aacb
fix vbulletin importer to hide soft-deleted posts ()
equal to theads posts can be soft-deleted which results in a visibile = 2 state. at the moment those posts will be imported fully visible.
2021-02-12 14:29:05 +01:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan 24347ace10 FIX: Properly associate user_profiles background urls via upload id.
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Régis Hanol afa22a0c6f REFACTOR: more 'fake_email' to base importer 2018-10-22 11:12:40 +02:00
Régis Hanol 8b20e2500a
Remove unnecessary line 2018-10-19 15:48:48 +02:00
Régis Hanol 637123ff6f Merge users based on their email in vBulletin importer 2018-10-19 15:16:45 +02:00
Régis Hanol 53aa0344bf FIX: properly import vBulletin's hashed password 2018-10-18 10:22:55 +02:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
discoursehosting fc973f9363 Improve the VBulletin importer () 2018-06-12 20:41:21 +02:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. () 2017-07-28 10:20:09 +09:00
Régis Hanol 9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Régis Hanol 0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
Jay Pfaffman 3a5c0c5605 add env variables for vBulletin import script 2017-01-31 10:31:18 +05:30
Mohamad Abras bc621a704e fix close topics vb4 importer 2016-12-15 14:20:05 +02:00
Arpit Jalan 59523aef9d more improvements to vBulletin import script 2016-12-05 17:44:18 +05:30
Mohamad Abras f68194cf8e fix nil for vb4 importer 2016-12-05 04:16:59 +02:00
Mohamad Abras 52749c0121 imporoving vb4 importer 2016-12-04 00:02:47 +02:00
Arpit Jalan 92e716a1fd fix vbulletin import script 2016-09-14 08:15:48 +05:30
Arpit Jalan 201d344a2d improve vBulletin import script 2016-07-10 14:51:31 +05:30
Gerhard Schlager 6238a43f93 Fix base and vBulletin importer 2015-10-21 19:07:31 +02:00
Kane York 821124f3fd FEATURE: Skip batches if all records exist
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
Erick Guan 7737cc2828 vBulletin: ability to import attachments from filesystem 2015-05-15 19:39:15 +08:00
Gerhard Schlager a412e9bede Fix gem load order for all importers
https://meta.discourse.org/t/migrating-from-mybb/25563/8
2015-03-18 20:31:02 +01:00
Gerhard Schlager b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
Daniel Phin c35cdfcf6b Update vbulletin.rb 2015-03-10 00:31:12 +10:30
Régis Hanol 31340de446 don't break the vBulletin importer when PrettyText times out 2015-01-26 20:35:30 +01:00
Régis Hanol eecc573fbc FIX: don't break import when raw can't be preprocessed (vBulletin importer) 2015-01-21 09:36:46 +01:00
Régis Hanol 6c4d852011 Improve vBulletin importer
- FEATURE: TopicCreator now supports 'pinned_at' parameter
- FIX: 🐛 FIX TopicQuerySQL to support pinned topic older than 2010
- FIX: 🐛 Properly remove all HTML Entities from Usernames/Titles/Category Names/Groups in vBulletin importer
- FIX: 🐛 Properly handle specific vBulletin BBCode (quotes/mentions)
- FIX: 🐛 Make sure we generate a username from the name of the user instead of a fake email
- FEATURE: Allow for custom timezone in vBulletin importer
- FEATURE: Support for profile pictures/background in vBulletin importer
- FIX: 🐛 merge the categories tree to only 2 levels in vBulletin importer
2015-01-19 15:00:55 +01:00
Régis Hanol 5045d8677c new vBulletin importer (uses mysql dump instead of multiple csv files) 2014-12-22 13:22:16 +01:00
Régis Hanol d1608bdfe9 FIX: better error handling in vBulletin importer 2014-09-08 11:02:54 +02:00
Régis Hanol 4907053cc4 FIX: vBulletin pre-processing regexes order 2014-09-05 18:44:34 +02:00
Régis Hanol 85cbb001ae FIX: properly close topics in vBulletin importer
FEATURE: add backtrace when an exception happen (importers)
FIX: post-processing should also happen on first posts (vBulletin
importer)
PERF: faster topic bypass when already imported
2014-09-04 17:55:05 +02:00
Régis Hanol e823f568a7 FIX: preprocessing regexes in vBulletin importer 2014-08-29 13:11:56 +02:00
Régis Hanol 9b86d1507d VBULLETIN: add some stats about the imported groups 2014-08-25 12:33:08 +02:00
Régis Hanol 7aaf718cf3 update bbcode dialect and fix vBulletin importer 2014-08-25 10:48:29 +02:00
Régis Hanol 841d547a48 FIX: vBulletin wasn't importing posters... 2014-08-20 23:04:43 +02:00
Régis Hanol 66aaa9329e FEATURE: vBulletin importer 2014-08-18 13:04:08 +02:00
Régis Hanol fab2c8f816 Create vbulletin.rb 2014-07-09 22:44:06 +02:00