Commit Graph

129 Commits

Author SHA1 Message Date
Kane York 869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Martin Brennan 867bc3b48e
FIX: Change base importer to create new Bookmark records (#9603)
Also add a spec and fixture with a mock importer that we can use to test the create_X methods of the base importer
2020-05-01 11:34:55 +10:00
Gerhard Schlager d216483c53 FIX: Importing with pgbouncer failed
Checking if all records have been imported uses a temp table in PostgreSQL. This fails when pgbouncer is used unless the temp table is created inside a transaction.
2020-03-26 16:41:09 +01:00
Gerhard Schlager 5b2b769eb7 DEV: Ensure uploads aren't deleted during imports
Sidekiq might delete uploads if you, for some reason, create upload records before using them in posts.
2020-03-24 17:14:16 +01:00
Gerhard Schlager ba1b840816 DEV: Don't deactivate suspended users during import
Otherwise a cleanup job might delete those deactivated users.
2020-03-14 00:00:36 +01:00
Gerhard Schlager edc8d58ac3 FEATURE: Add site setting to disable staged user cleanup
... and disabled the cleanup during imports, otherwise a running Sidekiq might delete users before posts are created
2020-03-09 10:26:41 +01:00
Régis Hanol 0843e3e6ce FIX: add support for sub-sub-categories in base_importer
Also delegates 'post_already_imported?' and 'user_already_imported?' to the base importer.
2020-02-05 10:40:28 +01:00
Sam Saffron 0c52537f10 DEV: update rubocop to version 0.77
We like to stay as close as possible to latest with rubocop cause the cops
get better.

This update required some code changes, specifically the default is to avoid
explicit returns where implicit is done

Also this renames a few rules
2019-12-10 11:48:39 +11:00
Gerhard Schlager b48ca9dee9 DEV: Simplify username validation in base importer
The `UsernameValidator` does already all the hard work. No need to do any additional checks in the import script. The checks were out-of-date anyway.
2019-10-01 20:33:09 +02:00
Gerhard Schlager a61ff16740 DEV: Make attachment markdown reusable 2019-07-25 14:04:18 +02:00
Gerhard Schlager c70d0c6659 Use an invalid domain for fake email addresses in importers 2019-05-30 22:22:24 +02:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Sam Saffron 9be70a22cd DEV: introduce new API to look up dynamic site setting
This removes all uses of both `send` and `public_send` from consumers of
SiteSetting and instead introduces a `get` helper for dynamic lookup

This leads to much cleaner and safer code long term as we are always explicit
to test that a site setting is really there before sending an arbitrary
string to the class

It also removes a couple of risky stubs from the auth provider test
2019-05-07 11:00:30 +10:00
Robin Ward b58867b6e9 FEATURE: New 'Reviewable' model to make reviewable items generic
Includes support for flags, reviewable users and queued posts, with REST API
backwards compatibility.

Co-Authored-By: romanrizzi <romanalejandro@gmail.com>
Co-Authored-By: jjaffeux <j.jaffeux@gmail.com>
2019-03-28 12:45:10 -04:00
Penar Musaraj 9334d2f4f7
FEATURE: add more granular user option levels for email notifications (#7143)
Migrates email user options to a new data structure, where `email_always`, `email_direct` and `email_private_messages` are replace by

* `email_messages_level`, with options: `always`, `only_when_away` and `never` (defaults to `always`)
* `email_level`, with options: `always`, `only_when_away` and `never` (defaults to `only_when_away`)
2019-03-15 10:55:11 -04:00
Gerhard Schlager 941e096df4 Fix error in base import script
Follow-up to 655a08dbbd
2019-03-06 21:58:25 +01:00
maulkin 655a08dbbd FIX: Return actual errors if PostCreator fails (#7096) 2019-03-06 21:29:37 +01:00
Gerhard Schlager 8d5dfe1e01 FIX: Don't import parts of the email address as name 2019-02-17 22:59:18 +01:00
Régis Hanol 1e67bcb456
PERF: bulk feature topic users & reset topic counters after an import 2019-01-17 21:48:23 +01:00
Régis Hanol 788719d271 DEV: speed up posts base imports 2019-01-04 15:30:17 +01:00
Arpit Jalan 71a5369fef FIX: do not convert quote tags to markdown 2018-12-11 20:09:46 +05:30
Régis Hanol c39a1022cc PERF: user imports would slow down the more users were imported 2018-10-22 11:14:13 +02:00
Régis Hanol afa22a0c6f REFACTOR: more 'fake_email' to base importer 2018-10-22 11:12:40 +02:00
Régis Hanol 5f2fb0fe33 Show original options when an error happens while importing an user 2018-10-18 10:21:12 +02:00
Neil Lalonde 15f657309a FEATURE: Zendesk importer that uses its API to get data 2018-08-28 10:21:39 -04:00
Vinoth Kannan 1390eb2957 Disable bootstrap mode before start importing 2018-07-25 12:12:26 +05:30
Gerhard Schlager fb022098f6 Base importer: Calculate category colors depending on parent category 2018-06-27 20:27:11 +02:00
Sam 5f64fd0a21 DEV: remove exec_sql and replace with mini_sql
Introduce new patterns for direct sql that are safe and fast.

MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API

- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder

See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Arpit Jalan b4e0cddcc9 disable all outgoing emails in base importer 2018-06-07 22:49:38 +05:30
Arpit Jalan f9ab3848ed FEATURE: support disabling emails for non-staff users 2018-06-07 18:31:08 +05:30
Régis Hanol aeb511e8ff FEATURE: SMF1 importer 2018-05-28 11:02:19 +02:00
Gerhard Schlager eceeef8413 Imported categories use colors from settings instead of brown 2018-05-22 21:55:54 +02:00
Neil Lalonde 6e8bf1271f increase purge site settings for imports in case jobs like clean_up_uploads cause a mess 2018-04-09 12:44:59 -04:00
Gerhard Schlager 2834705bd1 Improve base import script
* Make sure the category description is imported correctly
  (the about topic usually had the wrong excerpt).
* Allow import scripts to mark topics as closed or archived.
* Allow import scripts to store the topic's original id.
  It will be stored in topic_custom_fields as import_topic_id.
2018-03-16 22:39:06 +01:00
Arpit Jalan 25ec077eca rename 'min_private_message_{post/title}_length' to 'min_personal_message_{post/title}_length' 2018-02-01 13:25:29 +05:30
Gerhard Schlager bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
Gerhard Schlager a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
Gerhard Schlager 94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
Neil Lalonde aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
Régis Hanol 0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
Jay Pfaffman 20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Arpit Jalan ed5d322196 FIX: handle existing users in imports 2017-07-24 16:22:56 +05:30
Guo Xiang Tan d38b55e0c8 Merge pull request #4959 from farmdawgnation/minor-mbox-import-fix
Add a nil check on the connection before attempting to exec it
2017-07-21 09:45:15 +09:00
Leo McArdle d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
Matt Farmer 5b11391588 Add a nil check on the connection before attempting to exec it 2017-07-07 11:11:43 -04:00
Régis Hanol e155cb6db1 FIX: JIVE API Importer
- Added the 'hierarchical=false' query parameter to retrieve comments (replies) chronologically
- Calling 'create_post' (singular) wasn't updating the LookupContainer and thus we couldn't populate the 'reply_to_post_number' column
2017-04-26 15:36:35 +02:00
Régis Hanol 0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
Dmitry Demenchuk f687bfb7e2 Allow to process imported content. (add additional callbacks) 2016-10-26 16:51:34 +01:00
Jay Pfaffman 30f2081c1b importer/base.rb: improve progress reporting 2016-09-20 11:27:45 -07:00