Commit Graph

94 Commits

Author SHA1 Message Date
Gerhard Schlager bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
Gerhard Schlager a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
Gerhard Schlager 94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
Neil Lalonde aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
Régis Hanol 0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
Jay Pfaffman 20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Arpit Jalan ed5d322196 FIX: handle existing users in imports 2017-07-24 16:22:56 +05:30
Guo Xiang Tan d38b55e0c8 Merge pull request #4959 from farmdawgnation/minor-mbox-import-fix
Add a nil check on the connection before attempting to exec it
2017-07-21 09:45:15 +09:00
Leo McArdle d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
Matt Farmer 5b11391588 Add a nil check on the connection before attempting to exec it 2017-07-07 11:11:43 -04:00
Régis Hanol e155cb6db1 FIX: JIVE API Importer
- Added the 'hierarchical=false' query parameter to retrieve comments (replies) chronologically
- Calling 'create_post' (singular) wasn't updating the LookupContainer and thus we couldn't populate the 'reply_to_post_number' column
2017-04-26 15:36:35 +02:00
Régis Hanol 0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
Dmitry Demenchuk f687bfb7e2 Allow to process imported content. (add additional callbacks) 2016-10-26 16:51:34 +01:00
Jay Pfaffman 30f2081c1b importer/base.rb: improve progress reporting 2016-09-20 11:27:45 -07:00
Robin Ward 72d7c055f4 Lots of ugprades to the Disqus importer script 2016-08-17 15:10:46 -04:00
Gerhard Schlager fccc0c93c5 Store the original username during import (#4349) 2016-07-27 22:30:15 +02:00
Régis Hanol 56d2626187 support 'read_restricted' categories in importers 2016-07-27 18:38:23 +02:00
Neil Lalonde 3b119dac6d FIX: importer performance of update_tl0, and give feedback during vanilla_mysql's create_permalinks 2016-07-06 10:58:43 -04:00
Robin Ward b42f28d4c3 Improved mailing list import.
Now uses a SQLite database to store messages rather than JSON files
for performance and memory considerations.
2016-06-14 11:45:17 -04:00
Neil Lalonde f07b3e7df9 PERF: base importer updates user_stats with sql (mostly) 2016-05-17 18:39:04 -04:00
Régis Hanol b716886240 update bbPress importer 2016-03-07 18:22:15 +01:00
Gerhard Schlager 8a486d8cea Allow importers to set empty names 2016-02-21 23:35:47 +01:00
Gerhard Schlager eb166e78b6 Don't try to import invalid websites 2016-02-21 17:38:04 +01:00
Neil Lalonde 08e10c2a9d FEATURE: Zoho importer 2015-12-04 15:06:46 -05:00
Régis Hanol 26006f12a2 Increased 'error resistance' in base importer class 2015-12-01 15:38:21 +01:00
Robin Ward 6c6d406e63 Small fixes to import base. 2015-10-29 16:47:27 -04:00
Gerhard Schlager 6238a43f93 Fix base and vBulletin importer 2015-10-21 19:07:31 +02:00
Sam 606aeb9d55 improvements to importer
- improve perf of test for existing posts
- always use a system guardian when importing posts
- for lithuim importer requery raw (transform is not repeatable)
2015-10-16 10:41:21 +11:00
Gerhard Schlager 29d27ec2ef FIX: Prevent cast error when import_id contains strings and numbers
Some importers insert numbers and strings as import_id into the same
custom_fields table. This change prevents cast errors during imports.
2015-09-30 16:38:06 +02:00
Kane York 821124f3fd FEATURE: Skip batches if all records exist
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
Kane York 200ee15991 FIX: Importer not handling usernames correctly 2015-09-21 16:27:47 -07:00
Gerhard Schlager 8c03dd16af Adds support for importing password hashes used by "migratepassword" plugin
Adds setting to phpBB3 importer for importing passwords (default: off)
Plugin: https://github.com/discoursehosting/discourse-migratepassword
2015-08-20 22:22:47 +02:00
Robin Ward bb93a345eb UX: Use smaller messages for moderator actions. 2015-07-27 10:09:21 -04:00
Gerhard Schlager 2dd01c61b0 Improves the base importer
- Move some methods into their own classes in order to make it easier
  to reuse them outside of classes extending the base importer. For
  compatibility reasons the old methods are still in the base importer
  and delegate to the new objects. The following methods and hashes were
  extracted:
    - all the lookup maps for existing and imported data
    - all the methods used for uploads and attachments
- No need to store failed users and groups. This information wasn't
  used anyway.
- Print progress instead of category names when importing categories.
- Allow importers to override if bbcode_to_md should be used (until now
  it always used ARGV)
- Allow importers to add additional site settings that automatically get
  restored after the importer finishes.
- Show how many posts and messages are imported per minute. This should
  help detecting when the import is slowing down and needs to be
  restarted.
- Use max_image_width and max_image_height from settings instead of
  hard-coded values for uploaded images.
2015-07-16 15:28:42 +02:00
Sam bfd28ca9a9 ignore deleted ... too much trouble 2015-07-16 15:11:28 +10:00
Régis Hanol dcc4267490 FIX: base script import to handle category color/text color 2015-05-19 12:40:35 +02:00
Neil Lalonde e8527c5e3e it's ok if import script can't lower trust level of an admin user 2015-04-17 11:34:21 -04:00
Régis Hanol 78f5fe190a FIX: base importer should warn when there's an error in the user_profile 2015-04-17 16:31:46 +02:00
Neil Lalonde f8080c0138 add a spinner for importers 2015-03-30 12:29:48 -04:00
Neil Lalonde b1d78a4fd5 FIX: use batches for speed, and show progress from importers' update_tl0 2015-03-20 17:05:14 -04:00
Régis Hanol 5084e2bdf1 Merge pull request #3284 from techAPJ/patch-1
FEATURE: new site setting min_first_post_length
2015-03-19 19:55:27 +01:00
Arpit Jalan b706307ac7 FEATURE: new site setting min_first_post_length 2015-03-20 00:20:38 +05:30
Gerhard Schlager 9913dfa6c6 Imported users shouldn't get an email immediately after the import
This sets the last_emailed_at for imported users to now which postpones the sending of digest emails for the configured amount of time. Therefore, the first digest email gets sent after 1 week (the default value of default_digest_email_frequency).
As suggested in https://meta.discourse.org/t/dont-trigger-digests-on-migration/26345/7
2015-03-18 21:02:40 +01:00
Gerhard Schlager 53591ade00 Adds a few enhancements to the base importer
- Allows importing globally pinned topics
- Doesn't restore the original value of a SiteSetting anymore if an importer changed its value
- Allows all file extensions during the import
- Adds the ability to import bookmarks
2015-03-13 21:24:11 +01:00
Gerhard Schlager b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
Michael C. Grant 6030fe1f9d Added location.present? test to user profile updating. 2015-03-07 08:42:42 -06:00
Neil Lalonde de167165e3 FIX: don't change default_digest_email_frequency during import 2015-02-20 10:30:30 -05:00
Neil Lalonde ae2f861101 FEATURE: import scripts update last_seen_at on users based on created date and last post date 2015-02-12 12:25:01 -05:00
Neil Lalonde 07af79c85b Ning import: users who have never posted are trust level 0 2015-02-06 16:03:41 -05:00