Commit Graph

146 Commits

Author SHA1 Message Date
Gerhard Schlager 2834705bd1 Improve base import script
* Make sure the category description is imported correctly
  (the about topic usually had the wrong excerpt).
* Allow import scripts to mark topics as closed or archived.
* Allow import scripts to store the topic's original id.
  It will be stored in topic_custom_fields as import_topic_id.
2018-03-16 22:39:06 +01:00
Arpit Jalan 25ec077eca rename 'min_private_message_{post/title}_length' to 'min_personal_message_{post/title}_length' 2018-02-01 13:25:29 +05:30
Gerhard Schlager bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
Gerhard Schlager a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
Gerhard Schlager 94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
Neil Lalonde aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
Régis Hanol 0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
Jay Pfaffman 20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Arpit Jalan ed5d322196 FIX: handle existing users in imports 2017-07-24 16:22:56 +05:30
Guo Xiang Tan d38b55e0c8 Merge pull request #4959 from farmdawgnation/minor-mbox-import-fix
Add a nil check on the connection before attempting to exec it
2017-07-21 09:45:15 +09:00
Leo McArdle d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
Matt Farmer 5b11391588 Add a nil check on the connection before attempting to exec it 2017-07-07 11:11:43 -04:00
Régis Hanol e155cb6db1 FIX: JIVE API Importer
- Added the 'hierarchical=false' query parameter to retrieve comments (replies) chronologically
- Calling 'create_post' (singular) wasn't updating the LookupContainer and thus we couldn't populate the 'reply_to_post_number' column
2017-04-26 15:36:35 +02:00
Régis Hanol 0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
Dmitry Demenchuk f687bfb7e2 Allow to process imported content. (add additional callbacks) 2016-10-26 16:51:34 +01:00
Jay Pfaffman 30f2081c1b importer/base.rb: improve progress reporting 2016-09-20 11:27:45 -07:00
Robin Ward 72d7c055f4 Lots of ugprades to the Disqus importer script 2016-08-17 15:10:46 -04:00
Gerhard Schlager fccc0c93c5 Store the original username during import (#4349) 2016-07-27 22:30:15 +02:00
Régis Hanol 56d2626187 support 'read_restricted' categories in importers 2016-07-27 18:38:23 +02:00
Neil Lalonde 3b119dac6d FIX: importer performance of update_tl0, and give feedback during vanilla_mysql's create_permalinks 2016-07-06 10:58:43 -04:00
Robin Ward b42f28d4c3 Improved mailing list import.
Now uses a SQLite database to store messages rather than JSON files
for performance and memory considerations.
2016-06-14 11:45:17 -04:00
Neil Lalonde f07b3e7df9 PERF: base importer updates user_stats with sql (mostly) 2016-05-17 18:39:04 -04:00
Régis Hanol b716886240 update bbPress importer 2016-03-07 18:22:15 +01:00
Gerhard Schlager 8a486d8cea Allow importers to set empty names 2016-02-21 23:35:47 +01:00
Gerhard Schlager eb166e78b6 Don't try to import invalid websites 2016-02-21 17:38:04 +01:00
Neil Lalonde 08e10c2a9d FEATURE: Zoho importer 2015-12-04 15:06:46 -05:00
Régis Hanol 26006f12a2 Increased 'error resistance' in base importer class 2015-12-01 15:38:21 +01:00
Robin Ward 6c6d406e63 Small fixes to import base. 2015-10-29 16:47:27 -04:00
Gerhard Schlager 6238a43f93 Fix base and vBulletin importer 2015-10-21 19:07:31 +02:00
Sam 606aeb9d55 improvements to importer
- improve perf of test for existing posts
- always use a system guardian when importing posts
- for lithuim importer requery raw (transform is not repeatable)
2015-10-16 10:41:21 +11:00
Gerhard Schlager 29d27ec2ef FIX: Prevent cast error when import_id contains strings and numbers
Some importers insert numbers and strings as import_id into the same
custom_fields table. This change prevents cast errors during imports.
2015-09-30 16:38:06 +02:00
Kane York 821124f3fd FEATURE: Skip batches if all records exist
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
Kane York 200ee15991 FIX: Importer not handling usernames correctly 2015-09-21 16:27:47 -07:00
Gerhard Schlager 8c03dd16af Adds support for importing password hashes used by "migratepassword" plugin
Adds setting to phpBB3 importer for importing passwords (default: off)
Plugin: https://github.com/discoursehosting/discourse-migratepassword
2015-08-20 22:22:47 +02:00
Robin Ward bb93a345eb UX: Use smaller messages for moderator actions. 2015-07-27 10:09:21 -04:00
Gerhard Schlager 2dd01c61b0 Improves the base importer
- Move some methods into their own classes in order to make it easier
  to reuse them outside of classes extending the base importer. For
  compatibility reasons the old methods are still in the base importer
  and delegate to the new objects. The following methods and hashes were
  extracted:
    - all the lookup maps for existing and imported data
    - all the methods used for uploads and attachments
- No need to store failed users and groups. This information wasn't
  used anyway.
- Print progress instead of category names when importing categories.
- Allow importers to override if bbcode_to_md should be used (until now
  it always used ARGV)
- Allow importers to add additional site settings that automatically get
  restored after the importer finishes.
- Show how many posts and messages are imported per minute. This should
  help detecting when the import is slowing down and needs to be
  restarted.
- Use max_image_width and max_image_height from settings instead of
  hard-coded values for uploaded images.
2015-07-16 15:28:42 +02:00
Sam bfd28ca9a9 ignore deleted ... too much trouble 2015-07-16 15:11:28 +10:00
Régis Hanol dcc4267490 FIX: base script import to handle category color/text color 2015-05-19 12:40:35 +02:00
Neil Lalonde e8527c5e3e it's ok if import script can't lower trust level of an admin user 2015-04-17 11:34:21 -04:00
Régis Hanol 78f5fe190a FIX: base importer should warn when there's an error in the user_profile 2015-04-17 16:31:46 +02:00
Neil Lalonde f8080c0138 add a spinner for importers 2015-03-30 12:29:48 -04:00
Neil Lalonde b1d78a4fd5 FIX: use batches for speed, and show progress from importers' update_tl0 2015-03-20 17:05:14 -04:00
Régis Hanol 5084e2bdf1 Merge pull request #3284 from techAPJ/patch-1
FEATURE: new site setting min_first_post_length
2015-03-19 19:55:27 +01:00
Arpit Jalan b706307ac7 FEATURE: new site setting min_first_post_length 2015-03-20 00:20:38 +05:30
Gerhard Schlager 9913dfa6c6 Imported users shouldn't get an email immediately after the import
This sets the last_emailed_at for imported users to now which postpones the sending of digest emails for the configured amount of time. Therefore, the first digest email gets sent after 1 week (the default value of default_digest_email_frequency).
As suggested in https://meta.discourse.org/t/dont-trigger-digests-on-migration/26345/7
2015-03-18 21:02:40 +01:00
Gerhard Schlager 53591ade00 Adds a few enhancements to the base importer
- Allows importing globally pinned topics
- Doesn't restore the original value of a SiteSetting anymore if an importer changed its value
- Allows all file extensions during the import
- Adds the ability to import bookmarks
2015-03-13 21:24:11 +01:00
Gerhard Schlager b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
Michael C. Grant 6030fe1f9d Added location.present? test to user profile updating. 2015-03-07 08:42:42 -06:00
Neil Lalonde de167165e3 FIX: don't change default_digest_email_frequency during import 2015-02-20 10:30:30 -05:00
Neil Lalonde ae2f861101 FEATURE: import scripts update last_seen_at on users based on created date and last post date 2015-02-12 12:25:01 -05:00
Neil Lalonde 07af79c85b Ning import: users who have never posted are trust level 0 2015-02-06 16:03:41 -05:00
Régis Hanol 0e5c9b2590 small upload code refactor 2015-02-03 18:44:18 +01:00
Gerhard Schlager 827daf7f0f FIX: The order of includes in the base importer was wrong.
https://meta.discourse.org/t/importer-for-simple-machines-2-forums/17656/58
2015-01-31 15:42:39 +01:00
Gerhard Schlager 72755970c6 Improve the base importer
- FEATURE: Print the time it took for the import to finish
- FEATURE: Allow importing of unactivated users
- FIX: Remove unused variables
- FIX: Accessing an undefined variable is not a good idea
2015-01-23 22:44:00 +01:00
Neil Lalonde a2099110aa FEATURE: Ning importer 2015-01-23 12:19:46 -05:00
Régis Hanol 6c4d852011 Improve vBulletin importer
- FEATURE: TopicCreator now supports 'pinned_at' parameter
- FIX: 🐛 FIX TopicQuerySQL to support pinned topic older than 2010
- FIX: 🐛 Properly remove all HTML Entities from Usernames/Titles/Category Names/Groups in vBulletin importer
- FIX: 🐛 Properly handle specific vBulletin BBCode (quotes/mentions)
- FIX: 🐛 Make sure we generate a username from the name of the user instead of a fake email
- FEATURE: Allow for custom timezone in vBulletin importer
- FEATURE: Support for profile pictures/background in vBulletin importer
- FIX: 🐛 merge the categories tree to only 2 levels in vBulletin importer
2015-01-19 15:00:55 +01:00
Sam 1d4daca75d add override for post creation in importer 2014-10-31 15:19:40 +11:00
Neil Lalonde cea2fe5ee9 FEATURE: import attachments in phpbb3 importer 2014-10-07 16:47:41 -04:00
Robin Ward 47573fcd99 FIX: error importing some topics where `bumped_at` coudln't be
determined.
2014-10-06 09:26:52 -04:00
Robin Ward f4f54fd49b First stab at Importer for Drupal QA 2014-09-10 14:27:40 -04:00
Régis Hanol 78bcea5247 PERF: improve perf on consistency query in importers 2014-09-08 22:08:41 +02:00
Régis Hanol 9964955e88 FEATURE: fix consistency of users.last_posted_at in importers 2014-09-08 19:36:55 +02:00
Sam 59d04c0695 Internal renaming of elder,leader,regular,basic to numbers
Changed internals so trust levels are referred to with

TrustLevel[1], TrustLevel[2] etc.

This gives us much better flexibility naming trust levels, these names
are meant to be controlled by various communities.
2014-09-05 15:20:52 +10:00
Régis Hanol 7a7a38c78e FIX: always show the progress bars when skipping posts/users in importers 2014-09-04 19:18:22 +02:00
Régis Hanol a0569a22be FEATURE: reset topic counters at the end of the import 2014-09-04 19:08:57 +02:00
Régis Hanol 85cbb001ae FIX: properly close topics in vBulletin importer
FEATURE: add backtrace when an exception happen (importers)
FIX: post-processing should also happen on first posts (vBulletin
importer)
PERF: faster topic bypass when already imported
2014-09-04 17:55:05 +02:00
Gerhard Schlager 5a769c7c48 FIX: Reset changed site settings after import is complete
After an import has finished the site settings should be reset to their
previous values. For example, since 'disable_emails' was set to false
before any import, it wasn't possible to login after the import
(sending email for password reset didn't work).
2014-08-31 18:09:21 +02:00
Régis Hanol 9b86d1507d VBULLETIN: add some stats about the imported groups 2014-08-25 12:33:08 +02:00
Régis Hanol 7aaf718cf3 update bbcode dialect and fix vBulletin importer 2014-08-25 10:48:29 +02:00
Régis Hanol 76824cda31 FIX: completely disable emails in importers 2014-08-23 11:12:01 +02:00
Régis Hanol 071325ab98 VANILLA: the name is actually the username (WAT) 2014-08-22 10:11:12 +02:00
Régis Hanol d58d04f595 disable digest emails when importing a forum 2014-08-20 19:29:17 +02:00
Régis Hanol 66aaa9329e FEATURE: vBulletin importer 2014-08-18 13:04:08 +02:00
Neil Lalonde 5d9a389966 FIX: handle importing usernames that are too short, too long, or have invalid characters 2014-08-14 14:04:28 -04:00
Régis Hanol 61842157bf FIX: avatar_url should only be a custom field 2014-08-14 09:43:32 +02:00
Régis Hanol c97f18dec0 FIX: users are 'active' by default in imports 2014-08-13 23:57:07 +02:00
Régis Hanol 4c4ce05964 FIX & PERF: vanilla import
PERF: disabled refresh_avatar callback when importing users
PERF: avoid using UsernameSuggester when not needed
FIX: categories wasn't working
FIX: posts from deleted users are now from the system user
2014-08-13 22:20:06 +02:00
Régis Hanol d0095ce041 FEATURE: migrate vanilla importer to new import architecture 2014-08-11 18:44:17 +02:00
Neil Lalonde 8cb4442527 FIX: import scripts: bbcode_to_md should not be escaping html in post.raw 2014-07-23 15:16:01 -04:00
Jens Maier 7b5e6ff744 FEATURE: smf2 import script 2014-07-16 20:02:58 +02:00
Neil Lalonde a9788bc4fe FIX: instructions in import script 2014-07-11 13:36:11 -04:00
Neil Lalonde c113b5d270 FEATURE: phpbb 3 import script 2014-07-09 11:18:29 -04:00
Neil Lalonde 7d5d5862c1 Import optimizations for topic creation: Prevent queuing of most jobs when importing posts and topics. Only do some recalculations at the end of the import. 2014-07-03 16:24:47 -04:00
Neil Lalonde 4e1f76d3b9 Merge pull request #2497 from BenLubar/benlubar-import-id-on-imported-posts
set import_id on imported posts
2014-07-03 10:22:25 -04:00
Neil Lalonde 67b9fec204 FEATURE: option to use ruby-bbcode-to-md to convert bbcode to markdown before importing posts 2014-07-02 16:53:53 -04:00
Ben Lubar 1dafad580d set import_id on imported posts 2014-06-26 12:33:56 -05:00
Andrew Bezzub e3fcd3fa80 update import script to save bio to UserProfile 2014-06-13 14:55:32 -04:00
Neil Lalonde 916252483b FIX: in import scripts, fix a case where post author lookup fails for some reason 2014-06-11 13:59:21 -04:00
Neil Lalonde 1e0d29273a Import scripts: handle when a topic has been closed 2014-06-10 17:12:50 -04:00
Neil Lalonde f085770ed4 Import scripts: add update_feature_topic_users 2014-06-05 15:31:05 -04:00
Neil Lalonde 9bd76b0c16 Import scripts: relax our posting rules before the import starts 2014-06-05 15:31:05 -04:00
Neil Lalonde 2f706d084f FIX: in import scripts, don't generate names for users based on their emails. It's basically the same as publicizing their emails. 2014-06-05 14:15:52 -04:00
Neil Lalonde 08346974cf Import scripts: don't include moderator_action posts when calculating bumped_at 2014-06-04 18:22:19 -04:00
Neil Lalonde 35834a9c40 Import scripts: add method close_inactive_topics 2014-06-04 10:38:25 -04:00
Neil Lalonde daceea73a8 Import scripts for Drupal and Kunena/Joomla. A new base class for import scripts. 2014-06-02 14:38:40 -04:00