Commit Graph

456 Commits

Author SHA1 Message Date
Régis Hanol 07bc52dc69 import topic views in phpbb3 2018-03-07 19:27:03 +01:00
Gerhard Schlager dc32ee5cbf Improvements to mbox import script
* Ignore errors during indexing and show information about the message causing the problem
* Always activate imported users if they aren't staged
2018-03-06 11:32:12 +01:00
Régis Hanol 49400337b6 Improve IPBoard3 importer
- fix encoding issues
- close topics
- don't import queued & not approved topics
- don't import deleted posts
2018-03-05 10:24:42 +01:00
Régis Hanol ae9b1e8554 FEATURE: IP.Board 3 importer 2018-03-01 09:43:32 +01:00
Régis Hanol 82e68670bf FIX: SimplePress importer wasn't handling increment imports properly 2018-02-24 00:42:17 +01:00
Sam 720e1965e3 FEATURE: add category suppress from latest
In the past we used suppress_from_homepage, it had mixed semantics
it would remove from category list if category list was on home and
unconditionally remove from latest.

New setting explicitly only removes from latest list but leaves the
category list alond
2018-02-22 09:56:35 +11:00
Gerhard Schlager 479f7ed18f Ignore case when removing mailing list name from subject 2018-02-12 21:41:58 +01:00
Régis Hanol 31f5ca6180 improve simple press importer 2018-02-07 17:37:51 +01:00
Arpit Jalan 25ec077eca rename 'min_private_message_{post/title}_length' to 'min_personal_message_{post/title}_length' 2018-02-01 13:25:29 +05:30
Neil Lalonde cc830f7119 sorry rubocop 2018-01-30 14:37:15 -05:00
Jay Pfaffman db17c531d6 Feature: Invision Power Board Importer 2018-01-30 08:44:02 -08:00
Gerhard Schlager 192a0886e2 FIX: BBCode to Markdown conversion in phpBB3 importer was broken
This fixes the conversion for quotes, code blocks and lists (except for nested lists). It also discourages the usage of the ruby-bbcode-to-md gem.
2018-01-30 12:50:42 +01:00
Gerhard Schlager c26db2116c FIX: phpBB3 importer imported some users as anonymous users 2018-01-30 12:50:42 +01:00
Neil Lalonde db287ad58e FIX: invalid SQL in vanilla_mysql import script 2018-01-19 11:53:07 -05:00
Gerhard Schlager 6500343431 FIX: mbox importer didn't detected already indexed files 2018-01-17 17:03:53 +01:00
Gerhard Schlager bb54eb1192 Improvements to mbox importer
* store time it took to index message in DB (to find performance issues)
* ignore listserv specific files
* better examples for split_regex
* first email in mbox shouldn't contain the split string
* always lock the DB in exclusive mode
* save email within transaction
* messages can be grouped by subject and use original order (for Listserv)
* adds option to index emails without running the import
2018-01-17 12:04:57 +01:00
Yaw Anokwa 77a92e8878 Allow user staging via setting (#5468) 2018-01-04 09:17:35 +01:00
Gerhard Schlager a93bf186c5 FIX: validation of website failed in base importer 2017-12-28 14:51:43 +01:00
Jay Pfaffman cc46ab5bb1 smf2: deal with duplicate category names (#5435) 2017-12-27 16:56:36 +11:00
Gerhard Schlager cafe69caac Refactor mbox import script 2017-12-13 22:03:31 +01:00
Arpit Jalan 3190c13c22 import staged users as inactive in mbox import 2017-12-13 08:45:43 +05:30
Gerhard Schlager 16738cfb1b FEATURE: convert plain text emails to markdown 2017-12-06 01:47:51 +01:00
Gerhard Schlager 32dd1e66be improvements to the mbox import script
* ignores dot-files and empty emails
* new setting to prefer HTML over plaintext emails during import
* restore original site settings at the end of import
* elided content of HTML mails was not put inside details block
2017-11-18 17:16:44 +01:00
Gerhard Schlager 94ceb06b02 fix user lookup by email in disqus import script 2017-11-16 16:26:18 +01:00
Gerhard Schlager 06a6ddc3ba handle plaintext and HTML emails in mbox importer 2017-11-15 20:22:11 +01:00
Jon Bartlett 5d8508c523 New yahoo groups importer 2017-11-09 11:07:57 +01:00
Neil Lalonde 667b025d12 make rubocop happy 2017-11-07 14:20:46 -05:00
discoursehosting 4f0bdec370 some improvements for importers (#5295)
* decode html entities within code blocks

* Only import users that actually participated in the bbpress part of Wordpress; import password hashes

* create permalinks for topics

* Better handling of [code] blocks
2017-11-07 17:50:43 +01:00
Jay Pfaffman 9cbb90c5ed add importer for modx forum (#5239) 2017-10-31 10:50:34 +11:00
Neil Lalonde aec5de451b FIX: compare imported ids as string instead of converting to integer, same as LookupContainer 2017-10-23 14:01:57 -04:00
Régis Hanol 0d257058c9 make rubocop 🤖 happy 2017-10-23 10:21:50 +02:00
Jay Pfaffman 20e7e285d6 add users with invalid email addresses (#5224)
* add users with invalid email addresses

* start to add suspend user stff

* don't create a suspend_user function

* include reason for suspension
2017-10-23 10:18:44 +02:00
Gerhard Schlager 6c829c24d7 escaping the subject isn't needed in the mbox importer 2017-10-19 15:25:20 +02:00
Gerhard Schlager c41880ab19 Improvements to the experimental mbox importer
* Disable journaling to improve performance in Docker
* Use the email cooking method
* Store IncomingEmail in order find related posts by Message-ID
* Escape HTML in imported messages
2017-10-19 14:27:40 +02:00
Gerhard Schlager e9fb2c5756
phpBB3 importer: Fix validation of site settings 2017-08-29 01:48:30 +02:00
Neil Lalonde 3f8e535692 FEATURE: bbpress importer can import private messages 2017-08-21 16:39:57 -04:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Gerhard Schlager 32ef30cff9
phpBB3 importer: Permalink for first post of each topic was missing 2017-07-25 23:29:56 +02:00
Gerhard Schlager 5be52fa821
phpBB3 importer: Use permalink normalization for category links 2017-07-25 21:14:46 +02:00
Arpit Jalan ed5d322196 FIX: handle existing users in imports 2017-07-24 16:22:56 +05:30
Guo Xiang Tan d38b55e0c8 Merge pull request #4959 from farmdawgnation/minor-mbox-import-fix
Add a nil check on the connection before attempting to exec it
2017-07-21 09:45:15 +09:00
Leo McArdle d0b027d88d FEATURE: phase 1 of supporting multiple email addresses 2017-07-20 11:22:27 +09:00
Sam Saffron d0c5205a52 Feature: Change markdown engine to markdown it
This commit removes the old evilstreak markdownjs engine.

- Adds specs to WhiteLister and changes it to stop using globals
    (Fixes large memory leak)
- Fixes edge cases around bbcode handling
- Removes mdtest which is no longer valid (to be replaced with
    CommonMark)
- Updates MiniRacer to correct minor unmanaged memory leak
- Fixes plugin specs
2017-07-17 11:41:34 -04:00
Sam 79a084dd58 Revert "remove old markdown engine work-in-progress"
This reverts commit ee470b5317.
2017-07-12 18:10:51 -04:00
Sam Saffron ee470b5317 remove old markdown engine work-in-progress 2017-07-12 17:44:40 -04:00
Matt Farmer 5b11391588 Add a nil check on the connection before attempting to exec it 2017-07-07 11:11:43 -04:00
Jay Pfaffman 1c9cab0908 mybb: disable outgoing mail 2017-06-30 16:28:14 -07:00
Jay Pfaffman a7d89d2a8a mybb: add permalink support 2017-06-30 14:11:49 -07:00
Melroy Neil Dsouza 420abce549 Optimize import script - vanilla_mysql.rb
Use WHERE id > last_id instead of OFFSET because OFFSET causes performance issues
2017-06-19 10:30:55 +05:30
Guo Xiang Tan d82dbd565b FIX: Remove dependency on `rest-client`. 2017-06-16 09:42:41 +09:00
Régis Hanol 4e8e9c2d2f FIX: ensure we never collide ids in the JIVE API importer 2017-06-06 16:13:07 +02:00
Régis Hanol b93a4e369d add support for importing content by id in JIVE API importer 2017-06-03 21:49:30 +02:00
Gerhard Schlager 8299e7e8c3
Add new, experimental version of mbox importer 2017-05-29 20:59:18 +02:00
lead4good 0d36fb0c2b bbpress import script: Adding support for anonymous users 2017-05-25 13:15:50 +02:00
Robin Ward cdbe027c1c Refactor `FileHelper` to use keyword arguments. 2017-05-24 13:54:26 -04:00
Régis Hanol 2826f98bb9 update JIVE API importer 2017-05-16 10:22:44 +02:00
Régis Hanol 9641d2413d REFACTOR: upload workflow creation into UploadCreator
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Robin Ward 71414f7ba2 We don't need to log downloading avatars that are too large 2017-05-09 14:46:42 -04:00
Régis Hanol 4b9cc1ad17 import likes & bookmarks in JIVE API importer 2017-05-03 16:00:23 +02:00
Régis Hanol e155cb6db1 FIX: JIVE API Importer
- Added the 'hierarchical=false' query parameter to retrieve comments (replies) chronologically
- Calling 'create_post' (singular) wasn't updating the LookupContainer and thus we couldn't populate the 'reply_to_post_number' column
2017-04-26 15:36:35 +02:00
Gerhard Schlager 8d9b5ed70a
phpBB3 importer: Disable file size validators during import 2017-04-26 01:05:36 +02:00
Régis Hanol 29ddb3a611 update JIVE API importer 2017-04-24 22:03:12 +02:00
Régis Hanol ff6a98183f WIP: JIVE importer using the API 2017-04-12 22:30:36 +02:00
michael@discoursehosting.com 17b5303449 Convert nested quotes 2017-04-02 20:38:23 +00:00
michael@discoursehosting.com 6168bcb4c4 Someone broke/disabled? the php check 2017-04-02 20:35:43 +00:00
michael@discoursehosting.com 7e27559f3e We need to get rid of the ignore_quotes thing as well 2017-04-02 20:33:48 +00:00
michael@discoursehosting.com 60be3e0d38 Get rid of the weird and broken message graph 2017-04-02 20:31:06 +00:00
michael@discoursehosting.com 7d09af141e Apparently need to save 2017-04-02 20:28:16 +00:00
michael@discoursehosting.com a72f56c3a7 Add migratepassword plugin hash 2017-04-02 20:27:57 +00:00
michael@discoursehosting.com 332d655a1d Use html_for_upload instead of broken v8 stuff 2017-04-02 20:25:08 +00:00
michael@discoursehosting.com e788ac3351 Add permalinks creator for PrettyURL plugin 2017-04-02 20:23:03 +00:00
Robin Ward 14410b71fb Convert server side paths to use `/u/` 2017-03-30 10:23:24 -04:00
David McClure ed627c8233 FEATURE: Socialcast Importer: Set category and tags based on group 2017-02-26 04:54:07 -08:00
Gerhard Schlager 1fb5dac1bc
phpBB3 importer: Fix the link regex
Thanks to Leo Davidson for the improved regex.
https://meta.discourse.org/t/importing-from-phpbb3/30810/153
2017-02-05 17:13:31 +01:00
Gerhard Schlager 14a0dff469
phpBB3 importer: Fix import of anonymous users 2017-02-05 15:26:50 +01:00
Gerhard Schlager b79eef32e7
phpBB3 importer: Make it harder to set wrong table prefix 2017-02-05 14:13:25 +01:00
Gerhard Schlager 5569afc48b
phpBB3 importer: Don't ignore users that belong to non-existent group 2017-02-05 14:13:25 +01:00
Régis Hanol 0d250c3935 50% faster vBulletin 4 importer 2017-02-01 14:33:09 +01:00
Jay Pfaffman 3a5c0c5605 add env variables for vBulletin import script 2017-01-31 10:31:18 +05:30
Robin Ward 4ed46a45ed FIX: In the case of a blank name, use a SHA1 2017-01-19 14:31:50 -05:00
Robin Ward 8d4cff642e FIX: Use the username scrubber when generating the emails 2017-01-18 14:23:23 -05:00
Robin Ward 775b70227d FIX: Better standardizing on username 2017-01-18 13:22:45 -05:00
Robin Ward f1c242f61d A generic JSON importer that can be used for simple imports 2017-01-18 13:22:45 -05:00
Jay Pfaffman ffbaf374c8 use .presence rather than DIY checking 2017-01-11 12:55:25 -08:00
Jay Pfaffman e307bbccf9 Merge branch 'master' of github.com:discourse/discourse into bbpress-missing-display-name 2017-01-11 11:28:38 -08:00
Jay Pfaffman c5d6bfe7e2 bbpress: Use nicename if display_name is missing 2017-01-11 11:26:55 -08:00
Rafael dos Santos Silva 3a3a464a32 Merge pull request #4642 from miromichalicka/master
Add support for import from Drupal 6
2017-01-11 12:56:52 -02:00
Arpit Jalan 6506c0d900 use table prefix in bbpress import script 2017-01-10 16:07:13 +05:30
Miroslav Michalicka a313ddf8e3 Fix typos 2017-01-08 10:16:32 +01:00
Miroslav Michalicka bde42e397c Migration script from Drupal 6 2017-01-08 10:07:46 +01:00
Mohamad Abras bc621a704e fix close topics vb4 importer 2016-12-15 14:20:05 +02:00
Arpit Jalan 59523aef9d more improvements to vBulletin import script 2016-12-05 17:44:18 +05:30
Mohamad Abras f68194cf8e fix nil for vb4 importer 2016-12-05 04:16:59 +02:00
Mohamad Abras 52749c0121 imporoving vb4 importer 2016-12-04 00:02:47 +02:00
Jay Pfaffman 10f82a9a8e new importer: my little forum 2016-11-01 12:17:36 -07:00
Dmitry Demenchuk f687bfb7e2 Allow to process imported content. (add additional callbacks) 2016-10-26 16:51:34 +01:00
Régis Hanol 095170d6df Merge pull request #4493 from pfaffman/mbox-email-to-username
MBOX: replace email address with @username
2016-10-12 22:45:31 +02:00
Jay Pfaffman 4ccd224c26 Add link to HOWTO 2016-10-12 13:29:19 -07:00
Jay Pfaffman df3ac5e2e9 MBOX: replace email address with @username 2016-10-12 13:18:34 -07:00
Sam 47733d4f94 Merge pull request #4480 from vinkas0/import/fluxbb
FluxBB import script
2016-10-10 16:38:53 +11:00