Commit Graph

326 Commits

Author SHA1 Message Date
Sam bfd28ca9a9 ignore deleted ... too much trouble 2015-07-16 15:11:28 +10:00
Sam 16e69d16b8 add like import capabilities 2015-07-16 15:11:28 +10:00
Sam e0eb7f0016 Improve lithuim importer
- grab unique id for each post
- clean up html quotes and lines with nbsp
2015-07-15 15:34:25 +10:00
Sam b00554d662 initial lithuim importer (work in progress) 2015-07-14 17:09:43 +10:00
thiesen 38864a8303 remove setup_dev script to avoid confusion 2015-07-13 13:56:26 -03:00
Sam 4b5354a99e simple redis memory analysis script 2015-07-10 09:00:01 +10:00
Guo Xiang Tan efff3351bf Update Disqus importer. 2015-07-01 21:51:19 +08:00
Sam c8ec1ad2be improve analyzer to allow you to specify a limit in minutes 2015-06-26 13:37:05 +10:00
Sam 8252f4e110 FEATURE: allow use of redis sentinel via redis_sentinels
Use: DISCOURSE_REDIS_SENTINELS and DISCOURSE_REDIS_HOST to configure redis
sentinel
2015-06-25 16:51:48 +10:00
Arpit Jalan ef0f1b2dbd add XenForo importer 2015-06-22 18:02:42 +05:30
Sam 283459e496 add more routes 2015-06-22 09:17:06 +10:00
Sam c4224b8966 add mobile breakdown to script 2015-06-22 09:09:20 +10:00
Sam c034dca844 improve nginx report to include counts
and break down user traffic
2015-06-18 12:29:06 +10:00
Sam 0bb78ff53c Work in progress, performance report 2015-06-17 15:07:17 +10:00
Arpit Jalan 73b7b8a059 FEATURE: add Muut importer 2015-06-12 21:36:12 +05:30
Régis Hanol 189cb3ff12 FEATURE: move migrate_to_new_scheme into a background job
- new hidden site setting 'migrate_to_new_scheme' (defaults to false)
- new rake tasks to toggle migration to new scheme
- FIX: migrate_to_new_scheme also works with CDN
- PERF: improve perf of the DbHelper.remap method
- REFACTOR: UrlHelper is now a class
2015-06-12 12:07:57 +02:00
Robin Ward c6cd1928be Simple "cook" for email imports from mailing lists 2015-06-05 11:46:36 -04:00
Robin Ward ab50d039bb Importer for nabble 2015-05-22 15:41:26 -04:00
Régis Hanol efb992c9d9 FEATURE: t-nation importer 2015-05-19 12:40:47 +02:00
Régis Hanol dcc4267490 FIX: base script import to handle category color/text color 2015-05-19 12:40:35 +02:00
Régis Hanol b25f223bfc remove old vBulletin import script 2015-05-19 12:39:00 +02:00
Arpit Jalan 6a730a8283 Add PunBB import script 2015-05-18 23:08:05 +05:30
Erick Guan 7737cc2828 vBulletin: ability to import attachments from filesystem 2015-05-15 19:39:15 +08:00
Marcus Rückert fb25985b91 Mark all files with a shebang line as executable
This is a bit weird with the Rakefiles but makes
it consistent. Found with rpmlint.
2015-05-12 13:21:32 +02:00
Régis Hanol 5329403a71 use first part of email address as username in SFN import 2015-05-10 23:12:09 +02:00
Gerhard Schlager dc8759962b Fixes the language codes in the pull_translations script 2015-05-06 17:03:07 +02:00
Gerhard Schlager 1023badd90 Simplify the pull_translations script
- There's no need for configuring "trans.<lang>" for each resource.
  Using the global "lang_map" in the Transifex client config is enough.
- The Transifex client can rename locales. No need to do this manually.
- Pull all translation files for each locale that has a file named
  config/locales/client.<lang>.yml. Up until now the script did only
  update existing files, but didn't add missing ones. That is why there
  are so many translation files for the poll plugin missing.
2015-04-30 22:32:12 +02:00
Jens Maier 6dc7cafe00 phpBB importer: fix greedy smiley regex 2015-04-23 12:03:21 +02:00
Régis Hanol c4e813cc11 update SFN importer to handle merge with existing instance 2015-04-22 10:41:40 +02:00
Neil Lalonde e8527c5e3e it's ok if import script can't lower trust level of an admin user 2015-04-17 11:34:21 -04:00
Régis Hanol 58a7faad01 FIX: sfn importer 2015-04-17 16:32:01 +02:00
Régis Hanol 78f5fe190a FIX: base importer should warn when there's an error in the user_profile 2015-04-17 16:31:46 +02:00
Régis Hanol 32e02411bd add custom importer for sfn.org 2015-04-14 18:16:42 +02:00
Robin Ward 3c189c9064 Updates to the Drupal import script
- Allow the VID and DB to be configurable by ENV vars.
- Make importing the blog optional.
2015-04-13 13:06:26 -04:00
Neil Lalonde f8080c0138 add a spinner for importers 2015-03-30 12:29:48 -04:00
Jens Maier f8e1c97441 Update existing user's create time 2015-03-27 14:38:27 +01:00
Erick Guan 186c99a1a7 update regex to remove the bbcode tags 2015-03-24 17:59:08 +08:00
Neil Lalonde 0124187e3a FIX: Ning json files sometimes have an extra ] at the end 2015-03-23 15:14:50 -04:00
Sam 8cb380354e leftover debug code 2015-03-23 15:49:02 +11:00
Sam 23513eaf87 get satisfaction has topics with no body 2015-03-23 12:16:43 +11:00
Sam 24e4808aeb ensure titles exist 2015-03-23 11:58:17 +11:00
Neil Lalonde b1d78a4fd5 FIX: use batches for speed, and show progress from importers' update_tl0 2015-03-20 17:05:14 -04:00
Neil Lalonde 3c4cee5f62 FIX: improve imported code blocks from bbPress importer 2015-03-20 15:54:54 -04:00
Sam c5dd11f64c we dont need much of the escaping magic cause for whatever crazy reason
this CSV format does not allow newlines
2015-03-20 16:51:46 +11:00
Régis Hanol 5084e2bdf1 Merge pull request #3284 from techAPJ/patch-1
FEATURE: new site setting min_first_post_length
2015-03-19 19:55:27 +01:00
Arpit Jalan b706307ac7 FEATURE: new site setting min_first_post_length 2015-03-20 00:20:38 +05:30
Sam 85df7436bb added get satisfaction importer for gradle 2015-03-19 16:09:25 +11:00
Régis Hanol 29a0603850 Merge pull request #3279 from gschlager/importers_01
Fix gem load order for all importers
2015-03-18 21:41:37 +01:00
Gerhard Schlager 9913dfa6c6 Imported users shouldn't get an email immediately after the import
This sets the last_emailed_at for imported users to now which postpones the sending of digest emails for the configured amount of time. Therefore, the first digest email gets sent after 1 week (the default value of default_digest_email_frequency).
As suggested in https://meta.discourse.org/t/dont-trigger-digests-on-migration/26345/7
2015-03-18 21:02:40 +01:00
Gerhard Schlager a412e9bede Fix gem load order for all importers
https://meta.discourse.org/t/migrating-from-mybb/25563/8
2015-03-18 20:31:02 +01:00
Robin Ward 50bf066afd Add subcategory support to BBPress import 2015-03-16 13:18:20 -04:00
Régis Hanol f84503315e Merge pull request #3275 from dandv/patch-3
Add support for MyBB table prefix
2015-03-16 09:46:36 +01:00
Régis Hanol 35c94e8416 Merge pull request #3273 from Elberet/smf-importer
Fix gem load order for SFM2 importer
2015-03-16 09:46:18 +01:00
Dan Dascalescu 9cd59d382d Add support for MyBB table prefix 2015-03-14 22:31:08 -07:00
Dan Dascalescu b8d855227d Fix broken mysql2 load order in mybb import script 2015-03-14 21:29:17 -07:00
Jens Maier 1cc7db2370 Fix gem load order for SFM2 importer 2015-03-15 01:57:46 +01:00
Gerhard Schlager 53591ade00 Adds a few enhancements to the base importer
- Allows importing globally pinned topics
- Doesn't restore the original value of a SiteSetting anymore if an importer changed its value
- Allows all file extensions during the import
- Adds the ability to import bookmarks
2015-03-13 21:24:11 +01:00
Neil Lalonde 1e339ad527 FIX: handle missing post body in json in Ning importer 2015-03-12 18:46:22 -04:00
Gerhard Schlager b5426763e4 FIX: Importers should allow categories with existing name if the parent category is different
This changes the content of `@categories_lookup` from `Category` objects
to IDs since the category names aren't needed anymore. The lookup
method has been renamed too.
2015-03-12 21:15:02 +01:00
Daniel Phin c35cdfcf6b Update vbulletin.rb 2015-03-10 00:31:12 +10:30
Régis Hanol b6f0b29314 Merge pull request #3259 from mcg1969/askbot
Initial commit of askbot.rb
2015-03-07 17:38:34 +01:00
Michael C. Grant e03a1f3532 Initial commit of askbot.rb 2015-03-07 08:48:42 -06:00
Michael C. Grant 6030fe1f9d Added location.present? test to user profile updating. 2015-03-07 08:42:42 -06:00
Sushruth Sivaramakrishnan cdd4608403 typo correction
that -> than
2015-03-04 19:31:55 +05:30
Sam a55a6fb703 Merge pull request #3166 from fantasticfears/bbpress
add table prefix constant to bbpress import script & fix user imports
2015-02-23 14:23:15 +11:00
Neil Lalonde de167165e3 FIX: don't change default_digest_email_frequency during import 2015-02-20 10:30:30 -05:00
Neil Lalonde ae2f861101 FEATURE: import scripts update last_seen_at on users based on created date and last post date 2015-02-12 12:25:01 -05:00
Régis Hanol 38279c79c9 Merge pull request #3196 from fantasticfears/discuz
find more reply syntax during processing in discuzx importer
2015-02-11 09:57:25 +01:00
Erick Guan b1e69d1f95 find more reply syntax during processing in discuzx importer 2015-02-10 21:43:16 -08:00
Gerhard Schlager 519c875d87 Fixes the attachment import of the phpBB3 importer 2015-02-08 21:38:28 +01:00
Neil Lalonde 07af79c85b Ning import: users who have never posted are trust level 0 2015-02-06 16:03:41 -05:00
Neil Lalonde ef482591bf FIX: Ning json can have square brackets instead of commas 2015-02-06 13:47:11 -05:00
Erick Guan 36e550ffea add Discuz! X importer 2015-02-06 05:01:43 -08:00
Erick Guan 58024a8f4d add table prefix constant to bbpress import script & fix user imports 2015-02-03 10:06:35 -08:00
Régis Hanol 0e5c9b2590 small upload code refactor 2015-02-03 18:44:18 +01:00
Neil Lalonde a064bad9a3 fix Ning import script when resuming an import 2015-02-02 16:01:48 -05:00
Gerhard Schlager 827daf7f0f FIX: The order of includes in the base importer was wrong.
https://meta.discourse.org/t/importer-for-simple-machines-2-forums/17656/58
2015-01-31 15:42:39 +01:00
Neil Lalonde c3c202f59e import approval state of users in Ning import script 2015-01-27 15:22:38 -05:00
Régis Hanol 31340de446 don't break the vBulletin importer when PrettyText times out 2015-01-26 20:35:30 +01:00
Gerhard Schlager 72755970c6 Improve the base importer
- FEATURE: Print the time it took for the import to finish
- FEATURE: Allow importing of unactivated users
- FIX: Remove unused variables
- FIX: Accessing an undefined variable is not a good idea
2015-01-23 22:44:00 +01:00
Neil Lalonde a2099110aa FEATURE: Ning importer 2015-01-23 12:19:46 -05:00
Régis Hanol eecc573fbc FIX: don't break import when raw can't be preprocessed (vBulletin importer) 2015-01-21 09:36:46 +01:00
Régis Hanol 6c4d852011 Improve vBulletin importer
- FEATURE: TopicCreator now supports 'pinned_at' parameter
- FIX: 🐛 FIX TopicQuerySQL to support pinned topic older than 2010
- FIX: 🐛 Properly remove all HTML Entities from Usernames/Titles/Category Names/Groups in vBulletin importer
- FIX: 🐛 Properly handle specific vBulletin BBCode (quotes/mentions)
- FIX: 🐛 Make sure we generate a username from the name of the user instead of a fake email
- FEATURE: Allow for custom timezone in vBulletin importer
- FEATURE: Support for profile pictures/background in vBulletin importer
- FIX: 🐛 merge the categories tree to only 2 levels in vBulletin importer
2015-01-19 15:00:55 +01:00
Jens Maier 30710215bc FIX: handle timezone detection errors 2015-01-13 16:44:25 +01:00
Guo Xiang Tan ac72c23799 Exit if a non zero status code is returned. 2014-12-23 14:29:44 +08:00
Régis Hanol 5045d8677c new vBulletin importer (uses mysql dump instead of multiple csv files) 2014-12-22 13:22:16 +01:00
Lourens Naudé fb60daa867 Introduce support for dumping Rails process heap at the end of a benchmark run 2014-12-07 22:55:37 +00:00
Lourens Naudé c273a6d1d4 Rails -> Ruby verbiage change in script/measure.rb 2014-12-07 22:55:37 +00:00
Sam c46b4bd12c Merge pull request #3020 from bear-metal/master
Minor additions to script/bench.rb
2014-12-04 14:22:32 +11:00
Lourens Naudé 096138d0f7 Ensure tmp/pids exist when using unicorn 2014-12-04 01:30:00 +00:00
Sam 6d1321192a micro bench template 2014-12-04 09:31:04 +11:00
Lourens Naudé 5ddcd30e75 Introduce support for other tuneable GC vars 2014-12-03 15:11:03 +00:00
Régis Hanol 65c106325c remove old & useless import script 2014-11-27 19:30:19 +01:00
Régis Hanol 5b90ceb71d FEATURE: rolls up 1.2.*.* IP ranges when number of entries > 10 2014-11-27 19:29:30 +01:00
Sam 4aec3c8c4c correct import script 2014-11-20 14:53:30 +11:00
Stefan Tatschner 3fdda8c2b7 Retrieve first_post_id manually
I had imported a phpbb forum to mybb previously. The importer script
messed up the firstpost ids. In this case most topics cannot be imported
to discourse as the script is not able to determine the topic post.

In case you are infected of this problem just uncomment this mysql
query. It will retrieve the first_post_id manually but it will slow down
the import.
2014-11-14 14:52:16 +01:00
Stefan Tatschner 2ffd3d29d7 Order posts by dateline
The post_ids in my mybb installation did not be in the right order. Post
with ID 2 was posted after post 5. I think it has something to do with a
previous import from phpbb to mybb.
2014-11-14 14:30:16 +01:00
Stefan Tatschner 54d450156f Remove mybb video tags
mybb embedds videos like this:

[video=provider]http://link-to-video.com[/video]

Let's remove these video tags as discourse does this automatically.
2014-11-14 14:26:13 +01:00
Sam 34180eba93 FIX: official benchmark crashing while populating db 2014-11-11 10:28:17 +11:00
Godfrey Chan b1a0cd417d Avoid a deprecation warning by poly-filling #deliver_now and #deliver_now 2014-11-10 01:05:46 -08:00