Commit Graph

608 Commits

Author SHA1 Message Date
Jarek Radosz 2fc70c5572
DEV: Correctly tag heredocs (#16061)
This allows text editors to use correct syntax coloring for the heredoc sections.

Heredoc tag names we use:

languages: SQL, JS, RUBY, LUA, HTML, CSS, SCSS, SH, HBS, XML, YAML/YML, MF, ICS
other: MD, TEXT/TXT, RAW, EMAIL
2022-02-28 20:50:55 +01:00
Jarek Radosz 6f6406ea03
DEV: Fix random typos (#16066) 2022-02-28 10:20:58 +08:00
Michael Brown 3bf3b9a4a5 DEV: pull email address validation out to a new EmailAddressValidator
We validate the *format* of email addresses in many places with a match against
a regex, often with very slightly different syntax.

Adding a separate EmailAddressValidator simplifies the code in a few spots and
feels cleaner.

Deprecated the old location in case someone is using it in a plugin.

No functionality change is in this commit.

Note: the regex used at the moment does not support using address literals, e.g.:
* localpart@[192.168.0.1]
* localpart@[2001:db8::1]
2022-02-17 21:49:22 -05:00
Gerhard Schlager 6394d7cddf
DEV: Improve phpBB3 import script (#15956)
* Optional import of custom user fields from phpBB 3.1+
* Optional import of likes from phpBB3
  Requires the phpBB "Thanks for posts" extension
* Fix import of bookmarks from phpBB3
* Update `created_at` of existing user
* Support mapping of phpBB forums to existing Discourse categories
  This is in addition to the ability of merging phpBB forums and importing into newly created Discourse categories.
2022-02-16 13:04:31 +01:00
Gerhard Schlager 33d6ed60a4
DEV: Don't import year of birth (#15937)
The cakeday plugin doesn't use the year.
2022-02-14 18:10:35 +01:00
Gerhard Schlager 6a41ec179c
FIX: Default settings for phpBB3 import were broken (#15913) 2022-02-11 18:18:54 +01:00
Canapin ea2fd75d10
DEV: Fix some regexes in phpBB3 import script (#15829)
1. bbcode hashes don't always have exactly 8 characters.

2. colors aren't always hex values, it can be a color string ("red", "blue", etc).

3. The closing tag of smileys doesn't always include a `:` character (the start of the regex was already right for this particular issue)
2022-02-07 16:16:46 +01:00
Peter Zhu c5fd8c42db
DEV: Fix methods removed in Ruby 3.2 (#15459)
* File.exists? is deprecated and removed in Ruby 3.2 in favor of
File.exist?
* Dir.exists? is deprecated and removed in Ruby 3.2 in favor of
Dir.exist?
2022-01-05 18:45:08 +01:00
Leonardo Mosquera 48a08cc397
FIX: Vanilla importer fixes (#14699)
Import script was out of date
2021-10-27 14:22:37 +02:00
Dan Ungureanu 69f0f48dc0
DEV: Fix rubocop issues (#14715) 2021-10-27 11:39:28 +03:00
David Taylor 46d96c9feb
DEV: Apply rubocop to script/import_scripts/phorum.rb (#14727)
Followup to b24002018a
2021-10-26 19:16:52 +01:00
Jeremy Waters b24002018a Update phorum.rb
Add attachment/file/upload handling to bring them in from phorum to discourse
2021-10-26 12:41:50 -04:00
Theodore Diamantidis 97178cd777
FIX: phpbb import - attachments not embedded in posts (#14570) 2021-10-11 14:27:54 +02:00
Constanza a413a1e015
DEV: process image uploads in the Zendesk API import script (#14524) 2021-10-06 12:24:12 -04:00
Vinoth Kannan cd9262b7d3
DEV: minor improvements in the vanilla import script. (#14026)
We're parsing the post raw based on the record format now.
2021-08-12 15:07:44 +05:30
Ruoxin Wang f9aaed7020
FIX: MyBB importer exposes deleted posts (#13700)
The MyBB importer exposes posts marked as deleted in MyBB. This may lead to a privacy issue.
2021-07-22 09:55:02 +02:00
Alan Guo Xiang Tan 37b8ce79c9
FEATURE: Add last visit indication to topic view page. (#13471)
This PR also removes grey old unread bubble from the topic badges by
dropping `TopicUser#highest_seen_post_number`.
2021-07-05 14:17:31 +08:00
Dan Ungureanu 6ea4bbd2ec
DEV: Prefer .pluck_first over .pluck.first (#13607) 2021-07-02 10:03:54 +08:00
Bianca Nenciu 5efed91128 FIX: Set random values for digest_attempted_at
Setting a random value in the interval 1 week ago ... now works better
because this spreads digest scheduling over a week because digests are
sent one week from the date of the last digest.
2021-06-22 12:05:15 +08:00
Arpit Jalan 365d339985
DEV: fix Flarum import script (#13385) 2021-06-15 19:08:55 +05:30
Lecter 4bf195d502
FEATURE: Flarum import script (#13139) 2021-05-27 02:30:50 +02:00
Bianca Nenciu d0779a87bb
FIX: Use max_category_nesting when importing categories (#13105)
It allowed for a parent category and a sub-category.
2021-05-26 12:40:26 +03:00
Josh Soref 59097b207f
DEV: Correct typos and spelling mistakes (#12812)
Over the years we accrued many spelling mistakes in the code base. 

This PR attempts to fix spelling mistakes and typos in all areas of the code that are extremely safe to change 

- comments
- test descriptions
- other low risk areas
2021-05-21 11:43:47 +10:00
Pilou e7892df10d
FIX: handle charset=windows-1252 in mbox import script (#12832)
Co-authored-by: Loïc Dachary <loic@dachary.org>
2021-04-27 15:43:31 +02:00
Michael Maroszek 5bec0e5763
fix vbulletin importer to import unreferenced attachments (#12187) 2021-04-19 21:05:16 +02:00
Justin DiRose e302e32a3f
DEV: Add Higher Logic import script (#12623)
Wrote up a new script to import from Higher Logic. Nothing too crazy going on here. Two major things about this script:

    It requires you to convert a Microsoft SQL file to a format MySQL can read.
    Higher Logic stores posts (at least in the case of the import I ran) with the email thread shown in the post body. The script does its best to truncate this out, but the logic may need to be improved on future imports. For the import I ran, it worked just fine as is. 🤷‍♂️
2021-04-06 16:53:55 -05:00
Justin DiRose 1aaf588fb7
DEV: Improvements to vanilla_mysql importer (#12308)
Made some improvements to the Vanilla MySQL script -- mainly because not all SQL imports require use of the VanillaBodyParser. Still left it as an option to turn on and use if so desired. Also added subcategory support, importing of likes, and solve status.
2021-03-11 10:21:56 -06:00
Blake Erickson dbcda617b3
DEV: Add a CSV importer for restoring deleted users (#12147)
This is an importer I wrote to restore some users that were
accidentally deleted for being purged as old staged users or old
unactivated users.

It reads from CSV files exported from a discourse sql backup.
2021-02-19 13:46:54 -07:00
Blake Erickson ed0e4582a1
DEV: If disabled do not change setting after import (#12142)
When running an import script there are many site settings that are
changed but we reset them back to where they were originally before the
import. However, there are two settings that we don't roll back:

```
purge_unactivated_users_grace_period_days
purge_deleted_uploads_grace_period_days
```

which could have some unintended consequences.

My first question is do we *really* have to change these settings? I'm
not a huge fan of changing someones settings without them really knowing
they were changed.

If we really do have to change these settings here is my proposed PR
where we don't alter the `purge_unactivated_users_grace_period_days` if
it has been disabled.

As I'm writing this another change we could make is that we don't change
either of these site settings if we detect that they aren't set to the
default values.

The drive behind this PR is that there is a discourse instance which
relies on staged users as part of their workflow and this setting was
changed by accident via the import script causing users to be deleted
that shouldn't have been.
2021-02-19 09:33:35 -07:00
Michael Maroszek 144584aacb
fix vbulletin importer to hide soft-deleted posts (#12057)
equal to theads posts can be soft-deleted which results in a visibile = 2 state. at the moment those posts will be imported fully visible.
2021-02-12 14:29:05 +01:00
Bianca Nenciu a71b219c9a
Improvements to phpBB3 import script (#10999)
* FEATURE: Import attachments

* FEATURE: Add support for importing multiple forums in one

* FEATURE: Add support for category and tag mapping

* FEATURE: Import groups

* FIX: Add spaces around images

* FEATURE: Custom mapping of user rank to trust levels

* FIX: Do not fail import if it cannot import polls

* FIX: Optimize existing records lookup

Co-authored-by: Gerhard Schlager <mail@gerhard-schlager.at>
Co-authored-by: Jarek Radosz <jradosz@gmail.com>
2021-01-14 21:44:43 +02:00
Arpit Jalan bd7cbcd8f8
Improve Vanilla import script. (#11701)
- import groups and group users
- import uploads/attachments
- improved code tag parsing
- improved text formatting
- mark topics as solved
2021-01-13 23:10:00 +05:30
Régis Hanol a85d5edbf1
DEV: set digest_attempted_at during migrations (#11369) 2020-12-14 10:58:14 +11:00
Jarek Radosz 2f4a1ff61b
DEV: Update rubocop-discourse from 2.3.2 to 2.4.0 (#11079)
Also fixes whitespace related issues raised by rubocop.
2020-10-30 15:04:29 +01:00
Shane Dela Rosa 8b678b5dbe Update vanilla_body_parser for stability 2020-09-23 09:54:26 -04:00
Rachel Carvalho 812e0d6b5e
FIX: improve Vanilla importing (#10478)
* ensure emails don't have spaces
* import banned users as suspended for 1k yrs
* upgrade users to TL2 if they have comments
* topic: import views, closed and pinned info
* import messages
* encode vanilla usernames for permalinks. Vanilla usernames can contain spaces and special characters.
* parse Vanilla's new rich body format
2020-08-24 16:19:57 -04:00
Justin DiRose d266baff2b
DEV: Improve q2a import script (#10346)
The changes here are largely pulled from the below gist by RGJ on meta.discourse.org. Thanks much for the changes!

https://gist.github.com/discoursehosting/769eff2014d5482f0ab776de03dc3349
2020-07-31 12:04:03 -05:00
Krzysztof Kotlarek e0d9232259
FIX: use allowlist and blocklist terminology (#10209)
This is a PR of the renaming whitelist to allowlist and blacklist to the blocklist.
2020-07-27 10:23:54 +10:00
Régis Hanol 7a6d772ad2 DEV: couple bug fixes in getsatisfaction importer
- Ensure we don't modify a frozen string
- Ensure we have a slug before trying to create a permalink
2020-07-06 17:41:28 +02:00
Gerhard Schlager 1f053173a4 FEATURE: Import script for jForum 2020-06-21 12:12:42 +02:00
Bernhard Suttner e31471585a
DEV: allow to have duplicate topic titles if categegory is different (#10034)
Co-authored-by: Robin Ward <robin.ward@gmail.com>

Co-authored-by: Robin Ward <robin.ward@gmail.com>
2020-06-18 11:19:47 -04:00
Justin DiRose be28fc73a0
DEV: Improvements to Drupal script (#10016)
Refactors script to follow conventions of other importers and adds some features including like import, processing of post raw text, and, if needed, SSO import.
2020-06-10 10:59:17 -05:00
Arpit Jalan a93d24501c FIX: base import script was not updating first_post_created_at column
FEATURE: new rake task to update first_post_created_at column

The not-equal operator (`<>`) in PostgreSQL does not compare values
with NULL. We should instead use `IS DISTINCT FROM` when comparing
values with NULL.
2020-06-04 11:26:40 +05:30
Kane York 869f9b20a2
PERF: Dematerialize topic_reply_count (#9769)
* PERF: Dematerialize topic_reply_count

It's only ever used for trust level promotions that run daily, or compared to 0. We don't need to track it on every post creation.

* UX: Add symbol in TL3 report if topic reply count is capped

* DEV: Drop user_stats.topic_reply_count column
2020-05-14 15:42:00 -07:00
Francesco Frassinelli c6f68e4006
Fix syntax error in fluxbb.rb (#9727) 2020-05-11 11:07:57 -04:00
Krzysztof Kotlarek 9bff0882c3
FEATURE: Nokogumbo (#9577)
* FEATURE: Nokogumbo

Use Nokogumbo HTML parser.
2020-05-05 13:46:57 +10:00
Martin Brennan 867bc3b48e
FIX: Change base importer to create new Bookmark records (#9603)
Also add a spec and fixture with a mock importer that we can use to test the create_X methods of the base importer
2020-05-01 11:34:55 +10:00
Sam Saffron d0d5a138c3
DEV: stop freezing frozen strings
We have the `# frozen_string_literal: true` comment on all our
files. This means all string literals are frozen. There is no need
to call #freeze on any literals.

For files with `# frozen_string_literal: true`

```
puts %w{a b}[0].frozen?
=> true

puts "hi".frozen?
=> true

puts "a #{1} b".frozen?
=> true

puts ("a " + "b").frozen?
=> false

puts (-("a " + "b")).frozen?
=> true
```

For more details see: https://samsaffron.com/archive/2018/02/16/reducing-string-duplication-in-ruby
2020-04-30 16:48:53 +10:00
discoursehosting 094ddb1c1f
VBulletin5 importer improvements (#9477)
- no more hard coded contenttypes
- permalinks for topics, categories, subcategories
- better uploads handling
- tag support
2020-04-22 22:04:59 +02:00
Jay Pfaffman 3c72cbc5de
FIX: Google groups import changed login URL (#9432)
I'm not clear why changing only the `wait_for_url` address was necessary and not also the `get` a few lines above, but this change seems to work for me on both literatecomputing.com Groups and a public group.
2020-04-15 16:45:14 -04:00