Gerhard Schlager
0f3c3bc309
Make import scripts work with frozen strings
2019-05-30 22:22:24 +02:00
Sam Saffron
30990006a9
DEV: enable frozen string literal on all files
...
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.
Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan
24347ace10
FIX: Properly associate user_profiles background urls via upload id.
...
`Upload#url` is more likely and can change from time to time. When it
does changes, we don't want to have to look through multiple tables to
ensure that the URLs are all up to date. Instead, we simply associate
uploads properly to `UserProfile` so that it does not have to replicate
the URLs in the table.
2019-05-02 14:58:24 +08:00
Régis Hanol
3c9c95ac83
Update Rubocop to 0.60
2018-12-04 10:48:16 +01:00
David Taylor
9248ad1905
DEV: Enable `Style/SingleLineMethods` and `Style/Semicolon` in Rubocop ( #6717 )
2018-12-04 11:48:13 +08:00
Régis Hanol
afa22a0c6f
REFACTOR: more 'fake_email' to base importer
2018-10-22 11:12:40 +02:00
Guo Xiang Tan
434035f167
FIX: Link post to uploads in `PostCreator`.
...
* This ensures that uploads are linked to their post on creation
instead of a background job which may be delayed if Sidekiq
is facing difficulties.
2018-09-06 11:18:11 +08:00
Vinoth Kannan
652b32484f
Assign default value for message template matches
2018-06-26 05:16:03 +05:30
Vinoth Kannan
f3011c709b
Extract html content from lithium message template
2018-06-26 05:07:32 +05:30
Sam
5f64fd0a21
DEV: remove exec_sql and replace with mini_sql
...
Introduce new patterns for direct sql that are safe and fast.
MiniSql is not prone to memory bloat that can happen with direct PG usage.
It also has an extremely fast materializer and very a convenient API
- DB.exec(sql, *params) => runs sql returns row count
- DB.query(sql, *params) => runs sql returns usable objects (not a hash)
- DB.query_hash(sql, *params) => runs sql returns an array of hashes
- DB.query_single(sql, *params) => runs sql and returns a flat one dimensional array
- DB.build(sql) => returns a sql builder
See more at: https://github.com/discourse/mini_sql
2018-06-19 16:13:36 +10:00
Vinoth Kannan
4ffa4a28b0
FIX: duplicate_emails get overridden in new batch of import
2018-06-19 00:21:48 +05:30
Vinoth Kannan
750367007c
REFACTOR: Import user visits from lithium database
2018-06-18 20:38:57 +05:30
Vinoth Kannan
ef4a86456b
Add attachment folder name in prefix for lithium import
2018-06-18 18:29:14 +05:30
Vinoth Kannan
2a0f409b9d
Use lowercased email addresses to check duplicates
2018-06-16 20:34:37 +05:30
Vinoth Kannan
ac44374a59
Import user visits from user_log table
2018-06-16 19:10:55 +05:30
Vinoth Kannan
620a1524cb
Use plus addressing email address for duplicates
2018-06-07 19:11:55 +05:30
Vinoth Kannan
bb12fa3fdc
Migrate user mentions in lithium import
2018-05-21 18:19:22 +05:30
Vinoth Kannan
b229c112f6
FIX: variable name typo
2018-05-21 13:47:30 +05:30
Vinoth Kannan
09151190f9
FIX: Use avatar_dir to import user avatars
2018-05-21 13:43:23 +05:30
Vinoth Kannan
c9c3a83261
Importing lithium post images and attachments
2018-05-21 13:34:52 +05:30
Vinoth Kannan
f3385a74cb
Importing lithium topic tags
2018-05-19 11:24:48 +05:30
Vinoth Kannan
ba0dd5889d
Improvements in importing the lithium pms
2018-05-18 22:57:15 +05:30
Vinoth Kannan
9f92fdded0
Improvements in lithium topic and post import
2018-05-18 18:53:18 +05:30
Vinoth Kannan
9d4d6276b7
Import user profile fields and avatars
2018-05-18 17:11:20 +05:30
Vinoth Kannan
0639b902dc
Import lithium nodes into categories
2018-05-18 01:34:31 +05:30
Vinoth Kannan
ade7c94b43
Import lithium roles into groups
2018-05-16 17:26:15 +05:30
Guo Xiang Tan
5012d46cbd
Add rubocop to our build. ( #5004 )
2017-07-28 10:20:09 +09:00
Sam Saffron
d0c5205a52
Feature: Change markdown engine to markdown it
...
This commit removes the old evilstreak markdownjs engine.
- Adds specs to WhiteLister and changes it to stop using globals
(Fixes large memory leak)
- Fixes edge cases around bbcode handling
- Removes mdtest which is no longer valid (to be replaced with
CommonMark)
- Updates MiniRacer to correct minor unmanaged memory leak
- Fixes plugin specs
2017-07-17 11:41:34 -04:00
Sam
79a084dd58
Revert "remove old markdown engine work-in-progress"
...
This reverts commit ee470b5317
.
2017-07-12 18:10:51 -04:00
Sam Saffron
ee470b5317
remove old markdown engine work-in-progress
2017-07-12 17:44:40 -04:00
Régis Hanol
9641d2413d
REFACTOR: upload workflow creation into UploadCreator
...
- Automatically convert large-ish PNG/BMP to JPEG
- Updated fast_image to latest version
2017-05-11 00:16:57 +02:00
Sam
a6be3d7cbe
skip post processing stuff we did not import
2015-10-16 10:41:21 +11:00
Sam
606aeb9d55
improvements to importer
...
- improve perf of test for existing posts
- always use a system guardian when importing posts
- for lithuim importer requery raw (transform is not repeatable)
2015-10-16 10:41:21 +11:00
Kane York
821124f3fd
FEATURE: Skip batches if all records exist
...
Update all import scripts to take advantage of all_records_exist?
2015-09-21 17:13:02 -07:00
Sam
02a38eebbb
correct logic in importer
2015-08-03 18:30:26 +10:00
Sam
a1f02d4baa
correct logic, add missing permalink creator
2015-08-03 17:35:35 +10:00
Sam
fd82107df8
correct bugs in lithium importer
2015-08-03 17:16:19 +10:00
Sam
64bbf2c1c4
correct closing logic for wd importer
2015-08-03 16:18:28 +10:00
Sam
f7ecfb3a83
handle image uploads
2015-07-29 18:34:10 +10:00
Sam
c699a6fca9
PMs now import for my current data dump
2015-07-28 18:16:29 +10:00
Sam
7891ed8145
Work in progress lithium importer
...
* PM import partially implemented
* Proper category names
* Smiley mappings
2015-07-28 17:46:17 +10:00
Sam
3a54923116
FIX: permalink normalization not applied at constraint
...
implement permalink import for lithium
2015-07-22 13:40:45 +10:00
Sam
d873e68059
bypass table when normalizing.
2015-07-21 10:45:18 +10:00
Sam
c552f846ae
add index
2015-07-20 16:56:46 +10:00
Sam
24eb12778f
work in progress lithium importer
2015-07-17 17:16:47 +10:00
Sam
bbfd18aa59
wip lithuim importer
2015-07-17 12:43:45 +10:00
Sam
35e1b52d54
remove 0 view filter
2015-07-16 16:57:55 +10:00
Sam
88578f74d0
another subst
2015-07-16 16:24:46 +10:00
Sam
e449bf531c
more post cleanup rules
2015-07-16 16:23:14 +10:00
Sam
5db345705b
allow views to be specified in import mode
2015-07-16 16:11:20 +10:00