Truly testing for JSON validity would require defining a new postgres function. Checking just the first character should take care of all the cases of invalid historic data that we've seen.
In the old custom_field-based system, it was possible for a url to be both 'downloaded' and 'broken'. The new table enforces uniqueness, so we need to drop invalid data.
On some installations, this would fail with 'index row size exceeds btree version 4 maximum'. This commit replaces the (post_id, url)` index with a `(post_id, md5(url))` index, which is much more space efficient.
This will make future changes to the 'pull hotlinked images' system easier. This commit should not introduce any functional change.
For now, the old post_custom_field data is kept in the database. This will be dropped in a future commit.
David Taylor