Discource-C
/
discourse
mirror of https://github.com/discourse/discourse.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

FIX: Move downloading of Maxmind databases to after `assets:precompile` (#28157) 

We have been seeing `ZLib::BufError` when running the `assets:precompile` rake
task.

```
I, [2024-07-30T05:19:58.807019 #1059]  INFO -- : Writing /var/www/discourse/public/assets/scripts/discourse-test-listen-boot-9b14a0fc65c689577e6a428dcfd680205516fe211700a71c7adb5cbcf4df2cc5.js
rake aborted!
Zlib::BufError: buffer error (Zlib::BufError)
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb💯in `<<'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache/file_store.rb💯in `set'
/var/www/discourse/vendor/bundle/ruby/3.3.0/gems/sprockets-3.7.3/lib/sprockets/cache.rb:212:in `set'
```

The hypothesis here is that some thread unsafe issue is causing the
problem since we download the Maxmind databases in a thread and run
decompression operations once the gzip file is downloaded.

In the near term, we plan to move downloading of Maxmind databases out
of the Rake task into a scheduled job so this patch should be considered
a temporary solution.

The trade-off here is that build time will slightly increase since we
are not longer downloading Maxmind databases while precompiling assets
at the same time.
This commit is contained in:
Alan Guo Xiang Tan 2024-07-31 08:55:01 +08:00 committed by GitHub
parent cb3943a1f6
commit e9c0a6dffe
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
2 changed files with 16 additions and 23 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
lib/tasks/assets.rake
View File

@ -325,14 +325,11 @@ task "assets:precompile:theme_transpiler": "environment" do
end
# Run these tasks **before** Rails' "assets:precompile" task
task "assets:precompile": %w[
assets:precompile:before
maxminddb:refresh
assets:precompile:theme_transpiler
]
task "assets:precompile": %w[assets:precompile:before assets:precompile:theme_transpiler]
# Run these tasks **after** Rails' "assets:precompile" task
Rake::Task["assets:precompile"].enhance do
Rake::Task["assets:precompile:compress_js"].invoke
Rake::Task["assets:precompile:css"].invoke
Rake::Task["maxminddb:refresh"].invoke
end

32
lib/tasks/maxminddb.rake
View File

@ -64,25 +64,21 @@ task "maxminddb:refresh": "environment" do
puts "Downloading MaxMindDB..."
maxmind_thread =
Thread.new do
name = "unknown"
begin
GEOLITE_DBS.each do |db|
name = db
DiscourseIpInfo.mmdb_download(db)
end
name = "unknown"
if GlobalSetting.maxmind_backup_path.present?
copy_maxmind(DiscourseIpInfo.path, GlobalSetting.maxmind_backup_path)
end
rescue OpenURI::HTTPError => e
STDERR.puts("*" * 100)
STDERR.puts("MaxMindDB (#{name}) could not be downloaded: #{e}")
STDERR.puts("*" * 100)
Rails.logger.warn("MaxMindDB (#{name}) could not be downloaded: #{e}")
end
begin
GEOLITE_DBS.each do |db|
name = db
DiscourseIpInfo.mmdb_download(db)
end
at_exit { maxmind_thread.join }
if GlobalSetting.maxmind_backup_path.present?
copy_maxmind(DiscourseIpInfo.path, GlobalSetting.maxmind_backup_path)
end
rescue OpenURI::HTTPError => e
STDERR.puts("*" * 100)
STDERR.puts("MaxMindDB (#{name}) could not be downloaded: #{e}")
STDERR.puts("*" * 100)
Rails.logger.warn("MaxMindDB (#{name}) could not be downloaded: #{e}")
end
end