Commit Graph

63 Commits

Author SHA1 Message Date
Alan Guo Xiang Tan cadf5eafe6 DEV: Move Discourse app specific concern out of unicorn conf. 2021-06-04 09:13:34 +08:00
David Taylor cac7725e28
DEV: Improve `bin/unicorn` boot time in development environment (#12900)
This commit makes a few changes to improve boot time in development environments. It will have no effect on production boot times.

- Skip the SchemaCache warmup. In development mode, the SchemaCache is refreshed every time there is a code change, so warmup is of limited use.

- Skip warming up PrettyText. This adds ~2s to each web worker's boot time. The vast majority of requests do not use PrettyText, so it is more efficient to defer its warmup until it's needed

- Skip the intentional 1 second pause during Unicorn worker forking. The comment (which also exists in Unicorn's documentation) suggests this works around a Unix signal handling bug, but I haven't been able to locate any more information. Skipping it in dev will significantly speed up boot. If we start to see issues, we can revert this change.

On my machine, this improves `/bin/unicorn` boot time from >10s to ~4s
2021-04-30 11:32:13 +01:00
Jarek Radosz 956f849250 DEV: Enable unicorn logger in test environment 2021-02-11 15:24:15 +01:00
David Taylor 1d024f77a6
FEATURE: Allow plugins to register demon processes (#11493)
This allows plugins to call `register_demon_process` with a Class inheriting from Demon::Base. The unicorn master process will take care of spawning, monitoring and restarting the process. This API should be used with extreme caution, but it is significantly cleaner than spawning processes/threads in an `after_initialize` block.

This commit also cleans up the demon spawning logging so that it uses the same format as unicorn worker logging. It also switches to the block form of `fork` to ensure that Demons exit after running, rather than returning execution to where the fork took place.
2020-12-16 09:43:39 +00:00
Matt Palmer 1ad270965c
FEATURE: Allow the specification of an arbitrary unicorn listen address
Useful if you want to, say, have your unicorn listen on a Unix domain
socket, rather than a TCP port, or you want to be able to bind to a
single address other than 127.0.0.1.
2020-07-28 13:03:17 +10:00
Martin Brennan a9905ef7e5
FIX: Add enable_email_sync_demon global variable and disable EmailSync demon by default (#10304)
Demon::EmailSync is used in conjunction with the SiteSetting.enable_imap to sync N IMAP mailboxes with specific groups. It is a process started in unicorn.conf, and it spawns N threads (one for each multisite connection) and for each database spans another N threads (one for each configured group).

We want this off by default so the process is not started when it does not need to be (e.g. development, test, certain hosting tiers)
2020-07-24 17:09:29 +10:00
Guo Xiang Tan bb8f1ce8b1
DEV: Consolidate Unicorn error backtraces when logstash is enabled. 2020-07-21 15:35:41 +08:00
Dan Ungureanu c72bc27888
FEATURE: Implement support for IMAP and SMTP email protocols. (#8301)
Co-authored-by: Joffrey JAFFEUX <j.jaffeux@gmail.com>
2020-07-10 12:05:55 +03:00
Guo Xiang Tan b66f2187f1
DEV: Don't use logstash for unicorn if ENV is blank. 2020-06-11 15:58:18 +08:00
Guo Xiang Tan e4cd4f7e0b
DEV: Avoid reaching for `Redis#_client` which is considered deprecated. 2020-06-02 11:46:55 +08:00
David Taylor ed6b3b82bd
FIX: Reopen sidekiq log files after rotation (#9429)
Unicorn uses the USR1 to indicate that log files should be reopened. This commit implements the same functionality for our forked sidekiq workers:

- USR1 is intercepted in the unicorn master, and re-issued to all child processes
- USR1 is trapped in the sidekiq processes, and `Unicorn::Util.reopen_logs` is used to re-open log files
2020-04-16 12:13:13 +01:00
Jarek Radosz 85e03a7f68
DEV: Replace `Time.new` with `Time.now` (#9142)
(or `Time.zone.now`)
2020-03-09 17:37:49 +01:00
David Taylor 7b6bafc651 Revert "DEV: Bind to ipv6 loopback address in addition to ipv4 (#8544)"
This reverts commit 1002bf62e5.
2019-12-16 15:51:04 +00:00
David Taylor 1002bf62e5
DEV: Bind to ipv6 loopback address in addition to ipv4 (#8544) 2019-12-16 11:01:02 +00:00
Joffrey JAFFEUX 0d3d2c43a0
DEV: s/\$redis/Discourse\.redis (#8431)
This commit also adds a rubocop rule to prevent global variables.
2019-12-03 10:05:53 +01:00
Jeff Wong 64d51ab45e DEV: add UNICORN_BIND_ALL
listen on local interface by default.
2019-10-28 14:11:19 -07:00
Sam Saffron 8d5f47dded PREF: optimise preloading application
We preload to ensure as much memory as possible is reused from unicorn master
to various workers using copy-on-write (sidekiq, unicorn)

This migrates the preloading code into the Discourse module for easier
reuse and adds 3 notable preloading changes

1. We attempt to localize a string on each site, ensuring we warmup
the i18n

2. We preload all our templates (compiling .erb to class)

3. We warm-up our search tokenizer which uses cppjieba which is a large
memory consumer, this will only cause a warmup on CJK sites or sites with
the special site setting enabled.
2019-10-07 00:33:37 -04:00
Sam Saffron e0a403edfc PERF: ensure we warm up schema cache in the entire multisite
This makes sure that all processes that fork off the master have a fully
operation schema cache.

In Rails 6, schema cache is now bolted to the connection pool. This change
ensures the cache on all pools is fully populated prior to forking.

The bolting of cache to connection pool does lead to some strange cases
where a connection can "steal" the cache from another connection, which
can cause stuff to possibly hang or deadlock. This change minimizes the risk
of this happening cause it is already primed.

We make a STRONG assumption that the schema is always the same on all sites
when we spin up a multisite cluster.
2019-09-16 17:38:13 +10:00
David Taylor e2449f9f23 Revert "Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)"""
This reverts commit c3497559be.
2019-08-30 11:26:16 +01:00
Sam Saffron c3497559be Revert "Revert "FIX: Heartbeat check per sidekiq process (#7873)""
This reverts commit e805d44965.
We now have mechanisms in place to ensure heartbeat will always
be scheduled even if the scheduler is overloaded per: 098f938b
2019-08-30 10:12:10 +10:00
OsamaSayegh e805d44965 Revert "FIX: Heartbeat check per sidekiq process (#7873)"
This reverts commit 340855da55.
2019-08-27 11:56:23 +00:00
Osama Sayegh 340855da55
FIX: Heartbeat check per sidekiq process (#7873)
* FIX: Heartbeat check per sidekiq process

* Rename method

* Remove heartbeat queues of previous bootups

* Regis feedback

* Refactor before_start

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Update lib/demon/sidekiq.rb

Co-Authored-By: Régis Hanol <regis@hanol.fr>

* Expire redis keys after 3600 seconds

* Don't use redis to store the list of queues
2019-08-26 09:33:49 +03:00
Sam Saffron 76173dea87 DEV: ensure we never fork v8 contexts from unicorn
v8 forking is not supported and can lead to memory leaks.

This commit handles the most common case which is the unicorn master forking

There are still some cases related to backup where we fork, however those
forks are usually short lived so the memory leak is not severe, burning
the contexts in the master process could break sidekiq or web process that
do the actual forking
2019-05-16 09:50:34 +10:00
Sam Saffron 30990006a9 DEV: enable frozen string literal on all files
This reduces chances of errors where consumers of strings mutate inputs
and reduces memory usage of the app.

Test suite passes now, but there may be some stuff left, so we will run
a few sites on a branch prior to merging
2019-05-13 09:31:32 +08:00
Guo Xiang Tan 4774633dac DEV: Remove `StatsSocket`.
Removed in favor of https://github.com/discourse/discourse-prometheus.
2019-03-26 18:16:58 +08:00
Guo Xiang Tan edbcc992d4 Allow unicorn timeout to be configurable via ENV. 2018-09-04 13:21:41 +08:00
Sam 1172e141cd adjust timeouts in dev 2018-08-15 11:13:43 +10:00
Angus McLeod 6c41b54b2e FIX: create tmp if it doesn't exist when creating tmp/pids
I get this error if I stop a dev server, ``rm -rf tmp`` and start it again:
```
`mkdir': No such file or directory @ dir_s_mkdir - /Users/angusmcleod/discourse/discourse/tmp/pids (Errno::ENOENT)
```
This fixes it.

See: f3549291a3 (diff-26ac62db6c6a4582de3bbf2615790c23R22)
2018-08-08 14:49:09 +10:00
Sam f3549291a3 DEV: use unicorn in development
This commit also cleans up a bunch of pointless noise each time we boot app

- narrative was loading i18n cause redefinition of consts
- discourse.rb was loaded twice as was auth
- bin/unicorn now does all the smart things and boots unicron in dev
- bin/rails s will boot unicorn with no params
- remove bin/puma which only causes confusion
2018-08-07 17:13:47 +10:00
Guo Xiang Tan 805fd17b23 ActiveRecord in Rails 5.2 discards connection pools after fork. 2018-06-12 09:30:52 +08:00
Guo Xiang Tan 01f126e38f Simplify unicorn config. 2018-06-07 09:03:16 +08:00
Guo Xiang Tan a4e6662833 FIX: Disconnects all connections in the pool before forking.
* We were leaking connections as a result. Connections opened
  before the fork were never closed.
2018-06-06 14:45:05 +08:00
Guo Xiang Tan 45fe5dc793 `$redis.client` -> `$redis._client`.
See c239abb43c
2018-04-20 13:01:17 +08:00
Guo Xiang Tan db68434db1 More control over Unicorn logstash logger. 2017-11-15 12:28:36 +08:00
Guo Xiang Tan 89d9ffa884 EXPERIMENTAL: Allow logs to be shipped via different methods.
See https://github.com/dwbutler/logstash-logger#uri-configuration for
a list of available methods.
2017-11-15 09:11:33 +08:00
Guo Xiang Tan c9df21e131 FEATURE: Allow Unicorn logs to be JSON formatted. 2017-11-06 12:46:14 +08:00
Sam 220a961898 correct unicorn sidekiq messaging 2017-10-23 17:12:25 +11:00
Sam 15932f391e FEATURE: Add events when web and sidekiq forks start 2017-10-23 15:40:45 +11:00
Guo Xiang Tan 91dfe23d6f Exclude versions table when loading models in Unicorn. 2017-10-12 17:14:36 +08:00
Sam a4d4db4f0c PERF: code not correctly caching git commands
Every check for Discourse version could result in shelling out.
2017-10-04 14:22:38 +11:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Guo Xiang Tan 87e77267fa Add ENV variable to specify path to write unicorn's pid. 2017-06-06 11:46:01 +09:00
Sam 20778fbf58 fix handling of stats socket 2017-04-21 12:37:28 -04:00
Sam 0b3aec9c94 FEATURE: set UNICORN_STATS_SOCKET_DIR for status socket
eg:

sam@ubuntu stats_sockets % socat - UNIX-CONNECT:9622.sock
gc_stat
{"count":46,"heap_allocated_pages":2459,"heap_sorted_length":2460,"heap_allocatable_pages":0,"heap_available_slots":1002267,"heap_live_slots":647293,"heap_free_slots":354974,"heap_final_slots":0,"heap_marked_slots":503494,"heap_swept_slots":498773,"heap_eden_pages":2459,"heap_tomb_pages":0,"total_allocated_pages":2459,"total_freed_pages":0,"total_allocated_objects":4337014,"total_freed_objects":3689721,"malloc_increase_bytes":6448248,"malloc_increase_bytes_limit":29188387,"minor_gc_count":36,"major_gc_count":10,"remembered_wb_unprotected_objects":19958,"remembered_wb_unprotected_objects_limit":39842,"old_objects":462019,"old_objects_limit":895782,"oldmalloc_increase_bytes":6448696,"oldmalloc_increase_bytes_limit":19350882}
2017-04-21 11:37:03 -04:00
Sam f9f38873a2 FEATURE: add support for SIGTSTP which stops sidekiqs
Out of the box this signal "suspends" the process, but
we already use usr1 and usr2 and this is for an edge case
where the end user suspends it by typing "stop"
2017-03-10 11:36:10 -05:00
Guo Xiang Tan f69f225f65 FIX: Don't prevent unicorn worker from starting if warmup fails. 2016-09-06 14:02:08 +08:00
Sam Saffron 6777bd2629 warm up v8 after fork 2016-07-16 15:11:34 +10:00
Robin Ward 2d1d9a78b8 Revert "Try warming up pretty text before we fork"
This reverts commit 2e71e6fc6f.
2016-07-15 18:10:22 -04:00
Robin Ward 2e71e6fc6f Try warming up pretty text before we fork 2016-07-13 17:07:39 -04:00
Sam a3885a18f8 extra logic to force kill orphan sidekiqs 2015-06-18 15:32:04 +10:00