b87a089822
FIX: don't block api requests when whitelisted_crawler_user_agents is set
2018-09-14 15:40:20 -04:00
168ffd8384
FEATURE: group warnings about IP level rate limiting
2018-08-13 14:38:20 +10:00
0b7ed8ffaf
FEATURE: backend support for user-selectable components
...
* FEATURE: backend support for user-selectable components
* fix problems with previewing default theme
* rename preview_key => preview_theme_id
* omit default theme from child themes dropdown and try a different fix
* cache & freeze stylesheets arrays
2018-08-08 14:46:34 +10:00
379384ae1e
FIX: never block /srv/status which is used for health checks
...
This route is also very cheap so blocking it is not required
It is still rate limited and so on elsewhere
2018-07-18 12:37:01 +10:00
decf1f27cf
FEATURE: Groundwork for user-selectable theme components
...
* Phase 0 for user-selectable theme components
- Drops `key` column from the `themes` table
- Drops `theme_key` column from the `user_options` table
- Adds `theme_ids` (array of ints default []) column to the `user_options` table and migrates data from `theme_key` to the new column.
- Removes the `default_theme_key` site setting and adds `default_theme_id` instead.
- Replaces `theme_key` cookie with a new one called `theme_ids`
- no longer need Theme.settings_for_client
2018-07-12 14:18:21 +10:00
e72fd7ae4e
FIX: move crawler blocking into anon cache
...
This refinement of previous fix moves the crawler blocking into
anonymous cache
This ensures we never poison the cache incorrectly when blocking crawlers
2018-07-04 11:14:43 +10:00
7f98ed69cd
FIX: move crawler blocking to app controller
...
We need access to site settings in multisite, we do not have access
yet if we attempt to get them in request tracker middleware
2018-07-04 10:30:50 +10:00
e8a6323bea
remove crawler blocking until multisite support
2018-07-03 17:54:45 -04:00
035312d501
FIX: specify path for dosp cookie
2018-04-24 11:07:58 -04:00
ded84a4b58
PERF: improve performance once logged in rate limiter hits
...
If "logged in" is being forced anonymous on certain routes, trigger
the protection for any requests that spend 50ms queueing
This means that ...
1. You need to trip it by having 3 requests take longer than 1 second in 10 second interval
2. Once tripped, if your route is still spending 50m queueuing it will continue to be protected
This means that site will continue to function with almost no delays while it is scaling up to handle the new load
2018-04-23 11:55:25 +10:00
4810ce3607
correct regression
2018-04-18 21:04:08 +10:00
59cd7894d9
FEATURE: if site is under extreme load show anon view
...
If a particular path is being hit extremely hard by logged on users,
revert to anonymous cached view.
This will only come into effect if 3 requests queue for longer than 2 seconds
on a *single* path.
This can happen if a URL is shared with the entire forum base and everyone
is logged on
2018-04-18 16:58:57 +10:00
b87fa6d749
FIX: blacklisted crawlers could get through by omitting the accept header
2018-04-17 12:39:30 -04:00
9980f18d86
FEATURE: track request queueing as early as possible
2018-04-17 18:06:17 +10:00
4d12ff2e8a
when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler.
2018-03-27 13:44:14 -04:00
a84bb81ab5
only applies to get html requests
2018-03-22 17:57:44 -04:00
ced7e9a691
FEATURE: control which web crawlers can access using a whitelist or blacklist
2018-03-22 15:41:02 -04:00
0134e41286
FEATURE: detect when client thinks user is logged on but is not
...
This cleans up an error condition where UI thinks a user is logged on
but the user is not. If this happens user will be prompted to refresh.
2018-03-06 16:49:31 +11:00
f0d5f83424
FEATURE: limit assets less that non asset paths
...
By default assets can be requested up to 200 times per 10 seconds
from the app, this includes CSS and avatars
2018-03-06 15:20:39 +11:00
f295a18e94
FIX: stop double counting net calls in logs
2018-02-28 10:45:11 +11:00
ca1a3f37e3
FEATURE: add instrumentation for all external net calls
2018-02-21 15:20:29 +11:00
3e835047da
Remove "already initialized" constant warning.
2018-02-13 08:55:15 +08:00
df8e43abdd
use lazy & instead of try
...
unregister ip skipper in test
raise if called when a skipper is in play
2018-02-06 10:38:15 +11:00
eefd226611
Add extensibility point to `request_tracker` to skip IP addresses
...
This is useful if you want to run a per IP rate limiter but want to be
able to skip some IPs with custom logic.
2018-02-05 17:49:40 -05:00
2437b0d531
FIX: regression, missing 404 page
2018-01-23 09:00:28 +11:00
f26ff290c3
FEATURE: Shorten setting name to max_reqs
...
So it is consistent with other settings
2018-01-22 13:18:30 +11:00
8bf91b8dca
correct tracking of x runtime
2018-01-19 17:51:19 +11:00
8ad43f01c2
FIX: correctly log topic timings as background
2018-01-19 10:37:43 +11:00
12872d03be
PERF: run post timings in background
...
This means that if a very large amount of registered users hit
a single topic we will handle it gracefully, even if db gets slow.
2018-01-19 08:27:29 +11:00
442a17bfb2
PERF: bypass omniauth unless in an auth path
2018-01-15 12:44:54 +11:00
49ed382c2a
FIX: return 429 when admin api key is limited on admin route
...
This also handles a general case where exceptions leak out prior to being handled by the application controller
2018-01-12 14:15:26 +11:00
cecd7d0d07
FEATURE: global rate limiter can bypass local IPs
2018-01-08 08:39:17 +11:00
715cb98e95
add better diagnostics for rate limits
2018-01-05 12:14:28 +11:00
bbc606988f
improve message
2017-12-20 10:12:33 +11:00
4986ebcf24
FEATURE: optional default off global per ip rate limiter
2017-12-11 17:52:57 +11:00
df84e1c358
Correctly track hijacked requests
2017-11-28 16:47:20 +11:00
a4c539bade
FEATURE: Allow registration of detailed request logger
...
Detailed request loggers can be used to gather rich timing info
from all requests (which in turn can be forwarded to monitoring solution)
Middleware::RequestTracker.detailed_request_logger(->|env, data| do
# do stuff with env and data
end
2017-10-18 12:10:30 +11:00
5012d46cbd
Add rubocop to our build. ( #5004 )
2017-07-28 10:20:09 +09:00
bdb848b4f3
Split the theme_key so we extract the key from seq
2017-06-15 14:09:44 -04:00
ac1f84d3e1
SECURITY: theme key should be an anon cache breaker
2017-06-15 09:36:27 -04:00
a3e8c3cd7b
FEATURE: Native theme support
...
This feature introduces the concept of themes. Themes are an evolution
of site customizations.
Themes introduce two very big conceptual changes:
- A theme may include other "child themes", children can include grand
children and so on.
- A theme may specify a color scheme
The change does away with the idea of "enabled" color schemes.
It also adds a bunch of big niceties like
- You can source a theme from a git repo
- History for themes is much improved
- You can only have a single enabled theme. Themes can be selected by
users, if you opt for it.
On a technical level this change comes with a whole bunch of goodies
- All CSS is now compiled using a custom pipeline that uses libsass
see /lib/stylesheet
- There is a single pipeline for css compilation (in the past we used
one for customizations and another one for the rest of the app
- The stylesheet pipeline is now divorced of sprockets, there is no
reliance on sprockets for CSS bundling
- CSS is generated with source maps everywhere (including themes) this
makes debugging much easier
- Our "live reloader" is smarter and avoid a flash of unstyled content
we run a file watcher in "puma" in dev so you no longer need to run
rake autospec to watch for CSS changes
2017-04-12 10:53:49 -04:00
ea9f7a41af
remove gctools (no longer used) add gctracer for debugging
2016-12-20 15:07:30 +11:00
39a524aac8
FEATURE: brotli cdn bypass for assets
...
Allow CDNS that strip out brotli encoding to use brotli regardless
2016-12-05 13:57:09 +11:00
497ff76a67
make sure 1 is a string
2016-10-27 18:08:01 +11:00
3e7190866a
make code a bit safer
2016-10-27 16:50:56 +11:00
8a477f1857
FEATURE: added X-Discourse-TrackView header
...
This header is set to 1 if the particular request is a tracked page view
2016-10-27 16:48:27 +11:00
a9823ab59a
FIX: Use a cookie to bypass the anon cache
2015-10-28 17:16:56 -04:00
ec4a1bb2c4
FIX: page tracking was not properly tracking transitions
...
PERF: move closure to self contained method so env is released earlier.
2015-09-17 11:06:33 +10:00
d7aa4e81d6
revert 8f435fcbf6
2015-07-31 15:22:30 +02:00
86cd1a19cc
FEATURE: page view stats for mobile view
2015-07-03 17:19:33 -04:00
1f9761e85d
FEATURE: add a header to denote an anonymous req was cached
...
(X-Discourse-Cached)
2015-06-16 10:30:06 +10:00
90eaad336d
FEATURE: allow users to pick a CDN for s3 assets
2015-05-26 11:13:12 +10:00
bb0c2813ac
FEATURE: generate (avatar) thumbnails in a background task
...
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar
PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
2015-05-25 17:59:00 +02:00
f5af4768eb
FEATURE: add clean support for running Discourse in a subfolder
...
To setup set DISCOURSE_RELATIVE_URL_ROOT to the folder you wish
2015-03-09 13:14:29 +11:00
cbe18eb0df
FEATURE: allow view exclusion using custom header
...
Set Discourse-Track-View to either "0" or "false" to exclude request
2015-02-26 11:41:11 +11:00
fce9e296e7
background reqs failed or not are always counted seperately
2015-02-12 09:47:46 +11:00
3cf87b94c9
whitespace
2015-02-11 09:39:04 +11:00
0ce6524153
correct brokeness
2015-02-10 17:05:24 +11:00
acda6ebd60
FIX: view tracking needs to release data earlier
...
retaining data during queuing was causing huge memory spikes
2015-02-10 17:03:33 +11:00
820ce8765e
refactor traffic report
...
split traffic report in 2, page view vs raw traffic
hide raw traffic report by default
improve flushing logic for application reqs
2015-02-06 14:39:16 +11:00
08b790b3c2
improve metrics gathered using in our traffic section
...
this also pulls out the middleware into its own home and inserts in front
2015-02-05 16:08:52 +11:00
8690c7c49f
defer counting to avoid race condition
2015-02-05 12:19:21 +11:00
c150c55e2d
FEATURE: rudimentary view tracking wired in
2015-02-04 16:15:16 +11:00
4f8dfd84b9
FIX: vary accept for cache, seems most correct
2014-09-09 10:25:49 +10:00
8646c21e89
FIX: anonymous cache could cache json for html requests
2014-09-09 09:46:26 +10:00
6301a43d57
Not initializing variable for looping if unused in loop
2014-08-15 03:24:55 +05:30
6019e3f257
FIX: remove hardcoding from middleware stack so we can control it
2014-07-10 17:01:21 +10:00
5032c96486
FIX: disable x accl redirect for CDN assets
...
We need to keep headers in tact
2014-07-10 16:32:46 +10:00
35952055e2
BUGFIX: web crawlers messing with anon caching
2014-04-29 10:48:09 +10:00
024b86e112
BUGFIX: incorrect OOBGC algorithm, not dynamically adjusting
2014-04-02 12:26:59 +11:00
f617086e91
FIX oobgc support for Ruby 2.1
2014-02-18 14:43:24 +11:00
a6f4d00670
Add oobgc compatible with Ruby 2.1
2014-02-18 14:10:03 +11:00
18e719b0af
In development mode, if an avatar is missing just serve up a placeholder
2014-01-17 11:47:23 -05:00
adc9a58f4a
BUGFIX: anon cache was mucking with params
2014-01-09 16:49:12 +11:00
74c1555885
BUGFIX: fix broken spec
2014-01-09 15:11:04 +11:00
177983afe6
BUGFIX: mobile ui was being cached for anon views
2014-01-09 14:08:42 +11:00
b804f0bec9
add license for oobgc, should not be gpl
2013-11-22 17:04:32 +11:00
67222ce8a0
tune down ... 80k is over aggressive
2013-11-19 15:06:34 +11:00
518ae2c608
added a minimal number of free slots that should always exist
2013-11-19 14:58:27 +11:00
2841434e36
slow down decay to accommodate for long polling and other tiny reqs
2013-11-19 14:47:13 +11:00
ef64c1b757
more logging for miss
2013-11-19 14:27:18 +11:00
f61920565a
reimplemnt oobgc algorithm
2013-11-19 14:13:42 +11:00
2984d649f5
remove log
2013-11-15 18:22:02 +11:00
8a868bad58
time saving log
2013-11-15 18:21:19 +11:00
ce6b0ff22e
remove puts
2013-11-15 17:54:29 +11:00
c1d7bbdc79
more diags
2013-11-15 17:26:58 +11:00
a92ada0493
tune down min requests, tune up largest multiplier, tune up history
2013-11-15 17:17:08 +11:00
466459db51
more diags for a bit
2013-11-15 17:08:38 +11:00
8513164b85
lets not explode if init is called incorrectly
2013-11-15 17:03:03 +11:00
3219afd666
fix bug, num request should be zeroed, better logging
2013-11-15 16:56:21 +11:00
461972844e
unicorn out of band GC
2013-11-15 12:15:49 +11:00
3d647a4b41
remove rack cache, it has been causing trouble
...
instead implement an aggressive anonymous cache that is stored in redis
this cache is sitting in the front of the middleware stack enabled only in production
TODO: expire it more intelligently when stuff is created
2013-10-16 16:39:18 +11:00
21e08a423e
Added a little documentation to TurboDev middleware
2013-09-10 15:21:07 -04:00
858e51c8ab
Use `no-cache` in development mode for assets. Using `must-revalidate`, which is
...
Rails' default, seems to have Chrome sometimes not request assets in development
mode even though it's supposed to revalidate every time.
2013-09-09 16:55:41 -04:00
28d111d370
typo
2013-04-18 17:23:52 +10:00
8367951000
sprockets upgrade
...
stop bundling all js files in dev, replace with turbo boosted serving of assets
2013-04-18 16:33:25 +10:00
Neil Lalonde
Sam
Osama Sayegh
Guo Xiang Tan
Robin Ward
Guo Xiang Tan
Régis Hanol
Akshay