Commit Graph

146 Commits

Author SHA1 Message Date
Neil Lalonde b87a089822 FIX: don't block api requests when whitelisted_crawler_user_agents is set 2018-09-14 15:40:20 -04:00
Sam 168ffd8384 FEATURE: group warnings about IP level rate limiting 2018-08-13 14:38:20 +10:00
Osama Sayegh 0b7ed8ffaf FEATURE: backend support for user-selectable components
* FEATURE: backend support for user-selectable components

* fix problems with previewing default theme

* rename preview_key => preview_theme_id

* omit default theme from child themes dropdown and try a different fix

* cache & freeze stylesheets arrays
2018-08-08 14:46:34 +10:00
Sam 379384ae1e FIX: never block /srv/status which is used for health checks
This route is also very cheap so blocking it is not required

It is still rate limited and so on elsewhere
2018-07-18 12:37:01 +10:00
OsamaSayegh decf1f27cf FEATURE: Groundwork for user-selectable theme components
* Phase 0 for user-selectable theme components

- Drops `key` column from the `themes` table
- Drops `theme_key` column from the `user_options` table
- Adds `theme_ids` (array of ints default []) column to the `user_options` table and migrates data from `theme_key` to the new column.
- Removes the `default_theme_key` site setting and adds `default_theme_id` instead.
- Replaces `theme_key` cookie with a new one called `theme_ids`
- no longer need Theme.settings_for_client
2018-07-12 14:18:21 +10:00
Sam e72fd7ae4e FIX: move crawler blocking into anon cache
This refinement of previous fix moves the crawler blocking into
anonymous cache

This ensures we never poison the cache incorrectly when blocking crawlers
2018-07-04 11:14:43 +10:00
Sam 7f98ed69cd FIX: move crawler blocking to app controller
We need access to site settings in multisite, we do not have access
yet if we attempt to get them in request tracker middleware
2018-07-04 10:30:50 +10:00
Neil Lalonde e8a6323bea remove crawler blocking until multisite support 2018-07-03 17:54:45 -04:00
Sam 035312d501 FIX: specify path for dosp cookie 2018-04-24 11:07:58 -04:00
Sam ded84a4b58 PERF: improve performance once logged in rate limiter hits
If "logged in" is being forced anonymous on certain routes, trigger
the protection for any requests that spend 50ms queueing

This means that ...

1. You need to trip it by having 3 requests take longer than 1 second in 10 second interval
2. Once tripped, if your route is still spending 50m queueuing it will continue to be protected

This means that site will continue to function with almost no delays while it is scaling up to handle the new load
2018-04-23 11:55:25 +10:00
Sam 4810ce3607 correct regression 2018-04-18 21:04:08 +10:00
Sam 59cd7894d9 FEATURE: if site is under extreme load show anon view
If a particular path is being hit extremely hard by logged on users,
revert to anonymous cached view.

This will only come into effect if 3 requests queue for longer than 2 seconds
on a *single* path.

This can happen if a URL is shared with the entire forum base and everyone
is logged on
2018-04-18 16:58:57 +10:00
Neil Lalonde b87fa6d749 FIX: blacklisted crawlers could get through by omitting the accept header 2018-04-17 12:39:30 -04:00
Sam 9980f18d86 FEATURE: track request queueing as early as possible 2018-04-17 18:06:17 +10:00
Neil Lalonde 4d12ff2e8a when writing cache, remove elements from the user agents list. also return a message and content type when blocking a crawler. 2018-03-27 13:44:14 -04:00
Neil Lalonde a84bb81ab5 only applies to get html requests 2018-03-22 17:57:44 -04:00
Neil Lalonde ced7e9a691 FEATURE: control which web crawlers can access using a whitelist or blacklist 2018-03-22 15:41:02 -04:00
Sam 0134e41286 FEATURE: detect when client thinks user is logged on but is not
This cleans up an error condition where UI thinks a user is logged on
but the user is not. If this happens user will be prompted to refresh.
2018-03-06 16:49:31 +11:00
Sam f0d5f83424 FEATURE: limit assets less that non asset paths
By default assets can be requested up to 200 times per 10 seconds
from the app, this includes CSS and avatars
2018-03-06 15:20:39 +11:00
Sam f295a18e94 FIX: stop double counting net calls in logs 2018-02-28 10:45:11 +11:00
Sam ca1a3f37e3 FEATURE: add instrumentation for all external net calls 2018-02-21 15:20:29 +11:00
Guo Xiang Tan 3e835047da Remove "already initialized" constant warning. 2018-02-13 08:55:15 +08:00
Sam Saffron df8e43abdd use lazy & instead of try
unregister ip skipper in test
raise if called when a skipper is in play
2018-02-06 10:38:15 +11:00
Robin Ward eefd226611 Add extensibility point to `request_tracker` to skip IP addresses
This is useful if you want to run a per IP rate limiter but want to be
able to skip some IPs with custom logic.
2018-02-05 17:49:40 -05:00
Sam 2437b0d531 FIX: regression, missing 404 page 2018-01-23 09:00:28 +11:00
Sam f26ff290c3 FEATURE: Shorten setting name to max_reqs
So it is consistent with other settings
2018-01-22 13:18:30 +11:00
Sam 8bf91b8dca correct tracking of x runtime 2018-01-19 17:51:19 +11:00
Sam 8ad43f01c2 FIX: correctly log topic timings as background 2018-01-19 10:37:43 +11:00
Sam 12872d03be PERF: run post timings in background
This means that if a very large amount of registered users hit
a single topic we will handle it gracefully, even if db gets slow.
2018-01-19 08:27:29 +11:00
Sam 442a17bfb2 PERF: bypass omniauth unless in an auth path 2018-01-15 12:44:54 +11:00
Sam 49ed382c2a FIX: return 429 when admin api key is limited on admin route
This also handles a general case where exceptions leak out prior to being handled by the application controller
2018-01-12 14:15:26 +11:00
Sam cecd7d0d07 FEATURE: global rate limiter can bypass local IPs 2018-01-08 08:39:17 +11:00
Sam 715cb98e95 add better diagnostics for rate limits 2018-01-05 12:14:28 +11:00
Sam bbc606988f improve message 2017-12-20 10:12:33 +11:00
Sam 4986ebcf24 FEATURE: optional default off global per ip rate limiter 2017-12-11 17:52:57 +11:00
Sam df84e1c358 Correctly track hijacked requests 2017-11-28 16:47:20 +11:00
Sam a4c539bade FEATURE: Allow registration of detailed request logger
Detailed request loggers can be used to gather rich timing info
from all requests (which in turn can be forwarded to monitoring solution)

Middleware::RequestTracker.detailed_request_logger(->|env, data| do
   # do stuff with env and data
end
2017-10-18 12:10:30 +11:00
Guo Xiang Tan 5012d46cbd Add rubocop to our build. (#5004) 2017-07-28 10:20:09 +09:00
Sam bdb848b4f3 Split the theme_key so we extract the key from seq 2017-06-15 14:09:44 -04:00
Sam ac1f84d3e1 SECURITY: theme key should be an anon cache breaker 2017-06-15 09:36:27 -04:00
Sam a3e8c3cd7b FEATURE: Native theme support
This feature introduces the concept of themes. Themes are an evolution
of site customizations.

Themes introduce two very big conceptual changes:

- A theme may include other "child themes", children can include grand
children and so on.

- A theme may specify a color scheme

The change does away with the idea of "enabled" color schemes.

It also adds a bunch of big niceties like

- You can source a theme from a git repo

- History for themes is much improved

- You can only have a single enabled theme. Themes can be selected by
    users, if you opt for it.

On a technical level this change comes with a whole bunch of goodies

- All CSS is now compiled using a custom pipeline that uses libsass
    see /lib/stylesheet

- There is a single pipeline for css compilation (in the past we used
    one for customizations and another one for the rest of the app

- The stylesheet pipeline is now divorced of sprockets, there is no
   reliance on sprockets for CSS bundling

- CSS is generated with source maps everywhere (including themes) this
    makes debugging much easier

- Our "live reloader" is smarter and avoid a flash of unstyled content
   we run a file watcher in "puma" in dev so you no longer need to run
   rake autospec to watch for CSS changes
2017-04-12 10:53:49 -04:00
Sam ea9f7a41af remove gctools (no longer used) add gctracer for debugging 2016-12-20 15:07:30 +11:00
Sam 39a524aac8 FEATURE: brotli cdn bypass for assets
Allow CDNS that strip out brotli encoding to use brotli regardless
2016-12-05 13:57:09 +11:00
Sam 497ff76a67 make sure 1 is a string 2016-10-27 18:08:01 +11:00
Sam 3e7190866a make code a bit safer 2016-10-27 16:50:56 +11:00
Sam 8a477f1857 FEATURE: added X-Discourse-TrackView header
This header is set to 1 if the particular request is a tracked page view
2016-10-27 16:48:27 +11:00
Robin Ward a9823ab59a FIX: Use a cookie to bypass the anon cache 2015-10-28 17:16:56 -04:00
Sam ec4a1bb2c4 FIX: page tracking was not properly tracking transitions
PERF: move closure to self contained method so env is released earlier.
2015-09-17 11:06:33 +10:00
Régis Hanol d7aa4e81d6 revert 8f435fcbf6 2015-07-31 15:22:30 +02:00
Neil Lalonde 86cd1a19cc FEATURE: page view stats for mobile view 2015-07-03 17:19:33 -04:00
Sam 1f9761e85d FEATURE: add a header to denote an anonymous req was cached
(X-Discourse-Cached)
2015-06-16 10:30:06 +10:00
Sam 90eaad336d FEATURE: allow users to pick a CDN for s3 assets 2015-05-26 11:13:12 +10:00
Régis Hanol bb0c2813ac FEATURE: generate (avatar) thumbnails in a background task
FIX: keep the "uploading..." indicator until the server replies via the MessageBus
FIX: text was disapearing when uploading an avatar

PERF: always use a region for S3 (defaults to 'us-east-1')
FEATURE: ApplyCDN middleware when using S3
FIX: use the same pattern to store files on S3 and locally
PERF: keep a local cache of uploads when generating thumbnails
FEATURE: migrate_to_s3 rake task
2015-05-25 17:59:00 +02:00
Sam f5af4768eb FEATURE: add clean support for running Discourse in a subfolder
To setup set DISCOURSE_RELATIVE_URL_ROOT to the folder you wish
2015-03-09 13:14:29 +11:00
Sam cbe18eb0df FEATURE: allow view exclusion using custom header
Set Discourse-Track-View to either "0" or "false" to exclude request
2015-02-26 11:41:11 +11:00
Sam fce9e296e7 background reqs failed or not are always counted seperately 2015-02-12 09:47:46 +11:00
Sam 3cf87b94c9 whitespace 2015-02-11 09:39:04 +11:00
Sam 0ce6524153 correct brokeness 2015-02-10 17:05:24 +11:00
Sam acda6ebd60 FIX: view tracking needs to release data earlier
retaining data during queuing was causing huge memory spikes
2015-02-10 17:03:33 +11:00
Sam 820ce8765e refactor traffic report
split traffic report in 2, page view vs raw traffic
hide raw traffic report by default
improve flushing logic for application reqs
2015-02-06 14:39:16 +11:00
Sam 08b790b3c2 improve metrics gathered using in our traffic section
this also pulls out the middleware into its own home and inserts in front
2015-02-05 16:08:52 +11:00
Sam 8690c7c49f defer counting to avoid race condition 2015-02-05 12:19:21 +11:00
Sam c150c55e2d FEATURE: rudimentary view tracking wired in 2015-02-04 16:15:16 +11:00
Sam 4f8dfd84b9 FIX: vary accept for cache, seems most correct 2014-09-09 10:25:49 +10:00
Sam 8646c21e89 FIX: anonymous cache could cache json for html requests 2014-09-09 09:46:26 +10:00
Akshay 6301a43d57 Not initializing variable for looping if unused in loop 2014-08-15 03:24:55 +05:30
Sam 6019e3f257 FIX: remove hardcoding from middleware stack so we can control it 2014-07-10 17:01:21 +10:00
Sam 5032c96486 FIX: disable x accl redirect for CDN assets
We need to keep headers in tact
2014-07-10 16:32:46 +10:00
Sam 35952055e2 BUGFIX: web crawlers messing with anon caching 2014-04-29 10:48:09 +10:00
Sam 024b86e112 BUGFIX: incorrect OOBGC algorithm, not dynamically adjusting 2014-04-02 12:26:59 +11:00
Sam f617086e91 FIX oobgc support for Ruby 2.1 2014-02-18 14:43:24 +11:00
Sam a6f4d00670 Add oobgc compatible with Ruby 2.1 2014-02-18 14:10:03 +11:00
Robin Ward 18e719b0af In development mode, if an avatar is missing just serve up a placeholder 2014-01-17 11:47:23 -05:00
Sam adc9a58f4a BUGFIX: anon cache was mucking with params 2014-01-09 16:49:12 +11:00
Sam 74c1555885 BUGFIX: fix broken spec 2014-01-09 15:11:04 +11:00
Sam 177983afe6 BUGFIX: mobile ui was being cached for anon views 2014-01-09 14:08:42 +11:00
Sam b804f0bec9 add license for oobgc, should not be gpl 2013-11-22 17:04:32 +11:00
Sam 67222ce8a0 tune down ... 80k is over aggressive 2013-11-19 15:06:34 +11:00
Sam 518ae2c608 added a minimal number of free slots that should always exist 2013-11-19 14:58:27 +11:00
Sam 2841434e36 slow down decay to accommodate for long polling and other tiny reqs 2013-11-19 14:47:13 +11:00
Sam ef64c1b757 more logging for miss 2013-11-19 14:27:18 +11:00
Sam f61920565a reimplemnt oobgc algorithm 2013-11-19 14:13:42 +11:00
Sam 2984d649f5 remove log 2013-11-15 18:22:02 +11:00
Sam 8a868bad58 time saving log 2013-11-15 18:21:19 +11:00
Sam ce6b0ff22e remove puts 2013-11-15 17:54:29 +11:00
Sam c1d7bbdc79 more diags 2013-11-15 17:26:58 +11:00
Sam a92ada0493 tune down min requests, tune up largest multiplier, tune up history 2013-11-15 17:17:08 +11:00
Sam 466459db51 more diags for a bit 2013-11-15 17:08:38 +11:00
Sam 8513164b85 lets not explode if init is called incorrectly 2013-11-15 17:03:03 +11:00
Sam 3219afd666 fix bug, num request should be zeroed, better logging 2013-11-15 16:56:21 +11:00
Sam 461972844e unicorn out of band GC 2013-11-15 12:15:49 +11:00
Sam 3d647a4b41 remove rack cache, it has been causing trouble
instead implement an aggressive anonymous cache that is stored in redis
this cache is sitting in the front of the middleware stack enabled only in production
TODO: expire it more intelligently when stuff is created
2013-10-16 16:39:18 +11:00
Robin Ward 21e08a423e Added a little documentation to TurboDev middleware 2013-09-10 15:21:07 -04:00
Robin Ward 858e51c8ab Use `no-cache` in development mode for assets. Using `must-revalidate`, which is
Rails' default, seems to have Chrome sometimes not request assets in development
mode even though it's supposed to revalidate every time.
2013-09-09 16:55:41 -04:00
Sam 28d111d370 typo 2013-04-18 17:23:52 +10:00
Sam 8367951000 sprockets upgrade
stop bundling all js files in dev, replace with turbo boosted serving of assets
2013-04-18 16:33:25 +10:00