DEV: Use message-bus chunked encoding in development (#19878)

This was previously disabled because of incompatibility with the ember-cli proxy. This commit fixes that incompatibility, and restores the development behaviour to match production.

There were three issues at play:

1. Our bootstrap-js addon handles the forwarding of most requests in the ember-cli proxy. This is not built to handle streaming responses. Solution: skip our custom request processing for `/message-bus/*` and use ember-cli's default `http-proxy`.

2. The request/response size-limiting middleware (`rawMiddleware`) would apply even to unhandled paths, causing request and response bodies to be buffered. Solution: skip it for any paths which are not handled by our custom addon.

3. Expressjs servers will buffer/compress responses. Solution: add `Cache-Control: no-transform` to message-bus responses. For now I've done this in development only, but it may be useful to add it to message-bus's default headers in future

app/assets/javascripts/bootstrap-json/index.js

@@ -463,6 +463,13 @@ to serve API requests. For example:
     baseURL = rootURL === "" ? "/" : cleanBaseURL(rootURL || baseURL);
 
     const rawMiddleware = express.raw({ type: () => true, limit: "100mb" });
+    const pathRestrictedRawMiddleware = (req, res, next) => {
+      if (this.shouldHandleRequest(req, baseURL)) {
+        return rawMiddleware(req, res, next);
+      } else {
+        return next();
+      }
+    };
 
     app.use(
       "/favicon.ico",
@@ -474,9 +481,9 @@ to serve API requests. For example:
       )
     );
 
-    app.use(rawMiddleware, async (req, res, next) => {
+    app.use(pathRestrictedRawMiddleware, async (req, res, next) => {
       try {
-        if (this.shouldForwardRequest(req, baseURL)) {
+        if (this.shouldHandleRequest(req, baseURL)) {
           await handleRequest(proxy, baseURL, req, res);
         } else {
           // Fixes issues when using e.g. "localhost" instead of loopback IP address
@@ -497,7 +504,7 @@ to serve API requests. For example:
     });
   },
 
-  shouldForwardRequest(request, baseURL) {
+  shouldHandleRequest(request, baseURL) {
     if (
       [
         `${baseURL}tests/index.html`,
@@ -513,6 +520,10 @@ to serve API requests. For example:
       return false;
     }
 
+    if (request.path.startsWith(`${baseURL}message-bus/`)) {
+      return false;
+    }
+
     return true;
   },
 };

app/assets/javascripts/discourse/app/initializers/message-bus.js

@@ -86,8 +86,7 @@ export default {
     messageBus.baseUrl =
       siteSettings.long_polling_base_url.replace(/\/$/, "") + "/";
 
-    messageBus.enableChunkedEncoding =
-      isProduction() && siteSettings.enable_chunked_encoding;
+    messageBus.enableChunkedEncoding = siteSettings.enable_chunked_encoding;
 
     if (messageBus.baseUrl !== "/") {
       messageBus.ajax = function (opts) {

config/initializers/004-message_bus.rb

@@ -62,6 +62,11 @@ def setup_message_bus_env(env)
 
     extra_headers["Discourse-Logged-Out"] = "1" if env[Auth::DefaultCurrentUserProvider::BAD_TOKEN]
 
+    if Rails.env.development?
+      # Adding no-transform prevents the expressjs ember-cli proxy buffering/compressing the response
+      extra_headers["Cache-Control"] = "no-transform, must-revalidate, private, max-age=0"
+    end
+
     hash = {
       extra_headers: extra_headers,
       user_id: user_id,
@@ -100,6 +105,8 @@ MessageBus.on_middleware_error do |env, e|
     [403, {}, ["Invalid Access"]]
   elsif RateLimiter::LimitExceeded === e
     [429, { "Retry-After" => e.available_in.to_s }, [e.description]]
+  elsif Errno::EPIPE === e
+    [422, {}, ["Closed by Client"]]
   end
 end