[ML] Increase timeouts for named pipe connections to native processes (elastic/x-pack-elasticsearch#987)

It has been observed that Amazon EBS volumes created from snapshots can
have very high latency the first time a given block is accessed.  This
can lead to named pipes taking longer than 2 seconds to create.
Since the native processes create their named pipes immediately after
startup, and this only takes a fraction of a second on a local disk, 2
seconds was considered a generous timeout, but it seems that in the case
of a remote NAS with lazy provisioning it's not long enough.  During
debugging a latency of just over 3 seconds was observed.  The timeouts
have been increased to 10 seconds.

relates elastic/x-pack-elasticsearch#922

Original commit: elastic/x-pack-elasticsearch@c90434c948

plugin/src/main/java/org/elasticsearch/xpack/ml/job/process/NativeController.java

@@ -31,8 +31,9 @@ import java.util.regex.Pattern;
 public class NativeController {
     private static final Logger LOGGER = Loggers.getLogger(NativeController.class);
 
-    // The controller process should already be running by the time this class tries to connect to it, so the timeout can be short
-    private static final Duration CONTROLLER_CONNECT_TIMEOUT = Duration.ofSeconds(2);
+    // The controller process should already be running by the time this class tries to connect to it, so the timeout
+    // can be short (although there's a gotcha with EBS volumes restored from snapshot, so not TOO short)
+    private static final Duration CONTROLLER_CONNECT_TIMEOUT = Duration.ofSeconds(10);
 
     private static final String START_COMMAND = "start";
 

plugin/src/main/java/org/elasticsearch/xpack/ml/job/process/autodetect/NativeAutodetectProcessFactory.java

@@ -40,7 +40,7 @@ public class NativeAutodetectProcessFactory implements AutodetectProcessFactory
 
     private static final Logger LOGGER = Loggers.getLogger(NativeAutodetectProcessFactory.class);
     private static final NamedPipeHelper NAMED_PIPE_HELPER = new NamedPipeHelper();
-    private static final Duration PROCESS_STARTUP_TIMEOUT = Duration.ofSeconds(2);
+    private static final Duration PROCESS_STARTUP_TIMEOUT = Duration.ofSeconds(10);
 
     private final Client client;
     private final Environment env;

plugin/src/main/java/org/elasticsearch/xpack/ml/job/process/normalizer/NativeNormalizerProcessFactory.java

@@ -26,7 +26,7 @@ public class NativeNormalizerProcessFactory implements NormalizerProcessFactory
 
     private static final Logger LOGGER = Loggers.getLogger(NativeNormalizerProcessFactory.class);
     private static final NamedPipeHelper NAMED_PIPE_HELPER = new NamedPipeHelper();
-    private static final Duration PROCESS_STARTUP_TIMEOUT = Duration.ofSeconds(2);
+    private static final Duration PROCESS_STARTUP_TIMEOUT = Duration.ofSeconds(10);
 
     private final Environment env;
     private final Settings settings;