Add some debug tips for debugging peons (#12697)

* add some debug tips

* address comments

* fix typo

dev/intellij-setup.md

@@ -69,6 +69,21 @@ Before running or debugging the apps, you should do a `mvn clean install -Pdist
 
 You may also add `-Ddruid.console.skip=true` to the command if you're focusing on backend servers instead of frontend project. This option saves great building time.
 
+## Debug a running Druid cluster using Intellij
+Intellij IDEA debugger can attach to a local or remote Java process (Druid process). 
+Follow these steps to debug a Druid process using the IntelliJ IDEA debugger.
+1. Enable debugging in the Druid process
+    1. For Druid services (such as Overlord, Coordinator, etc), include the following as JVM config `-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=<PORT>` where `<PORT>` is any available port. Note that different port values should be chosen for each Druid service.
+    2. For the peons (workers on Middlemanager), include the following `agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0` in runtime properties `druid.indexer.runner.javaOpts` of the Middle Manager. Note that `address=0` will tell the debugger to assign ephemeral port.
+2. Find the port assigned to the Druid process.
+    1. For Druid services, the port value is what you chose in the JVM argument of step 1i above. The port value can also be found in the first line of each Druid service log.
+    2. For the peons (workers on Middlemanager), you can find the assigned ephemeral port by checking the first line of the task log.
+3. Attach Intellij IDEA debugger to the running process
+    1. Create a Remote configuration in the Run/Debug Configurations dialog of Intellij (see: https://www.jetbrains.com/help/idea/tutorial-remote-debug.html#debugger_rc)
+    2. Set the port value using the value from step 2 for the Druid service and/or peon (worker on Middlemanager) that you want to debug
+    3. Start (debug) the above remote configuration
+    4. Repeat step 3i to 3iii for each Druid service and/or peon (worker on Middlemanager) that you want to debug
+
 ## XML App Def
 You can configure application definitions in XML for import into IntelliJ. Below are a few examples. These should be placed in an XML file in `.idea/runConfigurations` in the Druid source code.
 

integration-tests/README.md

@@ -248,6 +248,7 @@ For your convenience, Druid processes running inside Docker have been debugging
 | Historical | 5007 |
 | Middlemanager | 5008 |
 | Overlord | 5009 |
+| Peons (Workers on Middlemanager) |  Ephemeral port assigned by debugger (check task log for port assigned to each task) |
 
 You can use remote debugger(such as via IntelliJ IDEA's Remote Configuration) to debug the corresponding Druid process at above port.
 

integration-tests/docker/environment-configs/middlemanager

@@ -27,7 +27,7 @@ SERVICE_DRUID_JAVA_OPTS=-server -Xmx64m -Xms64m -XX:+UseG1GC -agentlib:jdwp=tran
 druid_host=druid-middlemanager
 druid_server_http_numThreads=100
 druid_storage_storageDirectory=/shared/storage
-druid_indexer_runner_javaOptsArray=["-server", "-Xmx256m", "-Xms256m", "-XX:NewSize=128m", "-XX:MaxNewSize=128m", "-XX:+UseG1GC", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Dlog4j.configurationFile=/shared/docker/lib/log4j2.xml"]
+druid_indexer_runner_javaOptsArray=["-server", "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=0", "-Xmx256m", "-Xms256m", "-XX:NewSize=128m", "-XX:MaxNewSize=128m", "-XX:+UseG1GC", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-Dlog4j.configurationFile=/shared/docker/lib/log4j2.xml"]
 
 druid_indexer_fork_property_druid_processing_buffer_sizeBytes=25000000
 druid_indexer_fork_property_druid_processing_numThreads=1