There are multiple and sundry changes in here.
First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.
In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module. I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.
In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api. They are all API-level things, however, so they really belong in druid-api instead.
Lastly, I removed the druid-realtime module and put it all in druid-server.
JettyServerModule had been eagerly instantiating the Server object, which was causing things that didn't care about an HTTP interface to all of a sudden require host and port parameters. The change makes the JettyServerModule only setup the bindings without eagerly instantiating the Jetty Server. Each cli needs to register the Server class with the Lifecycle in order to make it actually get used.
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about. This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need. Guice tries to fail-fast and verifies that all
bindings exist before it does anything, which is a problem because the extension bind
some objects that don't necessarily have all of their dependencies bound in all processes.
The fix for this is to build a single Injector with all bindings in it and let each of the
processes only load the things that they care about. This also requires the use of
Module overrides and other such interesting things, which are node done.
In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided. This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.
In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
fjy
Xavier Léauté
Gian Merlino
cheddar