# Docker images for building Hadoop This folder contains the Dockerfiles for building Hadoop on various platforms. # Dependency management The mode of installation of the dependencies needed for building Hadoop varies from one platform to the other. Different platforms have different toolchains. Some packages tend to be polymorphic across platforms and most commonly, a package that's readily available in one platform's toolchain isn't available on another. We thus, resort to building and installing the package from source, causing duplication of code since this needs to be done for all the Dockerfiles pertaining to all the platforms. We need a system to track a dependency - for a package - for a platform. Thus, there's a lot of diversity that needs to be handled for managing package dependencies and `pkg-resolver` caters to that. ## Supported platforms `pkg-resolver/platforms.json` contains a list of the supported platforms for dependency management. ## Package dependencies `pkg-resolver/packages.json` maps a dependency to a given platform. Here's the schema of this JSON. ```json { "dependency_1": { "platform_1": "package_1", "platform_2": [ "package_1", "package_2" ] }, "dependency_2": { "platform_1": [ "package_1", "package_2", "package_3" ] } } ``` The root JSON element contains unique _dependency_ children. This in turn contains the name of the _ platforms_ and the list of _packages_ to be installed for that platform. Just to give an example of how to interpret the above JSON - 1. For `dependency_1`, `package_1` needs to be installed for `platform_1`. 2. For `dependency_2`, `package_1` and `package_2` needs to be installed for `platform_2`. 3. For `dependency_2`, `package_1`, `package_3` and `package_3` needs to be installed for `platform_1`. ## Standalone packages Most commonly, some packages are not available across the toolchains in various platforms. Thus, we would need to build and install them. Since we need to do this across all the Dockerfiles for all the platforms, it could lead to code duplication and managing them becomes a hassle. Thus, we put the build steps in a `pkg-resolver/install-.sh` and invoke this in all the Dockerfiles.