Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic. Some of its key features include:
- Web-based user interface
- Seamless experience for design, control, and monitoring
MiNiFi is a child project effort of Apache NiFi. It is a complementary data collection approach that supplements the core tenets of [NiFi](https://nifi.apache.org/) in dataflow management, focusing on the collection of data at the source of its creation.
Specific goals for MiNiFi are comprised of:
- small and lightweight footprint
- central management of agents
- generation of data provenance
- integration with NiFi for follow-on dataflow management and full chain of custody of information
Perspectives of the role of MiNiFi should be from the perspective of the agent acting immediately at, or directly adjacent to, source sensors, systems, or servers.
To run:
- Change directory to 'minifi-assembly'. In the target directory, there should be a build of minifi.
$ cd minifi-assembly
$ ls -lhd target/minifi*
drwxr-xr-x 3 user staff 102B Jul 6 13:07 minifi-1.14.0-SNAPSHOT-bin
-rw-r--r-- 1 user staff 39M Jul 6 13:07 minifi-1.14.0-SNAPSHOT-bin.tar.gz
-rw-r--r-- 1 user staff 39M Jul 6 13:07 minifi-1.14.0-SNAPSHOT-bin.zip
- For testing ongoing development you could use the already unpacked build present in the directory
named "minifi-*version*-bin", where *version* is the current project version. To deploy in another
location make use of either the tarball or zipfile and unpack them wherever you like. The
distribution will be within a common parent directory named for the version.
$ mkdir ~/example-minifi-deploy
$ tar xzf target/minifi-*-bin.tar.gz -C ~/example-minifi-deploy
$ ls -lh ~/example-minifi-deploy/
total 0
drwxr-xr-x 10 user staff 340B Jul 6 01:06 minifi-1.14.0-SNAPSHOT
To run MiNiFi:
- Change directory to the location where you installed MiNiFi and run it.
- For help building your first data flow and sending data to a NiFi instance see the System Admin Guide located in the docs folder or making use of the minifi-toolkit, which aids in adapting NiFi templates to MiNiFi YAML configuration file format.
- If you are testing ongoing development, you will likely want to stop your instance.
$ cd ~/example-minifi-deploy/minifi-*
$ ./bin/minifi.sh stop
### Docker Build
To build:
- Run a full NiFi build (see above for instructions). Then from the minifi/ subdirectory, execute `mvn -P docker clean install`. This will run the full build, create a docker image based on it, and run docker-compose integration tests. After it completes successfully, you should have an apacheminifi:${minifi.version} image that can be started with the following command (replacing ${minifi.version} with the current maven version of your branch):
```
docker run -d -v YOUR_CONFIG.YML:/opt/minifi/minifi-${minifi.version}/conf/config.yml apacheminifi:${minifi.version}
Registry—a subproject of Apache NiFi—is a complementary application that provides a central location for storage and management of shared resources across one or more instances of NiFi and/or MiNiFi.
### Getting Registry Started
1) Build nifi
See [Gettin Started](#getting-started) for NiFi
or
Build only the Registry subproject:
cd nifi/nifi-registry
mvn clean install
If you wish to enable style and license checks, specify the contrib-check profile:
mvn clean install -Pcontrib-check
3) Start Registry
cd nifi-registry/nifi-registry-assembly/target/nifi-registry-<VERSION>-bin/nifi-registry-<VERSION>/
./bin/nifi-registry.sh start
Note that the application web server can take a while to load before it is accessible.
4) Accessing the application web UI
With the default settings, the application UI will be available at [http://localhost:18080/nifi-registry](http://localhost:18080/nifi-registry)
5) Accessing the application REST API
If you wish to test against the application REST API, you can access the REST API directly. With the default settings, the base URL of the REST API will be at `http://localhost:18080/nifi-registry-api`. A UI for testing the REST API will be available at [http://localhost:18080/nifi-registry-api/swagger/ui.html](http://localhost:18080/nifi-registry-api/swagger/ui.html)
6) Accessing the application logs
Logs will be available in `logs/nifi-registry-app.log`
### Database Testing
In order to ensure that NiFi Registry works correctly against different relational databases,
the existing integration tests can be run against different databases by leveraging the [Testcontainers framework](https://www.testcontainers.org/).
Spring profiles are used to control the DataSource factory that will be made available to the Spring application context.
DataSource factories are provided that use the Testcontainers framework to start a Docker container for a given database and create a corresponding DataSource.
If no profile is specified then an H2 DataSource will be used by default and no Docker containers are required.
Assuming Docker is running on the system where the build is running, then the following commands can be run:
| MySQL 5.6 | `mvn verify -Pcontrib-check -Dspring.profiles.active=mysql-56` |
| MySQL 5.7 | `mvn verify -Pcontrib-check -Dspring.profiles.active=mysql-57` |
| MySQL 8 | `mvn verify -Pcontrib-check -Dspring.profiles.active=mysql-8` |
When one of the Testcontainer profiles is activated, the test output should show logs that indicate a container has been started, such as the following:
2019-05-15 16:14:45.078 INFO 66091 --- [ main] 🐳 [mysql:5.7] : Creating container for image: mysql:5.7
2019-05-15 16:14:45.145 INFO 66091 --- [ main] o.t.utility.RegistryAuthLocator : Credentials not found for host (index.docker.io) when using credential helper/store (docker-credential-osxkeychain)
2019-05-15 16:14:45.646 INFO 66091 --- [ main] 🐳 [mysql:5.7] : Starting container with ID: ca85c8c5a1990d2a898fad04c5897ddcdb3a9405e695cc11259f50f2ebe67c5f
2019-05-15 16:14:46.437 INFO 66091 --- [ main] 🐳 [mysql:5.7] : Container mysql:5.7 is starting: ca85c8c5a1990d2a898fad04c5897ddcdb3a9405e695cc11259f50f2ebe67c5f
2019-05-15 16:14:46.479 INFO 66091 --- [ main] 🐳 [mysql:5.7] : Waiting for database connection to become available at jdbc:mysql://localhost:33051/test?useSSL=false&allowPublicKeyRetrieval=true using query 'SELECT 1'
The Flyway connection should also indicate the given database:
2019-05-15 16:15:02.114 INFO 66091 --- [ main] o.a.n.r.db.CustomFlywayConfiguration : Determined database type is MYSQL
2019-05-15 16:15:02.115 INFO 66091 --- [ main] o.a.n.r.db.CustomFlywayConfiguration : Setting migration locations to [classpath:db/migration/common, classpath:db/migration/mysql]
2019-05-15 16:15:02.373 INFO 66091 --- [ main] o.a.n.r.d.CustomFlywayMigrationStrategy : First time initializing database...
2019-05-15 16:15:02.380 INFO 66091 --- [ main] o.f.c.internal.license.VersionPrinter : Flyway Community Edition 5.2.1 by Boxfuse
([archive](https://lists.apache.org/list.html?dev@nifi.apache.org)). For more interactive discussions, community members can often be found in the following locations:
To submit a feature request or bug report, please file a Jira at [https://issues.apache.org/jira/projects/NIFI/issues](https://issues.apache.org/jira/projects/NIFI/issues). If this is a **security vulnerability report**, please email [security@nifi.apache.org](mailto:security@nifi.apache.org) directly and review the [Apache NiFi Security Vulnerability Disclosure](https://nifi.apache.org/security.html) and [Apache Software Foundation Security](https://www.apache.org/security/committers.html) processes first.
