Apache NiFi
Go to file
Joe Skora 376af83a3d NIFI-3055 StandardRecordWriter Can Throw UTFDataFormatException
* Updated StandardRecordWriter, even though it is now deprecated to consider the encoding behavior of java.io.DataOutputStream.writeUTF() and truncate string values such that the UTF representation will not be longer than that DataOutputStream's 64K UTF format limit.
* Updated the new SchemaRecordWriter class to similarly truncate long Strings that will be written as UTF.
* Add tests to confirm handling of large UTF strings and various edge conditions of UTF string handling.

Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #1469.
2017-02-03 20:52:32 +00:00
.github NIFI-1615 Removing Apache License header and adding exclusion for Pull Request template for RAT check as we cannot provide comments without cluttering PR dialog. 2016-10-05 11:14:24 -04:00
nifi-api NIFI-957 Added the possibility to use DefaultSchedule annotation in reporting tasks 2017-01-30 08:35:39 -05:00
nifi-assembly NIFI-2615 polishing 2017-01-25 13:22:50 -05:00
nifi-bootstrap NIFI-2761 Fixed error message in bootstrap 2017-01-17 01:43:15 +11:00
nifi-commons NIFI-3055 StandardRecordWriter Can Throw UTFDataFormatException 2017-02-03 20:52:32 +00:00
nifi-docker NIFI-3314 Adjusting Dockerfile for Docker Hub to use defaults for ARGs 2017-01-29 11:16:22 -05:00
nifi-docs NIFI-3300 Implemented usage of ZooKeeper chroot capability in the connect string 2017-02-01 17:02:17 -05:00
nifi-external NIFI-3100-rc2 prepare for next development iteration 2016-11-25 23:49:27 -05:00
nifi-framework-api NIFI-2695: - Providing more granular and meaningful authorization error messages. 2017-01-04 13:06:19 -05:00
nifi-maven-archetypes NIFI-3100-rc2 prepare for next development iteration 2016-11-25 23:49:27 -05:00
nifi-mock NIFI-3161 added ASM exclusion 2017-01-30 22:06:40 +01:00
nifi-nar-bundles NIFI-3055 StandardRecordWriter Can Throw UTFDataFormatException 2017-02-03 20:52:32 +00:00
nifi-toolkit NIFI-3300 Implemented usage of ZooKeeper chroot capability in the connect string 2017-02-01 17:02:17 -05:00
.gitignore NIFI-857 fixing gitignore list 2015-08-16 16:25:56 -04:00
.travis.yml NIFI-2728 - Improve travis-ci build settings. This closes #985. 2016-09-20 13:46:23 -04:00
KEYS Adding pgp key for pvillard@apache.org to KEYS file 2017-02-01 11:36:18 +01:00
LICENSE NIFI-3291: 2017-01-17 14:54:59 -05:00
NOTICE NIFI-3104 This closes #1273. Removed tests derived from HAPI test data 2016-11-25 19:39:36 -05:00
README.md NIFI-1846 updated reamdme.md to mention need for git client 2016-08-14 15:47:19 -04:00
appveyor.yml NIFI-2814 - Add ASF header to AppVeyor settings file and adjust apache-rat accordingly 2016-09-26 11:30:44 -04:00
pom.xml NIFI-2615 polishing 2017-01-25 13:22:50 -05:00

README.md

Apache NiFi Build Status

Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data.

Table of Contents

Features

Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic. Some of its key features include:

  • Web-based user interface
    • Seamless experience for design, control, and monitoring
    • Multi-tenant user experience
  • Highly configurable
    • Loss tolerant vs guaranteed delivery
    • Low latency vs high throughput
    • Dynamic prioritization
    • Flows can be modified at runtime
    • Back pressure
    • Scales up to leverage full machine capability
    • Scales out with zero-master clustering model
  • Data Provenance
    • Track dataflow from beginning to end
  • Designed for extension
    • Build your own processors and more
    • Enables rapid development and effective testing
  • Secure
    • SSL, SSH, HTTPS, encrypted content, etc...
    • Pluggable fine-grained role-based authentication/authorization
    • Multiple teams can manage and share specific portions of the flow

Requirements

  • JDK 1.8 or newer
  • Apache Maven 3.1.0 or newer
  • Git Client (used during build process by 'bower' plugin)

Getting Started

  • Read through the quickstart guide for development. It will include information on getting a local copy of the source, give pointers on issue tracking, and provide some warnings about common problems with development environments.
  • For a more comprehensive guide to development and information about contributing to the project read through the NiFi Developer's Guide.

To build:

  • Execute mvn clean install or for parallel build execute mvn -T 2.0C clean install. On a modest development laptop that is a couple of years old, the latter build takes a bit under ten minutes. After a large amount of output you should eventually see a success message.

      laptop:nifi myuser$ mvn -T 2.0C clean install
      [INFO] Scanning for projects...
      [INFO] Inspecting build with total of 115 modules...
          ...tens of thousands of lines elided...
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD SUCCESS
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 09:24 min (Wall Clock)
      [INFO] Finished at: 2015-04-30T00:30:36-05:00
      [INFO] Final Memory: 173M/1359M
      [INFO] ------------------------------------------------------------------------
    

To deploy:

  • Change directory to 'nifi-assembly'. In the target directory, there should be a build of nifi.

      laptop:nifi myuser$ cd nifi-assembly
      laptop:nifi-assembly myuser$ ls -lhd target/nifi*
      drwxr-xr-x  3 myuser  mygroup   102B Apr 30 00:29 target/nifi-1.0.0-SNAPSHOT-bin
      -rw-r--r--  1 myuser  mygroup   144M Apr 30 00:30 target/nifi-1.0.0-SNAPSHOT-bin.tar.gz
      -rw-r--r--  1 myuser  mygroup   144M Apr 30 00:30 target/nifi-1.0.0-SNAPSHOT-bin.zip
    
  • For testing ongoing development you could use the already unpacked build present in the directory named "nifi-version-bin", where version is the current project version. To deploy in another location make use of either the tarball or zipfile and unpack them wherever you like. The distribution will be within a common parent directory named for the version.

      laptop:nifi-assembly myuser$ mkdir ~/example-nifi-deploy
      laptop:nifi-assembly myuser$ tar xzf target/nifi-*-bin.tar.gz -C ~/example-nifi-deploy
      laptop:nifi-assembly myuser$ ls -lh ~/example-nifi-deploy/
      total 0
      drwxr-xr-x  10 myuser  mygroup   340B Apr 30 01:06 nifi-1.0.0-SNAPSHOT
    

To run NiFi:

  • Change directory to the location where you installed NiFi and run it.

      laptop:~ myuser$ cd ~/example-nifi-deploy/nifi-*
      laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh start
    
  • Direct your browser to http://localhost:8080/nifi/ and you should see a screen like this screenshot: image of a NiFi dataflow canvas

  • For help building your first data flow see the NiFi User Guide

  • If you are testing ongoing development, you will likely want to stop your instance.

      laptop:~ myuser$ cd ~/example-nifi-deploy/nifi-*
      laptop:nifi-1.0.0-SNAPSHOT myuser$ ./bin/nifi.sh stop
    

Getting Help

If you have questions, you can reach out to our mailing list: dev@nifi.apache.org (archive). We're also often available in IRC: #nifi on irc.freenode.net.

Documentation

See http://nifi.apache.org/ for the latest documentation.

License

Except as otherwise noted this software is licensed under the Apache License, Version 2.0

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Export Control

This distribution includes cryptographic software. The country in which you currently reside may have restrictions on the import, possession, use, and/or re-export to another country, of encryption software. BEFORE using any encryption software, please check your country's laws, regulations and policies concerning the import, possession, or use, and re-export of encryption software, to see if this is permitted. See http://www.wassenaar.org/ for more information.

The U.S. Government Department of Commerce, Bureau of Industry and Security (BIS), has classified this software as Export Commodity Control Number (ECCN) 5D002.C.1, which includes information security software using or performing cryptographic functions with asymmetric algorithms. The form and manner of this Apache Software Foundation distribution makes it eligible for export under the License Exception ENC Technology Software Unrestricted (TSU) exception (see the BIS Export Administration Regulations, Section 740.13) for both object code and source code.

The following provides more details on the included cryptographic software:

Apache NiFi uses BouncyCastle, Jasypt, JCraft Inc., and the built-in java cryptography libraries for SSL, SSH, and the protection of sensitive configuration parameters. See http://bouncycastle.org/about.html http://www.jasypt.org/faq.html http://jcraft.com/c-info.html http://www.oracle.com/us/products/export/export-regulations-345813.html for more details on each of these libraries cryptography features.