Commit Graph

326 Commits

Author SHA1 Message Date
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Joe Witt ed324058c7
NIFI-7021 cacheFile not reliable on builds with latest puppycrawl checkstyle 2020-01-19 13:49:56 -05:00
Joe Witt 67c36f3c3e
NIFI-7021 apache parent pom and common dep maintenance for build 2020-01-19 13:32:37 -05:00
Pierre Villard ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00
Bryan Bende 5cfc68d26d
NIFI-6896 Standardize hadoop.version for everything that uses nifi-hadoop-libraries
NIFI-6896 Adding hadoop-aws, hadoop-azure, and hadoop-azure-datalake to nifi-hadoop-libraries-nar and nifi-ranger-nar so other filesystem impls will be available

NIFI-6896 Introduce profiles to optionally include additional hadoop modules to reduce default build size

NIFI-6896 Updating licenses and notices to account for new optional dependencies

NIFI-6896 Add profile for hadoop-cloud-storage

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3900
2019-11-22 11:14:46 -05:00
Mark Payne 5f4ce8f431
NIFI-6707: Addition of nifi-system-tests module. Updates to EmbeddedNiFi, NiFi and SystemBundle to properly support classloader isolation that is needed for creating system-level tests and isolating the classpath of the tests themselves from the NiFi classpath. Fixed bootstrap listener to ensure that it always closes socket before calling shutdown(). Failure to do so can result in RunNiFi sometimes timing out while waiting for a response from NiFi
NIFI-6707: Added System Test to verify behavior when services depend on one another

NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them

NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group

NIFI-6707: Updated to include java11 subdirectory under lib/

NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT

NIFI-6707: Updates to see if it will help travis build correctly

NIFI-6707: Commenting out java commands in bootstrap.conf files

This closes #3831.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-11-08 12:33:16 -05:00
Matt Gilman 8dadebafdc NIFI-6843:
- Bumping ranger version to 2.0.0.
2019-11-05 16:58:37 -05:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
samhjelmfelt ea1becac4f
NIFI-6703: Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes

This closes #3795.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-17 15:46:32 -04:00
Joe Witt 0f02de6002
NIFI-6733 updating key apache commons dependencies and apache base dependency for build
This closes #3791.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2019-10-07 22:20:48 -04:00
Pierre Villard 7cb87328c8 NIFI-6709 - Upgrade jackson-databind direct dependencies to 2.9.10
This closes #3765.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
2019-09-27 16:07:59 -04:00
Mark Payne 59734dad88
NIFI-6028: Updates to ignore changes between local version of a flow and a remote version of a flow if the difference is the addition of a new property that has the default value
NIFI-6028: Code refactoring to address review feedback

This closes #3544.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-28 10:38:56 -04:00
Bryan Bende cc88dd428f
NIFI-6158 Fix conversion of Avro fixed type with logicalType decimal
This closes #3665.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 10:57:28 +09:00
Jeff Storck 33187f8058 NIFI-5176 NiFI builds on Java 11
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build.  Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
  Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
  Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
  URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.

NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service

NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11

This closes #3404.
2019-08-13 18:41:39 -04:00
Jeff Storck 4783b12a9c
NIFI-5254 Upgraded Groovy to version 2.5.4
Updated pom files regarding groovy dependencies.  groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter

This closes #3547

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-07-11 14:17:35 -04:00
Troy Melhase e973cacb2f
NIFI-5973 Adds ShellUserGroupProvider.
NIFI-5973 More comments and better defaults for the shell provider.
NIFI-5973 Fixed bug where user was being retrieved by identifier when identity was provided.
NIFI-5973 Fixed a formatting string in the OS X shell commands.
Updated testing conditions to run IT in OS X environment.
Changed unit test to provide identity rather than identifier.

This closes #3537.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-20 16:22:42 -07:00
thenatog e7aac049ef
NIFI-6372 - This closes #3532. Upgrading jackson-databind version to 2.9.9.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-06-19 12:41:18 -04:00
thenatog 64b0d88d83
NIFI-6330 - This closes #3534.Updated Jetty version to 9.4.18.v20190429
NIFI-6330 - Removed jetty.version property declaration in nifi-prometheus-reporting-task pom.xml. This version should be inherited from the root pom.

NIFI-6330 - New release of Jetty came out 2 days ago. Upgraded and verified NiFi functionality again with smoke tests including ListenHTTP, secure clustering, site-to-site

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-06-19 12:39:15 -04:00
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
Bryan Bende c7768c909a
NIFI-6164 - Added CLI commands related to NiFi Registry extension bun… (#3492)
* NIFI-6164 - Added CLI commands related to NiFi Registry extension bundles
- Adding CLI commands fro uploading extension bundles
- Adding commands for listing extension bundles and their versions
- Adding checksums to UploadNarBundles command
- Adding sha256 to UploadExtensionBundle command, and refactoring result classes to be better organized
- Improvements for CLI commands to interact with extension registry.
- Added download-extension-bundle commands.
- Refactoring ProcessGroupBox

* Refactoring upload-nars command to be upload-bundles

* Rename ListTagCounts to ListExtensionTags and ListExtensionsWithTags to ListExtensions

* Update to catch more specific exception in UploadBundles

Co-Authored-By: Kevin Doran <kdoran@apache.org>
2019-06-17 08:22:19 -04:00
thenatog 0e1a37fcb9
NIFI-6362 - Upgraded com.puppycrawl.tools:checkstyle to 8.18. Made small checkstyle changes to nifi-kafka.
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.

NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.

NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.

NIFI-6362 - Grep pattern no longer required.

This closes #3522.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-17 17:19:09 +09:00
Andy LoPresto ed19f61682
NIFI-6331 Removed nifi-testharness module.
This closes #3506.
2019-05-31 13:01:46 -07:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Bryan Bende b09b84226c
NIFI-6311 Upgrade to nifi-registry-client 0.4.0
This closes #3485.
2019-05-23 12:48:33 -04:00
samhjelmfelt 146689b36f NIFI-5922: Renaming from NiFi-Fn to NiFi-Stateless
Fixed docker image and moved it into the nifi-docker project

Fixed Docker container, YARN runtime, and OpenWhisk runtime
2019-05-21 11:19:17 -04:00
Mark Payne cbafd8f834 NIFI-5922: Addressed checkstyle violations; added to README Updates to nifi-fn proposal:
- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows
- Rebased against master and updated to version 1.10.0-SNAPSHOT
- Removed dependency on nifi-framework-core
- Added LICENSE/NOTICE files
2019-05-21 11:19:17 -04:00
Bryan Bende 7825e40d15
NIFI-6262 - Upgrade to nifi-nar-maven-plugin 1.3.1
This closes #3463.
2019-05-14 11:00:21 -04:00
Jan Hentschel 535d65370f NIFI-1451 disable-doclint profile now gets activated for JDK 1.8 and newer
This closes #2858

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-11 19:02:59 -04:00
Jeff Storck 65c41ab917 NIFI-6196: Upgrade Jetty version to 9.4.15.v20190215
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore.  Also updated the tests to separately test one-way and two-way SSL

NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.

This closes #3426.
2019-05-09 12:05:12 -04:00
Joe Witt 1bdab73201 NIFI-6121 Updated the easy dependencies to update
This closes #3374

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-22 09:01:21 -04:00
Bryan Bende 9d21a106ad NIFI-6120 Move nifi.registry.version property to root pom so it can be shared between nifi-framework and nifi-toolkit
This closes #3369.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-03-20 10:56:10 +09:00
Peter G. Horvath 43235724e2 NIFI-5318 Implement NiFi test harness: initial commit
NIFI-5318 Implement NiFi test harness: replaced original sample feed payload with synthetic content
NIFI-5318 Implement NiFi test harness: fixed test harness run crash issue; better reporting of paths
NIFI-5318 Implement NiFi test harness: added further states where NiFi version can be queried
NIFI-5318 Implement NiFi test harness: fixed incorrect class reference
NIFI-5318 Implement NiFi test harness: added type parameter bounding to setClassOfSingleProcessor to prevent configuring obviously incorrect classes
NIFI-5318 Updated project version.

This closes #3165

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-16 10:08:44 -04:00
thenatog 36bbc77723 NIFI-6097 - Upgraded the fasterxml jackson version to 2.9.8. Ensure that the version is consistent across modules using a maven property defined in the root pom.
This closes #3347

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-04 19:30:35 -05:00
Joe Skora 270b6d5897 NIFI-4166 Create nifi-toolkit-api.
NIFI-4166 Adjusted dependencies
NIFI-4166 Excluded RAT and checkstyle for generated-sources. (Added by Mike Thomsen)

This closes #1994

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-21 07:09:25 -05:00
joewitt 0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt 45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Mark Payne 1ac5b93144
NIFI-5859: Added XML-based documentation writer that can be used to document a component.
- Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class.

NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin

NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s)

This closes #3192.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-12-10 11:16:01 -05:00
Bryan Bende fdd8cdbb31 NIFI-5673 Set up property/assembly for new auto-load directory
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface

This closes #3119.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-11-01 14:41:10 -04:00
Jeff Storck c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck 98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
Takanobu Asanuma ec8afd18d6 NIFI-5446: Specify the file encoding option to UTF-8 in pom.xml
This closes #2910.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-10-18 15:09:23 +09:00
Jeff Storck 01e2098d24 NIFI-5585 A node that was previously offloaded can now be reconnected to the cluster and queue flowfiles again
Added Spock test for NonLocalPartitionPartitioner
Updated NOTICE files for FontAwesome with the updated version (4.7.0) and URL to the free license
Updated package-lock.json with the updated version of FontAwesome (4.7.0)
Added method to FlowFileQueue interface to reset an offloaded queue
Queues that are now immediately have the offloaded status reset once offloading finishes
SocketLoadBalancedFlowFileQueue now ignores back-pressure when offloading flowfiles
Cleaned up javascript in nf-cluster-table.js when creating markup for the node operation icons
Fixed incorrect handling of a heartbeat from an offloaded node.  Heartbeats from offloading or offloaded nodes will now be reported as an event, the heartbeat will be removed and ignored.
Added unit tests and integration tests to cover offloading nodes
Updated Cluster integration test class with accessor for the current cluster coordinator
Updated Node integration test class's custom NiFiProperties implementation to return the load balancing port and a method to assert an offloaded node
Added exclusion to top-level pom for ITSpec.class
2018-10-11 09:23:01 -04:00
joewitt 8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
joewitt a27ccd8a56 NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we can avoid a new jetty bug with META-INF loading logic.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:11 +09:00
Bryan Bende 3fd45ac4c4 NIFI-5354 Changing ranger.version to 1.0.0 and ranger.hadoop.version to 3.0.0
Signed-off-by: Yolanda Davis <ymdavis@apache.org>

This closes #2827
2018-06-29 23:36:54 -04:00
Andy LoPresto d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto 99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Kevin Doran fc902234b6 NIFI-4942 Fixes Travis CI build
- Fixes unit test salt assertion regex
- Adds RAT excludes and reenables console output for unapproved files

NIFI-4942 Temporarily disables tests that are failing on Linux

This closes #2648.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-04-20 05:15:47 +00:00
joewitt d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Mike Thomsen aa947e4d3e NIFI-4325 Added new processor that uses the JSON DSL.
NIFI-4325 Cleaned up how ElasticSearch client service builds SSLContext, added query attribute to flowfiles and other changes requested in a code review.

This closes #2113.

Signed-off-by: Joe Percivall <JPercivall@apache.org>
2018-03-26 15:01:37 -04:00
Mark Payne 0b0aebe148
NIFI-4917: Externalize Keytab and Principal configuration from Processors to a Controller Service. This gives us the ability to allow users to interact with those Keytabs/Principals to which they've been given access without allowing them access to all Keytabs and Principals
- Addressed review feedback; rebased against master

This closes #2552.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-21 14:23:41 -04:00
Matt Gilman 32a8db2253
NIFI-4936:
- Moving definition of ranger version into root pom.

This closes #2532.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-03-12 10:39:54 -04:00
joewitt c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
2018-03-09 16:34:53 -05:00
Matt Gilman 8f66454455
NIFI-4945:
- Upgrading spring security version.

This closes #2524.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-03-09 11:50:26 -08:00
Andy LoPresto e9d585a785
NIFI-4870 Upgraded activemq-client and activemq-broker versions to 5.15.3.
- Updated copyright dates in NOTICE files.
- This closes #2469.
2018-03-09 13:59:13 -05:00
Andy LoPresto bff5b7ab71 NIFI-4928 Updated BouncyCastle dependencies to version 1.59.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2520.
2018-03-08 09:02:57 +01:00
Bryan Bende 9cf9e866ba NIFI-4839 Creating nifi-toolkit-cli to provide a CLI for interacting with NiFi and NiFi Registry 2018-02-28 17:22:28 +01:00
mans2singh f7fe2da106 NIFI-4289 - InfluxDB Put processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2101.
2018-02-24 14:46:35 +01:00
Koji Kawamura dbbf78f22c
NIFI-4848: Update HttpComponents version
- httpclient 4.5.3 -> 4.5.5
- httpcore 4.4.4 -> 4.4.9
  - ThreadSafe annotation is removed since 4.4.5, HTTPCLIENT-1743.
    Removed the annotation from DebugFlow processor.
- httpasyncclient 4.1.2 -> 4.1.3
- This closes #2453
2018-02-07 08:55:43 -05:00
Matthew Burgess 7e29103995 NIFI-4428: This closes #2310. Updated POM versions to current master branch
Signed-off-by: joewitt <joewitt@apache.org>
2018-01-16 13:27:29 -05:00
vvaks 7fa0a34aba NIFI-4428: - Implement PutDruid Processor and Controller
update

added provenance report

added parameters for batch control

WIP
2018-01-16 13:07:56 -05:00
joewitt 41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt 36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt 46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
Matt Gilman 20b539aac3
NIFI-4436:
- Bumping registry version to 0.1.0.
- Addressing PR comments.
2018-01-08 12:44:56 -05:00
Mark Payne 6b00dff1a8
NIFI-4436: Integrate with actual Flow Registry via REST Client - Store Bucket Name, Flow Name, Flow Description for VersionControlInformation - Added endpoint for determining local modifications to a process group - Updated authorizations required for version control endpoints - Add state and percent complete fields ot VersionedFlowUpdateRequestDTO
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2018-01-08 12:44:53 -05:00
Mark Payne 6a58d780d7
NIFI-4436:
- Initial checkpoint: able ot start version control and detect changes, in standalone mode, still 'crude' implementation
- Checkpoint: Can place flow under version control and can determine if modified
- Checkpoint: Change version working in some cases. Does not work if processor removed because COMPONENT_REMOVED type has ComponentA whose ID is the VersionedComponentID but we are trying to call ProcessorDAO.get() with this ID
- Checkpoint: Able to change flow from Version 1 to Version 2 and back. Not yet tested with controller services. Have not tried changing/removing connections. Not cluster-friendly yet. All inline, not in background. Have not taken into account ports, funnels, remote ports, etc. Have not tested with Labels yet
- Checkpoint after implementing ClusterReplicationComponentLifecycle instead of JerseyClientComponentLifecycle
- Checkpoint: Updated to allow starting version control and updating version in clustered mode
- Checkpoint: Updated versioning endpoint so that when version of a flow is updated, the bundle information is populated and the snapshot is replicated to the cluster.
- Checkpoint: Implemented endpoint for reverting to previously sync'ed version of a flow and updated version control endpoint so that Process Group can be pushed as a new version to existing flow instead of only creating a new flow
- Checkpoint: Updated so that if a Process Group is under Version Control and it has a child Process Group, which is also under Version Control, we can handle that gracefully. Not yet tested because it depends on updates to the nifi-registry module, which can't be compiled due to maven dependency conflicts
2018-01-08 12:44:52 -05:00
joewitt 6ee6b5e57c NIFI-4734 refactored couchbase bundle, jms bundle, removed jms test-lib usage, fixed broken SeeAlso and service links causing docs warns. Corrected L&N in each bundle. This closes #2369.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-01-04 09:16:17 -05:00
joewitt 7467bb7b0f
NIFI-4730 restructure of nifi-gcp-bundle
This closes #2367.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-01-03 11:21:52 -05:00
Koji Kawamura fc73c60924 NIFI-3709: Export NiFi flow dataset lineage to Apache Atlas 2017-12-18 12:23:58 -05:00
Matthew Burgess 2192138b06 NIFI-4683: Add ability to execute Spark jobs via Livy 2017-12-14 16:23:41 -05:00
joewitt 7e61c6333c
NIFI-2375 updated to latest Tika across codebase. Fixed
tests and updated L&N for Tika parsers.  Turned unstable grpc test to IT
 test. Updated deps due to Tika testing findings. This closes #2320
2017-12-06 11:53:15 -05:00
joewitt cdc1facf39
NIFI-4664, NIFI-4662, NIFI-4660, NIFI-4659 moved tests which are timing/threading/network dependent and brittle to integration tests and un-ignored tests that are IT. Updated travis to reduce impact on infra and appveyor now skips test runs so is just to prove build works on windows. This closes #2319
squash
2017-12-06 10:53:09 -05:00
Mark Payne 00b11e82b7 NIFI-4600: This closes #2312. Added nifi-kafka-1-0-nar and nifi-kafka-1-0-processors modules
Signed-off-by: joewitt <joewitt@apache.org>
2017-12-04 16:51:59 -05:00
joewitt 2c68d0e098
NIFI-4652 updating to latest apache version and updating key maven plugins, fixing versions for grpc to avoid extranous build dir, correcting checkstyle issues, resolving build problem with incorrect usage of activeByDefault in maven. Key change is now for the 'dir-only' profile it is now activated via a system property -Ddir-only instead of profile. No other activeByDefault profiles exist except to minifi the javascript but that usage is acceptible as-is. This closes #2313 2017-12-04 09:59:23 -05:00
Joey Frazee a458a139b3 NIFI-4650 This closes #2308. Added independent hadoop.http.core.version property
This closes #2308

Signed-off-by: joewitt <joewitt@apache.org>
2017-12-02 12:58:29 -05:00
dlukyanov c7a5a09b8a groovyx initial version
NIFI-3688 license update to ASF
NIFI-3688 add groovyx dependency
NIFI-3688 comments from @alopresto
- reformat code https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-EclipseUsers
- enabled java-doc
NIFI-3688
- try fix mess with licenses
- remove commented import
- remove unused var `sql`
NIFI-3688 the properties/skip javadoc/src
NIFI-3688 remove not used sql var
NIFI-3688
- first test case
NIFI-3688
+ 2 more tests based on current groovy tests
NIFI-3688 comment from @mattyb149
- processor renamed to ExecuteGroovyScript
NIFI-3688 fix script path validation
NIFI-3688
- refactor to compile on validation
- prepare 4 test cases with database
NIFI-3688 new test cases with groovy and sql
NIFI-3688 documentation
NIFI-3688
- refactor groovy extended methods
- add more test cases
NIFI-3688
- codestyle
- javadoc
- refactor flowfile voids to self-reference
- fix test cases
NIFI-3688 minor comments changes
NIFI-3688 rename additional documentation according to processor classname
NIFI-3688 exclude json test file from rat check
NIFI-3688 codestyle
NIFI-3688 add nar bundle to root pom.xml and to nifi-assembly/pom.xml
NIFI-3688
- fix & extend additional documentation
- fix pom.xml to bundle groovy into nar
NIFI-3688 add examples into additional processor documentation
NIFI-3688 fix pom.xml to exclude unnecessary libs from nar bundle
NIFI-3688 add restricted annotation
NIFI-3688 change version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT after rebase
NIFI-3688 new method in ProcessSession: public OutputStream write(FlowFile source)

NIFI-3688 change version from 1.3.0-SNAPSHOT to 1.4.0-SNAPSHOT after rebase

NIFI-3688 fix for @mattyb149 comment: The bundles referred to here and below are not included with this NAR and should be removed.

NIFI-3688 fix for @mattyb149 comment: This unused line can be removed

NIFI-3688
- removed `require flowfile` property
- fixed test cases according to deprecated property

change version to 1.5.0-SNAPSHOT

[NIFI-3688] Commented by mistake

[NIFI-3688] remove unused class

[NIFI-3688] fix javadoc comments

[NIFI-3688] refactor CTL & SQL properties

b3eecec901

2916ce1ec8

8e15392e2f

NIFI-3688: Checkstyle and typo fixes

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #1662
2017-11-13 14:32:58 -05:00
Koji Kawamura d914ad2924 NIFI-4547: Add ProvenanceEventConsumer utility class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2236.
2017-10-30 09:50:44 +01:00
Christopher Currie c894246e03 NIFI-3950 Refactor AWS bundle
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2140.
2017-10-21 12:58:40 -07:00
Matt Gilman 6baea8ccff
NIFI-4444:
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.

This closes #2206.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-10-12 10:27:02 -07:00
Omer Hadari 6201c06c99
NIFI-4392 Create a MetricReportingTask with GraphiteMetricService
This closes #2171.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-10-08 10:44:11 -04:00
Andy LoPresto d4168f5ff1
NIFI-4297
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes #2084
2017-10-05 15:23:52 -04:00
Matt Gilman 697c382d9b
NIFI-4328:
- Upgrading swagger and the corresponding maven plugin to address issues discovered in recursive domain models.
2017-10-03 09:50:11 -04:00
Jeff Storck a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Jeff Storck 502adc6e8a NIFI-4412 Removed duplicate dependency management definition for nifi-mongodb-client-service-api. 2017-09-28 11:08:47 -04:00
Matthew Burgess 1ca4fa3a83
NIFI-4345: Removed lookup-services dependency from MongoDB Services NAR, cleaned up deps
- Added nifi-lookup-service-api to nifi-nar-bundles POM

This closes #2174.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-09-25 14:52:33 -04:00
Mark Payne 3fb704c58f NIFI-4201: This closes #2024. Implementation of processors for interacting with Kafka 0.11
Signed-off-by: joewitt <joewitt@apache.org>
2017-09-22 22:08:19 -04:00
Mike Thomsen 9a8e6b2eb1 NIFI-4345 Added a MongoDB controller service and a lookup service.
NIFI-4345: Changed Lookup Key to Lookup Value Field
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2123
2017-09-22 13:33:07 -04:00
cam e3da44fb62 [NiFi-3973] Add PutKudu processor to ingest data to Kudu 2017-08-25 10:53:31 -04:00
Bryan Bende 60d4672195 NIFI-4304 Extracting HWX Schema Registry client version to a property and bumping to latest 0.3.0 release
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2096
2017-08-17 16:07:03 -04:00
Matt Gilman 528b82634f
NIFI-4210:
- Introducing support for OpenId Connect.
- Updating REST API and UI to support the authorization code flow.
- Adding/fixing documentation.
- Implementing time constant equality checks where appropriate.
- Corrected error handling during startup and throughout the OIDC login sequence.
- Redacting the token values from the user log.
- Defaulting to RS256 when not preferred algorithm is specified.
- Marking the OIDC endpoints as non-guaranteed in to allow for minor adjustments if/when additional SSO techniques are introduced.

This closes #2047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-08-10 11:15:35 -07:00
Matt Gilman 743c6b9c17 NIFI-4032: - Introducing the ManagedRangerAuthorizer. - Introducing the AuthorizationAuditor. - Updating authorization requests to utilize Authorizable where ever possible so allow for a singular place to audit resource not found as denied when the parent authorizable is null (no more inheritance). - Updating unit tests as appropriate. - Addressing issues with broken web-api integration tests.
NIFI-4032: - Generating the appropriate fingerprint for the ManagedRangerAuthorizer based on whether the UserGroupProvider is configurable. - Adding unit tests.

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #2019
2017-07-24 15:51:27 -04:00
mans2singh affc88e599 NIFI-4118 RethinkDB bundle with put processor
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1942.
2017-07-09 11:10:23 -07:00
Mark Payne 3906d4e1d2 NIFI-1763: Initial implementation of ConfluentSchemaRegistry.
NIFI-1763: Fixed bug where the Confluent Schema Registry Schema Access Writer was not being created

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1938
2017-07-06 15:52:07 -04:00
m-hogue 58a623dfa2 NIFI-4037 added InvokeGRPC processor, with proto service IDL
NIFI-4038 added ListenGRPC processor

This closes #1947

Signed-off-by: Tony Kurc <tkurc@apache.org>
2017-06-29 22:01:22 -04:00