Commit Graph

158 Commits

Author SHA1 Message Date
exceptionfactory fe424a2d42
NIFI-9805 Refactored Distributed Cache Servers using Netty
- Added Map and Set Cache Servers based on nifi-event-transport components
- Removed custom servers and unused socket stream components
- Reduced duplication on protocol classes
- Added checks for readable bytes
- Added mark and reset handling for buffer reads

This closes #6040
Signed-off-by: Paul Grey <greyp@apache.org>
2022-05-18 12:40:16 -04:00
Joe Witt 84340962d4
NIFI-9927 Initial Commit to clean up all extraneous version/group declarations.
This closes #5969

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-04-16 12:05:07 -04:00
Joe Witt 873d25585c
NIFI-9780 Merge branch 'NIFI-9780-RC3' 2022-04-04 08:35:23 -07:00
exceptionfactory 0fa4490a98
NIFI-9842 Refactored nifi-commons using JUnit 5
- Refactored nifi-bootstrap using JUnit 5
- Refactored nifi-maven-archetypes using JUnit 5
- Refactored nifi-stateless using JUnit 5

This closes #5912

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2022-03-28 16:00:23 -04:00
Joe Witt 0419dc2939
NIFI-9780-RC3 prepare for next development iteration 2022-03-21 11:58:27 -07:00
Joe Witt b019a9191f
NIFI-9780-RC3 prepare release nifi-1.16.0-RC3 2022-03-21 11:58:09 -07:00
sedadgn e603b0179b NIFI-7865 amqp$header is splitted in the wrong way for "," and "}"
This PR introduces 2 new properties for the ConsumeAMQP processor
And one new property for PublishAMQP

This allows to configure the processors to use escaping for commas and to consistently not use curly braces in the amqp$header attribute.

The default values ensure backwards compatibility.

This closes #5458.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2022-01-18 16:45:45 -05:00
Mark Payne d9b863a84b
NIFI-9549: Delegate NonFlushableOutputStream write methods to wrapped OutputStream
Ensure that we delegate calls to write(byte[]) and write(byte[], int, int) to the underlying OutputStream for NonFlushableOutputStream, instead of allowing FilterOutputStream to iterate over every byte

This closes #5642

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2022-01-06 22:46:55 -06:00
markap14 839fbf7d19
NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which H… (#5533)
* NIFI-9382: Created a new ClassloaderIsolationKey mechanism by which Hadoop related processors (and potentially others) can indicate that they need full classloaders to be cloned but can share with other instances in certain circumstances
- Added system tests

* NIFI-9382: Renamed interface based on review feedback

* NIFI-9382: Removed ReentrantKerberosUser.
2021-11-22 14:55:19 -05:00
Joe Witt c055895952
NIFI-9344-RC3 prepare for next development iteration 2021-11-03 08:53:37 -07:00
Joe Witt 7fdc07cccd
NIFI-9344-RC3 prepare release nifi-1.15.0-RC3 2021-11-03 08:53:32 -07:00
Mike Thomsen 8d513c5ed3
NIFI-9080 Converted nifi-commons to use JUnit 5
This closes #5332

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-25 21:07:54 -05:00
Nathan Gough 309ab8f4ab
NIFI-8792 - Modified ListenRELP to use Netty
- Refactored RELP encoders and decoders

This closes #5398

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-10-25 15:53:58 -05:00
Paul Grey 720e7ef353
NIFI-8919 - Refactor Distributed Cache Clients using Netty
This closes #5311

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-09-27 21:00:58 -05:00
Mark Payne e29c8af708
NIFI-9182: When calling ProcessSession.append(), do not allow calls from processor to flush the underlying BufferedOutputStream. Instead, wrap in a NonFlushableOutputStream and only flush when session commit is called
This closes #5359

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-09-22 18:54:32 -05:00
exceptionfactory f5a421ccb1
NIFI-9067 Upgraded JAXB Runtime from 2.3.0 to 2.3.5
- Replaced old com.sun.xml.bind:jaxb-impl and jaxb-core with current org.glassfish.jaxb:jaxb-runtime
- Replaced old javax.xml.bind:jaxb-api with current jakarta.xml.bind-api
- Removed unnecessary dependency references to javax.activation-api

This closes #5320

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-09-07 13:27:53 -04:00
Mark Payne fb4edfa0d6
NIFI-8950: Added ability to optionally use a file-system backed Content Repository for Stateless NiFi
This closes #5254

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-08-11 12:01:07 -05:00
Timea Barna 462306369f NIFI-8668 ConsumeAzureEventHub NiFi processors need to support storage SAS token authentication
This closes #5136.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-08-09 12:57:56 +02:00
Joe Witt 97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
exceptionfactory a3365c8833 NIFI-8462 Refactored PutSyslog and ListenSyslog using Netty
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog

NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues

NIFI-8462 Removed unused import of ExpressionLanguageScope

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5044.
2021-05-25 10:54:15 -04:00
Lehel Boér a3eaf0a37a NIFI-8325: Complete SNMP refactor: SNMP GET and SET processors reworked, unit tests added
This closes #5028.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-19 23:26:31 +02:00
Paul Grey cc554a6b11
NIFI-8478 Addressed i18n test issues; CI supplies locale to surefire
This closes #5040

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-03 13:40:01 -05:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Mark Payne 91313a2e75
NIFI-8360: Fixed an overflow issue where we used an integer to store the number of bytes encountered when reading data and searching for a given pattern
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4929.
2021-03-24 15:30:22 +01:00
Mark Payne 528fce2407
NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches

NIFI-7646: Removed TODO and unused Jackson dependency

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4818
2021-02-23 10:57:54 -05:00
exceptionfactory 418e2cc2cb NIFI-8132 Replaced framework uses of MD5 with SHA-256
NIFI-8132 Added FileDigestUtils in nifi-nar-utils to avoid dependency on nifi-utils

NIFI-8132 Removed unused imports from NarUnpacker

NIFI-8132 Removed MD5 references from FileUtils documentation

NIFI-8132 Replaced StringBuffer with StringBuilder and made new DigestUtils classes final

NIFI-8132 Replaced Collections.sort() with Stream.sorted()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4788.
2021-02-22 17:52:24 -05:00
Joe Witt 88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt 4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt 487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
exceptionfactory e7c6bdad42 NIFI-7937 Added StandardFlowFileMediaType enum to replace string references to FlowFile Media Types 2021-01-04 14:20:05 -05:00
tlsmith 112b893bf2 NIFI-7225 FetchSFTP processor routing to not.found error given when Private Key Path property is invalid 2021-01-04 13:41:03 -05:00
Mark Payne 3c9d8a7007
NIFI-7897: Refactoring to create a new nifi-framework-components module.
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup

NIFI-7897: Addressing review feedback

NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly

This closes #4669.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-24 11:37:45 -05:00
Andy LoPresto 9370571131
NIFI-7804 Split nifi-security-utils into sub-module for nifi-security… (#4533)
* NIFI-7804 Split nifi-security-utils into sub-module for nifi-security-utils-api (no external dependencies).
Separated interface and implementation of TlsConfiguration.
Reabsorbed nifi-security-xml-config into nifi-security-utils.

* NIFI-7804 Resolved failing unit test on Java 8.
Removed accidental module dependency.

* NIFI-7804 Resolved failing unit test.

* NIFI-7804 Removed legacy dependency.

* NIFI-7804 Marked nifi-security-utils-api as provided and overrode with compile scope in specific modules which are not children of nifi-standard-services-api-nar.
2020-09-17 12:52:22 -04:00
Otto Fowler bdfe1f2370 NIFI-7766 Getting initialize class org.apache.nifi.util.text.RegexDateTimeMatcher error in jsontreereader
check for null values returned by getZoneStrings()
2020-08-26 10:50:49 -04:00
Joe Witt 8baa5c9940
NIFI-7692 updating for next dev release 1.13.0 2020-08-18 14:48:02 -07:00
Joe Witt fb57bcbc11
NIFI-7692-RC1 prepare for next development iteration 2020-08-13 09:20:39 -07:00
Joe Witt 303d6c59ba
NIFI-7692-RC1 prepare release nifi-1.12.0-RC1 2020-08-13 09:20:36 -07:00
Peter Turcsanyi c980b64bf5
NIFI-7340: Adding ListAzureDataLakeStorage
Also added validator for Directory Name property in AbstractAzureDataLakeStorageProcessor
Fix Tracking Entities strategy: use milliseconds for lastModified

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4438.
2020-08-03 19:27:21 +02:00
Joey c18b27af18
NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment (#4341)
* NIFI-7540: Fix TestListenSMTP and TestListFile on macOS build environment

This also fixes NIFI-4760.

* NIFI-7540: Remove duplicate mail.smtp.starttls.enable from TestListenSMTP

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-06-16 17:45:37 -07:00
Tamas Palfy e2716a6c94 NIFI-7280 ReportLineageToAtlas recognizes 'atlas.metadata.namespace' from Atlas config file.
Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property.
Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties.

NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor.
NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate.

This closes #4213.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-20 22:34:03 +02:00
Otto Fowler d9ac9e44a7
add validator for lists that ensure the element validator is called for empty entries
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4116
2020-04-02 09:29:06 -04:00
Otto Fowler f1c6e92df5
NIFI-7055 handle empty split evaluations, which contain only ,
add explict test for " , "

updated with counting validator

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4012
2020-03-04 18:34:18 -05:00
Otto Fowler 6e8f10c4f6
NIFI-7109 Unit tests should be able to determine if item validator was called
- Create a mock Validator to track count of calls to validate().
We cannot use Mockito for this, because it can't mock all the
StandardValidators

refactor based on review comments

fix naming in comments

moved to main based on review

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4043
2020-02-12 11:22:31 -05:00
Joe Witt 3de77ebacc
NIFI-7021-RC3 prepare for next development iteration 2020-01-19 14:14:40 -05:00
Joe Witt 633408bce7
NIFI-7021-RC3 prepare release nifi-1.11.0-RC3 2020-01-19 14:14:38 -05:00
Mark Payne 5f4ce8f431
NIFI-6707: Addition of nifi-system-tests module. Updates to EmbeddedNiFi, NiFi and SystemBundle to properly support classloader isolation that is needed for creating system-level tests and isolating the classpath of the tests themselves from the NiFi classpath. Fixed bootstrap listener to ensure that it always closes socket before calling shutdown(). Failure to do so can result in RunNiFi sometimes timing out while waiting for a response from NiFi
NIFI-6707: Added System Test to verify behavior when services depend on one another

NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them

NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group

NIFI-6707: Updated to include java11 subdirectory under lib/

NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT

NIFI-6707: Updates to see if it will help travis build correctly

NIFI-6707: Commenting out java commands in bootstrap.conf files

This closes #3831.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-11-08 12:33:16 -05:00
Mark Payne a9db5a8cb7
NIFI-6841: Fixed bug that resulted in the wrong number of 'Bytes Read' being reported by ByteCountingInputStream in the event that #skip was called between calls to #mark and #reset
This closes #3868
2019-11-05 13:06:06 -05:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00