Commit Graph

5754 Commits

Author SHA1 Message Date
Michael Hogue 421bfdd5ff
NIFI-6919: Added relationship attribute to DistributeLoad
NIFI-6919: Cleaned up docs

NIFI-6919: Cleanup

NIFI-6919: Cleanup

NIFI-6919: added negative unit test

NIFI-6919: Removed unnecesary feature flag

Updated attribute description

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3939
2020-01-14 16:14:48 -05:00
Matthew Burgess 824cc0ed77
NIFI-7014: This closes #3985. Add RecordReader/Writer access in ExecuteGroovyScript
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-14 12:58:21 -05:00
Matthew Burgess 6dbcf9d521
NIFI-6902: This closes #3986. Added unit test for original author's patch
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-14 11:59:08 -05:00
Tom eb37e7475f
NIFI-6902 Update PrometheusReportingTask to supply correct fields for OutputPort 2020-01-14 11:58:45 -05:00
Martin Šimek 2601f722b3
NIFI-6998 This closes #3972. Batch & Partitioning key
+ Asynchronous sending in limited batches
+ Property to determine attribute name carrying partitioning key
+ Maximum batch size property
+ Carrier object
- Unit test fakery
NIFI-6998 Attributes to User Defined properties
NIFI-6998 Unit tests
NIFI-6998 Review corrections
+ Interruption propagation (& test)
+ Final carrier members
+ Unnecessary generic modifiers removed from generic container
NIFI-6998 checkstyle corrections
+ Tabs to spaces, trailnig spaces
+ Absolute Imports
+ Braces locations
NIFI-6998 imports & license

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 22:37:46 -05:00
Mark Payne 66d5ab80eb
NIFI-7011: This closes #3983. SwappablePriorityQueue contains two internal data structures: activeQueue, swapQueue. activeQueue is intended to be pulled from for processing. swapQueue is intended to hold FlowFiles that are waiting to be swapped out. SinWe want to ensure that we first swap in any data that has already been swapped out before processing the swap queue, in order to ensure that we process the data in the correct order. This fix ddresses an issue where data was being swapped out by writing the lowest priority data to a swap file, then adding the highest priority data to activeQueue and the 'middle' priority data back to swapQueue. As a result, when polling from the queue we got highest priority data, followed by lowest priority data, followed by middle priority data. This is addressed by avoiding putting anything back on swapQueue when we swap out. Instead, write data to the swap file, then push everything else to activeQueue. This way, any new data that comes in will still go to the swapQueue, as it should, but all data that didn't get written to the Swap file will be processed before the low priority data in the swap file.
NIFI-7011: Addressed corner case where data could be inserted out of order still if added while swapping was taking place
NIFI-7011: Fixed ordering issue with swap queue that can occur if data is migrated from swap queue to active queue instead of being swapped out
2020-01-13 22:02:05 -05:00
nagasivanath 4e2b61efe4
NIFI-6652 UI Fix overflowing text in variables dialog
This closes #3982.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-13 12:12:34 -08:00
John Pierce 0dc6439ddb
NIFI-6964 This closes #3975. Use compression level for xz-lzma2 format of the CompressContent processor
NIFI-6965 adding resource warning on memory and highlighting this with the compression level property

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 15:08:13 -05:00
Tamas Palfy b7fb94723c NIFI-6884 - Native library loading fixed/improved: NarClassLoader and InstanceClassLoader can load libraries from their own or their ancestors' NAR-INF/bundled-dependencies/native directory.
They also scan directories defined via java.library.path system property.
InstanceClassLoader also checks additional classpath resources defined by PropertyDescriptors with "dynamicallyModifiesClasspath(true)".
Added tests for loading native libraries. Supports mac only.
Added support for loading native libs from additional resources in AbstractHadoopProcessor.
Updated javadoc for PropertyDescriptor.dynamicallyModifiesClasspath.

This closes #3894.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-01-13 13:59:11 -05:00
mdayakar ccb85cf8b7
NIFI-6923:If FlowFile content is truncated in the Content Repository, NiFi does not throw Exception when reading the content.
NIFI-6923 fixing checkstyle
2020-01-13 12:45:59 -05:00
Tamas Palfy 26fcf8158a NIFI-6992 - Add "Batch Size" property to GetHDFSFileInfo processor - Added "Batch Size property", takes effect only if "Destination" is set to "Content" and Grouping" is set to "None"
NIFI-6992 - Add "Batch Size" property to GetHDFSFileInfo processor - Added validation for 'Batch Size' in 'GetHDFSFileInfo'.
NIFI-6992 - Changed 'GetHDFSFileInfo.BATCH_SIZE' validator from 'NON_NEGATIVE_INTEGER_VALIDATOR' to 'POSITIVE_INTEGER_VALIDATOR'. Added more tests.
NIFI-6992 - Removed 'AllowEmptyValidator'. 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String.
NIFI-6992 - 'Batch Size' in 'GetHDFSFileInfo' allows null but not empty String - cont.
NIFI-6992 - Fix: Unused import.

This closes #3966.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-01-13 17:50:33 +01:00
Joe Witt bec1f772db
NIFI-6852 added missing license header 2020-01-13 11:43:17 -05:00
Shawn Weeks 2c18cf22ad
NIFI-6852 This closes #3973. Don't Validate Processors that accept any ControllerService Implementation
NIFI-6852 Add Integration Tests for Controller Service API Validation

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-13 11:39:39 -05:00
Andy LoPresto 2cc467eb58
NIFI-3833 Added encrypted flowfile repository implementation.
Added EncryptedSchemaRepositoryRecordSerde.
Refactored CryptoUtils utility methods for repository encryption configuration validation checks to RepositoryEncryptorUtils.
Added FlowFile repo encryption config container.
Added more logging in cryptographic and serialization operations.
Generalized log messages in shared encryption services.
Added encrypted serde factory.
Added marker impl for encrypted WAL.
Moved validation of FF repo encryption config earlier in startup process.
Refactored duplicate property lookup code in NiFiProperties.
Added title case string helper.
Added validation and warning around misformatted encryption repo properties.
Added unit tests.
Added documentation to User Guide & Admin Guide.
Added screenshot for docs.
Added links to relevant sections of NiFi In-Depth doc to User Guide.
Added flowfile & content repository encryption configuration properties to default nifi.properties.

Signed-off-by: Joe Witt <joewitt@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3968.
2020-01-10 10:44:59 -08:00
Peter Turcsanyi b8ffb54612
NIFI-6945: Use minExtInfo=true to reduce the amount of data queried from Atlas
This closes #3970
2020-01-10 09:03:07 -05:00
Joe Witt fc2e6b3471 NIFI-7006 Changed the Kudu controller service test to an integration test since it starts a server and is not portable across build environments (#3976)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 19:10:03 -08:00
mtien 0ae200252f
NIFI-6978 added code and unit test to throw IllegalArgumentException when improper value given
This closes #3960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 13:50:27 -08:00
mtien a6f0498cf9
NIFI-6830 Change code to allow empty KeyPassword.
Add unit test verifying empty keyPassword.

This closes #3963.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2020-01-09 11:53:15 -05:00
Bryan Bende 08d0352ac1
NIFI-6985 Use correct versioned parameter contexts when child process groups are version controlled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3962.
2020-01-09 17:35:25 +07:00
Matthew Burgess 8d214f99a4
NIFI-6960: Add Provenance Table to MetricsSqlQueryService
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3945.
2020-01-09 17:26:43 +07:00
Matthew Burgess 28aa155fd4
NIFI-6989: Removed call to cancel() in SiteToSiteReportingSink
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3969.
2020-01-09 17:16:23 +07:00
Yolanda Davis ebd33452c4 NIFI-6962 - moved HashAlgorithm and HashService to nifi-security-utils (#3947) 2020-01-08 11:10:58 -08:00
Peter Turcsanyi cc74534bc0
NIFI-6939: Upgrade Atlas client dependency to 2.0.0
NIFI-6939: Review changes

This closes #3944
2020-01-08 13:25:45 -05:00
Matthew Burgess c604923c0b NIFI-6947: Add PutRecord processor using RecordSinkService (#3943)
* NIFI-6947: Add PutRecord processor using RecordSinkService

* NIFI-6947: Incorporated review comments
2020-01-08 04:25:14 -04:00
Matthew Burgess f641a15c5f
NIFI-6989: Fixed SiteToSiteReportingRecordSink when transaction sent no data
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3965.
2020-01-08 13:56:42 +07:00
Matthew Burgess f4ef45678e
NIFI-6963: Fix ValidateRecord handling of missing required array fields
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3948.
2020-01-08 02:02:04 +07:00
markap14 209c560ff5
Merge pull request #3946 from IlyaKovalev/NIFI-6955
NIFI-6955 close streams in Reader/writer mock
2020-01-07 10:55:27 -05:00
markap14 596e0ffbdc
Merge pull request #3932 from mtien-apache/NIFI-6336
NIFI-6336 Added code to catch port value when it is 0.
2020-01-07 10:48:50 -05:00
Matthew Burgess 9ed4623817
NIFI-6886: Fixed SiteToSiteReportingRecordSink, refactored mocks
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3959.
2020-01-07 15:15:54 +07:00
Eduardo Fontes 1ee6dba00a
NIFI-6886 - Bugfix
Attribute peerPersistence can be null generating Bulletin WARNs "Unable to refresh Remote Group's peers due to null".

Rollback

The fix is inside site-to-site-reporting-task-bundle

Modify getClient()

Get ConfigurationContext and ReportingContext to provide a StateManager.

Modify OnScheduled setup()

The OnSchedule setup() now save the ConfigurationContext to lazily create a SiteToSiteClient with ReportingContext through an overloaded setup().

Modify OnTrigger

Lazily creates SiteToSiteClient to provide a StateManager

Modify OnTrigger

Lazily create SiteToSiteClient to provide a StateManager

Modify OnTrigger

Lazily create SiteToSiteClient to provide a StateManager

Modify OnTrigger

Lazily create SiteToSiteClient to provide a StateManager

Retry compile

Fix maven-checkstyle-plugin

Fix maven-checkstyle-plugin

Fix maven-checkstyle-plugin

Fix maven-checkstyle-plugin

Update AbstractSiteToSiteReportingTask.java

Removed the OnSchedule setup(ConfigContext) because it is not needed.

Update SiteToSiteUtils.java

Removed ConfigContext from getClient parameters because ReportContext share the same properties.
2020-01-07 15:15:44 +07:00
Ubuntu 65ca8e6c8d
NIFI-6942 This closes #3934. Added a reporting task to push provenance data to azure log analytics.
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-01-06 15:42:15 -05:00
markap14 e4bdc79ea6
Merge pull request #3853 from markap14/NIFI-6822
NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we p…
2020-01-06 15:01:42 -05:00
Mark Payne c958deb5b0 NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we properly merge those maps during a checkpoint instead of overwriting existing values 2020-01-06 13:05:19 -05:00
Mark Payne b03e5b0520
NIFI-6983: Ensure that if any call to kafka's Producer throws a ProducerFencedException that it's handled properly by poisoning the lease, which in turn will close the client
This closes #6983.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-01-06 11:34:51 -05:00
Shawn Weeks b470db620b
NIFI-6979 Add record.index field to UpdateRecord
This closes #3955

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-01-03 19:25:01 -05:00
Endre Zoltan Kovacs c15876494a
NIFI-6897: making nifi-toolkit >= java 8 compliant
fixing cli.bat
cygwin fix: on windows ${JAVA} can have spaces in it.
...must wrap in quotes
some jvms doesn't print a dot after the major version

...removing that from patter matching

This closes #3908

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-12-30 08:10:37 -05:00
Peter Turcsanyi 41b47a5156
NIFI-6922: Fix typo in fs.permissions.umask-mode Hadoop property name in PutHDSF
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3911.
2019-12-30 10:01:25 +01:00
r65535 82b4fb0633
NIFI-6957 - Added REGEX header property, and option to allow illegal chars in header names
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3942.
2019-12-30 09:55:39 +01:00
Pierre Villard ad636789f0
NIFI-6928 - add connect/read timeout to RestLookupService
This closes #3920

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-12-29 19:10:30 -05:00
Mark Payne 5f0f801e46
NIFI-6879: Added system tests for variables. Fixed bug that resulted in Variable Registry not being updated if a Processor in a child group referenced it
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3949.
2019-12-26 18:05:08 +01:00
Andrew Lim 7f49c6b76a
NIFI-6907 Added nifi.bootstrap.sensitive.key to Bootstrap Properties section in Admin Guide
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3927.
2019-12-26 16:41:31 +01:00
mans2singh ec5bc7ea7a
NIFI-6971 - Minor typo in capabilities description
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3952.
2019-12-26 10:58:54 +01:00
Nathan Gough 42cb6e8489
NIFI-6948 - Removing some debug logging.
This closes #3935.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-12-23 16:07:52 -05:00
Mark Payne 2277d08c8e
NIFI-6949: When a Controller Service is removed, ensure that any other service that it references is obtained via the ControllerServiceProvider to ensure that it is obtain to obtain and remove the reference, even if the service is in a higher-level Process Group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3933.
2019-12-20 16:17:48 +00:00
KovalevIV 8f9fc49ca0 NIFI-6955 close streams in Reader/writer mock 2019-12-19 22:26:31 +03:00
mtien 2959c436b1 NIFI-6336 Fixed the Build Failure issue. 2019-12-19 10:47:23 -08:00
Matthew Burgess 0024668dd5
NIFI-6952: Evaluate EL for Hive3ConnectionPool properties
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3937.
2019-12-19 11:51:18 +01:00
Matthew Burgess 3a5a2da73c
NIFI-6954: Evaluate EL for DBCPConnectionPool properties
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3940.
2019-12-19 11:44:55 +01:00
mtien ab15536976 NIFI-6336 Fixed Exception message to refer to the correct port. 2019-12-18 15:06:45 -08:00
Joe Ferner 058883091c
NIFI-6872:
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful

NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication

This closes #3931
2019-12-18 16:37:47 -05:00