Commit Graph

6436 Commits

Author SHA1 Message Date
exceptionfactory be7233d252
NIFI-8304 Changed TestListenTCPRecord to loop until files found
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4904.
2021-03-17 11:41:21 +04:00
exceptionfactory 2ad88bbfff
NIFI-8304 This closes #4900. Improved Socket test reliability for several Processors
- Refactored TestPutTCP to single class
- Improved TestListenRELP
- Improved TestListenTCP
- Improved TestListenUDP
- Improved TestListenTCPRecord
- Changed OnUnscheduled to OnStopped in AbstractListenEventProcessor

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-16 15:30:11 -07:00
Peter Turcsanyi f00f0ad269 NIFI-8314: Generate warning for long-running processor tasks 2021-03-16 14:36:51 -04:00
exceptionfactory b1f513a2ca
NIFI-8304 Refactored InvokeHTTP unit tests using OkHttp MockWebServer
- Replaced Joda Time with java.time for date formatting
- Replaced Guava Files with java.nio.file.Files for cache directory
- Updated PutTCP test server to close connection when testing connection per FlowFile

NIFI-8304 Removed Thread wrapper for TestListenHTTP client requests

NIFI-8304 Disabled InvokeHTTP Connection Pooling for testing

NIFI-8304 Set 60 second timeout for testing TLS connections

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4892.
2021-03-15 21:28:40 +04:00
Bence Simon 57cca88eea NIFI-8289 Refine QuestDB status repository rollover and add time zone support
This closes #4883

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-12 09:10:02 -08:00
Matthew Burgess b039606cf8
NIFI-8297: Report incompatible value if it doesn't fit in an Integer
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4886.
2021-03-12 19:40:19 +04:00
Mark Payne 931f8fe798
NIFI-8316: Pass empty collections instead of null when calling write(ConfigurableComponent component)
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4891.
2021-03-12 12:24:06 +04:00
exceptionfactory 09692c5f69
NIFI-8313 Upgraded net.lingala.zip4j:zip4j to 2.7.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4889.
2021-03-12 12:21:07 +04:00
Otto Fowler 9f0b47af77
NIFI-2702 Support named captures in ExtractText
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4384.
2021-03-12 12:11:09 +04:00
Joe Witt 97b2e69101 NIFI-8309 updating to latest Jetty 2021-03-10 17:18:04 -05:00
Mark Payne a9b435e8d3
NIFI-8307: When StandardControllerServiceProvider.enableControllerServiceAndDependencies is called, ensure that it recursively calls itself for any dependent controller services. Updated toString() of ControllerServiceNode and passed itself to the ServiceStateTransition in order to facilitate better logging
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4882.
2021-03-10 20:13:42 +04:00
Mike Thomsen 160297f6ba NIFI-7717 Add ARGs to Docker build for parent image and tag
This closes #4460

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-10 01:11:41 +00:00
Bryan Bende 97536e9070
NIFI-8303 Exclude com.io7m.xom:xom when depending on spring-security-saml
Update LICENSE files to account for esapi BSD license

This closes #4878

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-09 14:39:46 -07:00
Mark Payne f09f2a6c85
NIFI-8212: When instantiating a temp component, ensure that nar class loader is set as the context class loader
This closes #4877

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-09 11:43:07 -06:00
Mark Payne 79121a142e
NIFI-6518: If processor is scheduled to run on Primary Node Only, and 'this' node is not primary node, yield for the bored yield duration.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4875.
2021-03-09 11:51:52 +04:00
Mark Payne 8c0f0a40cc
NIFI-8301: When we create a PreparedQuery in the StandardProcessContext, we do so with the effective property values. As a result, the Parameters have already been evaluated, so we need to use Query.prepareWithParametersPreEvaluated instead of Query.prepare
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4874.
2021-03-09 11:48:44 +04:00
Mark Payne dc94ef568a
NIFI-8299: Because we now iterate over stored events in an unordered map, we need to update maxEventId to Math.max( <current value>, <next value> ) instead of just assigning the next value.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4873.
2021-03-09 11:44:51 +04:00
exceptionfactory 98ab42aebc NIFI-8302 Corrected FingerprintFactory to use decrypt() method removing wrapping
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4876.
2021-03-08 22:16:21 -05:00
Mark Payne b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
Matthew Burgess 0965523bab NIFI-8266: Fix NPE when validating supported properties against validation context
- Removed schema properties from FreeFormTextRecordSetWriter causing validation issues

This closes #4851.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-03-08 11:30:13 -05:00
Pierre Villard c5b5be89a2
NIFI-8267 - Fix for HWX Schema Registry to use integers (#4850) 2021-03-08 09:12:37 -05:00
tpalfy f2a03fca2a
NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment. (#4868)
* NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment.

* NIFI-8285 Improve exception handling.
2021-03-08 09:10:12 -05:00
Peter Turcsanyi 9266212117
NIFI-8284: Removed diff merge leftover from the assembly NOTICE file (#4865) 2021-03-08 09:08:34 -05:00
Matthew Burgess 29ebec071e
NIFI-4974: Add try/catch around blob.free() to support JDBC spec
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4861.
2021-03-07 18:51:07 +04:00
Matthew Burgess ff93ec42c3
NIFI-8280: Remove Lumberjack NAR from assembly
This closes #4864

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-03-04 19:55:44 -05:00
Mark Payne e2e137fced
NIFI-8291: Support escaping unicode characters for fields that previously allowed escaping only of \t, \n\, \r
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4871.
2021-03-03 21:26:18 +04:00
mtien b9a0991157
NIFI-8260 [WIP] Created Upload File front end and server side logic.
NIFI-8260 [WIP] Fixed server side logic to upload a flow file. Cleaned up the front end logic.
NIFI-8260 [WIP] Finished the server side upload logic.
Added a client ID parameter to the endpoint.
Added JSON parsing error response.
Fixed the client side file form to reset after submit.
Fixed the canvas to instantly update and show the process group after submitting the file.
Changed the Add Processor Group dialog UI based on design notes.
Changed the Upload File link to an icon and moved to the process group name input.
Changed the Registry Import link to say 'Import from Registry' and moved to the bottom of the dialog.
Display the filename when a file is selected.

NIFI-8260 [WIP] Added a cancel file button to the Process Group dialog.
Fixed some CSS styles.

NIFI-8260 - Removed accessing the snapshot metadata to avoid an NPE.
Added a title attribute to the html of the dialog file cancel button.

NIFI-8260 - Disabled the dialog 'Add' button.
Revised based on PR feedback.
Refactored the upload file endpoint and client side filename extraction methods.
Fixed some CSS.
Reverted some unnecessary changes.

NIFI-8260 - Revised based on PR feedback.
Refactored uploadProcessGroup.
Fixed some exception handling.
Hid the Upload File button when grouping components.
Refactored nf-ng-group-component.js replacing jquery selectors with variables.
Extracted the resetValues function to clear dialog values.

NIFI-8260 - Fixed the cluster replicate request.
Created a new endpoint to handle the cluster replicate request.
Created ProcessGroupUploadEntity.
Renamed positionX and positionY parameters.

NIFI-8260 - Fixed a checkstyle error.
Removed unnecessary httpServletRequest parameter.
Reverted some re-ordering of imports.

NIFI-8260 - Changed the ProcessGroupUploadEntity to be consistent with other Entity and DTO models.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4846.
2021-03-03 14:38:45 +04:00
Nathan Gough c5abf2ba54
NIFI-7127 - Allow choosing a SecureHasher for Flow Fingerprint checking
- Configuration based on Sensitive Properties Algorithm defaults to Argon2
- Added SensitiveValueEncoder interface
- Standard implementation uses existing approach with HmacSHA256

This closes #4867

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-02 21:38:30 -06:00
Janosch Woschitz 481046f5be
NIFI-8286 Extended CertificateUtils to allow parsing of CNs conforming to RFC5280
This closes #4866

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-02 12:33:18 -06:00
Mark Payne 91fed2a993 Fixed minor typo in docs 2021-03-02 09:17:44 -05:00
Matthew Burgess 0d473bacf6
NIFI-8270: Fixed issue to show property dependent on default value of another
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4860.
2021-03-02 12:41:50 +04:00
Peter Turcsanyi 7876f412fb
NIFI-8258: Add support for Service Principal authentication in ADLS processors
- Removed Expression Language support indicators from sensitive properties

This closes #4843

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-01 22:34:59 -06:00
Pierre Villard ea8727a278
NIFI-8274 - add EL consideration in XXEValidator
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4859
2021-03-01 16:34:24 -05:00
exceptionfactory bbd37b8db7
NIFI-8264 Replaced commons-logging with jcl-over-slf4j in framework modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4848
2021-03-01 15:18:42 -05:00
Peter Gyori 4a27b23b1f
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4847.
2021-03-01 20:14:34 +04:00
Peter Gyori a54f739229 NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
NIFI-7969: Documentation update
Clarified that the Force Types From Schema property applies to the data read, whereas the Strict Type Checking property applies to the validation.
NIFI-7969: Documentation update - updated the property name in additionalDetails.html

This closes #4825.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-01 16:04:11 +01:00
exceptionfactory 5608f4389a NIFI-7668 Implemented support for additional AEAD property encryption methods
- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor

NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor

NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor

NIFI-7668 Added support for bcrypt secure hashing algorithm

NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4809.
2021-02-25 17:00:32 -05:00
Mark Payne 99fe548e6b NIFI-8113: Minor updates to admin guide to clarify documentation of some new properties 2021-02-25 16:08:07 -05:00
Bence Simon 7730777d66 NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-25 15:59:01 -05:00
Mark Payne 82eedea659
NIFI-8261: When gathering the states of affected components, make sure that we properly obtain the state of Input Ports and Output Ports
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4844.
2021-02-25 22:21:03 +04:00
Koji Kawamura 68d38dd0a6
NIFI-6752 Add ASN.1 RecordReader
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.

NIFI-6752 Removed obsolete TODOs.

NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).

NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.

NIFI-6752 ASN.1 reader - Updated ASN1.xml template.

NIFI-6752 ASN.1 reader - Updated versions.

NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.

NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4577
2021-02-25 12:58:05 -05:00
Matthew Burgess 29cc3886fc NIFI-8259: Infer CSV field types as strings if no records are present 2021-02-25 12:45:12 -05:00
Peter Turcsanyi 93b1a05dc3
NIFI-8030: Added property to ReportLineageToAtlas to configure file/directory level logging of [hd]fs_path entities to Atlas
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4813.
2021-02-25 18:37:19 +04:00
Matthew Burgess 34fc94454f NIFI-8249: Fixed issue with error during multiple FlowFile results in ExecuteSQL processors 2021-02-24 10:10:32 -05:00
Stan Antyufeev 2322b2cddf NIFI-6113 Refactoring GetHDFSFileInfo to remove instance level variable to make it stateless.
This closes #4837.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-24 13:15:12 +01:00
Matthew Burgess 2dedd8bf0f
NIFI-8253: Restore call to session.commit() in GenerateTableFetch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4840.
2021-02-24 15:57:16 +04:00
Peter Turcsanyi f101a2bba5
NIFI-8198: ConsumeAMQP detects if Queue is deleted on server
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4805.
2021-02-24 12:04:52 +04:00
Christian Gumpert f2555f27f1
NIFI-8119: properly free database resources when done with the processing of BLOBs
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4791
2021-02-23 16:47:32 -05:00
Peter Gyori d049bca794
NIFI-8205: Documentation improvements for the Wait processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4808.
2021-02-23 21:22:19 +04:00
Mark Payne 528fce2407
NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches

NIFI-7646: Removed TODO and unused Jackson dependency

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4818
2021-02-23 10:57:54 -05:00