Commit Graph

6471 Commits

Author SHA1 Message Date
Mark Payne 8c0f0a40cc
NIFI-8301: When we create a PreparedQuery in the StandardProcessContext, we do so with the effective property values. As a result, the Parameters have already been evaluated, so we need to use Query.prepareWithParametersPreEvaluated instead of Query.prepare
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4874.
2021-03-09 11:48:44 +04:00
Mark Payne dc94ef568a
NIFI-8299: Because we now iterate over stored events in an unordered map, we need to update maxEventId to Math.max( <current value>, <next value> ) instead of just assigning the next value.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4873.
2021-03-09 11:44:51 +04:00
exceptionfactory 98ab42aebc NIFI-8302 Corrected FingerprintFactory to use decrypt() method removing wrapping
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4876.
2021-03-08 22:16:21 -05:00
Mark Payne b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
Matthew Burgess 0965523bab NIFI-8266: Fix NPE when validating supported properties against validation context
- Removed schema properties from FreeFormTextRecordSetWriter causing validation issues

This closes #4851.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-03-08 11:30:13 -05:00
Pierre Villard c5b5be89a2
NIFI-8267 - Fix for HWX Schema Registry to use integers (#4850) 2021-03-08 09:12:37 -05:00
tpalfy f2a03fca2a
NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment. (#4868)
* NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment.

* NIFI-8285 Improve exception handling.
2021-03-08 09:10:12 -05:00
Peter Turcsanyi 9266212117
NIFI-8284: Removed diff merge leftover from the assembly NOTICE file (#4865) 2021-03-08 09:08:34 -05:00
Matthew Burgess 29ebec071e
NIFI-4974: Add try/catch around blob.free() to support JDBC spec
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4861.
2021-03-07 18:51:07 +04:00
Matthew Burgess ff93ec42c3
NIFI-8280: Remove Lumberjack NAR from assembly
This closes #4864

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-03-04 19:55:44 -05:00
Mark Payne e2e137fced
NIFI-8291: Support escaping unicode characters for fields that previously allowed escaping only of \t, \n\, \r
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4871.
2021-03-03 21:26:18 +04:00
mtien b9a0991157
NIFI-8260 [WIP] Created Upload File front end and server side logic.
NIFI-8260 [WIP] Fixed server side logic to upload a flow file. Cleaned up the front end logic.
NIFI-8260 [WIP] Finished the server side upload logic.
Added a client ID parameter to the endpoint.
Added JSON parsing error response.
Fixed the client side file form to reset after submit.
Fixed the canvas to instantly update and show the process group after submitting the file.
Changed the Add Processor Group dialog UI based on design notes.
Changed the Upload File link to an icon and moved to the process group name input.
Changed the Registry Import link to say 'Import from Registry' and moved to the bottom of the dialog.
Display the filename when a file is selected.

NIFI-8260 [WIP] Added a cancel file button to the Process Group dialog.
Fixed some CSS styles.

NIFI-8260 - Removed accessing the snapshot metadata to avoid an NPE.
Added a title attribute to the html of the dialog file cancel button.

NIFI-8260 - Disabled the dialog 'Add' button.
Revised based on PR feedback.
Refactored the upload file endpoint and client side filename extraction methods.
Fixed some CSS.
Reverted some unnecessary changes.

NIFI-8260 - Revised based on PR feedback.
Refactored uploadProcessGroup.
Fixed some exception handling.
Hid the Upload File button when grouping components.
Refactored nf-ng-group-component.js replacing jquery selectors with variables.
Extracted the resetValues function to clear dialog values.

NIFI-8260 - Fixed the cluster replicate request.
Created a new endpoint to handle the cluster replicate request.
Created ProcessGroupUploadEntity.
Renamed positionX and positionY parameters.

NIFI-8260 - Fixed a checkstyle error.
Removed unnecessary httpServletRequest parameter.
Reverted some re-ordering of imports.

NIFI-8260 - Changed the ProcessGroupUploadEntity to be consistent with other Entity and DTO models.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4846.
2021-03-03 14:38:45 +04:00
Nathan Gough c5abf2ba54
NIFI-7127 - Allow choosing a SecureHasher for Flow Fingerprint checking
- Configuration based on Sensitive Properties Algorithm defaults to Argon2
- Added SensitiveValueEncoder interface
- Standard implementation uses existing approach with HmacSHA256

This closes #4867

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-02 21:38:30 -06:00
Janosch Woschitz 481046f5be
NIFI-8286 Extended CertificateUtils to allow parsing of CNs conforming to RFC5280
This closes #4866

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-02 12:33:18 -06:00
Mark Payne 91fed2a993 Fixed minor typo in docs 2021-03-02 09:17:44 -05:00
Matthew Burgess 0d473bacf6
NIFI-8270: Fixed issue to show property dependent on default value of another
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4860.
2021-03-02 12:41:50 +04:00
Peter Turcsanyi 7876f412fb
NIFI-8258: Add support for Service Principal authentication in ADLS processors
- Removed Expression Language support indicators from sensitive properties

This closes #4843

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-01 22:34:59 -06:00
Pierre Villard ea8727a278
NIFI-8274 - add EL consideration in XXEValidator
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4859
2021-03-01 16:34:24 -05:00
exceptionfactory bbd37b8db7
NIFI-8264 Replaced commons-logging with jcl-over-slf4j in framework modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4848
2021-03-01 15:18:42 -05:00
Peter Gyori 4a27b23b1f
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4847.
2021-03-01 20:14:34 +04:00
Peter Gyori a54f739229 NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
NIFI-7969: Documentation update
Clarified that the Force Types From Schema property applies to the data read, whereas the Strict Type Checking property applies to the validation.
NIFI-7969: Documentation update - updated the property name in additionalDetails.html

This closes #4825.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-01 16:04:11 +01:00
exceptionfactory 5608f4389a NIFI-7668 Implemented support for additional AEAD property encryption methods
- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor

NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor

NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor

NIFI-7668 Added support for bcrypt secure hashing algorithm

NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4809.
2021-02-25 17:00:32 -05:00
Mark Payne 99fe548e6b NIFI-8113: Minor updates to admin guide to clarify documentation of some new properties 2021-02-25 16:08:07 -05:00
Bence Simon 7730777d66 NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-25 15:59:01 -05:00
Mark Payne 82eedea659
NIFI-8261: When gathering the states of affected components, make sure that we properly obtain the state of Input Ports and Output Ports
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4844.
2021-02-25 22:21:03 +04:00
Koji Kawamura 68d38dd0a6
NIFI-6752 Add ASN.1 RecordReader
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.

NIFI-6752 Removed obsolete TODOs.

NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).

NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.

NIFI-6752 ASN.1 reader - Updated ASN1.xml template.

NIFI-6752 ASN.1 reader - Updated versions.

NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.

NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4577
2021-02-25 12:58:05 -05:00
Matthew Burgess 29cc3886fc NIFI-8259: Infer CSV field types as strings if no records are present 2021-02-25 12:45:12 -05:00
Peter Turcsanyi 93b1a05dc3
NIFI-8030: Added property to ReportLineageToAtlas to configure file/directory level logging of [hd]fs_path entities to Atlas
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4813.
2021-02-25 18:37:19 +04:00
Matthew Burgess 34fc94454f NIFI-8249: Fixed issue with error during multiple FlowFile results in ExecuteSQL processors 2021-02-24 10:10:32 -05:00
Stan Antyufeev 2322b2cddf NIFI-6113 Refactoring GetHDFSFileInfo to remove instance level variable to make it stateless.
This closes #4837.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-24 13:15:12 +01:00
Matthew Burgess 2dedd8bf0f
NIFI-8253: Restore call to session.commit() in GenerateTableFetch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4840.
2021-02-24 15:57:16 +04:00
Peter Turcsanyi f101a2bba5
NIFI-8198: ConsumeAMQP detects if Queue is deleted on server
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4805.
2021-02-24 12:04:52 +04:00
Christian Gumpert f2555f27f1
NIFI-8119: properly free database resources when done with the processing of BLOBs
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4791
2021-02-23 16:47:32 -05:00
Peter Gyori d049bca794
NIFI-8205: Documentation improvements for the Wait processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4808.
2021-02-23 21:22:19 +04:00
Mark Payne 528fce2407
NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches

NIFI-7646: Removed TODO and unused Jackson dependency

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4818
2021-02-23 10:57:54 -05:00
Matthew Burgess 200c04c6d0
NIFI-8237: This closes #4835. Added missing SQL types to getDataTypeFromSQLTypeValue(), added defensive code
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:08:09 -07:00
Pierre Villard 9101160cbf
NIFI-8250 - This closes #4838. UTF8 encoding for FTP processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:05:32 -07:00
Chris Sampson 3cb26aec72
NIFI-8232 CSV Parsers optionally allow/reject duplicate header names
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4828.
2021-02-23 09:50:18 +04:00
exceptionfactory 418e2cc2cb NIFI-8132 Replaced framework uses of MD5 with SHA-256
NIFI-8132 Added FileDigestUtils in nifi-nar-utils to avoid dependency on nifi-utils

NIFI-8132 Removed unused imports from NarUnpacker

NIFI-8132 Removed MD5 references from FileUtils documentation

NIFI-8132 Replaced StringBuffer with StringBuilder and made new DigestUtils classes final

NIFI-8132 Replaced Collections.sort() with Stream.sorted()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4788.
2021-02-22 17:52:24 -05:00
Bryan Bende ebef823cb9
NIFI-8241 This closes #4833. Set the SAML context provider to use the EmptyStorageFactory
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-22 08:53:32 -07:00
Celso Marques 703c3cfe63
NIFI-5906 - Removed FILE_EXISTS_VALIDATOR on JARs and Files to be added in the session
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4810.
2021-02-22 13:04:03 +04:00
Lehel Boér 80da0cf47d
NIFI-7496: Regex pattern changed to match dashes in filters.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4811.
2021-02-22 12:59:03 +04:00
Matthew Burgess b08d6071dd
NIFI-8224: Add LoggingRecordSink controller service
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4820.
2021-02-22 12:55:53 +04:00
Bryan Bende 6ed496c714
NIFI-8242 Ensure NiFi URL for redirects has a trailing slash in AccessResource
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4834.
2021-02-22 12:48:42 +04:00
exceptionfactory 51e7bdf940
NIFI-3669 Add SSL Support to CaptureChangeMySQL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4594.
2021-02-22 11:29:59 +04:00
Timea Barna 59ad21fbaa NIFI-8200 Modified PutAzureDataLakeStorage to delete temp file if exception was thrown in uploadContent()
This closes #4815

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-02-21 13:35:28 -08:00
Grant Henke 662a35b7de
NIFI-8240: Unify Kudu versions
The Kudu versions for the controller service and processors
currently doesn’t match. This patch unifies the version in
the parent bundle pom to ensure they match going forward.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4832.
2021-02-19 20:42:30 +04:00
Mark Payne cd6418cdaf
NIFI-8238: When FlowFile Concurrency/Outbound Policy is changed on child group, and child group is directly under version control, ensure that those fields are maintained on the generated VersionedProcessGroup when nested components are stripped out
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4831.
2021-02-19 14:44:42 +04:00
exceptionfactory f532b3ae1d
NIFI-5623 Upgraded OkHttp3 to 4.9.1 and updated unit tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4826.
2021-02-19 14:42:16 +04:00
Tamas Palfy aa726040c5 NIFI-8188 - Add 'Run Once' for processors in context menu.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-18 14:51:28 -05:00