Commit Graph

1887 Commits

Author SHA1 Message Date
Joe Witt 77c353219b
NIFI-8421 This closes #4996. increase commit frequency to ensure we commit often during these tests to minimize race conditions
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-19 07:03:58 -07:00
Pierre Villard 4fb86d79e4
NIFI-8428 - increased actions column width for controller services
This closes #5000

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-14 15:56:06 -05:00
Bryan Bende 1ab4cf5307
NIFI-8419 This closes #4994. Fix issues with contoller services during parameter context update in a secure instance
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-04-12 20:35:44 -07:00
Joe Gresock 2298953f90
NIFI-8410: Enabling TLS in RedisStateProvider
This closes #4990

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-12 11:38:47 -05:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
sjyang18 365dde2d11 NIFI-7573 Added Azure Active Directory User Group Provider
This closes #4367

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-09 15:06:14 -07:00
Mark Payne 14e6dc3dc6
NIFI-8405: Added debug logging around how long it takes to establish connections/query dns/read and write headers and body when replication requests; added additional timing around Ranger audits and authorizations and monitoring of long-running tasks because those run often and frequently show up in the logs at the same time as the long requests
This closes #4983

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-08 15:41:44 -05:00
Matthew Burgess 33ec8c8427 NIFI-8400: Use longs in SystemDiagnostics, add null checks
This closes #4980.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 10:06:26 +02:00
Mohammed Nadeem 77a73a603b NIFI-8394 Fix NPE error when NiFi connects to external zookeeper insecurely
This closes #4974

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-07 11:23:25 -07:00
Nathan Gough 9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Mark Payne b79987918a
NIFI-8380: Allow for an extensions.directory property to specify where to place downloaded files. Also fixed an issue that was encountered, when a Source Processor is scheduled for Primary Node Only but more than 1 task is set. In that case, even though only a single task will should be scheduled, an Exception was getting thrown because @OnScheduled methods of Processors were still called. To avoid this, moved the initialization of the dataflow outside of the creation of the dataflow so that initialization can be triggered only when appropriate.
NIFI-8380: Removed requirement in validation for working directory and extensions directory to exist; removed auto-creation of directories in validation

NIFI-8380: Fixed a few thrading bugs, so that if we have multiple threads trying to download/unpack extensions, we properly synchronize the unpacking and unpack into the correct sub-directory under the working directory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4950.
2021-04-07 13:28:12 +02:00
Jose Luis Pedrosa f21c1be60f
NIFI-8374 Refactor readOnly to accessMode in ApiModelProperty usages
This closes #4945

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-04-02 14:22:07 -04:00
Rob Fellows 87b561f026
[NIFI-8387] - Use the bulletins from the referencing components rathe… (#4969)
* [NIFI-8387] - Use the bulletins from the referencing components rather than making extraneous http calls to get them.

* Moving the spinner next to the Parameter/Variable lables

This closes #4969
2021-04-02 09:59:34 -04:00
Mark Payne 7db1b8d564
NIFI-8386: Ensure that we set (and merge) bulletins when creating AffectedComponent entities and ControllerService Reference entities
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes

This closes #4955
2021-04-01 14:11:32 -04:00
Eric Olson bff3e94c01
NIFI-7172 Trim trailing whitespace from NiFi properties
This closes #4854

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-25 15:40:35 -05:00
exceptionfactory 1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Peter Turcsanyi 4473d23ccd
NIFI-8356: Add unit test for LongRunningTaskMonitor.
This closes #4925

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-24 10:29:59 -05:00
Mark Bean 1719e36165 NIFI-8348: upgrade jersey version to one fully compatible with Java 11 2021-03-24 09:18:31 -04:00
markap14 321e979673
NIFI-8353: When receiving data via load-balanced connection, throw an Exception (resulting in an ABORT_TRANSACTION status code) when attempting to add received FlowFiles to the FlowFile queue, if the node is not currently connected to cluster. In this case, ensure that we remove the received FlowFiles from the FlowFile Repository and emit a DROP event to the repository (with appropriate details) to coincide with the already-committed Provenance events. Also ensure that when a node is disconnected that we don't keep sending to that node until it reconnects. While testing the fixes via LoadBalancedQueueIT, also noticed that unit test logging was not working properly in nifi-framework-core because of the (erroneous) dependency on logback-class; removed this dependency and updated unit tests that were created that depended on its existence. (#4924) 2021-03-23 14:17:54 -04:00
Nathan Gough 07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00
Bence Simon 4d68509212 NIFI-8308 Remove unnecessary field adjustments from QuestDB row creation
This closes #4888.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-19 17:55:11 +01:00
s9514171 fedeafb43f NIFI-8300 Make non-source processors invalid if scheduled for Primary Node only
This closes #4913.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-03-19 10:48:29 -04:00
Peter Turcsanyi 105a76b7b7
NIFI-8314: Add controller-level bulletin message for long-running tasks.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4906.
2021-03-18 02:40:39 +04:00
Mark Payne cf4e966d91
NIFI-8337: This closes #4910. Fixed bug in StandardProcessSession where the session didn't account for FlowFile's contentClaimOffset when seeking to the appropriate location in the stream.
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-17 14:12:35 -07:00
Peter Turcsanyi f00f0ad269 NIFI-8314: Generate warning for long-running processor tasks 2021-03-16 14:36:51 -04:00
Bence Simon 57cca88eea NIFI-8289 Refine QuestDB status repository rollover and add time zone support
This closes #4883

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-12 09:10:02 -08:00
Mark Payne a9b435e8d3
NIFI-8307: When StandardControllerServiceProvider.enableControllerServiceAndDependencies is called, ensure that it recursively calls itself for any dependent controller services. Updated toString() of ControllerServiceNode and passed itself to the ServiceStateTransition in order to facilitate better logging
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4882.
2021-03-10 20:13:42 +04:00
Bryan Bende 97536e9070
NIFI-8303 Exclude com.io7m.xom:xom when depending on spring-security-saml
Update LICENSE files to account for esapi BSD license

This closes #4878

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-09 14:39:46 -07:00
Mark Payne f09f2a6c85
NIFI-8212: When instantiating a temp component, ensure that nar class loader is set as the context class loader
This closes #4877

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-09 11:43:07 -06:00
Mark Payne 79121a142e
NIFI-6518: If processor is scheduled to run on Primary Node Only, and 'this' node is not primary node, yield for the bored yield duration.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4875.
2021-03-09 11:51:52 +04:00
Mark Payne 8c0f0a40cc
NIFI-8301: When we create a PreparedQuery in the StandardProcessContext, we do so with the effective property values. As a result, the Parameters have already been evaluated, so we need to use Query.prepareWithParametersPreEvaluated instead of Query.prepare
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4874.
2021-03-09 11:48:44 +04:00
exceptionfactory 98ab42aebc NIFI-8302 Corrected FingerprintFactory to use decrypt() method removing wrapping
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4876.
2021-03-08 22:16:21 -05:00
Mark Payne b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
mtien b9a0991157
NIFI-8260 [WIP] Created Upload File front end and server side logic.
NIFI-8260 [WIP] Fixed server side logic to upload a flow file. Cleaned up the front end logic.
NIFI-8260 [WIP] Finished the server side upload logic.
Added a client ID parameter to the endpoint.
Added JSON parsing error response.
Fixed the client side file form to reset after submit.
Fixed the canvas to instantly update and show the process group after submitting the file.
Changed the Add Processor Group dialog UI based on design notes.
Changed the Upload File link to an icon and moved to the process group name input.
Changed the Registry Import link to say 'Import from Registry' and moved to the bottom of the dialog.
Display the filename when a file is selected.

NIFI-8260 [WIP] Added a cancel file button to the Process Group dialog.
Fixed some CSS styles.

NIFI-8260 - Removed accessing the snapshot metadata to avoid an NPE.
Added a title attribute to the html of the dialog file cancel button.

NIFI-8260 - Disabled the dialog 'Add' button.
Revised based on PR feedback.
Refactored the upload file endpoint and client side filename extraction methods.
Fixed some CSS.
Reverted some unnecessary changes.

NIFI-8260 - Revised based on PR feedback.
Refactored uploadProcessGroup.
Fixed some exception handling.
Hid the Upload File button when grouping components.
Refactored nf-ng-group-component.js replacing jquery selectors with variables.
Extracted the resetValues function to clear dialog values.

NIFI-8260 - Fixed the cluster replicate request.
Created a new endpoint to handle the cluster replicate request.
Created ProcessGroupUploadEntity.
Renamed positionX and positionY parameters.

NIFI-8260 - Fixed a checkstyle error.
Removed unnecessary httpServletRequest parameter.
Reverted some re-ordering of imports.

NIFI-8260 - Changed the ProcessGroupUploadEntity to be consistent with other Entity and DTO models.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4846.
2021-03-03 14:38:45 +04:00
Nathan Gough c5abf2ba54
NIFI-7127 - Allow choosing a SecureHasher for Flow Fingerprint checking
- Configuration based on Sensitive Properties Algorithm defaults to Argon2
- Added SensitiveValueEncoder interface
- Standard implementation uses existing approach with HmacSHA256

This closes #4867

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-02 21:38:30 -06:00
Matthew Burgess 0d473bacf6
NIFI-8270: Fixed issue to show property dependent on default value of another
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4860.
2021-03-02 12:41:50 +04:00
exceptionfactory bbd37b8db7
NIFI-8264 Replaced commons-logging with jcl-over-slf4j in framework modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4848
2021-03-01 15:18:42 -05:00
exceptionfactory 5608f4389a NIFI-7668 Implemented support for additional AEAD property encryption methods
- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor

NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor

NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor

NIFI-7668 Added support for bcrypt secure hashing algorithm

NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4809.
2021-02-25 17:00:32 -05:00
Bence Simon 7730777d66 NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-25 15:59:01 -05:00
Mark Payne 82eedea659
NIFI-8261: When gathering the states of affected components, make sure that we properly obtain the state of Input Ports and Output Ports
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4844.
2021-02-25 22:21:03 +04:00
Mark Payne 528fce2407
NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches

NIFI-7646: Removed TODO and unused Jackson dependency

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4818
2021-02-23 10:57:54 -05:00
exceptionfactory 418e2cc2cb NIFI-8132 Replaced framework uses of MD5 with SHA-256
NIFI-8132 Added FileDigestUtils in nifi-nar-utils to avoid dependency on nifi-utils

NIFI-8132 Removed unused imports from NarUnpacker

NIFI-8132 Removed MD5 references from FileUtils documentation

NIFI-8132 Replaced StringBuffer with StringBuilder and made new DigestUtils classes final

NIFI-8132 Replaced Collections.sort() with Stream.sorted()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4788.
2021-02-22 17:52:24 -05:00
Bryan Bende ebef823cb9
NIFI-8241 This closes #4833. Set the SAML context provider to use the EmptyStorageFactory
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-22 08:53:32 -07:00
Lehel Boér 80da0cf47d
NIFI-7496: Regex pattern changed to match dashes in filters.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4811.
2021-02-22 12:59:03 +04:00
Bryan Bende 6ed496c714
NIFI-8242 Ensure NiFi URL for redirects has a trailing slash in AccessResource
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4834.
2021-02-22 12:48:42 +04:00
Mark Payne cd6418cdaf
NIFI-8238: When FlowFile Concurrency/Outbound Policy is changed on child group, and child group is directly under version control, ensure that those fields are maintained on the generated VersionedProcessGroup when nested components are stripped out
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4831.
2021-02-19 14:44:42 +04:00
exceptionfactory f532b3ae1d
NIFI-5623 Upgraded OkHttp3 to 4.9.1 and updated unit tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4826.
2021-02-19 14:42:16 +04:00
Tamas Palfy aa726040c5 NIFI-8188 - Add 'Run Once' for processors in context menu.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-18 14:51:28 -05:00
Joe Witt 88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt 4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00