Commit Graph

4577 Commits

Author SHA1 Message Date
Matthew Burgess 33ec8c8427 NIFI-8400: Use longs in SystemDiagnostics, add null checks
This closes #4980.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-08 10:06:26 +02:00
Mohammed Nadeem 77a73a603b NIFI-8394 Fix NPE error when NiFi connects to external zookeeper insecurely
This closes #4974

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-04-07 11:23:25 -07:00
Nathan Gough 9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Mark Payne b79987918a
NIFI-8380: Allow for an extensions.directory property to specify where to place downloaded files. Also fixed an issue that was encountered, when a Source Processor is scheduled for Primary Node Only but more than 1 task is set. In that case, even though only a single task will should be scheduled, an Exception was getting thrown because @OnScheduled methods of Processors were still called. To avoid this, moved the initialization of the dataflow outside of the creation of the dataflow so that initialization can be triggered only when appropriate.
NIFI-8380: Removed requirement in validation for working directory and extensions directory to exist; removed auto-creation of directories in validation

NIFI-8380: Fixed a few thrading bugs, so that if we have multiple threads trying to download/unpack extensions, we properly synchronize the unpacking and unpack into the correct sub-directory under the working directory

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4950.
2021-04-07 13:28:12 +02:00
Pierre Villard f4ad658fae NIFI-8389 - Grok Reader - raw data strategy when no match
added allowable value in property
added docs in additionalDetails

This closes #4966.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-04-07 09:11:44 +02:00
Peter Turcsanyi 4b852ba7c8
NIFI-8390: Handle HBase namespaces in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4977.
2021-04-07 08:57:29 +02:00
Jose Luis Pedrosa fa3da2aa1c
There was some minor changes required to stop using deprecated APIs byu accumulo, inspecting the deprecated makes the fixes really simple
* writerFactory.createWriter receives an extra parameteres with a map, in the implementation it sends an empty one if not passed.
* tests were relying on  getConnector, that is replaced by createAccumuloClient

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4942.
2021-04-06 11:10:01 +02:00
exceptionfactory 64b12176b2
NIFI-8388 Upgrade Hazelcast from 4.0.1 to 4.2
- Removes dependency on Guava using ByteBuffer methods for serializing revision

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4967.
2021-04-05 11:46:32 +02:00
Jose Luis Pedrosa f21c1be60f
NIFI-8374 Refactor readOnly to accessMode in ApiModelProperty usages
This closes #4945

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2021-04-02 14:22:07 -04:00
Rob Fellows 87b561f026
[NIFI-8387] - Use the bulletins from the referencing components rathe… (#4969)
* [NIFI-8387] - Use the bulletins from the referencing components rather than making extraneous http calls to get them.

* Moving the spinner next to the Parameter/Variable lables

This closes #4969
2021-04-02 09:59:34 -04:00
Mark Payne 7db1b8d564
NIFI-8386: Ensure that we set (and merge) bulletins when creating AffectedComponent entities and ControllerService Reference entities
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes

This closes #4955
2021-04-01 14:11:32 -04:00
Mark Payne 2edf5514b7 NIFI-8344: If configured to tail a file for some period of time post-rollover, ensure that we only consume lines that are fully written (i.e., end with a newline). Once we stop tailing that file post-rollover, consume any data from that file that has not yet been consumed, up to the end of the file, even if there is no newline.
This closes #4937.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-31 13:40:08 +02:00
exceptionfactory 89dda0b6a4
NIFI-8364 This closes #4953. Set static date for TestQuerySolr
- Static date for TestQuerySolr avoids the potential for missing values in facet range queries

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 16:05:04 -07:00
exceptionfactory 02ea103c6b
NIFI-8379 This closes #4949. Improved reliability and performance of DBCPServiceTest
- Removed unreliable H2 network connection test methods
- Removed duplicate and ignored test methods
- Reduced duplication of setting common service properties

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-30 07:24:35 -07:00
Matthew Burgess a5dbf56114
NIFI-8368: If decimal scale > precision, set precision = scale
This closes #4938

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-29 10:53:46 -05:00
Eric Olson bff3e94c01
NIFI-7172 Trim trailing whitespace from NiFi properties
This closes #4854

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-25 15:40:35 -05:00
exceptionfactory 1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Florimond Manca 66e2741871
NIFI-5535 NIFI-4713 This closes #4858. Change metric tagging in DataDogReportingTask
Remove low-value default tags

Send processor name as tag instead of as part of metric name

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-25 09:59:59 -07:00
exceptionfactory 54d2222fb0
NIFI-8364 This closes #4935. Refactored TestQuerySolr and TestGetSolr to reuse embedded SolrClient
- Replaced Gson with Jackson for parsing JSON
- Corrected assertion argument ordering
- Simplified relative path determination for EmbeddedSolrServerFactory
- Replaced SimpleDateFormat with java.time.Instant parsing and formatting

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-25 09:33:19 -07:00
exceptionfactory 36e0187648 NIFI-8347 Set Thread Context ClassLoader for provided jetty-server classes
- Updated unit test with WebSocket connect method

NIFI-8347 Replaced init() method with ServletContextHandler.setClassLoader()

This closes #4918.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-25 15:51:22 +01:00
Timea Barna 9df717f69b
NIFI-8283 Value handling in ScanAccumulo processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4931.
2021-03-25 13:00:04 +01:00
Hsin-Ying Lee 0f28702b47
NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file… (#4923)
* NIFI-8354 ExecuteStreamCommand processor doesn't delete the temp file if the process start failed
* NIFI-8354 Record the log when delete file failed


This closes #4923

Signed-off-by: Otto Fowler <otto@apache.org>
2021-03-24 12:26:56 -04:00
Peter Turcsanyi 4473d23ccd
NIFI-8356: Add unit test for LongRunningTaskMonitor.
This closes #4925

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-24 10:29:59 -05:00
Mark Payne 74ea3840ac NIFI-8357: Updated Kafka 2.6 processors to automatically handle recreating Consumer Lease objects when an existing one is poisoned, even if using statically assigned partitions
This closes #4926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-24 14:59:13 +01:00
Mark Payne 2f08d1f466 NIFI-8357: Updated Kafka 2.0 processors to automatically handle recreating Consumer Lease objects when an existing one is poisoned, even if using statically assigned partitions
This closes #4926.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-24 14:58:51 +01:00
Dmitry Ibragimov 057b4af482
NIFI-8296 - Use API to retrieve all subjects associated with a schema id for Confluent Schema Registry v5.4.0+
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4872.
2021-03-24 14:23:41 +01:00
Mark Bean 1719e36165 NIFI-8348: upgrade jersey version to one fully compatible with Java 11 2021-03-24 09:18:31 -04:00
Joe Witt 7fe8858d22
NIFI-8359 resolving unused import 2021-03-23 14:30:28 -07:00
Joe Witt 788abe9eb3 NIFI-8359 removing test that cannot be reliable. Random data doesn't guarantee it will be detected as octet-stream and there is no obvious 'will never match' set of bytes so this test is not reliable or valuable 2021-03-23 17:18:54 -04:00
Denes Arvay 22fcb594d9 NIFI-8346 Route to failure in PutAzureBlobStorage for auth, IO exceptions
This closes #4917

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-23 11:28:04 -07:00
markap14 321e979673
NIFI-8353: When receiving data via load-balanced connection, throw an Exception (resulting in an ABORT_TRANSACTION status code) when attempting to add received FlowFiles to the FlowFile queue, if the node is not currently connected to cluster. In this case, ensure that we remove the received FlowFiles from the FlowFile Repository and emit a DROP event to the repository (with appropriate details) to coincide with the already-committed Provenance events. Also ensure that when a node is disconnected that we don't keep sending to that node until it reconnects. While testing the fixes via LoadBalancedQueueIT, also noticed that unit test logging was not working properly in nifi-framework-core because of the (erroneous) dependency on logback-class; removed this dependency and updated unit tests that were created that depended on its existence. (#4924) 2021-03-23 14:17:54 -04:00
exceptionfactory e505e8b42d
NIFI-8352 This closes #4922. Changed assertServerConnections to loop and sleep while evaluating
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-23 07:15:48 -07:00
Mark Payne e1b9548ab6 NIFI-8344: Introduced new Rollover Tail Period property
This closes #4916.

Signed-off-by: Tamas Palfy <tamas.bertalan.palfy@gmail.com>
2021-03-22 18:22:00 +01:00
Nathan Gough 07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00
exceptionfactory 0e659981d4 NIFI-8298 Refactored Kerberos and Socket classes from security-utils to new modules
- Created nifi-security-socket-ssl
- Created nifi-security-kerberos
- Removed nifi-security-utils dependency from nifi-processor-utils
- Updated modules to reference new dependencies
- Eliminated unnecessary transitive dependencies on bcprov-jdk15on from over 30 modules

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4881.
2021-03-19 13:12:19 -04:00
Bence Simon 4d68509212 NIFI-8308 Remove unnecessary field adjustments from QuestDB row creation
This closes #4888.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-19 17:55:11 +01:00
s9514171 fedeafb43f NIFI-8300 Make non-source processors invalid if scheduled for Primary Node only
This closes #4913.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-03-19 10:48:29 -04:00
Matthew Burgess 446401b6f1
NIFI-8328: Allow RestLookupService to use FlowFile attributes in header properties
This closes #4908

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 09:26:00 -05:00
Peter Turcsanyi 105a76b7b7
NIFI-8314: Add controller-level bulletin message for long-running tasks.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4906.
2021-03-18 02:40:39 +04:00
exceptionfactory a7ba5bb4f8
NIFI-8319 Added AES/CBC/NoPadding for decryption in EncryptContent Processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4894.
2021-03-18 01:40:18 +04:00
exceptionfactory 8202bffc98
NIFI-8304 This closes #4907. Replaced HttpsURLConnection with OkHttpClient in TestListenHTTP
NIFI-8304 Updated TestPutTCP to shutdown server before checking connections
NIFI-8304 Changed TestListenTCP to send messages in one byte array
NIFI-8304 Added check for expected jdk.tls.disabledAlgorithms
2021-03-17 14:15:25 -07:00
Mark Payne cf4e966d91
NIFI-8337: This closes #4910. Fixed bug in StandardProcessSession where the session didn't account for FlowFile's contentClaimOffset when seeking to the appropriate location in the stream.
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-17 14:12:35 -07:00
Matthew Burgess e16cc9df46
NIFI-8336: Change BULLETINS table bulletinTimestamp column to Long
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4909.
2021-03-17 22:58:25 +04:00
Peter Turcsanyi 04cd418618
NIFI-8312: Support PKCS12 and BCFKS truststores in Atlas reporting task
- Generate ssl-client.xml on NiFi side in order to be able to configure non-JKS truststores.
- Close FileOutputStream in tests to prevent error during clean-up.
- Removed generating Hadoop Credential Store.
- The credential store is not related to Atlas REST API SSL connection but would eliminate a warning from Atlas Kafka client. Removed because it caused test failure on Windows due to missing Hadoop native libraries.

This closes #4893

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-17 13:53:47 -05:00
exceptionfactory be7233d252
NIFI-8304 Changed TestListenTCPRecord to loop until files found
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4904.
2021-03-17 11:41:21 +04:00
exceptionfactory 2ad88bbfff
NIFI-8304 This closes #4900. Improved Socket test reliability for several Processors
- Refactored TestPutTCP to single class
- Improved TestListenRELP
- Improved TestListenTCP
- Improved TestListenUDP
- Improved TestListenTCPRecord
- Changed OnUnscheduled to OnStopped in AbstractListenEventProcessor

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-16 15:30:11 -07:00
Peter Turcsanyi f00f0ad269 NIFI-8314: Generate warning for long-running processor tasks 2021-03-16 14:36:51 -04:00
exceptionfactory b1f513a2ca
NIFI-8304 Refactored InvokeHTTP unit tests using OkHttp MockWebServer
- Replaced Joda Time with java.time for date formatting
- Replaced Guava Files with java.nio.file.Files for cache directory
- Updated PutTCP test server to close connection when testing connection per FlowFile

NIFI-8304 Removed Thread wrapper for TestListenHTTP client requests

NIFI-8304 Disabled InvokeHTTP Connection Pooling for testing

NIFI-8304 Set 60 second timeout for testing TLS connections

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4892.
2021-03-15 21:28:40 +04:00
Bence Simon 57cca88eea NIFI-8289 Refine QuestDB status repository rollover and add time zone support
This closes #4883

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-03-12 09:10:02 -08:00
exceptionfactory 09692c5f69
NIFI-8313 Upgraded net.lingala.zip4j:zip4j to 2.7.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4889.
2021-03-12 12:21:07 +04:00
Otto Fowler 9f0b47af77
NIFI-2702 Support named captures in ExtractText
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4384.
2021-03-12 12:11:09 +04:00
Mark Payne a9b435e8d3
NIFI-8307: When StandardControllerServiceProvider.enableControllerServiceAndDependencies is called, ensure that it recursively calls itself for any dependent controller services. Updated toString() of ControllerServiceNode and passed itself to the ServiceStateTransition in order to facilitate better logging
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4882.
2021-03-10 20:13:42 +04:00
Bryan Bende 97536e9070
NIFI-8303 Exclude com.io7m.xom:xom when depending on spring-security-saml
Update LICENSE files to account for esapi BSD license

This closes #4878

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-03-09 14:39:46 -07:00
Mark Payne f09f2a6c85
NIFI-8212: When instantiating a temp component, ensure that nar class loader is set as the context class loader
This closes #4877

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-09 11:43:07 -06:00
Mark Payne 79121a142e
NIFI-6518: If processor is scheduled to run on Primary Node Only, and 'this' node is not primary node, yield for the bored yield duration.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4875.
2021-03-09 11:51:52 +04:00
Mark Payne 8c0f0a40cc
NIFI-8301: When we create a PreparedQuery in the StandardProcessContext, we do so with the effective property values. As a result, the Parameters have already been evaluated, so we need to use Query.prepareWithParametersPreEvaluated instead of Query.prepare
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4874.
2021-03-09 11:48:44 +04:00
Mark Payne dc94ef568a
NIFI-8299: Because we now iterate over stored events in an unordered map, we need to update maxEventId to Math.max( <current value>, <next value> ) instead of just assigning the next value.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4873.
2021-03-09 11:44:51 +04:00
exceptionfactory 98ab42aebc NIFI-8302 Corrected FingerprintFactory to use decrypt() method removing wrapping
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4876.
2021-03-08 22:16:21 -05:00
Mark Payne b523698534
NIFI-8212: Refactored StandardExtensionDiscoveringManager to avoid using ServiceLoader
Instead, it will look at the ServiceLoader file and read the names of the classes but avoid instantiating all of the objects or loading the classes into memory.
- Updated Doc Generation so that if the documentation for a given NAR already exists, it doesn't delete it and re-generate it. This was necessary because we are no longer instantiating an instance of each component and instead lazily creating the components as necessary.
- Removed stateless version of extension registry because it's no longer necessary

This closes #4852

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-08 12:21:19 -06:00
Matthew Burgess 0965523bab NIFI-8266: Fix NPE when validating supported properties against validation context
- Removed schema properties from FreeFormTextRecordSetWriter causing validation issues

This closes #4851.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-03-08 11:30:13 -05:00
Pierre Villard c5b5be89a2
NIFI-8267 - Fix for HWX Schema Registry to use integers (#4850) 2021-03-08 09:12:37 -05:00
tpalfy f2a03fca2a
NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment. (#4868)
* NIFI-8285 Prevent HBase client services to throw NPE in non-kerberized environment.

* NIFI-8285 Improve exception handling.
2021-03-08 09:10:12 -05:00
Matthew Burgess 29ebec071e
NIFI-4974: Add try/catch around blob.free() to support JDBC spec
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4861.
2021-03-07 18:51:07 +04:00
Mark Payne e2e137fced
NIFI-8291: Support escaping unicode characters for fields that previously allowed escaping only of \t, \n\, \r
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4871.
2021-03-03 21:26:18 +04:00
mtien b9a0991157
NIFI-8260 [WIP] Created Upload File front end and server side logic.
NIFI-8260 [WIP] Fixed server side logic to upload a flow file. Cleaned up the front end logic.
NIFI-8260 [WIP] Finished the server side upload logic.
Added a client ID parameter to the endpoint.
Added JSON parsing error response.
Fixed the client side file form to reset after submit.
Fixed the canvas to instantly update and show the process group after submitting the file.
Changed the Add Processor Group dialog UI based on design notes.
Changed the Upload File link to an icon and moved to the process group name input.
Changed the Registry Import link to say 'Import from Registry' and moved to the bottom of the dialog.
Display the filename when a file is selected.

NIFI-8260 [WIP] Added a cancel file button to the Process Group dialog.
Fixed some CSS styles.

NIFI-8260 - Removed accessing the snapshot metadata to avoid an NPE.
Added a title attribute to the html of the dialog file cancel button.

NIFI-8260 - Disabled the dialog 'Add' button.
Revised based on PR feedback.
Refactored the upload file endpoint and client side filename extraction methods.
Fixed some CSS.
Reverted some unnecessary changes.

NIFI-8260 - Revised based on PR feedback.
Refactored uploadProcessGroup.
Fixed some exception handling.
Hid the Upload File button when grouping components.
Refactored nf-ng-group-component.js replacing jquery selectors with variables.
Extracted the resetValues function to clear dialog values.

NIFI-8260 - Fixed the cluster replicate request.
Created a new endpoint to handle the cluster replicate request.
Created ProcessGroupUploadEntity.
Renamed positionX and positionY parameters.

NIFI-8260 - Fixed a checkstyle error.
Removed unnecessary httpServletRequest parameter.
Reverted some re-ordering of imports.

NIFI-8260 - Changed the ProcessGroupUploadEntity to be consistent with other Entity and DTO models.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4846.
2021-03-03 14:38:45 +04:00
Nathan Gough c5abf2ba54
NIFI-7127 - Allow choosing a SecureHasher for Flow Fingerprint checking
- Configuration based on Sensitive Properties Algorithm defaults to Argon2
- Added SensitiveValueEncoder interface
- Standard implementation uses existing approach with HmacSHA256

This closes #4867

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-02 21:38:30 -06:00
Matthew Burgess 0d473bacf6
NIFI-8270: Fixed issue to show property dependent on default value of another
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4860.
2021-03-02 12:41:50 +04:00
Peter Turcsanyi 7876f412fb
NIFI-8258: Add support for Service Principal authentication in ADLS processors
- Removed Expression Language support indicators from sensitive properties

This closes #4843

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-01 22:34:59 -06:00
Pierre Villard ea8727a278
NIFI-8274 - add EL consideration in XXEValidator
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4859
2021-03-01 16:34:24 -05:00
exceptionfactory bbd37b8db7
NIFI-8264 Replaced commons-logging with jcl-over-slf4j in framework modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4848
2021-03-01 15:18:42 -05:00
Peter Gyori 4a27b23b1f
NIFI-8263: Maximum Thread Pool Size property introduced in ListenHTTP
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4847.
2021-03-01 20:14:34 +04:00
Peter Gyori a54f739229 NIFI-7969: ValidateRecord enhanced with Force Types From Schema property
NIFI-7969: Documentation update
Clarified that the Force Types From Schema property applies to the data read, whereas the Strict Type Checking property applies to the validation.
NIFI-7969: Documentation update - updated the property name in additionalDetails.html

This closes #4825.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-03-01 16:04:11 +01:00
exceptionfactory 5608f4389a NIFI-7668 Implemented support for additional AEAD property encryption methods
- Added support for PBKDF2 and Scrypt property encryption methods in addition to Argon2
- Refactored StringEncryptor class to PropertyEncryptor interface with implementations
- Added PasswordBasedCipherPropertyEncryptor and KeyedCipherPropertyEncryptor
- Replaced direct instantiation of encryptor with PropertyEncryptorFactory
- Refactored applicable unit tests to use mocked PropertyEncryptor

NIFI-7668 Consolidated similar methods to CipherPropertyEncryptor

NIFI-7668 Updated AbstractTimeBasedSchedulingAgent with PropertyEncryptor

NIFI-7668 Added support for bcrypt secure hashing algorithm

NIFI-7668 Updated comments to clarify implementation of bcrypt key derivation

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4809.
2021-02-25 17:00:32 -05:00
Bence Simon 7730777d66 NIFI-8113 Adding persistent status history repository backed by embedded QuestDB
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-25 15:59:01 -05:00
Mark Payne 82eedea659
NIFI-8261: When gathering the states of affected components, make sure that we properly obtain the state of Input Ports and Output Ports
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4844.
2021-02-25 22:21:03 +04:00
Koji Kawamura 68d38dd0a6
NIFI-6752 Add ASN.1 RecordReader
NIFI-6752 Refactored type and value conversion logic. Added support for more types. Added more tests.
Removed 'parent' from 'Recursive'. (Caused issues. The recursive nature is still there as it has a child with the same type).
Updated jasn1 1.11.2 to asn1bean 1.12.0. If an asn field name is a Java reserved keyword, the field gets a trailing "_" but the getter remains normal. In JASN1Utils adjusted logic when looking for the getter.
Added support for inherited types. OctetStrings are converted to Strings instead of byte arrays.
Service takes care of the compilation of the ASN files. Test sources are generated and removed from source control.

NIFI-6752 Removed obsolete TODOs.

NIFI-6752 Updated nifi-asn1-nar version to 1.13.0-SNAPSHOT. Fixed checkstyle violations (unused imports).

NIFI-6752 ASN.1 reader - ASN.1 bundle requires 'include-asn1' profile to be active to be part of assembly.

NIFI-6752 ASN.1 reader - Updated ASN1.xml template.

NIFI-6752 ASN.1 reader - Updated versions.

NIFI-6752 ASN.1 reader - Update example generator. Updated ASN1.xml template. Updated (fixed) nifi-asn1-nar version in pom.xml.

NIFI-6752 ASN.1 reader - Added missing license for ASN1.xml.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4577
2021-02-25 12:58:05 -05:00
Matthew Burgess 29cc3886fc NIFI-8259: Infer CSV field types as strings if no records are present 2021-02-25 12:45:12 -05:00
Peter Turcsanyi 93b1a05dc3
NIFI-8030: Added property to ReportLineageToAtlas to configure file/directory level logging of [hd]fs_path entities to Atlas
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4813.
2021-02-25 18:37:19 +04:00
Matthew Burgess 34fc94454f NIFI-8249: Fixed issue with error during multiple FlowFile results in ExecuteSQL processors 2021-02-24 10:10:32 -05:00
Stan Antyufeev 2322b2cddf NIFI-6113 Refactoring GetHDFSFileInfo to remove instance level variable to make it stateless.
This closes #4837.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-24 13:15:12 +01:00
Matthew Burgess 2dedd8bf0f
NIFI-8253: Restore call to session.commit() in GenerateTableFetch
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4840.
2021-02-24 15:57:16 +04:00
Peter Turcsanyi f101a2bba5
NIFI-8198: ConsumeAMQP detects if Queue is deleted on server
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4805.
2021-02-24 12:04:52 +04:00
Christian Gumpert f2555f27f1
NIFI-8119: properly free database resources when done with the processing of BLOBs
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4791
2021-02-23 16:47:32 -05:00
Peter Gyori d049bca794
NIFI-8205: Documentation improvements for the Wait processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4808.
2021-02-23 21:22:19 +04:00
Mark Payne 528fce2407
NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches

NIFI-7646: Removed TODO and unused Jackson dependency

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4818
2021-02-23 10:57:54 -05:00
Matthew Burgess 200c04c6d0
NIFI-8237: This closes #4835. Added missing SQL types to getDataTypeFromSQLTypeValue(), added defensive code
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:08:09 -07:00
Pierre Villard 9101160cbf
NIFI-8250 - This closes #4838. UTF8 encoding for FTP processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-23 08:05:32 -07:00
Chris Sampson 3cb26aec72
NIFI-8232 CSV Parsers optionally allow/reject duplicate header names
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4828.
2021-02-23 09:50:18 +04:00
exceptionfactory 418e2cc2cb NIFI-8132 Replaced framework uses of MD5 with SHA-256
NIFI-8132 Added FileDigestUtils in nifi-nar-utils to avoid dependency on nifi-utils

NIFI-8132 Removed unused imports from NarUnpacker

NIFI-8132 Removed MD5 references from FileUtils documentation

NIFI-8132 Replaced StringBuffer with StringBuilder and made new DigestUtils classes final

NIFI-8132 Replaced Collections.sort() with Stream.sorted()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4788.
2021-02-22 17:52:24 -05:00
Bryan Bende ebef823cb9
NIFI-8241 This closes #4833. Set the SAML context provider to use the EmptyStorageFactory
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-22 08:53:32 -07:00
Celso Marques 703c3cfe63
NIFI-5906 - Removed FILE_EXISTS_VALIDATOR on JARs and Files to be added in the session
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4810.
2021-02-22 13:04:03 +04:00
Lehel Boér 80da0cf47d
NIFI-7496: Regex pattern changed to match dashes in filters.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4811.
2021-02-22 12:59:03 +04:00
Matthew Burgess b08d6071dd
NIFI-8224: Add LoggingRecordSink controller service
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4820.
2021-02-22 12:55:53 +04:00
Bryan Bende 6ed496c714
NIFI-8242 Ensure NiFi URL for redirects has a trailing slash in AccessResource
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4834.
2021-02-22 12:48:42 +04:00
exceptionfactory 51e7bdf940
NIFI-3669 Add SSL Support to CaptureChangeMySQL
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4594.
2021-02-22 11:29:59 +04:00
Timea Barna 59ad21fbaa NIFI-8200 Modified PutAzureDataLakeStorage to delete temp file if exception was thrown in uploadContent()
This closes #4815

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-02-21 13:35:28 -08:00
Grant Henke 662a35b7de
NIFI-8240: Unify Kudu versions
The Kudu versions for the controller service and processors
currently doesn’t match. This patch unifies the version in
the parent bundle pom to ensure they match going forward.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4832.
2021-02-19 20:42:30 +04:00
Mark Payne cd6418cdaf
NIFI-8238: When FlowFile Concurrency/Outbound Policy is changed on child group, and child group is directly under version control, ensure that those fields are maintained on the generated VersionedProcessGroup when nested components are stripped out
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4831.
2021-02-19 14:44:42 +04:00
exceptionfactory f532b3ae1d
NIFI-5623 Upgraded OkHttp3 to 4.9.1 and updated unit tests
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4826.
2021-02-19 14:42:16 +04:00
Tamas Palfy aa726040c5 NIFI-8188 - Add 'Run Once' for processors in context menu.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-18 14:51:28 -05:00
Zsihovszki Krisztina 6801704846 NIFI-8235 UnsupportedOpertationException in PutAccumuloRecord 2021-02-18 13:12:09 -05:00
Joey Frazee 042b34cb43 NIFI-8231 Clarify behavior of allowed values for Memory Pools in MonitorMemory 2021-02-18 09:58:48 -05:00
Matthew Burgess fd50d6f042 NIFI-3383 Check if usage operations are supported in MonitorMemory
This closes #4790

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-02-16 17:25:28 -06:00
Joe Witt 88fab00e29
NIFI-7873 merging release branch to latest and updating to 1.14.0-SNAPSHOT 2021-02-15 12:09:32 -07:00
Joe Witt 4afb2ba743
NIFI-7873-RC4 prepare for next development iteration 2021-02-15 12:09:31 -07:00
Joe Witt 487280bee9
NIFI-7873-RC4 prepare release nifi-1.13.0-RC4 2021-02-15 12:09:30 -07:00
Pierre Villard a1c82a66a1
NIFI-7361 - increase actions column width for reporting tasks
This closes #4214

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-15 10:42:19 -06:00
Matthew Burgess d08f02428d
NIFI-8223: This closes #4819. Use column datatype in PutDatabaseRecord when calling setObject()
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-10 18:01:43 -07:00
Nathan Gough 8057f8f6c5
NIFI-8221 - Set the default HTTP listening interface to 127.0.0.1.
This closes #4817

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-10 14:46:37 -06:00
Bryan Bende 1d82fb8e01
NIFI-8218 This closes #4816. Use proxy headers when available when getting request values while processing SAML responses
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-02-10 13:34:57 -07:00
Mark Payne d5d520764d
NIFI-8210: When Index Reader/Searcher is used, do not allow the Lucene Index to be deleted until the reader/searcher is finsihed being used and closed.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4812
2021-02-09 15:32:15 -05:00
Peter Gyori b9b131239c NIFI-8215 --wait-for-init in nifi.sh refactored for POSIX sh
This fixes #4814

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-02-09 11:10:15 -08:00
Nathan Gough 4ebeb4e146
NIFI-8181 - Added Disable HTTP/2 property to InvokeHTTP
This closes #4804

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-08 14:53:35 -06:00
Peter Turcsanyi 40d8c41656
NIFI-8193: Fixed PutSQL reordering FlowFiles
Also removed sorting of transferred MockFlowFiles in test framework.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4803
2021-02-05 16:17:10 -05:00
Mark Payne 749d05840b
NIFI-8204, NIFI-7866: Send revision update count in heartbeats. If update count in heartbeat is greater than that of cluster coordinator, request that node reconnect to get most up-to-date revisions. Cannot check exact equality, as the values may change between the time a heartbeat is created and the time the cluster coordinator receives it. However, it should be safe to assume that the revision won't be greater than that of the cluster coordinator. There is a tiny window in which it could be, as the sending node may update its revision, create the heartbeat, send it, and cluster coordinator process it before updating its own revision. However, this window is incredibly small and would only result in the sending node reconnecting, which will resolve itself. Also, when testing this fix, encountered NIFI-7866 and addressed that NullPointerException.
This closes #4806.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-02-05 15:18:44 -05:00
Mark Payne 03fd59eb2f
NIFI-8196: When node is reconnected to cluster, ensure that it re-registers for election of cluster coordinator / primary node. On startup, if cluster coordinator is already registered and is 'this node' then register silently as coordinator and do not join the cluster until there is no Cluster Coordinator or another node is elected. This allows the zookeeper session timeout to elapse.
Signed-off-by: Bryan Bende <bbende@apache.org>
2021-02-05 15:18:07 -05:00
Matthew Burgess b77dbd5030
NIFI-8172: Provide schema name to getPrimaryKeys call in PutDatabaseRecord
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4782.
2021-02-06 00:00:58 +04:00
Mark Bean a9b8635ac9
NIFI-6522: Referencing Components link from Reporting Task Constroller Services links to proper Controller Settings tab
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4794.
2021-02-05 23:53:29 +04:00
Jon Kessler 3f26e54521 NIFI-8194 Updated log level from WARN to DEBUG for various authentication protocols not being configured 2021-02-05 13:20:37 -05:00
Tamas Palfy b55998afc1 NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Time Window
NIFI-8081 Added new Listing Strategy to ListFTP and ListSFTP: Adjusted Time Window. User can specify the time zone or time difference (compared to where NiFi runs) of the system hosting the files and based on the calculates the current time there. Lists files modified before this adjusted current time (and after the last listing).
NIFI-8081 'Time Adjustment' validated not to be set if listing strategy is not 'Adjusted Time Window'. Extracted validator to a separate class. Added more tests. Minor refactor. Typo fix.
NIFI-8081 Improved validation.
NIFI-8081 'Time Adjustment' is not necessary - in fact it can cause problems. SFTP (and usually FTP - which has a more general bug at the moment) returns a timestamp that doesn't really need adjustment. (SFTP in particular returns the an 'epoch' time.) Everything remains the same - the new listing strategy relies on a sliding time window, but without the unnecessary option to adjust for the modification time.
NIFI-8081 Resolved conflicts after rebasing to main.
NIFI-8081 Renamed 'AbstractListProcessor.listByAdjustedSlidingTimeWindow' to 'listByTimeWindow'. Post main rebase correction.
NIFI-8081 Updated user doc for the BY_TIME_WINDOW strategy to warn user on it's reliance of accurate time.

This closes #4721.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-02-04 17:04:37 +01:00
Denes Arvay c1f88ec740 NIFI-8183 TailFile intermittently creates records with ascii NULL after rollover
This closes #4792.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-03 13:09:00 -05:00
mtien 6e1f737c53
NIFI-1355 Implemented new methods in KeyStoreUtils to programmatically-generate certificates, Keystores, and Truststores and return it wrapped in a TLS configuration.
Updated TestInvokeHTTP, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use new Keystore functionality.

NIFI-1355 Refactored and removed unnecessary unit tests in KeyStoreUtilsGroovyTest.

NIFI-1355 Added a password requirement when creating a new truststore.
Handled exception when loading a passwordless truststore type of Bouncy Castle PKCS12.

This closes #4801

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-02-03 08:53:47 -06:00
Peter Gyori 5a8b18b12d NIFI-8123: Added support for --wait-for-init when NiFi started with the 'start' command
This closes #4748.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-02-03 09:40:09 -05:00
Joe Witt d826416217
NIFI-8192 updating Copyright years for nifi things to include 2021
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4802.
2021-02-03 12:56:28 +04:00
Peter Turcsanyi 77eb8af275
NIFI-8187: Fixed PutSQL duplicating provenance events
Also fixed provenance event mocking in test framework.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4796
2021-02-02 19:02:02 -05:00
Bryan Bende e0a8b479fd NIFI-8190 Protect against property that references missing controller service 2021-02-01 15:47:52 -05:00
exceptionfactory b4e213cb2c
NIFI-8186 Excluded bcprov-ext-jdk15on from spring-security-saml2-core dependency (#4793) 2021-02-01 13:21:46 -05:00
Grant Henke 29a1d6badc NIFI-7987: Support ignore operations in the PutKudu processor 2021-02-01 12:10:19 -05:00
Pierre Villard d72bfc76b5
NIFI-8155 - add banner text in page title
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4768
2021-01-29 19:20:34 -05:00
Matthew Burgess 5e84cf6ff9 NIFI-8175: Add WindowsEventLogReader controller service
This closes #4785

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-01-29 15:16:19 -06:00
exceptionfactory 11e9ff3773
NIFI-8178 This closes #4787. Replaced StandardSSLContextService in unit tests with SslContextUtils
- Removed references to StandardSSLContextService from nifi-standard-processors
- Removed TestGetHTTPGroovy and TestPostHTTPGroovy since these are testing deprecated processors
- Optimized HandleHttpRequest, GetHTTP, PostHTTP to use SSLContextService.createContext()
NIFI-8178 Changed TestGetHTTP to ITGetHTTP since GetHTTP is deprecated
NIFI-8178 Changed TestPostHTTP to ITPostHTTP since PostHTTP is deprecated

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 17:48:06 -07:00
exceptionfactory 8c1fe84f62
NIFI-8177 This closes #4786. Added timeout to Sender Pool polling to avoid blocking indefinitely
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-27 15:04:23 -07:00
snorlaxa 04fb1aca47
NIFI-8142: Add "Insert Ignore" option to PutDatabaseRecord
NIFI-8142: Fix Checkstyle error

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4778
2021-01-27 11:50:40 -05:00
Waleed Al Aibani 813ac539b3
NIFI-7263 This closes #4167. Add No Tracking Listing Strategy to ListFile/ListFTP processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 22:13:26 -07:00
Mark Payne 4ff9cddf15
NIFI-8173, NIFI-8174: This closes #4784. Updated Parameter Contexts to allow for unsetting a parameter value / setting it to null. Allow Parameters to make use of Expression Language. Updated docs to illustrated how EL is evaluated
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 21:55:18 -07:00
exceptionfactory abb6ed3128
NIFI-8171 This closes #4779. Upgraded Bouncy Castle libraries to 1.68 and centralized dependency version
NIFI-8171 Increased response and idle timeouts for HTTP unit tests
NIFI-8171 Increased TestServer idle timeout to 45 seconds for HTTP unit tests
NIFI-8171 Adjusted timeout and sleep on TestPutTCPCommon.testPruneSenders
NIFI-8171 Increased TestServer idle timeout to 60 seconds and removed 500ms Thread.sleep() in TestInvokeHttpSSL
NIFI-8171 Optimized OkHttpClientUtils to avoid reading trust store twice during initialization
NIFI-8171 Added static variable for server startup sleep
NIFI-8171 Increased TestInvokeHTTP Connect Timeout and TestListenHTTP Response Timeout to 30 seconds
NIFI-8171 Refactored unit tests for InvokeHTTP and ListenHTTP to optimize SSLContext creation
NIFI-8171 Updated TestListenHTTP for static creation of SSLContext
NIFI-8171 Added started check for ListenHTTP Server in TestListenHTTP
NIFI-8171 Refactored TestPutTCP classes to optimize SSLContext creation
NIFI-8171 Increased TestListenHTTP timeout for server start to 120 seconds and added exception when not connected
NIFI-8171 Increased Connect and Read Timeouts for InvokeHTTP SSL unit tests

Signed-off-by: Joe Witt <joewitt@apache.org>
2021-01-26 21:24:07 -07:00
exceptionfactory 2cdb0fb6a3 NIFI-8088 Removed deprecation warning log for PKCS12 trust stores
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4727.
2021-01-26 16:50:33 -05:00
Peter Turcsanyi 67d06003b7 NIFI-8023: Convert java.sql.Date between UTC/local time zone normalized forms before/after database operations
This closes #4781

Signed-off-by: David Handermann <exceptionfactory@gmail.com>
2021-01-26 14:39:02 -06:00
Bence Simon 0a10557dd5 NIFI-7801 Adding support for HTTP based Splunk put and indexed acknowledgement
This closes #4714.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-26 15:20:20 -05:00
Mark Payne f2a16cd02e NIFI-8163: When counting number of components, we traverse into all Process Groups, but then call findAllRemoteProcessGroups, which is a recursive call, instead of calling getRemoteProcessGroups(). This results in counting the Process Groups many times. So fixed that.
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4775.
2021-01-21 15:09:42 -05:00
Troy Melhase 76648bdc0b NIFI-7356 - Config TLS for embedded ZooKeeper when NiFi TLS enabled.
NIFI-7356 - Addresses PR feedback.

NIFI-7356 - Additional changes from PR feedback.

NIFI-7356 - Adding integration tests for ZooKeeperStateServer for TLS.

NIFI-7356 - TLS + Zookeeper now working with single and quorum. Needs code cleanup, need to fix IT tests and docs.

NIFI-7356 - Fixed up tests and removed some irrelevant ones. Refactored some of ZooKeeperStateServer. Tested successfully with a secure and insecure 3 node NiFi + Quorum.

NIFI-7356 - Checkstyle fixes.

NIFI-7356 - Updated administration guide with embedded ZooKeeper TLS configuration.

NIFI-7356 - Updated the way ZooKeeper TLS properties are set/mapped from NiFi properties.

NIFI-7356 - Updated per review, using NiFiProperties keystore strings, classname for ocnnection factory, adjusted TLS configuration checks in NiFiProperties.

NIFI-7356 - Updated configuration validation logic and added tests.

NIFI-7356 - Codestyle check fixes.

NIFI-7356 - Updated some of the log messages.

NIFI-7356 - Updated as per code review.

NIFI-7356 - Fixed max port number.

NIFI-7356 - Updated admin guide and updated small code issues as per code review.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4753.
2021-01-21 13:47:53 -05:00
Mark Payne 6741317cc4 NIFI-8162: Fixed outdated docs. Self-merging as this is a trivial docs fix.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-21 12:52:30 -05:00
exceptionfactory 8c49f0688b
NIFI-7243 Upgraded com.hierynomus.sshj to 0.30.0 to resolve authentication race condition in version 0.27.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4774.
2021-01-21 21:21:47 +04:00
Mark Payne 525e4105eb NIFI-8095: Created StatelessNiFi Sink Connector and Source Connector. Minor updates to stateless nifi api to accommodate.
Self-merging PR based on comments from @joewitt
2021-01-21 10:56:28 -05:00
Pierre Villard 93a5823f8a NIFI-7890 - Added record support to ConsumeMQTT processor
This closes #4738.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-21 11:16:07 +00:00
Mike Thomsen 953327cdf5
NIFI-8156 Fixed byte handling bug in cassandra.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4771.
2021-01-21 11:23:14 +04:00
Matthew Burgess fb2a8b5820
NIFI-8043: Quote update key column names in PutDatabaseRecord
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4772.
2021-01-21 11:17:13 +04:00
Andrew Lim 27f57e6463
NIFI-8150 Change Download flow to Download flow definition for process groups (#4766) 2021-01-20 16:00:41 -05:00
Mark Payne 803ba882aa
NIFI-8146: Ensure that we close the Connection/Statement/PreparedStatement objects in finally blocks or try-with-resources
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4770
2021-01-20 13:48:57 -05:00
Mark Payne 25ab050ed7
NIFI-8149: Updated Apache Calcite version from 1.21.0 to 1.26.0
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4765.
2021-01-20 13:04:08 +04:00
Pierre Villard 88de03aebb NIFI-7895 - Fix NPE in ConsumeMQTT with truststore only SSL CS
This closes #4587.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-20 00:29:30 +01:00
Mark Payne 930e8d9e0e
NIFI-7698: Added a merge.reason attribute for flowfiles indicating why they were merged by MergeContent. Also updated logs to indicate the reason and added additional documentation for processor
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4764
2021-01-19 17:20:06 -05:00
s9514171 b4ddeb8997 NIFI-8116 The old peers do not be deleted when the URIs of the remote process group are changed
This closes #4745.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-19 16:25:09 -05:00
Mark Payne f02079c2fa
NIFI-8146: Added RecordPath for Statement Type and to specify where to find the data within the incoming Record. Also performed some minor code refactoring to avoid lots of large lambdas, as it results in code that is more readable and more maintainable. Updated some properties to use dependsOn() notation now that this is available.
NIFI-8146: Updated PutDatabaseRecord to avoid using the functional style framework that had previously been used. Doing so resulted in code that was difficult to understand and maintain. Added additional unit tests and improved MockRecordParser so that it could throw configurable types of exceptions

NIFI-8146: Fixed checkstyle issues

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4763
2021-01-19 13:23:54 -05:00
eduardofontes 3e72dfe421
NIFI-8109 - Update MongoDBControllerService to externalize username and password
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4759.
2021-01-19 21:21:44 +04:00
Chris Sampson 40f193098b
NIFI-7726 correct versionedFlowState field name in SiteToSiteStatusReportingTask schema
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4762.
2021-01-19 10:40:38 +04:00
Mark Payne 097edf4f7c
NIFI-8136: Added getState/setState/replaceState/clearState methods to ProcessSession, updated processors to use these methods instead of StateManager version where appropriate
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4757
2021-01-15 10:02:09 -05:00
Matthew Burgess b9076ca26e NIFI-7989: Add Update Field Names and Record Writer to UpdateHiveTable processors
NIFI-7989: Only rewrite records if a field name doesn't match a table column name exactly
NIFI-7989: Rewrite records for created tables if Update Field Names is set

This closes #4750.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-13 23:26:44 +01:00
Nissim Shiman 3cc8d767b3 NIFI-7738 Reverse Provenance Query
This closes #4563.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-13 09:35:05 -05:00
Matthew Burgess 382439c1d0 NIFI-7973: Add default precision and scale properties to remaining SQL-based components
This closes #4682.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-13 08:41:12 -05:00
tlsmith 0b966cdb14
NIFI-7991 Flow Configuration History displays "annotation data not found/available" from "Advanced" changes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4668
2021-01-12 16:09:38 -05:00
Matthew Burgess 2d69179184
NIFI-8138: Add Sensitive Property value to property tooltip
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4752.
2021-01-13 00:44:03 +04:00
exceptionfactory 7d76bcd520 NIFI-8094 Added support for BCFKS Keystore Type
NIFI-8094 Updated Administration Guide to include BCFKS

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4729.
2021-01-12 10:52:49 -05:00
Mike Thomsen 82c84492ce
NIFI-7906 Fixed a bug in how record paths were handled.
NIFI-7906 Fixed bugs in unit tests.

NIFI-7906 Updated a few things based on a code review.

NIFI-7906: Fixed typo in record.count attribute

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4739
2021-01-12 10:32:08 -05:00
exceptionfactory 5f7558cecf NIFI-8069 Set invalid status when Controller Services are enabling
This closes #4710.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-11 13:42:52 -05:00
Peter Turcsanyi 84b561ad33
NIFI-8131: Support aws_s3_v2_directory in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4751.
2021-01-11 21:53:12 +04:00
exceptionfactory d41f2e1d0a NIFI-8121 Updated ListenHTTP with inferred Client Authentication Policy
- Added default property value for automatic determination of Client Authentication Policy based on SSLContextService Trust Store properties
- Added new ClientAuthentication enum with values specific to ListenHTTP

This closes #4749.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-11 18:02:09 +01:00
exceptionfactory a4027e8e77 NIFI-8120 Added RuntimeException handling on HttpContextMap.complete()
NIFI-8120 Renamed exception variable and reordered log statements

This closes #4747.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-08 17:13:10 +00:00
sjyang18 f330078fff NIFI-7924 Add fallback claims for identifying user to OIDC provider
This closes #4630

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2021-01-07 14:57:57 -06:00
exceptionfactory 817f621d6f NIFI-8096 Deprecated ClientAuth references in SSLContextService
- Added SSLContextService.createContext() and refactored referencing components
- Removed references to ClientAuth from SslContextFactory methods

This closes #4737.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-06 16:24:26 -05:00
Mark Payne 783633cac5
NIFI-8021: Fixed bug in ConsumeKafka_2_6 and ConsumeKafkaRecord_2_6 where explicit partition assignment causes issues with more than 1 concurrent task. Also fixed bug that prevented more nifi nodes than partitions because it didn't properly handle empty string for the list of partitions
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4744.
2021-01-06 19:28:09 +01:00
Mark Payne ea80dad16a
NIFI-8021: Fixed bug in ConsumeKafka_2_0 and ConsumeKafkaRecord_2_0 where explicit partition assignment causes issues with more than 1 concurrent task. Also fixed bug that prevented more nifi nodes than partitions because it didn't properly handle empty string for the list of partitions
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4744.
2021-01-06 19:27:33 +01:00
tlsmith 7214dc0f85 NIFI-6242 PutFileTransfer generating incorrect provenance event 2021-01-06 12:10:20 -05:00
Mark Bean 954c09d0c3 NIFI-7387: update ComponentLog API to accept Object varargs instead of array
This closes #4225.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-01-06 11:09:07 -05:00
Pierre Villard 80eb570ee1 NIFI-7894 - ConsumeMQTT - allow EL on Client ID property with shared subscription
This closes #4586.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-06 16:46:17 +01:00
Pierre Villard c9d778a8ee NIFI-7889 - ConsumeMQTT - use offer instead of add
This closes #4578.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-06 14:15:27 +01:00
Joey Frazee 1e227ca643 NIFI-7964 Force PutAzureBlobStorage to stream writes
NIFI-7964 Remove unused import
NIFI-7964 Reverted to using BufferedInputStream along with unmarkable
NIFI-7964 Made UnmarkableInputStream static and added factory method
NIFI-7964 Remove unnecessary factory method from UnmarkableInputStream

This closes #4632.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-01-06 13:47:05 +01:00
exceptionfactory bb178f371b
NIFI-6576 Added Basic Authentication support to ConfluentSchemaRegistry
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4743.
This closes #4508.
This closes #4224.
2021-01-06 09:34:43 +01:00
Peter Turcsanyi 584adc3b91
NIFI-8067 - Fix 1-way SSL in GRPC processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4733.
2021-01-05 11:31:00 +01:00
Otto Fowler ceb9dff3b9
NIFI-7761 Allow HandleHttpRequest to add specified form data to FlowFile attributes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4513.
2021-01-05 09:42:41 +01:00
exceptionfactory e7c6bdad42 NIFI-7937 Added StandardFlowFileMediaType enum to replace string references to FlowFile Media Types 2021-01-04 14:20:05 -05:00
Kevin Barranco 0f667cb4f3 MonitorActivity THRESHOLD to use Variables
When using MonitorActivity, it would be interesting to use variables in "Threshold Duration", which will help maintain Version Control in ProcessGroups (Avoiding "breaking" versions when including the value manually). 

I've included the option expressionLanguageSupported(ExpressionLanguageScope.VARIABLE_REGISTRY), so we can use variables in "Threshold Duration".
2021-01-04 14:14:29 -05:00
tlsmith 112b893bf2 NIFI-7225 FetchSFTP processor routing to not.found error given when Private Key Path property is invalid 2021-01-04 13:41:03 -05:00
Shane Ardell 5c79553af6 NIFI-8062: make tabs functional with keyboard 2021-01-04 11:58:47 -05:00
Matthew Burgess de88ce7a61 NIFI-7906: Implemented RecordSetWriter support for ExecuteGraphQueryRecord
This closes #4704

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-23 12:41:42 -05:00
Denes Arvay 31c5dc5ab4 NIFI-8102 TailFile can skip characters if tailing a file on NFS mount
- Force the TailFile processor to recreate and reposition the reader
  by setting it to null in case of a NulCharacterEncounteredException
- Updated the TestTailFile.testNULContent() to not initialize the
  processor when calling the second run()

Update the reader's position instead of abandoning it.

This closes #4736.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-12-18 14:20:41 -05:00
sjyang18 942030579a NIFI-7386 Added Azure Azurite emulator credentials service
This closes #4286

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-12-16 18:11:06 -06:00
Tamas Palfy 989287adaf
NIFI-8027 In AbstractExecuteSQL make 'SQL Pre-Query' and 'SQL Post-Query' properties support escaping semicolons to allow them be part of statements.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4732
2020-12-16 14:57:59 -05:00
Peter Turcsanyi 02665d7561
NIFI-8092: Fix ITPutS3Object test failure due to Outposts storage class
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4728.
2020-12-16 12:43:24 +01:00
Matthew Burgess f29d6a6046 NIFI-7989: Add support to UpdateHiveTable for creating external tables
NIFI-7989: Add support for creating partitions, quote identifiers
NIFI-7989: Quote table name when getting description

This closes #4697.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-12-15 22:05:53 +01:00
exceptionfactory 28ca7478d6 NIFI-8019 Added TlsPlatform to provide runtime TLS protocol configuration
NIFI-8019 Renamed getDefaultProtocols() to getSupportedProtocols()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4673.
2020-12-14 19:34:58 -05:00
exceptionfactory 5bcb5abc86 NIFI-1930 Updated ListenHTTP to set TLS included protocols based on SSLContextService
NIFI-1930 Removed unused import in SSLContextServiceTest

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4673.
2020-12-14 14:31:08 -05:00
Mark Payne a73cd6a610
NIFI-7856: If a Provenance Event File is ready to be rolled over due to the maximum amount of time having elapsed, avoid rolling over unless there is at least one event written to the event file. Otherwise, we can have multiple RecordWriters / RecordWriterLeases pointing to the same file. This can result in data being overwritten, as well as failing to compress the event file upon rollover. Also added significant DEBUG/TRACE level logging.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4580
2020-12-14 11:49:47 -05:00
Peter Turcsanyi 91f6b42985
NIFI-8085 Use poll(Duration) in ConsumeKafka_2_x processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4725.
2020-12-13 10:23:35 +01:00
Peter Turcsanyi ec3d5f89f0
NIFI-8086: Fix file separator char in PutS3Object multipart state directory path
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4726.
2020-12-13 10:18:32 +01:00
exceptionfactory 7bff64b3cf NIFI-7913 Added getEnabledProtocols() to TlsConfiguration and updated ListenSMTP to set enabled protocols on SSL Sockets
NIFI-7913 Changed order of supported protocols to match existing comments in SSLContextService

This closes #4599

Signed-off-by: Nathan Gough <thenatog@gmail.com>
2020-12-10 11:53:20 -05:00
Mike Thomsen 8e4948322e
NIFI-8074 Switched ReplaceText to use line by line evaluation as its default evaluation mode.
This closes #4711

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-10 10:30:11 -05:00
Matthew Burgess 1f8b4e4779
NIFI-8080: Move script compilation to configurator init method
This closes #4718

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-09 21:05:44 -05:00
Sushil Kumar 1d31434c48 NIFI-8055 Fix validation message class-name for AzureCosmosDBClientService
This closes #4695

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-12-09 17:13:25 -06:00
Peter Turcsanyi 8d2fd39bdb
NIFI-8066: Bump GRPC dependency versions (#4716) 2020-12-09 09:37:01 -05:00
exceptionfactory 0af736dc0b
NIFI-7885 Added Environment Variable to deny LFS access using Hadoop (#4712)
* NIFI-7885 Added Environment Variable to deny LFS access using Hadoop

* NIFI-7885 Changed Hadoop Processor validation to check LFS access
2020-12-09 09:29:42 -05:00
Matthew Burgess 2c0671cf80
NIFI-8080: Compile Jython scripts before evaluating
This closes #4717

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-09 08:57:21 -05:00
exceptionfactory cfbcecc4c6
NIFI-7884 Added and applied Distributed File System permissions (#4713) 2020-12-08 15:56:46 -05:00
Bryan Bende 039a09923a
NIFI-8073 Correcting isBackPressureEnabled in PrometheusMetricsUtil 2020-12-07 10:03:16 -05:00
DataMoTrain d7ca4c920c
NIFI-8073 corrected condition for label isBackPressureEnabled
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-12-07 10:03:04 -05:00
Matthew Burgess 405c393cb5
NIFI-8031: Add UPSERT capability for MySQL in PutDatabaseRecord
This closes #4678

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-07 08:51:07 -05:00
r65535 2f99d6fce4
NIFI-4540 - Added FIFO options to PutSQS
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4705.
2020-12-07 13:11:22 +01:00
Mark Payne 37c2284d04
NIFI-8068: Ensure that when we determine the best of multiple possible types in a UNION that we handle Arrays of Records properly. Also refactored code to be a bit cleaner by extracting blocks of it into appropriately named methods
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4706.
2020-12-04 19:31:23 +01:00
Matthew Burgess 71d5162965
NIFI-6878: Added 'Use statement.type Attribute' to ConvertJSONToSQL
This closes #3893

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-04 07:39:08 -05:00
Mark Payne 29ea872f2c
NIFI-8042: Fixed bug that was escaping Expression Language references for use in a Regular Expression (i.e., Pattern.quote) even though it wasn't being used in a Regular Expression
This closes #4685

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-04 07:30:09 -05:00
Matthew Burgess 04aaf25131 NIFI-1121: Use display name for dependent property documentation 2020-12-03 16:39:46 -05:00
mtien 8055c47a84
NIFI-8058 Fixed a UI error to correctly delete dynamic properties while configuring processors.
Changed to check the length of all unfiltered properties instead of only filtered properties.
Added additional check if descriptor is a dynamic property.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4707
2020-12-03 15:44:56 -05:00
Bryan Bende 312fa8e85e NIFI-8064 Convert TestSecureClientZooKeeperFactory to integration test 2020-12-03 14:46:13 -05:00
Bryan Bende 2b1359a808 NIFI-8060 Addressed review feedback 2020-12-03 14:42:50 -05:00
Bryan Bende 8ac8a2bd1f NIFI-8060 Added minimal VolatileProvenanceRepository to nifi-stateless-engine and remove dependency on nifi-volatile-provenance-repo module 2020-12-03 14:42:50 -05:00
Mike Thomsen b90a6e893d
NIFI-7906 This closes #4701. Updated test case to fix a windows-centric bug.
NIFI-7906 Removed unused test code.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-12-03 07:22:40 -07:00
Levi Lentz c29cced269
NIFI-7906: parameterized graph query
NIFI-7906: addressing PR concerns
NIFI-7906: code styling fixes
NIFI-7906: adding in license information to new files
+ enables processor in META-INF
NIFI-7906: exclude test files from RAT
NIFI-7906: PR refactor to streamline graph response
NIFI-7906: removing ERRORS output
Unused after refactor
Did a few cleanups for the contributor.

This closes #4638

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-01 11:13:10 -05:00
Matthew Burgess 64e3599f05
NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services
This closes #4147

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-01 09:42:14 -05:00
exceptionfactory 1e13b62e78
NIFI-8047 Added documentation for sensitive DBCP properties
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4696
2020-11-30 18:33:32 -05:00
markap14 aaa1452d04
NIFI-8054: Updated ReflectionUtils to use a WeakHashMap for the mapping of annotations to methods with that annotation. This way, the ReflectionUtils class will not hold a reference to Classes that are no longer referenced elsewhere. (#4694) 2020-11-30 16:14:12 -05:00
exceptionfactory fe53f8090d
NIFI-8047 Added support for sensitive dynamic properties in DBCP
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4692
2020-11-30 14:55:20 -05:00
Bence Simon 39f8a008d4 NIFI-8039 Adding properties to ListenTCP in order to allow refine behaviour under higher load; Refining thread pool for better scalability
NIFI-8039 Review findings; refining thread pool to be able to scale down properly when not under load
NIFI-8039 Answers to PR comments

This closes #4689.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-30 10:17:15 +01:00
Mike Thomsen 3625a73d91
NIFI-7821 Added Cassandra-based DMC.
NIFI-7821 Updated configuration documentation.

NIFI-7821 Fixed getAndPutIfAbsent and added int test.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4635
2020-11-29 18:58:57 -05:00
Matthew Burgess fe950131c3 NIFI-8046: Fix issue with ResultSetRecordSet on DB2
This closes #8046.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-27 19:23:44 +01:00
Koji Kawamura 5ea18f39b5
NIFI-8048 PutElasticsearchRecord should retry with 503 response
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4690.
2020-11-26 10:49:52 +01:00
Chris Sampson 124cdbd3fe
NIFI-6403 and NIFI-6404: Elasticsearch 7 support
Addressed PR#4153 comments; removed ES Version property and made Type optional in all ES HTTP/Record processors, applying sensible default values where required; use _source queyr parameter instead of _source_include/s as it's compatible between ES versions

Fix unit test compilation to use JDK8-compatible library/method

Better optional type and id handling for PutElasticsearchRecord; update nifi-elasticsearch-client-service build dependencies to use latest versions of Elasticsearch in each supported major version (5/6/7); addressed several warnings in ElasticSearchClientServiceImpl

This closes #4667.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-11-26 15:08:13 +09:00
Joe Gresock a3d845a38f
NIFI-6403: Adding Elasticsearch7 support to HTTP processors
This closes #4153.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-11-26 15:08:05 +09:00
Mark Payne 7ad9520079
NIFI-8041: Refactored API for stateless nifi so that calling StatelessDataflow.trigger() returns an object that allows the caller to wait for the result, cancel the result, etc. and then optionally acknowledge the completion. This allows the caller to block the completion of ProcessSession.commit() until it has handled the output of the dataflow execution.
NIFI-8038: Fixed deadlock that can occur when updating Parameter Context

This closes #4684.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-24 15:12:51 -05:00
Mark Payne 3c9d8a7007
NIFI-7897: Refactoring to create a new nifi-framework-components module.
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup

NIFI-7897: Addressing review feedback

NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly

This closes #4669.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-24 11:37:45 -05:00
Arek Burdach 87fd9f968f
NIFI-7958 ConvertAvroToORC - decimal support
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4628
2020-11-23 13:00:54 -05:00
Denes Arvay 2bf5d1a56d NIFI-8017 NPEx at org.apache.nifi.provenance.index.lucene. IndexDirectoryManager:486
Added an extra null-check to avoid NullPointerException
2020-11-19 16:01:42 -05:00
exceptionfactory dc6ba2541f NIFI-8024 Added null claim check to EncryptedFileSystemRepository.read() to avoid EOFException 2020-11-19 09:11:49 -05:00
humpfhumpf 40fdcd8144 NIFI-7771 - Infinite loop on WebUI when node stopped in cluster (PB1)
This commit resolves "Problem 1" described in issue #7771.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4496.
2020-11-18 14:56:59 -05:00
Bryan Bende dcc4fb00a5
NIFI-7888 Added support for authenticating via SAML
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page

- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests

- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm

- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client

- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding

- Made atomic replace methods for storing groups and saml credential in database

- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata

- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias

- Fixed unit test

- Added property to specify an optional identity attribute which would be used instead of NameID

- Cleaned up logging

- Fallback to keystore password when key password is blank

- Make signature and digest default to SHA-256 when no value provided in nifi.properties

This closes #4614
2020-11-18 12:44:02 -05:00
Mark Payne 7848ba59a2
NIFI-8021: Provide the ability to pin partitions to particular hosts when using ConsumeKafka processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4672.
2020-11-18 16:03:24 +01:00
Matthew Burgess edc060bd92 NIFI-7989: Add UpdateHiveTable processors for data drift capability
NIFI-7989: Allow for optional blank line after optional column and partition headers
NIFI-7989: Incorporated review comments
NIFI-7989: Close Statement when finishing processing
NIFI-7989: Remove database name property, update output table attribute

This closes #4653.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-18 00:39:07 +01:00
Tamas Palfy 4aaec5aa38
NIFI-8020 Make sure TailFile doesn't leave FileChannel open when handling NUL characters
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4671.
2020-11-18 00:26:27 +01:00
Kevin Doran a0328ff8d1
NIFI-7744 Add Support for Unicode in X-ProxiedEntitiesChain (#4664)
- Adds detection and encoding of non-ascii characters to creation of chain
- Adds unit tests that use proxied entities with Unicode
2020-11-17 10:23:30 -05:00
r65535 d5dc63ded9
NIFI-7982 Added tags to FlattenJSON processor
Adding missing imports

This closes #4662

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-17 07:05:52 -05:00
Tamas Palfy 47cc2867ba NIFI-7972 Add a boolean property by which the user can tell the processor to yield (and try again later) whenever it encounters a NUL character. 2020-11-16 15:16:12 -05:00
Chris Sampson 5c925371c2
NIFI-7474 Allow index operation type to be provided for PUT Elasticsearch processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4665.
2020-11-16 18:19:35 +01:00
Denes Arvay 83948db989
NIFI-7996 Conversion with ConvertRecord to avro results in invalid date
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4666.
2020-11-16 17:40:40 +01:00
Peter Gyori e66362194d NIFI-7922: Added support for GET in ListenHTTP for health check
NIFI-7922: Fixes based on review comments

ListenHTTP: changed if(!sslRequired) to if(sslRequired) so that the positive case comes first.
HealthCheckServlet: response body for GET contains "OK".
ContentAcknowledgmentServlet: super.doDelete() is called when a DELETE should be rejected because of port mismatch.

NIFI-7922: Refactored, based on review comments

NIFI-7922: Fixed a checkstyle violation (organized the imports)

This closes #4603.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-16 12:20:30 +01:00
s9514171 c57d0abcd3 NIFI-6820 If Processor adjusts a counter but does not interact with any FlowFiles, the counter isn't updated 2020-11-13 11:25:34 -05:00
Nathan Gough 8a4079cd78
NIFI-7993 - Upgraded Jetty dependency and fixed a minor issue with Jetty security filter tests.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4658.
2020-11-13 13:27:12 +01:00
Mark Payne c79ad1502e
NIFI-7999: Do not call NioAsyncLoadBalanceClient.nodeDisconnected() if node was already in a disconnected state. Doing so was resulting in that method being called constantly on startup, and with the synchronization in place that can result in a huge performance hit on startup. Also updated RemoteQueuePartition to move a small predicate into its own method. This was done because the predicate was previously defined within a synchronized method, which meant that invoking that predicate required synchronization.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4657.
2020-11-12 17:44:46 +01:00
Peter Turcsanyi 018778a25d NIFI-7994: Fixed ReplaceText concurrency issue
Moving RegexReplace.additionalAttrs field to local variable in replace() method.
Concurrent usage of the additionalAttrs map caused data corruption.
2020-11-12 10:45:10 -05:00
Nathan Gough 479ee6e3db
NIFI-7819 - Added ZooKeeperStateProvider TLS properties.
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.

This closes #4613.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-10 17:05:05 -05:00
Mark Payne badcfe1ab7
NIFI-7992: Periodically check disk usage for content repo to see if backpressure should be applied. Log progress in background task. Improve performance of background cleanup task by not using an ArrayList Iterator and constantly calling remove but instead wait until the end of our cleanup loop and then removed from the list all elements that should be removed in a single update
This closes #4652.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-10 15:27:51 -05:00
tpalfy 940bc3056c
NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() (#4629)
* NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() and taking care of TGT renewal.

* NIFI-7954 Simplified SecurityUtil.callWithUgi a little.

* NIFI-7954 Simplified SecurityUtil.callWithUgi more.

* NIFI-7954 Removed unnecessary code.
2020-11-09 09:00:20 -05:00
Pierre Villard 14ec02f21d
NIFI-7981 - add support for enum type in avro schema
This closes #4648

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-05 18:19:55 -05:00
Siyao Meng 9a3a659c44
NIFI-7949: Add tag HCFS to HDFS Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4633.
2020-11-05 13:21:11 +01:00
Bryan Bende 1366d017b8
NIFI-7968 Ensure the status login user is set in UserGroupInformation after creating a UGI from a Subject
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4643.
2020-11-05 13:18:35 +01:00
Bence Simon 59e00c4b6f NIFI-7959 Handling node disconnection in MonitorActivity processor
- Make reporting in clustered scope to dependent of expected cluster state in order to prevent unexpected flow file emission

This closes #4642.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-11-04 10:09:46 -05:00
Matthew Burgess 71a5735f63
NIFI-7976: Infer ints in JSON when all values fit in integer type
NIFI-7976: Fixed unit test

This closes #4645

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-03 15:43:15 -05:00
Bryan Bende d773521ee0
NIFI-1121 Fix Schema Name and Schema Branch properties
This closes #4512.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:38:41 -05:00
mtien 42c2cda9a2
NIFI-1121 Fixed a dependent value check error.
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:51 -05:00
Mark Payne 4b9014b959
NIFI-1121: Updated backend to perform appropriate validation. Added tests. Updated documentation writer. Updated dev guide to explain how PropertyDescriptor.Builder#dependsOn affects validation. Updated JavaDocs for PropertyDescriptor.Builder#dependsOn
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:42 -05:00
mtien 535cab3167
NIFI-1121: Added an additional check for hidden properties to account for transitive dependent properties.
- Added a 'dependent' attribute to determine whether or not to save dependent property values

Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:33 -05:00
Mark Payne e2e901b6b9
NIFI-1121: Added property dependencies to MergeContent
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:26 -05:00
mtien 4bd9d7b413
NIFI-1121 Show and hide properties that depend on another property.
Co-authored-by: Scott Aslan <scottyaslan@gmail.com>
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:16 -05:00
Mark Payne f7f336a4b0
NIFI-1121: Added API changes for having one Property depend on another
Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:37:08 -05:00
r65535 d8d9aa9cec Added regex support for attribute header selection on HandleHTTPResponse 2020-11-03 12:22:41 -05:00
mtien 8b78277a45
NIFI-7914 Bumped H2 dependency to 1.4.199.
Bumped icu4j dependency to 60.2.
Replaced jackson-mapper-asl dependency with jackson-databind.
Fixed an error comparing key identities in TestKeyService.
Replaced jackson-mapper-asl ObjectMapper with jackson-databind ObjectMapper in LivySessionController.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4640.
2020-11-03 11:46:29 +01:00
sjyang18 80f49eb7bd NIFI-7406 Added PutAzureCosmosRecord processor for Azure Cosmos DB
This closes #4253

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-11-03 01:54:33 -06:00
Peter Turcsanyi 2b461bbf29
NIFI-7452: Support adls_gen2_directory in Atlas reporting task
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4636.
2020-10-30 13:50:53 +01:00
Mark Bean a115894984 NIFI-7963: fix package name for TestZooKeeperStateServer
This closes #4631

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-10-28 21:09:10 -05:00
Mark Payne 2232e28052
NIFI-7956: This closes #4626. Added option of rolling back session on error instead of routing to failure for PutKudu
Signed-off-by: Joe Witt <joewitt@apache.org>
2020-10-27 21:23:59 -07:00
Mark Payne c610aab3cb NIFI-7953: Updated ConsumeKafka_2_0/ConsumeKafkaRecord_2_0/ConsumeKafka_2_6/ConsumeKafkaRecord_2_6 to allow separating records by key 2020-10-27 19:04:19 -04:00
Mark Payne 74968991d5 NIFI-7952: Allow RecordPath to be used for specifying the Insertion Operation and the data to be inserted into Kudu 2020-10-27 19:03:35 -04:00
mtien bf962f6227 NIFI-7584 Added OIDC logout mechanism.
Added method to validate the OIDC Access Token for the revoke endpoint.
Created a new callback URI of oidc/logoutCallback to handle certain OIDC logout cases.
Changed method to exchange the Authorization Code for a Login Authentication Token.
Added a new method to exchange the AuthN Code for an Access Token.
Changed method to convert OIDC Token to a Login AuthN Token instead of a NiFi JWT.
Created new OidcServiceGroovyTest class.

NIFI-7584-rebase Added test.

NIFI-7584 Fixed a checkstyle issue.

NIFI-7584 Removed a dependency not in use.

NIFI-7584 Made revisions based on PR review.
Refactored revoke endpoint POST request to a private method.
Removed unnecessary dependencies.
Fixed Regex Pattern to search for literal dot character.
Fixed logging the Exception message.
Fixed caught Exception.
Changed timeout value to a static variable.
Changed repeating error messages to a static string.
Reduced sleep duration in unit test.
Refactored cookie generation to private method.

NIFI-7584 Fixed the snapshot version.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4593.
2020-10-27 17:18:05 -04:00
Mark Payne bcf625f4c8 NIFI-7920: When node that is connected to cluster is asked to reconnect, ensure that it relinquishes role of Cluster Coordinator and Primary Node 2020-10-27 14:30:13 -04:00
Bryan Bende e7f244f267
NIFI-7923 Catch Throwable instead of Exception in NarAutoLoaderTask so that thread won't die (#4623) 2020-10-27 12:33:08 -04:00
Bryan Bende 77b4abc21e
NIFI-7984 Use UGI doAs in DeleteHDFS (#4622) 2020-10-27 11:24:48 -04:00
markap14 67d1b73a85
NIFI-7944: When checking if a Component can reference a Controller Service, NiFi would sometimes fail to find that Controller Service because it was looking based on the wrong Process Group. Fixed that. (#4619) 2020-10-27 08:57:05 -04:00
Pierre Villard 9c1656dbff NIFI-7854 - Add proxy username/password support to SQS processors
This closes #4600.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-10-27 00:22:15 +01:00
Peter Turcsanyi 3dd024fb66
NIFI-7825: Support native library loading via absolute path
Use an AspectJ aspect and agent to intercept the load native library calls
at runtime, copy the native library file to temp folder and proceed with
the newly created file in order to provide classloader isolation.

Remove AspectJ jars from lib directory, move the necessary jar to lib/aspectj subdirectory.

This closes #4540.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-10-26 16:21:16 -04:00
Fabio Torchetti 9a11d23c83
NIFI-7943 - Add application properties to GetAzureEventHub
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4617.
2020-10-26 19:36:27 +01:00
Bence Simon c8ea7523ef
NIFI-7942 Fixing ordering issue when counter based metrics are added by component
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4621.
2020-10-26 19:31:05 +01:00
Bence Simon b980a8ea8c NIFI-7549 Adding Hazelcast based DistributedMapCacheClient support
NIFI-7549 Refining documentation; Changing explicit HA mode; Smaller review comments
NIFI-7549 Code review responses about license, documentation and dependencies
NIFI-7549 Fixing issue when explicit HA; Some further review based adjustments
NIFI-7549 Response to code review comments
NIFI-7549 Adding extra serialization test
NIFI-7549 Minor changes based on review comments
NIFI-7549 Adding hook point to the shutdown

This closes #4510.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-10-22 19:22:49 +02:00
Denes Arvay d05d0c6240 NIFI-7925: ValidateRecord reports false positive for avro arrays with null elements 2020-10-21 09:17:29 -04:00
Joey Frazee 55cb8d73cb NIFI-7401 Add ZooKeeper client TLS to CuratorLeaderElectionManager
NIFI-7401 Rebased to 1.13.0-SNAPSHOT and simplified tests

NIFI-7401 Added keystore types and changed properties to match nifi.security.*

NIFI-7401 Removed dead code from SecureClientZooKeeperFactory test

NIFI-7401 Renamed bean methods, moved helper code into NiFiProperties

NIFI-7401 Changed connection socket constants to use .class.getName()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4592.
2020-10-20 11:20:43 -04:00
eylonronen b442df58e2
NIFI-7935 Add reason to flow file on error
This closes #4608

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-16 18:44:27 -04:00
Indrajeet Gour 4a370eae8c
NIFI-7933 - Set proper property descriptor for CSV Escape
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4606.
2020-10-16 18:13:29 +02:00
Dan Kim 5b5b9a7800
NIFI-7916 refactor CountText to ignore whitespace words when counting
This closes #4595

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-14 09:09:42 -04:00
dependabot[bot] 8aa001a918
Bump junit from 4.13 to 4.13.1
Bumps [junit](https://github.com/junit-team/junit4) from 4.13 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.13...r4.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-12 21:54:52 +00:00
Matthew Burgess 4c235f0405
NIFI-7909: Change DataTypeUtils.toInteger() to use Math.toIntExact()
This closes #4596

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-12 17:53:27 -04:00
Pierre Villard c2960998ac
NIFI-7869 - adding a GCP profile for GCS connector (#4565) 2020-10-09 09:42:20 -04:00
kpsngh1998 a95ffce6bd
NIFI-7534 Consume MQTT Processor. Support Expression Language for Topic Filter property
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4573.
2020-10-08 15:38:37 +02:00
Andy LoPresto 82d3fd6cd6 NIFI-7898 Fixed unit test which failed with simultaneous services running on localhost:80. 2020-10-08 08:58:14 -04:00
Peter Gyori 668c278e30
NIFI-7815: Enhance the logging in MergeContent processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4534.
2020-10-08 11:44:52 +02:00
exceptionfactory efb629e37d
NIFI-7777 Removed support for Expression Language from Password property
Added unit test for no password configured on Zip files

This closes #4572.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-10-07 10:59:50 -07:00
exceptionfactory ea6b01159d
NIFI-7777 Added optional Password property to UnpackContent for decrypting Zip archives
This closes #4572.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-10-07 10:56:23 -07:00
M Tien de39a3d9f4
NIFI-7892 Created a Logout page to inform users of a complete logout when OIDC is configured. (#4579) 2020-10-07 12:17:00 -04:00
Matthew Burgess 325a49556c
NIFI-7592: Allow NiFi to be started without a GUI/REST interface (#4509)
* NIFI-7592: Allow NiFi to be started without a GUI/REST interface

* NIFI-7592: Enable all controller services when starting headless

* NIFI-7592: Marked duplicate dependencies as provided

* NIFI-7592: Incorporated additional review comments
2020-10-07 09:12:16 -04:00
Matthew Burgess 7cc3713389
NIFI-7796: Add Prometheus counters for total bytes sent/received (#4522)
* NIFI-7796: Add Prometheus metrics for total bytes sent/received, fixed read/written metrics

* NIFI-7796: Incorporated review comments
2020-10-06 09:26:27 -04:00
Ofirmgr be72dbb815
rename onConfigured to OnEnabled to make code more clear
This closes #4547

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-10-04 14:55:57 -04:00
Mark Payne 5d37809d30
NIFI-7874: Fixed bug that results in buffering all S3 elements in heap before emitting any of them. Added Batch Size property to ListS3
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4566.
2020-10-02 19:12:40 +02:00
Denes Arvay 6990f0d3a9
NIFI-7685: Add UTF8 support for FetchFTP
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4446
2020-10-01 17:35:46 -04:00
Mohammed Nadeem e2ccfbbacf
NIFI-7859: Support for capturing execution duration of query run as attributes in SelectHiveQL processors
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4560
2020-10-01 09:06:01 -04:00