Commit Graph

5804 Commits

Author SHA1 Message Date
eduardofontes c7edcd68e1
NIFI-7463
Create empty relationship for RunMongoAggregation

Fix default autoterminate and condition to redirect to REL_EMPTY

Change from new relationship to write an empty FlowFile to RESULT

Fix MONGO_URI

This closes #4281

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-05-20 19:54:24 -04:00
Mark Payne d195702ee2 Fixed a couple of typos in the RecordPath guide 2020-05-20 17:09:40 -04:00
Pierre Villard a9e9e5d137 NIFI-6571 Check token length on TLS toolkit server startup
This closes #3659.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-20 15:17:39 -05:00
KovalevIV 7034d7e44c NIFI-7471 fix bug with property validation 2020-05-20 10:49:49 -04:00
Andy LoPresto 441781cec5
NIFI-7407 Replaced SSLContextFactory references to "TLS" with "TLSv1.2" (in shared constant).
Changed JettyServer default SSL initialization and updated unit test.
Removed SecurityStoreTypes (unused).
Added StringUtils inverted blank and empty checks.
Added TlsConfiguration container object.
Enhanced KeystoreType enum.
Added clean #createSSLContext() method to serve as base method for special cases/other method signatures.
Added utility methods in KeyStoreUtils.
Added generic TlsException for callers that cannot resolve TLS-specific exceptions.
Added utility methods for component object debugging.
Enforced TLS protocol version on cluster comms socket creation.
Added utility method for SSL server socket creation.
Refactored (Server)SocketConfigurationFactoryBean to store relevant NiFiProperties in TlsConfiguration instead of stateful SSLContextFactory (Cluster comms now enforce modern TLS protocol version).
Removed duplicate SSLContextFactory.
Switched duplicate SslContextFactory to wrap shared SSLContextFactory.
Refactored SslContextFactoryTest for clarity (will move any unique tests to nifi-security-utils class test).
Added further validation & boundary checking in uses of TlsConfiguration.
Provided SSLSocketFactory accessor in SslContextFactory.
Refactored OkHttpReplicationClient tuple method.
Refactored OcspCertificateValidator TLS logic.
Added utility method to apply TLS configs to OkHttpClientBuilder.
Removed references to duplicate SslContextFactory.
Removed unnecessary SslContextFactory.
Moved OkHttpClientUtils to nifi-web-util module.
Updated module dependencies.
Removed now empty nifi-security module.
Enforced TLS protocol selection on LB server socket.
Enforced TLS protocol selection on S2S server socket.
Applied specified TLS protocol versions to S2S socket creation.
Completed removal of legacy SSLContext creation methods from only remaining SslContextFactory.
Replaced references to creation methods throughout codebase.
Replaced references to unnecessary NiFiProperties file reads throughout tests.
Removed duplicate ClientAuth enum from SSLContextService and changed all references to SslContextFactory.ClientAuth.
Suppressed repeated TLS exceptions in cluster, S2S, and load balance socket listeners.
Cleaned up legacy code.
Added external timing check to timing test assertion.
Made RestrictedSSLContextService TLS protocol versions allowable values explicit.
Enabled TLSv1.3 on Java 11.
Added explanations of TLS protocol versions in StandardSSLContextService and StandardRestrictedSSLContextService.
Resolved additional Java 11 test failures for NiFi internal classes that don't support TLSv1.3. Filed NIFI-7468 as follow on task.

This closes #4263.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-19 12:56:59 -07:00
Sushil Kumar 5826a09a83 NIFI-6911 Removed default Blob value for PutAzureBlobStorage
This closes #3906

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-05-19 13:47:08 -05:00
Peter Gyori d6240a1074 NIFI-7336: Add tests for DeleteAzureDataLakeStorage
DeleteAzureDataLakeStorage now throws exception if fileSystem or fileName is empty string

NIFI-7336: Add tests for DeleteAzureDataLakeStorage - typos fixed
NIFI-7336: Add tests for DeleteAzureDataLakeStorage - fixed a test case

This closes #4272.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 18:35:33 +02:00
sjyang18 852715aadd NIFI-7409: Azure managed identity support to Azure Datalake processors
NIFI-7409: review changes
NIFI-7409: ordering import statements
NIFI-7409: changed validateCredentialProperties logic

This closes #4249.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 11:50:08 +02:00
mtien 3fec4d8c27
NIFI-7331 Fixed grammatical errors in log output.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4283.
2020-05-19 10:38:45 +02:00
Peter Gyori 9aae58f117 NIFI-7446: FetchAzureDataLakeStorage processor now throws exception when the specified path points to a directory
A newer version (12.1.1) of azure-storage-file-datalake is imported.

This closes #4273.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-18 23:50:05 +02:00
Matthieu Cauffiez 179675f0b4 NIFI-7380 - fix for controller service validation in NiFi Stateless
This closes #4264.

Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 16:11:40 -04:00
Matthew Burgess a3cc2c58ff NIFI-6497: Allow FreeFormTextRecordSetWriter to access FlowFile Attributes
This closes #4275.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 15:11:33 -04:00
Yolanda M. Davis 13418ccb91
NIFI-7437 - created separate thread for preloading predictions, refactors for performance
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4274
2020-05-18 14:57:54 -04:00
Mark Payne c51b9051a8 NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows 2020-05-15 16:16:39 -04:00
Matthew Burgess 53a161234e
NIFI-7448: Fix quoting of DDL table name in PutORC
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4269.
2020-05-14 11:41:35 +02:00
Peter Gyori 9608fe33fa NIFI-7367: Add tests for FetchAzureDataLakeStorage
NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage
FetchAzureDataLakeStorage throws exception when filesystem or filename is blank.
Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors.
testFetchDirectory test case marked as ignored.

This closes #4257.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-13 14:50:31 +02:00
Nathan Gough 302a42185c
NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-05-12 14:19:23 -07:00
Peter Gyori 6b73ea48bc
NIFI-7413: Documented REMOTE_INVOCATION provenance event type in user/dev guides
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4267
2020-05-12 16:00:41 -04:00
Sushil Kumar 851359c1fc
NIFI-6913: PutAzureBlobStorage processor will create container if not exists
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4237.
2020-05-09 15:37:14 +02:00
mtien 647a9a60b7
NIFI-7423 Upgraded jquery dependency version.
NIFI-7423 Upgraded jquery dependency version to latest 3.5.1.

This closes #4258
2020-05-08 13:13:38 -04:00
Bence Simon 66b175f405
NIFI-7390 Covering Avro type conversion in case of map withing Record
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4256
2020-05-08 13:11:19 -04:00
mtien 6e2f86d716 NIFI-7398 Upgraded jackson-databind dependency version to 2.9.10.4 at root pom.xml.
Upgraded tika-parsers dep in nifi-media-processors.
Upgraded jackson-databind dep in nifi-graph-processors.
Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api.
Upgraded jackson-databind dep in in nifi-easyrules-service.
Upgraded calcite-core dep in nifi-sql-reporting-tasks.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4252.
2020-05-07 14:19:55 -04:00
Matthew Burgess b7c81d6007
NIFI-7428: Switch hive.version property to set Hive 3 version
This closes #4259
2020-05-06 19:38:23 -04:00
Otto Fowler f22cd26cf4 NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large
This closes #4251.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-06 17:20:08 -04:00
Alan Jackoway effa930151
NIFI-7425 Log Message for ReplaceText Over Buffer Size
Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4255.
2020-05-06 09:40:46 +02:00
Mark Payne 0448e23a96
NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz
NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered

This closes #4244
2020-05-04 14:49:15 -04:00
Peter Gyori 2a44f0ff2f NIFI-7412: Fixed provenance event types in Azure Fetch/Delete processors
This closes #4245.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-01 22:57:02 +02:00
Matthew Burgess aa986e0bfb
NIFI-7404: Fixed invalid script processors upon thread termination
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4238.
2020-05-01 16:46:52 +02:00
Matthew Burgess e66844e6c8
NIFI-7415: Add .asf.yaml to configure GitHub integrations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4246.
2020-05-01 16:19:55 +02:00
Peter Turcsanyi bce82a5e6f
NIFI-7416: Update travis-ci to GitHub Actions in the PR template
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4247.
2020-05-01 16:17:07 +02:00
sjyang18 aa1e272052 NIFI-6149: Azure EventHub Managed identities support patch
review changes
additional review changes
NIFI-6149: typo fixes

This closes #4226.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-30 23:17:20 +02:00
Andy LoPresto 7d494011ea
Added note about unique initial user identity names to walkthrough doc. 2020-04-30 12:36:30 -07:00
Matt Gilman e81960f8e8 NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.
2020-04-30 15:20:19 -04:00
Yolanda M. Davis 7784178abd
NIFI-7408 - added percent used metrics for connections
NIFI-7408 - return double value for utilization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4240
2020-04-29 13:11:06 -04:00
Otto Fowler 659a383723 NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
  without the flowfile
- change the content name and content file name from dynamic properties
  to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests

This closes #4234.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-04-29 13:01:06 -04:00
Matthew Burgess 1259bd5bd1
NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)
* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances
2020-04-28 16:56:49 -04:00
Peter Turcsanyi a7008903ed
NIFI-7298: PutAzureDataLakeStorage tests.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4227.
2020-04-28 19:03:56 +02:00
Andy LoPresto 148537d64a
NIFI-7377 Cleaned up nifi-stateless logs.
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.

This closes #4222.

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 17:29:42 -07:00
Sushil Kumar 996688b419
NIFI-7389 Makes Missable heartbeat counts configurable
This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 13:21:21 -07:00
Sushil Kumar 6c2701abef Do not update status for stopping a deleted node 2020-04-25 11:01:08 -04:00
Matthew Burgess cd10435b9f
NIFI-7378: Ensure label values are not null in Prometheus metrics (#4219) 2020-04-22 10:37:04 -04:00
Tamas Palfy e2716a6c94 NIFI-7280 ReportLineageToAtlas recognizes 'atlas.metadata.namespace' from Atlas config file.
Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property.
Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties.

NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor.
NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate.

This closes #4213.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-20 22:34:03 +02:00
Mark Payne c19db9d623
NIFI-7375: This closes #4218. Fixed a bug that caused Provenance Events not to show up in specific situations when clicking View Provenance for a Processor.
- Added System-level tests for Provenance repository to reproduce behavior.
- Added a Provenance Client to the CLI, which is necessary for System-level tests.
- Added small additional configuration for Provenance repository to simplify development of system tests
- Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-04-20 13:40:07 -04:00
Bence Simon 923a07a5db NIFI-7300 Allowing narrow numeric types to fit againt schema check with wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302) 2020-04-20 11:19:21 -04:00
Opher Shachar 99e69f0252
NIFI-7318 - Allow 'docker stop' to gracefully shutdown
As it is issuing 'docker stop' will immediatly exit the container.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4182.
2020-04-18 16:25:05 +02:00
Jérémie 8e9dbd7335
NIFI-7366 - ConsumeEWS Processor parse EML
https://issues.apache.org/jira/browse/NIFI-7366

This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4215.
2020-04-17 15:59:15 +02:00
muazmaz 58118cf904 NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.
NIFI-7334 Update to FetchDataLakeStorage Processor

This closes #4212.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-16 11:49:44 +02:00
EndzeitBegins 8e3f42051f
NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration
This closes #4201.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-04-16 15:56:40 +09:00
Alexandre Vautier b113a022e4
NIFI-7359 Fix parent id on process metrics for Prometheus
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4209
2020-04-15 20:27:44 -04:00
Andy LoPresto 2224ace968
Added Jira and security reporting links to README.md 2020-04-15 12:34:03 -07:00