Commit Graph

5947 Commits

Author SHA1 Message Date
sjyang18 852715aadd NIFI-7409: Azure managed identity support to Azure Datalake processors
NIFI-7409: review changes
NIFI-7409: ordering import statements
NIFI-7409: changed validateCredentialProperties logic

This closes #4249.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-19 11:50:08 +02:00
mtien 3fec4d8c27
NIFI-7331 Fixed grammatical errors in log output.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4283.
2020-05-19 10:38:45 +02:00
Peter Gyori 9aae58f117 NIFI-7446: FetchAzureDataLakeStorage processor now throws exception when the specified path points to a directory
A newer version (12.1.1) of azure-storage-file-datalake is imported.

This closes #4273.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-18 23:50:05 +02:00
Matthieu Cauffiez 179675f0b4 NIFI-7380 - fix for controller service validation in NiFi Stateless
This closes #4264.

Signed-off-by: Matthieu Cauffiez <matthieu.cauffiez@bell.ca>
Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 16:11:40 -04:00
Matthew Burgess a3cc2c58ff NIFI-6497: Allow FreeFormTextRecordSetWriter to access FlowFile Attributes
This closes #4275.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-18 15:11:33 -04:00
Yolanda M. Davis 13418ccb91
NIFI-7437 - created separate thread for preloading predictions, refactors for performance
NIFI-7437 - reduced scheduler to 15 seconds, change cache to expire after no access vs expire after write

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4274
2020-05-18 14:57:54 -04:00
Mark Payne c51b9051a8 NIFI-7460: Avoid NPE when a VersionedProcessor has a null value for autoTerminatedRelationships. Added additional logging and improved error handling around syncing with invalid flows 2020-05-15 16:16:39 -04:00
Matthew Burgess 53a161234e
NIFI-7448: Fix quoting of DDL table name in PutORC
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4269.
2020-05-14 11:41:35 +02:00
Peter Gyori 9608fe33fa NIFI-7367: Add tests for FetchAzureDataLakeStorage
NIFI-7367: Negative test cases for expression language in FetchAzureDataLakeStorage
FetchAzureDataLakeStorage throws exception when filesystem or filename is blank.
Fixed logged error messages in all 3 of the Delete, Fetch and Put ADLS processors.
testFetchDirectory test case marked as ignored.

This closes #4257.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-13 14:50:31 +02:00
Nathan Gough 302a42185c
NIFI-7321 - Allow NiFi admins to configure whether Jetty will send the Jetty server version in responses.
Fixed a checkstyle error.
Added property to nifi.properties.
Changed property to a variable that is set with the pom.xml.
Added setting the version variable to another HTTPConfiguration to fix the version being sent in docs context.
Fixed typo error.

This closes #4192.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-05-12 14:19:23 -07:00
Peter Gyori 6b73ea48bc
NIFI-7413: Documented REMOTE_INVOCATION provenance event type in user/dev guides
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4267
2020-05-12 16:00:41 -04:00
Sushil Kumar 851359c1fc
NIFI-6913: PutAzureBlobStorage processor will create container if not exists
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4237.
2020-05-09 15:37:14 +02:00
mtien 647a9a60b7
NIFI-7423 Upgraded jquery dependency version.
NIFI-7423 Upgraded jquery dependency version to latest 3.5.1.

This closes #4258
2020-05-08 13:13:38 -04:00
Bence Simon 66b175f405
NIFI-7390 Covering Avro type conversion in case of map withing Record
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4256
2020-05-08 13:11:19 -04:00
mtien 6e2f86d716 NIFI-7398 Upgraded jackson-databind dependency version to 2.9.10.4 at root pom.xml.
Upgraded tika-parsers dep in nifi-media-processors.
Upgraded jackson-databind dep in nifi-graph-processors.
Upgraded jackson-databind dep in nifi-elasticsearch-client-service-api.
Upgraded jackson-databind dep in in nifi-easyrules-service.
Upgraded calcite-core dep in nifi-sql-reporting-tasks.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4252.
2020-05-07 14:19:55 -04:00
Matthew Burgess b7c81d6007
NIFI-7428: Switch hive.version property to set Hive 3 version
This closes #4259
2020-05-06 19:38:23 -04:00
Otto Fowler f22cd26cf4 NIFI-7420 remove the http.param attribute. It contained both the query parameters already captured in http.query.param _and_ the multipart form data names and values, which are captured in the part data, and could be very very large
This closes #4251.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-05-06 17:20:08 -04:00
Alan Jackoway effa930151
NIFI-7425 Log Message for ReplaceText Over Buffer Size
Adds a log message when ReplaceText sends a flowfile to the failure relationship because
it is larger than the max buffer size.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4255.
2020-05-06 09:40:46 +02:00
Mark Payne 0448e23a96
NIFI-7414: Escape user-defined values that contain invalid XML characters before writing flow.xml.gz
NIFI-7414: Updated StandardFlowSerializerTest to include testing for variable names and values being filtered

This closes #4244
2020-05-04 14:49:15 -04:00
Peter Gyori 2a44f0ff2f NIFI-7412: Fixed provenance event types in Azure Fetch/Delete processors
This closes #4245.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-05-01 22:57:02 +02:00
Matthew Burgess aa986e0bfb
NIFI-7404: Fixed invalid script processors upon thread termination
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4238.
2020-05-01 16:46:52 +02:00
Matthew Burgess e66844e6c8
NIFI-7415: Add .asf.yaml to configure GitHub integrations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4246.
2020-05-01 16:19:55 +02:00
Peter Turcsanyi bce82a5e6f
NIFI-7416: Update travis-ci to GitHub Actions in the PR template
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4247.
2020-05-01 16:17:07 +02:00
sjyang18 aa1e272052 NIFI-6149: Azure EventHub Managed identities support patch
review changes
additional review changes
NIFI-6149: typo fixes

This closes #4226.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-30 23:17:20 +02:00
Andy LoPresto 7d494011ea
Added note about unique initial user identity names to walkthrough doc. 2020-04-30 12:36:30 -07:00
Matt Gilman e81960f8e8 NIFI-7170:
- Adding a flag to nifi.properties to disable anonymous authentication.

NIFI-7170:
- Fixing checkstyle issues.

NIFI-7170:
- Adding missing license header.

NIFI-7170:
- Initial PR feedback.

NIFI-7170:
- Fixing broken integration tests.
- Creating new integration tests for verifying allowing and preventing anonymous access.

NIFI-7170:
- Ensuring the new anonymous authentication property is considered for proxied requests.

NIFI-7170 - Fixed comment.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4099.
2020-04-30 15:20:19 -04:00
Yolanda M. Davis 7784178abd
NIFI-7408 - added percent used metrics for connections
NIFI-7408 - return double value for utilization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4240
2020-04-29 13:11:06 -04:00
Otto Fowler 659a383723 NIFI-7394: Add support for sending Multipart/FORM data to InvokeHTTP.
By using dynamic properties with a prefix naming scheme, allow
definition of the parts, including the name to give the Flowfile content
part, and optionally it's file name.
After review:
- change so that we can send just the form content or just form data
  without the flowfile
- change the content name and content file name from dynamic properties
  to properties
- change the dynamic name to be an invalid http header "post:form:xxxx"
- add validation and more tests

This closes #4234.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-04-29 13:01:06 -04:00
Matthew Burgess 1259bd5bd1
NIFI-7379: Support multiple instances of Prometheus registries/metrics (#4229)
* NIFI-7379: Support multiple instances of Prometheus registries/metrics

* NIFI-7379: Refactored Prometheus objects to support multiple instances
2020-04-28 16:56:49 -04:00
Peter Turcsanyi a7008903ed
NIFI-7298: PutAzureDataLakeStorage tests.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4227.
2020-04-28 19:03:56 +02:00
Andy LoPresto 148537d64a
NIFI-7377 Cleaned up nifi-stateless logs.
Refactored masking logic to CipherUtility and indicated masking with label and Base64 output.
Added JSON masking logic to nifi-stateless module.
Added argument masking functionality to Program.
Moved groovy unit tests to proper Maven directory structure.
Modified plain argument output to use filtering/masking methods in provided utility.
Refactored utility methods.
Updated unit tests.

This closes #4222.

Co-authored-by: Pierre Villard <pierre.villard.fr@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 17:29:42 -07:00
Sushil Kumar 996688b419
NIFI-7389 Makes Missable heartbeat counts configurable
This closes #4236.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-04-27 13:21:21 -07:00
Sushil Kumar 6c2701abef Do not update status for stopping a deleted node 2020-04-25 11:01:08 -04:00
Matthew Burgess cd10435b9f
NIFI-7378: Ensure label values are not null in Prometheus metrics (#4219) 2020-04-22 10:37:04 -04:00
Tamas Palfy e2716a6c94 NIFI-7280 ReportLineageToAtlas recognizes 'atlas.metadata.namespace' from Atlas config file.
Still recognizes 'atlas.cluster.name' as well, but takes lower precedence than the new property.
Also Atlas URL can be provided via the 'atlas.rest.address' property in the atlas-application.properties.

NIFI-7280 In ReportLineageToAtlas improved documentation and adjusted property ordering for better user experience. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended documentation. Minor refactor.
NIFI-7280 In ReportLineageToAtlas amended more documentation. More minor refactor.
NIFI-7280 - In Atlas reporting: complete clusterName -> namespace overhaul where appropriate.

This closes #4213.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-20 22:34:03 +02:00
Mark Payne c19db9d623
NIFI-7375: This closes #4218. Fixed a bug that caused Provenance Events not to show up in specific situations when clicking View Provenance for a Processor.
- Added System-level tests for Provenance repository to reproduce behavior.
- Added a Provenance Client to the CLI, which is necessary for System-level tests.
- Added small additional configuration for Provenance repository to simplify development of system tests
- Minor improvements to system tests (such as ability to destroy environment between tests) needed for Provenance repository based system tests

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-04-20 13:40:07 -04:00
Bence Simon 923a07a5db NIFI-7300 Allowing narrow numeric types to fit againt schema check with wider type; Allowing doubles with value within float precision to be considered as valid floats (NIFI-7302) 2020-04-20 11:19:21 -04:00
Opher Shachar 99e69f0252
NIFI-7318 - Allow 'docker stop' to gracefully shutdown
As it is issuing 'docker stop' will immediatly exit the container.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4182.
2020-04-18 16:25:05 +02:00
Jérémie 8e9dbd7335
NIFI-7366 - ConsumeEWS Processor parse EML
https://issues.apache.org/jira/browse/NIFI-7366

This commit allows to retrieve ItemAttachement (such as EML) file when pulling mail.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4215.
2020-04-17 15:59:15 +02:00
muazmaz 58118cf904 NIFI-7334 Adding FetchDataLakeStorage Processor to provide native support for Azure Data lake Gen 2 Storage.
NIFI-7334 Update to FetchDataLakeStorage Processor

This closes #4212.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-04-16 11:49:44 +02:00
EndzeitBegins 8e3f42051f
NIFI-7348 Wait - Removes WAIT_START_TIMESTAMP after expiration
This closes #4201.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-04-16 15:56:40 +09:00
Alexandre Vautier b113a022e4
NIFI-7359 Fix parent id on process metrics for Prometheus
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4209
2020-04-15 20:27:44 -04:00
Andy LoPresto 2224ace968
Added Jira and security reporting links to README.md 2020-04-15 12:34:03 -07:00
Bence Simon 83400789f6
NIFI-7292 Preventing file listing from fail because of insufficient privileges
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4195.
2020-04-15 18:54:51 +02:00
Rob Fellows c263daf20b
[NIFI-7358] - Fix: Sorting on 'Estimated Time to Back Pressure' in the Connection summary table does not work properly
- fix style issues
- review feedback

This closes #4208
2020-04-15 10:06:38 -04:00
sjyang18 9df53e7204
NIFI-6977 - Change the reporting behavior of Azure Reporting task to report report the time when metrics are generated
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4211.
2020-04-15 12:53:18 +02:00
Mark Payne a61a4c2b58
NIFI-7347: Fixed NullPointerException that can happen if a bin is merged due to timeout and has no records
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4210.
2020-04-15 12:39:13 +02:00
Matthew Burgess c9f48e3788
NIFI-7354: Allow analytics properties to be set via environment variables in docker scripts
This closes #4203.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2020-04-13 09:15:05 -04:00
Andy LoPresto 07a8311b4c
NIFI-7319 Add walkthrough document (#4193)
* NIFI-7319 Added first draft of walkthroughs doc.

* NIFI-7319 Added instructions and screenshots for securing standalone NiFi instance.

* NIFI-7319 Added instructions and screenshots for instructing OS & browser to trust self-signed certificate.

* NIFI-7319 Added instructions and screenshots for securing NiFi with externally-provided certificates.

* NIFI-7319 Added instructions and screenshots for building NiFi from source.

* NIFI-7319 [WIP] Converting secure cluster instructions to match format.
Fixed instructions regarding embedded ZooKeeper configuration.

* NIFI-7319 Completed secure cluster walkthrough.

* NIFI-7319 Added walkthroughs to documentation navigation list.

* NIFI-7319 Incorporated PR feedback on broken links.

* NIFI-7319 Removed line number helpers from update sections.

* NIFI-7319 Incorporated final PR review items.

Co-authored-by: Sandra Pius <spiusapache@gmail.com>
2020-04-10 22:25:40 -07:00
Mark Payne c99a187018 NIFI-7346: Ensure that the Provenance Repository doesn't delete the Active Event File 2020-04-09 13:34:37 -04:00