Commit Graph

631 Commits

Author SHA1 Message Date
Joe Gresock 2daac5714a
NIFI-8695: Adding context to sensitive property providers
This closes #5206

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-07-27 15:37:29 -05:00
exceptionfactory e16bf644e3
NIFI-8948 Upgraded Spring to 5.3.9 and Security to 5.5.1
- Upgrades Spring Framework from 5.3.8 to 5.3.9
- Upgrades Spring Security from 5.4.6 to 5.5.1
- Upgrades Spring Boot from 2.5.1 to 2.5.2 in Registry
- Upgrades Spring Data Redis from 2.5.1 to 2.5.3

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5252.
2021-07-27 10:21:27 +02:00
Emilio Setiadarma d4a560c59a
NIFI-6325 Added AWS KMS Sensitive Properties Provider
This closes #5202

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-07-22 08:24:52 -05:00
Bryan Bende 74c0a91b6c
NIFI-8933 Configure Jersey's ObjectMapper to ingnore unknown fields
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5236.
2021-07-21 11:46:45 +02:00
exceptionfactory c668d3df1b NIFI-8782 Added Rate-Limiting for Access Token Requests
- Added Jetty DoSFilter configured for /access/token
- Added nifi.web.max.access.token.requests.per.second property with default value of 25

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5215.
2021-07-14 14:24:45 -04:00
Joe Witt 7356332852
Merge branch 'NIFI-8767-RC2' 2021-07-14 09:07:22 -05:00
exceptionfactory 05d7867a16
NIFI-8768 Added toLocalDate() for convertType() handling of DATE fields
- Updated PutKudu to use DataTypeUtils.toLocalDate() for DATE fields
- Updated PutDatabaseRecord to remove convertDateToLocalTZ() since convertType() uses toLocalDate()
- Updated PutElasticsearchHttpRecord to use default time zone format for DATE fields
- Updated WriteXMLResult to use default time zone format for DATE fields
- Updated WriteJsonResult to use default time zone format for DATE fields
- Updated AvroTypeUtil to use toLocalDate() for logical DATE fields
- Updated JdbcCommon to avoid conversion to UTC for logical DATE fields
- Updated Processor and RecordReader unit tests for consistency in DATE comparison

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5210
2021-07-13 16:17:36 -04:00
Joe Witt 97feacc181
NIFI-8767-RC2 prepare for next development iteration 2021-07-10 12:17:09 -07:00
Joe Witt fcbf1d5f97
NIFI-8767-RC2 prepare release nifi-1.14.0-RC2 2021-07-10 12:17:05 -07:00
exceptionfactory aedbd0d753 NIFI-8511 Added KeyStore implementation of KeyProvider
- KeyStoreKeyProvider supports PKCS12 and BCFKS
- Refactored KeyProvider and implementations to nifi-security-kms
- Updated Admin Guide and User Guide with KeyStoreKeyProvider details

NIFI-8511 Improved documentation and streamlined several methods

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5110.
2021-07-09 12:12:25 -04:00
Peter Turcsanyi f11c99f9ec
NIFI-8475: Fixed EL query parser when input contains multiple escaped expressions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5031
2021-06-25 13:59:11 -04:00
Joe Gresock 726082ffa6
NIFI-8447 Added HashiCorp Vault Transit Sensitive Properties Provider
- Added default bootstrap-hashicorp-vault.conf
- Updated Toolkit Guide documentation with HashiCorp Vault properties

This closes #5154

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-24 22:06:42 -05:00
exceptionfactory 6a83115d6a NIFI-7468 Updated SSLSocketChannel to support TLS 1.3
- Handling additional FINISHED Handshake Status for TLS 1.3 Post-Handshake Messages per RFC 8446 Section 4.6
- Removed clearing buffers after handshake to avoid losing packets
- Updated read() method to check Handshake Status after SSLEngine.unwrap()
- Changed SSLSocketChannelSender to close SSLSocketChannel before other resources
- Added ChannelStatus enum and convenience logging methods for tracing status
- Added unit tests for TLS 1.2 and 1.3 using Netty server and client handlers

NIFI-8704 Updated netty-handler to 4.1.65.Final

NIFI-7468 Corrected SSLSocketChannel.read() to return byte read

NIFI-7468 Adjusted comment formatting

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5152.
2021-06-22 22:29:16 -04:00
exceptionfactory 2efc2b6b8b
NIFI-8718 Upgraded Apache Commons IO to 2.10.0
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5169
2021-06-21 11:29:27 -04:00
rhavermans 9fd8042cfb
NIFI-8664 Update json-path lib from 2.4.0 -> 2.6.0
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5132
2021-06-21 11:01:10 -04:00
exceptionfactory 9be9b9dbe2
NIFI-8708 Upgraded Spring Framework to 5.3.8 for several extensions
- nifi-vault-utils
- nifi-email-bundle
- nifi-redis-bundle

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #5163.
2021-06-16 18:18:54 +02:00
markap14 96a8b2d090
NIFI-8666: Allow users to escape parameter names in Expression Language using quotes. (#5133) 2021-06-15 09:37:34 -04:00
Peter Turcsanyi 48befe22f6
NIFI-8645: Disable LongRunningTaskMonitor by default
This closes #5111

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-14 08:50:34 -05:00
Joe Gresock 6f04b45773
NIFI-8643: Correcting nifi.cluster.load.balance.address in properties (#5146)
NIFI-8643: Correcting nifi.cluster.load.balance.address in properties
2021-06-11 16:58:20 -04:00
Joe Gresock 1ccc4fbb0f
NIFI-8651: Refactor Sensitive Properties Providers for extension
This closes #5131

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-06-11 15:44:37 -05:00
Mark Payne 64f600d0ce
NIFI-8689: This closes #5150. Avoid flushing the socket buffer unnecessarily when sending a series of FlowFiles via site-to-site
Signed-off-by: Joe Witt <joewitt@apache.org>
2021-06-11 13:15:03 -07:00
exceptionfactory db40989b48 NIFI-8516 Enabled HTTPS and Single User Authentication in default configuration
- Set default HTTPS Port to 9443
- Set default authorizer to single-user-authorizer
- Set default login-identity-provider to single-user-provider
- Updated README.md with authentication instructions using generated credentials
- Updated default URL and port information in Administration and User Guides
- Updated Getting Started Guide with authentication and URL changes
- Updated Docker images to set HTTPS as default configuration
- Updated default HTTPS port to 8443
- Set Cluster Protocol secure property in Docker start scripts
- Added set-single-user-credentials command
- Refactored shared classes to nifi-single-user-utils
- Updated Getting Started documentation and logging
- Updated documentation and TLS Toolkit default ports
- Updated Toolkit Guide and Administration Guide
- Updated README.md with HTTPS links
2021-06-11 14:18:53 -04:00
Mark Payne c9dee30294
NIFI-8658: Allow Filter Functions and expressions to be specified as a RecordPaths
NIFI-8658: Addressed issue where the RecordField that was provided from Function Filters were not accurate

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5125
2021-06-04 17:17:33 -04:00
Chris Sampson e883aa6b0b
Rename RecordPath un/escape JSON functions (#5124)
Signed-off-by: Otto Fowler <ottobackwards@gmail.com>

This closes #5124.
2021-06-04 11:56:28 -04:00
Chris Sampson bc5204d4df
NIFI-8137 Record Path EscapeJson/UnescapeJson functions (#4756)
* NIFI-8137 Record Path EscapeJson/UnescapeJson functions

* Correct jackson-databind dependency version

* Add negative tests for RecordPath JSON handling; rename RecordPath JSON classes to better match existing functions

Signed-off-by: Otto Fowler <ottobackwards@gmail.com>

This closes #4756.
2021-06-04 09:19:24 -04:00
exceptionfactory a3365c8833 NIFI-8462 Refactored PutSyslog and ListenSyslog using Netty
- Added nifi-event-transport module encapsulating Netty classes
- Refactored unit tests for PutSyslog and ListenSyslog
- Removed integration tests for PutSyslog and ListenSyslog

NIFI-8462 Added context.yield() in PutSyslog when no FlowFiles and addressed other issues

NIFI-8462 Removed unused import of ExpressionLanguageScope

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5044.
2021-05-25 10:54:15 -04:00
exceptionfactory 01783a295c NIFI-8037 Changed SSLContextServices to use runtime supported protocols for TLSv1.3
Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4827.
2021-05-23 23:52:01 -04:00
Lehel Boér a3eaf0a37a NIFI-8325: Complete SNMP refactor: SNMP GET and SET processors reworked, unit tests added
This closes #5028.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2021-05-19 23:26:31 +02:00
Joe Gresock ed591e0f22
NIFI-8445: Implemented HashiCorpVaultCommunicationService in nifi-vault-utils
This closes #5034

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-14 12:27:01 -05:00
Bence Simon 51aae5bcf6 NIFI-8519 Adding HDFS support for NAR autoload
- Refining classloader management with the help of @markap14

This closes #5059

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-14 13:14:06 -04:00
exceptionfactory 6776765a92
NIFI-8538 Upgraded Apache Commons IO to 2.8.0
- Upgraded direct dependencies from 2.6 to 2.8.0
- Added dependency management configuration to use 2.8.0 for some modules
- Updated scripted Groovy tests to avoid copying unnecessary files

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5073
2021-05-14 09:31:48 -04:00
Sandro Tolaini fdda3582e3
NIFI-8470: Fix DataTypeUtils for CHAR() sequences
NIFI-8470: Added unit test

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #5027
2021-05-11 17:46:38 -04:00
Matthew Burgess 284322feed
MINIFI-554: Move OSUtils to nifi-bootstrap-utils for MiNiFi and NiFi
This closes #5062

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-10 08:39:10 -05:00
Matthew Burgess 30fc26647e
NIFI-6061: Fix CLOB/BLOB handling in PutDatabaseRecord
NIFI-6061: Force getBytes() in BLOB handling to use UTF-8 charset

NIFI-6061: Use setClob(), added unit tests, incorporated review comments

This closes #5049

Co-authored-by: zhangcheng <zhangchengk@foxmail.com>
Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-04 14:58:14 -05:00
exceptionfactory 13d5be622b NIFI-8230 Removed default Sensitive Properties Key and added random generation
- Retained legacy default Sensitive Properties Key in ConfigEncryptionTool to support migration
- Streamlined default file path and moved key generation conditional
- Refactored with getDefaultProperties()
- Cleared System Property in ConfigEncryptionToolTest
- Added checking and error handling for clustered status
- Added set-sensitive-properties-key command
- Refactored PropertyEncryptor classes to nifi-property-encryptor
- Added nifi-flow-encryptor
- Refactored ConfigEncryptionTool to use FlowEncryptor for supporting AEAD algorithms
- Added Admin Guide section Updating the Sensitive Properties Key

This closes #4857.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2021-05-04 08:49:34 -04:00
Paul Grey cc554a6b11
NIFI-8478 Addressed i18n test issues; CI supplies locale to surefire
This closes #5040

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-05-03 13:40:01 -05:00
Joe Gresock 54a0e27c93
NIFI-7134: Adding auto-reloading of Keystore and Truststore
- NIFI-7261 Included TrustStoreScanner for auto-reloading of truststore

This closes #4991

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-29 08:29:42 -05:00
Joe Gresock 90c7d03ed3
NIFI-8403: Generating Self-signed cert on startup when applicable (#4986)
* NIFI-8403: Implementing auto-generated certificates for secure startup

* Adding check for passwords in SecureNiFiConfigUtil
2021-04-27 09:34:44 -04:00
Paul Grey 17fa0cf3c1
NIFI-3580 Add configurable TLS Cipher Suite properties
This closes #5018

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-22 11:31:19 -05:00
exceptionfactory ed6d5bacba NIFI-8451 Updated KeyStoreUtils to use KeyStore.getInstance() with provider
- Refactored and consolidated KeyStoreUtils unit tests
- Corrected KeyStoreUtils.loadEmptyKeyStore() to use KeyStoreUtils.getKeyStore()

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #5015.
2021-04-21 13:45:50 -04:00
Tamas Palfy a50957161c NIFI-8365 Fix JSON AbstractJsonRowRecordReader to handle deep CHOICE-typed records properly: change the logic that selects the first compatible schema which can have missing fields compared to the real value and search for a more strict match first and fallback to the existing logic only if not one found.
- AbstractJsonRowRecordReader - Handle (meaning log a warning and not fail completely) multi-array CHOICE type when data has extra fields (not defined by the schema) and can't determine correct type.
- AvroTypeUtil - Allow multiple different record types in avro union type. Minor refactors. Added documentation fro EqualsWrapper.
2021-04-19 12:56:09 -04:00
Mark Payne 7d1d536da6
NIFI-8206: Added identifiesExternalResource() method to PropertyDescriptor.Builder and implemented functionality.
- Updated components to make use of new feature

NIFI-8206: Added a ResourceType of TEXT. This requires that the ResourceReferenceFactory know which types are allowed in order to create the ResourceReference. PropertyValue needs to then have the PropertyDescriptor available to it. This resulted in highlighting many bugs in unit tests where components were not exposing property descriptors via getSupportedPropertyDescriptors() or were evaluating Expression Language using the wrong scope, so fixed many unit tests/components to properly declare Expression Language scope when using it

NIFI-8206: Removed problematic unit test that required directory names with special characters that are not allowed on some operating systems

This closes #4890.

Signed-off-by: Bryan Bende <bbende@apache.org>
2021-04-12 11:48:04 -04:00
Nathan Gough 9da3b1ec01
NIFI-7912 - Added properties to configure DoSFilter timeout and whitelisted addresses
- Added nifi.web.request.ip.whitelist property to set DoSFilter.ipWhitelist
- Added nifi.web.request.timeout property to set DoSFilter.maxRequestMs with default of 60 seconds

This closes #4972

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-04-07 07:40:03 -05:00
Mark Payne 7db1b8d564
NIFI-8386: Ensure that we set (and merge) bulletins when creating AffectedComponent entities and ControllerService Reference entities
NIFI-8386: Addressed review feedback: removed unused call to determine permissions, null out bulletins in standalone mode if permissions not allowed. Also fixed automated tests that were failing due to changes

This closes #4955
2021-04-01 14:11:32 -04:00
noblenumbat360 ee1386db02
NIFI-8317 heap and non_heap usage Prometheus metrics now return -1.0 when undefined.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4920
2021-03-30 11:06:52 -04:00
Matthew Burgess a5dbf56114
NIFI-8368: If decimal scale > precision, set precision = scale
This closes #4938

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-29 10:53:46 -05:00
exceptionfactory 1cd3fbb4eb NIFI-8288 Removed OkHttpClientUtils to reduce reliance on nifi-security-utils
- Added createTrustManager() on SSLContextService
- Removed nifi-security-utils and okhttp dependencies from nifi-web-utils

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4869.
2021-03-25 15:38:24 -04:00
Mark Payne 91313a2e75
NIFI-8360: Fixed an overflow issue where we used an integer to store the number of bytes encountered when reading data and searching for a given pattern
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4929.
2021-03-24 15:30:22 +01:00
Mark Bean 1719e36165 NIFI-8348: upgrade jersey version to one fully compatible with Java 11 2021-03-24 09:18:31 -04:00
Nathan Gough 07a4966d10
NIFI-8329 - Updated dependencies with no build failures
NIFI-8329 - Removed unnecessary jackson.version from azure bundle to use the global property instead.

NIFI-8329 - Updated jackson/jackson-databind version and removed the 'jackson-databind.version' pom property in favor of 'jackson.version'

Updated dependencies include the following:

- jackson-core
- jackson-databind
- icu4j
- snakeyaml
- spring-integration-mail
- spring-core and framework modules
- activemq-client
- activemq-broker
- xercesImpl

This closes #4911

Signed-off-by: David Handermann <exceptionfactory@apache.org>
2021-03-19 14:46:33 -05:00