Commit Graph

6147 Commits

Author SHA1 Message Date
Mark Payne d84583690f NIFI-8070: Added coalesce function to RecordPath 2020-12-03 17:23:11 -05:00
Matthew Burgess 04aaf25131 NIFI-1121: Use display name for dependent property documentation 2020-12-03 16:39:46 -05:00
mtien 8055c47a84
NIFI-8058 Fixed a UI error to correctly delete dynamic properties while configuring processors.
Changed to check the length of all unfiltered properties instead of only filtered properties.
Added additional check if descriptor is a dynamic property.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4707
2020-12-03 15:44:56 -05:00
Bryan Bende 312fa8e85e NIFI-8064 Convert TestSecureClientZooKeeperFactory to integration test 2020-12-03 14:46:13 -05:00
Bryan Bende 2b1359a808 NIFI-8060 Addressed review feedback 2020-12-03 14:42:50 -05:00
Bryan Bende 8ac8a2bd1f NIFI-8060 Added minimal VolatileProvenanceRepository to nifi-stateless-engine and remove dependency on nifi-volatile-provenance-repo module 2020-12-03 14:42:50 -05:00
Mike Thomsen b90a6e893d
NIFI-7906 This closes #4701. Updated test case to fix a windows-centric bug.
NIFI-7906 Removed unused test code.

Signed-off-by: Joe Witt <joewitt@apache.org>
2020-12-03 07:22:40 -07:00
Levi Lentz c29cced269
NIFI-7906: parameterized graph query
NIFI-7906: addressing PR concerns
NIFI-7906: code styling fixes
NIFI-7906: adding in license information to new files
+ enables processor in META-INF
NIFI-7906: exclude test files from RAT
NIFI-7906: PR refactor to streamline graph response
NIFI-7906: removing ERRORS output
Unused after refactor
Did a few cleanups for the contributor.

This closes #4638

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-01 11:13:10 -05:00
Matthew Burgess 64e3599f05
NIFI-7260: Fix error handling and re-evaluate Module Directory property on changed for scripted controller services
This closes #4147

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-12-01 09:42:14 -05:00
exceptionfactory 1e13b62e78
NIFI-8047 Added documentation for sensitive DBCP properties
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4696
2020-11-30 18:33:32 -05:00
markap14 aaa1452d04
NIFI-8054: Updated ReflectionUtils to use a WeakHashMap for the mapping of annotations to methods with that annotation. This way, the ReflectionUtils class will not hold a reference to Classes that are no longer referenced elsewhere. (#4694) 2020-11-30 16:14:12 -05:00
exceptionfactory fe53f8090d
NIFI-8047 Added support for sensitive dynamic properties in DBCP
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4692
2020-11-30 14:55:20 -05:00
Bence Simon 39f8a008d4 NIFI-8039 Adding properties to ListenTCP in order to allow refine behaviour under higher load; Refining thread pool for better scalability
NIFI-8039 Review findings; refining thread pool to be able to scale down properly when not under load
NIFI-8039 Answers to PR comments

This closes #4689.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-30 10:17:15 +01:00
Mike Thomsen 3625a73d91
NIFI-7821 Added Cassandra-based DMC.
NIFI-7821 Updated configuration documentation.

NIFI-7821 Fixed getAndPutIfAbsent and added int test.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4635
2020-11-29 18:58:57 -05:00
Matthew Burgess fe950131c3 NIFI-8046: Fix issue with ResultSetRecordSet on DB2
This closes #8046.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-27 19:23:44 +01:00
Koji Kawamura 5ea18f39b5
NIFI-8048 PutElasticsearchRecord should retry with 503 response
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4690.
2020-11-26 10:49:52 +01:00
Chris Sampson 124cdbd3fe
NIFI-6403 and NIFI-6404: Elasticsearch 7 support
Addressed PR#4153 comments; removed ES Version property and made Type optional in all ES HTTP/Record processors, applying sensible default values where required; use _source queyr parameter instead of _source_include/s as it's compatible between ES versions

Fix unit test compilation to use JDK8-compatible library/method

Better optional type and id handling for PutElasticsearchRecord; update nifi-elasticsearch-client-service build dependencies to use latest versions of Elasticsearch in each supported major version (5/6/7); addressed several warnings in ElasticSearchClientServiceImpl

This closes #4667.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-11-26 15:08:13 +09:00
Joe Gresock a3d845a38f
NIFI-6403: Adding Elasticsearch7 support to HTTP processors
This closes #4153.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2020-11-26 15:08:05 +09:00
zhangcheng 857eeca3c7
NIFI-8032: fix record-path-guide.adoc
This closes #4679

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-24 18:01:16 -05:00
Mark Payne 7ad9520079
NIFI-8041: Refactored API for stateless nifi so that calling StatelessDataflow.trigger() returns an object that allows the caller to wait for the result, cancel the result, etc. and then optionally acknowledge the completion. This allows the caller to block the completion of ProcessSession.commit() until it has handled the output of the dataflow execution.
NIFI-8038: Fixed deadlock that can occur when updating Parameter Context

This closes #4684.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-24 15:12:51 -05:00
Mark Payne 3c9d8a7007
NIFI-7897: Refactoring to create a new nifi-framework-components module.
- Refactored nifi-stateless to make use of nifi-framework-components
- Removed requirement for nifi-framework-nar to be provided.
- Refactored stateless nifi into api, engine, nar, and bootstrap modules, with a parent 'bundle' module
- Creation of nifi-stateless-system-tests
- Added unit tests and logging
- Changed flow configuration to use properties file instead of json
- Allow for -p parameter to specify parameters on command line
- Moved implementations of Authorizer, NiFiUser, and UserGroupProviders to new module named nifi-framework-authorization-providers so that those that depend on nifi-framework-authorization don't have to bring in the providers. This way, we can have stateless not bring in those providers, as we otherwise get warnings on startup about the provider already being registered. Additionally, it avoids needing dependencies on spring-security-core
- Updated bin/nifi.sh script to run new stateless bootstrap
- Added Reporting Tasks to stateless.
- Download bundles as necessary on stateless nifi startup

NIFI-7897: Addressing review feedback

NIFI-7897: Fixed typos in README and also addressed issue that caused parameters with spaces not to be parsed properly

This closes #4669.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-24 11:37:45 -05:00
Arek Burdach 87fd9f968f
NIFI-7958 ConvertAvroToORC - decimal support
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #4628
2020-11-23 13:00:54 -05:00
Mike Thomsen 932496e106 NIFI-7879 Created record path function for UUID v5
This closes #4570.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2020-11-20 12:10:01 -06:00
Denes Arvay 2bf5d1a56d NIFI-8017 NPEx at org.apache.nifi.provenance.index.lucene. IndexDirectoryManager:486
Added an extra null-check to avoid NullPointerException
2020-11-19 16:01:42 -05:00
exceptionfactory dc6ba2541f NIFI-8024 Added null claim check to EncryptedFileSystemRepository.read() to avoid EOFException 2020-11-19 09:11:49 -05:00
humpfhumpf 40fdcd8144 NIFI-7771 - Infinite loop on WebUI when node stopped in cluster (PB1)
This commit resolves "Problem 1" described in issue #7771.

Signed-off-by: Nathan Gough <thenatog@gmail.com>

This closes #4496.
2020-11-18 14:56:59 -05:00
Bryan Bende dcc4fb00a5
NIFI-7888 Added support for authenticating via SAML
- Add dependency on spring-security-saml2-core
- Updated AccessResource with new SAML end-points
- Updated Login/Logout filters to handle SAML scenario
- Updated logout process to track a logout request using a cookie
- Added database storage for cached SAML credential and user groups
- Updated proxied requests when clustered to send IDP groups in a header
- Updated X509 filter to process the IDP groups from the header if present
- Updated admin guide
- Fixed logout action on error page

- Updated UserGroupProvider with a default method for getGroupByName
- Updated StandardManagedAuthorizer to combine groups from request with groups from lookup
- Updated UserGroupProvider implementations with more efficient impl of getGroupByName
- Added/updated unit tests

- Ensure signing algorithm is applied to all signatures and not just metadata signatures
- Added property to specify signature digest algorithm

- Added option to specify whether JDK truststore or NiFi's truststore should be used when connecting to IDP over https
- Added properties to configure connect and read timeouts for http client

- Added URL encoding of issuer when generating JWT to prevent potential issue with the frontend performing base64 decoding

- Made atomic replace methods for storing groups and saml credential in database

- Added properties to control AuthnRequestsSigned and WantAssertionsSigned in the generated service provider metadata

- Dynamically determine the private key alias from the keystore and remove the property for specifying the signing key alias

- Fixed unit test

- Added property to specify an optional identity attribute which would be used instead of NameID

- Cleaned up logging

- Fallback to keystore password when key password is blank

- Make signature and digest default to SHA-256 when no value provided in nifi.properties

This closes #4614
2020-11-18 12:44:02 -05:00
Mark Payne 7848ba59a2
NIFI-8021: Provide the ability to pin partitions to particular hosts when using ConsumeKafka processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4672.
2020-11-18 16:03:24 +01:00
Matthew Burgess edc060bd92 NIFI-7989: Add UpdateHiveTable processors for data drift capability
NIFI-7989: Allow for optional blank line after optional column and partition headers
NIFI-7989: Incorporated review comments
NIFI-7989: Close Statement when finishing processing
NIFI-7989: Remove database name property, update output table attribute

This closes #4653.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-18 00:39:07 +01:00
Tamas Palfy 4aaec5aa38
NIFI-8020 Make sure TailFile doesn't leave FileChannel open when handling NUL characters
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4671.
2020-11-18 00:26:27 +01:00
Kevin Doran a0328ff8d1
NIFI-7744 Add Support for Unicode in X-ProxiedEntitiesChain (#4664)
- Adds detection and encoding of non-ascii characters to creation of chain
- Adds unit tests that use proxied entities with Unicode
2020-11-17 10:23:30 -05:00
r65535 d5dc63ded9
NIFI-7982 Added tags to FlattenJSON processor
Adding missing imports

This closes #4662

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-17 07:05:52 -05:00
Tamas Palfy 47cc2867ba NIFI-7972 Add a boolean property by which the user can tell the processor to yield (and try again later) whenever it encounters a NUL character. 2020-11-16 15:16:12 -05:00
Chris Sampson 5c925371c2
NIFI-7474 Allow index operation type to be provided for PUT Elasticsearch processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4665.
2020-11-16 18:19:35 +01:00
Denes Arvay 83948db989
NIFI-7996 Conversion with ConvertRecord to avro results in invalid date
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4666.
2020-11-16 17:40:40 +01:00
Peter Gyori e66362194d NIFI-7922: Added support for GET in ListenHTTP for health check
NIFI-7922: Fixes based on review comments

ListenHTTP: changed if(!sslRequired) to if(sslRequired) so that the positive case comes first.
HealthCheckServlet: response body for GET contains "OK".
ContentAcknowledgmentServlet: super.doDelete() is called when a DELETE should be rejected because of port mismatch.

NIFI-7922: Refactored, based on review comments

NIFI-7922: Fixed a checkstyle violation (organized the imports)

This closes #4603.

Signed-off-by: Peter Turcsanyi <turcsanyi@apache.org>
2020-11-16 12:20:30 +01:00
s9514171 c57d0abcd3 NIFI-6820 If Processor adjusts a counter but does not interact with any FlowFiles, the counter isn't updated 2020-11-13 11:25:34 -05:00
Nathan Gough 8a4079cd78
NIFI-7993 - Upgraded Jetty dependency and fixed a minor issue with Jetty security filter tests.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4658.
2020-11-13 13:27:12 +01:00
Mark Payne c79ad1502e
NIFI-7999: Do not call NioAsyncLoadBalanceClient.nodeDisconnected() if node was already in a disconnected state. Doing so was resulting in that method being called constantly on startup, and with the synchronization in place that can result in a huge performance hit on startup. Also updated RemoteQueuePartition to move a small predicate into its own method. This was done because the predicate was previously defined within a synchronized method, which meant that invoking that predicate required synchronization.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4657.
2020-11-12 17:44:46 +01:00
Peter Turcsanyi 018778a25d NIFI-7994: Fixed ReplaceText concurrency issue
Moving RegexReplace.additionalAttrs field to local variable in replace() method.
Concurrent usage of the additionalAttrs map caused data corruption.
2020-11-12 10:45:10 -05:00
Nathan Gough 479ee6e3db
NIFI-7819 - Added ZooKeeperStateProvider TLS properties.
- Added tests for TLS with ZooKeeperStateProvider.
- Added docs to administration guide.
- Small fixes for PR comments.
- Changed the ZooKeeperStateProvider to receive configuration from the nifi.properties file. Uses the Zookeeper TLS properties or if they are not declared, uses the standard NiFi TLS properties.
- Updated administration-guide.
- Fixed some boolean literalsl. Set the ZooKeeper watcher to null. Removed stacktrace prints to standard out. Added getPreferredProperty for key/truststore types.
- Removing some unused code. Fixing up NiFi properties methods. Removed whitespace.
- Added some tests for getPreferredProperty().
- Checkstyle fixes.
- Passing through nifi properties to the state provider using an annotation to avoid ZooKeeper references in the StateManagerProvider.
- Fixed comment.
- Added CLIENT_SECURE property to isZooKeeperTlsConfigurationPresent() check.
- Small change to getPreferredProperty, added more tests.
- Added checkstyle fix.
- Moved StateProviderContext to nifi-framework-api.
- Changed combine properties to handle null NiFiProperties. Inject NiFiProperties object for tests.
- Checkstyle fix.
- Changed the connect string in state-management.xml to be required. Rearranged order of property validation to validate before initialization.
- Rearranged the way ZooKeeperClientConfig is initialized and added a non blank validator to connect string.
- Minor change to ZooKeeperClientConfig member variable set and get.

This closes #4613.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-10 17:05:05 -05:00
Mark Payne badcfe1ab7
NIFI-7992: Periodically check disk usage for content repo to see if backpressure should be applied. Log progress in background task. Improve performance of background cleanup task by not using an ArrayList Iterator and constantly calling remove but instead wait until the end of our cleanup loop and then removed from the list all elements that should be removed in a single update
This closes #4652.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-10 15:27:51 -05:00
tpalfy 940bc3056c
NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() (#4629)
* NIFI-7954 Wrapping HBase_*_ClientService calls in getUgi().doAs() and taking care of TGT renewal.

* NIFI-7954 Simplified SecurityUtil.callWithUgi a little.

* NIFI-7954 Simplified SecurityUtil.callWithUgi more.

* NIFI-7954 Removed unnecessary code.
2020-11-09 09:00:20 -05:00
Pierre Villard 14ec02f21d
NIFI-7981 - add support for enum type in avro schema
This closes #4648

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-05 18:19:55 -05:00
Siyao Meng 9a3a659c44
NIFI-7949: Add tag HCFS to HDFS Processors
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4633.
2020-11-05 13:21:11 +01:00
Bryan Bende 1366d017b8
NIFI-7968 Ensure the status login user is set in UserGroupInformation after creating a UGI from a Subject
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #4643.
2020-11-05 13:18:35 +01:00
Bence Simon 59e00c4b6f NIFI-7959 Handling node disconnection in MonitorActivity processor
- Make reporting in clustered scope to dependent of expected cluster state in order to prevent unexpected flow file emission

This closes #4642.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2020-11-04 10:09:46 -05:00
Dustin Rodrigues 0805670263
NIFI-7978 - update Homebrew getting started instructions
NIFI-7978 Removed a few more OS X references.

This closes #4647

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-04 07:20:21 -05:00
Matthew Burgess 71a5735f63
NIFI-7976: Infer ints in JSON when all values fit in integer type
NIFI-7976: Fixed unit test

This closes #4645

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2020-11-03 15:43:15 -05:00
Bryan Bende d773521ee0
NIFI-1121 Fix Schema Name and Schema Branch properties
This closes #4512.

Signed-off-by: Bryan Bende <bbende@apache.org>
2020-11-03 15:38:41 -05:00