Commit Graph

1617 Commits

Author SHA1 Message Date
mtien 0ae200252f
NIFI-6978 added code and unit test to throw IllegalArgumentException when improper value given
This closes #3960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2020-01-09 13:50:27 -08:00
Bryan Bende 08d0352ac1
NIFI-6985 Use correct versioned parameter contexts when child process groups are version controlled
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3962.
2020-01-09 17:35:25 +07:00
Mark Payne c958deb5b0 NIFI-6822: Ensure that when we manage a Map of ID -> Count, that we properly merge those maps during a checkpoint instead of overwriting existing values 2020-01-06 13:05:19 -05:00
Mark Payne 5f0f801e46
NIFI-6879: Added system tests for variables. Fixed bug that resulted in Variable Registry not being updated if a Processor in a child group referenced it
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3949.
2019-12-26 18:05:08 +01:00
Mark Payne 2277d08c8e
NIFI-6949: When a Controller Service is removed, ensure that any other service that it references is obtained via the ControllerServiceProvider to ensure that it is obtain to obtain and remove the reference, even if the service is in a higher-level Process Group
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3933.
2019-12-20 16:17:48 +00:00
Joe Ferner 058883091c
NIFI-6872:
- Added UI versioned flow supportsDownload functionality with download flow menu item
- Added VersionsResource endpoint for downloading versioned flow with registry-related info removed
- Added ProcessGroupResource endpoint for downloading current flow with registry-related info removed
- Added StandardNifiServiceFacade functionality for downloading both current and versioned flow
- Added XmlTransient markers on variables introduced by Instantiated model classes so they do not appear in serialized download
- Updated NiFiRegistryFlowMapper.mapParameterContexts to handle mapping nested parameter contexts for use in producing a complete VersionedFlowSnapshot
- Added ability for NiFiRegistryFlowMapper to map nested process groups ignoring versioning for use in producing a complete VersionedFlowSnapshot
- Added unit tests where helpful

NIFI-6872: PR response...
- Updated mapParameterContext to return a Map to handle uniqueness of contexts by name since ultimately everything converted it to a map anyway. The VersionedParameterContext class from the registry model doesn't support hashcode/equals currently so returning a Set wouldn't work.
- Updated assert calls to put expected value as first parameter and actual as second parameter
- Added one time password (OTP) support for flow download endpoint to support non cert based authentication

This closes #3931
2019-12-18 16:37:47 -05:00
Mark Payne e05d11c0b4 NIFI-6787: Added a comment to the UI tooltip, user guide, and javadocs to indicate that round robin may skip a node if that node is not receiving the data as fast as other nodes in the cluster, in order to maximum throughput. 2019-12-13 15:09:15 -05:00
Tamas Palfy 7731609582 NIFI-6787 - Before: When checking if a load balanced connection queue is full, we compare the totalSize.get() and getMaxQueueSize().
After: Go over all partitions and see if all of them are full.
Wrapping RoundRobinPartitioner in a (new) AvailableSeekingPartitioner which selects a new partition if the currently selected one is full.
2019-12-13 14:36:44 -05:00
Iván Rodriguez 65ba4a2d93 NIFI-6853 - flowFileEvent combineCounters hashmap overwritten
- Added unit test to verify behavior of contribution

This closes #3875.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-12-13 14:23:50 -05:00
Mark Payne 452ca98c29
NIFI-6924: When seeking to the appropriate offset for a content claim, ensure that if there are not enough bytes in the underlying resource claim that a ContentNotFoundException is thrown. Also cleaned up error-handling case in StandardProcessSession to ensure that we close the existing InputStream before calling handleContenttNotFoundException, since this method may itself throw an Exception
This closes #3924.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-12-10 15:27:53 -05:00
Bryan Bende dd620680b1
NIFI-6904 Moving ClassLoader creation before Authorizer instantiation
This closes #3903
2019-11-25 13:35:21 -05:00
Pierre Villard ac5bacccb8
NIFI-6839 - Upgrade jackson-databind direct dependencies
This closes #3870

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-11-25 10:58:22 -05:00
Andrew Lim 5bfc86737d
NIFI-6747 Updated images used by Swagger REST API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3901.
2019-11-22 14:05:36 +01:00
Joe Ferner 3f3b778f24 NIFI-6893
Upgraded angular from 1.7.2 to 1.7.9

This closes #3899

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-11-21 16:45:33 -05:00
Mark Payne ea31c1fcbb
NIFI-6868: Ensure that when a property value is obtained by calling ProcessContext.getPropertyValue(...).evaluateAttributeExpressions(...).getValue() that we still retain the appropriate value even if the property references a sensitive parameter
* Added additional unit tests and updated ParameterEvaluator to make sure that it follows the same logic as the ParameterExpression in that sensitive parameters cannot be access from within EL

This closes #3889
Signed-off-by: Rob Fellows <rob.fellows@gmail.com>
2019-11-14 15:37:39 -05:00
Joe Ferner c947e57d5e
NIFI-6814:
- Restored JAXB Adapter

This closes #3880
2019-11-08 15:43:13 -05:00
Mark Payne 5f4ce8f431
NIFI-6707: Addition of nifi-system-tests module. Updates to EmbeddedNiFi, NiFi and SystemBundle to properly support classloader isolation that is needed for creating system-level tests and isolating the classpath of the tests themselves from the NiFi classpath. Fixed bootstrap listener to ensure that it always closes socket before calling shutdown(). Failure to do so can result in RunNiFi sometimes timing out while waiting for a response from NiFi
NIFI-6707: Added System Test to verify behavior when services depend on one another

NIFI-6707: Ensure that when tearing down flow after test, we wait for all processors and controller services to complete stop/disable before attempting to delete them

NIFI-6707: Fixed bug in RemoteProcessGroupIT that caused the test to fail intermittently based on timing. Improved logging for Process Group, Standard Process Group

NIFI-6707: Updated to include java11 subdirectory under lib/

NIFI-6707: Rebased against master and changed dependencies from 1.10.0-SNAPSHOT to 1.11.0-SNAPSHOT

NIFI-6707: Updates to see if it will help travis build correctly

NIFI-6707: Commenting out java commands in bootstrap.conf files

This closes #3831.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-11-08 12:33:16 -05:00
Mark Payne 49b7a7cd6b
NIFI-6846: If an Exception is thrown while a Processor is writing to a FlowFile, but that Content Claim is not yet eligible for destruction, mark it as a transient claim on the RepositoryRecord so that if it's available when the FlowFile Repository is checkpointed, then it will be cleaned up then
This closes #3872
2019-11-06 09:53:25 -05:00
Andy LoPresto 12bb284453
NIFI-6841 Fixed checkstyle error introduced in a9db5a8. 2019-11-05 13:12:40 -08:00
Mark Payne a9db5a8cb7
NIFI-6841: Fixed bug that resulted in the wrong number of 'Bytes Read' being reported by ByteCountingInputStream in the event that #skip was called between calls to #mark and #reset
This closes #3868
2019-11-05 13:06:06 -05:00
Jan Hentschel d6b4bce668 NIFI-4758 Changed 'Name' to 'Flow Name' in Import Version dialog 2019-11-04 15:26:25 -05:00
mtien 7f1886adad NIFI-4757 Capitalized 'version' in tool tip. 2019-11-04 15:22:52 -05:00
Joe Witt f8c3d877cf
NIFI-6733 updating to next release version for master branch 2019-11-04 13:31:39 -05:00
Joe Witt 4119279c03
Merge branch 'NIFI-6733-RC3' 2019-11-04 13:24:29 -05:00
Joe Witt 363b751310
Revert "NIFI-6733 bumping master to 1.11.0-SNAPSHOT"
This reverts commit a7c56cfd9a.
2019-11-04 13:24:13 -05:00
Joe Witt a7c56cfd9a
NIFI-6733 bumping master to 1.11.0-SNAPSHOT 2019-11-04 07:54:17 -05:00
Mark Payne 0b3a60ae3e NIFI-6832: When receiving a 0-byte FlowFile via site-to-site, avoid writing to the FlowFile at all 2019-11-01 16:22:17 -04:00
Kotaro Terada 245e055d5d
NIFI-6828 There is an unintentional string in "conf/logback.xml"
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3863.
2019-10-31 11:20:21 +01:00
mtien 3d56e2f26d
NIFI-4348 handled NPE in isExpressionLanguagePresent.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3857.
2019-10-30 09:58:55 +01:00
Mark Payne ce5eae5b2c NIFI-6800: Added unit test to verify behavior of contribution 2019-10-29 10:17:27 -04:00
Ivan Ezequiel Rodriguez 67f1677212 NIFI-6800 - Fix hashmap counter overwritten in highThroughputSession
This closes #3837.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-29 10:16:49 -04:00
Joe Witt 418179f5b2
NIFI-6733-RC3 prepare for next development iteration 2019-10-28 15:13:13 -07:00
Joe Witt b217ae20ad
NIFI-6733-RC3 prepare release nifi-1.10.0-RC3 2019-10-28 15:12:57 -07:00
Joe Witt dda29cd90b
NIFI-6733 updating zk to latest incremental 2019-10-28 12:40:03 -07:00
Jan Hentschel ccf85777c4
NIFI-6816 Removed duplicated dependency declarations
This closes #3851

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-28 12:49:08 -04:00
Matt Gilman 9364f7d2e8
NIFI-6811:
- Allowing enterGroup to complete prior to loading the controller service table.

This closes #3846.
2019-10-25 12:07:03 -04:00
Mark Payne 3f270f184c
NIFI-6807: When a Controller Service's state is transitioned to ENABLING, complete the Future successfully, even if the Controller Service is not valid. The Controller Service will remain in the ENABLING state until it is made valid, at which point it will ENABLE (unless explicitly disabled first). This allows us to Enable a Controller Service and its referencing components, even if the referencing component is still invalid due to it not yet recognizing the the referenced service has been enabled.
This closes #3841
2019-10-25 11:38:34 -04:00
Mark Payne 81fc2768c9
NIFI-6798: When checking that referenced Controller Services are valid, use ValidationContext.getProperties() rather than ComponentNode.getSupportedPropertyDescriptors() because the latter does not include dynamic properties
This closes #3836.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-25 09:52:44 -04:00
Mark Payne 8079d2531e
NIFI-6805, NIFI-6806: Updated toString() of StandardControllerServiceNode so that it includes the component's type and UUID, updated the more generic Exception Mappers to include stack traces regardless of the log level that is enabled
This closes #3840

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-24 19:27:30 -04:00
Yolanda M. Davis 147365285c
NIFI-6801 - fix to ensure unique model instance is created for each connection. Also increased default query interval.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3838
2019-10-24 08:51:50 -04:00
Matthew Burgess ace23c35b5 NIFI-6780: Add QueryNiFiReportingTask, RecordSinkService, S2S and DB impls (#3826)
* NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils

Added QueryNiFiReportingTask to SQL query NiFi status and metrics

Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS

Add CONNECTION_STATUS_PREDICTIONS

check for null predictions

Add ConnectionStatusRecursiveIterator

Fix issue w/ duplicate iterator outputs

Refactored query interfaces, fixed assembly POM

Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors

Add DatabaseRecordSink service (#13)

* Add DatabaseRecordSink service

* Incorporated review comments

* NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions

* Added bundle profile, remove predictions table if not enabled

* Added doc for which tables are available when

This closes #3826.
2019-10-22 08:56:42 -04:00
Andy LoPresto d148fb1854
NIFI-3834 This closes #3821. Added encrypted content repository implementation.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-21 13:03:59 -04:00
Matt Gilman 8b23929638
NIFI-6786:
- Upgrading nimbus-jose-jwt.

This closes #3830.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-21 11:41:29 -04:00
samhjelmfelt ea1becac4f
NIFI-6703: Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes

This closes #3795.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-17 15:46:32 -04:00
Mark Payne b004f1f94c
NIFI-6779: When resolving node identifier, if NodeIdentifier has a different value for hostname/port for web api or cluster protocol, assume that node is correct about itself instead of assuming that Cluster Coordinator knows best about the other node
NIFI-6779: Remove any conflicting Node Identifiers when a new Node ID is encountered

This closes #3819
2019-10-17 11:21:17 -04:00
Mark Payne 9dd0dda688
NIFI-6772: Improved the information that is emitted about Garbage Collection when a Diagnostics Dump is performed
This closes #3809
2019-10-14 11:32:47 -04:00
Joe Witt a273ff10f9
NIFI-6758 fixing checkstyle issues 2019-10-10 15:04:16 -04:00
Matt Gilman 7db6ba841f
NIFI-6768:
- Ensuring the parameter context resources are available via the /resources endpoint.

This closes #3808.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:49:39 -04:00
Endre Zoltan Kovacs 5238dc20b1
NIFI-6758: allow pre-unpacked nar's to be recorded in ExtensionMapping
- fixing resource leak

This closes #3806.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:36:29 -04:00
Matt Gilman ed1e843609
NIFI-6769:
- Ensuring the users policy listing can handle when the specified policy is unknown.

This closes #3807.
2019-10-10 11:30:14 -04:00