Commit Graph

3710 Commits

Author SHA1 Message Date
Mark Payne 3543b9c42c
NIFI-6797: Add support for specifying Partition via EL or RecordPath for PublishKafka(Record)_1_0 and PublishKafka(Record)_2_0
This closes #3834.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-23 14:59:33 -04:00
Matthew Burgess ace23c35b5 NIFI-6780: Add QueryNiFiReportingTask, RecordSinkService, S2S and DB impls (#3826)
* NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils

Added QueryNiFiReportingTask to SQL query NiFi status and metrics

Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS

Add CONNECTION_STATUS_PREDICTIONS

check for null predictions

Add ConnectionStatusRecursiveIterator

Fix issue w/ duplicate iterator outputs

Refactored query interfaces, fixed assembly POM

Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors

Add DatabaseRecordSink service (#13)

* Add DatabaseRecordSink service

* Incorporated review comments

* NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions

* Added bundle profile, remove predictions table if not enabled

* Added doc for which tables are available when

This closes #3826.
2019-10-22 08:56:42 -04:00
Andy LoPresto d148fb1854
NIFI-3834 This closes #3821. Added encrypted content repository implementation.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-21 13:03:59 -04:00
Yolanda M. Davis 7c9d34f820
NIFI-6778 - Added rules engine service api and Easy Rules Implementation
NIFI-6778 - added comments and component description annotations

NIFI-6778 - added license and notice details

NIFI-6778 - added additional details section and updated description of service

NIFI-6778 - fixed checkstyle error

NIFI-6778 - addressed review comments for documentation and description corrections

NIFI-6778 - added include-rules profile for assembly build

NIFI-6778 - add MVEL and ASM to license and notice

NIFI-6778 - switch to use Jackson for NiFi Rules Json deserialization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3824
2019-10-21 12:16:11 -04:00
Matt Gilman 8b23929638
NIFI-6786:
- Upgrading nimbus-jose-jwt.

This closes #3830.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-21 11:41:29 -04:00
samhjelmfelt ea1becac4f
NIFI-6703: Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes

This closes #3795.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-17 15:46:32 -04:00
Mark Payne b004f1f94c
NIFI-6779: When resolving node identifier, if NodeIdentifier has a different value for hostname/port for web api or cluster protocol, assume that node is correct about itself instead of assuming that Cluster Coordinator knows best about the other node
NIFI-6779: Remove any conflicting Node Identifiers when a new Node ID is encountered

This closes #3819
2019-10-17 11:21:17 -04:00
Andrew Rodriguez cd90b1d3e1
NIFI-6773 This closes #3820. Adding lz4-framed option for compression/decompression
NIFI-6773 Adding two unit tests for lz4-framed compression/decompression
NIFI-6773 Adding compressed lz4 file for decompression test

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-16 17:45:32 -04:00
Mark Bean c4ccd5f505
NIFI-6750: This closes #3810. add FETCH provenance event to FetchFile
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-15 14:52:39 -04:00
Bryan Bende d570fe8154
NIFI-4820 This closes #3813. Improving security configuration for Kafka 2.0 processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-15 14:05:25 -04:00
Matthew Burgess 77e9bda0ff
NIFI-6764: Recreate GroovyShell in ExecuteGroovyScript only when necessary
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3803.
2019-10-15 11:55:52 +02:00
Mark Payne 9dd0dda688
NIFI-6772: Improved the information that is emitted about Garbage Collection when a Diagnostics Dump is performed
This closes #3809
2019-10-14 11:32:47 -04:00
JF Beauvais 08fe648b1c
[NIFI-6374] ConsumeEWS fails when email attachment has no content type - ensure non null content type
This closes #3596.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-10-11 15:10:11 +09:00
Bryan Bende af81afce40
NIFI-5753 Make use of keyPassword optional and only used when keystore is used, bump registry client version
This closes #3126.
2019-10-10 15:09:28 -04:00
Grzegorz Kołakowski 5d65e6aba4
NIFI-5753 Add SSL support to HortonworksSchemaRegistry service
Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 15:09:16 -04:00
Joe Witt a273ff10f9
NIFI-6758 fixing checkstyle issues 2019-10-10 15:04:16 -04:00
Matt Gilman 7db6ba841f
NIFI-6768:
- Ensuring the parameter context resources are available via the /resources endpoint.

This closes #3808.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:49:39 -04:00
Endre Zoltan Kovacs 5238dc20b1
NIFI-6758: allow pre-unpacked nar's to be recorded in ExtensionMapping
- fixing resource leak

This closes #3806.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:36:29 -04:00
Matt Gilman ed1e843609
NIFI-6769:
- Ensuring the users policy listing can handle when the specified policy is unknown.

This closes #3807.
2019-10-10 11:30:14 -04:00
markap14 d64f4fa942 NIFI-6760: When writing/reading the length of a DataFrame, do so usin… (#3801)
* NIFI-6760: When writing/reading the length of a DataFrame, do so using a 4-byte integer instead of a 2-byte short. When using uncompressed data, we know that the length of the DataFrame will be no more than 64 KB so a 2-byte short is sufficient. However, if data is compresed, there's a chance that the compressed form of the data will be larger than the uncompressed form (for example, with random binary data or with encrypted data). In this situation, we can end up overflowing the 2-byte short, which causes the length that is written to be wrong. Using a 4-byte integer avoids this situation.

* NIFI-6760: Fixed unit tests
2019-10-10 10:19:21 -04:00
Matt Gilman 4e8dd6557f
NIFI-6756:
- Fixing issue opening a users policy listing when there is a policy for a specific parameter context.

This closes #3805.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-10 00:55:47 -04:00
Matt Gilman 99e9010b32
NIFI-6766:
- Ensuring policy label is properly escaped when populating the user's access policy listing.
This closes #3804.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-10 00:19:22 -04:00
Mark Payne 5414cd5016 NIFI-6759: When encountering Exception during load-balanced connection communications, after closing socket, return from method rather than continuing loop
This closes #3800.
2019-10-09 16:05:44 -04:00
Matt Gilman 9a496fe9d2
NIFI-6751:
- Fixing the identifier on the user table. In a previous task, this was changed to utilize the URI but that does not work with other code interacting with this table.

This closes #3798.
2019-10-09 12:00:22 -04:00
Mark Payne 47f45abdf9
NIFI-6748: Fixed bug in Parameter Contexts' affected components where if a Controller Service referenced a Parameter, any component that references that service should also be considered an affected component but wasn't. Also fixed a bug in how we handled stopping a Processor that was in the STARTING phase.
This closes #3794.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-09 10:20:45 -04:00
Mark Payne e394f6683a
NIFI-6753: Fixed bug where all values being provided to the CSV Writer were String objects, which resulted in the CSV Writer improperly quoting numeric values when the schema indicates that the value is a number. Now, we will only convert the value to a String if the value is not a Number and/or the schema does not indicate a numeric ty type
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3797
2019-10-09 09:54:45 -04:00
Kevin Doran 2d90145842
NIFI-6749 This closes #3792. Fix Swagger for ControllerServiceReferencingComponentDTO
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-08 11:48:10 -04:00
Joe Witt 0f02de6002
NIFI-6733 updating key apache commons dependencies and apache base dependency for build
This closes #3791.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2019-10-07 22:20:48 -04:00
Matthew Burgess b588073cbb
NIFI-6741: Add RecordPath support to scripting components
This closes #3790.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-04 17:37:53 -07:00
Peter Turcsanyi ba141690c5
NIFI-6734: Fixed S3 multipart upload in case of SSE S3 and CSE* encryptions.
Removed unnecessary code from S3 CSE* encryptions.
S3 Encryption Service documentation fixes and improvements.
Renamed region property of StandardS3EncryptionService to kms-region.
Renamed Client-side Customer Master Key in StandardS3EncryptionService.
Use Client-side Customer Key on the GUI / documentation (similar to
Server-side Customer Key).
Use C suffix in constants and class names (similar to SSE_C).
Fixed / extended StandardS3EncryptionService validation.
FetchS3Object encryption strategy changes.
Disable SSE S3 and SSE KMS for FetchS3Object. In case of fetching the
S3 object, these strategies are handled implicitly / automatically.
Set the encryption strategy on the fetched FF that was used to store
the S3 object, instead of the one that is used to read the object (eg.
non-encrypted or SSE S3 encrypted objects can be fetched with a CSE client).
Typo fix.

This closes #3787.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-04 15:57:06 -07:00
Rob Fellows df90c65246 NIFI-6735 - validate components before restarting processors following parameter context update.
This closes #3782.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-04 15:48:09 -04:00
Peter Wicks 555004cdde
NIFI-6684 - Fixing checkstyle violation (#3788)
signed-off by: Peter Wicks <patricker@gmail.com>
2019-10-04 08:41:56 -06:00
Mark Payne 03473b922b NIFI-6001: When adding Controller Services from a Versioned Flow, make a second iteration over any newly added Controller Services and update them again. This is done so that any linkage between the Controller Services happens properly, once all services have been created. 2019-10-03 16:41:55 -04:00
Kotaro Terada d7ca37d065
NIFI-6723: Enrich processor-related and JVM GC metrics in Prometheus Reporting Task
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3771
2019-10-03 13:17:42 -04:00
Mike 02d3b7e92b NIFI-6656 Added a default visibility expression configuration item to… (#3723)
* NIFI-6656 Added a default visibility expression configuration item to the HBase map cache client services.

* NIFI-6656 Fixed validation bug.

* NIFI-6656 Added stylecheck fix.

* NIFI-6656 Deleted shared interface and moved property descriptor into both services.

* NIFI-6656 Removed dependency from hbase api.
2019-10-03 12:53:53 -04:00
Bryan Bende 9c9f9c10a9 NIFI-5910 Retain external service reference when already set to an existing external service
This closes #3783.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-10-03 12:32:46 -04:00
James Cheng f1d35f46ce NIFI-6684 Add more property to Hive3ConnectionPool (#3763)
* NiFi-6684 Add more property to Hive3ConnectionPool

signed-off by: Peter Wicks <patricker@gmail.com>
2019-10-03 07:24:29 -06:00
Pierre Villard b12a9ad446
NIFI-6159 - Add BigQuery processor using the Streaming API
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3394.
2019-10-03 11:37:00 +02:00
Kotaro Terada a0a66839c4
NIFI-6715: Metrics of removed/renamed components continues to remain in PrometheusReportingTask
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3766.
2019-10-03 11:33:49 +02:00
Kotaro Terada 261a395992
NIFI-6716: Prometheus Metrics Server returns 500 when a Remote Process Group used
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3767.
2019-10-03 11:07:58 +02:00
Mark Payne 99cf87c330
NIFI-6736: Create thread on demand to handle incoming request from client for load balancing. This allows us to avoid situations where we don't have enough threads and we block on the server side, waiting for data, when clients are trying to send data in another connection
This closes #3784.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-02 14:29:00 -04:00
Mark Payne 2f6f852915 NIFI-6589: Addressed NPE 2019-10-02 10:55:55 -04:00
Jeff Storck 8d748223ff
NIFI-6275 ListHDFS now ignores scheme and authority when uses "Full Path" filter mode
Updated description for "Full Path" filter mode to state that it will ignore scheme and authority
Added tests to TestListHDFS for listing an empty and nonexistent dirs
Updated TestListHDFS' mock file system to track state properly when FileStatus instances are added, and updated listStatus to work properly with the underlying Map that contains FileStatus instances
Updated ListHDFS' additional details to document "Full Path" filter mode ignoring scheme and authority, with an example
Updated TestRunners, StandardProcessorTestRunner, MockProcessorInitializationContext to support passing in a logger.

NIFI-6275 Updated the "Full Path" filter mode to check the full path of a file with and without its scheme and authority against the filter regex
Added additional documentation for how ListHDFS handles scheme and authority when "Full Path" filter mode is used
Added test case for "Full Path" filter mode with a regular expression that includes scheme and authority

This closes #3483.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-10-02 17:08:37 +09:00
Alessandro D'Armiento 39a258dc38
NIFI-6727 Fixed the MockProcessContext decrypt bug and added unit test.
This closes #3773.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-01 12:53:44 -07:00
Joe Witt 8abf330328
Revert NIFI-6703 as it creates startup issues or logs a confusing message to the user. Details on JIRA
This reverts commit 61baa41e7b.
2019-10-01 15:33:07 -04:00
Koji Kawamura 6541eac625
NIFI-6598 Storing peers into managed-state
- Fixed checkstyle errors.
- Added PeerPersistence interface.
- Expose RemoteProcessGroup state via REST API
- Made stateManager transient.

This closes #3677.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-01 09:56:10 -04:00
samhjelmfelt 61baa41e7b
NIFI-6703: This closes #3762. Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 23:23:15 -04:00
Mark Payne fbd6200ab3
NIFI-6589: This closes #3670. Cache results from zookeeper when determining the leader
NIFI-6589: Updated CuratorLeaderElectionManager to cache results for no more than 5 seconds per review feedback

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 16:47:27 -04:00
samhjelmfelt 2ab99970b7
NIFI-6662: Adding Kudu Lookup Service
NIFI-6662: Cleaning up Kudu logic
NIFI-6662: Minor enhancements and build fixes
NIFI-6662: This closes #3732.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 14:38:33 -04:00
Rob Fellows 2493665c27 NIFI-6587 - Fix: Unable save sensitive property value that equals masked value
This closes #3778

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-30 13:18:08 -04:00
Mark Payne 6dccdd586e
NIFI-6726: Updated fingerprint logic to ensure that Controller Services are fingerprinted when housed within a ProcessGroup
This closes #3775.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-30 11:55:46 -04:00
Mark Payne 5562c587fe
NIFI-6132: This closes #3776. Do not default ot G1GC because it has known issues with Java 8 and earlier versions
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-30 11:51:27 -04:00
David Mollitor 48759be974
Updated license information dnsjava 2019-09-30 10:34:02 -04:00
David Mollitor d5a7fc5daa
NIFI-6542: Upgrade nifi-hadoop-bundles Hadoop version dependency to 3.2.0 2019-09-30 10:33:50 -04:00
Peter Turcsanyi a4c3ca50dd
NIFI-6550: Create controller service for Azure Storage Credentials
Added AzureStorageCredentialsService controller service interface
with 2 implementations:
 - AzureStorageCredentialsControllerService: holds the credential properties
   (account name, account key, sas token)
 - AzureStorageCredentialsControllerServiceLookup: can be used to dynamically
   look up another AzureStorageCredentialsService (similar to
   DBCPConnectionPoolLookup)

The controller service can be used by the Azure Storage processors:
{List|Fetch|Put|Delete}AzureBlobStorage, {Get|Put}AzureQueueStorage

NIFI-6550: Review changes.

NIFI-6550: Review changes #2.

This closes #3742.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-27 01:36:28 +09:00
archongum aba450d658
NIFI-6695: ExtractEmailAttachments/ExtractEmailHeaders should not validate 'SendDate'
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3757.
2019-09-24 20:51:57 +02:00
eduardofontes c7a2885c0c
NIFI-6679 - MoveHDFS removes original file when destination exists
- Include testPutWhenAlreadyExisting test
- Resolve dependencies

This closes #3746.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-24 14:29:25 -04:00
Mark Payne c721a9ee5f
NIFI-6696: Ensured that callback to RemoteQueuePartition do not attempt to obtain the Partition Read Lock. The Read Lock is not necessary as long as the 'partitioner' is volatile, because it doesn't matter whether or not the actual partitions themselves change, since the only partition that would be touched is the Rebalancing Partition,on, which is fixed. Obtaining the partition read lock can lead to a deadlock as outlined in the Jira description.
This closes #3760.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-23 10:07:21 -04:00
Joe Witt 6d8968cc87
NIFI-6520 update to tika 1.22
This closes #3756

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-23 08:18:55 -04:00
samhjelmfelt 17a16ba68b
NIFI-6692: Fixing NullPointerException when not using Kerberos
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3755.
2019-09-21 21:37:25 +02:00
Rob Fellows 870a9529c5
NIFI-6693 - This closes #3759. replace assertTrue with assertEquals. use isNull matcher.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-20 15:40:29 -04:00
Scott Aslan 219a83b023
[NIFI-6689] reset private var on dialog close
This closes #3752
2019-09-20 15:21:38 -04:00
thenatog 282c271c7c NIFI-6578 - Upgraded zookeeper framework version. Some code changes required. This change also required a change to the embedded zookeeper.
- Updating tests to account for the new node /zookeeper/config
Upgraded Zookeeper and Curator versions for zookeeper migrator in toolkit. Updated tests to allow for new znode /zookeeper/config
- Added documentation changes to reflect changes to the zookeeper.properties file going foward.

This closes #3715.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-20 10:49:04 -04:00
Mark Payne e18d9ce1e8
NIFI-6369: This closes #3560. Updated JSON Readers to convert String values to Date/Time/Timestamp objects when appropriate according to the schema and the configured pattern
NIFI-6939: Fix to WriteJsonRecord to deal with date/time/timestamp when no format is explicitly set
2019-09-19 23:16:34 -04:00
Mark Payne d0371a5ef1
NIFI-6691: This closes #3754. Fixed bug in MapRecord's algorithm for incorporating Inactive Fields, by recursing into any 'child fields' that contain records; also fixed bug in UpdateRecord that caused it to incorrectly map the first result to all elements if the key of the property points to an array 2019-09-19 22:53:39 -04:00
Joe Witt 13381c2254
NIFI-5816 converting test to integration test as it is test environment specific 2019-09-19 21:46:06 -04:00
Tamas Palfy 34112519c2 NIFI-6640 - UNION/CHOICE types not handled correctly
3 important changes:
1. FieldTypeInference had a bug when dealing with multiple datatypes for
 the same field where some (but not all) were in a wider-than-the-other
 relationship.
 Before: Some datatypes could be lost. String was wider than any other.
 After: Consistent behaviour. String is NOT wider than any other.
2. Choosing a datatype for a value from a ChoiceDataType:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.
3. Conversion of a value of avro union type:
 Before it chose the first compatible datatype as the basis of conversion.
 After change it tries to find the most suitable datatype.

Change: In the RecordFieldType enum moved TIMESTAMP ahead of DATE.

This closes #3724.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-19 11:02:30 -04:00
Bryan Bende ff6a7d9561
NIFI-5816 This closes #3726. Remove unused SFTP classes that reference Jsch
NIFI-5816 Switching SFTP processors from JSCH to SSHJ
NIFI-5816 LICENSE/NOTICIE updates to reflect changing from JSch to SSHJ

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-19 10:58:53 -04:00
FaizFizy Rosle f6842a5200
Fix syntax error in schema example
This closes #3745

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-19 07:56:42 -04:00
Matt Gilman 171e37ddda
NIFI-6400 Fixing checkstyle issue. 2019-09-18 14:01:52 -04:00
Troy Melhase 4596ef7c8a
NIFI-6400 Better options, consistent ids for ShellUserGroupProvider.
NIFI-6400 Fixes to address PR feedback.
NIFI-6400 Accepts proposed changes.

This closes #3637
2019-09-18 09:08:41 -04:00
Mark Payne fd3f0707c6 NIFI-6677: Update ListHDFS to clear state (when appropriate) in an @OnScheduled method, just as AbstractListProcessor does, instead of doing it in onTrigger. Doing it in onTrigger is problematic because in a cluster, the Primary Node may run for some period of time, perhaps days or months. Then, when the Primary Node chagnes, onTrigger gets called for the first time on the new Primary Node, and this triggers the processor to clear state. 2019-09-17 10:02:06 -04:00
Rob Fellows 9e2a959778 NIFI-6659 - Open create new parameter context dialog in edit mode.
This closes #3729

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-16 11:16:25 -04:00
Rob Fellows 7e9277a2bc
NIFI-6671 - sort parameters by name in the 'Reference Parameter' dropdown.
This closes #3736
2019-09-16 10:03:10 -04:00
Rob Fellows 5ca3655dbf
NIFI-6381 - Make Parameters and Parameter Contexts searchable in UI
NIFI-6381 - remove wildcard imports

This closes #3728
2019-09-13 16:52:14 -04:00
archon 8a8852e73d
NIFI-6480: PutORC/PutParquet can't overwrite file even if set 'Overwrite Files' to true
This closes #3599.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 12:08:27 -04:00
Mark Payne 7623e6f5a1
NIFI-6024: When fetching names of buckets and flows for registry, sort them alphanumerically
This closes #3709.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 11:21:32 -04:00
Mark Payne eb6085a31d
NIFI-6658: Implement new bin/nifi.sh diagnostics command that is responsible for obtaining diagnostic information about many different parts of nifi, the operating system, etc.
This closes #3727.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-13 10:45:53 -04:00
Koji Kawamura e659e3b606 NIFI-5952 Refactor RAW S2S from nio to socket 2019-09-13 10:30:04 -04:00
Mark Payne af94b035cb NIFI-6660: Fixed ordering of directory creation and timestamp gathering
This closes #3731.
2019-09-13 09:34:13 -04:00
korir 3d1bb09ff8
fix deserialization issues with NiFiRecordSerDe for hive3streaming
NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types

NIFI-6295: Incorporated review comments

NIFI-6295: Refactored unit tests for NiFiRecordSerDe
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3684
2019-09-13 09:08:57 -04:00
Troy Melhase 93e6f195d9
NIFI-6596 Moves AmazonS3EncryptionService interface
to `nifi-aws-service-api` package.

This closes #3694.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-13 08:55:05 +09:00
Yolanda M. Davis 5106b764da
NIFI-6649 - repaired test due to min zero fix
NIFI-6649 - added check if logging debug is enabled

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3730
2019-09-12 16:08:31 -04:00
Rob Fellows 75c47388a6 NIFI-6630 - Add a goto action in the property table for properties that reference parameters.
NIFI-6630 - base convert and goto parameter logic on any reference to a parameter contained in the text of the property value.

This closes #3718

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-12 13:23:32 -04:00
Rob Fellows 2f7448c9e5
NIFI-6634 - Indicate variables are no longer recommended and favor parameters
This closes #3721
2019-09-12 12:44:52 -04:00
Yolanda M. Davis 8e1452a3f3
NIFI-6649 - added separate query interval configuration for observation queries
NIFI-6649 - documentation update

NIFI-6649 - add debug logging for score and prediction information

NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3719
2019-09-12 12:13:19 -04:00
Scott Aslan c187292fd9
[NIFI-6629] calc correct width for usage ellipsis
This closes #3708
2019-09-11 15:06:56 -04:00
Scott Aslan d9c8d0c5a7
[NIFI-6625] add ellipsis and tooltip to list of parameters in usage list
This closes #3707
2019-09-11 13:27:55 -04:00
Scott Aslan 250e1b0297
[NIFI-6639] consistent ux for checkboxes and their correspoinding field
This closes #3706
2019-09-11 13:09:32 -04:00
Hsin-Ying Lee 758035b964 NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some cir… (#3673)
* NIFI-6567 HandleHttpRequest does not shutdown HTTP server in some circumstances

signed-off by: Peter Wicks <patricker@gmail.com>
2019-09-11 09:27:52 -06:00
Rob Fellows 29708d8362 NIFI-6641 - fix issue with dialog state when adding a new parameter context. added ellipsis styling and titles to name and id fields.
This closes #3710

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-10 10:50:52 -04:00
Rob Fellows 70af2acc06
NIFI-6641 - support read only mode for parameter context dialog 2019-09-10 10:03:06 -04:00
Rob Fellows 5ddc01eb9f
NIFI-6644 - Consider param context permissions when deciding to allow property conversion to parameter.
NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings.

This closes #3713
2019-09-09 16:11:20 -04:00
Mark Payne f89ea8cf21
NIFI-6645: Fixed problem in unit test for StandardParameterContext because behavior changed; fixed issue in StandardParameterContext around adding a parameter with no description
This closes #3714.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-09 15:21:21 -04:00
Peter Turcsanyi 21a27c8bb0 NIFI-6636: Fixed ListGCSBucket file duplication error
ListGCSBucket duplicated files if they arrived not in alphabetical order.
The set storing the name of the latest blob (which was loaded with the highest
timestamp during the previous run of the processor) was cleared too early.

Also changed the state persisting logic: it is now saved only once at the end
of onTrigger() (similar to ListS3). Some inconsistent state (only blob names
without the timestamp) was also saved earlier.

This closes #3702.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-09 13:32:17 -04:00
Andy I. Christianson 8a8b9c1d08
NIFI-6510 - Analytics framework (#3681)
* NIFI-6510 Implement initial analytic engine

* NIFI-6510 Implemented basic linear regression model for queue counts

* NIFI-6510 Initial analytics REST endpoint and supporting objects

* NIFI-6510 Connect the dots for StatusAnalytics -> API

* NIFI-6510 Added poc engine with prediction model caching

(cherry picked from commit e013b91)

DFA-9 - updated logging and corrected logic for checking if not in backpressure

(cherry picked from commit a1f8e70)

* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics

(cherry picked from commit 050e0fc)

(cherry picked from commit 9fd365f)

* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id

(cherry picked from commit 14854ff)

DFA-9 - reduced snapshot interval to 1 minute

(cherry picked from commit 36abb0a)

* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions

* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly

* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"

This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.

* NIFI-6510 Added prediction fields for use by UI, still need to be populated

* NIFI-6510 Analytics Framework Introduction (#10)

* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.

DFA-9 - added logging

* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval

* DFA-9 - checkstyle fixes

* NIFI-6510 Add prediction percent values and predicted interval seconds

(cherry picked from commit e60015d)

* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.

(cherry picked from commit 6fae058)

* NIFI-6510 Added tests for engine

(cherry picked from commit 6d7a13b)

* NIFI-6150 Added tests for connection status analytics class, corrected variable names

(cherry picked from commit 58c7c81)

* NIFI-6150 Make checkstyle happy

(cherry picked from commit b6e35ac)

* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing

* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics

* NIFI-6510 Adjusted interval and incorporated R-squared check

Updates to support multiple variables for features, clearing cached regression model based on r-squared values

Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).

Ratcheck fixes

Added test for SimpleRegression. Minor fix for OLS model

fixed test errors

fixed checkstyle errors

(cherry picked from commit fab411b)

* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)

* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties

NIFI-6566 - changes to allow scoring configurations for model in nifi.properties

NIFI-6566 - added default implementation value to NiFiProperties

NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3663

* NIFI-6585 - Refactored tests to use mocked models and extract functions.  Added check in ConnectionStatusAnalytics to confirm expected model by type

* NIFI-6586 - documentation and comments

This closes NIFI-6586

Signed-off-by: Andrew I. Christianson <andy@andyic.org>

* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns

* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.

* NIFI-6510 - add an enable/disable property for analytics

* NIFI-6510 - documentation updates for enable/disable property

* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)

* NIFI-6510 - admin guide updates to further describe model functionality

* NIFI-6510 - code quality fixes (if statement and constructor)

* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold

* NIFI-6510 Extract out predictions into their own DTO

* NIFI-6510 Optimize imports

* NIFI-6510 Fix formatting

* NIFI-6510 Optimize imports

* NIFI-6510 Optimize imports

* NIFI-6510 - Notice updates for Commons math and Caffeine

* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)

* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3705

* NIFI-6510 Rip out useless members

* NIFI-6510 - dto updates to check for -1 value

* NIFI-6510 - checkstyle fix

* NIFI-6510 - rolled back last change and applied minNonNegative method

* NIFI-6510 Rip out useless members
2019-09-09 15:37:11 +00:00
Matt Gilman 625d4a13ca NIFI-6612:
- Updating context menu to only show the Parameters menu item when the user has the appropriate permissions.
2019-09-09 10:24:28 -04:00
Matt Gilman f867e92606 NIFI-6384:
- Adding auditing of parameter contexts.

This closes #3699.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-06 15:20:24 -04:00
Rob Fellows 7af710f168
NIFI-6637 - Display Parameter Context Id on Update Parameter Context dialog.
This closes #3701
2019-09-06 14:15:38 -04:00
Rob Fellows b9c0f51356 NIFI-6632 - Add tooltip to parameter value field indication it does not support EL or parameter references.
This closes #3703

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-06 13:42:23 -04:00
Mark Payne 731df87f22
NIFI-6626: Fixed the order of arguments passed to the ComponentDetails constructor when creating the Component Details for a connection. Also noticed that null fields are being added to the map when calling ComponentDetails.toMap() and this causes the UI to display these fields even though they have no value, so avoided populating the map with fields whose values are null
This closes #3696
2019-09-06 13:24:10 -04:00
samhjelmfelt a672294f3f
NIFI-6621: Add support for Druid schema-less dimensions
NIFI-6621: Small change to fix failing tests

NIFI-6621: Minor style changes
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3693
2019-09-06 13:18:34 -04:00
Mark Payne 4868308fd8
NIFI-6380: Fixed partial updates to parameters
This closes #3692
2019-09-06 11:28:53 -04:00
Rob Fellows 9aed19137c NIFI-6623 - enter process group before navigating to the controller service
This closes #3695

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-06 09:31:46 -04:00
Rob Fellows 710c02c673 NIFI-6623 - Support linking to unauthorized components from referencing components 2019-09-06 08:10:28 -04:00
Arpad Boda f01668e66a
NIFI-6530 - HTTP SiteToSite server returns 201 in case no data is available
This closes #3647.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-06 19:27:59 +09:00
Rob Fellows 92fd3129ba NIFI-6602 - Fix: Apply button should be greyed out if no changes have been made
This closes #3691

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-05 14:54:47 -04:00
Rob Fellows fa8a4462e7 NIFI-6506 - don't show 'Sensitive value set' for null, sensitive params'
This closes #3683

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-05 13:39:48 -04:00
Rob Fellows 3c918e25bd NIFI-6506 - better empty-string/null handling for edit parameters 2019-09-05 13:39:48 -04:00
Rob Fellows 5cbade7b1d NIFI-6505 - fix for editing parameter value. simplify logic in edit dialog for disabled state of the apply button. 2019-09-05 13:39:48 -04:00
Rob Fellows a52d17628a NIFI-6506 - review feedback 2019-09-05 13:39:47 -04:00
Rob Fellows 183ece194e NIFI-6506 - update parameter context list after successful update. 2019-09-05 13:39:47 -04:00
Rob Fellows 93d99820b7 NIFI-6506 - more review fixes 2019-09-05 13:39:47 -04:00
Rob Fellows 5936fe3487 NIFI-6506 - navigate to process group when selecting to go to controller service from referencing components on parameter context dialog. 2019-09-05 13:39:47 -04:00
Rob Fellows 385c067507 NIFI-6505 - fixing issues uncovered in code review. 2019-09-05 13:39:47 -04:00
Rob Fellows 10b4f6fd89 NIFI-6506 - review feedback
* fixed issues converting sensitive props to parameters
* only include parameter value if it has changed (even for non-sensitive params)
* do not allow null parameter values when editing parameters.
2019-09-05 13:39:47 -04:00
Rob Fellows aa07378a03 NIFI-6505 - Addressing review feedback: reuse single method for getting parameter field values. 2019-09-05 13:39:47 -04:00
Rob Fellows f3a04521c2 NIFI-6506 - Fix syntax/style issues. Apply same logic for disabled state when converting prop to param. 2019-09-05 13:39:47 -04:00
Rob Fellows eddc5624d8 NIFI-6506 - Add ability to convert properties to parameters
* open existing add param dialog from property table
* convert prop to param, update param context
* keep add parameter dialog open until update of the context completes when converting property to param
* Updating parameter context while converting property to parameter.
* Disable the Apply button while updating
* When Cancel is pressed and there is an update in progress, issue the DELETE request to cancel the update.
* Show some status in the Add Parameter dialog when parameter context is updating following a property conversion.
* enforce character restrictions on property names.
* Add convert property to parameter to controller service config.
* Properly set sensitive setting when converting sensitive properties
* Allow converting of properties from controller services configuration into parameters
* Refactor addNewParameter method to be able to share common parts with convertPropertyToParameter
2019-09-05 13:39:47 -04:00
Brandon 78d9f13ded
NIFI-6416: This closes #3654. Removing swap locations on SWAP_IN
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-09-04 17:47:59 -04:00
Bryan Bende 6731e4c8ee
NIFI-6089 Improve javadoc for newly added inputLength argument in RecordReaderFactory 2019-09-04 11:49:19 -04:00
Bryan Bende 76c2c3fee2
NIFI-6089 Add Parquet record reader and writer
NIFI-5755 Allow PutParquet processor to specify avro write configuration
Review feedback
Additional review feedback

This closes #3679

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-09-04 08:52:17 -04:00
Alessandro D'Armiento 9071e5baa7
- Removed unused AUTOCREATE_PARTITIONS from PutHive3Streaming
- Renamed PARTITION_VALUES to STATIC_PARTITION_VALUES for correctness and better understanding
- STATIC_PARTITION_VALUES descriptions clearly states that having that property filler implies Hive Static Partitioning

NIFI-6536: Additional documentation for Static Partition Values

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3631
2019-09-03 15:45:08 -04:00
Mike Thomsen 768a7b8c00
NIFI-6425 Made executeQuery able to reconnect to the Gremlin cluster.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3572
2019-09-03 15:06:22 -04:00
Alessandro D'Armiento 4c6c1cbb14
NIFI-6536 EncryptContent accepts configurable PGP symmetric cipher
Additional test case in TestEncryptContent
Apply suggestions from code review
nit picking
Apply suggestions from code review
nitpicking from mgaido91
Small fixes
Additional test case to check the provided cipher is actually used by inferring it from the ciphertext
Updated `EncryptContent` to use a pre-defined list of allowable PGP ciphers
Updated `EncryptContent` to check the validity of `PGP_SYMMETRIC_ENCRYPTION_CIPHER`
Updated test cases
Minor fixes
NIFI-6536 Fixed typo and strengthened validity checking for PGP symmetric cipher.

This closes #3664.

Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-09-03 10:59:20 -07:00
Mark Payne a2edc2aa87
NIFI-6025: Fixed bug that caused Flows to indicate that they have been locally modified (when they hadn't been) when running against versions of the registry older than 0.5.0
This closes #3687.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-09-03 13:48:05 -04:00
Scott Aslan 459ef24e99
[NIFI-6606] disable PC for which the user does not have read permissions when configuring a PG. Also, sort the list of PCs for the combo dropdown in the PG configuration
This closes #3690
2019-09-03 13:28:06 -04:00
Mark Payne 4851e00f1d
NIFI-6607: Ensure that when creating ParameterContextUpdateRequestDTO that we populate the AffectedComponentEntities with the most up-to-date version of the component
This closes #3689
2019-09-03 12:55:55 -04:00
Matt Gilman 07e1c11520 NIFI-6608:
- Conditionally support parameters in controller service configuration.

This closes #3688

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-09-03 12:08:19 -04:00
Bryan Bende 180a188096 NIFI-6028 Protect against null VersionedComponent when finding relevant process group
This closes #3686.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-09-03 11:32:03 -04:00
Mark Payne 9ba280680f
NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occur with site-to-site. Fixed bug that caused site-to-site listener not to accept connections if there are no input/output ports on the root group - this used to be a valid check and was done to prevent spawning extra threads and doing extra work if no ports exist but now that we have site-to-site ports outside of the root group it's no longer a reasonable condition to check.
This closes #3658.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-02 11:51:03 +09:00
SandishKumarHN 26b203616e
NIFI-6552 - Kudu Put Operations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3610.
2019-09-01 13:34:55 +02:00
Mike Thomsen fc3477bd69
NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher services.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3571
2019-08-30 14:01:44 -04:00
Matt Gilman bfbf3b7532 NIFI-6290:
- Addressing review feedback.

This closes #3636

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-08-30 13:53:12 -04:00
Matt Gilman c6e500dfa4 NIFI-6290:
- Ensuring the first allowable value is selected.
2019-08-30 13:51:37 -04:00
Matt Gilman f318a8c685 NIFI-6290:
- Ensuring the first allowable value is selected.
2019-08-30 13:51:37 -04:00
Matt Gilman e5850fa1a3 NIFI-6290:
- Addressing timing issue when parameters listing request is not immediately resolved.
2019-08-30 13:51:37 -04:00
Matt Gilman 5b409d6fac NIFI-6418:
- Updating property table editors to not support parameter referencing when necessary.
2019-08-30 13:51:37 -04:00
Matt Gilman d18c98ce78 NIFI-6290:
- Updating comboEditor to allow for parameter reference.
2019-08-30 13:51:37 -04:00
Matt Gilman d93ae47afc NIFI-6346:
- Updating nfel to support referencing parameters.
2019-08-30 13:51:37 -04:00
Matt Gilman 7228496801 NIFI-6345:
- Updating property table editor to distinguish between available modes (el vs parameter reference).
2019-08-30 13:51:37 -04:00
Matt Gilman 75e452b45c NIFI-6277:
- Allow parameters to be referenced from properties that do not support EL.
2019-08-30 13:51:37 -04:00
Yolanda Davis 07baf723f7 NIFI-6429 - Added optional field to allow variables with null values to be included in reporting records. Default functionality will exclude variables with null values for backwards compatibility.
NIFI-6429 - checkstyle corrections
NIFI-6429 - change property descriptor name
2019-08-30 13:38:38 -04:00
Mark Payne 254a84d74d
NIFI-6371: Fixed bug that caused root-level Controller Services to stay cached and therefore continually be validated, even after being deleted. Also cleaned up code that looked for specific text in Exceptions instead of of catching specific Exceptions
This closes #3614.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-29 16:24:40 -04:00
Scott Aslan f678c75d70
[NIFI-6282] manage parameters and parameter contexts
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
2019-08-29 14:48:53 -04:00
Mark Payne 4ae1fec78a
NIFI-6025: Include Processor 'scheduled state' (i.e., Enabled or Disabled) in the VersionedProcessor when pushing to Flow Registry and take into account when updating flows on the NiFi side
NIFI-6025: Include difference in Scheduled State as a Local Flow Difference

This closes #3546.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-29 10:35:03 -04:00
Evan Reynolds e2ca50e66a
NIFI-6367 - This closes #3563. more error handling for FetchS3Object
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-08-28 19:42:47 -04:00
Bryan Bende b78aeb67a1 NIFI-6382 Fix verifyUpdate for parameter contexts to account for deletions
This closes #3661.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-28 11:29:00 -04:00
Mark Payne 59734dad88
NIFI-6028: Updates to ignore changes between local version of a flow and a remote version of a flow if the difference is the addition of a new property that has the default value
NIFI-6028: Code refactoring to address review feedback

This closes #3544.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-28 10:38:56 -04:00
Mark Payne 3c9426d287
NIFI-6595: Fixed bug in TailFile that caused it not to properly honor the Initial Start Position after state has been restored
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3675.
2019-08-28 10:33:13 +02:00
Andy LoPresto fbd201f116
NIFI-6298 Upgraded versions of Spring Framework in nifi-spring-processors and nifi-email-processors.
NIFI-6412 Upgraded Spring Security version in nifi-framework-bundle pom.xml.
NIFI-6412 Updated version for spring.security.version in nifi-kerberos-iaa-providers and nifi-ldap-iaa-providers pom.xml files.

This closes #3564.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
2019-08-27 18:58:04 -04:00
Tamas Palfy 3de81d618e
NIFI-6509 - Date related issue in unit test VolatileComponentStatusRepositoryTest
As NIFI-6508 has already solved the original issue, turned this PR into a refactor within in the affected Test class.

This closes #3621.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 11:02:35 +09:00
Bryan Bende cc88dd428f
NIFI-6158 Fix conversion of Avro fixed type with logicalType decimal
This closes #3665.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 10:57:28 +09:00
Peter Wicks 0b74795578
NIFI-6562 Expose ExecutionNode in ProcessContext
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3656.
2019-08-20 11:13:55 +02:00
Malthe Borch 00f0f84856
NIFI-6508 - Fix test failure caused by timezone and/or DST difference
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-20 11:55:54 +09:00
Koji Kawamura 1cb1b00c09
NIFI-6477 Delete operate policy when its component is deleted
+1 from markobean.

This closes #3653.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-20 10:59:00 +09:00
Mike Thomsen e822a1612b
NIFI-6572 Added a workaround for GetMongoRecordIT that fixes a weird issue with Groovy where the Groovy addons don't appear to be getting recognized for java.util.Date. 2019-08-19 21:15:56 -04:00
Matt Gilman 326bc91405
NIFI-5839:
- Introducing case insensitive group membership.
- Ensuring user identity and group name mapping is applied when the user identity or group name is inferred through group membership decisions.

This closes #3657.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-19 17:04:05 -04:00
Koji Kawamura e32689a602 NIFI-6436 Fix NPE at StandardPublicPort
Removing unnecessary ProcessGroup from PublicPort constructor.

This closes #3580.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-19 15:33:26 -04:00
Mark Payne bd8342c5e0
NIFI-6382: On flow import and version change, create parameter context where necessary and add parameters where necessary
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context

NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry

NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException

This closes #3642.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:49 -04:00
Mark Payne bcf373a049
NIFI-6382: Allow Parameters to have null values
Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:27 -04:00
andrew 9c8f40415e
NIFI-6554 fix oid parsing for PutMongo processor
This closes #3650

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-08-16 05:45:48 -04:00
Brandon Devries 5f696482b0 NIFI-4775: Updating documentation and updating rocksdb version
NIFI-4775: Rolling back fix for NIFI-6416

NIFI-4775: Documentation improvements

+1 from markobean

This closes #3648

Signed-off-by: Brandon <devriesb@apache.org>
2019-08-15 15:28:48 +00:00
Jeff Storck 33187f8058 NIFI-5176 NiFI builds on Java 11
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build.  Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
  Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
  Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
  URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.

NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service

NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11

This closes #3404.
2019-08-13 18:41:39 -04:00
Peter Turcsanyi b99cecd4e7
NIFI-6468: TestListS3 assertion fix
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3649.
2019-08-14 00:16:17 +02:00
Troy Melhase e841f4d5b7
NIFI-4256 Adds AWS Encryption Controller Service.
NIFI-4256 Adds AWS S3 FlowFile encryption attributes, more javadocs,
better names.

This closes #3574.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-08-13 12:18:46 -07:00
Joe Gresock 32c46f0bdd
NIFI-6468: Adding AWS S3 'requester pays' to Fetch and List processors.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3601.
2019-08-13 09:15:10 +02:00
Brandon Devries 7d77b464cc NIFI-4775: FlowFile Repository implementation based on RocksDB
+l from markobean.

This closes #3638.

Signed-off-by: Brandon <devriesb@apache.org>
2019-08-12 19:03:38 +00:00
Tamas Palfy 2baafcc2e6
NIFI-6534 Improve logging in PutHive3Streaming processor
Added logging for PutHive3Streaming when routing to failure

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3640
2019-08-09 17:19:26 -04:00
Alessandro D'Armiento 35d79eaf0f NIFI-6420 Controller Service references not properly tracked if multiple references from same component
<h1>Changes</h1>

-  Instead of keeping a Set of referencing `ComponentNode`, keep instead a Map which stores, for each referencing component, also the name of the property for which the service is referenced
   -  This is done to keep the `addReference()` idempotent
- `StandardControllerService.referencingComponents` changed from HashSet to HashMap
    -  `Key` is the combination of `ComponentNode.hashCode()` and property name
    -  `Value` is the referencing `ComponentNode`
- `ControllerServiceNode.addReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.removeReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.getReferences()` signature is left untouched, when it's called, the values of the `referencingComponents` map are turned into a Set, meaning a referencing component will keep being returned by this method until at least one of its properties still reference the `ControllerService`
- `StandardSchedulingContext.leaseControllerService()` uses a dedicated constant
- `FrameworkIntegrationTest` has a test which create a processor with two optional controller service properties and then verifies that having them referencing and dereferencing the same controller service doesn't cause the reported bug anymore

Removed SchedulingContext from ClojureScriptEngineConfigurator

Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3600.
2019-08-09 15:22:14 -04:00
Peter Turcsanyi a5bdecbd25
NIFI-5478: PutS3Object support for new Storage Classes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3608.
2019-08-09 10:37:29 +02:00
Jeff Storck b938fc20a3
NIFI-6479 Fixes timezone issues in TestJdbcCommon
Dates calculated for method testConvertToAvroStreamForDateTimeAsLogicalType now handle timezone correctly
Added debug logging of date/time calculations and expectations

This closes #3630.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-09 16:29:40 +09:00
Jeff Storck 403c07d4e8
NIFI-6529 Updated TestFormatUtils.testFormatDataSize to use DecimalFormatSymbols when verifying results of FormatUtils.formatDataSize
NIFI-6529 Updated tests that fail with non en_US locales

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3639.
2019-08-08 21:39:20 +02:00
Mark Payne 164e0c4186
NIFI-6380: Fixed NPE that is thrown when clustered and using parameters from Controller Services
This closes #3635
2019-08-06 10:19:20 -04:00
thenatog 52645e8aa7
NIFI-6516 - Upgraded solrj to 6.6.6.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3629.
2019-08-06 09:25:23 +02:00
Koji Kawamura 05cbf8b24f
NIFI-6507 Unsubscribe if WindowsEventLog subscription encounters an error
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3617.
2019-08-05 10:30:50 +02:00
Matt Gilman 944d256d31
NIFI-6380:
- Fixing checkstyle issue.
2019-08-02 14:21:56 -04:00
Mark Payne 4f50f30ad7
NIFI-6380: Introduced the notion of Parameters and Parameter Contexts to the code base.
- Added nifi-parameter module
- Added Parameter to nifi-api
- Added ParameterContext and ParameterLookup
- Updated EL to support Parameters
- Updated backend so that any property can make use of Parameters
- Added web endpoint for Parameter Contexts
- Updated Templates to make use of Parameter Contexts
- Updated Versioned Flows to make use of Parameter Contexts
- Updated Stateless NiFi to support Parameters and take Parameters as part of the configuration, not Variables.

NIFI-6380: Addressed review feedback and fixed bugs; added additional unit and integration tests to verify

NIFI-6380: Added Description to Parameter Context

NIFI-6380: Fixed checkstyle violations

NIFI-6380: Fixed bug that caused updating a Parameter Context to fail if the name is provided and unchanged

NIFI-6380: If parameter is being deleted, don't worry about its sensitivity flag. This addresses a bug where the deletion of a Sensitive Parameter would require that the Parameter be submitted with a value of null and a sensitivity flag of true; else it would provide an error indicating that the parameter can't be changed from sensitive to non-sensitive. Now, the sensitivity flag is ignored.

NIFI-6380: Fixed bug around unsetting Process Group's Parameter Context

NIFI-6380: Moved lastRefreshed timestamp from ParameterContextDTO to ParameterContextsEntity and renamed to currentTime to match the pattern of ControllerServicesEntity rather than FlowHistory. Added parameterContextId to ProcessGroupFlowDTO

NIFI-6380: Added additional integration tests around escaped parameter references

NIFI-6380: Additional tests and bug fix for referencing EL from within another EL Expression

NIFI-6380: Created ParameterEntity to house a 'canWrite' flag for parameters and updated ParameterContext to use it. Updated ParameterContextUpdateRequestDTO to include a Set<AffectedComponentEntity> indicating the components that are affected by the update

NIFI-6380: Addressed review feedback

NIFI-6380: Addressed additional review feedback, mostly around code cleanup

NIFI-6380: Bug fix

NIFI-6380: Addressed more review feedback; fixed a couple of minor bugs encountered when testing

NIFI-6380: Bug fix around Parameter escaping for properties that support Expression Language. Fixed inconcsistency in Authorization hierarchy between /parameter-contexts/1234 and /parameter-contexts  to ensure that the parent of /parameter-contexts is /controller, regardless of how we arrive at the /parameter-contexts resource

NIFI-6380: Fixed but around using ProcessContext#newPropertyValue(String) that previously resulted in causing the Parameters to be evaluated a second time, thereby ignoring escape characters. Updated Integration Test to verify behavior. Also fixed bug in LoadBalanceQueueIT as it was a one-liner and noticed it while running tests to verify other behavior

NIFI-6380: Ensure that if Processor or Controller Services references or de-references a Parameter that user making the change has READ policy on the Parameter Context

NIFI-6380: Verify permissions on Parameter Context when creating processor/controller service and when moving between groups, insantiating template, reverting/change flow version

NIFI-6380: Addressed problem that resulted from rebasing against master; fixed bug in showing the Affected Components for a particular Parameter Context Update

NIFI-6380: Fixed NPE that occurs when attempting to update controller-level Controller Service

NIFI-6380: Updated validation error text when referencing parameter from controller-level controller service or reporting task

NIFI-6380: Ensuring permissions for copy/paste when components reference parameters.

NIFI-6380: Do not require READ policy on Parameter Context in order to un-reference a parameter, only to reference one.

NIFI-6380: Restricted parameter names to match those in the Feature Proposal. Fixed bug that allowed copy & paste of a PG for which user does not have READ policy for a child/descendant PG's parameter context

This closes #3536
2019-08-02 10:45:39 -04:00
Ferenc Szabó f851a9e96f
NIFI-6519 GrokReader does not evaluate EL while validating "Grok Pattern File" property
added evaluateAttributeExpressions call where it was missing.

tested manually

This closes #3627

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2019-08-02 09:03:49 -04:00
Mark Payne a9a4b765b1
NIFI-6517: Ensure that if IOException is thrown from LoadBalanceSession that we properly catch the Exception, mark session as complete, and then re-throw. There was one condition in which this was not occurring. This commit addresses that situation.
This closes #3626.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-01 15:21:53 -04:00
Pierre Villard 639e81e5a1
NIFI-6407 - added support for useAvroLogicalTypes in PutBigQueryBatch
fix Maven dep

This closes #3592.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-31 17:07:37 +09:00
JF Beauvais cdee1d8c09
NIFI-6487 Add S3 User Metadata to ListS3 processor
Fix imports auto formatted by intellij

NIFI-6487 Fix WriteAttribute documentation

This closes #3603.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-31 16:47:54 +09:00
Alessandro D'Armiento 1d22e8a86d
NIFI-6490 MergeRecord supports Variable Registry for MIN_RECORDS and MAX_RECORDS
Unified unit tests

Added custom validation cases for MIN_RECORDS and MAX_RECORDS enforcing they are greater than zero.
While MIN_RECORDS > 0 can fail individually, MAX_RECORDS > 0 validator cannot fail without having also at least another validation step (either the MIN_RECORDS > 0 or the MAX_RECORDS > MIN_RECORDS) to fail, since MIN_RECORDS is a required property with default value 1

This closes #3607.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-31 16:36:42 +09:00
Mark Payne f8e93186f5
NIFI-5856: When exporting a flow that references a Controller Service, also include the ExternalControllerServiceReference's in the snapshot. When importing/updating a flow, if a Controller Service that is inherited is referenced, attempt to find any one ControllerService that gets inherited whose name matches the reference and if found link to that Controller Service.
NIFI-5856: Fixed checkstyle violation

NIFI-5856: Addressed review feedback

NIFI-5856: When assigning a matching Controller Service at a higher level for Process Group import/update, ensure that we consider the User's read permissions when selecting the Controller Service

This closes #3553.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-07-29 10:28:53 -04:00
Mark Owens d1c30d1477 NIFI-6433 Update getStatusHistory to honor filter values.
This modification updates the getStatusHistory method of
VolatileComponentStatusRepository to utilize the start, end, and
preferredDataPoints parameters when retrieving status histories.

When calling the various get<XXXX>StatusHistory methods of
VolatileComponentStatusRepository, they are all passed start, end,
and preferredDataPoints values to allow the filtering of histories
according to the provided parameters. But the follow-on method calls ignore
those values completely. This ticket updates those methods to honor the
parameters and filter the returned Dates accordingly.

This closes #3579.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-07-26 11:27:06 -04:00
Rob Fellows 7dda04e895 NIFI-6455 - Fix issue preventing scrolling all the way to the bottom of certain tables. (#3591)
This closes #3591.

Signed-off-by: Peter Wicks <patricker@gmail.com>
2019-07-25 13:54:12 -06:00
Peter Turcsanyi c6e6a418aa
NIFI-6318: Support EL in CSV formatting properties
CSVReader and CSVRecordSetWriter controller services and also ConvertExcelToCSVProcessor
support EL for Value Separator, Quote Character and Escape Character properties.

NIFI-6318: Fixed null checks and compound OR expression.

NIFI-6318: RecordSetWriterFactory.createWriter() changes.

NIFI-6318: Initialize CSVFormat in onEnabled() if there are no dynamic formatting properties.

NIFI-6318: Comment Marker supports EL.

NIFI-6318: Various review changes.

This closes #3504.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-25 17:04:22 +09:00
archon 3bfef3356e
NIFI-6442: ExecuteSQL/ExecuteSQLRecord convert to Avro date type incorrectly when set 'Use Avro Logical Types' to true
This closes #3584.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-19 14:47:07 +09:00
Pierre Villard 630c651226
NIFI-6334 - fix custom validate error in PutBigqueryBatch
This closes #3589.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-19 13:49:23 +09:00
Peter Wicks 2b0b96e59e NIFI-6439 - Revert Spring Framework HTTPS XSD (#3590) 2019-07-18 11:08:07 -07:00
Matthew Burgess d8388c1887
NIFI-6409: Fixed issue with PutDatabaseRecord when driver doesn't support setObject() without type
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3585.
2019-07-18 16:16:34 +02:00
mike 3fb4454375
NIFI-6443 - ParseSyslog5424 n-th SD-ELEMENT isn't parsed
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3588.
2019-07-18 16:01:32 +02:00
Bryan Bende f2db1539a8
NIFI-6432 Updating get, containsKey, and remove in HBase_1_1_2_ClientMapCacheService and HBase_2_ClientMapCacheService so that they use the column family and qualifier specified in the service
This closes #3581.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-16 13:43:26 +09:00
Phillip Grenier e277545cea
NIFI-6419: Flush the buffers on close to ensure all data is written.
This closes #3568.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-12 17:07:11 +09:00
Peter Turcsanyi 24e50953a3
NIFI-6419: Fixed AvroWriter single record with external schema results in data loss
This closes #3573.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-12 17:07:02 +09:00
Jeff Storck 4783b12a9c
NIFI-5254 Upgraded Groovy to version 2.5.4
Updated pom files regarding groovy dependencies.  groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter

This closes #3547

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-07-11 14:17:35 -04:00
Matthew Burgess 0ccc346aec NIFI-5417: Add missing component status and metrics to S2SStatusReportingTask and PrometheusReportingTask (#3554)
* NIFI-5417: Add missing component status and metrics to S2SReportingTask and PrometheusReportingTask

* NIFI-5417: Added executionNode to schema and doc

This closes #3554

Signed-off-by: Yolanda M. Davis <yolanda.m.davis@gmail.com>
2019-07-10 23:46:13 -04:00
Gary Teichrow 15b0f1cfc2
NIFI-6320. This closes #3494. Added a check whereby, if the user is using the "Use Segment Names" option in the ExtractHL7Attributes processor and the HL7 message being processed contains custom user segments (e.g., Z Segments), the code will downgrade to returning the Segment Label as it does when this option is not used. Without this a null pointer is thrown at line 288.
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-07-10 12:51:46 -04:00
avseq1234 fa1ed16e2b
NIFI-6271, fix issue that incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set
NIFI-6271, fix incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set

NIFI-6271, fix incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set

replace getAttribute(uuid) with getAttribute(CoreAttributes.UUID.key()

fix checkstyle violation

This closes #3575.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-10 11:38:20 +09:00
thenatog 95f5b2278c
Removed guava test dependency as it doesn't look like any of the test code requires it.
NIFI-5562 - Removed unnecessary usage of Guava

NIFI-5562 - Updated Guava version to 28.0-jre

NIFI-5562 - Upgraded Guava to 28.0-jre and Curator to 4.2.0

This closes #3577

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-07-09 12:08:27 -04:00
Koji Kawamura 31097c96d6 NIFI-6385 Added signal.id penalization
- Add additional doc about best practices.

This closes #3540.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-07-09 09:44:04 -04:00
Mike Thomsen f851ed4887 NIFI-4406 Updated ExecuteScript to use Searchable.
This closes #3513.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-07-08 12:25:59 -04:00
Kemix Koo 5127cb209a
NIFI-6392: Avoid accessing the dir for /nifi-docs
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-07-06 18:39:29 -04:00
Travis Neeley 44b84a678f
NIFI-6387 RetryFlowFile
NIFI-6387 RetryFlowFile

NIFI-6387 Maximum Retries support FLOWFILE_ATTRIBUTES scope

NIFI-6387 Fixed reuses descriptions for clarity

This closes #3541.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-05 15:02:05 +09:00
Koji Kawamura 0857d5f89d
NIFI-6413 Remove nifi-ssl-context-service dependency from nifi-prometheus-nar
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3565
2019-07-03 13:30:12 -04:00
Rob Fellows 846c9b0a3f
NIFI-6401 - Long List of Users Overflows onto Date
This closes #3567
2019-07-02 16:02:39 -04:00
DimDroll 41663929a4
NIFI-6174: Add ClientAuth property to ListenBeats
This closes #3539.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-02 11:54:44 +09:00
Ferenc Szabó 3d21059e95 NIFI-6405 GetHDFSFileInfo ignores files in the root/start directory
modifying test cases, fixing file processing.

Testing done:
- Unit tests
- Manual run against a hadoop instance

This closes #3558
2019-07-01 16:35:24 -04:00
adyoun2 f1ae05974e
NIFI-6004 PutFile created directory permissions
NIFI-6004 Improve testing of PutFile file and directory permissions

NIFI-6004 Typo in regex

NIFI-6004 Updates based on review

This closes #3294.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-07-01 17:58:22 +09:00
Troy Melhase e973cacb2f
NIFI-5973 Adds ShellUserGroupProvider.
NIFI-5973 More comments and better defaults for the shell provider.
NIFI-5973 Fixed bug where user was being retrieved by identifier when identity was provided.
NIFI-5973 Fixed a formatting string in the OS X shell commands.
Updated testing conditions to run IT in OS X environment.
Changed unit test to provide identity rather than identifier.

This closes #3537.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-20 16:22:42 -07:00
Mike Thomsen 58ae7d4f92 NIFI-5537 Updated Jackson to latest version. 2019-06-20 19:10:22 -04:00
Mike Thomsen 82f177c632
NIFI-5537 Refactored to be named Cypher Bundle instead.
NIFI-5537 Added skeletons of services, apis, etc.
NIFI-5537 Renamed classes to reflect that they are for cypher and not neo4j specific
NIFI-5537 Added start of client API for doing streaming execution.
NIFI-5537 Completed refactor of unit and integration tests.
NIFI-5537 Added OpenCypherClientService.
NIFI-5537 Updated to gremlin-core 3.3.5 to get better SSL configuration options.
NIFI-5537 Added SSL support to OpenCypher client service.
NIFI-5537 Updated L&N.

NIFI-5537 Updated to 1.10.0-SNAPSHOT.

NIFI-5537 Removed Mac-specific documentation in int test.

NIFI-5537 Updated a few properties to add EL.

NIFI-5537 Added GremlinClientService to support gremlin as well.

NIFI-5537 refactored everything to be a graph bundle.

NIFI-5537 Updated documentation.

NIFI-5537 Fixed assembly.

NIFI-5537 Updated Jackson.

NIFI-5537 Moved MIT section per code review guidance.

NIFI-5537 Changed provenance event and made it more explicit.

NIFI-5537 Updated processor config file.

NIFI-5537 Made changes requested in a code review.

NIFI-5537 Added documentation about driver configuration.

NIFI-5537 Remove output flowfile if an exception is thrown.

NIFI-5537 Updated configuration properties.

NIFI-5537 Made port and path configurable in the tinkerpop family of components.
NIFI-5537 Upgraded to OpenCypher cypher-gremlin-neo4j-driver 0.9.13 because it's the most recent version that JanusGraph supports.

NIFI-5537 Added warning about Janus 0.3.X to int test.

NIFI-5537 Added TinkerPopClientService interface.

NIFI-5537 Changed to mockito-core from mockito-all.

NIFI-5537 Added documentation about Gremlin for first time users w/ NiFi.
2019-06-20 13:39:20 -04:00
mans2singh 1b28c7b45c
NIFI-5537 Create Neo4J cypher execution processor
NIFI-5537 Renamed bundle based on review comments

NIFI-5537 Updated neo4j setup and version documentation and default values based on review comments

NIFI-5537 - Changed exception, logging, and json parser based on review comments

NIFI-5537 - Check debug enabled before logging at debug level

NIFI-5537 - Updated property displayname/description and corrected test file names

NIFI-5537 - Added note about dummy password required in case authentication is disabled on server

NIFI-5537 - Use SSLContext
2019-06-20 13:38:59 -04:00
Mike Thomsen c352c2cc3e
NIFI-6228 Fixed order of operations bug that prevented SSLContext from being handled correctly.
NIFI-6228 Added unit test for SSL configuration.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3535
2019-06-20 12:34:43 -04:00
Jeff Storck 82a6c430a4
NIFI-5373 Created new module nifi-metrics
In nifi-metrics, added build-helper-maven-plugin to enable groovy tests to run, since there are no java tests or src/test/java path in the module
Replaced yammer/dropwizard's VirtualMachineMetrics usage in multiple modules with JvmMetrics interface and JmxJvmMetrics implementation in nifi-metrics
Updated nifi-datadog-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-scripting-bundle to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTask to use JmxJvmMetrics instead of VirtualMachineMetrics
Updated nifi-reporting-utils pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-ambari-reporting-task pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated SiteToSiteMetricsReportingTask to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTaskGroovyTest to use regular strings instead of GStrings due to class cast exception
Updated nifi-prometheus-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated PrometheusMetricsUtil to use LF endings instead of CRLF
Updated nifi-metrics-reporting-bundle and its submodules to use nifi-metrics:1.10.0-SNAPSHOT
nifi-metrics-reporting-task metrics-graphite version upgraded from 3.1.2 to 4.1.0
Removed unused property yammer.metrics.version from nifi-jolt-record-bundle pom
Updated root NOTICE to include derived work in nifi-metrics
Removed duplicate nifi-assembly NOTICE clauses for yammer/dropwizard metrics
Removed unnecessary inclusion of Yammer Metrics in Kafka 0-9, 0-10, 0-11, 1-0, and 2-0 NARs' NOTICE files since the Kafka dependency that transitively includes it is test-scoped.
Removed unnecessary mock interaction in GraphiteMetricReporterServiceTest, Mockito 2.28.2 operates in strict mode and flags unused unused interactions.

NIFI-6354: Added LICENSE and NOTICE for nifi-kafka-2-0-nar
Updated NOTICE in kafka nar modules to reflect the actual module names at the top of the NOTICE
Added yammer metrics NOTICE clauses to nifi-kafka-0-11-nar
Added yammer metrics NOTICE clauses to nifi-kafka-1-0-nar

NIFI-6355: Extended NOTICE clauses for yammer/dropwizard metrics in multiple modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3520
2019-06-19 15:50:42 -04:00
thenatog 64b0d88d83
NIFI-6330 - This closes #3534.Updated Jetty version to 9.4.18.v20190429
NIFI-6330 - Removed jetty.version property declaration in nifi-prometheus-reporting-task pom.xml. This version should be inherited from the root pom.

NIFI-6330 - New release of Jetty came out 2 days ago. Upgraded and verified NiFi functionality again with smoke tests including ListenHTTP, secure clustering, site-to-site

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-06-19 12:39:15 -04:00
Jeff Storck 1d560e2b02 NIFI-6360 Updated Mockito to 2.28.2, PowerMock to 2.0.2
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated.  Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests.  Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version.  Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup

This closes #3533

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-06-17 12:21:07 -04:00
thenatog 0e1a37fcb9
NIFI-6362 - Upgraded com.puppycrawl.tools:checkstyle to 8.18. Made small checkstyle changes to nifi-kafka.
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.

NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.

NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.

NIFI-6362 - Grep pattern no longer required.

This closes #3522.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-17 17:19:09 +09:00
Bryan Bende 2102d8a0bd
NIFI-6328 Allowing force-commit when in LOCALLY_MODIFIED_AND_STALE state
NIFI-6328 Addressing review feedback
NIFI-6328 Additional validation on action

This closes #3499
2019-06-13 10:42:13 -04:00
thenatog 75fb34c8ee
NIFI-6301 - Added a SafeXMLConfiguration which disables XML DTDs which may call external entities.
NIFI-6301 - Fixed unit test. Added comments.
NIFI-6301 - Removed unused rule from test.
NIFI-6301 - Changed read() methods to use a boolean instead. Updated comments.
NIFI-6301 - Fixing checkstyle errors.
NIFI-6301 - Added an XXE Validator to do a simple regex check on files that contain XXE declarations. Added unit tests and related XML test files.
NIFI-6301 - Forgot license headers. Fixed a unit test.
NIFI-6301 - Refactored duplicate code for error handling.
NIFI-6301 - Removed unused imports.
NIFI-6301 - Slightly changed messaging language.
NIFI-6301 - Added logic to catch multiline XXE attacks.
Added logging messages.
Updated unit test.

This closes #3507.

Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-12 16:40:49 -07:00
Mike Thomsen 3e7816da43
NIFI-5902 Added GeoEnrichIPRecord.
NIFI-5902 Updated GeoEnrichIPRecord with more error handling.

NIFI-5902 Made changes requested in a review.

NIFI-5902 Updated EL support on database path.

NIFI-5902 Made updates based on code review.

NIFI-5902 Made two changes from a code review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3231
2019-06-12 15:25:42 -04:00
Mark Payne 8bed249f37
NIFI-6353: If LoadBalanceSession determines remote node's queue is full, mark session complete and penalize the Partition, rather than just penalizing the session. This allows the client to move on to another session, which avoids an issue where backpressure on one queue can prevent data from being transferred between nodes in another queue
This closes #3529.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 11:32:41 -04:00
Mark Payne 2465f67dfe
NIFI-6339: Only call @OnConfigurationRestored method on components when they are created if FlowController has been initialized
This closes #3530.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-12 10:56:04 -04:00
Shawn Weeks e913f5706f
NIFI-6243 Add Support for AtomicDistributedCache to the HBase 1.x and 2.x Map Cache Services
This closes #3462.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 15:31:39 +09:00
Matthew Burgess 05f3cadee8 NIFI-6370: Allow multiple SQL statements in PutDatabaseRecord
This closes #3528.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-12 10:32:11 +09:00
Mark Payne f15332ff87 NIFI-6182: Updated dependency on Lucene to Lucene 8.0.0. Updated code necessary to use the new API. Updated WriteAheadProvenanceRepository so that upon startup if provenance indexes are written using the old Lucene format, , they are considered 'defunct' and the events are re-indexed in a background thread into a new index that uses the Lucene 8 format and the old index is then removed. Added Provenance Repository that consists of about 30 events and added integration test to ensure that the repo can be started up/initialized when pointing to a prov repo that was written using the old lucene format. 2019-06-11 09:45:46 -04:00
Andres Garagiola 1975101292 NIFI-6361: Add fix to PutFile processor
When PutFile uses 'replace' conflict resolution and max files, there is an issue when the folder has X files, and the limit is also X. The processor fails instead of replacing it, leaving X files. This commit fixes that issue.

This closes #3524.

Signed-off-by: Andres Garagiola  <andresgaragiola@gmail.com>

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 15:45:19 +09:00
Evan Reynolds 99350b761d NIFI-6349 Fix to MergeRecords when handling fragments over multiple iterations
Fixed fragment count attribute check

This closes #3517.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-10 14:33:11 +09:00
Matthew Burgess 99d6ed244c
NIFI-6352: Add ability to report all component metrics to Prometheus
This closes #3519.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-06-07 15:30:38 -04:00
Ferenc Szabó aefdc9377c NIFI-6218 Support setting transactional.id in PublishKafka/PublishKafkaRecord
Added new property to the processors. If it is empty, then the behaviour is the same as before.
Adde unit test to check if KafkaProducer receives the new property.

Testing Done:
Unit tests.
Connecting to kafka and verifying transactionalId-s in kafka server log.

fixing review findings

This closes #3439.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-07 17:55:52 +09:00
Peter Wicks 2d3fce81c1
NIFI-6350 Fixed misleading flow controller startup error message.
This closes #3516.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-06-05 19:25:13 -07:00
Matthew Burgess 1d18735076
NIFI-6348: Added Custom ORDER BY Column property to GenerateTableFetch
NIFI-6348: Fixed doc

This closes #3515.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-05 10:00:12 -04:00
Mark Payne 60b5c13ce9
NIFI-6341: Addressed bug that can result in a deadlock if Thread 1 adds a Controller Service to a Process Group (which obtains group's write lock and AbstractComponent's Lock) while at the same time Thread 2 attempts to peform validation on the service (which obtains Controller Service's write lock, then AbstractComponent's Lock)
This closes #3510.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-06-04 09:55:26 -04:00
Matthew Burgess 08d2e69878
Revert "NIFI-6244: Provide default ORDER BY clause in MSSQL adapters"
This reverts commit 7fc49c2cb8.

This closes #3512
2019-06-04 09:02:03 -04:00
Bryan Bende e7f4af6801 NIFI-6315 Ensuring remote ports get tracked correctly when saving/retrieving versioned flows
NIFI-6315 Renaming public ports during import/change-version when name alreadye exists, and ignorning name change from flow diffs

NIFI-6315 Ensuring public ports get renamed during copy/paste

NIFI-6315 Set max concurrent tasks on Port based on value from VersionedPort

This closes #3488.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-06-03 11:25:04 +09:00
Mike Thomsen 407add7847 NIFI-5916 Added an option to enable empty flowfiles to be sent if there are no results from a query.
NIFI-5916 Fixed potential NPE.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3315
2019-06-01 11:27:53 -04:00
Andy LoPresto a9d1dd7b08 NIFI-6337 Resolved groovy unit test execution problems in nifi-properties-loader and fixed failing unit tests.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3508
2019-05-31 08:44:17 -04:00
slyouts 097b9c3075 NIFI-6022 Added try/catch to protect against secondary exception in a primary exception catch clause
NIFI-6022 mods per PR review

NIFI-6022 mods per PR review

This closes #3476.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-05-31 08:51:36 +09:00
Satwik 8560eb995a NIFI-5435 Added Prometheus reporting task.
NIFI-5435 Fixed Rebase Issue
NIFI-5435 changes in capability description of reporting task
NIFI-5435 Changes in capability description of reporting task
NIFI-5435 Changes to build Property Descriptors in static constructor
NIFI-5435 Indentation changes
NIFI-5435 Test code changes
NIFI-5435 Minor code changes
NIFI-5435 Changes related to client authentication
NIFI-5435 Prometheus metrics changes
NIFI-5435 Added process group name and id to metrics

This closes #3257

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-30 08:08:07 -04:00
Matt Gilman 999cfdcac9
NIFI-6302:
- Updating integration tests to verify pruned results.

This closes #3487.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-29 14:45:00 -07:00
Andy LoPresto e6c843f465
NIFI-6323 Changed URLs for repositories, project description, and mailing lists to use HTTPS.
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.

This closes #3497
2019-05-29 14:36:40 -04:00
Koji Kawamura 99b20ac2d1 NIFI-6303 CSVWriter should write schema defined fields first, then extra ones
This closes #3484

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-26 16:52:12 -04:00
Peter Turcsanyi 2bfffd35b2 NIFI-6317: HandleHttpRequest timeout handling issues
- set cleanup thread scheduling to 5 seconds in order to prevent stale request staying in the cache for a long time
- disable timeout handling in HandleHttpRequest in order to prevent a new flowfile being generated at request expiration
- use HttpServletResponse.SC* status codes everywhere in HandleHttpRequest, get rid of importing javax.ws.rs
- add some more error logging to make bug investigations easier
- add a short message to 503 error responses to make bug investigations easier

This closes #3490.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-24 14:43:50 -04:00
Rob Fellows 0a31c38790
NIFI-6253 - Remove invalid/unnecessary argument in call to nfCanvasUtil.reload
This closes #3491
2019-05-24 11:02:09 -04:00
Rob Fellows 410854f53e
NIFI-6316 - Upgrade jQuery to the latest version (3.4.1).
This closes #3489
2019-05-24 09:52:02 -04:00
thenatog fe68d43e1d
NIFI-6280 - Broke out the matching for /access/knox/** and /access/oidc/** to allow the Jetty security filters to be applied in the /access/oidc/logout and /access/knox/logout cases.
NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.

This closes #3482.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-05-22 17:05:06 -07:00
Koji Kawamura 8a50cb10b2 NIFI-2933 Remote input/output ports at any PG
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.

This closes #3351.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 15:44:16 -04:00
Matt Gilman a97766d62f NIFI-6302:
- Ensuring Process Group contents are pruned when appropriate.

This closes #3477.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-05-22 14:12:50 -04:00
Aaron Leon 81ddd02ca8 NIFI-6100 Use setBytes in JdbcCommon for binary types
This closes #3410

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-05-21 14:22:56 -04:00
Mark Payne cbafd8f834 NIFI-5922: Addressed checkstyle violations; added to README Updates to nifi-fn proposal:
- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows
- Rebased against master and updated to version 1.10.0-SNAPSHOT
- Removed dependency on nifi-framework-core
- Added LICENSE/NOTICE files
2019-05-21 11:19:17 -04:00