Commit Graph

5424 Commits

Author SHA1 Message Date
David Mollitor dccbde40f3
NIFI-5452: Allow ignore block locality in HDFS
Fixed formatting. Fallback to autoboxing

This closes #3652.
2019-10-28 14:23:44 -07:00
Peter Turcsanyi 5fd8df5780
NIFI-6818: This closes #3852. Set service api dependencies to provided in AWS and Azure bundles
The controller service api jars have their own bundles, they do not need to be
packaged into the processor nars.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-28 12:56:06 -07:00
Joe Witt c6e8bf204c
NIFI-6794 resolving pom issue creating resolving merge conflict 2019-10-28 12:55:26 -07:00
Mike Thomsen d5fbfd1a12
NIFI-6794 This closes #3828. Moved to the latest Elasticsearch 5.X client and upgraded Apache HttpClient to the latest 4.5.X release to try to get around a SSL bug reported on the mailing list. 2019-10-28 12:42:37 -07:00
Joe Witt dda29cd90b
NIFI-6733 updating zk to latest incremental 2019-10-28 12:40:03 -07:00
mysunnytime 5c69faf9bb NIFI-6597: Upgrade Azure Event Hub version and code.
- Lazy instantiation for PutAzureEventHub. Also add explaination for thread pool needed for EventHubClient.
2019-10-28 15:33:32 -04:00
Jan Hentschel ccf85777c4
NIFI-6816 Removed duplicated dependency declarations
This closes #3851

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-28 12:49:08 -04:00
Otto Fowler cfcf2e698d
NIFI-6682 SplitJson does not work if the input is null
This closes #3772

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-25 17:43:13 -04:00
mtien 4d25b6341c
NIFI-1474 Added method to check p boundary. Added a unit test.
NIFI-1474 Added unit test for constructor p boundaries. Added p boundary conditions.
NIFI-1474 Updated documentation regarding p boundary.
NIFI-1474 Added r validation and unit tests. Added Javadoc for new methods.

This closes #3844.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-25 14:28:46 -07:00
Matt Gilman 4f5718bf50
NIFI-6812: This closes #3847
- Establishing the L&N for the azure-services-api-nar.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-25 13:30:54 -04:00
Matt Gilman 9364f7d2e8
NIFI-6811:
- Allowing enterGroup to complete prior to loading the controller service table.

This closes #3846.
2019-10-25 12:07:03 -04:00
Mark Payne 3f270f184c
NIFI-6807: When a Controller Service's state is transitioned to ENABLING, complete the Future successfully, even if the Controller Service is not valid. The Controller Service will remain in the ENABLING state until it is made valid, at which point it will ENABLE (unless explicitly disabled first). This allows us to Enable a Controller Service and its referencing components, even if the referencing component is still invalid due to it not yet recognizing the the referenced service has been enabled.
This closes #3841
2019-10-25 11:38:34 -04:00
Mark Payne 81fc2768c9
NIFI-6798: When checking that referenced Controller Services are valid, use ValidationContext.getProperties() rather than ComponentNode.getSupportedPropertyDescriptors() because the latter does not include dynamic properties
This closes #3836.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-25 09:52:44 -04:00
Matt Gilman 2cef010b06
NIFI-6687:
- Using correct NiFi Ranger Service name in unit tests.

This closes #3842.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-25 09:31:34 -04:00
Aldrin Piri 20f70357a3
NIFI-6809 Make use of ASF shortened URL to point to community SLack invite link.
This closes #3845

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-25 09:09:28 -04:00
Koji Kawamura 87838e6c31
NIFI-6793 Fix NumberFormatException at CaptureChangeMySQL
This closes #3827

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-25 05:48:50 -04:00
Mark Payne 8079d2531e
NIFI-6805, NIFI-6806: Updated toString() of StandardControllerServiceNode so that it includes the component's type and UUID, updated the more generic Exception Mappers to include stack traces regardless of the log level that is enabled
This closes #3840

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-10-24 19:27:30 -04:00
Matthew Burgess 8771c35f4a NIFI-6799: Add PrometheusRecordSink and reset() method to RecordSinkService (#3839)
This closes #3839
2019-10-24 11:53:28 -04:00
Yolanda M. Davis 147365285c
NIFI-6801 - fix to ensure unique model instance is created for each connection. Also increased default query interval.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3838
2019-10-24 08:51:50 -04:00
Matt Gilman c2da6e7aa2
NIFI-6687:
- Updating exclusions and overrides accordingly for Ranger 2.0.

This closes #3835.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-23 15:02:05 -04:00
Mark Payne 3543b9c42c
NIFI-6797: Add support for specifying Partition via EL or RecordPath for PublishKafka(Record)_1_0 and PublishKafka(Record)_2_0
This closes #3834.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-23 14:59:33 -04:00
Matthew Burgess ace23c35b5 NIFI-6780: Add QueryNiFiReportingTask, RecordSinkService, S2S and DB impls (#3826)
* NIFI-6780: Introduce RecordSinkService to separate format and destination, refactor common S2S utils

Added QueryNiFiReportingTask to SQL query NiFi status and metrics

Add PROCESSOR_STATUS and PROCESS_GROUP_STATUS

Add CONNECTION_STATUS_PREDICTIONS

check for null predictions

Add ConnectionStatusRecursiveIterator

Fix issue w/ duplicate iterator outputs

Refactored query interfaces, fixed assembly POM

Rebased v master, fixed isBackPressureEnabled and Checkstyle/RAT errors

Add DatabaseRecordSink service (#13)

* Add DatabaseRecordSink service

* Incorporated review comments

* NIFI-6780: Add/fix docs, cleanup warnings, fixed some table definitions

* Added bundle profile, remove predictions table if not enabled

* Added doc for which tables are available when

This closes #3826.
2019-10-22 08:56:42 -04:00
Andy LoPresto d148fb1854
NIFI-3834 This closes #3821. Added encrypted content repository implementation.
Added skeleton implementation of EncryptedFileSystemRepository.
Added new impl to META-INF registry.
Added investigation comments to FileSystemRepository.
Implemented RepositoryObject block and stream encryptors.
Added passing unit test for encryption and decryption of multiple content writes (large buffered file) for AES-CTR encryptor.
Refactored shared logic from AES CTR and G/CM encryptors to abstract parent.
Added working unit test for writing/reading via encrypted file system repository.
Added stream wrappers.
Added encryptor.
Added working unit test for writing/reading multiple pieces of content via encrypted file system repository.
Added unit test skeleton for writing/reading multiple pieces of content with different keys via encrypted file system repository.
Implemented key management skeleton for encrypted content repository.
Multiple content claims can now be encrypted with different keys on the same resource claim and retrieved.
Implemented validation on setting active key id.
Added content repository encryption properties to NiFiProperties.
Implemented configuration of encryption services from NiFiProperties.
Refactored NiFiPropertiesLoader functionality to CryptoUtils for availability in other modules.
Added RepositoryEncryptionConfiguration and repo-specific subclasses for data containers.
Continued refactoring of CryptoUtils and RepositoryEncryptorUtils library methods.
Exposed some internal state of FileSystemRepository via protected getters so encrypted implementation could access.
Refactored EncryptedFileSystemRepository to extend rather than duplicate FSR.
Refactored EFSR to use ECROS which now extends extracted ContentRepositoryOutputStream protected inner class in FSR.
Added unit test to encrypt & decrypt image resource.
Added smaller image resource for easier unit test debugging.
Added importFrom method to resolve issue where GetFile would not encrypt content persisted to repository.
Added text test resource for tests around exporting claim subsets.
Added exportTo methods to handle decrypting encrypted content.
Performed large unit test refactoring, moving shared logic to helper methods.
Added unit test for merged content claim with header/footer/demarcator.
Added unit test for merging content claims each encrypted with a different key.
Ignored non-deterministically failing firewall DNS test.
Added documentation to User and Admin Guide for Encrypted Content Repository.
Added image.
Added refactored utility method for shared ROEM extraction and validation logic in AbstractAESEncryptor.
Replaced ad-hoc generation of ciphertext stream and byte[] for testing with static initialization from pre-generated serialized form for performance.
Cleaned up unused test code.
Cleaned up Javadoc and code comments.
Refactored shared logic.
Fixed checkstyle issue.
Fixed test failure due to error message change.
Added experimental warning to repository implementation classes and User Guide documentation.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-21 13:03:59 -04:00
Yolanda M. Davis 7c9d34f820
NIFI-6778 - Added rules engine service api and Easy Rules Implementation
NIFI-6778 - added comments and component description annotations

NIFI-6778 - added license and notice details

NIFI-6778 - added additional details section and updated description of service

NIFI-6778 - fixed checkstyle error

NIFI-6778 - addressed review comments for documentation and description corrections

NIFI-6778 - added include-rules profile for assembly build

NIFI-6778 - add MVEL and ASM to license and notice

NIFI-6778 - switch to use Jackson for NiFi Rules Json deserialization

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3824
2019-10-21 12:16:11 -04:00
Matt Gilman 8b23929638
NIFI-6786:
- Upgrading nimbus-jose-jwt.

This closes #3830.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-21 11:41:29 -04:00
Andrew Lim e9717c76ac
NIFI-6553 Updated Admin Guide with a section that recommends antivirus scan exclusions
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3793.
2019-10-19 16:41:02 +02:00
Andrew Lim a30eb3a08a NIFI-6457 Add Upgrading NiFi section to Admin Guide (#3818)
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-10-18 13:25:48 -07:00
samhjelmfelt ea1becac4f
NIFI-6703: Add Stateless NiFi to CLI
NIFI-6703: Fixed extracted nar directory and marked api as experimental
NIFI-6703: Moving nifi-stateless into nifi-framework
NIFI-6703: Refactored to fix jetty/spring issues
NIFI-6703: checkstyle fix
NIFI-6703: updated to mirror traditional NiFi's bootstrap process and java11 dependency management
NIFI-6703: minor changes
NIFI-6703: Documentation fixes

This closes #3795.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-17 15:46:32 -04:00
Mark Payne b004f1f94c
NIFI-6779: When resolving node identifier, if NodeIdentifier has a different value for hostname/port for web api or cluster protocol, assume that node is correct about itself instead of assuming that Cluster Coordinator knows best about the other node
NIFI-6779: Remove any conflicting Node Identifiers when a new Node ID is encountered

This closes #3819
2019-10-17 11:21:17 -04:00
Andrew Rodriguez cd90b1d3e1
NIFI-6773 This closes #3820. Adding lz4-framed option for compression/decompression
NIFI-6773 Adding two unit tests for lz4-framed compression/decompression
NIFI-6773 Adding compressed lz4 file for decompression test

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-16 17:45:32 -04:00
Mark Bean c4ccd5f505
NIFI-6750: This closes #3810. add FETCH provenance event to FetchFile
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-15 14:52:39 -04:00
Bryan Bende d570fe8154
NIFI-4820 This closes #3813. Improving security configuration for Kafka 2.0 processors
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-15 14:05:25 -04:00
Matthew Burgess 77e9bda0ff
NIFI-6764: Recreate GroovyShell in ExecuteGroovyScript only when necessary
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3803.
2019-10-15 11:55:52 +02:00
Bahlul Haider 266a59d05a
NIFI-6755 Fix broken link to Quartz documentation
Update the links to the quartz documentation in the User Guide section
for cron trigger scheduling.

This closes #3802.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2019-10-14 15:54:03 -04:00
Mark Payne 9dd0dda688
NIFI-6772: Improved the information that is emitted about Garbage Collection when a Diagnostics Dump is performed
This closes #3809
2019-10-14 11:32:47 -04:00
JF Beauvais 08fe648b1c
[NIFI-6374] ConsumeEWS fails when email attachment has no content type - ensure non null content type
This closes #3596.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-10-11 15:10:11 +09:00
Bryan Bende af81afce40
NIFI-5753 Make use of keyPassword optional and only used when keystore is used, bump registry client version
This closes #3126.
2019-10-10 15:09:28 -04:00
Grzegorz Kołakowski 5d65e6aba4
NIFI-5753 Add SSL support to HortonworksSchemaRegistry service
Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 15:09:16 -04:00
Joe Witt a273ff10f9
NIFI-6758 fixing checkstyle issues 2019-10-10 15:04:16 -04:00
Matt Gilman 7db6ba841f
NIFI-6768:
- Ensuring the parameter context resources are available via the /resources endpoint.

This closes #3808.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:49:39 -04:00
Endre Zoltan Kovacs 5238dc20b1
NIFI-6758: allow pre-unpacked nar's to be recorded in ExtensionMapping
- fixing resource leak

This closes #3806.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-10 11:36:29 -04:00
Matt Gilman ed1e843609
NIFI-6769:
- Ensuring the users policy listing can handle when the specified policy is unknown.

This closes #3807.
2019-10-10 11:30:14 -04:00
markap14 d64f4fa942 NIFI-6760: When writing/reading the length of a DataFrame, do so usin… (#3801)
* NIFI-6760: When writing/reading the length of a DataFrame, do so using a 4-byte integer instead of a 2-byte short. When using uncompressed data, we know that the length of the DataFrame will be no more than 64 KB so a 2-byte short is sufficient. However, if data is compresed, there's a chance that the compressed form of the data will be larger than the uncompressed form (for example, with random binary data or with encrypted data). In this situation, we can end up overflowing the 2-byte short, which causes the length that is written to be wrong. Using a 4-byte integer avoids this situation.

* NIFI-6760: Fixed unit tests
2019-10-10 10:19:21 -04:00
Matt Gilman 4e8dd6557f
NIFI-6756:
- Fixing issue opening a users policy listing when there is a policy for a specific parameter context.

This closes #3805.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-10 00:55:47 -04:00
Matt Gilman 99e9010b32
NIFI-6766:
- Ensuring policy label is properly escaped when populating the user's access policy listing.
This closes #3804.

Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-10 00:19:22 -04:00
Mark Payne 5414cd5016 NIFI-6759: When encountering Exception during load-balanced connection communications, after closing socket, return from method rather than continuing loop
This closes #3800.
2019-10-09 16:05:44 -04:00
Matt Gilman 9a496fe9d2
NIFI-6751:
- Fixing the identifier on the user table. In a previous task, this was changed to utilize the URI but that does not work with other code interacting with this table.

This closes #3798.
2019-10-09 12:00:22 -04:00
Mark Payne 47f45abdf9
NIFI-6748: Fixed bug in Parameter Contexts' affected components where if a Controller Service referenced a Parameter, any component that references that service should also be considered an affected component but wasn't. Also fixed a bug in how we handled stopping a Processor that was in the STARTING phase.
This closes #3794.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-10-09 10:20:45 -04:00
Mark Payne e394f6683a
NIFI-6753: Fixed bug where all values being provided to the CSV Writer were String objects, which resulted in the CSV Writer improperly quoting numeric values when the schema indicates that the value is a number. Now, we will only convert the value to a String if the value is not a Number and/or the schema does not indicate a numeric ty type
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3797
2019-10-09 09:54:45 -04:00
Kevin Doran 2d90145842
NIFI-6749 This closes #3792. Fix Swagger for ControllerServiceReferencingComponentDTO
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-10-08 11:48:10 -04:00