Commit Graph

286 Commits

Author SHA1 Message Date
Bryan Bende ff7283d65d NIFI-4506 Adding toDate and format functions to record path. This closes #2221.
Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-01-03 11:01:33 -05:00
Mark Payne c91d99884a NIFI-4717: Several minor bug fixes and performance improvements around record-oriented processors
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2359
2017-12-29 10:43:21 -05:00
Mark Payne f772f2f093 NIFI-4671: This closes #2328. Ensure that Avro Schemas that are created properly denote fields as being nullable iff the schemas says they are, for non-top-level fields
Signed-off-by: joewitt <joewitt@apache.org>
2017-12-11 11:46:15 -05:00
Andy LoPresto 5d643edfab
NIFI-4501
- Changed request header handling logic.
- Removed unnecessary Maven dependency.
- This closes #2279
2017-11-17 20:36:55 -05:00
Matthew Burgess 53de8c93ba NIFI-4603 - Externalize Max HTTP Header Size to nifi.properties
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2271.
2017-11-15 10:12:39 +01:00
Mark Payne af3a578711
NIFI-4598: When we retrieve the 'controller' from a remote NiFi instance in order to determine which ports are available, cache those results for up to some configurable amount of time (default 30 secs) so that we don't constantly issue HTTP Requests to the remote nifi
This closes #2270.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-11-13 14:46:38 -05:00
Mike Thomsen c70a86eac0 NIFI-4575 Added ability to convert from a date to a timestamp within the record API.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2255
2017-11-13 12:57:39 -05:00
Matt Gilman 6baea8ccff
NIFI-4444:
- Upgrading to Jersey 2.x.
- Updating NOTICE files where necessary.
- Fixing checkstyle issues.

This closes #2206.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-10-12 10:27:02 -07:00
Andy LoPresto d4168f5ff1
NIFI-4297
- Upgraded immediately actionable dependency versions from Meterian report.
- Upgraded jackson-core test dependencies for HBase and Elasticsearch modules.
- Only 3 instances of jackson-core < 2.8.6 (Google Cloud Platform and Spark Receiver modules).
- Upgraded version of poi dependency in nifi-email-processors to 3.16.
- Resolving dependency issues after rebasing against 1.5.0-SNAPSHOT.
- Removed jackson-databind from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.version} in all referenced artifacts.
- Removed jackson-mapper-asl from <dependencyManagement> block in nifi/pom.xml and added explicit reference to ${jackson.old.version} in all referenced artifacts.
- Removed Jasypt from <dependencyManagement> and added explicit version in test dependency for legacy compatibility.
- This closes #2084
2017-10-05 15:23:52 -04:00
Mark Payne 8741b6f6a5
NIFI-4461: When reading from socket channel use blocking mode instead of sleeping; when writing, use a far smaller sleep duration
This closes #2193.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-10-04 11:45:35 -04:00
Jeff Storck 2694adcca9 Merge branch 'NIFI-4412-RC2' 2017-10-02 13:58:54 -04:00
Jeff Storck a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Pierre Villard bdab3cda0a NIFI-4340 - fix record path evaluation when array is [ null ]
This closes #2122.
2017-09-28 13:39:55 -04:00
Matt Gilman 6c798d18ef NIFI-4382:
- Adding support for KnoxSSO.
- Updated the docs for nifi.security.user.knox.audiences.
- The KnoxSSO cookie is removed prior to request replication.

This closes #2177
2017-09-27 16:22:18 -04:00
Andy LoPresto 2c1f5b49e4 NIFI-3116 This closes #2108. Added initial regression test for StringEncryptor to ensure continued functionality during removal of Jasypt.
Added external compatibility regression test for StringEncryptor to ensure continued functionality during removal of Jasypt.
Documents custom salt lengths and iteration counts for each encryption method.
Added (ignored) failing tests for keyed encryption (Jasypt does not support keyed encryption).
Changed StringEncryptor to non-final class and added protected default constructor.
Added failing test for initialization status.
Added utility methods in CipherUtility.
Moved PBE cipher providers (and tests) from nifi-standard-processors to nifi-security-utils module.
Implemented PBE and keyed encryption/decryption logic.
Moved Scrypt unit test back into scrypt package.
Resolved test failures in limited strength cryptographic environment.
Implemented keyed encryption/decryption and enabled unit tests.
Removed Jasypt dependency from production scope (kept in test scope for backward compatibility tests).

Signed-off-by: joewitt <joewitt@apache.org>
2017-09-26 10:55:29 -04:00
Andy LoPresto 9e2c7be7d3
NIFI-4353
- Added XmlUtils class.
- Added unit test.
- Added XXE test resource.
- Refactored JAXB unmarshalling globally to prevent XXE attacks.
- Refactored duplicated/legacy code.
- Cleaned up commented code.
- Switched from FileInputStream back to StreamSource in AuthorizerFactoryBean.
- This closes #2134
2017-09-22 14:31:38 -04:00
Mark Payne e52e9acc59 NIFI-4377: Added a fieldName() function to RecordPath and addressed an issue that caused //* to not work
NIFI-4377: Updated RecordPath Guide to include the new fieldName() function and improved unit test for the function

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2147.
2017-09-14 09:40:36 +02:00
Pierre Villard 5957f6ee95
NIFI-4341 - add provenance repository storage usage in UI
- addressing review comments
- This closes #2119
2017-09-11 13:16:53 -04:00
Mark Payne 5cd8e93beb
NIFI-4224:
- Initial implementation of Process Group level Variable Registry
- Updated to incorporate PR Feedback
- Changed log message because slf4j-simple apparently has a memory leak; passing a String instead of passing in the Controller Service object as an argument addresses this.
- This closes #2051
2017-08-17 10:42:29 -04:00
Mark Payne 451f9cf124 NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields)
Signed-off-by: joewitt <joewitt@apache.org>
2017-08-11 22:01:46 -07:00
Matt Gilman 528b82634f
NIFI-4210:
- Introducing support for OpenId Connect.
- Updating REST API and UI to support the authorization code flow.
- Adding/fixing documentation.
- Implementing time constant equality checks where appropriate.
- Corrected error handling during startup and throughout the OIDC login sequence.
- Redacting the token values from the user log.
- Defaulting to RS256 when not preferred algorithm is specified.
- Marking the OIDC endpoints as non-guaranteed in to allow for minor adjustments if/when additional SSO techniques are introduced.

This closes #2047.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-08-10 11:15:35 -07:00
Andy LoPresto 675d989003
NIFI-4139
- Moved key provider interface and implementations from nifi-data-provenance-utils module to nifi-security-utils module.
- Refactored duplicate byte[] concatenation methods from utility classes and removed deprecation warnings from CipherUtility.
- Created KeyProviderFactory to encapsulate key provider instantiation logic.
- Added logic to handle legacy package configuration values for key providers.
- Added unit tests.
- Added resource files for un/limited strength cryptography scenarios.
- Added ASL to test resources.
- Moved legacy FQCN handling logic to CryptUtils.
- Added unit tests to ensure application startup logic handles legacy FQCNs.
- Moved master key extraction/provision out of FBKP.
- Removed nifi-security-utils dependency on nifi-properties-loader module.
- Added unit tests.
2017-08-07 15:21:58 -04:00
Yolanda M. Davis afd4f9e034
NIFI-4022 - Initial update for SASL support for cluster management in Zookeeper
NIFI-4022 - adding sasl documentation update and update to test

This closes #2046.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-04 14:19:35 -04:00
Wesley-Lawrence 40cde0466a NIFI-4215 NiFi can now parse an Avro schema of a record that references an already defined record, including itself.
This closes #2052.
2017-08-03 15:13:07 -04:00
James Wing 2502b79bae NIFI-4215 Revert Complex Avro Schema Changes
This reverts commit cf49a58ee7.
2017-08-01 21:03:04 -07:00
Wesley-Lawrence cf49a58ee7 NIFI-4215 Allow Complex Avro Schema Parsing
NiFi can now parse an Avro schema of a record that references an already defined record, including itself.

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2034.
2017-07-30 16:31:39 -07:00
m-hogue c54b2ad81c NIFI-3736: change to honor nifi.content.claim.max.appendable.size and nifi.content.claim.max.flow.files properties. Added 100 MB cap for NiFiProperties.MAX_APPENDABLE_CLAIM_SIZE
This closes #2010.
2017-07-21 09:32:45 -04:00
Pierre Villard a3b72f1bb7
NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS. This closes #1962 2017-07-06 15:38:55 -04:00
Mark Payne ba56774fa1
NIFI-4151: Updated UpdateAttribute to only create JAXB Context once; Minor performance tweaks to standard validators and StatusMerge.prettyPrint; updated AbstractConfiguredComponent to not create a new ValidationContext each time that validate is called but only when needed; updated FlowController, StandardControllerServiceProvider, and StandardProcessGroup so that component lookups can be performed using a ConcurrentMap at FlowController level instead of having to perform a depth-first search through all ProcessGroups when calling findProcessor(), findProcessGroup(), findXYZ()
This closes #1979
2017-07-06 13:43:56 -04:00
Matt Gilman 6bc6f955c0 NIFI-4059:
- Introducing the LdapUserGroupProvider.
- Updating documentation accordingly.
- Moving the IdentityMapping utilities so they were accessible.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1923.
2017-06-19 19:25:33 +02:00
Matt Gilman cc741d2be6
NIFI-3997:
- Bumping to next minor version.
2017-06-08 15:22:51 -04:00
Matt Gilman 1bf0a1a849
Merge branch 'NIFI-3997-RC1' 2017-06-08 14:30:10 -04:00
Andy LoPresto f31f2a5f26
NIFI-3836 Added check for root user in test that depends on unreadable file.
This closes #1865.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-06-05 18:58:07 -07:00
Matt Gilman 6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Mark Payne 32314d70fd NIFI-4009: Added support for several key functions in RecordPath
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1881
2017-06-02 14:06:05 -04:00
Mark Payne 239bbfbb9d NIFI-3989: This closes #1869. Avoid calling Long.parseLong, etc. in order to determine if a String is a valid number because we don't want the Exceptions that occur if it is not
Signed-off-by: joewitt <joewitt@apache.org>
2017-06-02 10:09:38 -04:00
Matt Gilman 7bcccb10f4 NIFI-3719:
- Removing the usage of SimpleDateFormat when formatting hours/minutes/seconds as the current timezone could cause unintended results.

This closes #1871.
2017-05-30 15:01:39 -04:00
Joey Frazee 118f4e8cca NIFI-3978 Increase threadpool size for S2S HTTP tests
This closes #1858.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-05-26 14:12:02 +09:00
Mark Payne 5aa3baca79 NIFI-3568: This closes #1577. Use a cached thread pool in order to allow ThreadPoolRequestReplicator to scale up the number of threads to some configurable max
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-24 21:21:45 -04:00
Matt Gilman f97b3fe455
NIFI-3963: - Ensuring the RemoteGroupPort yields when the details cannot be refreshed from any of the configured remote instances.
This closes #1853.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-24 15:58:22 -04:00
Mark Payne 6937a6cf64 NIFI-3953: This closes #1837. Allow multiple schemas on same kafka topic/partition for ConsumeKafkaRecord_0_10
Also, updated record writers to ensure that they write the schema as appropriate if not using a RecordSet. Updated ConsumeKafkaRecord to allow for multiple schemas to be on same topic and partition

Signed-off-by: joewitt <joewitt@apache.org>
2017-05-22 14:37:53 -04:00
Mark Payne 1a3c525dd6
NIFI-3951: Fixed bug that calculated the index incorrectly when filtering for ArrayIndexPath
This closes #1835.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-22 12:39:32 -04:00
Mark Payne c49933f03d NIFI-3948: This closes #1834. Added flush() method to RecordWriter and call it when writing a single record to OutputStream for PublishKafkaRecord. Also removed no-longer-used class WriteAvroResult
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-19 23:05:04 -04:00
Mark Payne ae9953db64 NIFI-3857: This closes #1825. Added PartitionRecord processor
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-19 02:08:52 -04:00
Mark Payne 9bd0246a96 NIFI-3863: Initial implementation of Lookup Services. Implemented LookupRecord processors. This required some refactoring of RecordSetWriter interface, so refactored that interface and all implementations and references of it 2017-05-19 01:02:41 -04:00
Koji Kawamura 36e7bd6164 NIFI-3894: This closes #1820. Fixed close and consume order with compression.
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown as the Inflater is closed before input stream is consumed.

Also, calling close from AbstractTransaction.receive is removed, because the DataPacket is exposed as its return value and this class will not be able to know when to close the stream.

Signed-off-by: joewitt <joewitt@apache.org>
2017-05-17 22:57:33 -04:00
Koji Kawamura 77a676bf92 NIFI-3894: Call Inflater/Deflater.end to free up memory
This closes #1796.
2017-05-16 13:49:31 -04:00
Joey Frazee e717fb3f79 NIFI-3771 Move HL7 test docs into class members to avoid git rewriting CRs
This closes: #1731

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-05-14 22:29:37 +10:00