Commit Graph

5804 Commits

Author SHA1 Message Date
Mark Payne 9ba280680f
NIFI-6569, NIFI-6570: Fixed bug that caused read timeouts not to occur with site-to-site. Fixed bug that caused site-to-site listener not to accept connections if there are no input/output ports on the root group - this used to be a valid check and was done to prevent spawning extra threads and doing extra work if no ports exist but now that we have site-to-site ports outside of the root group it's no longer a reasonable condition to check.
This closes #3658.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-09-02 11:51:03 +09:00
SandishKumarHN 26b203616e
NIFI-6552 - Kudu Put Operations
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3610.
2019-09-01 13:34:55 +02:00
Mike Thomsen fc3477bd69
NIFI-6424 Created a proper transit URL for Gremlin and OpenCypher services.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3571
2019-08-30 14:01:44 -04:00
Matt Gilman bfbf3b7532 NIFI-6290:
- Addressing review feedback.

This closes #3636

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2019-08-30 13:53:12 -04:00
Matt Gilman c6e500dfa4 NIFI-6290:
- Ensuring the first allowable value is selected.
2019-08-30 13:51:37 -04:00
Matt Gilman f318a8c685 NIFI-6290:
- Ensuring the first allowable value is selected.
2019-08-30 13:51:37 -04:00
Matt Gilman e5850fa1a3 NIFI-6290:
- Addressing timing issue when parameters listing request is not immediately resolved.
2019-08-30 13:51:37 -04:00
Matt Gilman 5b409d6fac NIFI-6418:
- Updating property table editors to not support parameter referencing when necessary.
2019-08-30 13:51:37 -04:00
Matt Gilman d18c98ce78 NIFI-6290:
- Updating comboEditor to allow for parameter reference.
2019-08-30 13:51:37 -04:00
Matt Gilman d93ae47afc NIFI-6346:
- Updating nfel to support referencing parameters.
2019-08-30 13:51:37 -04:00
Matt Gilman 7228496801 NIFI-6345:
- Updating property table editor to distinguish between available modes (el vs parameter reference).
2019-08-30 13:51:37 -04:00
Matt Gilman 75e452b45c NIFI-6277:
- Allow parameters to be referenced from properties that do not support EL.
2019-08-30 13:51:37 -04:00
Yolanda Davis 07baf723f7 NIFI-6429 - Added optional field to allow variables with null values to be included in reporting records. Default functionality will exclude variables with null values for backwards compatibility.
NIFI-6429 - checkstyle corrections
NIFI-6429 - change property descriptor name
2019-08-30 13:38:38 -04:00
mans2singh 5df6b0edbb
NIFI-6546 - Add JsonPath set value support
NIFI-6546 - Addressed review comments (double semi-colon, javadoc, error message, null init)
NIFI-6546 - Refactored test and added log error for JsonPath exception based on review
NIFI-6546 - Refactored tests based on review comments
NIFI-6546 - Removed redundant phone check and added constant for empty path
NIFI-6546 - Added brackets based on review comments

This closes #3646

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-08-29 18:13:09 -04:00
Mark Payne 254a84d74d
NIFI-6371: Fixed bug that caused root-level Controller Services to stay cached and therefore continually be validated, even after being deleted. Also cleaned up code that looked for specific text in Exceptions instead of of catching specific Exceptions
This closes #3614.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-29 16:24:40 -04:00
Scott Aslan f678c75d70
[NIFI-6282] manage parameters and parameter contexts
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
2019-08-29 14:48:53 -04:00
Mike Thomsen 2e554cbb67 NIFI-6588 Updated assembly pom file to move all graph components into a profile to save space in the convenience binary. 2019-08-29 10:58:25 -04:00
Mark Payne 4ae1fec78a
NIFI-6025: Include Processor 'scheduled state' (i.e., Enabled or Disabled) in the VersionedProcessor when pushing to Flow Registry and take into account when updating flows on the NiFi side
NIFI-6025: Include difference in Scheduled State as a Local Flow Difference

This closes #3546.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-29 10:35:03 -04:00
Andy LoPresto 01503b5187
NIFI-6187 - Added checkbox task for building on JDK 8 and JDK 11.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3680.
2019-08-29 11:42:19 +02:00
Evan Reynolds e2ca50e66a
NIFI-6367 - This closes #3563. more error handling for FetchS3Object
Signed-off-by: Joe Witt <joewitt@apache.org>
2019-08-28 19:42:47 -04:00
Bryan Bende b78aeb67a1 NIFI-6382 Fix verifyUpdate for parameter contexts to account for deletions
This closes #3661.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-28 11:29:00 -04:00
Mark Payne 59734dad88
NIFI-6028: Updates to ignore changes between local version of a flow and a remote version of a flow if the difference is the addition of a new property that has the default value
NIFI-6028: Code refactoring to address review feedback

This closes #3544.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-28 10:38:56 -04:00
Mark Payne 3c9426d287
NIFI-6595: Fixed bug in TailFile that caused it not to properly honor the Initial Start Position after state has been restored
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3675.
2019-08-28 10:33:13 +02:00
Andy LoPresto fbd201f116
NIFI-6298 Upgraded versions of Spring Framework in nifi-spring-processors and nifi-email-processors.
NIFI-6412 Upgraded Spring Security version in nifi-framework-bundle pom.xml.
NIFI-6412 Updated version for spring.security.version in nifi-kerberos-iaa-providers and nifi-ldap-iaa-providers pom.xml files.

This closes #3564.

Signed-off-by: Nathan Gough <thenatog@gmail.com>
2019-08-27 18:58:04 -04:00
Tamas Palfy 3de81d618e
NIFI-6509 - Date related issue in unit test VolatileComponentStatusRepositoryTest
As NIFI-6508 has already solved the original issue, turned this PR into a refactor within in the affected Test class.

This closes #3621.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 11:02:35 +09:00
Bryan Bende cc88dd428f
NIFI-6158 Fix conversion of Avro fixed type with logicalType decimal
This closes #3665.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-26 10:57:28 +09:00
Alessandro D'Armiento bb9758be2c
NIFI-6500 (Original commit messages left to preserve authorship credit for multiple contributors)
Expression Language now supports padding functions:
- padLeft:
  -  attr:padLeft(Int n, Char c) will prepend to the `attr` attributes the `c` character until the size `n` is reached
  -  attr:padLeft(Int n) will prepend to the `attr` attributes the `'_'` character until the size `n` is reached
- padRight:
  -  attr:padRight(Int n, Char c) will append to the `attr` attributes the `c` character until the size `n` is reached
  -  attr:padRight(Int n) will append to the `attr` attributes the `'_'` character until the size `n` is reached

- In both cases, the padding function returns the `attr` `String` as is if its length is already equal of higher than the desired size `n`
- Returns null if `attr` does not exist or in case desiredLenght is higher than Integer.MAX_INT

Further test cases:
- Returns null if the input string is null
- Returns a string full of padding if the input string is empty
Supports PaddingString instead of PaddingCharacter
Apply suggestions from code review
Applying style suggestions
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
style fixes
style fixes
Padding returns input string instead of null in case desired length is missing, is negative, or is overflowing
Better tests
doc update
less verbose parser notation
Doc and style fixes
Fixed `StringEvaluator.evaluate()` after rebase
Applying nitpicking suggestion
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Fixed `PaddingEvaluator` constructor issue
Removed unused import

This closes #3615

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-08-23 22:34:39 -04:00
Peter Wicks 0b74795578
NIFI-6562 Expose ExecutionNode in ProcessContext
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3656.
2019-08-20 11:13:55 +02:00
Malthe Borch 00f0f84856
NIFI-6508 - Fix test failure caused by timezone and/or DST difference
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-20 11:55:54 +09:00
Koji Kawamura 1cb1b00c09
NIFI-6477 Delete operate policy when its component is deleted
+1 from markobean.

This closes #3653.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-20 10:59:00 +09:00
Mike Thomsen e822a1612b
NIFI-6572 Added a workaround for GetMongoRecordIT that fixes a weird issue with Groovy where the Groovy addons don't appear to be getting recognized for java.util.Date. 2019-08-19 21:15:56 -04:00
Alessandro D'Armiento 81136bf550
NIFI-6502 RecordPath padding functions:
- PadLeft(label, desiredLength, paddingChar) prepends the paddingChar (or the default value '_' if a paddingChar is not provided) to the label string until desiredLength is reached.
- PadRight(label, desiredLength, paddingChar) appends the paddingChar (or the default value '_' if a paddingChar is not provided) to the label string until desiredLength is reached.
Added Apache license disclaimers
checkstyle fixes
Replaced functional interface with abstract method
Fixes and Further test cases:
- Returns null if the input string is null
- Returns a string full of padding if the input string is empty
wip support padding string
In order to be consistent with the feature introduced in #3615, RecordPath padLeft and padRight supports String padding.
Since nifi-record-path doesn't have the Apache Commons StringUtils dependency, the padding methods have been added to the available NiFi commons StringUtils class.
NIFI-6502 Updated top level NOTICE file to include citation for code borrowed from commons-lang3.
borrowed pad methods from lang3 StringUtils
Replaced `PadLeft` and `PadRight` record path functions with borrowed Apache Lang `StringUtils` padding methods and updated `nifi-assembly/NOTICE` accordingly

This closes #3613

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-08-19 17:26:19 -04:00
Matt Gilman 326bc91405
NIFI-5839:
- Introducing case insensitive group membership.
- Ensuring user identity and group name mapping is applied when the user identity or group name is inferred through group membership decisions.

This closes #3657.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-19 17:04:05 -04:00
Koji Kawamura e32689a602 NIFI-6436 Fix NPE at StandardPublicPort
Removing unnecessary ProcessGroup from PublicPort constructor.

This closes #3580.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-19 15:33:26 -04:00
Bryan Bende 16f3dbdc6a NIFI-6364 - Add CLI commands for parameter contexts
This closes #3628.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2019-08-19 11:16:44 -04:00
Mark Payne bd8342c5e0
NIFI-6382: On flow import and version change, create parameter context where necessary and add parameters where necessary
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context

NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry

NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException

This closes #3642.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:49 -04:00
Mark Payne bcf373a049
NIFI-6382: Allow Parameters to have null values
Signed-off-by: Bryan Bende <bbende@apache.org>
2019-08-16 16:47:27 -04:00
andrew 9c8f40415e
NIFI-6554 fix oid parsing for PutMongo processor
This closes #3650

Signed-off-by: Mike Thomsen <mthomsen@apache.org>
2019-08-16 05:45:48 -04:00
Brandon Devries 5f696482b0 NIFI-4775: Updating documentation and updating rocksdb version
NIFI-4775: Rolling back fix for NIFI-6416

NIFI-4775: Documentation improvements

+1 from markobean

This closes #3648

Signed-off-by: Brandon <devriesb@apache.org>
2019-08-15 15:28:48 +00:00
Jeff Storck f671a37aed
NIFI-5505 Added Java 11 build to Travis CI
Introduced stages to cache dependencies before build and test
Added --fail-never to dependency cache stage to allow dependency:go-offline to finish and update the cache with any dependencies that were able to be downloaded
Added download of AdoptOpenJDK 11.0.4 for the Java 11 build
Removed setting environment variables to enable easier sharing of cache between build, region/language settings are passed in via the command line when invoking maven
Formatted scripts to allow multiline commands for more readability
Replaced "exit ${PIPESTATUS[0]}" with "test ${PIPESTATUS[0]} -eq 0" to avoid using "exit" in script, which prevents Travis from completing the cache upload after the build completes
Install Maven 3.6.1 for the -ntp (no-transfer-progress) feature
Updated all build stages to exclude non-api nar and assembly modules to decrease build time
Dependency cache stages check if the cached maven repository doesn't exist or is empty before attempting to download dependencies

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3644.
2019-08-15 00:37:49 +02:00
Jeff Storck 33187f8058 NIFI-5176 NiFI builds on Java 11
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build.  Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
  Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
  Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
  URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.

NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service

NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11

This closes #3404.
2019-08-13 18:41:39 -04:00
Peter Turcsanyi b99cecd4e7
NIFI-6468: TestListS3 assertion fix
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3649.
2019-08-14 00:16:17 +02:00
Troy Melhase e841f4d5b7
NIFI-4256 Adds AWS Encryption Controller Service.
NIFI-4256 Adds AWS S3 FlowFile encryption attributes, more javadocs,
better names.

This closes #3574.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2019-08-13 12:18:46 -07:00
Joe Gresock 32c46f0bdd
NIFI-6468: Adding AWS S3 'requester pays' to Fetch and List processors.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3601.
2019-08-13 09:15:10 +02:00
Brandon Devries 7d77b464cc NIFI-4775: FlowFile Repository implementation based on RocksDB
+l from markobean.

This closes #3638.

Signed-off-by: Brandon <devriesb@apache.org>
2019-08-12 19:03:38 +00:00
Tamas Palfy 2baafcc2e6
NIFI-6534 Improve logging in PutHive3Streaming processor
Added logging for PutHive3Streaming when routing to failure

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3640
2019-08-09 17:19:26 -04:00
Alessandro D'Armiento 35d79eaf0f NIFI-6420 Controller Service references not properly tracked if multiple references from same component
<h1>Changes</h1>

-  Instead of keeping a Set of referencing `ComponentNode`, keep instead a Map which stores, for each referencing component, also the name of the property for which the service is referenced
   -  This is done to keep the `addReference()` idempotent
- `StandardControllerService.referencingComponents` changed from HashSet to HashMap
    -  `Key` is the combination of `ComponentNode.hashCode()` and property name
    -  `Value` is the referencing `ComponentNode`
- `ControllerServiceNode.addReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.removeReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.getReferences()` signature is left untouched, when it's called, the values of the `referencingComponents` map are turned into a Set, meaning a referencing component will keep being returned by this method until at least one of its properties still reference the `ControllerService`
- `StandardSchedulingContext.leaseControllerService()` uses a dedicated constant
- `FrameworkIntegrationTest` has a test which create a processor with two optional controller service properties and then verifies that having them referencing and dereferencing the same controller service doesn't cause the reported bug anymore

Removed SchedulingContext from ClojureScriptEngineConfigurator

Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>

This closes #3600.
2019-08-09 15:22:14 -04:00
Peter Turcsanyi a5bdecbd25
NIFI-5478: PutS3Object support for new Storage Classes
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3608.
2019-08-09 10:37:29 +02:00
Jeff Storck b938fc20a3
NIFI-6479 Fixes timezone issues in TestJdbcCommon
Dates calculated for method testConvertToAvroStreamForDateTimeAsLogicalType now handle timezone correctly
Added debug logging of date/time calculations and expectations

This closes #3630.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2019-08-09 16:29:40 +09:00
Jeff Storck 403c07d4e8
NIFI-6529 Updated TestFormatUtils.testFormatDataSize to use DecimalFormatSymbols when verifying results of FormatUtils.formatDataSize
NIFI-6529 Updated tests that fail with non en_US locales

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3639.
2019-08-08 21:39:20 +02:00