NIFI-6295: Refactored NiFiRecordSerDe to handle nested complex types
NIFI-6295: Incorporated review comments
NIFI-6295: Refactored unit tests for NiFiRecordSerDe
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3684
NIFI-6630 - base convert and goto parameter logic on any reference to a parameter contained in the text of the property value.
This closes#3718
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
NIFI-6649 - documentation update
NIFI-6649 - add debug logging for score and prediction information
NIFI-6649 - fix to ensure counts return minimum value of 0 if not infinite or NaN
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3719
NIFI-6644 - when deciding to show the convert prop option: enforce user has read and write permissions to the param context & the props are not for a controller service defined in the controller settings.
This closes#3713
ListGCSBucket duplicated files if they arrived not in alphabetical order.
The set storing the name of the latest blob (which was loaded with the highest
timestamp during the previous run of the processor) was cleared too early.
Also changed the state persisting logic: it is now saved only once at the end
of onTrigger() (similar to ListS3). Some inconsistent state (only blob names
without the timestamp) was also saved earlier.
This closes#3702.
Signed-off-by: Mark Payne <markap14@hotmail.com>
* NIFI-6510 Implement initial analytic engine
* NIFI-6510 Implemented basic linear regression model for queue counts
* NIFI-6510 Initial analytics REST endpoint and supporting objects
* NIFI-6510 Connect the dots for StatusAnalytics -> API
* NIFI-6510 Added poc engine with prediction model caching
(cherry picked from commit e013b91)
DFA-9 - updated logging and corrected logic for checking if not in backpressure
(cherry picked from commit a1f8e70)
* NIFI-6510 Updated objects and interfaces to reflect 4 prediction metrics
(cherry picked from commit 050e0fc)
(cherry picked from commit 9fd365f)
* NIFI-6510 adjustments for interface updates, added call to StandardEventAccess, updated interface to use connection id
(cherry picked from commit 14854ff)
DFA-9 - reduced snapshot interval to 1 minute
(cherry picked from commit 36abb0a)
* NIFI-6510 Split StatusAnalytics interface into Engine and per-Connection versions
* NIFI-6510 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly
* NIFI-6510 Revert "DFA-9 Remove redundant connection prediction interfaces as we can just use ConnectionStatusAnalytics directly"
This reverts commit 5b9fead1471059098c0e98343fb337070f1c75c1.
* NIFI-6510 Added prediction fields for use by UI, still need to be populated
* NIFI-6510 Analytics Framework Introduction (#10)
* DFA-9 - Initial refactor for Status Analytics - created additional interfaces for models, refactored callers to use StatusAnalytics objects with connection context. Implemented SimpleRegression model.
DFA-9 - added logging
* DFA-9 - relocated query window to CSA from model, adding the prediction percentages and time interval
* DFA-9 - checkstyle fixes
* NIFI-6510 Add prediction percent values and predicted interval seconds
(cherry picked from commit e60015d)
* NIFI-6510 Changes to inject flowManager instead of flow controller, also changes to properly reflect when predictions can be made vs not.
(cherry picked from commit 6fae058)
* NIFI-6510 Added tests for engine
(cherry picked from commit 6d7a13b)
* NIFI-6150 Added tests for connection status analytics class, corrected variable names
(cherry picked from commit 58c7c81)
* NIFI-6150 Make checkstyle happy
(cherry picked from commit b6e35ac)
* NIFI-6150 Fixed NaN check and refactored time prediction. Switched to use non caching engine for testing
* NIFI-6510 Fixed checkstyle issue in TestConnectionStatusAnalytics
* NIFI-6510 Adjusted interval and incorporated R-squared check
Updates to support multiple variables for features, clearing cached regression model based on r-squared values
Added ordinary least squares model, which truly uses multivariable regression. Refactor of interfaces to include more general interface for variate models (that include scoring support).
Ratcheck fixes
Added test for SimpleRegression. Minor fix for OLS model
fixed test errors
fixed checkstyle errors
(cherry picked from commit fab411b)
* NIFI-6510 Added property to nifi.properties - Prediction Interval for connection status analytics (#11)
* NIFI-6566 - Refactor to decouple model instance from status analytics object. Also allow configurable model from nifi.properties
NIFI-6566 - changes to allow scoring configurations for model in nifi.properties
NIFI-6566 - added default implementation value to NiFiProperties
NIFI-6566 - correction to default variable name in NiFiProperties, removed unnecessary init method from ConnectionStatusAnalytics
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3663
* NIFI-6585 - Refactored tests to use mocked models and extract functions. Added check in ConnectionStatusAnalytics to confirm expected model by type
* NIFI-6586 - documentation and comments
This closes NIFI-6586
Signed-off-by: Andrew I. Christianson <andy@andyic.org>
* NIFI-6568 - Surface time-to-back-pressure and initial predictions in the UI
* Add multi-line tooltips with detail for connection queue back pressure graphics.
* Add estimated time to back pressure to connections summary table.
* Add back pressure prediction ticks.
* add moment.js to format predicted time to back pressure
* tweak summary table headings to match data displayed. re-order connection summary columns
* NIFI-6568 - Properly sort the min estimated time to back pressure in the connection summary table. Also added a js doc comment.
* NIFI-6510 - add an enable/disable property for analytics
* NIFI-6510 - documentation updates for enable/disable property
* NIFI-6510 - UI: handle the scenario where backpressure predictions are disabled (#3685)
* NIFI-6510 - admin guide updates to further describe model functionality
* NIFI-6510 - code quality fixes (if statement and constructor)
* NIFI-6510 - log warnings when properties could not be retrieved. fixed incorrect property retrieval for score threshold
* NIFI-6510 Extract out predictions into their own DTO
* NIFI-6510 Optimize imports
* NIFI-6510 Fix formatting
* NIFI-6510 Optimize imports
* NIFI-6510 Optimize imports
* NIFI-6510 - Notice updates for Commons math and Caffeine
* NIFI-6510 - UI updates to account for minor API changes for back pressure predictions (#3697)
* NIFI-6510 - Fix issue displaying estimated time to back pressure in connection summary table when only one of the predictions is known.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3705
* NIFI-6510 Rip out useless members
* NIFI-6510 - dto updates to check for -1 value
* NIFI-6510 - checkstyle fix
* NIFI-6510 - rolled back last change and applied minNonNegative method
* NIFI-6510 Rip out useless members
* fixed issues converting sensitive props to parameters
* only include parameter value if it has changed (even for non-sensitive params)
* do not allow null parameter values when editing parameters.
* open existing add param dialog from property table
* convert prop to param, update param context
* keep add parameter dialog open until update of the context completes when converting property to param
* Updating parameter context while converting property to parameter.
* Disable the Apply button while updating
* When Cancel is pressed and there is an update in progress, issue the DELETE request to cancel the update.
* Show some status in the Add Parameter dialog when parameter context is updating following a property conversion.
* enforce character restrictions on property names.
* Add convert property to parameter to controller service config.
* Properly set sensitive setting when converting sensitive properties
* Allow converting of properties from controller services configuration into parameters
* Refactor addNewParameter method to be able to share common parts with convertPropertyToParameter
- Renamed PARTITION_VALUES to STATIC_PARTITION_VALUES for correctness and better understanding
- STATIC_PARTITION_VALUES descriptions clearly states that having that property filler implies Hive Static Partitioning
NIFI-6536: Additional documentation for Static Partition Values
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3631
Additional test case in TestEncryptContent
Apply suggestions from code review
nit picking
Apply suggestions from code review
nitpicking from mgaido91
Small fixes
Additional test case to check the provided cipher is actually used by inferring it from the ciphertext
Updated `EncryptContent` to use a pre-defined list of allowable PGP ciphers
Updated `EncryptContent` to check the validity of `PGP_SYMMETRIC_ENCRYPTION_CIPHER`
Updated test cases
Minor fixes
NIFI-6536 Fixed typo and strengthened validity checking for PGP symmetric cipher.
This closes#3664.
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
[NIFI-6282] when creating a parameter context inline during PG configuration set the newly created parameter context as the selected option
[NIFI-6282] If a request to update a parameter context fails, then update the button model to give the user the ability to Apply or Cancel again.
[NIFI-6282] address pr review comments
[NIFI-6282] remove es6 let
[NIFI-6282] update marshall parameters logic and add comments
[NIFI-6282] deterministic parameter usage listing, update CS status on PG PC change, expand all twisties by default, remove es6 const
[NIFI-6282] update regex and
[NIFI-6282] update parameter loading/serialization/marshalling
[NIFI-6282] use referencingComponents instead of affectedComponents
[NIFI-6282] activate Apply button for sensitive parameter set empty string change
[NIFI-6282] fix bug with PG parameters context menu enable
[NIFI-6282] only allow delete and recreate of a parameter with equivalent sensitivity
[NIFI-6282] display referencing components during parameter management as well as during the parameter context update
[NIFI-6282] display no value set in parameter table when parameter value is null
[NIFI-6282]
- Add ellipsis to referencing component names.
- Addressing issues canceling update requests.
- Addressing issues with incorrect service scope.
- Addressing issue showing the affected parameters.
NIFI-6412 Upgraded Spring Security version in nifi-framework-bundle pom.xml.
NIFI-6412 Updated version for spring.security.version in nifi-kerberos-iaa-providers and nifi-ldap-iaa-providers pom.xml files.
This closes#3564.
Signed-off-by: Nathan Gough <thenatog@gmail.com>
As NIFI-6508 has already solved the original issue, turned this PR into a refactor within in the affected Test class.
This closes#3621.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Introducing case insensitive group membership.
- Ensuring user identity and group name mapping is applied when the user identity or group name is inferred through group membership decisions.
This closes#3657.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-6382: Fixed bugs reported in code review: sensitive properties referencing parameters should be exported to registry; when importing a versioned flow, any new parameters should be added to the parameter context
NIFI-6382: Updated logic for handling parameter references with sensitive properties when integrating with nifi registry
NIFI-6382: If user does not have permissions to READ a parameter context with a given name when importing a flow or changing flow version, ensure that NiFi properly generates an AccessDenied messages instead of throwing NullPointerException
This closes#3642.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-4775: Rolling back fix for NIFI-6416
NIFI-4775: Documentation improvements
+1 from markobean
This closes#3648
Signed-off-by: Brandon <devriesb@apache.org>
Added "jigsaw" profile to multiple modules, which auto-activates when building with Java 11 and adds several dependencies: jaxb, activation, and annotation-api.
Updated SslContextFactory to return tuple of socket factory with trust manager for issue with okhttp api changes in java 9+
Updated TestGet/PostHTTPGroovy to use default SSL context to get default cipher suites
Updated StandardTemplateDAOSpec.groovy, was using a 37-character UUID, shortened to 36 characters.
Multiple tests updated to specifically use TLSv1.2, since two-way TLSv1.3 for some component tests fail during the Java 11 build. Needs more investigation.
Replaced GString with String concatenation for map key's value in ScriptedReportingTaskGroovyTest to avoid runtime error of casting GString to String when retrieved from a map that is storing the GString
Removed nifi-toolkit-api plugin configuration of maven-compiler-plugin, it is inherited
Resolved deprecation errors during Groovy compile for bouncycastle, Extension should be used in place of X509Extension
Fixed JNAOverridingJUnitRunner's creation of the classpath for the custom classloader created to be able to mock jna classes
Removed import of IOUtils (from the wrong package) from InferenceSchemaStrategy
Updated maven-compiler-plugin version to 3.8.1
TLS (default in Java 11 is TLSv1.3) working for Site-to-Site client tests after upgrading JDK installation to JDK 11.0.3, httpclient5 5.0-beta4, and httpasyncclient 4.1.4
HttpNotificationService updated to find and use the first found X509TrustManager rather than casting directly from the array of TrustManagers returned from SslContextFactory
Removed unnecessary throws declaration from getSslSocketFactory method
Replaced DnsContextFactory.class.getName() with a string to avoid having to export/open the jdk.naming.dns module
Updated TestGetIgniteCache and TestPutIgniteCache.java to skip tests on Java 11+ (via Assume), and noted the check should be removed once a version of Ignite is released that supports Java 11
Updated SpringContextProcessor to create proper ClassLoader and uses URLClassloader.getResource() instead of URLClassloader.findResource() in isConfigResolvable.
Due to the application classloader no longer being a URLClassLoader in Java 9+, the URLClassLoader created in isConfigResolvable no longer has explicit access to the parent's resources as URLs.
URLClassLoader.getResource() searches the parent classloaders, and must be used instead of URLClassLoader.findResource() which only searches the URLs in the URLClassLoader and does not search the parent classloaders.
NIFI-5176 Moved exclusion of jdk.tools to the jigsaw profile in the POMs of nifi-hbase_1_1_2-client-service and nifi-hbase_2-client-service
NIFI-5176 Updated site-to-site-client's POM to use properties to establish the correct httpclient dependency for when building with Java 11
This closes#3404.
<h1>Changes</h1>
- Instead of keeping a Set of referencing `ComponentNode`, keep instead a Map which stores, for each referencing component, also the name of the property for which the service is referenced
- This is done to keep the `addReference()` idempotent
- `StandardControllerService.referencingComponents` changed from HashSet to HashMap
- `Key` is the combination of `ComponentNode.hashCode()` and property name
- `Value` is the referencing `ComponentNode`
- `ControllerServiceNode.addReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.removeReference()` now requires a tuple (referring `ComponentNode`, property name)
- `ControllerServiceNode.getReferences()` signature is left untouched, when it's called, the values of the `referencingComponents` map are turned into a Set, meaning a referencing component will keep being returned by this method until at least one of its properties still reference the `ControllerService`
- `StandardSchedulingContext.leaseControllerService()` uses a dedicated constant
- `FrameworkIntegrationTest` has a test which create a processor with two optional controller service properties and then verifies that having them referencing and dereferencing the same controller service doesn't cause the reported bug anymore
Removed SchedulingContext from ClojureScriptEngineConfigurator
Co-Authored-By: Marco Gaido <marcogaido91@gmail.com>
Signed-off-by: Mark Payne <markap14@hotmail.com>
This closes#3600.
Dates calculated for method testConvertToAvroStreamForDateTimeAsLogicalType now handle timezone correctly
Added debug logging of date/time calculations and expectations
This closes#3630.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Added nifi-parameter module
- Added Parameter to nifi-api
- Added ParameterContext and ParameterLookup
- Updated EL to support Parameters
- Updated backend so that any property can make use of Parameters
- Added web endpoint for Parameter Contexts
- Updated Templates to make use of Parameter Contexts
- Updated Versioned Flows to make use of Parameter Contexts
- Updated Stateless NiFi to support Parameters and take Parameters as part of the configuration, not Variables.
NIFI-6380: Addressed review feedback and fixed bugs; added additional unit and integration tests to verify
NIFI-6380: Added Description to Parameter Context
NIFI-6380: Fixed checkstyle violations
NIFI-6380: Fixed bug that caused updating a Parameter Context to fail if the name is provided and unchanged
NIFI-6380: If parameter is being deleted, don't worry about its sensitivity flag. This addresses a bug where the deletion of a Sensitive Parameter would require that the Parameter be submitted with a value of null and a sensitivity flag of true; else it would provide an error indicating that the parameter can't be changed from sensitive to non-sensitive. Now, the sensitivity flag is ignored.
NIFI-6380: Fixed bug around unsetting Process Group's Parameter Context
NIFI-6380: Moved lastRefreshed timestamp from ParameterContextDTO to ParameterContextsEntity and renamed to currentTime to match the pattern of ControllerServicesEntity rather than FlowHistory. Added parameterContextId to ProcessGroupFlowDTO
NIFI-6380: Added additional integration tests around escaped parameter references
NIFI-6380: Additional tests and bug fix for referencing EL from within another EL Expression
NIFI-6380: Created ParameterEntity to house a 'canWrite' flag for parameters and updated ParameterContext to use it. Updated ParameterContextUpdateRequestDTO to include a Set<AffectedComponentEntity> indicating the components that are affected by the update
NIFI-6380: Addressed review feedback
NIFI-6380: Addressed additional review feedback, mostly around code cleanup
NIFI-6380: Bug fix
NIFI-6380: Addressed more review feedback; fixed a couple of minor bugs encountered when testing
NIFI-6380: Bug fix around Parameter escaping for properties that support Expression Language. Fixed inconcsistency in Authorization hierarchy between /parameter-contexts/1234 and /parameter-contexts to ensure that the parent of /parameter-contexts is /controller, regardless of how we arrive at the /parameter-contexts resource
NIFI-6380: Fixed but around using ProcessContext#newPropertyValue(String) that previously resulted in causing the Parameters to be evaluated a second time, thereby ignoring escape characters. Updated Integration Test to verify behavior. Also fixed bug in LoadBalanceQueueIT as it was a one-liner and noticed it while running tests to verify other behavior
NIFI-6380: Ensure that if Processor or Controller Services references or de-references a Parameter that user making the change has READ policy on the Parameter Context
NIFI-6380: Verify permissions on Parameter Context when creating processor/controller service and when moving between groups, insantiating template, reverting/change flow version
NIFI-6380: Addressed problem that resulted from rebasing against master; fixed bug in showing the Affected Components for a particular Parameter Context Update
NIFI-6380: Fixed NPE that occurs when attempting to update controller-level Controller Service
NIFI-6380: Updated validation error text when referencing parameter from controller-level controller service or reporting task
NIFI-6380: Ensuring permissions for copy/paste when components reference parameters.
NIFI-6380: Do not require READ policy on Parameter Context in order to un-reference a parameter, only to reference one.
NIFI-6380: Restricted parameter names to match those in the Feature Proposal. Fixed bug that allowed copy & paste of a PG for which user does not have READ policy for a child/descendant PG's parameter context
This closes#3536
Fix imports auto formatted by intellij
NIFI-6487 Fix WriteAttribute documentation
This closes#3603.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Unified unit tests
Added custom validation cases for MIN_RECORDS and MAX_RECORDS enforcing they are greater than zero.
While MIN_RECORDS > 0 can fail individually, MAX_RECORDS > 0 validator cannot fail without having also at least another validation step (either the MIN_RECORDS > 0 or the MAX_RECORDS > MIN_RECORDS) to fail, since MIN_RECORDS is a required property with default value 1
This closes#3607.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5856: Fixed checkstyle violation
NIFI-5856: Addressed review feedback
NIFI-5856: When assigning a matching Controller Service at a higher level for Process Group import/update, ensure that we consider the User's read permissions when selecting the Controller Service
This closes#3553.
Signed-off-by: Bryan Bende <bbende@apache.org>
This modification updates the getStatusHistory method of
VolatileComponentStatusRepository to utilize the start, end, and
preferredDataPoints parameters when retrieving status histories.
When calling the various get<XXXX>StatusHistory methods of
VolatileComponentStatusRepository, they are all passed start, end,
and preferredDataPoints values to allow the filtering of histories
according to the provided parameters. But the follow-on method calls ignore
those values completely. This ticket updates those methods to honor the
parameters and filter the returned Dates accordingly.
This closes#3579.
Signed-off-by: Mark Payne <markap14@hotmail.com>
CSVReader and CSVRecordSetWriter controller services and also ConvertExcelToCSVProcessor
support EL for Value Separator, Quote Character and Escape Character properties.
NIFI-6318: Fixed null checks and compound OR expression.
NIFI-6318: RecordSetWriterFactory.createWriter() changes.
NIFI-6318: Initialize CSVFormat in onEnabled() if there are no dynamic formatting properties.
NIFI-6318: Comment Marker supports EL.
NIFI-6318: Various review changes.
This closes#3504.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Updated pom files regarding groovy dependencies. groovy-all was changed to a pom type dependency
Added Groovy Bintray as a plugin repository in root pom
Upgraded spock-core to version 1.3-groovy-2.5 and added exclusion of groovy dependencies to force spock to use the version from dependencyManagement
Updated groovy-eclipse-batch to use the groovy.version property to determine the version
Updated groovy-eclipse-compiler to 3.4.0-01
Updated maven-compiler-plugin to use maven.compiler.source and maven.compiler.target properties for source and target configuration properties
Removed configuration and dependencies sections from nifi-toolkit-admin and nifi-toolkit-encrypt-config maven-compiler-plugin configurations so that the configuration from the root pom's maven-compiler-plugin is inherited
Removed dependencyManagement from nifi-jetty module, version of groovy modules will be inherited from root pom
Removed maven-compiler-plugin configuration from nifi-toolkit-api so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated spock-core dependencyManagement to 1.3-groovy-2.5
Fixed AESSensitivePropertyProviderTest issue with Groovy creating KEY_256_HEX slice of the wrong size due to BigDecimal being used as the result of the division; using intdiv to force an integer result creates the correctly sized array
Added groovy-json test dependency to nifi-web-security
Removed maven-compiler-plugin configuration from nifi-lookup-services, nifi-mock-record-utils, and nifi-web-utils so that the configuration from the root pom's maven-compiler-plugin is inherited
Updated root pom pluginManagement to specify version 3.8.0 of maven-compiler-plugin
Added maven-compiler-plugin config to nifi-toolkit-admin and nifi-toolkit-encrypt-config to use groovy-eclipse-compiler during the compile phase so that the groovy-based tools are compiled
Addressed deprecated CliBuilder and OptionAccessor usage in nifi-toolkit-encrypt-config, those classes were moved from groovy.util to groovy.cli.commons
Removed getInner() usage from nifi-toolkit-encrypt-config, method no longer exists causing the tests to crash
Updated CryptographicHashAttributeTest to use java.time classes instead of java.util.Date
Updated nifi root POM's groovy-test dependency to be test-scoped
Added properties for specifying groovy versions for several modules: nifi, nifi-groovyx-bundle, nifi-scripting-bundle, nifi-toolkit
Established dependency management for groovy-all:pom:2.5.4, and added that dependency to several modules: nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-admin, nifi-toolkit-encrypt-config
Added groovy version property usage to several modules that established a dependency on a groovy submodule that was not listed in its own or inherited dependency management
Removed unused build-helper-maven-plugin from nifi-toolkit-api's POM
Removed unnecessary groovy-eclipse-compiler build plugin config from nifi-web-utils' POM to use the inherited config for that plugin
Updated several modules' NOTICEs to include appropriate Groovy NOTICE content
Updated to list groovy-all:pom:2.5.4 and its transitive submodule dependencies to nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, and nifi-toolkit-assembly NOTICEs
Added missing groovy-all🫙2.1.6 NOTICE to nifi-hive-nar and nifi-hive_1_1-nar NOTICEs
Added missing groovy-all🫙2.4.11 NOTICE to nifi-hive3-nar NOTICE
Updated to list groovy-all🫙2.4.16 nifi-other-graph-services-nar NOTICE
Removed Groovy NOTICE content from nifi-record-serialization-services-nar NOTICE, no Groovy modules are included in the NAR
NIFI-5254 Updated several modules' NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with Groovy modules which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-hive-nar, nifi-hive3-nar, nifi-hive_1_1-nar, nifi-other-graph-services-nar, nifi-scripting-nar, nifi-toolkit-assembly
NIFI-5254 Updated NOTICEs and LICENSEs to include appropriate content from the LICENSEs and NOTICEs bundled with modules used by Groovy 2.5.4 which in turn are bundled with NiFi binary artifacts: nifi-assembly, nifi-groovyx-nar, nifi-scripting-nar, nifi-toolkit-assembly.
The following modules' LICENSE and NOTICE content were added: Apache Ant, Apache Commons CLI, JLine, JUnit Platform/Jupiter
This closes#3547
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
* NIFI-5417: Add missing component status and metrics to S2SReportingTask and PrometheusReportingTask
* NIFI-5417: Added executionNode to schema and doc
This closes#3554
Signed-off-by: Yolanda M. Davis <yolanda.m.davis@gmail.com>
NIFI-6271, fix incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set
NIFI-6271, fix incoming flowfile attributes don't copy into output flowfiles when Output Batch Size is set
replace getAttribute(uuid) with getAttribute(CoreAttributes.UUID.key()
fix checkstyle violation
This closes#3575.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5562 - Removed unnecessary usage of Guava
NIFI-5562 - Updated Guava version to 28.0-jre
NIFI-5562 - Upgraded Guava to 28.0-jre and Curator to 4.2.0
This closes#3577
Signed-off-by: Mike Thomsen <mthomsen@apache.org>
NIFI-6387 RetryFlowFile
NIFI-6387 Maximum Retries support FLOWFILE_ATTRIBUTES scope
NIFI-6387 Fixed reuses descriptions for clarity
This closes#3541.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6004 Improve testing of PutFile file and directory permissions
NIFI-6004 Typo in regex
NIFI-6004 Updates based on review
This closes#3294.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5973 More comments and better defaults for the shell provider.
NIFI-5973 Fixed bug where user was being retrieved by identifier when identity was provided.
NIFI-5973 Fixed a formatting string in the OS X shell commands.
Updated testing conditions to run IT in OS X environment.
Changed unit test to provide identity rather than identifier.
This closes#3537.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-5537 Added skeletons of services, apis, etc.
NIFI-5537 Renamed classes to reflect that they are for cypher and not neo4j specific
NIFI-5537 Added start of client API for doing streaming execution.
NIFI-5537 Completed refactor of unit and integration tests.
NIFI-5537 Added OpenCypherClientService.
NIFI-5537 Updated to gremlin-core 3.3.5 to get better SSL configuration options.
NIFI-5537 Added SSL support to OpenCypher client service.
NIFI-5537 Updated L&N.
NIFI-5537 Updated to 1.10.0-SNAPSHOT.
NIFI-5537 Removed Mac-specific documentation in int test.
NIFI-5537 Updated a few properties to add EL.
NIFI-5537 Added GremlinClientService to support gremlin as well.
NIFI-5537 refactored everything to be a graph bundle.
NIFI-5537 Updated documentation.
NIFI-5537 Fixed assembly.
NIFI-5537 Updated Jackson.
NIFI-5537 Moved MIT section per code review guidance.
NIFI-5537 Changed provenance event and made it more explicit.
NIFI-5537 Updated processor config file.
NIFI-5537 Made changes requested in a code review.
NIFI-5537 Added documentation about driver configuration.
NIFI-5537 Remove output flowfile if an exception is thrown.
NIFI-5537 Updated configuration properties.
NIFI-5537 Made port and path configurable in the tinkerpop family of components.
NIFI-5537 Upgraded to OpenCypher cypher-gremlin-neo4j-driver 0.9.13 because it's the most recent version that JanusGraph supports.
NIFI-5537 Added warning about Janus 0.3.X to int test.
NIFI-5537 Added TinkerPopClientService interface.
NIFI-5537 Changed to mockito-core from mockito-all.
NIFI-5537 Added documentation about Gremlin for first time users w/ NiFi.
NIFI-5537 Renamed bundle based on review comments
NIFI-5537 Updated neo4j setup and version documentation and default values based on review comments
NIFI-5537 - Changed exception, logging, and json parser based on review comments
NIFI-5537 - Check debug enabled before logging at debug level
NIFI-5537 - Updated property displayname/description and corrected test file names
NIFI-5537 - Added note about dummy password required in case authentication is disabled on server
NIFI-5537 - Use SSLContext
In nifi-metrics, added build-helper-maven-plugin to enable groovy tests to run, since there are no java tests or src/test/java path in the module
Replaced yammer/dropwizard's VirtualMachineMetrics usage in multiple modules with JvmMetrics interface and JmxJvmMetrics implementation in nifi-metrics
Updated nifi-datadog-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-scripting-bundle to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTask to use JmxJvmMetrics instead of VirtualMachineMetrics
Updated nifi-reporting-utils pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated nifi-ambari-reporting-task pom to use nifi-metrics:1.10.0-SNAPSHOT
Updated SiteToSiteMetricsReportingTask to use nifi-metrics:1.10.0-SNAPSHOT
Updated ScriptedReportingTaskGroovyTest to use regular strings instead of GStrings due to class cast exception
Updated nifi-prometheus-reporting-task to use nifi-metrics:1.10.0-SNAPSHOT
Updated PrometheusMetricsUtil to use LF endings instead of CRLF
Updated nifi-metrics-reporting-bundle and its submodules to use nifi-metrics:1.10.0-SNAPSHOT
nifi-metrics-reporting-task metrics-graphite version upgraded from 3.1.2 to 4.1.0
Removed unused property yammer.metrics.version from nifi-jolt-record-bundle pom
Updated root NOTICE to include derived work in nifi-metrics
Removed duplicate nifi-assembly NOTICE clauses for yammer/dropwizard metrics
Removed unnecessary inclusion of Yammer Metrics in Kafka 0-9, 0-10, 0-11, 1-0, and 2-0 NARs' NOTICE files since the Kafka dependency that transitively includes it is test-scoped.
Removed unnecessary mock interaction in GraphiteMetricReporterServiceTest, Mockito 2.28.2 operates in strict mode and flags unused unused interactions.
NIFI-6354: Added LICENSE and NOTICE for nifi-kafka-2-0-nar
Updated NOTICE in kafka nar modules to reflect the actual module names at the top of the NOTICE
Added yammer metrics NOTICE clauses to nifi-kafka-0-11-nar
Added yammer metrics NOTICE clauses to nifi-kafka-1-0-nar
NIFI-6355: Extended NOTICE clauses for yammer/dropwizard metrics in multiple modules
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3520
NIFI-6330 - Removed jetty.version property declaration in nifi-prometheus-reporting-task pom.xml. This version should be inherited from the root pom.
NIFI-6330 - New release of Jetty came out 2 days ago. Upgraded and verified NiFi functionality again with smoke tests including ListenHTTP, secure clustering, site-to-site
Signed-off-by: Joe Witt <joewitt@apache.org>
Fixed test failures in nifi-couchbase-processors, BinaryDocument matcher replaced with ByteArrayDocument
Fixed test failures in nifi-riemann-processors, anyInt() matcher replaced with anyLong() matcher, calling method passes a long, not int
Removed unnecessary method mocks from nifi-toolkit-tls tests, TlsCertificateAuthorityServiceHandlerTest and TlsCertificateSigningRequestPerformerTest, since those were flagged by Mockito as unnecessary (they're unused)
Removed explicit mockito dependency version in nifi-gcp-processors pom to inherit version from nifi's pom.xml
Updated ArgumentMatchers in Kafka 0.10, 0.11, 1.0, and 2.0 processor tests, since in Mockito 2.x, the "any" matchers no longer allow nulls
Updated ArgumentMatchers in nifi-jolt-transform-json-ui, since in Mockito 2.x, the "any" matchers no longer allow nulls
Removed unnecessary method mocks from MetricsReportingTaskTest
Updated TestStandardRemoteGroupPort to return Long instead of Integer for test flowfile.size() invocations
Updated AbstractCassandraProcessor to include keyspaceProperty.getValue() in null check
Updated SimpleProcessLogger and TestSimpleProcessLogger, vararg matching does not work the same in Java 8 and 11
Updated TestStandardProcessScheduler to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated TestPutHiveStreaming to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated FetchParquetTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Updated ControllerSearchServiceTest to allow null values during mock invocations, Mockito 2.x no longer allows nulls in those matchers
Removed usage of Whitebox from GetAzureEventHubTest due to Mockito 2.x, replaced with FieldUtils
Removed usage of Whitebox from StandardOidcIdentityProviderTest due to Mockito 2.x, replaced with FieldUtils
Updated apache-rat-plugin configuration in root POM to make use of useIdeaDefaultExcludes which makes the rat plugin exclude IntelliJ artifacts
Updated several modules to use mockito-core instead of mockito-all (discontinued in Mockito 2.x)
Updated nifi-site-to-site-reporting-task tests to be compatible with Mockito 2.x
Ignored TestPutJMS tests; the tests need to be refactored to work with Mockito 2.x, but the processor is deprecated. Refactor may be done in a separate PR.
Adjusted several mock interaction iterations to 0 for TestPublishKafkaRecord_* tests. Mockito 2.x flagged several interactions as unused and were adjusted to 0 interactions.
Updated PowerMock and Mockito dependencies to exclude transitive dependency on bytebuddy, added explicit dependency on bytebuddy 1.9.10 so that PowerMock and Mockito use the same version. Bytebuddy 1.9.3 (used by PowerMock 2.0.2) did not allow for the mocking of final/private classes, bytebuddy 1.9.10 (used by Mockito 2.28.2) does.
Updated TestSiteToSiteProvenanceReportingTask use of InvocationOnMock.getArgument to use objects for the resulting object rather than primitives
Removed unnecessary stubs from evtx tests, Mockito 2.x defaults to strict mocks
Fixed classloader issue with tests in nifi-windows-event-log-processors module that use JNAJUnitRunner when Mockito mocked JNA classes (Kernel32)
Addressed Mockito-related deprecation warnings
Import cleanup
This closes#3533
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6362 - Upgraded both checkstyle plugins to latest. Added a checkstyleCache file and property.
NIFI-6362 - Downgraded maven-checkstyle-plugin to 3.0.0 instead as there was an issue with the latest version and @throws Javadoc checkstyle module. Fixed some checkstyle issues. Changed checkstyleCache file to .checkstyleCache. Added to .gitignore.
NIFI-6362 - Changed checkstyleCache file location into target/, and is no longer a hidden file. Removed .checkstyleCache from .gitignore as it is no longer required.
NIFI-6362 - Grep pattern no longer required.
This closes#3522.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6301 - Fixed unit test. Added comments.
NIFI-6301 - Removed unused rule from test.
NIFI-6301 - Changed read() methods to use a boolean instead. Updated comments.
NIFI-6301 - Fixing checkstyle errors.
NIFI-6301 - Added an XXE Validator to do a simple regex check on files that contain XXE declarations. Added unit tests and related XML test files.
NIFI-6301 - Forgot license headers. Fixed a unit test.
NIFI-6301 - Refactored duplicate code for error handling.
NIFI-6301 - Removed unused imports.
NIFI-6301 - Slightly changed messaging language.
NIFI-6301 - Added logic to catch multiline XXE attacks.
Added logging messages.
Updated unit test.
This closes#3507.
Co-authored-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-5902 Updated GeoEnrichIPRecord with more error handling.
NIFI-5902 Made changes requested in a review.
NIFI-5902 Updated EL support on database path.
NIFI-5902 Made updates based on code review.
NIFI-5902 Made two changes from a code review.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3231
When PutFile uses 'replace' conflict resolution and max files, there is an issue when the folder has X files, and the limit is also X. The processor fails instead of replacing it, leaving X files. This commit fixes that issue.
This closes#3524.
Signed-off-by: Andres Garagiola <andresgaragiola@gmail.com>
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Added new property to the processors. If it is empty, then the behaviour is the same as before.
Adde unit test to check if KafkaProducer receives the new property.
Testing Done:
Unit tests.
Connecting to kafka and verifying transactionalId-s in kafka server log.
fixing review findings
This closes#3439.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6315 Renaming public ports during import/change-version when name alreadye exists, and ignorning name change from flow diffs
NIFI-6315 Ensuring public ports get renamed during copy/paste
NIFI-6315 Set max concurrent tasks on Port based on value from VersionedPort
This closes#3488.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5435 Fixed Rebase Issue
NIFI-5435 changes in capability description of reporting task
NIFI-5435 Changes in capability description of reporting task
NIFI-5435 Changes to build Property Descriptors in static constructor
NIFI-5435 Indentation changes
NIFI-5435 Test code changes
NIFI-5435 Minor code changes
NIFI-5435 Changes related to client authentication
NIFI-5435 Prometheus metrics changes
NIFI-5435 Added process group name and id to metrics
This closes#3257
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6323 Changed URLs for splunk.artifactoryonline.com to use HTTPS (certificate validity warning in browsers, but command-line connection using openssl s_client is successful).
NIFI-6323 Changed URLs for XMLNS schema locations to use HTTPS (the XMLNS and schema identifier remain http:// because they are not designed to be resolvable).
NIFI-6323 Fixed Maven XML schema descriptor URLs.
This closes#3497
- set cleanup thread scheduling to 5 seconds in order to prevent stale request staying in the cache for a long time
- disable timeout handling in HandleHttpRequest in order to prevent a new flowfile being generated at request expiration
- use HttpServletResponse.SC* status codes everywhere in HandleHttpRequest, get rid of importing javax.ws.rs
- add some more error logging to make bug investigations easier
- add a short message to 503 error responses to make bug investigations easier
This closes#3490.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6280 - Updated terminology in JwtAuthenticationFilter to authentication instead of authorization. Added stricter token parsing using an explicit regex pattern. Added tests.
NIFI-6280 - Updated terminology from Authorization to Authentication.
NIFI-6280 - Updated the access logout method to use getNiFiUserIdentity(). Updated javascript logout method to handle errors.
NIFI-6280 - Fixing checkstyle issues.
NIFI-6280 - Added some javadoc comments and logging. Renamed some variables for clarity. Fixed handling of exception when JWT does not match expected format.
NIFI-6280 - Cleaned up checkstyle, increased log severity level for logout action, and cleaned up Groovy syntax in test.
This closes#3482.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Specify remote access at port creation.
Incorporated comments, and finished refactoring.
Renamed RootGroupPort to PublicPort.
Fix error message for creating a connection from a child PG having only PublicPorts.
Enhanced ProcessGroup instances rendered in the parent ProcessGroup
Loosen Port move check, allow moving public port between PG.
Show 'Remote NiFi Instance' info on Connection dialogs
Make labels narrative.
'Within Remote Group'.
Fixed DTO (de)serialization.
Return null only if all values are null.
This closes#3351.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Separated into nifi-fn-core, nifi-fn-bootstrap, nifi-fn-nar, nifi-fn-assembly; fully unpacks nars and runs flows
- Rebased against master and updated to version 1.10.0-SNAPSHOT
- Removed dependency on nifi-framework-core
- Added LICENSE/NOTICE files
Updated NOTICE with current copyright year for Jetty dependencies
Updated Jetty SSLContextFactory usage, invoke setEndpointIdentificationAlgorithm(null) on server SslContextFactory instances
Updated TestInvokeHttpSSL to provide a separate client keystore, rather than reusing the server's keystore
Regenerated nifi-standard-processors keystore and truststore, added client keystore
Updated ITestHandleHttpRequest, TestInvokeHttpSSL, TestInvokeHttpTwoWaySSL, and TestListenHTTP to use a separate client keystore instead of reusing the server's keystore. Also updated the tests to separately test one-way and two-way SSL
NIFI-6196 - Setting the endpoint identification algorithm to null for sockets to ensure certificates without SANs applied correctly still work. We can change this in a new NiFi release with other potentially breaking changes.
This closes#3426.
NIFI-6085 - Updated logOut method to use NiFiUserUtils and updated tests.
NIFI-6085 - Added some more integration tests.
NIFI-6085 Suppressed stacktrace when token is used after being invalidated.
This closes#3362.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-5933 Removed unused file with sun.* import
NIFI-5933 Removed unsafe of sun.*.
NIFI-5933 Added openjdk8 to test
This closes#3234
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-6253 - simplifying loading of a new process group, handle failure.
NIFI-6253 - Expose nfCanvas.loadProcessGroup functionality through nfCanvasUtils.loadProcessGroup
NIFI-6253 - optional groupId approach to reload
This closes#3458
* NIFI-6228 Made ElasticSearchClientServiceImpl able to use just a trust store for SSL.
* NIFI-6228 Replaced buildSslContext with SSLContextService's helper method.
NIFI-6171 re-added lookupEmail() as fallback
NIFI-6171 additional OIDC scopes via nifi.properties
NIFI-6171 alternative user identification (instead of email) via nifi.properties
NIFI-6171 changed lookupEmail() so that any configured claim can be fetched fro the UserInfo endpoint
This closes#3398
This closes#2346
NIFI-5986 Refactored Stop & Configure feature to enable bulletin/thread notifications and terminate capability in processor dialogs. Also added feature as a menu item to the canvas context menu.
NIFI-5986 Refactored Stop & Configure feature to enable a status bar in a dialog that conveys bulletin/thread notifications and buttons.
NIFI-5986 Refactored Stop & Configure feature to decouple status bar from modal component, updated styling and revised graph synchronization process.
NIFI-5986 Refactored Stop & Configure feature to improve status bar button hide/show functionality.
NIFI-5986: Rebased and resolved conflicts.
NIFI-5986 - Refactored Stop & Configure statusbar observer, as well as processor dialogs to remove duplicative code.
This closes#3281
What this PR changed:
- Standardized required name error message to 'The name of the xxxx must
be specified'
- Added nfErrorHandler.handleConfigurationUpdateAjaxError
- Replaced existing custom error handling for 400 status code
- Replaced handleAjxError with this where appropriate
- Standardized config error dialog title to 'Configuration Error'
- Close the configuration dialog when configuration is successfully
updated
Above convention applied to following components:
- ProcessGroup
- Input/OutputPort
- RemoteProcessGroup
- RemoteGroupPort
- Template
- Connection
- ControllerService
- Label
- Processor
- ReportingTask
- NiFi Settings/GENERAL tab
- Variable
- User
NIFI-6170 Provide consistent UX at Connection config
Fixed Label.
Incorporated review comments
Removed unnecessary else block.
Disable canvas when the message pane is shown
This closes#3401
Updated usage of deprecated FormatUtils.getTimeDuration to FormatUtils.getPreciseTimeDuration
Implemented prioritized handling of appending the default realm
A realm-qualified principal will not be modified before authentication
A principal shortname will have Default Realm appended to it when it is not blank before authentication
A principal shortname will not be modified if Default Realm is blank, and the underlying kerberos implementation will append the default_realm configured in krb5.conf
In nifi-security-util
added KerberosPrincipalParser for determining the realm of a kerberos principal
added tests for KerberosPrincipalParser
updated pom with spock-core as a test dependency
This closes#3446.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Before this fix, NiFi loses information about created versioned flow in case of subsequent snapshot creation failure, and NiFi API returned an error response
- This commit makes:
- The created versioned Flow information is stored even if subsequent snapshot creation fails
- NiFi API to return a successful 200 response in that case, but return versioned flow status as SYNC_FAILURE with an explanation. NiFi UI shows a popup error dialog with the explanation.
- Versioned flow status will be LOCALLY_MODIFIED if the latest version is 0.
This closes#3134.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-6201 Fixed existing checkstyle complaint on missing Javadoc element.
NIFI-6201 Restored "connection refused" text to relevant registry sync error message.
NIFI-6201 Simplified error logging for bundle not found issues and removed unused bundle coordinate methods.
This closes#3427.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- handle multi-select copy/paste better.
- remove dimension info from getOrigin function
- handle pasting better when pasted items should be centered and not offset from the original.
- fixed strict mode and removed console.logs
- Fixing pasting logic when trying to center the items on the canvas.
This closes#3383
Fixed indentation errors to pass checkstyle-checks
Added Evaluation Modes as per discussion in PR thread
Adding exclusions of test files on rat plugin
Added new property 'Line-by-Line Evaluation Mode' and refactored common code
This closes#3375.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
* Added lodash utility library to leverage its throttle capability (and many other useful functions in the future).
* Made lodash available in all JSP's so it could be imported into nf-common (or any component for that matter).
* Added a throttle function to nf-common that just wraps _.throttle
This closes#3393
- hooking in property INDEX_RETRY_PERIOD to list of properties
- fixing property description
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3402
This change is based on https://github.com/apache/nifi/pull/2489
I have reproduced the issue with some additional test cases and test files then applied the original fix.
commit message from the original change:
Updated the EVTX FileHeader class to correctly check if there are more chunks in the file. Previously this would not process the last chunk.
Updated the EVTX ChunkHeader class to correctly check if there are additional records in the chunk. Previously this would only process the first record of each chunk. It was using the fileLastRecordNumber where it should have been using the logLastRecordNumber value.
Updated the EVTX unit tests to have the correct expected number of events and use the logLastRecordNumber.
refactoring duplicated code and magic numbers
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2489
This closes#3379
- Allowing user or group existence enforcement to be parameterized.
- Fixing error handling when loading user groups which may have resulted in stack trace leaking.
This closes#3377.
Signed-off-by: Kevin Doran <kdoran@apache.org>
Added an unit test representing the fixed issue.
And updated existing testDefragment test to illustrate
the remaining FlowFiles those did not meet the threshold.
- Make enriched fields nullable at LookupRecord.
- Removed unnecessary AvroConversionOptions and reader schema creation,
because ResultSetRecordSet can generate NiFi Record Schema from RS
directly. No Avro schema is needed to do that.
* Documentation : ConsumerKafka_2_0 - updated configuration details related to SASL_PLAINTEXT - SCRAM
* Updated rest of the related documents with SCRAM configurations and few NOTEs
* incorporated review comments
* incorporated review comments
- Added snap alignment for nf-label, label resize events, and nf-connection
- Shift key now disables snap alignment during the drag event.
- nf-connection load-balance-icon updated
This closes#3335
Processor for sending messages on Slack and optionally upload and attach
the FlowFile content (e.g. an image) to the message.
NIFI-6078: Remove username/icon properties.
NIFI-6078: Make Text property optional.
NIFI-6078: Documentation changes.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3339
Moved test classes into the proper folder.
Fixed outdated mocking and added assertions that actually tests the content.
This closes#3346.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-6056 Fixing cassandra bundle so nifi-mock is a test scoped and removing incorrect usage of ProviderCreation exception from nifi-framework-api
This closes#3322.
Signed-off-by: Mark Payne <markap14@hotmail.com>
NIFI-5660: Added lines to integration test to verify these header properties (added by Mike Moser)
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#3053
Suppressed deprecation warnings in TestGetHTTP, TestPostHTTP, TestGetHTTPGroovy, and TestPostHTTPGroovy. The annotation suppresses some warnings, but until Java 9, explicit FQCN imports are needed to avoid all warnings.
This closes#3312.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Updated exception message thrown when accessing the ListHDFS state to refer to the State Manager rather than the "Distributed Cache Service"
Removed usage and references to MockCacheClient from TestListHDFS
NIFI-6023: Minor update of the "Distributed Cache Service" description
This closes#3313.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5943 Added another unit test to verify list + map conversion to list of records. (Mike Thomsen)
This closes#3267
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-5947 Made the lookup key for ElasticSearchStringLookupService more unique to the service (was previously 'id'); change by Mike Thomsen
This closes#3268.
Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
NIFI-4975 Added changes requested in a code review.
NIFI-4975 Reverted some base Mongo changes.
NIFI-4975 Moved connection configuration to using Mongo client service.
NIFI-4975 Fixed a lot of style issues.
NIFI-4975 Removed an EL statement that was causing problems with the UI.
NIFI-4975 Added changes from code review.
NIFI-4975 Added additional details for FetchGridFS.
NIFI-4975 Added documentation for DeleteGridFS.
NIFI-4975 Added documentation for PutGridFS.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2546
resets worker if it doesn't work anymore for any reason. this will add "reconnect" capabilities. Will solve issues for following use cases:
- authentication changed after successful connection
- JNDI mapping changed and requires recaching.
- JMS server isn't available anymore or restarted.
improved controller reset on exception
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3261
NIFI-5903: Removed TODO comments that were done
NIFI-5903: Added support for working with MAP types to QueryRecord and associated RPATH functions
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3223
Use temporary values for port names when updated process groups as part of a Change Flow Version operation.
This avoids the potential for a name conflict between a ports during the update process.
Add a final step to the update process group logic to set the final name on all ports.
This closes#3301.
- Updates to make UpdateRecord and RecordPath automatically update Record schema when performing update and perform the updates on the first record in UpdateRecord before obtaining Writer Schema. This allows the Writer to to inherit the Schema of the updated Record instead of the Schema of the Record as it was when it was read.
- Updated JoltTransformRecord so that schema is inferred on the first transformed object before passing the schema to the Record Writer, so that if writer inherits schema from record, the schema that is inherited is the trans transformed schema
- Updated LookupRecord to allow for Record fields to be arbitrarily added
- Implemented ContentClaimInputStream
- Added controller service for caching schemas
- UpdatedQueryRecord to cache schemas automatically up to some number of schemas, which will significantly inprove throughput in many cases, especially with inferred schemas.
NIFI-5938: Updated AvroTypeUtil so that if creating an Avro Schema using a field name that is not valid for Avro, it creates a Schema that uses a different, valid field name and adds an alias for the given field name so that the fields still are looked up appropriately. Fixed a bug in finding the appropriate Avro field when aliases are used. Updated ContentClaimInputStream so that if mark() is called followed by multiple calls to reset(), that each reset() call is successful instead of failing after the first one (the JavaDoc for InputStream appears to indicate that the InputStream is free to do either and in fact the InputStream is even free to allow reset() to reset to the beginning of file if mark() is not even called, if it chooses to do so instead of requiring a call to mark()).
NIFI-5938: Added another unit test for AvroTypeUtil
NIFI-5938: If using inferred schema in CSV Reader, do not consider first record as a header line. Also addressed a bug in StandardConfigurationContext that was exposed by CSVReader, in which calling getProperty(PropertyDescriptor) did not properly lookup the canonical representation of the Property Descriptor from the component before attempting to get a default value
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3253
Refactored KerberosAction to return a result from execute()
Removed usage of ProcessContext.yield() from KerberosAction, which should instead be handled the component using the KerberosCredentialsService.
Updated SolrProcessor to yield a flowfile on error, rather than the KerberosAction invoking the yield.
NIFI-5985: Updated TestPutSolrContentStream.testUpdateWithKerberosAuth test case to match on PrivilegedExceptionAction instead of PrivilegedAction doAs arguments.
NIFI-5985: Moved kerberosUser logout after closing the datasource in the shutdown method.
NIFI-5985: Removed catching exceptions in DBCPConnectionPool.shutdown
Exception when closing the datasource is prioritized over an exception when logging out the kerberos principal
Added GroovyDBCPServiceTest tests to verify prioritizing datasource.close() exception over kerberosUser.logout() exception
This closes#3288.
Signed-off-by: Bryan Bende <bbende@apache.org>
Removed bad test.
Refactored filter creation method.
Ensure HSTS header is only applied if NiFi is secured with HTTPS
Small changes to header array list.
Fixed checkstyle errors.
This closes#3273.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
MQTT over websockets uses ws (and wss for secure websocket connections) schemes in the broker URI.
Added support for the same.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3289.
Unit test is implemented: testNoSplitterInString
NIFI-5974: Fixed Checkstyle violations
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3275
NIFI-5909 added content checks to the unit tests
NIFI-5937 use explicit long value for test dates/times (to not depend on the timezone of test executor)
NIFI-5937 tabs to spaces
Fixing checkstyle violations introduced by https://github.com/apache/nifi/pull/3249 PR)
NIFI-5937 adjusted property descriptions for consistency; limited EL scope to variable registry; added an appropriate validator along with its Maven dependency; moved format initialization to @OnScheduled
NIFI-5909 tabs to spaces
Signed-off-by: Ed <edward.berezitsky@gmail.com>
This closes#3227
NIFI-5937 added tests to verify that accented characters are preserved correctly
NIFI-5937 unfolding starred imports
NIFI-5937 unfolding starred imports (now with statics)
Signed-off-by: Ed <edward.berezitsky@gmail.com>
This closes#3250
- Ensuring exceptions are handled in the ldap user/group sync background thread.
- Adding additional logging around what users/groups were discovered.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3247.
Unit tests and functionality for tagging an object in S3.
Set FlowFile attributes directly from tags retrieved from S3.
Add guard clauses to ensure evaluated properties are not blank.
This closes#3239.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Clear the wait timestamp when transferring to failur or success
replace explicit attribute clear with function call, refactor and integrate into existing tests per review
This closes#3233.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-4892 - ValidateCSV: no doublequote escaping in invalid output
NIFI-5907 - unit test
This closes#2481.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixed proxy header support to use X-Forwarded-Host instead of X-ForwardedServer
- Added support for the context path header used by Traefik when proxying a service (X-Forwarded-Prefix)
- Added tests to ApplicationResourceTest for X-Forwarded-Context and X-Forwarded-Prefix
- Updated administration doc to include X-Forwarded-Prefix
- Added NIFI_WEB_PROXY_CONTEXT_PATH env var to dockerhub and dockermaven start.sh scripts
- Added documentation for NIFI_WEB_PROXY_CONTEXT_PATH to dockerhub README.md
- Updated ApplicationResource to handle a port specified in X-ProxyPort and X-Forwarded-Port headers
This closes#3129.
Signed-off-by: Kevin Doran <kdoran@apache.org>
- Found several instances of nifi-framework-api's ProviderException being thrown from processors. Changed those to IllegalStateException, as ProviderException is not an appropriate Exception in those cases, and extensions should not depend on nifi-framework-api.
- Performed some cleanup, moving Property Descriptors from Controller Service API's/specs into the implementations. Adding to the Service API results in bringing in nifi-utils to the nifi-standard-services-api-nar, which is a bad practice. The 'main' service api nar should not have a dependency on a util class.
NIFI-5859: Added javadocs. Fixed pom.xml that was left pointing to snapshot version of nar maven plugin
NIFI-5859: Addressing review feedback: adding component type, multiple additional details into separate file(s)
This closes#3192.
Signed-off-by: Bryan Bende <bbende@apache.org>
NIFI-5406 introduced the issue by trying to use the resetState variable for
different purposes. AbstractListProcessor should have had a different variable
to control whether to clear state for tracking entity strategy.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#3189.
NIFI-5833 Added unit test to demonstrate arbitrary decryption of sensitive values regardless of processor property sensitive status.
NIFI-5833 Updated GetTwitter documentation with note about 1.9.0+ marking Consumer Key and Access Token as sensitive.
Signed-off-by: joewitt <joewitt@apache.org>
NIFI-5788: Introduce batch size limit in PutDatabaseRecord processor
Renamed 'batch size' to 'Maximum Batch Size'.
Changed default value of max_batch_size to zero (INFINITE)
Fixed parameter validation.
Added unit tests
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3128
making the database session autocommit value a configurable property
adding custom validation to PutSQL processor so as to disallow 'supports transaction' and 'rollback on failure' to be true when the autocommit value has been set to true
fixing some style issues to conform to standards
This closes#3113.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-5752: Remove an unnecessary String.format
NIFI-5752: Remove an unnecessary block
This closes#3110.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
ListS3 used to update currentKeys within listing loop, that causes
duplicates. Because S3 returns object list in lexicographic order, if we
clear currentKeys during the loop, we cannot tell if the object has been
listed or not, in a case where newer object has a lexicographically
former name.
Signed-off-by: James Wing <jvwing@gmail.com>
This closes#3116, closes#2361.
- Set up NarAutoLoader to watch directory for new files
- Move NarAutoLoader to JettyServer since it will need access to ExtensionManager
- Created NarLoader to shared between NarAutoLoader and the framework
- Created nifi-framework-nar-loading-utils so we can use nifi-documentation to call DocGenerator
- Add additional bundles to overall map in NarClassLoaders as they are loaded
- Added handling of skipped NARs to include them in next iteration
- Added check of last modified timestamp on NARs
- Refactored JettyServer so we can load additional web contexts while the application is running
- Setting up unit tests
- Remove static use of ExtensionManager
- Adding unit tests for NarLoader
- Extracting interface for ExtensionManager and splitting discovery into it's own interface
This closes#3119.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Moved module appending (aka classpath in python) into init stage instead of running each time onTrigger.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#3117
NIFI-5761 Code review.
Remove startsWidth to check an exception. Added the dependency as provided.
NIFI-5761 Code review.
Remove provided.
NIFI-5761 Code review.
Improve logging.
This closes#3112.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
Rob Fellows
archon
Mark Payne
Koji Kawamura
korir
Troy Melhase
Yolanda M. Davis
Scott Aslan
Hsin-Ying Lee
Peter Turcsanyi
Andy I. Christianson
Matt Gilman
samhjelmfelt
Arpad Boda
Brandon
Bryan Bende
Alessandro D'Armiento
Mike Thomsen
SandishKumarHN
Evan Reynolds
Andy LoPresto
Tamas Palfy
Peter Wicks
Malthe Borch
Mike Thomsen
andrew
Jeff Storck
Joe Gresock
thenatog
Ferenc Szabó
Pierre Villard
JF Beauvais
Mark Owens
Matthew Burgess
mike
Phillip Grenier
Gary Teichrow
avseq1234
Kemix Koo
Travis Neeley
DimDroll
adyoun2
mans2singh
Shawn Weeks
Andres Garagiola
slyouts
Satwik
Aaron Leon
Otto Fowler
Michael Karpel
Endre Zoltan Kovacs
Jan Hentschel
Alex Savitsky
Mark Bean
Tomer Ghelber
simonl
Alex Aversa
Andrew Lim
Bryan Rosander
Sivaprasanna
Dustin Rodrigues
aeaversa
Rahul Patil
Lars Francke
Kourge
pushpavanthar
bdesert
dnsbtchr
ambah
Craig Davidson
Joe Witt
Ferenc Szabó
Sandish Kumar
Ryan Bower
Kevin Doran
Arpad Boda
Joe Percivall
Denes Arvay
Wietze
Corey Fritz
Aleksandr Salatich
Mike Moser
Patrice Freydiere
Key Miyauchi
Ed
Dorian Bugeja
Alex Goos
Sujesh Menon
a.durov
erichanson5
SavtechSolutions
Stephen Goodman
Ryan Whittington
gkkorir
Alexander Bukarev
luup2k
Kotaro Terada
Kumara M S Hemantha
Daniel Jimenez
yjhyjhyjh0
Colin Dean
Colin Dean
vadimar
Ed B
Arek Burdach
Vish Uma
Adam Lamar
Gardella Juan Pablo
juliansniff
rednikotin
Mohit Garg