- Only include Maximum Value columns in the type map.
- Squashed commits in the previous PR
- Rebased against the latest master
- Added stop method to GenerateTableFetch so that it refreshes the
column type map when it gets restarted
- Fixed whitespacing around if/for statement
- Updated expressionLanguageSupported value since it is not auto-merged
correctly
This closes#2618.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Fixing process group audit advice.
- Setting spring security user in background threads.
- Removing unnecessary overloaded methods.
This closes#2626.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Fixed dependency issue by providing a local JSON reader
Rebased + fixed conflict + updated versions in pom + EL scope
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2575
- take into account input requirement for documentation rendering
- Renamed variable registry scope and added comments
- Doc + change in mock framework to check scope + update of components + UI
- Processing properties and property descriptors in Controller Service referencing components unconditionally.
This closes#2602.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Ensuring appropriate response in checkAuthorization when user is null.
- Ensuring the user reference is passed down when applying variable changes.
This closes#2598.
Signed-off-by: Mark Payne <markap14@hotmail.com>
The bug is not reproducible in Jython (it handles modules with every script reload). But Groovy loads JARs and dirs with classes only on setup.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2584
NIFI-4325 Cleaned up how ElasticSearch client service builds SSLContext, added query attribute to flowfiles and other changes requested in a code review.
This closes#2113.
Signed-off-by: Joe Percivall <JPercivall@apache.org>
- Fixing required permission for PutParquet.
NIFI-5008:
- Ensuring all restricted components are tagged as such.
This closes#2583.
Signed-off-by: Bryan Bende <bbende@apache.org>
Refactor the two existing CSV classes to inherit from an abstract base class that parses the data
Add unit tests for calls to nextRecord that do not coerce types
This closes#2473.
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
If failed, use first method and throw error if not working.
Array.length can never be less then 0
This closes#2499.
Signed-off-by: Mark Payne <markap14@hotmail.com>
The feature allows users to convert from non-integral types to the correct underlying type. The
original behavior is maintained; however, now simple conversions take place automatically for some
logical types (date, time, and timestamp).
This closes#2526.
Signed-off-by: Derek Straka <derek@asterius.io>
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Fixing RPG port merging.
- Adding unit tests.
- Removing unecessary sorting that wasn't maintained while clustered.
This closes#2551.
Signed-off-by: Mark Payne <markap14@hotmail.com>
address review comments
NIFI-4809 - Added Record Writer property
added unit tests and additional details doc
review comments
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2430
Implemented review feedback. Refactored data model to make the API cleaner and delineate more along the lines of what permissions are required in order to see which details
Implementing review feedback
Removed sensitive information from the diagnostics reports
Fixed bug in merging logic for GCDiagnosticsSnapshots
This closes#2468
SelectHiveQL should emit FETCH instead of CONTENT_MODIFIED when it has
incoming connections.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#2543.
After first failure in customValidate, stop printing logs until any property is changed
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2537
- New processor for scanning HBase records based on verious params like range of rowkeys, range of timestamps. Supports result limit and reverse scan.
- Adds Atlas Support for ScanHBase processor
- Fixed not recent version of FF
- Formatting and Style changes
- Single line to multiline if-then statements
- Removed HTML formatting that is not used for doc generation
- Fixed issue with limitRows
- Fixed issue with filter expression
- Refactored "processRows"
- Fixed possible NPE for bulkSize var
- Changed provenance to "receive" to indicate new data from external source.
- Updated min/max timestamp custom validation
- JSON array support
- Removed in-memory caching for records. Now records are being written directly to FF
- Removed unfinished flowfile from session, transfered original to "failure". Test cases update as well
This closes#2478.
Signed-off-by: Bryan Bende <bbende@apache.org>
* Added low water mark for purgeOldEvents() to more quickly purge after a large spike in events.
* Adjusted rollover high water mark to avoid overrunning "nifi.provenance.repository.max.storage.size".
* Adjusted looping logic in mergeJournals() to use ".firstKey()" instead of ".entrySet().iterator().next()" to avoid unnecessary object creation.
Signed-off-by: Mike Moser <mosermw@apache.org>
This closes#1240
- Addressing memory leak from lingering authorization results that did not represent actual access attempts. This closes#2511.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Initial set of components marked with the HighResourceUsageScenario annotation.
- Added customized descriptions to SystemResourceConsideration annotations for MergeContent, SplitContent, SplitJson, SplitText, and SplitXml.
This closes#2475.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Added configurable character set encoding for JMS TextMessages
- Improved PublishJMS/ConsumeJMS documentation
- Validate character set in property validator instead of OnScheduled
...Search results
* Separated the search functionality.
* Added a unit test.
* Added the PG info to UI (a mere draft).
* Introduce the nearest versioned group
* Removed the top level group results in favour of the nearest versioned group.
* This closes#2364
NIFI-4772: Introduced a new thread-pool with 2 threads that will be used for monitoring lifecycle task. This means that if all threads in the timer-driven thead pool are blocked by processors that don't complete their @OnScheduled methods, we have a separate thread pool that at least gives us a chance of interrupting those threads
NIFI-4772: Remove unused import
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2403
HBase_1_1_2_ClientService performance dropped due to accessing HBase
admin upon every FlowFile transfer. Fixed that by getting master address
only once when the service is enabled and a connection is established.
This closes#2464
- Everywhere that we ignore adding remote ports we should ignore removing remote ports as well in flow diffs
This closes#2462.
Signed-off-by: Bryan Bende <bbende@apache.org>
- When Jetty initializes fails, performing a shutdown sequence to ensure all allocated resources are released.
This closes#2455.
Signed-off-by: Bryan Bende <bbende@apache.org>
Before this fix, MergeContent only processed the first bin even if there
were multiple bins.
There were two unit tests marked with Ignore those had been
failing because of this.
This closes#2444.
Signed-off-by: Mark Payne <markap14@hotmail.com>
- Applied the same scale adjustment not only to BigDecimal inputs, but
also to Double values.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2450
Adds stricter checks in AuthorizerFactoryBean for unique ids within
a given type of provider and requires unique providers in composite
and composite-configurable user group providers. Failed validation
checks cause startup to fail. Adds test cases for these new rules.
This closes#2419.
Signed-off-by: Bryan Bende <bbende@apache.org>
Added HostHeaderHandler regression unit tests before adding new functionality.
Added logic for parsing nifi.web.proxy.host property.
Added default property in nifi.properties and pom.xml.
Added logic for IPv6 and custom default hostnames.
Improved error messaging.
Added HostHeaderHandler unit tests.
Disabled HostHeaderSanitizationCustomizer in HTTP mode.
Fixed HTML escaping in error message.
Improved error message.
Added failing unit test for parsing custom hostnames.
Fixed custom hostname parsing.
Fixed unit tests.
Added TODO for IPv6 custom hostname parsing and unit test.
Added IPv6 custom hostname parsing and unit tests.
Fixed checkstyle issues.
Removed empty element in host list when no value defined.
Improved error message formatting.
Added unit tests.
Removed HostHeaderSanitizationCustomizer.
Removed InvalidPropertiesFormatException from NiFiProperties.
Removed InvalidPropertiesFormatException from HostHeaderHandler.
This closes#2415
using per-record flowfiles
NIFI-4428: Added initial L&N, fixed dependency hierarchy
NIFI-4428: Exposed additional ZK properties
NIFI-4428: More work to reduce dependencies in Druid artifacts
NIFI-4428: Removed inaccessible attribute
NIFI-4428: Fixed typo in property description
NIFI-4428: Updates to Druid NAR L&N
NIFI-4428: Updated LICENSE to refer to ICU License for icu4j
This commit add back the existing capability for those Processors to use
incoming FlowFile attributes to compute account name and account key,
which had been removed by NIFI-4004.
Also, the same capability is added for SAS token.
This closes#2400.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Only showing the import link in the new Process Group dialog when initiating from the toolbar.
This closes#2395.
Signed-off-by: Mark Payne <markap14@hotmail.com>
Adds support for NiFI Registry config files to the encrypt-config tool
in NiFi Toolkit.
Also adds decryption capability to encrypt-config tool.
This closes#2376.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
NIFI-4436: Fixed a bug that caused a flow not to successfully change version if a connection is added to an existing component and that component is running at time of version change
NIFI-4436: Fixed bug with ordering of controller services being enabled and disabled
NIFI-4436: Fixed bug that prevented local input and output ports from being stopped and started as needed
NIFI-4436: Fixed bugs around referencing controller services that are at a higher level than the versioned flow
NIFI-4436: Ensure that we clear components from FlowController's cache when removed and that they are added to cache when created.
NIFI-4436: Fixed error message coming back if component is invalid when trying to be restarted/re-enabled
NIFI-4436: Addressed issue with children of a removed process group not being considered 'affected components' and as a result not being stopped/disabled/restarted/re-enabled
This closes#2219.
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
- Minor tweak to when the max height for the options list is calculated per PR comments.
- Resolving logic issue in two phase commit when updating variable registry.
- Fixing variable visibility.
- Updating buckets permissions based on new model.
- Adding check to ensure that flow name is non null before checking the length.
- Adding versioned flow state to the Process Group tab in the Summary table.
- Fixing issue with navigating to Controller Services from the local changes dialog.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Always showing Process Group state to complement the aggregation counts.
- Adding the Process Group state to the top status bar.
- Code clean up.
- Improved error handling.
- Minor UX improvements.
- Adding message to indicate that variables do not support sensitive values.
- Preventing a user from changing the flow version to the current version.
- Only presenting buckets a user has appropriate permissions to.
- Adding basic auditing to the version control actions.
- Updating front end to use version control state/status.
- Fixing copy/paste issue during revert local changes.
- Code clean up in the breadcrumbs.
- Update VersionsResource authorization and two phase commit object usage.
- Addressing miscellaneous minor UX issues.
- Updating comments UX for all components.
- Updating the styling of PG and RPG to be more consistent.
- Adding the icons for nested versioned process groups.
- Calculating the number/states of nested versioned process groups.
- Fixing default border radius.
- Code clean up.
- Ensuring component visibility is updated after updating/reverting.
- Fixing sort on component name in local changes dialog
NIFI-4526:
- Added front end controls for updating RPG target URL.
- Clearing bucket/flow/versions when changing the selected registry/bucket.
- Using the versioned flow to get the group name when importing.
- Adding menu items for viewing local changes.
- Showing local changes during revert request.
- Adding the version number to the start version control, commit, and change version dialog.
- Showing a loading item in the combo's while querying for the registries, buckets, and flows.
- Adding tooltips to display version control information on the canvas.
- Adding progress bar dialogs for changing version and reverting local changes.
- Updating canvas and breadcrumb according to the version control state.
- Updating to use registry name, bucket name, and flow name where appropriate.
- Added the import dialog for importing a versioned flow into a new process group.
- Added the change version dialog for upgrading/downgrading a versioned flow.
- Adding support to save a version of a flow based on a selected Process Group.
- Adding support for revert changes back to the most recent version.
- Adding support to disconnect from version control.
- Moving the version control information out of the entity objects and into the dto's.
- Fixing checkstyle issues.
NIFI-4502:
- Updating the UI to allow for the user to register registry clients.
- Updating the version control menu item names.
- Initial checkpoint: able ot start version control and detect changes, in standalone mode, still 'crude' implementation
- Checkpoint: Can place flow under version control and can determine if modified
- Checkpoint: Change version working in some cases. Does not work if processor removed because COMPONENT_REMOVED type has ComponentA whose ID is the VersionedComponentID but we are trying to call ProcessorDAO.get() with this ID
- Checkpoint: Able to change flow from Version 1 to Version 2 and back. Not yet tested with controller services. Have not tried changing/removing connections. Not cluster-friendly yet. All inline, not in background. Have not taken into account ports, funnels, remote ports, etc. Have not tested with Labels yet
- Checkpoint after implementing ClusterReplicationComponentLifecycle instead of JerseyClientComponentLifecycle
- Checkpoint: Updated to allow starting version control and updating version in clustered mode
- Checkpoint: Updated versioning endpoint so that when version of a flow is updated, the bundle information is populated and the snapshot is replicated to the cluster.
- Checkpoint: Implemented endpoint for reverting to previously sync'ed version of a flow and updated version control endpoint so that Process Group can be pushed as a new version to existing flow instead of only creating a new flow
- Checkpoint: Updated so that if a Process Group is under Version Control and it has a child Process Group, which is also under Version Control, we can handle that gracefully. Not yet tested because it depends on updates to the nifi-registry module, which can't be compiled due to maven dependency conflicts
Signed-off-by: Matthew Burgess <mattyb149@apache.org>
This closes#2307
Replaced star imports, removed unused import
Added explanation for invalid Expression
Removes user existence check from FileUserGroupProvider when
group is created or updated. Replaces it with check in the
Authorizer Decorator class created by Authorizer Factory, so
that all providers are used.
Also fixes bug when searching for group membership by user
that returns results across all providers.
The reporting task used to hold a single AtlasClientV2 instance
throughout its runtime starting from being started until being stopped.
If it is configured to use Kerberos authentication for Atlas REST API, after
a published DelegationToken expires (10 hours by default), the reporting
task will not be able to recover from 401 Unauthorized state.
In order to avoid stucking in such situation, this commit changes the
way ReportLineageToAtlas uses AtlasClientV2 instance to create an
instance per onTrigger execution. It also addresses Kerberos ticket
expiration.
This approach incurs some overheads by initiating the client each time,
however, it should be insignificant from an overall processing time
perspective including analyzing NiFi flow and Provenance records.
Before this fix, PublishKafka (0.9) and PublishKafka_0_10 fail with empty incoming FlowFiles due to 'transfer relationship not specified' error.
Because the internal 'publish' method is not called as StreamDemarcator does not emit any token regardless whether demarcator is set or not.
As for PublishKafka_0_11 and PublishKafka_1_0, empty FlowFiles are transferred to 'success' relationship, however no Kafka message is sent to Kafka.
Since Kafka allows 0 byte body empty messages, NiFi should be able to send it, too.
This commit changes above current situation to the followings, with all PublishKafka_* processors:
- If demarcator is not set, then publish incoming FlowFile content as it is. This enables sending an empty Kafka message.
- If demarcator is set, send each token as a separate message.
Even if no token is found (empty incoming FlowFile), transfer the FlowFile to 'success'.
This closes#2362.
Signed-off-by: Mark Payne <markap14@hotmail.com>
JohannesDaniel
Joe Percivall
patricker
Matt Gilman
Pierre Villard
mans2singh
Matthew Burgess
joewitt
Vi On
Mike Thomsen
Andrew Lim
Mark Payne
Marco Gaido
Mike Moser
Koji Kawamura
Mark Bean
Bryan Bende
Arun A K
Scott Aslan
Ed
zenfenan
Otto Fowler
Derek Straka
sbouchex
Andy LoPresto
Internet
Julian Gimbel
Jorge Machado
Deon Huang
m-hogue
Joe Skora
Kevin Doran
gardellajuanpablo
Takanobu Asanuma
Jeff Storck
James Wing
yuri1969
jlwt90
dorian.bugeja
Peter G. Horvath
Joey Frazee
Senthilannaswamy
vvaks
Patrice Freydiere
Andy LoPresto
Kevin Doran
Andre F de Miranda
Aldrin Piri