- Added org.apache.nifi.processor.util.pattern package in nifi-processor-utils containing reusable functions to mix-in 'Rollback on Failure' capability.
- Created a process pattern classes, Put and PutGroup. It will be helpful to standardize Processor implementations.
- Applied Rollback on Failure to PutSQL, PutHiveQL, PutHiveStreaming and PutDatabaseRecord.
- Stop using AbstractProcessor for these processors, as it penalizes FlowFiles being processed when it rollback a process session. If FlowFiles are penalized, it will not be fetched again until penalization expires.
- Yield processor when a failure occurs and RollbackOnFailure is enabled. If we do not penalize nor yield, a failed FlowFile retries too frequently.
- When Rollback on Failure is enabled but processor is not transactional, discontinue when an error occurred after successful processes.
- Fixed existing issues on PutHiveStreaming:
- Output FlowFile Avro format was corrupted by concatenating multiple Avro files.
- Output FlowFile records had incorrect values because of reusing GenericRecord instance.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1658
NIFI-3704: Incorporated review comments
NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records
NIFI-3704: Added retry paths for transient SQL errors
NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException
NIFI-3704: Added better error handling and rollback
This closes#1677.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.
NIFI-3738 Added licenses to new unit tests.
This closes#1694.
Signed-off-by: Bryan Bende <bbende@apache.org>
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Addressed feedback from PR Review
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers
This closes#1652
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.
nifi-mock is modified to support FlowFile assertion using Prioritizer.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporated review comments, added displayNames.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
Incorporate review comments:
- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3414: Added EnforceOrder processor
When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1496
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Updated test to check selected column names
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1596
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate
- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.
Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up
- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.
Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services
- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.
Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework
- Finishing clean up following rebase.
Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests
- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.
Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle
- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.
Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled
- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.
Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory
- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.
Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times
Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.
Improving error messages when looking for bundles
Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.
Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.
Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0
Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster
NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.
Fixing checkstyles issue and adding back assume checks to distributed cache server test
Ensuring new component types are sorted correctly when shown initially.
This closes#1585.
- Refactored TestGetHTTPGroovy for componentization & customization of tests.
- Ignored some TestGetHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate GetHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Refactored TestPostHTTPGroovy for componentization & customization of tests.
- Ignored some TestPostHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate PostHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Restored one of the TestGetHTTPGroovy unit tests (GetHTTP vs. TLSv1/1.1/1.2) by connecting to https://nifi.apache.org which still supports all three protocol versions (and uses JVM cacerts as truststore).
- This closes#1601
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3481 Added support for Database Adapter Descriptions
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-3481 Updated Adapter Name
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1510
* Add test for failure not clear context map.
* Add handler to remove context map entry if ProcessException occurs during while exporting response.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1567.
- Support multiplle incoming FlowFiles to Wait processor, up to Wait
Buffer Count
- Added Releasable FlowFile Count, which controls how many FlowFiles can
be released when wait condition is met
- Added special meaning to Notify delta Zero(0) to clear a signal
counter back to zero
This closes#1554
Signed-off-by: Aldrin Piri <aldrin@apache.org>
Before this fix, files with the latest timestamp within a listing
iteration are always be held back one cycle no matter how old it is.
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
- Added Signal Counter Delta property
- Added Signal Buffer Count property
- Added processor property name and display name
- Changed IOException handling from routing it to failure to throw
RuntimeException, so that NiFi framework can yield the processor for a while and try again
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
This closes#1466.
use the FileNameFilter when not passing down explit jar paths
Filter out ^. files when reading lists of files from directories
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
- Support counters at Wait/Notify processors so that NiFi flow can be
configured to wait for N signals
- Extract Wait/Notify logics into WaitNotifyProtocol
- Added FragmentAttributes to manage commonly used fragment attributes
- Changed existing split processors to set 'fragment.identifier' and
'fragment.count', so that Wait can use those to wait for all splits
get processed
This closes#1420.
Signed-off-by: Bryan Bende <bbende@apache.org>
- Added unit test cases to simulate NiFi version update which fails without this fix.
- Added state object migration code, add file.0. prefix to state keys,
and add length from stored position.
This closes#1289
- Addressing comments from PR.
- Adding restricted tags to relevant components.
- Showing a restricted icon overlay on the processor node on the canvas. (+1 squashed commit)
Squashed commits:
[f487682] NIFI-3050:
- Introducing a Restricted annotation for components that require elevated privileges to use.
- Updating the new Processor, Controller Service, and Reporting Task dialogs to include these details and prevent unauthorized selection.
- Including the Restricted description in the generated component documentation.
- Updating processor access control integration test to verify restricted component creation.
- Updating the developer, user, and admin guide to include the restricted component policy.
This closes#1247.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2750 Tweaking Property Retrieval
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1001
- introduced org.apache.nifi.stream.io.util.TextLineDemarcator
- refactored SplitText to use org.apache.nifi.stream.io.util.TextLineDemarcator
- updated SplitText's capability discription to provide more clarity around splits with headers.
This closes#1147.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Fixed typos in error messages, renamed variables in test, and cleaned up unnecessary imports. (+1 squashed commit)
Squashed commits:
[e755cbd] NIFI-2919 improved GetFile to fail if target directory is inaccessible
ExecuteProcess ignores error stream when Redirect Error Stream is
false, this commit let it to be logged instead so that user can see it on
bulletin.
This closes#1143.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
Correction as suggested by users in JIRA.
+ adding a unit test to check attachments.
This closes: #1081
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2803: Updated unit tests for GenerateTableFetch
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#1052
* Corrected the DEFAULT_REGEX pattern.
* Added tests to isolate regex capture group problem and verify corrected functionality.
* Removed short circuit logic that masked configuration errors and created inconsistent processor behavior.
This closes#1007.
Added 'file.lastModifiedTime' attribute to ListFileTransfer, which is
the abstract class extended by ListSFTP.
String literal attribute names were replaced with static references to
attribute name constants in ListFile.
ListFileTransfer stores the 'file.lastModifiedTime' attribute in the
format specified in ListFile.FILE_MODIFY_DATE_ATTR_FORMAT
Updated WritesAttribute description for file last modify time attribute
to mirror the entry in ListFile
Signed-off-by: Joe Skora <jskora@gmail.com>
This closes#931.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2591
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2591
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2591 - Added Format option for binary data types. Updated unit tests.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#883
* Removed @Ignore annotation on class and unnecessary EOL translation of test data. Because ModifyBytes treat input as binary data, not text, line endings don't matter as long as they byte offsets are calculated correctly.
* Replace validator with .allowableValues.
This closes#890.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2582
Signed-off-by: Matt Burgess <mattyb149@apache.org>
NIFI-2582
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#877
With this commit, MonitorActivity can be configured as:
- Monitor activity per node individually
- Monitor cluster wide activity
- Send notification flow-file from all of nodes
- or only from a primary node
Changes:
- Added 'Monitoring Scope' property
- Added 'Reporting Node' property
- Falls back from cluster scope to node scope if necessary
- Persist the latest activity as Cluster state
- Examine cluster state on each node if necessary
- Only update the cluster state if current timestamp is later than existing
timestamp stored in Zookeeper
This closes#575
Signed-off-by: jpercivall <joepercivall@yahoo.com>
NIFI-2020 - updates to use lambdas/stream wherever possible and fix potential nullpointer issue.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#564
- Addressing access controls for the Controller resource.
- Addressing access controls for RAW site to site clients.
- Addressing access controls for downloading content (from provenance and queue).
- Addressing access controls for accessing queues.
- Addressing access controls for cluster endpoints.
- Addressing access controls for counter endpoints.
- Removing redundant authorization calls.
NIFI-2044:
- Requiring revision when creating components.
- Requiring component creation over POST requests.
NIFI-1901
- Continuing to restore access control tests.
- Converting access control tests to itegration tests.
- Restoring contrib check to travis build.
- This closes#567
Adds a "File Filter" property to the `UnpackContent` processor
to allow users to specify which files are eligible for extraction.
By default, all files will be extracted.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Refactor how Unpacker is initialized
Re-uses unpackers to avoid creating them each time a
flowfile is received. Moved the package formats into
an enum for clarity.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
Fix packaging format enum warning
The `AUTO_DETECT_FORMAT` enum for PackagingFormat
is not valid for initilization. When this value
is set, then we use the mime.type to determine
which packaging format to use.
We never pass `AUTO_DETECT_FORMAT` to the
`initUnpacker` method.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
This closes#248
Fixed IDE setting for import wildcarding on Groovy files. (+4 squashed commits)
Squashed commits:
[4c3b174] NIFI-1981 Lowered logging level of client auth setting on cluster connection receive.
[b50f473] NIFI-1981 Finished logic to suppress exception on missing client certificates when clientAuth is set to WANT.
Added unit tests for CertificateUtil methods.
[ace35a2] NIFI-1981 Added test scope dependency on BouncyCastle and BC PKIX modules for CertificateUtils tests.
[2c463d1] NIFI-1981 Added ClientAuth enum and CertificateUtil methods to extract this setting from an SSLSocket.
Added logic to compare X509Certificate DNs regardless of RDN element order.
Added logic to suppress peer certificate exceptions when client authentication is not required.
Removed duplicate dependency in pom.xml.
Fixed '$' escape logic (if not followed by number) to use Matcher.quoteReplacement(java.lang.String) to
suppress its special meaning so it could be treated as literal.
This closes#466.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
Koji Kawamura
Mark Payne
Matt Burgess
Andy LoPresto
Andre F de Miranda
joewitt
Yolanda M. Davis
Oleg Zhurakousky
Toivo Adams
Pierre Villard
Bryan Bende
Matt Gilman
Joe Percivall
ambud
Peddy
patricker
Joe Skora
Pere Urbon-Bayes
Joe Trite
Giovanni Lanzani
josephxsxn
Nick Carenza
Selim Namsi
Otto Fowler
Ryan Persaud
Giovanni Lanzani
Joe Gresock
Jeff Storck
Joe Skora
Joey Frazee
Mike Moser
Bryan Rosander
Davy De Waele
Wesley-Lawrence
Byunghwa Yun
Kathik Narayanan
Peter Wicks
James Wing
Bartosz Wozniak
Nicolas Dupont
Kirk Tarou
Joseph Percivall
Bryan Rosander
Joey Frazee
Koji Kawamura
Matt Brown
ricky
Adam Lamar
Joshi
Mark Bean
Thad Guidry
DomenicPuzio
Aldrin Piri