Commit Graph

446 Commits

Author SHA1 Message Date
Matt Burgess 4303e4742a NIFI-3704: Add PutDatabaseRecord processor
NIFI-3704: Incorporated review comments

NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records

NIFI-3704: Added retry paths for transient SQL errors

NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException

NIFI-3704: Added better error handling and rollback

This closes #1677.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-26 09:50:50 +09:00
Andy LoPresto ef5bac207e
NIFI-3738 Fixed NPE when ListenSyslog UDP datagram has zero length. Added default constructor to SyslogParser to allow map coercion for test. Added unit test.
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.

NIFI-3738 Added licenses to new unit tests.

This closes #1694.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-25 16:38:46 -04:00
Matt Burgess 9ed001869b NIFI-2323: Fixed Oracle timestamp handling in DB fetch processors. This closes #1664. 2017-04-25 09:59:43 -04:00
Mark Payne 50ea1083ec NIFI-3682: This closes #1682. Add Schema Access Strategy and Schema Write Strategy Record Readers and Writers; bug fixes.
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-24 17:02:45 -04:00
Mark Payne b93cf7bbdb NIFI-3658: Created processor for converting records between data formats with like schemas
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3658: Incorporated PR review feedback; added counter; clarified documentation

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1668
2017-04-19 16:12:14 -04:00
Andre F de Miranda cd260bab9f
NIFI-2747 - Introduce FuzzyHashContent processor
This closes #1619.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-04-18 10:27:15 -04:00
joewitt d08233b9fe NIFI-3694 updated latest logback,jetty,moved servlet-api and jetty-schemas to lib root as needed now
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1678
2017-04-18 07:41:21 -04:00
Yolanda M. Davis 5f73ba7f2c NIFI-3708 - removed typo from character escape regex
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1674
2017-04-14 22:13:09 -04:00
Oleg Zhurakousky 68c592ea43 NIFI-1280 added support for RecordSchema in SchemaRegistry
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Addressed feedback from PR Review

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers

This closes #1652
2017-04-11 19:29:35 -04:00
Mark Payne a88d3bfa3c NIFI-1280: Refactoring to make more generic so that other data types can be supported; created InputStreams to content on-demand so that multiple passes can be made over FlowFile content if required. Created new Controller Services for reading and writing specific data types
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-04-11 19:29:04 -04:00
Toivo Adams 4d5872a385 NIFI-1280 Create FilterCSVColumns Processor.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-04-11 19:29:04 -04:00
Koji Kawamura 9583ca99c1 NIFI-3414: Added EnforceOrder processor
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

Incorporated review comments, added displayNames.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

Incorporate review comments:

- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1496
2017-04-10 14:03:22 -04:00
Pierre Villard 02d38b55bf NIFI-2952 Fixed search value validation in ReplaceText
This closes: #1378

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 00:00:07 +10:00
Mark Payne a5d630672a NIFI-3635: This closes #1631. Avoid using a static member variable for the 'Grok' object. Code cleanup
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 13:22:46 -04:00
joewitt 6a64b3cd9c NIFI-3440 This closes #1638. fixing tests not written for windows to not run on windows 2017-03-31 01:59:35 -04:00
Mark Payne 83fa24b68f NIFI-3613: Fixed threading bug in JoltTransformJSON and adding caching of compiled Transforms
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1630
2017-03-29 10:45:34 -04:00
Pierre Villard ced6708d4b NIFI-3596 - added attributes to GenerateTableFetch processor
Signed-off-by: Matt Burgess <mattyb149@apache.org>

Updated test to check selected column names

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1596
2017-03-27 08:50:55 -04:00
Bryan Bende d90cf846b9 NIFI-3380 Bumping NAR plugin to 1.2.0-SNAPSHOT development to leverage changes from master, adding buildnumber-maven-plugin to nifi-nar-bundles to properly set build info in MANIFEST of NARs
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate

- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.

Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up

- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.

Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services

- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.

Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework

- Finishing clean up following rebase.

Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests

- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.

Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle

- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.

Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled

- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.

Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory

- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.

Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times

Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.

Improving error messages when looking for bundles

Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.

Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.

Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0

Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster

NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.

Fixing checkstyles issue and adding back assume checks to distributed cache server test

Ensuring new component types are sorted correctly when shown initially.

This closes #1585.
2017-03-24 11:06:44 -04:00
Andre F de Miranda a298f1a707 NIFI-3466 - Address non English locales in ParseCEF
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1614.
2017-03-23 09:04:16 +01:00
Matt Burgess d1ebddce98 NIFI-3627: Added removeByPattern() to DistributedMapCache interfaces
NIFI-3627: Updated unit tests that use MapCache interface(s)

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1609.
2017-03-22 19:22:08 +01:00
Matt Burgess 37a1e6d07f NIFI-3585: Add DatabaseAdapter impl for MS SQL 2008
This closes #1584

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2017-03-22 14:57:42 +00:00
Matt Burgess c4d0c0bbd1
NIFI-3626: Add support for DELETE in ConvertJSONToSQL
This closes #1605.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-21 17:20:27 -04:00
Andy LoPresto b3b65219a0
NIFI-3361:
- Refactored TestGetHTTPGroovy for componentization & customization of tests.
- Ignored some TestGetHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate GetHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Refactored TestPostHTTPGroovy for componentization & customization of tests.
- Ignored some TestPostHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate PostHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Restored one of the TestGetHTTPGroovy unit tests (GetHTTP vs. TLSv1/1.1/1.2) by connecting to https://nifi.apache.org which still supports all three protocol versions (and uses JVM cacerts as truststore).
- This closes #1601
2017-03-21 14:48:40 -04:00
Matt Gilman 8e61aa3c43
NIFI-3361:
- Upgrading to the latest Jetty version.
2017-03-21 13:44:26 -04:00
Joe Percivall 5daeba6e03 NIFI-3614 Adding "return" statement to HandleHttpResponse when statusCode is not a number
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1598.
2017-03-20 14:51:56 +01:00
ambud 9424836412 NIFI-3525 Add tablename attribute to QDT flowfiles
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1537.
2017-03-13 14:22:14 -07:00
Peddy ccd11816e4 NIFI-3034 This closes #1323. Update Counter and test case 2017-03-12 16:34:33 -04:00
patricker 18f05856ff NIFI-3481 - Add DB Adapter for MS SQL 2012+
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3481 Added support for Database Adapter Descriptions

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3481 Updated Adapter Name

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1510
2017-03-10 11:34:16 -05:00
Andre F de Miranda 5a6373b16c
NIFI-3442 Explicitly support EL on username, password, port.
Introduced basic test unit

This closes #1474.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-08 20:56:01 -08:00
Pierre Villard 37e9c5efc7 NIFI-3556 Fixed ValidateCsv schema parsing
Fixed schema parsing on comma boundaries.

This closes #1570.

Signed-off-by: James Wing <jvwing@gmail.com>
2017-03-08 14:57:33 -08:00
Joe Skora 90ff275bb9 NIFI-3517 If HandleHttpResponse cannot write response remove entry from HttpContextMap.
* Add test for failure not clear context map.
* Add handler to remove context map entry if ProcessException occurs during while exporting response.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1567.
2017-03-08 13:16:58 +01:00
Pierre Villard 70175816b3 NIFI-3541 NIFI-3545 - check style violations
Fix for checkstyle issues introduced by:
9e68f02f1f
000414e7ea

This closes #1569.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-03-07 22:17:23 +09:00
Koji Kawamura 000414e7ea
NIFI-3545: Release M FlowFilews once N signals arrive
- Support multiplle incoming FlowFiles to Wait processor, up to Wait
  Buffer Count
- Added Releasable FlowFile Count, which controls how many FlowFiles can
  be released when wait condition is met
- Added special meaning to Notify delta Zero(0) to clear a signal
  counter back to zero

  This closes #1554

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-06 09:29:44 -05:00
Mark Payne 7f5c0dfb54 NIFI-3553: If IOException is thrown when completing FTP transfer, do not update the FlowFile to point to the content that was streamed but instead leave FlowFile pointing to original version
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1562.
2017-03-05 15:27:48 +01:00
Pere Urbon-Bayes 33a702500c NIFI-3554 Fixing the jBCrypt dependency name from nifi-standard-processors/pom.xml file, so it uses the expected naming format from the maven repository
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-03-05 10:15:06 +11:00
Joe Trite f8cad0f8c9
NIFI-1750 Updated ScanAttribute dictionary file processor property description.
This closes #1543.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-03 17:36:40 -08:00
Giovanni Lanzani 5990db39ae
NIFI-3459 Fixed documentation for UnpackContent File Filter.
This closes #1558.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-02 16:19:11 -08:00
josephxsxn ac5a7960e8
NIFI-3547 Changing level from WARN to INFO for Wait Processor's expected behavior
This closes #1556.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-02 15:37:39 -05:00
patricker d689ac561e NIFI-3494: GenerateTableFetch Some JDBC Drivers Have Trouble With Name vs Label
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1517
2017-03-02 11:12:54 -05:00
patricker bf2f04fb5f NIFI-3540: QueryDatabaseTable Failing to Track MS SQL Max Values
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1547
2017-03-02 10:04:09 -05:00
Oleg Zhurakousky 8e17929d6a NIFI-2876 refactored demarcators into a common abstract class
This closes #1214.
2017-02-24 10:06:53 -05:00
Nick Carenza f9c4d3ce3a NIFI-3508: Added support for PATCH identical to that of PUT/POST
This closes #1522.

Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-02-21 16:23:23 +09:00
Oleg Zhurakousky ded18b94db NIFI-3255 removed dependency on session.merge from SplitText
NIFI-3255 addressed PR comments

NIFI-3255 fixed linkage for Split creation

This closes #1394
2017-02-17 12:31:21 -05:00
Koji Kawamura 095c04eda0 NIFI-3213: ListFile do not skip obviously old files
Before this fix, files with the latest timestamp within a listing
iteration are always be held back one cycle no matter how old it is.

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-18 00:56:46 +11:00
Koji Kawamura a90fa9c285 NIFI-3431: Support batch update in Notify processor
- Added Signal Counter Delta property
- Added Signal Buffer Count property
- Added processor property name and display name
- Changed IOException handling from routing it to failure to throw
  RuntimeException, so that NiFi framework can yield the processor for a while and try again

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1466.
2017-02-15 15:51:31 +01:00
Pierre Villard 71e2061b5d NIFI-2968 This closes #1377. Improved message when ExecuteSQL hits unsupported SQL types 2017-02-14 23:25:29 -05:00
Pierre Villard 6279fd4184 NIFI-3183 This closes #3183. Added command and arguments as FF attributes in ExecuteProcess processor 2017-02-14 22:08:09 -05:00
Koji Kawamura e46f4131f9 NIFI-3452: Add Wait processor Wait Mode property
Ensure back-pressure is active until downstream processing completes.

This closes #1490.
2017-02-14 19:15:27 +01:00
patricker 9cfc13423d NIFI-3455 Large row count paging
This closes #1499.
2017-02-14 19:09:12 +01:00
Selim Namsi 2ef7c15b5d NIFI-2565: add Grok parser
This closes #1108.

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-14 21:11:03 +11:00