Commit Graph

1972 Commits

Author SHA1 Message Date
Joe Gresock ea6320d621
NIFI-3674: Implementing SiteToSiteStatusReportingTask
Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-01 09:55:10 -04:00
Pierre Villard 7df5c2dc89 NIFI-3607 Allow multi files mode with fixed names
This closes: #1608

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-05-01 09:31:24 +10:00
Bryan Rosander 0054a9e35f
NIFI-3755 - Restoring Hive exception handling behavior
This closes #1711.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-28 11:45:43 -04:00
Matt Burgess aa4efb43ca NIFI-3749: Added database filtering to DDL events in CaptureChangeMySQL
This closes #1708.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-28 13:16:01 +09:00
Andrew Lim 2664ea093d NIFI-3756 This closes #1715. Update ConvertAvroToOrc documentation to note that processor does not support unions of nested data
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-28 00:02:03 -04:00
Matt Burgess d61f519326 NIFI-3742: Fixed handling of nulls in DELETE for PutDatabaseRecord
This closes #1709.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-28 11:34:43 +09:00
joewitt 8651d79778 NIFI-3757 grabs latest kafka client 0.10.2.1
This closes #1714.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-28 11:16:35 +09:00
Mark Payne 726d15d1f7 NIFI-3717 This closes #1707. Added HortonworksSchemaRegistry controller service for schema lookup
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-27 14:56:52 -04:00
Koji Kawamura d9acdb54be NIFI-3415: Add Rollback on Failure.
- Added org.apache.nifi.processor.util.pattern package in nifi-processor-utils containing reusable functions to mix-in 'Rollback on Failure' capability.
- Created a process pattern classes, Put and PutGroup. It will be helpful to standardize Processor implementations.
- Applied Rollback on Failure to PutSQL, PutHiveQL, PutHiveStreaming and PutDatabaseRecord.
- Stop using AbstractProcessor for these processors, as it penalizes FlowFiles being processed when it rollback a process session. If FlowFiles are penalized, it will not be fetched again until penalization expires.
- Yield processor when a failure occurs and RollbackOnFailure is enabled. If we do not penalize nor yield, a failed FlowFile retries too frequently.
- When Rollback on Failure is enabled but processor is not transactional, discontinue when an error occurred after successful processes.
- Fixed existing issues on PutHiveStreaming:
  - Output FlowFile Avro format was corrupted by concatenating multiple Avro files.
  - Output FlowFile records had incorrect values because of reusing GenericRecord instance.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1658
2017-04-27 13:44:56 -04:00
Mark Payne a1bffbcc87 NIFI-3659: This closes #1707. Added Processor to Split a FlowFile consisting of Record-oriented data into multiple FlowFiles, each containing a subset of the original FlowFile's records
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-27 12:51:01 -04:00
Yolanda M. Davis 55b8c7ddad
NIFI-3695
- added proxy dn flag to support providing authorized username for secured environments
- addressed pr comments including fix to ensure proxy info added when getting cluster info, showing cleaner error messaging  and improving help text. Also fixed potential issue with versioning comparison (mismatched lengths)
- Printing response body when requests fails.
- This closes #1697
2017-04-27 12:42:12 -04:00
Koji Kawamura a41a2a9b1a
NIFI-1202: Site-to-Site batch settings.
- Added batchCount, batchSize, batchDuration to limit flow files to be
  included in a single Site-to-Site transaction.
- Added batch throttling logic when StandardRemoteGroupPort transfers
  flow files to a remote input port using the batch limit configurations,
  so that users can limit batch not only for pulling data, but also pushing data.
- Added destination list shuffle to provide better load distribution.
  Previously, the load distribution algorithm produced the same host consecutively.
- Added new batch settings to FlowConfiguration.xsd.
- Added new batch settings to Flow Fingerprint.
- Added new batch settings to Audit.
- Sort ports by name at 'Remote Process Group Ports' dialog.
- Show 'No value set' when a batch configuration is not set
- Updated batch settings tooltip to clearly explain how it works the configuration works differently for input and output ports.
- Updated DTO by separating batch settings to BatchSettingsDTO to indicate count, size and duration are a set of configurations.
- This closes #1306
2017-04-27 10:35:07 -04:00
Matt Burgess cf0e8bb1d3 NIFI-3741: Improved connection error message for CaptureChangeMySQL
This closes #1704.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-27 10:40:35 +09:00
Matt Burgess 097548da9d NIFI-3745: Fixed Table caching / primary key logic in PutDatabaseRecord
This closes #1700.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-27 08:43:13 +09:00
Matt Burgess d66eac2ea1 NIFI-3743: Include RENAME TABLE events in CaptureChangeMySQL
This closes #1701.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-27 08:42:01 +09:00
Matt Burgess da0454d80f NIFI-3746: Fixed DDL event transfer when outside a transaction in CaptureChangeMySQL
This closes #1702.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-27 08:35:34 +09:00
Matt Burgess 24bb8cf95d NIFI-3735: Replaced 'Schema Change Event' references to 'DDL Event'
This closes #1703.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-27 08:26:33 +09:00
Bryan Rosander 97461657b1 NIFI-3744 - PutHiveStreaming cleanup null fixes
This closes #1698.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
2017-04-26 22:54:49 +02:00
Andre F de Miranda 5cacc52cfc NIFI-2661 - Abstract the GeoEnrichIP processor NIFI-2661 - Add accuracy radius to GeoEnrichIP NIFI-2661 - Introduce ISPEnrichIP Processor
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-2661 - Addresses PR comments

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-2661 - Addresses PR comments

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-2661: Fixed typos and reversed logging params

This closes #1650
2017-04-26 09:10:48 -04:00
Matt Burgess 4303e4742a NIFI-3704: Add PutDatabaseRecord processor
NIFI-3704: Incorporated review comments

NIFI-3704: Updated PutDatabaseRecord documentation, including limitations on Update records

NIFI-3704: Added retry paths for transient SQL errors

NIFI-3704: Updated to use renamed RecordReaderFactory and catch SchemaNotFoundException

NIFI-3704: Added better error handling and rollback

This closes #1677.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-26 09:50:50 +09:00
Andy LoPresto ef5bac207e
NIFI-3738 Fixed NPE when ListenSyslog UDP datagram has zero length. Added default constructor to SyslogParser to allow map coercion for test. Added unit test.
NIFI-3738 Fixed NPE when ParseSyslog UDP datagram has zero length.
Added unit test.

NIFI-3738 Added licenses to new unit tests.

This closes #1694.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-25 16:38:46 -04:00
Matt Burgess 49a62448ce
NIFI-3734: Add ScriptedReader and ScriptedRecordSetWriter
This closes #1691.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-04-25 10:57:21 -07:00
Bryan Rosander 879fe3efd6 NIFI-3737 - Ran optipng on all png files 2017-04-25 12:28:49 -04:00
Matt Burgess 26ca40c0d9 NIFI-3728: Detect 'truncate table' event, allow exclusion of schema change events in CaptureChangeMySQL
NIFI-3728: Changed 'Include Schema Change Events' to 'Include DDL Events', updated tests

This closes #1690.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-25 23:37:09 +09:00
Matt Burgess 9ed001869b NIFI-2323: Fixed Oracle timestamp handling in DB fetch processors. This closes #1664. 2017-04-25 09:59:43 -04:00
Anders Breindahl 995c7ce2fa
NIFI-2656: replace -k [password] with -K [passwordfile].
This approaches a proper solution on how to hand over the key from
RunNiFi to NiFi. Insofar the password file is pruned as part of the
startup, NiFi processors can't read it.

See also: NIFI-3045.

This closes #1302.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-04-25 00:32:58 -07:00
Yolanda M. Davis c0f0462e8b
NIFI-3695 - created the nifi admin toolkit which includes shell scripts and classes to support notification and basic node management in standalone and clustered nifi.
This closes #1669.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-04-24 23:08:37 -07:00
Matt Burgess ba2bdf8586 NIFI-3730: Allow exclusion of BEGIN/COMMIT event output in CaptureChangeMySQL
This closes #1689.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-25 14:40:24 +09:00
Mark Payne 50ea1083ec NIFI-3682: This closes #1682. Add Schema Access Strategy and Schema Write Strategy Record Readers and Writers; bug fixes.
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-24 17:02:45 -04:00
Scott Aslan 57ccf97c58
[NIFI-3035] disable deep linking when URL .searchParams and URL .search properties are unsupported by browser. This closes #1687 2017-04-21 14:37:27 -04:00
Matt Gilman 0b95ccb90a NIFI-3380:
- Fixing opening of Usage from the Controller Service and Reporting Task table.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1684
2017-04-21 09:48:24 -04:00
Scott Aslan 2f928490e3
[NIFI-3360] inject nf.ControllerServices into nf.ControllerService. This closes #1685 2017-04-21 08:24:14 -04:00
Jeremy Dyer 769e874677 Support for websocket multiplexing to all existing websocket
connections (Think chat to all clients instead of individual person).
The core change was a change in WebSocketMessageRouter.java where if a
sessionId is not present the message is sent to all connected clients.
So the key is leaving the sessionId to empty or null to send to all
clients. If the sessionId is specified the message will be sent just to
that session specified.

This closes #1649.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-04-21 08:21:56 +09:00
ShellyLC 816034bd01 NIFI-3543 This closes #1548. Added support for using EL when defining brokerURL in jms-cf-service
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-19 22:50:48 -07:00
Koji Kawamura 8ce2a1b3a7 NIFI-3657 This closes #1634. Fix HTTP S2S to use local address.
- Fixed SiteInfoProvider and HttpClient to use specified local address with its SiteToSiteRestApiClient
- Removed setupRequestConfig method call from connection and read timeout setter methods at SiteToSiteRestApiClient, because it created config object before local address was set
- Null clear StandardRemoteProcessGroup localAddress when user clears Local Network Interface

Signed-off-by: joewitt <joewitt@apache.org>
2017-04-19 22:19:41 -07:00
Koji Kawamura 0ddb90cd2d NIFI-3166 This closes #1324. Fix SocketRemoteSiteListener NPE.
- Refactored ServerProtocol.sendPeerList method signature to clarify the
  meaning of arguments, and avoid null pointer exception when converting null Integer to int.
- Refactored SocketRemoteSiteListener handleRequest method to make it
  more unit test friendly.
- Added more unit tests.
2017-04-19 22:18:43 -07:00
Mark Payne 0207f21ce4 NIFI-3273: Added nifi-toolkit-flowfile-repo that contains a simple Java class that is capable of recovering a FlowFile Repository manually in the case of an operating system crash that results in trailing 0's being dumped into the edit logs. Also refactored flowfile repo into some independent modules so that it additional capabilities can be added in the future to examine the flowfile repo 2017-04-19 21:44:23 -07:00
Matt Burgess 141334c3c9 NIFI-3718: Fixed TestAvroRecordReader to handle timezone differences
This closes #1683
2017-04-19 17:04:45 -04:00
Mark Payne b93cf7bbdb NIFI-3658: Created processor for converting records between data formats with like schemas
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3658: Incorporated PR review feedback; added counter; clarified documentation

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1668
2017-04-19 16:12:14 -04:00
Matt Gilman 7e7f9a5deb NIFI-3705:
- Using consistent logic when verifying connection creation. Removed some unecessary checks as verification has been changed to run in cluster and standalone mode.

This closes #1672.
2017-04-19 15:49:11 -04:00
ijokarumawak 220a870a1e NIFI-2697: Fix opening Remote Ports dialog.
If there is an unconnected remote port, right clicking a Remote Process Group
then select 'Remote Ports' causes a JS error saying 'Uncaught TypeError'.
Because there are few code paths those don't return the created
transmissionSwitch value from createPortOption function in
nf-remote-process-group-ports.js.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1679
2017-04-18 16:08:10 -04:00
joewitt 189b4ad63d NIFI-3694 removed the total cap size value as per review feedback 2017-04-18 14:07:38 -04:00
Andre F de Miranda cd260bab9f
NIFI-2747 - Introduce FuzzyHashContent processor
This closes #1619.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-04-18 10:27:15 -04:00
Matt Gilman 7e1ecad738
NIFI-3706: - Removing non-existent resource when converting a legacy authorized users file.
This closes #1673.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-18 09:46:21 -04:00
joewitt d08233b9fe NIFI-3694 updated latest logback,jetty,moved servlet-api and jetty-schemas to lib root as needed now
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1678
2017-04-18 07:41:21 -04:00
Matt Burgess edb9c644dd NIFI-3413: Standardize 'write' event to 'insert'
This closes #1675.

Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-04-18 00:55:17 +09:00
Yolanda M. Davis 5f73ba7f2c NIFI-3708 - removed typo from character escape regex
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1674
2017-04-14 22:13:09 -04:00
Scott Aslan deed25656f
[NIFI-3035] use URLSearchParams .toString() to update URL. This closes #1659 2017-04-13 14:32:30 -04:00
Oleg Zhurakousky 68c592ea43 NIFI-1280 added support for RecordSchema in SchemaRegistry
Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Updated SimpleKeyValueSchemaRegistry to make use of new CHOICE RecordFieldType - Update Record Readers to use SchemaRegistry controller service. Moved SchemaRegistry api into its own maven module and added to standard-services-api so that we can properly add dependencies on it. Code cleanup and bug fixes

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Fixed checkstyle violations and license exclusions for RAT plugin

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Addressed feedback from PR Review

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-1280: Additional changes/doc to support QueryFlowFile and Record Readers/Writers

This closes #1652
2017-04-11 19:29:35 -04:00
Mark Payne a88d3bfa3c NIFI-1280: Refactoring to make more generic so that other data types can be supported; created InputStreams to content on-demand so that multiple passes can be made over FlowFile content if required. Created new Controller Services for reading and writing specific data types
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-04-11 19:29:04 -04:00
Toivo Adams 4d5872a385 NIFI-1280 Create FilterCSVColumns Processor.
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-04-11 19:29:04 -04:00
joewitt 52cf9a7953 NIFI-3690 Updated nifi-gcp-nar L&N, cat-x deps
Updated L&N in nifi-gcp-nar and nifi-assembly and fixed nifi-gcp-processors
to avoid jsr305 and json.org cat-x dependencies

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1665.
2017-04-11 15:56:25 -07:00
Joey Frazee 4a8398bb31 NIFI-3692 This closes #1663. Use hbc-core instead of hbc-twitter4j in social-media bundle to resolve org.json cat-x
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-11 14:25:04 -04:00
Koji Kawamura 7c19796eb2 NIFI-3561: Fix WS port forwarding
Fixed service instance lookup to use local port so that it works when
it is accessed through port forwarding.

This fixes: #1568

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 21:27:53 +10:00
Pierre Villard 776a00b0be NIFI-2860 Use different properties for JVM Heap Max and Min values
This closes: #1382

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 21:18:49 +10:00
Joe Percivall 5419891fe7 NIFI-3633 This closes #1610. Adding HttpNotificationService.
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-11 00:37:12 -04:00
Mark Payne 8a5398eba7 NIFI-3679: This closes #1655. Added Avro Content Viewer
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-11 00:00:09 -04:00
Koji Kawamura 9583ca99c1 NIFI-3414: Added EnforceOrder processor
Use it with FirstInFirstOutPrioritizer, it can enforce original ordering
of 'out-of-order' FlowFiles.

nifi-mock is modified to support FlowFile assertion using Prioritizer.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

Incorporated review comments, added displayNames.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

Incorporate review comments:

- Moved nifi-standard-prioritizers dependency to top level nifi/pom.xml.
- Changed default initial order from 1 to 0.
- Fixed typos.
- Use session.get(batchCount).

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3414: Added EnforceOrder processor

When a FlowFile is transferred to success, remove attributes previously set when it was transferred to wait or failure.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1496
2017-04-10 14:03:22 -04:00
Pierre Villard 02d38b55bf NIFI-2952 Fixed search value validation in ReplaceText
This closes: #1378

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-04-11 00:00:07 +10:00
Francois Prunier 61c799d88b NIFI-3204 This closes #1561. fix handling deleting a path with a wildcard when the processor is invoqued via an incoming flowfile
applied Joseph Witts patch from NIFI-3204 JIRA

Signed-off-by: joewitt <joewitt@apache.org>
2017-04-10 08:51:10 -04:00
Jack Pickett 4bfb905f37 NIFI-2714 This closes #1549. Added regex support to ConsumeKafka_0_10
Enabled the ability to specify wildcard topics as a regular expression
as supported in the Kafka client library.

Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 21:52:33 -04:00
Mark Payne 778ba3957e NIFI-3631: This closes #1613. Do not change Active Directory in IndexDirectoryManager when it becomes full. Instead, wait until it is committed and is removed by the onIndexCommitted method. This resolved a bug where the index can exceed the configured limit but not yet be committed and as a result would no longer be the active index. As a result, this bug causes the IndexWriter never to get closed/removed from the IndexManager, and so a memory leak is created
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 14:42:18 -04:00
Mark Payne a5d630672a NIFI-3635: This closes #1631. Avoid using a static member variable for the 'Grok' object. Code cleanup
Signed-off-by: joewitt <joewitt@apache.org>
2017-04-07 13:22:46 -04:00
Matt Gilman 35f4f48f37 NIFI-3683:
- Ensuring we use the appropriate action based on the previously displayed policy.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1657
2017-04-07 13:03:01 -04:00
Matt Gilman e8771e59a7
NIFI-2940: - Fixing broken integration test following changes introduced in NIFI-2940.
This closes #1653.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-07 10:42:11 -04:00
Mark Payne 292dd1d66b
NIFI-3678: Ensure that we catch EOFException when reading header information from WAL Partition files; previously, we caught EOFExceptions when reading a 'record' from the WAL but not when reading header info
NIFI-3678: If we have a transaction ID but then have no more data written to Partition file, we end up with a NPE. Added logic to avoid this and instead return null for the next record when this happens

This closes #1656.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-07 10:28:27 -04:00
Bryan Bende 6a75ab1740
Fixing checkstyle error in FlowSnippetDTO 2017-04-07 09:58:15 -04:00
Oleg Zhurakousky c7ecaba234
NIFI-3509 Fixed ID duplication
Fixed the possibility of duplicated MSB part of the component during template creation. This closes #1651
2017-04-06 16:04:27 -04:00
Pierre Villard 614fa6a6c4
NIFI-3528 Added support for keytab/principal to Kafka 0.10 processors
This closes #1606.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-06 15:09:49 -04:00
Bryan Bende 556f309df0
NIFI-3520 Refactoring instance class loading
- Fixing FlowController to use appropriate class loader when instantiating processor
- Updating ExtensionManager to leverage new flag in MANIFEST from NAR plugin
- Adding ReloadComponent interface and refactoring instance class loading to use it
- Fixing FetchHDFS issue with TDE by using ugi.doAs
- Refactoring nifi-nar-utils so that ExtensionManager only lives in nifi-framework
- Caching temp components found during service loader in ExtensionManager
- Updating authorizables, docs, and fingerprinting to use the cached components
- Introducing a flag on @RequiresInstanceClassLoading to indicate if ancestor resources should be cloned
- Updating developer guide regarding cloneAncestorResources flag
- This closes #1635
2017-04-06 13:51:08 -04:00
Matt Burgess 8f37ad4512 NIFI-3413: Add GetChangeDataCaptureMySQL processor
NIFI-3413: Incorporated review comments
NIFI-3413: Changed GetChangeDataCaptureMySQL to CaptureChangeMySQL, fixed some bugs
NIFI-3413: Refactored setup() for better error handling, more review comments incorporated
NIFI-3413: Refactored CDC into its own module(s), updated assembly and top-level POMs
NIFI-3413: Added RECEIVE prov event and Server ID property

Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-04-07 00:44:42 +09:00
Matt Gilman f5f54c468c
NIFI-3664: - Updating the unmarshalling to consider the current date when parsing the time.
This closes #1647.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-05 13:22:35 -04:00
Matt Gilman a565484ddd
NIFI-2940: - Allowing access to configuration actions through the Controller when the underlying component has been removed.
This closes #1648.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-04-04 17:40:09 -04:00
Koji Kawamura 5e62b4ae72 NIFI-3668: Fix purging expired replicate requests.
This closes #1646.

Newly created async response is added before checking map size nor
purging expired ones. If there are already 100 remaining requests,
the added request will not be executed nor removed.
2017-04-03 09:50:47 -04:00
Andre F de Miranda 84f1fb3959 NIFI-3222 - Allow ExecuteScript,InvokeScriptedProcessor and ScriptedReporting to use ExpressionLanguage to define the scripting engine modules path
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1640
2017-03-31 23:41:02 -04:00
joewitt 6a64b3cd9c NIFI-3440 This closes #1638. fixing tests not written for windows to not run on windows 2017-03-31 01:59:35 -04:00
Koji Kawamura fd92999daf
NIFI-3189: ConsumeKafka 0.9 and 0.10 with downstream backpressure
Currently, NiFi Kafka consumer processors have following issue.

While downstream connections are full, ConsumeKafka is not scheduled to run onTrigger.
It stopps executing poll to tell Kafka server that this client is alive.
Thus, after a while in that situation, Kafka server rebalances the client.
When downstream connections back to normal, although ConsumeKafka is scheduled again,
the client is no longer a part of a consumer group.

If this happens, Kafka client succeeds polling messages when ConsumeKafka processor resumes, but fails to commit offset.
Received messages are already committed into NiFi flow, but since consumer offset is not updated, those will be consumed again, duplicated.

In order to address above issue:

- For ConsumeKafka_0_10, use latest client library

    Above issue has been addressed by KIP-62.
    The latest Kafka consumer poll checks if the client instance is still valid, and rejoin the group if not, before consuming messages.

- For ConsumeKafka (0.9), added manual retention logic using pause/resume

    Kafka client 0.9 doesn't have background thread heartbeat, so similar machanism is added manually.
    Use Kafka pause/resume consumer API to tell Kafka server that the client stops consuming messages but is still alive.
    Another internal thread is used to perform paused poll periodically based on the time passed since the last onTrigger(poll) is executed.

This closes #1527.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-30 16:29:40 -04:00
Matt Gilman 282e1a7b1a NIFI-3421: This closes #1620.
- On contextDestroyed, referencing beans created during contextInitialized to prevent successive attempts to create a bean if that bean failed to be created initially.

Signed-off-by: joewitt <joewitt@apache.org>
2017-03-30 15:14:41 -04:00
Matt Gilman e203535163 NIFI-3272 This closes #1616.
- Updating the garbage collection column to use a icon with tooltip to relay all garbage collections.
2017-03-30 14:55:17 -04:00
Mark Payne 3aa1db6ee5 NIFI-3636: Lazily copy FlowFile Attributes Hash Map instead of doing so eagerly.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3257: Added additional logging regarding timing information when replicating requests across cluster in order to glean insight as to what is taking so long when replicating some requests

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3649: Buffer node responses when replicating HTTP Requests up to a maximum buffer size

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3636: Added unit test to ensure that flowfile attribute maps are copied when appropriate

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3636: Removed patch file that should not have been in commit

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1612
2017-03-30 09:53:39 -04:00
Andre F de Miranda a7bf683a0d
NIFI-79 - Introduces the ability to set Bulletin Level to NONE 2017-03-30 08:54:24 -04:00
kedarchitale 457df93dd6 NIFI-3147 Build processor to parse CCDA into attributes
This closes #1312

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2017-03-29 20:30:57 +00:00
Mark Payne 83fa24b68f NIFI-3613: Fixed threading bug in JoltTransformJSON and adding caching of compiled Transforms
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1630
2017-03-29 10:45:34 -04:00
Gahan Rakholia 168fc72bba NIFI-3637 Fixed ListHDFS Documentation
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1633
2017-03-29 10:31:56 -04:00
Oleg Zhurakousky c324c9e21c NIFI-3441 This closes #1629. added @Ignore to integration tests due to intermittent failures 2017-03-28 16:17:17 -04:00
James Wing f9b5bcf7bb NIFI-2907 Fix Flow Schema Validation Errors
This closes #1587.
2017-03-28 10:53:16 -04:00
Matt Gilman 4432958a65
NIFI-3643: - Addressing incorrect capitalization in VersionInfoDTO in NiFiVersion.
This closes #1626.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-27 16:44:21 -04:00
Matt Gilman f15e0d8e1e
NIFI-3380: - Addressing issues with paths in generated documentation.
This closes #1627.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-27 16:43:15 -04:00
Pierre Villard ced6708d4b NIFI-3596 - added attributes to GenerateTableFetch processor
Signed-off-by: Matt Burgess <mattyb149@apache.org>

Updated test to check selected column names

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1596
2017-03-27 08:50:55 -04:00
Andre F de Miranda a26689318d
NIFI-814 - Introduce hidden header with name of Processor to allow those arriving to site via direct reference to know the name of the processor
This closes #1624.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-26 11:41:15 -04:00
Bryan Bende f0a1c14f02
NIFI-3380 Fixing parent pom of nifi-standard-prioritizers to point to correct parent after moving under nifi-framework 2017-03-25 11:13:58 -04:00
Bryan Bende d90cf846b9 NIFI-3380 Bumping NAR plugin to 1.2.0-SNAPSHOT development to leverage changes from master, adding buildnumber-maven-plugin to nifi-nar-bundles to properly set build info in MANIFEST of NARs
- Refactoring NarDetails to include all info from MANIFEST
- Adding the concept of a Bundle and refactoring NarClassLoaders to pass Bundles to ExtensionManager
- Adding logic to fail start-up when multiple NARs with same coordinates exist, moving Bundle classes to framework API
- Refactoring bundle API to classes and creating BundleCoordinate
- Updating FlowController to use BundleCoordinate

- Updating the UI and DTO model to support showing bundle details that loaded an extension type.
- Adding bundle details for processor canvas node, processor dialogs, controller service dialogs, and reporting task dialogs.
- Updating the formating of the bundle coordinates.
- Addressing text overflow in the configuration/details dialog.
- Fixing self referencing functions.
- Updating extension UI mapping to incorporate bundle coordinates.
- Discovering custom UIs through the supplied bundles.
- Adding verification methods for creating extensions through the rest api.
- Only returning extensions that are common amongst all nodes.
- Rendering the ghost processors using a dotted border.
- Adding bundle details to the flow.xml.
- Loading NiFi build and version details from the framework NAR.
- Removing properties for build and version details.
- Wiring together front end and back end changes.
- Including bundle coordinates in the component data model.
- Wiring together component data model and flow.xml.
- Addressing issue when resolve unvesioned dependent NARs.

Updating unit tests to pass based on framework changes
- Fixing logging of extension types during start up

- Allowing the application to start if there is a compatible bundle found. - Reporting missing bundle when the a compatible bundle is not found. - Fixing table height in new component dialogs.

Fixing chechstyle error and increasing test timeout for TestStandardControllerServiceProvider
- Adding ability to change processor type at runtime
- Adding backend code to change type for controller services

- Cleaning up instance classloaders for temp components.
- Creating a dialog for changing the version of a component.
- Updating the formatting of the component type and bundle throughout.
- Updating the new component dialogs to support selecting source group.
- Cleaning up new component dialogs.
- Cleaning up documentation in the cluster node endpoint.

Adding missing include in nifi-web-ui pom compressor plugin
- Refactoring so ConfigurableComponent provides getLogger() and so the nodes provide the ConfigurableComponent
- Creating LoggableComponent to pass around the component, logger, and coordinate with in the framework

- Finishing clean up following rebase.

Calling lifecycle methods for add and remove when changing versions of a component
- Introducing verifyCanUpdateBundle(coordinate) to ConfiguredComponent, and adding unit tests

- Ensuring documentation is available for all components. Including those of the same type that are loaded from different bundles.

Adding lookup from ClassLoader to Bundle, adding fix for instance class loading to include all parent NARs, and adding additional unit tests for FlowController
- Adding validation to ensure referenced controller services implement the required API
- Fixing template instantiation to look up compatible bundle

- Requiring services/reporting tasks to be disabled/stopped.
- Only supporting a change version option when the item has multiple versions available.
- Limiting the possible new controller services to the applicable API version.
- Showing the implemented API versions for Controller Services.
- Updating the property descriptor tooltip to indicate the required service requirements.
- Introducing version based sorting in the new component dialog, change version dialog, and new controller service dialog.
- Addressing remainder of the issues from recent rebase.

Ensuring bundles have been added to the flow before proposing a flow, and incorporating bundle information into flow fingerprinting
- Refactoring the way missing bundles work to retain the desired bundle if available
- Fixing logger.isDebugEnabled to be logger.isTraceEnabled

- Auditing when user changes the bundle. - Ensuring bundle details are present in templates.

Moving standard prioritizers to framework NAR and refactoring ExtensionManager logic to handle cases where an extension is in a JAR directly in the lib directory

- Ensuring all nodes attempt to instantiate the same template instance when the available bundles may differ. - Fixing the auditing of copy/paste and template instantiation. - Running addtional verification methods when running standalone.

Refactoring controller service invocation handler to allow updating the node used by the invocation handler
- Ensuring the bundles in a proposed flow are compatible with the current instance when the current instance has no flow is going to accept the proposed flow
- Merging whether multiple versions of the component are available
- Setting NAR plugin back to current released version
- Cleaning up DocGenerator to not process multiple times

Addressing incorrect usage of nf.Common. - Using formatType in the new component type dialogs.

Improving error messages when looking for bundles

Addressing comments from PR. - Fixing references to global nf namespace. - Fixing injection of nfProcessGroupConfiguration in nfComponentVersion. - Fixing web api integration tests.

Not rendering unversioned in help documentation. - Ensuring the isExtentionMissing flag is correct after changing the component type.

Adding synchronization in node classes to ensure changing component can't occur when component is running, introducing MissingBundleException for better reporting when a node can't join cluster due to a missing bundle, and bumping NAR plugin to released version 1.2.0

Adding concept of missing components to fingerprinting to ensure nodes agree on missing components when joining a cluster

NIFI-3380: NIFI-3520: - Fixing hive nar dependency. - Marking DBCPService as provided. - Skipping services that require instance classloading and are cobundled with their service API. - Skipping components that require instance classloading and reference service APIs that are cobundled. - Addressing UI issues in the new component dialogs when re-opening with a filter applied.

Fixing checkstyles issue and adding back assume checks to distributed cache server test

Ensuring new component types are sorted correctly when shown initially.

This closes #1585.
2017-03-24 11:06:44 -04:00
Andre F de Miranda a298f1a707 NIFI-3466 - Address non English locales in ParseCEF
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1614.
2017-03-23 09:04:16 +01:00
Scott Aslan a90a770244
[NIFI-3035] add deep linking URL to canvas components and PGs. This closes #1559 2017-03-22 14:23:02 -04:00
Matt Burgess d1ebddce98 NIFI-3627: Added removeByPattern() to DistributedMapCache interfaces
NIFI-3627: Updated unit tests that use MapCache interface(s)

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1609.
2017-03-22 19:22:08 +01:00
Pierre Villard 0b73715562 NIFI-3565 - Fixed BigDecimal instantiation and unit tests for specific locales
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1571
2017-03-22 13:40:59 -04:00
Matt Burgess 37a1e6d07f NIFI-3585: Add DatabaseAdapter impl for MS SQL 2008
This closes #1584

Signed-off-by: Joey Frazee <jfrazee@apache.org>
2017-03-22 14:57:42 +00:00
Matt Burgess c4d0c0bbd1
NIFI-3626: Add support for DELETE in ConvertJSONToSQL
This closes #1605.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-21 17:20:27 -04:00
Scott Aslan 40acd4a6e1
[NIFI-2697] revert transmission switch toggle on failed update request. This closes #1600 2017-03-21 15:17:02 -04:00
Andy LoPresto b3b65219a0
NIFI-3361:
- Refactored TestGetHTTPGroovy for componentization & customization of tests.
- Ignored some TestGetHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate GetHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Refactored TestPostHTTPGroovy for componentization & customization of tests.
- Ignored some TestPostHTTPGroovy tests because they cannot connect to a Jetty server running TLSv1 or TLSv1.1 (the new version of Jetty only supports TLSv1.2+). The tests are still valuable to demonstrate PostHTTP's capacity to connect to various TLS protocols, and should not be removed unless necessary.
- Restored one of the TestGetHTTPGroovy unit tests (GetHTTP vs. TLSv1/1.1/1.2) by connecting to https://nifi.apache.org which still supports all three protocol versions (and uses JVM cacerts as truststore).
- This closes #1601
2017-03-21 14:48:40 -04:00
Matt Gilman 8e61aa3c43
NIFI-3361:
- Upgrading to the latest Jetty version.
2017-03-21 13:44:26 -04:00
Puspendu Banerjee 552148e9e7 NIFI-3579: Incorporated Review comments. This closes #1602. 2017-03-20 10:15:24 -04:00
Joe Percivall 5daeba6e03 NIFI-3614 Adding "return" statement to HandleHttpResponse when statusCode is not a number
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1598.
2017-03-20 14:51:56 +01:00
Jeremy Dyer d05727b8c0 NIFI-2613 Apache POI processor for Excel to CSV
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #929.
2017-03-17 10:39:05 -07:00
Matt Gilman 47c6718fe2 NIFI-3615:
- Removing hidden control characters.

This closes #1599

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2017-03-17 11:46:25 -04:00
Matt Gilman 768e89b579 NIFI-3246:
- Preventing the editing of controller services that are defined in an ancestor process group.
- Adding a go to link for users to easily navigate to where services are actually defined.

This closes #1594

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2017-03-14 14:15:14 -04:00
Puspendu Banerjee 78382c66bc NIFI-3579 Fix for Windows FileStore issue
Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #1580
2017-03-14 13:58:06 -04:00
ambud 9424836412 NIFI-3525 Add tablename attribute to QDT flowfiles
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1537.
2017-03-13 14:22:14 -07:00
Joe Gresock bd91390105
NIFI-3582: Fixing PersistentMapCache bug that skipped persistent evictions
This closes #1592.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-13 15:08:25 -04:00
Matt Gilman 9d4239be1e NIFI-3245:
- Adding notes to the controller service tables to better define their availability.
- Clarifying support of Safari.

This closes #1593

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2017-03-13 13:48:11 -04:00
Matt Gilman 620c572703 NIFI-3244:
- Updating Controller Service scope header.
- Rendering the name of the parent when possible.
- Fixing the check if the parent Process Group is write-able when the parent is selected on the current canvas.

This closes #1590

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2017-03-13 12:35:46 -04:00
Jeff Storck a61f353051
NIFI-3520 Updated nifi-hdfs-processors POM to depend directly on hadoop-client
- Removed NAR dependency on nifi-hadoop-libraries-nar from nifi-hadoop-nar so that hadoop-client dependencies will be included directly in nifi-hadoop-nar
- Added RequiresInstanceClassLoading annotation to AbstractHadoopProcessor and HiveConnectionPool
- UGI relogins are now performed using doAs
- Added debug-level logging for UGI relogins in KerberosTicketRenewer and AbstractHadoopProcessor

This closes #1539.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-03-13 12:21:49 -04:00
Matt Gilman 9bb31a70d9 NIFI-3243:
- Rendering the selected Process Group in the header if possible.

This closes #1591

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
2017-03-13 12:18:22 -04:00
Peddy ccd11816e4 NIFI-3034 This closes #1323. Update Counter and test case 2017-03-12 16:34:33 -04:00
Matt Gilman 977aa6919c
NIFI-3575:
- Verify that when moving a snippet, we do not attempt to move a process group into itself.
- Update the UI to ensure a subsequent move event cannot be triggered while a previous move event is still in progress.

This closes #1582.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-10 17:56:57 -08:00
patricker 18f05856ff NIFI-3481 - Add DB Adapter for MS SQL 2012+
Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3481 Added support for Database Adapter Descriptions

Signed-off-by: Matt Burgess <mattyb149@apache.org>

NIFI-3481 Updated Adapter Name

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1510
2017-03-10 11:34:16 -05:00
Bryan Rosander cd8eb775e6 NIFI-3574 - PutHiveStreaming UGI fixes
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1578
2017-03-10 10:50:52 -05:00
joewitt 443803e729 NIFI-2564 corrected L&N to reference new BeanUtils dependency. 2017-03-09 22:00:41 -05:00
Joey Frazee c1e47f9b44 NIFI-2564 This closes #854. Add support for HL7 composite component names
Signed-off-by: joewitt <joewitt@apache.org>
2017-03-09 21:50:02 -05:00
Aldrin Piri c9097e7db3
NIFI-3541 Add copying of local network interface for DtoFactory RPG copy. This closes #1574 2017-03-09 08:23:28 -05:00
James Wing 3c862a1515
NIFI-3334 Fix Update Run Duration. This closes #1572 2017-03-09 08:13:46 -05:00
Andre F de Miranda 5a6373b16c
NIFI-3442 Explicitly support EL on username, password, port.
Introduced basic test unit

This closes #1474.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-08 20:56:01 -08:00
Andy LoPresto 7f6373239f
NIFI-3313 Added explicit Java runtime argument to default bootstrap.conf to avoid blocking on VM deployment.
This closes #1579.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-08 19:24:43 -08:00
Pierre Villard 37e9c5efc7 NIFI-3556 Fixed ValidateCsv schema parsing
Fixed schema parsing on comma boundaries.

This closes #1570.

Signed-off-by: James Wing <jvwing@gmail.com>
2017-03-08 14:57:33 -08:00
Koji Kawamura 27afd2e0cc NIFI-3163: Add new RPG settings to fingerprint
- Added timeout, yieldPeriod, transportProtocol, proxyHost, proxyPort,
  proxyUser and proxyPassword

- Removed unused fingerprint methods

FingerprintFactory has two types of fingerprinting method, from XML
elements and from DTO. However, the ones from DTO are not used by
anywhere. IDE didn't report those private methods unused because
addProcessGroupFingerprint and addSnippetFingerprint call each other,
but those are not used from outside actuallly.

This commit removes those private methods to keep the class clean to
avoid unnecessary code maintenance and tests.

This closes #1332.
2017-03-08 17:31:22 -05:00
Joe Skora 90ff275bb9 NIFI-3517 If HandleHttpResponse cannot write response remove entry from HttpContextMap.
* Add test for failure not clear context map.
* Add handler to remove context map entry if ProcessException occurs during while exporting response.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1567.
2017-03-08 13:16:58 +01:00
Andy LoPresto 1376e5fbd0
NIFI-3483 Added import statement to resolve failing tests.
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-07 20:07:59 -05:00
Pierre Villard 01979aebec
NIFI-3483 - fixed one unit test relying on locale
This closes #1566.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-07 16:41:45 -05:00
Pierre Villard 70175816b3 NIFI-3541 NIFI-3545 - check style violations
Fix for checkstyle issues introduced by:
9e68f02f1f
000414e7ea

This closes #1569.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-03-07 22:17:23 +09:00
jpercivall b7f946e847 NIFI-1682 Adding AttributeRollingWindow processor
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1328.
2017-03-06 19:42:25 +01:00
Matt Gilman 16bde02ed0
NIFI-3541: - Allowing the user to specify the network interface to send/receive data for a Remote Process Group.
This closes #1550.

Signed-off-by: Mark Payne <markap14@hotmail.com>
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-06 10:38:15 -05:00
Mark Payne 9e68f02f1f NIFI-3541: Add local network interface capability to site-to-site client and remote group and ports 2017-03-06 10:36:30 -05:00
Koji Kawamura 000414e7ea
NIFI-3545: Release M FlowFilews once N signals arrive
- Support multiplle incoming FlowFiles to Wait processor, up to Wait
  Buffer Count
- Added Releasable FlowFile Count, which controls how many FlowFiles can
  be released when wait condition is met
- Added special meaning to Notify delta Zero(0) to clear a signal
  counter back to zero

  This closes #1554

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-06 09:29:44 -05:00
Mark Payne 7f5c0dfb54 NIFI-3553: If IOException is thrown when completing FTP transfer, do not update the FlowFile to point to the content that was streamed but instead leave FlowFile pointing to original version
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1562.
2017-03-05 15:27:48 +01:00
Pere Urbon-Bayes 33a702500c NIFI-3554 Fixing the jBCrypt dependency name from nifi-standard-processors/pom.xml file, so it uses the expected naming format from the maven repository
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-03-05 10:15:06 +11:00
Joe Trite f8cad0f8c9
NIFI-1750 Updated ScanAttribute dictionary file processor property description.
This closes #1543.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-03 17:36:40 -08:00
Scott Aslan 9f669aa944
[NIFI-3550] inject nf.ConnectionConfiguration into the nf.Cannection module during nf.Graph init. This closes #1560 2017-03-03 09:38:29 -05:00
Mark Payne 6c44d6be2e NIFI-3523: Ensure that if we are unable to read Provenance events on startup that we don't prevent NiFi from starting
This closes #1542
2017-03-03 09:18:29 -05:00
Mark Payne 4ed64e7561
NIFI-3548: Fixed bug that caused failed requests to not get removed from 'request map' and also results in that preventing the purging logic, which would then unintentially throw exceptions. This closes #1555 2017-03-03 09:13:28 -05:00
Giovanni Lanzani 5990db39ae
NIFI-3459 Fixed documentation for UnpackContent File Filter.
This closes #1558.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-03-02 16:19:11 -08:00
Andre F de Miranda 7c6f2dc677 NIFI-3544 - Fixes minor typo n ExtractEmailHeaders log message
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1552.
2017-03-02 23:09:04 +01:00
josephxsxn ac5a7960e8
NIFI-3547 Changing level from WARN to INFO for Wait Processor's expected behavior
This closes #1556.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-03-02 15:37:39 -05:00
patricker d689ac561e NIFI-3494: GenerateTableFetch Some JDBC Drivers Have Trouble With Name vs Label
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1517
2017-03-02 11:12:54 -05:00
patricker bf2f04fb5f NIFI-3540: QueryDatabaseTable Failing to Track MS SQL Max Values
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1547
2017-03-02 10:04:09 -05:00
Pierre Villard 0d66b6dcda NIFI-1797 - CreateHadoopSequenceFile processor (including Pierre's forced push)
Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-03-02 22:52:10 +11:00
Pierre Villard dcdfd3dad9 NIFI-1797 - Added compression codec property to CreateHadoopSequenceFile processor
This closes: #1387

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-03-02 22:19:34 +11:00
Andre F de Miranda a32e1509b6 NIFI-3238 - Introduce ListenBeats processor and deprecates ListenLumberjack processor
NIFI-3238 - Rewords processor description

NIFI-3238: Removed unused imports

This closes #1418
2017-03-01 21:41:51 -05:00
Scott Aslan c3dbbb845f
[NIFI-3038] Update cluster summary dialog layout. This closes #1534 2017-02-28 09:47:47 -05:00
Bryan Rosander cfe899d189 NIFI-3530 - Ensuring configuration and ugi information is passed down to Hive in PutHiveStreaming
NIFI-3530 - closing renewer in PutHiveStreaming.cleanup(), making npe less likely in HiveWriter.toString()

This closes #1544
2017-02-27 15:01:59 -05:00
Scott Aslan 1978c986b5
[NIFI-3465] change CS and RT warning tooltip viewports to a visible and connected DOM element. This closes #1545 2017-02-27 13:18:14 -05:00
Scott Aslan d98426cc3d
[NIFI-2941] allow users to click checkbox labels to toggle the checkbox. This closes #1538 2017-02-27 09:57:43 -05:00
Scott Aslan a76641b252
[NIFI-2891] persist selected tab for connection configuration dialog, except for when creating a new connection the dialog will open with the details tab visible since users will need to select a connection relationship. This closes #1535 2017-02-27 09:21:55 -05:00
Oleg Zhurakousky 8e17929d6a NIFI-2876 refactored demarcators into a common abstract class
This closes #1214.
2017-02-24 10:06:53 -05:00
Scott Aslan fba17159a9
[NIFI-3358] remove duplicate calls to nfProcessGroupConfiguration.showConfiguration(). This closes #1540 2017-02-24 08:05:34 -05:00
Scott Aslan 2c374bafaa
[NIFI-3358] update naming conventions to avoid conflicts. This closes #1533 2017-02-23 09:14:33 -05:00
Mark Payne 64379d2007 NIFI-3522: When creating a clone of a FlowFile, ensure that we properly set the 'updated attributes' on the RepositoryRecord to all attributes on the FlowFile itself, the same way we do when calling create(FlowFile)
This closes #1532
2017-02-22 17:16:14 -05:00
Andre F de Miranda cefc02c5fb NIFI-3514 - Ensures ListGCSBucket triggers serially and forbids inputs
This closes #1526.

Signed-off-by: James Wing <jvwing@gmail.com>
2017-02-22 10:24:37 -08:00
Mark Payne 96ed405d70 NIFI-3356: Initial implementation of writeahead provenance repository
- The idea behind NIFI-3356 was to improve the efficiency and throughput of the Provenance Repository, as it is often the bottleneck. While testing the newly designed repository,
  a handful of other, fairly minor, changes were made to improve efficiency as well, as these came to light when testing the new repository:

- Use a BufferedOutputStream within StandardProcessSession (via a ClaimCache abstraction) in order to avoid continually writing to FileOutputStream when writing many small FlowFiles
- Updated threading model of MinimalLockingWriteAheadLog - now performs serialization outside of lock and writes to a 'synchronized' OutputStream
- Change minimum scheduling period for components from 30 microseconds to 1 nanosecond. ScheduledExecutor is very inconsistent with timing of task scheduling. With the bored.yield.duration
  now present, this value doesn't need to be set to 30 microseconds. This was originally done to avoid processors that had no work from dominating the CPU. However, now that we will yield
  when processors have no work, this results in slowing down processors that are able to perform work.
- Allow nifi.properties to specify multiple directories for FlowFile Repository
- If backpressure is engaged while running a batch of sessions, then stop batch processing earlier. This helps FlowFiles to move through the system much more smoothly instead of the
  herky-jerky queuing that we previously saw at very high rates of FlowFiles.
- Added NiFi PID to log message when starting nifi. This was simply an update to the log message that provides helpful information.

NIFI-3356: Fixed bug in ContentClaimWriteCache that resulted in data corruption and fixed bug in RepositoryConfiguration that threw exception if cache warm duration was set to empty string

NIFI-3356: Fixed NPE

NIFI-3356: Added debug-level performance monitoring

NIFI-3356: Updates to unit tests that failed after rebasing against master

NIFI-3356: Incorporated PR review feedback

NIFI-3356: Fixed bug where we would delete index directories that are still in use; also added additional debug logging and a simple util class that can be used to textualize provenance event files - useful in debugging

This closes #1493
2017-02-22 12:40:06 -05:00
Koji Kawamura 8d467f3d1f
NIFI-3505: Fix nifi.sh restart
The 'exec' command added by NIFI-2689 affected restart behavior
negatively as 'exec' command will not execute subsequent commands in the
shell script.
This commit changes 'exec' is added only when 'run' is specified.

This closes #1523.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-02-22 10:47:39 -05:00
Scott Aslan 364efa96cc
[NIFI-3501] Upgrade D3 to 3.5.17 and allow npm to pull it down at build time. This closes #1519 2017-02-21 10:17:34 -05:00
Nick Carenza f9c4d3ce3a NIFI-3508: Added support for PATCH identical to that of PUT/POST
This closes #1522.

Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-02-21 16:23:23 +09:00
Aldrin Piri 4f49095d75 NIFI-3294 Adjusting nifi.sh to invoke nifi-env.sh when running as another user such that properties are preserved across environments.
This closes #1405.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
Signed-off-by: ijokarumawak <ijokarumawak@apache.org>
2017-02-20 11:04:11 +09:00
Pierre Villard a8b8f1530b NIFI-2914 Fixed message error in AbstractMongoProcessor
This closes: #1380

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-19 21:25:08 +11:00
Scott Aslan 19302263c3
[NIFI-3181] add operate palette actions to context menu. This closes #1516 2017-02-17 15:40:23 -05:00
Oleg Zhurakousky 6a1854c975 NIFI-3354 Added support for simple AVRO/CSV/JSON transformers that utilize external Schema
Added support for simple Key/Value Schema Registry as Controller Service
Added support for registering multiple schemas as dynamic properties of Schema Registry  Controller Service
Added the following 8 processors
- ExtractAvroFieldsViaSchemaRegistry
- TransformAvroToCSVViaSchemaRegistry
- TransformAvroToJsonViaSchemaRegistry
- TransformCSVToAvroViaSchemaRegistry
- TransformCSVToJsonViaSchemaRegistry
- TransformJsonToAvroViaSchemaRegistry
- TransformJsonToCSVViaSchemaRegistry
- UpdateAttributeWithSchemaViaSchemaRegistry

polishing

NIFI-3354 Adding support for HDFS Schema Registry, unions and default values in the Avro Schema and NULL columns in the source CSV

NIFI-3354 Adding support for logicalTypes per the Avro 1.7.7 spec

NIFI-3354 polishing and restructuring CSVUtils

NIFI-3354 renamed processors to address PR comment

NIFI-3354 addressed latest PR comments
- removed HDFS-based ControllerService. It will be migrated into a separate bundle as a true extension.
- removed UpdateAttribute. . . processor
- added mime.type attribute to all Transform* processors

NIFI-3354 added missing L&N entries

This closes pr/1436
2017-02-17 14:32:06 -05:00
Oleg Zhurakousky ded18b94db NIFI-3255 removed dependency on session.merge from SplitText
NIFI-3255 addressed PR comments

NIFI-3255 fixed linkage for Split creation

This closes #1394
2017-02-17 12:31:21 -05:00
Koji Kawamura 095c04eda0 NIFI-3213: ListFile do not skip obviously old files
Before this fix, files with the latest timestamp within a listing
iteration are always be held back one cycle no matter how old it is.

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-18 00:56:46 +11:00
Scott Aslan d9828c240b
[NIFI-3360] resolve clashing variable name for connection and nf.Connection. This closes #1506 2017-02-16 15:54:00 -05:00
Koji Kawamura 1eb98aefee NIFI-3373: Add nifi.flow.configuration.archive.max.count
- Add 'nifi.flow.configuration.archive.max.count' in nifi.properties
- Change default archive limit so that it uses archive max time(30 days)
  and storage (500MB) if no limitation is specified
- Simplified logic to delete old archives

This closes #1460.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-02-16 14:19:39 +09:00
Aldrin Piri 76fcf5def1 NIFI-3385 - This closes #1440. Updating copyright statements to reflect the new year. 2017-02-16 00:10:02 -05:00
JJ 598563277e NIFI-3485: This closes #1514. Updated test to mock out use of InetAdress when looking up domain. There was an issue where unresolvable domains got routed to an ISP page. This should fix that issue. 2017-02-16 00:07:00 -05:00
Jeff Storck 8b90343715
NIFI-3355 Allows NiFi to bind to specific network interfaces, with separate interface lists for HTTP and HTTPS.
This closes #1508.

Signed-off-by: Bryan Rosander <brosander@apache.org>
2017-02-15 18:39:26 -05:00
João Henrique Ferreira de Freitas a62523c7f5 NIFI-3014 GetSNMP add textual oid attribute to flowfile
Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1196.
2017-02-15 20:20:30 +01:00
Mark Payne 39cfd03397 NIFI-3205: This closes #1336. Ensure that FlowFile Repository is updated with any Transient Content Claims when session rollback occurs 2017-02-15 13:15:56 -05:00
Bryan Bende b207397a11 NIFI-1784 This closes #1349. Initial commit for FetchHBaseRow processor 2017-02-15 12:59:38 -05:00
Koji Kawamura a90fa9c285 NIFI-3431: Support batch update in Notify processor
- Added Signal Counter Delta property
- Added Signal Buffer Count property
- Added processor property name and display name
- Changed IOException handling from routing it to failure to throw
  RuntimeException, so that NiFi framework can yield the processor for a while and try again

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1466.
2017-02-15 15:51:31 +01:00
patricker b7cdc6b382 NIFI-2779 - Add processor to GetEmail Supporting Exchange Web Services
This closes: #1326

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-15 23:48:34 +11:00
Pierre Villard 86fb67d55c NIFI-3405 - Add uptime to JVM section in System Diagnostics
renamed column for node connection date in cluster

This closes #1492.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-02-15 16:53:53 +09:00
Pierre Villard 71e2061b5d NIFI-2968 This closes #1377. Improved message when ExecuteSQL hits unsupported SQL types 2017-02-14 23:25:29 -05:00
Pierre Villard 2d58497c2e NIFI-2859 - This closes #1383. Ignore files starting with a dot in ListHDFS 2017-02-14 23:21:44 -05:00
Pierre Villard 6279fd4184 NIFI-3183 This closes #3183. Added command and arguments as FF attributes in ExecuteProcess processor 2017-02-14 22:08:09 -05:00
Andy LoPresto bd88e4335a
Refactored user identity parsing and proxied entity chain formatting.
Added unit tests.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-02-14 13:39:05 -08:00
Koji Kawamura 4bf267c8bb NIFI-3101: This closes #1271. Improve Get/Modify/PutHTMLElement URL
- Added detailed description about how the URL property works with
  GetHTMLElement
- Added Expression support with URL
- Made URL property dynamic with ModifyHTMLElement and PutHTMLElement,
  since it won't be used to alter HTML element and need not to be
  specified. Making it a dynamic property let existing processor configuration stays valid
2017-02-14 15:28:46 -05:00
JJ 3dc7a160ce NIFI-3478: This closes #1507. added tests for GeoEnrichIP and added support for expression language on 'IP Address Attribute' property. 2017-02-14 13:33:27 -05:00
Koji Kawamura e46f4131f9 NIFI-3452: Add Wait processor Wait Mode property
Ensure back-pressure is active until downstream processing completes.

This closes #1490.
2017-02-14 19:15:27 +01:00
patricker 9cfc13423d NIFI-3455 Large row count paging
This closes #1499.
2017-02-14 19:09:12 +01:00
Bryan Rosander 8ffa1703ba NIFI-3389 - Using long string type for attribute name and value in FlowFileSchema
This closes #1446.
2017-02-14 13:04:09 -05:00
Andre F de Miranda ca887308af
NIFI-2565 - Add Elastic copyright to notice due to the use of grok patterns during test
This closes #1505.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-02-14 12:58:06 -05:00
Hunter Morgan df91169512
NIFI-2689 - improve foreground run mode of nifi
* add exec to RUN_NIFI_CMD
* remove subshell for else
* tested compatible with runit with these changes

This closes #966.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-02-14 12:37:31 -05:00
Simon Elliston Ball 0aa4b72678 Added EL support to CSV properties
Clean up checkstyle errors

Added unit tests, fixed typo

This closes #709

Signed-off-by: Matt Burgess <mattyb149@apache.org>
2017-02-14 11:19:35 -05:00
Matt Burgess 7963df89eb NIFI-3288: Add Clojure support for ExecuteScript
This closes: #1402.

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-15 02:52:11 +11:00
Mike Moser 690130b063
NIFI-3362 update FlowConfiguration.xsd to allow all current time period units
This closes #1501.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-02-14 10:20:39 -05:00
Joe Skora bd3e0438a3 NIFI-2890 Provenance Repository Corruption (1.x)
* Corrected handling of corrupt journal file records that prevents instance startup and loss of records from corrupt files.  Specifically, exception handling was expanded to cover failures on records after the first the same as failures on the first record.
* Adjusted log messages  to reflect that the remainder or all of the journal will be skipped, not just the current record.

This closes #1485.
2017-02-14 10:02:16 -05:00
Pierre Villard 1e43694145 NIFI-3057 Added provenance events to PutElasticsearch and FetchElasticsearch
This closes: #1370.

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-15 01:37:14 +11:00
Selim Namsi 2ef7c15b5d NIFI-2565: add Grok parser
This closes #1108.

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
2017-02-14 21:11:03 +11:00
Andre F de Miranda 1a91ecc379
NIFI-3352 - Address issue where ListenLumberjack was sending long numbers back to clients - instead of int
This closes #1421.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-02-13 23:55:21 -08:00
Koji Kawamura b2290b24a6
NIFI-3107: Unit test failure for DBCPServiceTest
H2 and Kafka broker uses the same default port 9092.
If an user is running Kafka broker on the same machine, or run the unit
tests in parallel, DBCPServiceTest can fail since some of its test
methods connects to port 9092.

This closes #1504.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-02-13 23:09:07 -08:00
joewitt 7e97946c35
NIFI-3412 ignoring tests which depend on timing/network setup
This closes #1503.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-02-13 22:46:44 -08:00
Yolanda M. Davis f1d0a6b36b NIFI-3010 - This closes #1348. applied styling changes 2017-02-13 22:38:54 -05:00
Yolanda M. Davis 8d082d1aca NIFI-3010 - Incorporate expression language support for jolt-spec for processor and Advance UI. Upgraded jolt library to latest version for support of modify functions. Also refactored to interact with JoltTransform interface and updated styling 2017-02-13 22:16:22 -05:00