Apache
/
nifi
mirror of https://github.com/apache/nifi.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,984 Commits 143 Branches 197 Tags 391 MiB
Commit Graph

2300 Commits

Author SHA1 Message Date
yuri1969 505e93065e NIFI-1027 - Treat label and node of a provenance graph event as one unit 
The JIRA issue asks for treating both node and its label as one unit.
Described cursor issues seems to be fixed already.

However, there is an annoying dead space between a node and label
preventing displaying context menu, etc. Due to SVG group's nature
there has been added an opaque joint to remove the dead space.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2059.
 2017-08-10 10:54:47 +02:00
Andy LoPresto 5150dff70b NIFI-4274 Removed EL evaluation logic from custom file validator. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2071.
 2017-08-10 10:05:16 +02:00
Pierre Villard 6ff8321cf7
NIFI-4278 - add error message to invalid FFs in ValidateXml 
This closes #2069.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-08-09 21:20:08 -07:00
Matt Burgess 9c4fdd4ef3 NIFI-4192: Add more Avro/metadata merge options to MergeContent 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2067.
 2017-08-09 12:55:12 +02:00
Mike Thomsen 67819e5019
NIFI-4124 Added org.apache.nifi.mongo.PutMongoRecord. 
Added changes based on code review. Changed:
* Put record reader instantiation inside of try-with.
* Put a batch size for the insert List.
* Ensured that session.transfer() to the success relationship will always happen.
Removed an unused import to fix the style check.

This closes #1945.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-08-07 22:10:27 -07:00
Bryan Bende 0029f025f8 NIFI-4152 Initial commit of ListenTCPRecord 2017-08-07 22:44:11 +02:00
yuri1969 e44f436bd8 NIFI-4271 - Revert double-click behavior back... 
...to delete a bend point

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2064
 2017-08-07 15:58:48 -04:00
zhengsg ee3b58368d NIFI-4267 Change the class name for log in FileRemovalAction 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2061.
 2017-08-07 21:42:46 +02:00
Wil Selwood f7f64a572c NIFI-4268: Add fetch tag to FetchFile processor. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2062.
 2017-08-07 21:38:55 +02:00
Andy LoPresto 675d989003
NIFI-4139 
- Moved key provider interface and implementations from nifi-data-provenance-utils module to nifi-security-utils module.
- Refactored duplicate byte[] concatenation methods from utility classes and removed deprecation warnings from CipherUtility.
- Created KeyProviderFactory to encapsulate key provider instantiation logic.
- Added logic to handle legacy package configuration values for key providers.
- Added unit tests.
- Added resource files for un/limited strength cryptography scenarios.
- Added ASL to test resources.
- Moved legacy FQCN handling logic to CryptUtils.
- Added unit tests to ensure application startup logic handles legacy FQCNs.
- Moved master key extraction/provision out of FBKP.
- Removed nifi-security-utils dependency on nifi-properties-loader module.
- Added unit tests.
 2017-08-07 15:21:58 -04:00
Pierre Villard 8b54c2604c NIFI-4082 - Added EL on GetMongo properties 
NIFI-4082 - Added EL on DB, URI and Collection

NIFI-4082 - Added UT for EL evaluation (URI, DB, Collection) and fixed ex. message for document validator.

This closes #1969
 2017-08-07 13:40:27 -04:00
yuri1969 329b1caf82 NIFI-4266 - Allow double-click to display Event 
...details

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2058
 2017-08-07 08:44:13 -04:00
Bryan Bende f8f1cc8d0d
NIFI-4024 Ensuring InputStream gets closed and cleaning up complex field handling 
This closes #1961.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-08-04 14:54:05 -04:00
Mike Thomsen 496a32e12c
NIFI-4024 Added org.apache.nifi.hbase.PutHBaseRecord 
Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-08-04 14:54:01 -04:00
Yolanda M. Davis afd4f9e034
NIFI-4022 - Initial update for SASL support for cluster management in Zookeeper 
NIFI-4022 - adding sasl documentation update and update to test

This closes #2046.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-08-04 14:19:35 -04:00
Mark Payne 969bbe654c NIFI-4051: Ensure that we use the appropriate Lexical syntax for SQL query validation as we do when running 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2054.
 2017-08-04 16:31:56 +02:00
Wesley-Lawrence 40cde0466a NIFI-4215 NiFi can now parse an Avro schema of a record that references an already defined record, including itself. 
This closes #2052.
 2017-08-03 15:13:07 -04:00
Matt Burgess 2d33582147 NIFI-4253: Add Oracle 12+ database adapter 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2048.
 2017-08-03 09:59:29 +02:00
James Wing 2502b79bae NIFI-4215 Revert Complex Avro Schema Changes 
This reverts commit cf49a58ee7.
 2017-08-01 21:03:04 -07:00
yuri1969 bcf60aa556 Add double-click shortcut to RPG 
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2009
 2017-07-31 10:58:49 -04:00
yuri1969 ef9cb5be23 NIFI-1580 - Allow double-click to display config 
* Added double-click shortcut opening config/details dialog to processors,
connections, ports and labels.
* Created a base for further default action selection, disabling, etc.
* Omitted default action configuration UI - that might be a separate JIRA ticket.
 2017-07-31 10:36:37 -04:00
Wesley-Lawrence cf49a58ee7 NIFI-4215 Allow Complex Avro Schema Parsing 
NiFi can now parse an Avro schema of a record that references an already defined record, including itself.

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2034.
 2017-07-30 16:31:39 -07:00
mans2singh 0bb1411532 NIFI-4212 - RethinkDB Delete Processor 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2030.
 2017-07-29 14:49:22 -07:00
Leah Anderson 70878fe6d6 NIFI-4052 Add DynamicProperty annotation for LookupRecord 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2032.
 2017-07-28 17:12:06 +02:00
Mark Payne 1d6b486b63 NIFI-4232: Ensure that we handle conversions to Avro Arrays properly. Also, if unable to convert a value to the expected object, include in the log message the (fully qualified) name of the field that is problematic 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2040.
 2017-07-27 08:57:25 +02:00
Matt Burgess dc4006f423 NIFI-3335: Add initial.maxvalue support to GenerateTableFetch 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2039.
 2017-07-27 08:43:00 +02:00
Pierre Villard 759f81bc1b
NIFI-4221 - Print app startup in human-readable time 
This closes #2038.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-07-25 15:25:55 -07:00
Matt Burgess 9a5d4ff6b3 NIFI-4217 - Retain original DDL in CaptureChangeMySQL 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2036.
 2017-07-25 17:54:11 +02:00
Matt Gilman 743c6b9c17 NIFI-4032: - Introducing the ManagedRangerAuthorizer. - Introducing the AuthorizationAuditor. - Updating authorization requests to utilize Authorizable where ever possible so allow for a singular place to audit resource not found as denied when the parent authorizable is null (no more inheritance). - Updating unit tests as appropriate. - Addressing issues with broken web-api integration tests. 
NIFI-4032: - Generating the appropriate fingerprint for the ManagedRangerAuthorizer based on whether the UserGroupProvider is configurable. - Adding unit tests.

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #2019
 2017-07-24 15:51:27 -04:00
Pierre Villard 10692256d6 NIFI-917 Improve Rendering of Component Docs 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2025.
 2017-07-23 11:30:04 -07:00
Leah Anderson 3371e915cc NIFI-4087 This closes #2026. Fix to allow exclusion of filename from tika criteria. 2017-07-21 16:56:56 -04:00
Mark Payne 695e8aa98f
NIFI-106: 
- Expose processors' counters in Stats History
- Only include counters in Processors' Status History if user has read access to corresponding Processor
- Addressed review feedback. Found and addressed bug where a counter is not present in all of the aggregate snaphot values for status history, resulting in the UI not rendering the chart properly
- This closes #1872
 2017-07-21 14:18:11 -04:00
m-hogue c54b2ad81c NIFI-3736: change to honor nifi.content.claim.max.appendable.size and nifi.content.claim.max.flow.files properties. Added 100 MB cap for NiFiProperties.MAX_APPENDABLE_CLAIM_SIZE 
This closes #2010.
 2017-07-21 09:32:45 -04:00
Davide d334532b16 NIFI-4184: Enabled EL on PutHDFS Remote Group and Owner 
I needed to put some attributes on REMOTE_GROUP and REMOTE_OWNER, in order to achieve it i put expressionLanguageSupported(true) on the PropertyDescriptor of REMOTE_GROUP and REMOTE_OWNER

This closes #2007.

Signed-off-by: Davide <davidde85@hotmail.it>
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-07-21 16:31:46 +09:00
Andre F de Miranda b0be99036d NIFI-55 - Ensures ListenHTTP logs the source of an expired hold 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1623.
 2017-07-20 23:14:32 +02:00
Koji Kawamura b4e0a6e206 NIFI-4205: Avoid duplicated data from TailFile 
Before this fix, it is possible that TailFile to produce duplicated data
if an already tailed file has newer timestamp and fewer or the same
amout of data.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2021.
 2017-07-20 12:14:05 +02:00
James Wing 1f330b4fe4 NIFI-11 Fixing checkstyle issue - usused import 2017-07-19 22:46:15 -07:00
mans2singh ea1fe1a9c0 NIFI-4188 New RethinkDB Get processor 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2012.
 2017-07-19 20:56:47 -07:00
Andre F de Miranda 451a88df43
NIFI-11 
- Capture Exception to prevent failed evaluations from yielding processor
- Further capture evaluation exceptions as per PR feedback
- Adjust jUnit to new exception behavior
- This closes #1644
 2017-07-19 11:50:48 -04:00
Bryan Bende f87d2a2f57 NIFI-4157 Improvements to PutTCP 
Signed-off-by: Bryan Bende <bbende@apache.org>
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1989.
 2017-07-19 12:25:42 +02:00
joewitt 59754500d5 NIFI-2829 ignoring unit test that appears broken. Commented on JIRA about the problem observed 2017-07-18 22:23:10 -04:00
Andre F de Miranda a307ea3a00 NIFI-4098 This closes #2013. Adjust ParseCEF to re-use bean validator in order to improve performance 2017-07-18 22:02:39 -04:00
Matt Burgess 844dbe4edb NIFI-4156: Fixed fragment.count in SplitText to equal emitted flow files 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2014.
 2017-07-18 11:31:16 +02:00
deonhuang a6e94de0bb NIFI-2829 - Add Date and Time Format Support for PutSQL 
Fix unit test for Date and Time type time zone problem
Enhance Time type to record milliseconds

This closes #1983.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-07-18 14:16:10 +09:00
patricker 03bff7c9fc NIFI-2829 Date and Time Format Support for PutSQL 
This closes #1524.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-07-18 14:11:37 +09:00
Mark Payne a9529adab1 NIFI-4093: Fixed bug where we used the wrong Enum to determine how to process an update to the FlowFile Repo 
Signed-off-by: Matt Burgess <mattyb149@apache.org>
 2017-07-13 10:51:02 -04:00
Koji Kawamura 5c48655e69 NIFI-4112: Fix ConvertExcelToCSV to handle empty cells. 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1973
 2017-07-13 10:40:01 -04:00
Koji Kawamura 8acee02393 NIFI-1613 
- Truncate text data types only.
- Added conversion from a boolean to number.
 2017-07-12 17:02:55 -04:00
Koji Kawamura 3844a821f1 NIFI-1613: ConvertJSONToSQL truncates numeric value wrongly. 
- Changed boolean value conversion to use Boolean.valueOf.
- Updated comments in source code to reflect current situation more clearly.
- Updated tests those have been added since the original commits were made.
 2017-07-12 17:02:55 -04:00
Toivo Adams 3b2e43b75c NIFI-1613 Initial version, try to improve conversion for different SQL types. New test and refactored existing test to reuse DBCP service. 
nifi-1613 Adding numeric and Date/time types conversion and test.
 2017-07-12 17:02:55 -04:00
Mark Payne b603cb955d NIFI-4060: Initial implementation of MergeRecord 
NIFI-4060: Addressed threading issue with RecordBin being updated after it is completed; fixed issue that caused mime.type attribute not to be written properly if all incoming flowfiles already have a different value for that attribute

NIFI-4060: Bug fixes; improved documentation; added a lot of debug information; updated StandardProcessSession to produce more accurate logs in case of a session being committed/rolled back with open input/output streams
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1958
 2017-07-12 16:36:48 -04:00
Matt Gilman eefad29167 NIFI-4127: 
- Introducing composite ConfigurableUserGroupProvider and UserGroupProvider.
- Adding appropriate unit tests.
- Updating object model to support per resource (user/group/policy) configuration.
- Updating UI to support per resource (user/group/policy) configuration.
- Adding necessary documentation.
- Updating documentation to clarify integrity checks.
- Providing an example of configuring a composite implementation.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1978.
 2017-07-11 18:13:04 +02:00
Joe Skora 325fe53fa1 NIFI-3677 - ExtractMediaMetadata should close TikaInputStream 
* Added finally block to close TikaInputStream.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1993.
 2017-07-11 18:04:50 +02:00
Jeff Storck 58fc5f24b8
NIFI-1586 Removed check for distributed ZK quorum before starting the DatadirCleanupMananger to enable autopurge during standalone ZK server usage. This closes #2001 2017-07-11 10:04:08 -04:00
mans2singh 0fd51b4d12 NIFI-4145 Added writes attributes annotation 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1998.
 2017-07-10 21:25:26 -07:00
m-hogue 78fbb8f2ee
NIFI-3939: Reviewed and corrected all incorrect nifi-web-api resource response types. This closes #1999 2017-07-10 14:40:13 -04:00
Mark Payne 87e062ff55
NIFI-4167: StandardResourceClaimManager should not synchronize on a ResourceClaim in order to determine the claim count. This closes #1996 2017-07-10 14:33:07 -04:00
Tony Kurc 2dc45a4dd7 NIFI-4154 Fixing line endings in .java,.html files 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1982.
 2017-07-09 12:13:15 -07:00
mans2singh affc88e599 NIFI-4118 RethinkDB bundle with put processor 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1942.
 2017-07-09 11:10:23 -07:00
Mark Payne 9e296830ab
NIFI-4151: Ensure that we properly call invalidateValidationContext() when properties change; ensure that in the controller service provider we don't replace a controller service with a new node if the ID's match, as we won't be able to actually add the new one to the flow. This closes #1995 2017-07-07 16:41:32 -04:00
Yolanda M. Davis 6df97bbc88
NIFI-4135 - added hadoop-client and enhanced Authorizers entity to support classpath for resources entry 
NIFI-4135 - classpath under class

This closes #1956.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-07-07 10:08:48 -04:00
kwydler e89512e744 NIFI-4161: Adding expression evaluation to AWS PROXY_HOST and PROXY_HOST_PORT property usage 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1990.
 2017-07-07 13:28:33 +02:00
Koji Kawamura e84f9a2416 NIFI-4160: SFTPTransfer connection timeout for opening channel. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1991.
 2017-07-07 12:31:50 +02:00
Koji Kawamura 50c364a793 NIFI-4162: PutSQL batch update error message should include the cause 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1992.
 2017-07-07 12:25:10 +02:00
Pierre Villard 45f82dc855 NIFI-4111 - NiFi shutdown 
Fixed threads shutdown so that NiFi can shutdown gracefully

NIFI-4111 - Review - Handling SocketRemoteSiteListener (RAW S2S)

This closes #1963.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-07-07 14:23:18 +09:00
Mark Payne 3906d4e1d2 NIFI-1763: Initial implementation of ConfluentSchemaRegistry. 
NIFI-1763: Fixed bug where the Confluent Schema Registry Schema Access Writer was not being created

Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1938
 2017-07-06 15:52:07 -04:00
Pierre Villard a3b72f1bb7
NIFI-4143 - externalize MAX_CONCURRENT_REQUESTS. This closes #1962 2017-07-06 15:38:55 -04:00
Mark Payne ba56774fa1
NIFI-4151: Updated UpdateAttribute to only create JAXB Context once; Minor performance tweaks to standard validators and StatusMerge.prettyPrint; updated AbstractConfiguredComponent to not create a new ValidationContext each time that validate is called but only when needed; updated FlowController, StandardControllerServiceProvider, and StandardProcessGroup so that component lookups can be performed using a ConcurrentMap at FlowController level instead of having to perform a depth-first search through all ProcessGroups when calling findProcessor(), findProcessGroup(), findXYZ() 
This closes #1979
 2017-07-06 13:43:56 -04:00
Mark Payne cff81c0cd2
NIFI-4153: Use a LinkedBlockingQueue instead of a SynchronousQueue for Request Replicator's thread pool so that requests will queue when all threads are active, instead of throwing an Exception. This closes #1980 2017-07-06 10:11:48 -04:00
m-hogue e6b166a3a2 NIFI-552: added regex properties for include and ignore filters in LogAttribute 
This closes #1981

Signed-off-by: Tony Kurc <tkurc@apache.org>
 2017-07-05 21:21:50 -04:00
m-hogue 47eece5798 NIFI-3193: added ability to authenticate using cert common names 
This closes #1971.

Signed-off-by: Tony Kurc <tkurc@apache.org>
Also reviewed by Pierre Villard <pierre.villard.fr@gmail.com>
 2017-07-05 15:28:54 -04:00
Pierre Villard 7843b885ee NIFI-4144 - added min/max age to ListHDFS processor 
This closes #1966.

Signed-off-by: Tony Kurc <tkurc@apache.org>
 2017-07-04 21:18:43 -04:00
Pierre Villard 8b5342dec0 NIFI-3974 - Added charset in LogAttribute 
This closes #1967

Signed-off-by: Tony Kurc <tkurc@apache.org>
 2017-07-02 12:03:07 -04:00
Pierre Villard 902b6e205d NIFI-4131 - GFT warning message when badly setting relationships 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1954
 2017-06-30 15:12:10 -04:00
Mark Payne 7f4cfd51ea NIFI-3897: If swapping data into queue and an unexpected exception/error is thrown, do not lose track of the swap file 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1960.
 2017-06-30 13:53:05 +02:00
Mark Payne cdc154f7c8 NIFI-4046: If we are unable to parse out any records from a Kafka Mesaage with ConsumeKafkaRecord, then we should route all of the bytes received to 'parse.failure' 
NIFI-4046: Addressed issue of Record Writer failing with ConsumeKafkaRecord

This closes #1906.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-06-30 17:56:14 +09:00
m-hogue 58a623dfa2 NIFI-4037 added InvokeGRPC processor, with proto service IDL 
NIFI-4038 added ListenGRPC processor

This closes #1947

Signed-off-by: Tony Kurc <tkurc@apache.org>
 2017-06-29 22:01:22 -04:00
yuri1969 3bf1d12706
NIFI-4125 Added secure transform feature and configuration to TransformXML processor to mitigate XXE file system leaks. 
This closes #1946.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-29 10:40:33 -07:00
Mike Thomsen 5172797448 NIFI-4122 Added the ability to combine multiple Mongo result documents into a single output JSON array. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1948.
 2017-06-29 12:43:52 +02:00
Sreeram Garlapati 82ef671953 NIFI-4007 - Update azure EventHubs client to latest version 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1880.
 2017-06-29 09:30:39 +02:00
Michael Andre Pearce bdfd710692 NIFI-3983 - Support ability to make JMS 2.0 durable subscriptions on Topic 
Add logic in Consumer adding support for all topic consumer combinations, non-durable, durable, shared, durable-shared.
Add new optional config option to supply subscription name.
Add new optional config option to supply clientId.

This closes #1863.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-06-29 10:23:15 +09:00
Matt Burgess 9bfa7469cb NIFI-4137: Add support for UTF-8 in GetFTP and PutFTP 
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1957
 2017-06-28 16:41:14 -04:00
Matt Gilman c99c036c20 NIFI-4045: 
- Addressing issues causing the eventId to not be relayed when submitting a lineage request under certain conditions.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1903.
 2017-06-27 21:48:55 +02:00
Andre F de Miranda 1e7eceee84 NIFI-4126 - Add SupportBatching, SideEffectFree and EventDriven annotations to ExtractGrok 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1950.
 2017-06-27 20:56:56 +02:00
yuri1969 202eb5ccbe
NIFI-4119 - Improve UX of canvas label configuration by providing immediate focus to value field 
This closes #1940.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-27 10:38:35 -07:00
kwydler c37aed4de2 NIFI-4128 Added "All" MessageAttributeNames to GetSQS processor ReceiveMessageRequest 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1949.
 2017-06-26 22:25:16 -07:00
Scott Aslan 24e2981012 [NIFI-4117] update connection config prioritizers to be contained within their parent element 
This closes #1944.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
 2017-06-26 20:25:50 +02:00
Steve Champagne 68b42c9e54 NIFI-4055: Add a compression option to AvroRecordSetWriter 
This closes #1909.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>
 2017-06-26 19:16:31 +02:00
Koji Kawamura 91ed96f8c4 NIFI-4040: Continue if SFTP failed getting user home directory 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1898.
 2017-06-26 19:09:22 +02:00
Koji Kawamura 69613f29c9
NIFI-4050: Fixed NPE at AbstractConfiguredComponent.validate 
This closes #1905.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-23 13:58:22 -04:00
Pierre Villard 47165afc0c
NIFI-4073 - fix duplicated stack trace 
This closes #1916.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-23 11:20:07 -04:00
Mark Payne 8878d732dd NIFI-4103: Return an empty Optional if an IP address cannot be found 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1933
 2017-06-23 11:14:57 -04:00
Andy LoPresto ff65b42e88 NIFI-4110 Added Thread.sleep() to ensure provenance event indexing occurs before retrieval and unit test passes. 
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1935
 2017-06-23 08:48:34 -04:00
patricker 7c27d4c588 NIFI-3992: Added Original Headers to Include/Exclude 
Some headers can cause problems with message parsing, specifically the 'Content-Type' header.
If an email contains attachments, ConsumeEWS may generate emails where the attachments cannot be extracted.

This closes #1867.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-06-23 17:38:19 +09:00
Andy LoPresto 253ea2e73b NIFI-4095 Changed minimum capture group count in ExtractText from 1 to 0. 
Added unit test and removed obsolete test.
Added custom validation to enforce capture group if "include capture group 0" is false.
 2017-06-22 23:31:35 +02:00
Koji Kawamura 5c755c006b
NIFI-4017: Emit provenance event from Notify. 
This closes #1890.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-22 16:49:31 -04:00
Pierre Villard f54e146561
NIFI-4085 Add exception in log message when (S)FTP listing fails 
This closes #1929.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-22 16:06:18 -04:00
Niels Zeilemaker 5b82b735ab
NIFI-4106 Increase log level of user authentication rejection to warn. 
This closes #1934.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-22 14:03:25 -04:00
Jeff Storck 668a64cd56 NIFI-1586 Implements autopurging of transaction log and snapshot files 
When NiFi is clustered, and autopurge.purgeInterval is greater than 1, the DatadirCleanupManager will be started in order to automatically purge transaction log and snapshot files based on the autopurge settings in zookeeper.properties

This closes #1928.
 2017-06-22 09:30:38 -04:00
Jeff Storck c99100c934
NIFI-4010 Enables EL on Fetch/List/PutSFTP and List/Fetch/Put/DeleteHDFS processor properties 
FetchSFTP/ListSFTP/PutSFTP: Private Key Path
ListHDFS/FetchHDFS/PutHDFS/DeleteHDFS: Hadoop Configuration Resources, Kerberos Principal, Kerberos Keytab, Kerberos Relogin Period

This closes #1148
This closes #1930.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-21 17:14:49 -04:00
Maurizio Colleluori 59a32948ea
NIFI-2923 Added evaluation of attribute expressions for Kerberos principal and keytab 
Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-21 17:14:28 -04:00
Maurizio Colleluori 86fa1bba4f
NIFI-2923 Add expression language support to Kerberos parameters used by processors 
Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-21 17:14:27 -04:00
Mark Payne 73e601bc55 NIFI-4102: If first line read does not match Grok expression, skip line 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1932
 2017-06-21 15:10:22 -04:00
Joe Percivall 77e49b749e NIFI-4075 Updating 'addRemoteProcessGroup' in StandardFlowSerializer to properly use ScheduledStateLookup. This closes #1922. 2017-06-21 14:20:22 -04:00
Bryan Bende aabd4a25d2 NIFI-4043 Initial commit of nifi-redis-bundle 
NIFI-4061 Initial version of RedisStateProvider
- Adding PropertyContext and updating existing contexts to extend it
- Added embedded Redis for unit testing
- Added wrapped StateProvider with NAR ClassLoader in StandardStateManagerProvider
- Updating state-management.xml with config for RedisStateProvider
- Renaming tests that use RedisServer to be IT tests so they don't run all the time

This closes #1918.
 2017-06-21 12:47:19 -04:00
Matt Gilman 6bc6f955c0 NIFI-4059: 
- Introducing the LdapUserGroupProvider.
- Updating documentation accordingly.
- Moving the IdentityMapping utilities so they were accessible.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1923.
 2017-06-19 19:25:33 +02:00
Scott Aslan ad6af1d941 NIFI-3994 Add Create Template to the context menu 
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1883.
 2017-06-17 16:46:31 -07:00
Wesley-Lawrence 5a7e6c6ac1 NIFI-4064 Make sure that Funnels with queued incoming FlowFiles, but no outgoing connections yield rather than continually check to run. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1914.
 2017-06-17 15:50:40 +02:00
Chris Herrera 2800df30d3 NIFI-2692 
- Removing problematic timeout for SMTP Listen
- Converting anonymous method to lambda
- Adding debug and error logging

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1924.
 2017-06-17 15:22:33 +02:00
Matt Gilman 490e1da5db NIFI-4019: 
- Adding support for X-Forwarded-* headers.
- Unrelated code clean up.
- Addressing additional proxying issues.
- Addressing mistyped landing page.
- Handled trailing slashes in proxy headers for context path
 2017-06-15 14:17:35 -04:00
Mark Payne c0ad5bcc7b NIFI-1921: Fixed bug in how average lineage duration was calculated for cluster stat 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1912
 2017-06-13 12:29:03 -04:00
Bryan Bende 2b47ade0c4
NIFI-4049 Fixing unit test that doesn't run on OSX and caused Travis failure 2017-06-12 16:36:23 -04:00
Koji Kawamura de1b84e2aa
NIFI-4049: Refactor AtomicDistributedMapCacheClient 
To be used with cache engines that does not have revision number.

NIFI-4049: Refactor AtomicDistributedMapCacheClient

Removed old methods completely.

This closes #1904.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-12 10:34:48 -04:00
Koji Kawamura 7761903352 NIFI-4041: Add file.size attribute to ListFTP and ListSFTP 
This closes #1899.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-06-12 10:53:09 +09:00
Mark Payne e7dcb6f6c5 NIFI-3921: Allow Record Writers to inherit schema from Record 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1902
 2017-06-09 16:13:25 -04:00
Matt Gilman 4ed7511bee
NIFI-3653: - Introducing UserGroup and Policy provider interfaces. 
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.

This closes #1897.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-09 13:54:10 -04:00
Mark Payne d8dc2bb00b NIFI-4047: Addressed issue with inconsistent Lexical policy between used cached schemas and non-cached schemas 
This closes #1907

Signed-off-by: Joe Percivall <JPercivall@apache.org>
 2017-06-09 12:05:11 -04:00
Matt Gilman cc741d2be6
NIFI-3997: 
- Bumping to next minor version.
 2017-06-08 15:22:51 -04:00
Matt Gilman 1bf0a1a849
Merge branch 'NIFI-3997-RC1' 2017-06-08 14:30:10 -04:00
Bryan Bende b0c9428776 NIFI-4030 Populating default values on GenericRecord from Avro schema if not present in RecordSchema 
This closes #1896.
 2017-06-07 13:59:40 -04:00
Steve Champagne 45e035686f
NIFI-4029: Allow null Avro default values in HortonworksSchemaRegistry 
This closes #1894.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-07 12:03:53 -04:00
Matt Gilman c86190c513 NIFI-4027: - Fixing the positioning of the tooltips in the component toolbar. 
This closes #1893

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>
 2017-06-06 17:35:43 -04:00
Joe Gresock 3ef0fa48da NIFI-4014: Correcting Event Type text in S2S provenance reporting task 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1887.
 2017-06-06 12:06:04 +02:00
Koji Kawamura e1880785d0
NIFI-4021: Fix StandardFlowSynchronizerSpec to pass independently. 
This closes #1892.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-06-05 19:25:07 -07:00
Matt Gilman 6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Andre F de Miranda 082b99640e NIFI-4001 - Bump ParCEFone version 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1875.
 2017-06-03 12:11:38 +02:00
Andre F de Miranda 4d78052dd4 NIFI-1709 - Introduce logic to probe Linux version using /etc/os-release to nifi.sh 
Add explicit paths to support SLES 11 SP4 / OpenSUSE init.d layout

This closes #1794
 2017-06-03 08:12:52 +10:00
Mark Payne 32314d70fd NIFI-4009: Added support for several key functions in RecordPath 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1881
 2017-06-02 14:06:05 -04:00
Matt Burgess 0bddcfe730
NIFI-4002: Add PutElasticsearchHttpRecord processor 
This closes #1878

Signed-off-by: Bryan Rosander <brosander@apache.org>
 2017-06-02 13:01:13 -04:00
Mark Payne 37be0b9820 NIFI-3990: This closes #1870. Avoided creating garbage when we can avoid it in the JSON and csv readers and json writer 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-06-02 10:01:40 -04:00
Mark Payne 067e9dfeb0
NIFI-4003: Expose configuration option for cache size and duration NIFI-4003: Addressed remaining spots where client does not cache information 
This closes #1879.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-01 15:31:35 -04:00
Mark Payne 2595d816c4 NIFI-3986: This closes #1868. When we encounted EOFException from sequential record reader event iterator, just treat it as not having any more events 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-06-01 15:08:12 -04:00
Mark Payne 7035694e37
NIFI-3995: Updated Hwx Encoded Schema Ref Writer to write 13 bytes for header instead of 14; added unit test to verify 
This closes #1876.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-06-01 10:29:58 -04:00
Mark Payne a0b2311ff6 NIFI-3995: This closes #1873. No longer use the 14th byte in the header for hwx content-encoded schema reference 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-31 13:41:25 -04:00
Mark Payne 2b435cdfc6 NIFI-3985: This closes #1864. Added 'Starting Position' property to SiteToSiteReportingTask; also added additionalDetails.html that explains the schema and updated the reporting task to stop publishing when the user clicks 'stops' instead of running indefinitely until the reporting task has caught up 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-26 21:05:45 -05:00
Yolanda M. Davis dd50745a9f
NIFI-3984 - upgraded version of schema-registry client to 0.2.1 
This closes #1866.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-26 17:59:53 -04:00
Pierre Villard 80dfe0257f ExtractGrok - Include exception message when compile fails 
This closes: #1845

Signed-off-by: Andre F de Miranda <trixpan@users.noreply.github.com>
 2017-05-27 00:56:25 +10:00
Bryan Bende e85081ed20 NIFI-3979 Documenting how ListHDFS maintains state and performs listings 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1860.
 2017-05-26 10:40:22 +02:00
Mark Payne 5cb928131c
NIFI-3981: When serializing flow to cluster, use the Scheduled State of ports as they are configured to be, not the current state, since the current state may change as soon as the FlowController has finished initializing 
This closes #1861.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-25 16:45:20 -04:00
Koji Kawamura 13b59b5621 NIFI-3958: Decimal logical type with undefined precision and scale. 
- Oracle NUMBER can return 0 precision and -127 or 0 scale with variable scale NUMBER such as ROWNUM or function result
- Added 'Default Decimal Precision' and 'Default Decimal Scale' property to ExecuteSQL and QueryDatabaseTable to apply default precision and scale if those are unknown
- Coerce BigDecimal scale to field schema logical type, so that BigDecimals having different scale can be written

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1851
 2017-05-25 14:37:17 -04:00
Mark Payne 36911957dc NIFI-3972: This closes #1855. When enabling Controller Services on startup, wait until service is enabled (or until 30 seconds elapses, whichever comes first) before returning. This avoids having Service A depend on Service B and then attempting to start Service A before Service B is fully enabled 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-25 13:32:02 -04:00
Mark Payne eb25c8547a NIFI-3972: Ensure that we wait until service state becomes enabled before triggering completable future that says that it's enabled 2017-05-25 13:31:53 -04:00
Joseph Niemiec 44fdc0e4ef NIFI-3640 uri eventhub changes 
This closes #1617
 2017-05-25 13:14:51 -04:00
Koji Kawamura 4d0667380a NIFI-3404: Improved UX of LookupAttributes. 
- Added dependency notice.
- Added EL evaluation at SimpleKeyValueLookupService.
- Updated documentation.
- Updated CommonsConfigurationLookupService to throw LookupFailureException if it fails to get configuration so that error messages can be displayed at each processor bulletin.
- Added calling getConfiguration at OnEnabled of CommonsConfigurationLookupService, so that the service will stay in Enabling state if there is any issue.

Signed-off-by: Joey Frazee <jfrazee@apache.org>
 2017-05-25 10:05:32 -05:00
Joey Frazee 46e2420d74 NIFI-3404 Added LookupAttribute processor and lookup controller services 
Signed-off-by: Joey Frazee <jfrazee@apache.org>
 2017-05-25 10:05:32 -05:00
Koji Kawamura 23cbc3b346 NIFI-3976: This closes #1857. Fix ConsumePOP3 and ConsumeIMAP to delete messages. 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-25 10:59:53 -04:00
Pierre Villard dc5e032368 NIFI-3791 - added back pressure data into S2SStatusReportingTask 
This closes #1745.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-05-25 18:20:11 +09:00
Andre F de Miranda c07850aec3 NIFI-2702 - Deprecates nifi-standard-bundle/*JMS and document the recommended alternative 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1643.
 2017-05-25 08:44:39 +02:00
Pierre Villard e05005584d NIFI-3809 - Added HTTP mode and HTTP proxy for S2S Reporting Tasks 
This closes #1754.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-05-25 15:31:49 +09:00
Pierre Villard b6eb0ac0fb NIFI-3859 - Provide filtering options in S2SProvenanceReportingTask 
This closes #1777.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-05-25 14:10:50 +09:00
Mark Payne eaefec6d81 NIFI-3971: This closes #1854. Fixed bug in calculating content size that was transferred when cloning a relationship 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-24 22:14:56 -04:00
Mark Payne 5aa3baca79 NIFI-3568: This closes #1577. Use a cached thread pool in order to allow ThreadPoolRequestReplicator to scale up the number of threads to some configurable max 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-24 21:21:45 -04:00
Scott Aslan 86728bac7e
[NIFI-3925] for the jquery.combo plugin: update the calculation of the width of the combo box options overlay and also updated the plugins styles to now leverage the css calc() to determine the exact width of .combotext and .combo-option-text elements. This closes #1848 2017-05-24 16:15:42 -04:00
Matt Gilman f97b3fe455
NIFI-3963: - Ensuring the RemoteGroupPort yields when the details cannot be refreshed from any of the configured remote instances. 
This closes #1853.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-24 15:58:22 -04:00
Bryan Bende ae3db82303
NIFI-3644 Fixing the result handler in HBase_1_1_2_ClientMapCacheService to use the offsets for the value bytes 
This closes #1645.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-24 14:59:28 -04:00
baolsen 152f002abf
Completed initial development of HBase_1_1_2_ClientMapCacheService.java which is compatible with DetectDuplicate (and other processors) 
Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-24 14:59:23 -04:00
Mike Moser 232380dbfd NIFI-1452 on timer-driven yield, use the greater of yield duration or run schedule 
This closes #1832.
 2017-05-24 14:24:29 -04:00
Mark Payne b12cf8a6d2 NIFI-3671: This closes #1852. Ensure that we use the existing ResourceClaim (if it exists) when swapping data in, instead of creating a new one. Otherwise, if the ResourceClaim is still writable, then we may archive the data and then write to it, which can cause a NullPointerException in FileSystemRepository 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-24 14:21:18 -04:00
Pierre Villard fb7d6d1150 NIFI-3799 Add Input Requirement Info to Docs 
Added input requirement info for processor documentation

This closes #1756.

Signed-off-by: James Wing <jvwing@gmail.com>
 2017-05-24 10:45:45 -07:00
Mark Payne 08b66b5b6a NIFI-3969: Prevent merging flowfiles prematurely when all bins fill but some are already full and can be processed 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1850.
 2017-05-24 19:36:18 +02:00
Pierre Villard bb96b0f464 NIFI-3816 - Added correct mime.type to S2S bulletin and provenance reporting tasks 
This closes #1758.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-05-25 00:33:50 +09:00
Scott Aslan 7a46238baf
[NIFI-3943] align combo option item toolips to hover closer to the icon. This closes #1849 2017-05-24 11:14:48 -04:00
Pierre Villard ba49b8427c
NIFI-3191 - HDFS Processors Should Allow Choosing LZO Compression 
This closes #1802.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-24 11:04:00 -04:00
Pierre Villard d63fa8a822 NIFI-3878 This closes #1790. AmbariReportingTask - Default run schedule set to 1 minute 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-24 00:09:17 -04:00
Andy LoPresto 6d07ebf50c NIFI-3954 This closes #1841. Upgraded version of httpclient library to handle TLS issues. 
Added explicit dependency on httpcore for @ThreadSafe annotation.

Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-23 23:42:28 -04:00
Mark Payne 2c751a8e5b NIFI-3962: This closes #1846. Updated ConsumerLease to better handle batching of messages into a single FlowFile in the same way that it is handled for demarcated data 2017-05-23 23:20:13 -04:00
Matt Burgess fb925fc182 NIFI-3867: Fixed issue with getConnectionURL in HiveConnectionPool using Expression Language 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1847.
 2017-05-23 21:28:59 +02:00
James Wing 9238fdb493 NIFI-3903 Unit Tests for AWS Processors 
* Added non-integration unit tests for Lambda, S3, SNS, and SQS processors
* Moved non-integration tests out of integration test files
* Moved SQS integration tests to IT* files for clarification

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1814.
 2017-05-23 19:25:55 +02:00
Joe Skora 0a7d149656 NIFI-3932 Symbols in API comments cause problems with Swagger Codegen 
* Update ApiModelProperty descriptions to avoid problematic characters.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1821.
 2017-05-23 19:04:54 +02:00
Matt Gilman 3ec0aa947f NIFI-3955: 
- Filtering out the EventTime searchable field for all implementation of ProvenanceRepository. Prefer user query using the startDate/endDate fields.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1844.
 2017-05-23 18:48:29 +02:00
James Wing b64232117f NIFI-3707 Sort Attributes in FlowFile Details UI 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1843.
 2017-05-23 18:06:18 +02:00
Bryan Bende a8de27e69b NIFI-3732 Adding connect with timeout to StandardCommsSession and SSLCommsSession to avoid blocking 
This closes #1842.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
 2017-05-23 15:43:27 +09:00
Matt Gilman ded396f0ef NIFI-3933: 
- When monitoring heartbeats use the connected nodes as the basis for the check. This addresses the case when a node is terminated and no corresponding heartbeats exist.

This closes #1838.
 2017-05-22 16:51:13 -04:00
Matt Gilman d33c4c72d4 NIFI-3933: 
- When monitoring heartbeats use the connected nodes as the basis for the check. This addresses the case when a node is terminated and no corresponding heartbeats exist.
 2017-05-22 16:50:30 -04:00
Mark Payne a1b07b1e9c
NIFI-3949: Updated Grok Reader to allow for sub-patterns to be used when determining the schema 
This closes #1839.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-22 16:30:47 -04:00
Mark Payne 6937a6cf64 NIFI-3953: This closes #1837. Allow multiple schemas on same kafka topic/partition for ConsumeKafkaRecord_0_10 
Also, updated record writers to ensure that they write the schema as appropriate if not using a RecordSet. Updated ConsumeKafkaRecord to allow for multiple schemas to be on same topic and partition

Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-22 14:37:53 -04:00
Mark Payne 6d16fdf170
NIFI-3952: Updated UpdateRecord to pass field-related variables to the Expression Language 
This closes #1836.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-22 14:32:45 -04:00
Mark Payne 7f8987471d
NIFI-3946: Update LookupService to take a Map instead of a String for the input 
NIFI-3946: Fixed issues where null values were returned instead of empty optionals

This closes #1833.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-22 11:52:41 -04:00
Bryan Bende 71cd497fef NIFI-3942 Added retry logic if a lookup fails due to InvalidDatabaseException which occurs if the underlying file was modified before we could refresh the reader 
This closes #1831.
 2017-05-22 09:12:51 -04:00
Bryan Bende f35e0ecdd0 NIFI-3942 Making IPLookupService reload the database file on the fly when detecting the file has changed 2017-05-22 09:06:42 -04:00
Mark Payne c49933f03d NIFI-3948: This closes #1834. Added flush() method to RecordWriter and call it when writing a single record to OutputStream for PublishKafkaRecord. Also removed no-longer-used class WriteAvroResult 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 23:05:04 -04:00
Tim Reardon 58ce52d5d6 NIFI-3854 This closes #1773. Expand expression language support for Kafka processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 15:51:23 -04:00
Matt Burgess 9294a26139
NIFI-3938: Added ScriptedLookupService, some refactor for reusable scripting classes 
This closes #1828.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-05-19 11:20:38 -07:00
Bryan Bende d4f0c1d048 NIFI-3945 This closes #1829. Adding documentaion about security protocols to Kafka 0.10 processors 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 13:59:27 -04:00
Mark Payne ae9953db64 NIFI-3857: This closes #1825. Added PartitionRecord processor 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-19 02:08:52 -04:00
Mark Payne 9bd0246a96 NIFI-3863: Initial implementation of Lookup Services. Implemented LookupRecord processors. This required some refactoring of RecordSetWriter interface, so refactored that interface and all implementations and references of it 2017-05-19 01:02:41 -04:00
joewitt 3b98abbf41
NIFI-3940 upgraded to latest jruby which resolves an apparent lgpl bunding issue 
This closes #1827.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-05-18 21:37:25 -07:00
joewitt 762293c01c NIFI-3935 made the old persistent prov test more forgiving of timing issues and extended lucene test timeouts for slower environments. These should be integration tests really. 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1823
 2017-05-18 14:42:46 -04:00
joewitt be67f3552b
NIFI-3853 fixing tests to properly encapsulate their environments 
This closes #1822.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-18 13:36:02 -04:00
Koji Kawamura 36e7bd6164 NIFI-3894: This closes #1820. Fixed close and consume order with compression. 
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown as the Inflater is closed before input stream is consumed.

Also, calling close from AbstractTransaction.receive is removed, because the DataPacket is exposed as its return value and this class will not be able to know when to close the stream.

Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-17 22:57:33 -04:00
Jeff Storck 8e1c79eaaf
NIFI-1963 Allows a node reconnecting to the cluster to inherit non-fingerprinted processor settings 
Forces a node reconnecting to a cluster to serialize the updated flow to disk
Added most processor settings to the flow fingerprint (excluding name, style, comment, position, and schedule state)
Updated some test data for FingerprintFactoryTest to test for new fields added to the flow fingerprint
Updated StandardProcessorNode to allow processor comments and name to be settable while a processor is running
Updated StandardFlowSynchronizer to inherit non-fingerprinted processor settings (name, style, comment, and position) when flow is already synchronized
This closes #1812
 2017-05-17 17:35:33 -04:00
Andre F de Miranda 289dde098e
NIFI-3896 - Makes DeprecationNotice more intuitive. This closes #1799 
Update developers guide on how to deprecate a component
 2017-05-17 16:37:11 -04:00
Mark Payne 5cf4bf61b3
NIFI-3917: Use a default value of 'false' for CSV Reader's Skip Header Line property. This closes #1819 2017-05-17 13:31:53 -04:00
Koji Kawamura 3fd7d3b272 NIFI-3873: Improve Avro logical types readability 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1788
 2017-05-17 13:08:00 -04:00
Scott Aslan 2dde480029
[NIFI-3906] Introduce additional validation when handling user-supplied input to ensure appropriate values and output encoding within SlickGrid tables. This closes #1818 2017-05-17 13:04:31 -04:00
Mark Payne 9198bee038
NIFI-3923: Only start processors after repositories have been initialized. This closes #1817 2017-05-17 11:12:59 -04:00
Koji Kawamura 40a9cd4f2e NIFI-3919: Let AvroTypeUtil try every possible type 
Before this fix, AvroTypeUtil can throw an Exception before trying every possible data types defined within a union field.

This closes #1816.
 2017-05-17 10:53:27 -04:00
Koji Kawamura f019d509f3 NIFI-3918: Added Choice mapping to JsonTreeRowRecordReader. 2017-05-17 09:49:11 -04:00
Koji Kawamura 33dc3e36fb NIFI-3920: Remove unnecessary code from AvroTypeUtil 
- Removed remaining duplicate lines of code left by NIFI-3861 refactoring.
- Added test case that writes Avro record having union field.

This closes #1813.
 2017-05-17 09:45:21 -04:00
Matt Gilman c50d516076 NIFI-3848: 
- Allowing the revision check to be overridden when the state of a nodes connection to the cluster changes.

This closes #1811
 2017-05-16 17:09:48 -04:00
Matt Gilman 6b71b4cbb8 NIFI-3868: - Ensuring we do not attempt to group bulletins that lack permissions. - Only group bulletins when all nodes report the same message. - Retain the most recent bulletin. 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1801
 2017-05-16 16:54:30 -04:00
Mark Payne ce1bc42ac5 NIFI-3912: This closes #1809. Fixed NPE that would result in validation failure for FreeFormTextRecordSetWriter 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-16 16:17:04 -04:00
Mark Payne 2f3ba57dd6
NIFI-3914: Do not validate that remote group ports can start transmitting if there is no connection 
This closes #1810.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
 2017-05-16 16:10:00 -04:00
Bryan Bende 3a0004a665
NIFI-3904 Adding logic to only reload when incoming bundle is different and to use additional URLs from spec component 
NIFI-3908 Changing UI to submit filterType for CSs and filter for processors and reporting tasks
 2017-05-16 16:08:28 -04:00
Mark Payne 494a0e8928
NIFI-3900: Ensure that when we serialize a flow to send to Cluster Coordinator that we include the Scheduled State for processors as they are intended to be, not as they are currently because during startup they may not have been started yet. 
NIFI-3900: Rebased against master and updated new unit test to use new method signature for FlowSerializer.serialize

This closes #1804.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-16 16:04:09 -04:00
Matt Gilman ac8e57259f
NIFI-3907 Added response headers. 
Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-05-16 11:55:36 -07:00
Mark Payne fb94990e60 NIFI-3909: This closes #1806. If we have a FlowFile with 0 records, ensure that PublishKafkaRecord_0_10 handles the flowfile properly 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-16 14:42:42 -04:00
Scott Aslan f5f6cab646
NIFI-3906 Added formatting to tables. 
Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-05-16 11:17:29 -07:00
Koji Kawamura 77a676bf92 NIFI-3894: Call Inflater/Deflater.end to free up memory 
This closes #1796.
 2017-05-16 13:49:31 -04:00
Mark Payne 4fdea680ec NIFI-3905: This closes #1805. When a Provenance Query is submitted to WriteAheadProvenanceRepository, purge any obsolete queries from the internal state before rejecting the query due to 'too many outstanding queries' 
Signed-off-by: joewitt <joewitt@apache.org>
 2017-05-16 13:48:16 -04:00
Aldrin Piri 58cf15a912 NIFI-2199 NIFI-3112 Pipe stdout to /dev/null and allow stderr to propagate through. 
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1786
 2017-05-16 13:31:15 -04:00
Mark Payne 3f4b276b71 NIFI-3910: Fixed issue where CSVRecordReader throws an Exception if a field is missing instead of using a null value 
This closes #1807.
 2017-05-16 13:16:46 -04:00
Matt Gilman 824712bffe
NIFI-3901: 
- Addressing response code of POST /provenance/lineage.
- Ensuring cluster node details are set accordingly.

This closes #1803.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
 2017-05-16 12:36:27 -04:00
Mark Payne 603f713a40 NIFI-3860: Relax the constraint that the ProcessSession must always be given the most up-to-date version of a FlowFile 
This closes #1778.
 2017-05-16 08:51:41 -04:00
Matt Gilman 6ffb78d404
NIFI-3853: 
- Filtering out certain control characters and unpaired Unicode surrogate codepoints prior to saving the flow.xml.

This closes #1784.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-05-15 19:02:03 -07:00
Peter G. Horvath bf15502e19 NIFI-3763 Add new processor to log user defined messages built with NiFi Expression Language 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1737
 2017-05-15 14:18:57 -04:00
Koji Kawamura 1811ba5681 NIFI-2624: Avro logical types for ExecuteSQL and QueryDatabaseTable 
- Added Logical type support for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP column types.
- Added Logical type 'decimal' to AvroReader so that Avro records with logical types written by ExecuteSQL and QueryDatabaseTable can be consumed by AvroReader.
- Added JdbcCommon.AvroConversionOptions to consolidate conversion options.
- Added 'Use Avro Logical Types' property to ExecuteSQL and QueryDatabaseTable to toggle whether to use Logical types.
- Added 'mime.type' FlowFile attribute as 'application/avro-binary' so that output FlowFiles can be displayed by content viewer.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1798
 2017-05-15 14:15:23 -04:00
Tim Reardon 20a1fc24d7 NIFI-3885 DynamoDB Processor EL Support 
Add EL support to remaining Dynamo processor properties

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1793.
 2017-05-15 10:13:53 -07:00
Tim Reardon 196ca237e6 NIFI-3855 Support PutSlack attachments as dynamic properties 
NIFI-3855 Refactor deprecated API

NIFI-3855 Log separate error messages in PutSlack

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1774
 2017-05-15 09:08:43 -04:00
Andre F de Miranda d092551211
NIFI-3761 
- adjust testFullyDocumentedProcessor to correctly eval to false when needed
- Introduce the ability to deprecate a component
- Allow documentation to inform user about deprecation of a component
- This closes #1718

NIFI-391 - Add set/getDeprecationReson to DocumentedDTO and use it within DtoFactory'
 2017-05-15 08:43:32 -04:00
Matt Burgess 3353865ce9 NIFI-3867: Add Expression Language support to HiveConnectionPool properties 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1783.
 2017-05-15 13:46:29 +02:00
Koji Kawamura bc68eb754f NIFI-3895: Fixed AvroReader to use Schema Access Strategy default value properly. 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1797.
 2017-05-15 13:27:26 +02:00
Scott Aslan 09f6a7040c
[NIFI-3830] force browsers to use URLSearchParams polyfill do to bugs and inconsistent browser implementations. This closes #1785 2017-05-13 07:34:33 -04:00
Steve Champagne 382eef2183 NIFI-3879: Allow null Avro default values 
- Avro uses their own class for null values, so a standard check against
  null isn't picking them up.

This closes #1792.
 2017-05-12 14:58:43 -04:00
Tim Reardon e9848f4276 NIFI-3881 Fix PutHiveStreaming EL evaluation 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1791
 2017-05-12 14:06:32 -04:00
Tim Reardon afd2b04afd NIFI-3852 Add expression language support to Cassandra processors 
Remove unused import

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1770
 2017-05-12 13:52:07 -04:00
Mark Payne b1901d5fe0 NIFI-3838: Initial implementation of RecordPath and UpdateRecord processor 
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum

NIFI-3838: Addressed PR Review feedback

NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests

NIFI-3838: Addressed problems with documentation based on PR Review

NIFI-3838: Fixed checkstyle violation

NIFI-3838: Addressed issue of comparing different types of Number objects

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1772
 2017-05-12 12:36:52 -04:00
Scott Aslan 1c58e78ce9 [NIFI-3876] trim whitespace processor names before applying ellipsis 
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1789.
 2017-05-12 16:01:43 +02:00
Matt Gilman dd0306cce7 NIFI-3837: - Being more specific regarding the escaping of back references in evaluated expressions. 
Signed-off-by: Matt Burgess <mattyb149@apache.org>

fixed typo in comment

This closes #1781
 2017-05-11 22:16:46 -04:00
Koji Kawamura 0b0ac196ea NIFI-3861: AvroTypeUtil used different constant. 
Previous fix #1779 refactored the way to check Logical type to use string constants.
One of those refactoring used wrong constant mistakenly in normalizeValue method.

Fortunately, this defect is harmless since even though normalizeValue did not convert int to Time, DataTypeUtils.convertType does the same conversion.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1782
 2017-05-11 21:31:07 -04:00
Steve Champagne 6e4db6b11a NIFI-3871: Convert Avro map values 
This closes #1787.

- When converting from a raw value to an Avro object, convert the values
  of any Avro map types so that they can be complex types like other
  records.
 2017-05-11 20:25:19 -04:00
Matt Gilman cf497c9cf9
NIFI-3865: - Obtaining the appropriate input stream after accessing content. The stream was being consumed into a buffer and was not longer available. 
This closes #1780.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-10 15:51:00 -04:00
Tim Reardon dc44e30698 NIFI-3856 PutCloudWatchMetric Stats, Dimensions 
PutCloudWatchMetric support for StatisticSets and Dimensions

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1775.
 2017-05-10 12:07:04 -07:00
Tim Reardon 361a58c531 NIFI-3851 Support EL in AWS Endpoint Override URL 
Add expression language support to AWS Endpoint Override URL property

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1769.
 2017-05-10 10:37:39 -07:00
Koji Kawamura 72de1cbdef NIFI-3861: Fixed AvroReader nullable logical types issue 
- AvroReader did not convert logical types if those are defined with union
- Consolidated createSchema method in AvroSchemaRegistry and AvroTypeUtil as both has identical implementation and mai
ntaining both would be error-prone

This closes #1779.
 2017-05-10 10:08:22 -04:00
Koji Kawamura 744ecc3d83 NIFI-3832: Fixed AvroRecordSetWriter validation NPE. 
This closes #1768.
 2017-05-09 14:11:43 -04:00
Bryan Bende 3af53419af
NIFI-3770-RC2 prepare for next development iteration 2017-05-05 20:50:28 -04:00
Bryan Bende 3a605af8e0
NIFI-3770-RC2 prepare release nifi-1.2.0-RC2 2017-05-05 20:50:14 -04:00
joewitt f4f174b899
NIFI-3826 added proper NOTICE entries for apache calcite and maxmind source 
This closes #1766.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-05 20:00:22 -04:00
joewitt 829c02b058 NIFI-3825 fix L&N for jolt, json, and source deps 
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1765
 2017-05-05 18:43:04 -04:00
Jeremy Dyer 5f65b2561a NIFI-3600 Improve logging and relationship routing for failures in DeleteHDFS 
realized that the session should be cloned here because its inside a for loop and the original flow file would be transferred but not be the latest flow file if an error occurred in the for loop

@trixpan at a high level what do you think about this approach?

NIFI-3600: Added unit test

NIFI-3600: Removed the hdfs.error.code attribute
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1595
 2017-05-05 15:25:08 -04:00
Matt Gilman f0dfcc180d NIFI-3808: 
- Making the selector for the enable/disable controller service state more specific. Previously name was too general and was affect other parts of the application.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1759
 2017-05-05 15:19:28 -04:00
joewitt ee8dc7640f
NIFI-3820 added calcite to assembly notice and updated all copyright years for compress, avro, and jetty 
This closes #1764.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-05 14:45:40 -04:00
Mark Payne 12249e5a08
NIFI-2268: Instead of allowing HandleHttpRequest to run constantly calling BlockingQueue.poll() as fast as possible, call BlockingQueue.poll(2, TimeUnit.MILLISECONDS) to avoid overutilization of CPU when there is no work to be done 
This closes #1762.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-05 13:58:10 -04:00
Matt Gilman e6be5d3276
NIFI-3801: 
- Making the version text next to the component name more subtle.

This closes #1760.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
 2017-05-05 13:50:21 -04:00
Koji Kawamura af6f63691c
NIFI-3818: PutHiveStreaming throws IllegalStateException 
Changed from async append to sync as it breaks 'recursionSet' check in StandardProcessSession by updating it from multiple threads, resulting IllegalStateException to happen.

This closes #1761.

Signed-off-by: Bryan Bende <bbende@apache.org>
 2017-05-05 13:25:59 -04:00
Scott Aslan 85405dae15
[NIFI-3813] fix broken PG config dialog when user has read only perms on the PG. This closes #1757 2017-05-05 12:37:56 -04:00
Bryan Bende d9410d6404 NIFI-3795 Moved no-git profile to nifi-nar-bundles 
Removing buildnumber plugin and build-info-no-git profile from nifi-assembly
and adding build-info-no-git profile to nifi-nar-bundles pom

Signed-off-by: James Wing <jvwing@gmail.com>
 2017-05-04 10:09:25 -07:00
Andy LoPresto 4f40eca16c
NIFI-3788 Switched Amazon HTTP client instantiation from using null HostnameVerifier (which defaulted to Strict, which cannot handle wildcard certificate hostnames) to DefaultHostnameVerifier, which is fine. 
I still want to add unit tests and integration tests, but I ran a flow which had previously caused the reproducible exception and this worked fine (flow showed objects were put in S3, no exceptions, and I verified through AWS Web Console that new objects were present).

This closes #1753.

Signed-off-by: Bryan Rosander <brosander@apache.org>
 2017-05-04 12:09:52 -04:00