Commit Graph

1917 Commits

Author SHA1 Message Date
Matt Burgess 9294a26139
NIFI-3938: Added ScriptedLookupService, some refactor for reusable scripting classes
This closes #1828.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-05-19 11:20:38 -07:00
Bryan Bende d4f0c1d048 NIFI-3945 This closes #1829. Adding documentaion about security protocols to Kafka 0.10 processors
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-19 13:59:27 -04:00
Mark Payne ae9953db64 NIFI-3857: This closes #1825. Added PartitionRecord processor
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-19 02:08:52 -04:00
Mark Payne 9bd0246a96 NIFI-3863: Initial implementation of Lookup Services. Implemented LookupRecord processors. This required some refactoring of RecordSetWriter interface, so refactored that interface and all implementations and references of it 2017-05-19 01:02:41 -04:00
joewitt 3b98abbf41
NIFI-3940 upgraded to latest jruby which resolves an apparent lgpl bunding issue
This closes #1827.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-05-18 21:37:25 -07:00
joewitt 762293c01c NIFI-3935 made the old persistent prov test more forgiving of timing issues and extended lucene test timeouts for slower environments. These should be integration tests really.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1823
2017-05-18 14:42:46 -04:00
joewitt be67f3552b
NIFI-3853 fixing tests to properly encapsulate their environments
This closes #1822.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-18 13:36:02 -04:00
Koji Kawamura 36e7bd6164 NIFI-3894: This closes #1820. Fixed close and consume order with compression.
Before this fix, 'NullPointerException: Inflater has been closed' can be thrown as the Inflater is closed before input stream is consumed.

Also, calling close from AbstractTransaction.receive is removed, because the DataPacket is exposed as its return value and this class will not be able to know when to close the stream.

Signed-off-by: joewitt <joewitt@apache.org>
2017-05-17 22:57:33 -04:00
Jeff Storck 8e1c79eaaf
NIFI-1963 Allows a node reconnecting to the cluster to inherit non-fingerprinted processor settings
Forces a node reconnecting to a cluster to serialize the updated flow to disk
Added most processor settings to the flow fingerprint (excluding name, style, comment, position, and schedule state)
Updated some test data for FingerprintFactoryTest to test for new fields added to the flow fingerprint
Updated StandardProcessorNode to allow processor comments and name to be settable while a processor is running
Updated StandardFlowSynchronizer to inherit non-fingerprinted processor settings (name, style, comment, and position) when flow is already synchronized
This closes #1812
2017-05-17 17:35:33 -04:00
Andre F de Miranda 289dde098e
NIFI-3896 - Makes DeprecationNotice more intuitive. This closes #1799
Update developers guide on how to deprecate a component
2017-05-17 16:37:11 -04:00
Mark Payne 5cf4bf61b3
NIFI-3917: Use a default value of 'false' for CSV Reader's Skip Header Line property. This closes #1819 2017-05-17 13:31:53 -04:00
Koji Kawamura 3fd7d3b272 NIFI-3873: Improve Avro logical types readability
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1788
2017-05-17 13:08:00 -04:00
Scott Aslan 2dde480029
[NIFI-3906] Introduce additional validation when handling user-supplied input to ensure appropriate values and output encoding within SlickGrid tables. This closes #1818 2017-05-17 13:04:31 -04:00
Mark Payne 9198bee038
NIFI-3923: Only start processors after repositories have been initialized. This closes #1817 2017-05-17 11:12:59 -04:00
Koji Kawamura 40a9cd4f2e NIFI-3919: Let AvroTypeUtil try every possible type
Before this fix, AvroTypeUtil can throw an Exception before trying every possible data types defined within a union field.

This closes #1816.
2017-05-17 10:53:27 -04:00
Koji Kawamura f019d509f3 NIFI-3918: Added Choice mapping to JsonTreeRowRecordReader. 2017-05-17 09:49:11 -04:00
Koji Kawamura 33dc3e36fb NIFI-3920: Remove unnecessary code from AvroTypeUtil
- Removed remaining duplicate lines of code left by NIFI-3861 refactoring.
- Added test case that writes Avro record having union field.

This closes #1813.
2017-05-17 09:45:21 -04:00
Matt Gilman c50d516076 NIFI-3848:
- Allowing the revision check to be overridden when the state of a nodes connection to the cluster changes.

This closes #1811
2017-05-16 17:09:48 -04:00
Matt Gilman 6b71b4cbb8 NIFI-3868: - Ensuring we do not attempt to group bulletins that lack permissions. - Only group bulletins when all nodes report the same message. - Retain the most recent bulletin.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1801
2017-05-16 16:54:30 -04:00
Mark Payne ce1bc42ac5 NIFI-3912: This closes #1809. Fixed NPE that would result in validation failure for FreeFormTextRecordSetWriter
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-16 16:17:04 -04:00
Mark Payne 2f3ba57dd6
NIFI-3914: Do not validate that remote group ports can start transmitting if there is no connection
This closes #1810.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-05-16 16:10:00 -04:00
Bryan Bende 3a0004a665
NIFI-3904 Adding logic to only reload when incoming bundle is different and to use additional URLs from spec component
NIFI-3908 Changing UI to submit filterType for CSs and filter for processors and reporting tasks
2017-05-16 16:08:28 -04:00
Mark Payne 494a0e8928
NIFI-3900: Ensure that when we serialize a flow to send to Cluster Coordinator that we include the Scheduled State for processors as they are intended to be, not as they are currently because during startup they may not have been started yet.
NIFI-3900: Rebased against master and updated new unit test to use new method signature for FlowSerializer.serialize

This closes #1804.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-16 16:04:09 -04:00
Matt Gilman ac8e57259f
NIFI-3907 Added response headers.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-05-16 11:55:36 -07:00
Mark Payne fb94990e60 NIFI-3909: This closes #1806. If we have a FlowFile with 0 records, ensure that PublishKafkaRecord_0_10 handles the flowfile properly
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-16 14:42:42 -04:00
Scott Aslan f5f6cab646
NIFI-3906 Added formatting to tables.
Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-05-16 11:17:29 -07:00
Koji Kawamura 77a676bf92 NIFI-3894: Call Inflater/Deflater.end to free up memory
This closes #1796.
2017-05-16 13:49:31 -04:00
Mark Payne 4fdea680ec NIFI-3905: This closes #1805. When a Provenance Query is submitted to WriteAheadProvenanceRepository, purge any obsolete queries from the internal state before rejecting the query due to 'too many outstanding queries'
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-16 13:48:16 -04:00
Aldrin Piri 58cf15a912 NIFI-2199 NIFI-3112 Pipe stdout to /dev/null and allow stderr to propagate through.
Signed-off-by: Yolanda M. Davis <ymdavis@apache.org>

This closes #1786
2017-05-16 13:31:15 -04:00
Mark Payne 3f4b276b71 NIFI-3910: Fixed issue where CSVRecordReader throws an Exception if a field is missing instead of using a null value
This closes #1807.
2017-05-16 13:16:46 -04:00
Matt Gilman 824712bffe
NIFI-3901:
- Addressing response code of POST /provenance/lineage.
- Ensuring cluster node details are set accordingly.

This closes #1803.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2017-05-16 12:36:27 -04:00
Mark Payne 603f713a40 NIFI-3860: Relax the constraint that the ProcessSession must always be given the most up-to-date version of a FlowFile
This closes #1778.
2017-05-16 08:51:41 -04:00
Matt Gilman 6ffb78d404
NIFI-3853:
- Filtering out certain control characters and unpaired Unicode surrogate codepoints prior to saving the flow.xml.

This closes #1784.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-05-15 19:02:03 -07:00
Peter G. Horvath bf15502e19 NIFI-3763 Add new processor to log user defined messages built with NiFi Expression Language
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1737
2017-05-15 14:18:57 -04:00
Koji Kawamura 1811ba5681 NIFI-2624: Avro logical types for ExecuteSQL and QueryDatabaseTable
- Added Logical type support for DECIMAL/NUMBER, DATE, TIME and TIMESTAMP column types.
- Added Logical type 'decimal' to AvroReader so that Avro records with logical types written by ExecuteSQL and QueryDatabaseTable can be consumed by AvroReader.
- Added JdbcCommon.AvroConversionOptions to consolidate conversion options.
- Added 'Use Avro Logical Types' property to ExecuteSQL and QueryDatabaseTable to toggle whether to use Logical types.
- Added 'mime.type' FlowFile attribute as 'application/avro-binary' so that output FlowFiles can be displayed by content viewer.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1798
2017-05-15 14:15:23 -04:00
Tim Reardon 20a1fc24d7 NIFI-3885 DynamoDB Processor EL Support
Add EL support to remaining Dynamo processor properties

Signed-off-by: James Wing <jvwing@gmail.com>

This closes #1793.
2017-05-15 10:13:53 -07:00
Tim Reardon 196ca237e6 NIFI-3855 Support PutSlack attachments as dynamic properties
NIFI-3855 Refactor deprecated API

NIFI-3855 Log separate error messages in PutSlack

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1774
2017-05-15 09:08:43 -04:00
Andre F de Miranda d092551211
NIFI-3761
- adjust testFullyDocumentedProcessor to correctly eval to false when needed
- Introduce the ability to deprecate a component
- Allow documentation to inform user about deprecation of a component
- This closes #1718

NIFI-391 - Add set/getDeprecationReson to DocumentedDTO and use it within DtoFactory'
2017-05-15 08:43:32 -04:00
Matt Burgess 3353865ce9 NIFI-3867: Add Expression Language support to HiveConnectionPool properties
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1783.
2017-05-15 13:46:29 +02:00
Koji Kawamura bc68eb754f NIFI-3895: Fixed AvroReader to use Schema Access Strategy default value properly.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1797.
2017-05-15 13:27:26 +02:00
Scott Aslan 09f6a7040c
[NIFI-3830] force browsers to use URLSearchParams polyfill do to bugs and inconsistent browser implementations. This closes #1785 2017-05-13 07:34:33 -04:00
Steve Champagne 382eef2183 NIFI-3879: Allow null Avro default values
- Avro uses their own class for null values, so a standard check against
  null isn't picking them up.

This closes #1792.
2017-05-12 14:58:43 -04:00
Tim Reardon e9848f4276 NIFI-3881 Fix PutHiveStreaming EL evaluation
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1791
2017-05-12 14:06:32 -04:00
Tim Reardon afd2b04afd NIFI-3852 Add expression language support to Cassandra processors
Remove unused import

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1770
2017-05-12 13:52:07 -04:00
Mark Payne b1901d5fe0 NIFI-3838: Initial implementation of RecordPath and UpdateRecord processor
NIFI-3838: Updated version from 1.2.0-SNAPSHOT to 1.3.0-SNAPSHOT; removed unneeded value from AttributeExpression.ResultType enum

NIFI-3838: Addressed PR Review feedback

NIFI-3838: Allow for schemas to be merged together for a record; refactored RecordSetWriterFactory so that there is a method to obtain the schema and then the writer is created with that schema. Added additional unit tests

NIFI-3838: Addressed problems with documentation based on PR Review

NIFI-3838: Fixed checkstyle violation

NIFI-3838: Addressed issue of comparing different types of Number objects

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1772
2017-05-12 12:36:52 -04:00
Scott Aslan 1c58e78ce9 [NIFI-3876] trim whitespace processor names before applying ellipsis
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1789.
2017-05-12 16:01:43 +02:00
Matt Gilman dd0306cce7 NIFI-3837: - Being more specific regarding the escaping of back references in evaluated expressions.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

fixed typo in comment

This closes #1781
2017-05-11 22:16:46 -04:00
Koji Kawamura 0b0ac196ea NIFI-3861: AvroTypeUtil used different constant.
Previous fix #1779 refactored the way to check Logical type to use string constants.
One of those refactoring used wrong constant mistakenly in normalizeValue method.

Fortunately, this defect is harmless since even though normalizeValue did not convert int to Time, DataTypeUtils.convertType does the same conversion.

Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1782
2017-05-11 21:31:07 -04:00
Steve Champagne 6e4db6b11a NIFI-3871: Convert Avro map values
This closes #1787.

- When converting from a raw value to an Avro object, convert the values
  of any Avro map types so that they can be complex types like other
  records.
2017-05-11 20:25:19 -04:00
Matt Gilman cf497c9cf9
NIFI-3865: - Obtaining the appropriate input stream after accessing content. The stream was being consumed into a buffer and was not longer available.
This closes #1780.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-05-10 15:51:00 -04:00