Commit Graph

4172 Commits

Author SHA1 Message Date
Pierre Villard 6559604456 NIFI-4081 - Added raw message option in GrokReader
This closes #1921.
2017-09-08 12:44:37 -04:00
Koji Kawamura 1f67cbf628 NIFI-4004: Use RecordReaderFactory without FlowFile.
- Removed FlowFile from RecordReaderFactory, RecordSetWriterFactory and SchemaAccessStrategy.
- Renamed variable 'allowableValue' to 'strategy' to represent its meaning better.
- Removed creation of temporal FlowFile to resolve Record Schema from ConsumerLease.

- Removed unnecessary 'InputStream content' argument from
  RecordSetWriterFactory.getSchema method.

This closes #1877.
2017-09-08 12:37:40 -04:00
yuri1969 20d23e836e
NIFI-3941 - Clarify tab name for controller-level controller services dialog
* Changed the tab title since sharing the name makes things
less clear for newcomers.
* Suggested info sentence is omitted.

This closes #2124.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-09-07 18:29:13 -07:00
m-hogue 03e51ee8ac
NIFI-4335: Changed SSLContextService implementations to RestrictedSSLContextService for all Listen* processors
This closes #2131.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-09-07 17:34:00 -07:00
Pierre Villard c10ff574c4 NIFI-4257 - add custom WHERE clause in database fetch processors
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2050
2017-09-07 14:41:26 -04:00
patricker ae30c7f350 NIFI-3484: GenerateTableFetch Should Allow for Right Boundary
fix checkstyle issue, and added unit test showing data duplication issue, removed property

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2091
2017-09-07 11:05:38 -04:00
Mike Thomsen 527ce0b4ef NIFI-4269 Added the ability to serialize Mongo documents to a clean JSON view instead of just extended JSON.
NIFI-4269 incorporated changes from the code review.

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2063
2017-09-06 15:56:12 -04:00
Pierre Villard 9ac88d210a NIFI-4342 - Add EL support to PutHiveStreaming
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2120
2017-09-06 13:33:18 -04:00
Pierre Villard 458c987fe3 NIFI-4355 - query execution time as attribute of ExecuteSQL
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2129
2017-09-06 13:29:19 -04:00
Andy LoPresto 0536c3edf1
NIFI-4353 Implemented new JAXB logic.
Added unit test and template test resource.
Added RAT exclusion.
This closes #2128
2017-09-06 13:13:24 -04:00
Pierre Villard c3754c3921 NIFI-4320 - Escape quotes for JSON output in QueryCassandra
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2114
2017-09-06 13:05:18 -04:00
Mark Payne bfd6c0aef7 NIFI-4116: Allow fields of Record returned from Lookup Service to be placed into record in the input, instead of requiring that the 'wrapper record' returned from Lookup be included
Signed-off-by: James Wing <jvwing@gmail.com>

This closes #2110.
2017-09-05 21:29:16 -07:00
Matthew Burgess 20a6374bf7 NIFI-4334: Added Apache Commons Net to scripting NAR
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2117.
2017-08-31 09:22:22 +02:00
m-hogue d28e61c5dd
NIFI-2528 Added RestrictedSSLContextService interface with implementation. Changed ListenHTTP to allow only new StandardRestrictedSSLContextService.
This closes #1986.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-08-30 20:17:01 -07:00
Pierre Villard e62417ea6b NIFI-1923 - AttributesToJson regex property
added EL support

Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2099
2017-08-30 14:01:27 -04:00
Pierre Villard a53a37f9ca NIFI-4319 - Fixed ArrayIndexOutOfBoundsException in QueryCassandra
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2112
2017-08-30 13:52:05 -04:00
Matt Gilman 3de0b8edff NIFI-4224:
- Ensuring the parent permissions are used when building the variable registry for a process group ancestry. This closes #2116.
2017-08-30 11:54:31 -04:00
Scott Aslan e2b8be53cb
[NIFI-4327] Parameterize node and npm in poms. This closes #2115 2017-08-28 15:10:24 -04:00
Koji Kawamura e68ff153e8
NIFI-3332: ListXXX to not miss files with the latest processed timestamp
Before this fix, it's possible that ListXXX processors can miss files those have the same timestamp as the one which was the latest processed timestamp at the previous cycle. Since it only used timestamps, it was not possible to determine whether a file is already processed or not.

However, storing every single processed identifier as we used to will not perform well.
Instead, this commit makes ListXXX to store only identifiers those have the latest timestamp at a cycle to minimize the amount of state data to store.

NIFI-3332: ListXXX to not miss files with the latest processed timestamp

- Fixed TestAbstractListProcessor to use appropriate time precision.
  Without this fix, arbitrary test can fail if generated timestamp does
  not have the desired time unit value, e.g. generated '10:51:00' where
  second precision is tested.
- Fixed TestFTP.basicFileList to use millisecond time precision explicitly
  because FakeFtpServer's time precision is in minutes.
- Changed junit dependency scope to 'provided' as it is needed by
  ListProcessorTestWatcher which is shared among different modules.

This closes #1975.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-28 11:31:04 -04:00
Koji Kawamura 28ee70222b
NIFI-4069: Make ListXXX work with timestamp precision in seconds or minutes
- Refactored variable names to better represents what those are meant for.
- Added deterministic logic which detects target filesystem timestamp precision and adjust lag time based on it.
- Changed from using System.nanoTime() to System.currentTimeMillis in test because Java File API reports timestamp in milliseconds at the best granularity. Also, System.nanoTime should not be used in mix with epoch milliseconds because it uses arbitrary origin and measured differently.
- Changed TestListFile to use more longer interval between file timestamps those are used by testFilterAge to provide more consistent test result because sleep time can be longer with filesystems whose timestamp in seconds precision.
- Added logging at TestListFile.
- Added TestWatcher to dump state in case assertion fails for further investigation.
- Added Timestamp Precision property so that user can set if auto-detect is not enough
- Adjust timestamps for ages test

This closes #1915.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-28 11:31:03 -04:00
Matt Gilman a4e729c7a7 NIFI-4324:
- Ensuring that sub context menus are removed when hiding to ensure they are correctly (re)created during mouseenter events.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2109
2017-08-25 16:20:32 -04:00
cam e3da44fb62 [NiFi-3973] Add PutKudu processor to ingest data to Kudu 2017-08-25 10:53:31 -04:00
Yolanda M. Davis 0eda71a9a6 NIFI-4310 - added changes to support detection of reporting tasks and controller services during isEmpty flow check. Added testing scenarios. This closes #2107. 2017-08-25 10:15:55 -04:00
yuri1969 6c426f7a12 NIFI-2715 - Match style to 'detail-value'
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2100
2017-08-24 18:39:12 -04:00
yuri1969 a9ee2aa5e4 NIFI-2715 - Keep 'Content Claim' header visible 2017-08-24 18:38:29 -04:00
yuri1969 a7ea178cdc NIFI-2715 - When performing a Queue Listing, if...
...FlowFile has no content, UI does not indicate this fact

For FlowFiles with no content:
* Added a simple title replacing the Content Claim's one.
* Defaulted File Size to 0 bytes using common formatting.
2017-08-24 18:38:29 -04:00
Yolanda M. Davis acee2627ca NIFI-4255 - Added flag to allow migration of existing (source) acls to destination, update to documentation
Added build-helper-maven-plugin to nifi-toolkit-zookeeper-migrator pom to build groovy test code
Moved spock spec from src/test/java to src/test/groovy
Minor code formatting updates to ZooKeeperMigrator.java

This closes #2065
2017-08-23 15:38:44 -04:00
Bryan Bende cf57639396 NIFI-4311 Allowing umask to get set properly before initializing the FileSystem
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2106.
2017-08-22 22:40:26 +02:00
yuri1969 05700a2008 NIFI-3781 - Change tooltip for disabled state
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2102
2017-08-21 12:56:19 -04:00
yuri1969 a3401c3b95 NIFI-3781 - Controller Service Config Dialog...
..'View Details' button feels confusing and inconsistent with Processors

* Merged 'View Details' and 'Edit' buttons to a single 'Configure'/'View Details' one.
* Fixed confusing 'View Details' modal's title.
2017-08-21 12:56:12 -04:00
mans2singh 8cb501443b NIFI-4250 - Elasticsearch 5 delete processor
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #2045
2017-08-21 08:49:57 -04:00
Pierre Villard 41984bed13 NIFI-1835 - added text/xml to content viewer
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2098
2017-08-18 12:48:49 -04:00
Pierre Villard 0bcb19771f NIFI-1670 - typos/details in EL doc
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2097
2017-08-18 09:08:46 -04:00
Bryan Bende 60d4672195 NIFI-4304 Extracting HWX Schema Registry client version to a property and bumping to latest 0.3.0 release
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2096
2017-08-17 16:07:03 -04:00
Matt Burgess 6b5015e39b
NIFI-4218: Dynamic properties as query parameters in ESHttp processors
This closes #2049.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-17 14:29:05 -04:00
Mark Payne 69a08e78c2
NIFI-4295:
- When determining which controller services to return for a component, ensure that we don't show services that belong to 'child groups'
- Fixed a logic bug that determined which process group to use for obtaining controller services
- This closes #2087
2017-08-17 12:47:46 -04:00
Mark Payne 5cd8e93beb
NIFI-4224:
- Initial implementation of Process Group level Variable Registry
- Updated to incorporate PR Feedback
- Changed log message because slf4j-simple apparently has a memory leak; passing a String instead of passing in the Controller Service object as an argument addresses this.
- This closes #2051
2017-08-17 10:42:29 -04:00
Pierre Villard c1b99d584d NIFI-4028 - fix cache update when Wait releases flow files
NIFI-4028: Refactored Wait processor.

- Consolidated implementation for the cases of releasableFlowCount is 1 or more, in order to reduce complexity and behavior differences
- Added 'consumed' counter when total counter is used to release incoming FlowFiles
- Fixed method name typo, releaseCandidates

This closes #2055.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-08-17 08:30:07 +09:00
Pierre Villard 9f23ec360a NIFI-3931 - Added EL to properties in SFTP transfer
The properties are already evaluated against EL. Also added an
integration test that is failing without the change.

This closes #1968.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-08-17 08:20:30 +09:00
Arun Manivannan 6df112e4b1 NIFI-4062 Provide an option to disable DTD validation for EvaluateXPath and EvaluateXQuery
This closes #2093.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-08-17 08:07:37 +09:00
yuri1969 02c05bc203 NIFI-3780 - Should be able to specify Service...
...name when I create a new Controller Service within a Processor/Service configuration dialog

Changed CS name default's behavior according to PR feedback.

Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2081
2017-08-16 11:56:23 -04:00
yuri1969 79300de7a9 NIFI-3780 - Should be able to specify Service...
...name when I create a new Controller Service within a Processor/Service configuration dialog

I'm not sure how to integrate/style the text box. Suggestions are welcome.
2017-08-16 10:31:10 -04:00
Mark Payne abca9d1464
NIFI-4208: Fixed bug in SchemaRepositoryRecordSerde that would return null from deserializeEdit if there was no data; the interface documents that null cannot be returned from this method, as it is called only when data is expected to exist. As a result, if there is no data, we should throw EOFException instead, and the write-ahead log will handle that appropriately.
This closes #2086.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2017-08-15 12:10:43 -04:00
Pierre Villard a1706d12f5 NIFI-3281 - fix for (S)FTP processors when using EL against FFs
NIFI-3281 - Review - handle completePendingCommand return and added a unit test for ListFTP

NIFI-3281 - Review - Added flow file for EL evaluation in other methods and added unit test for NIFI-3590

This closes #1974.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2017-08-15 11:24:48 +09:00
Matt Burgess 50f22162b0
NIFI-4286: Fix NPE in GenerateTableFetch when Partition Size = 0
This closes #2078.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-08-14 15:28:07 -04:00
Pierre Villard 8aa4450084 NIFI-779 - fix for ReportingContext.getProperty(PropertyDescriptor)
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #2033
2017-08-14 13:41:10 -04:00
Andrew Lim bac175a00f NIFI-4276 Add Write Ahead Provenance section to User Guide
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #2074
2017-08-13 10:01:36 -04:00
Mark Payne 451f9cf124 NIFI-4142: This closes #2015. Refactored Record Reader/Writer to allow for reading/writing "raw records". Implemented ValidateRecord. Updated Record Reader to take two parameters for nextRecord: (boolean coerceTypes) and (boolean dropUnknownFields)
Signed-off-by: joewitt <joewitt@apache.org>
2017-08-11 22:01:46 -07:00
Mark Payne 84935d4f78 NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was aging off the wrong index. When it should age off Index 1, it was removing Index 2. As a result, the earliest index is never aged off, and the newest index could potentially be aged off before it is ready to be. Also addressed issue where a query that attempts to read an event that has aged off throws FileNotFoundException (NIFI-4077) instead of skipping over the event. The JavaDocs indicate that the EventIterator should skip records that it cannot find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead
Signed-off-by: joewitt <joewitt@apache.org>
2017-08-11 22:01:46 -07:00
Andy LoPresto ae940d8624 NIFI-4237 Added working test for StringEncryptor decryption of sensitive flow values in FlowFromDOMFactory.
NIFI-4237 Cleaned up unused alternate approaches.

NIFI-4237 Added failing unit test for better error message.

NIFI-4237 Added logic to capture unhelpful encryption exception and provide context in message. All tests pass.

This closes #2077
2017-08-11 23:57:15 -04:00