125 Commits

Author SHA1 Message Date
Mark Payne
91b2283f49
NIFI-6183: Implemented base class for Framework-level integration tests that is responsible for instantiating the key components for integration tests, including FlowController, WriteAheadFlowFileRepository, WriteAheadProvenanceRepository, FileSystemRepository, FileSystemSwapManager, StandardFlowFileQueue, StandardProcessScheduler, etc. Implemented several integration tests to span different parts of the framework
This closes #3423.

Signed-off-by: Bryan Bende <bbende@apache.org>
2019-04-10 16:47:08 -04:00
Joe Witt
1bdab73201 NIFI-6121 Updated the easy dependencies to update
This closes #3374

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-03-22 09:01:21 -04:00
joewitt
25cc7b4a1e
NIFI-6029 merging nifi 1.9.0 release into master 2019-02-19 22:55:49 -05:00
Denes Arvay
76e92c8682 NIFI-6052 Update NOTICE files to reflect 2019
This closes #3319

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2019-02-19 18:32:15 -05:00
joewitt
0e204f3576
NIFI-6029-RC2 prepare for next development iteration 2019-02-16 21:50:35 -05:00
joewitt
45bb53d2aa
NIFI-6029-RC2 prepare release nifi-1.9.0-RC2 2019-02-16 21:50:15 -05:00
Jeff Storck
c0182294ed NIFI-5720-RC3 prepare for next development iteration 2018-10-22 22:16:43 -04:00
Jeff Storck
98aabf2c50 NIFI-5720-RC3 prepare release nifi-1.8.0-RC3 2018-10-22 22:16:23 -04:00
joewitt
8e233ca2ef
NIFI-4806 updated tika and a ton of other deps as found by dependency versions plugin
This closes #3028
2018-10-04 09:32:04 -04:00
Mark Payne
aac2c6a60e
NIFI-5482: Made WriteAheadProvenanceRepository the default implementation
This closes #2960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-23 16:49:55 -07:00
Andy LoPresto
f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
Andy LoPresto
d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto
99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Matt Gilman
fe31a06fdc
NIFI-4907:
- Minor adjustments following PR.
- Avoiding additional find operation when authorizing components when populating component details.
- Requiring access to provenance events when downloading content or submitting a replay as they may provide events details.
- Updating the REST API docs detailing the required permissions.
- Updating the wording in the documentation regarding the provenance and data policies.
- Removed the event attributes from the authorization calls that were verifying access to provenance events.
- Only checking content availability when the user is authorized for the components data.
- Addressing typo in JavaDoc.

This closes #2703
2018-06-14 12:21:47 -04:00
Mark Bean
e27798797a
NIFI-4907: add 'view provenance' component policy
whitespace removed for checkstyle
2018-06-14 10:59:17 -04:00
joewitt
d511fe3e4b NIFI-4995-RC3 prepare for next development iteration 2018-04-03 08:28:34 -07:00
joewitt
f8466cb16d NIFI-4995-RC3 prepare release nifi-1.6.0-RC3 2018-04-03 08:28:15 -07:00
Mark Payne
641ba29cc8 NIFI-4993: Accept null user at querying ProvenanceRepository
Pass null user from ReportLineageToAtlas.

Address NullPointer dereferences

This closes #2589.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-03-28 12:15:05 -04:00
joewitt
7b5bf265a6 NIFI-4995 updating copyright year on all notices 2018-03-26 21:54:10 -04:00
Mark Payne
844da06344
NIFI-4849: Implemented REST Endpoint and associated backend code to generate a Diagnostics Report for a Processor
Implemented review feedback. Refactored data model to make the API cleaner and delineate more along the lines of what permissions are required in order to see which details
Implementing review feedback
Removed sensitive information from the diagnostics reports
Fixed bug in merging logic for GCDiagnosticsSnapshots
This closes #2468
2018-03-15 15:10:41 -04:00
Joe Skora
49c7af03c1 NIFI-3039 Provenance Repository - Fix PurgeOldEvent and Rollover Size Limits
* Added low water mark for purgeOldEvents() to more quickly purge after a large spike in events.
* Adjusted rollover high water mark to avoid overrunning "nifi.provenance.repository.max.storage.size".
* Adjusted looping logic in mergeJournals() to use ".firstKey()" instead of ".entrySet().iterator().next()" to avoid unnecessary object creation.

Signed-off-by: Mike Moser <mosermw@apache.org>
This closes #1240
2018-03-09 21:49:13 +00:00
joewitt
c71409fb5d
NIFI-4936 trying to quiet down the mvn output a bit so we dont exceed the travis-ci 4MB max
NIFI-4936 updated dependency handling pushing down delcarations where they belong
This closes #2512
2018-03-09 16:34:53 -05:00
Mark Payne
9f95a10df9 NIFI-4794: Updated event writers to avoid creating a lot of byte[] by reusing buffers. Also removed synchronization on EventWriter when rolling over the writer and just moved the writing of the header to happen before making the writer available to any other threads. This reduces thread contention during rollover.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2437
2018-02-19 09:31:11 -05:00
joewitt
41ce788812 NIFI-4751 changed to next minor release version snapshot 2018-01-12 15:15:32 -05:00
joewitt
42edfa75b7 Merge branch 'NIFI-4751-RC1' 2018-01-12 15:00:32 -05:00
Mark Payne
6153fb6465 NIFI-4752: Addressed issue with some event types having potentially the wrong FlowFile UUID listed (could have child UUID when it's supposed to have parent flowfile UUID). In testing fix, also found an issue with Search threads not being daemon and Re-Index threads not propertly being shutdown so addressed those as well.
This closes #2390.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-01-10 17:42:04 +09:00
joewitt
36405e888c NIFI-4751-RC1 prepare for next development iteration 2018-01-08 23:39:49 -07:00
joewitt
46d30c7e92 NIFI-4751-RC1 prepare release nifi-1.5.0-RC1 2018-01-08 23:39:32 -07:00
joewitt
2c68d0e098
NIFI-4652 updating to latest apache version and updating key maven plugins, fixing versions for grpc to avoid extranous build dir, correcting checkstyle issues, resolving build problem with incorrect usage of activeByDefault in maven. Key change is now for the 'dir-only' profile it is now activated via a system property -Ddir-only instead of profile. No other activeByDefault profiles exist except to minifi the javascript but that usage is acceptible as-is. This closes #2313 2017-12-04 09:59:23 -05:00
Mark Payne
2acf6bdf7a NIFI-3689: Fixed threading bug in TestWriteAheadStorePartition - multiple threads were simultaneously attempting to update HashMap. Changed impl to ConcurrentHashMap.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2214.
2017-10-20 18:15:02 +02:00
Mark Payne
b950eed1a5 NIFI-4439: This closes #2190. When a Provenance Event File is rolled over, we were failing to close the resource before attempting to compress it. Fixed that.
NIFI-4439: Addressed threading bug that can occur when rolling over provenance record writer

Signed-off-by: joewitt <joewitt@apache.org>
2017-10-16 17:18:01 -04:00
Jeff Storck
a57911d3db NIFI-4412-RC2 prepare for next development iteration 2017-09-28 13:45:36 -04:00
Jeff Storck
e6508ba7d3 NIFI-4412-RC2 prepare release nifi-1.4.0-RC2 2017-09-28 13:45:21 -04:00
Pierre Villard
5957f6ee95
NIFI-4341 - add provenance repository storage usage in UI
- addressing review comments
- This closes #2119
2017-09-11 13:16:53 -04:00
Mark Payne
84935d4f78 NIFI-4023, NIFI-4077 This closes #2075. Addressed issue where repository was aging off the wrong index. When it should age off Index 1, it was removing Index 2. As a result, the earliest index is never aged off, and the newest index could potentially be aged off before it is ready to be. Also addressed issue where a query that attempts to read an event that has aged off throws FileNotFoundException (NIFI-4077) instead of skipping over the event. The JavaDocs indicate that the EventIterator should skip records that it cannot find, but SelectiveRecordReaderEventIterator throw FileNotFoundException instead
Signed-off-by: joewitt <joewitt@apache.org>
2017-08-11 22:01:46 -07:00
zhengsg
ee3b58368d NIFI-4267 Change the class name for log in FileRemovalAction
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2061.
2017-08-07 21:42:46 +02:00
Andy LoPresto
675d989003
NIFI-4139
- Moved key provider interface and implementations from nifi-data-provenance-utils module to nifi-security-utils module.
- Refactored duplicate byte[] concatenation methods from utility classes and removed deprecation warnings from CipherUtility.
- Created KeyProviderFactory to encapsulate key provider instantiation logic.
- Added logic to handle legacy package configuration values for key providers.
- Added unit tests.
- Added resource files for un/limited strength cryptography scenarios.
- Added ASL to test resources.
- Moved legacy FQCN handling logic to CryptUtils.
- Added unit tests to ensure application startup logic handles legacy FQCNs.
- Moved master key extraction/provision out of FBKP.
- Removed nifi-security-utils dependency on nifi-properties-loader module.
- Added unit tests.
2017-08-07 15:21:58 -04:00
Matt Gilman
c99c036c20 NIFI-4045:
- Addressing issues causing the eventId to not be relayed when submitting a lineage request under certain conditions.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1903.
2017-06-27 21:48:55 +02:00
Andy LoPresto
ff65b42e88 NIFI-4110 Added Thread.sleep() to ensure provenance event indexing occurs before retrieval and unit test passes.
Signed-off-by: Scott Aslan <scottyaslan@gmail.com>

This closes #1935
2017-06-23 08:48:34 -04:00
Matt Gilman
4ed7511bee
NIFI-3653: - Introducing UserGroup and Policy provider interfaces.
- Introducing FileUserGroupProvider and FileAccessPolicyProvider.
- Refactoring FileAuthorizer to utilize the file based implementations.
- Introducing the StandardManagedAuthorizer.
- Decorating the configured ManagedAuthorizer to ensure integrity checks are still performed.
- Loading user groups if possible to use during access decisions.
- Merging responses for requests for AccessPolicies, Users, and UserGroups.
- Adding unit tests as appropriate.
- Adding methods to the User, Group, and AccessPolicy builder that more easily supports generating UUIDs.
- Fixing typo when seeding policies during startup.
- Fixing type in documentation and error messages.

This closes #1897.

Signed-off-by: Bryan Bende <bbende@apache.org>
2017-06-09 13:54:10 -04:00
Matt Gilman
cc741d2be6
NIFI-3997:
- Bumping to next minor version.
2017-06-08 15:22:51 -04:00
Matt Gilman
6ee12e9b47
NIFI-3997-RC1prepare for next development iteration 2017-06-05 11:07:43 -04:00
Matt Gilman
ddb73612bd
NIFI-3997-RC1prepare release nifi-1.3.0-RC1 2017-06-05 11:07:28 -04:00
Mark Payne
2595d816c4 NIFI-3986: This closes #1868. When we encounted EOFException from sequential record reader event iterator, just treat it as not having any more events
Signed-off-by: joewitt <joewitt@apache.org>
2017-06-01 15:08:12 -04:00
Matt Gilman
3ec0aa947f NIFI-3955:
- Filtering out the EventTime searchable field for all implementation of ProvenanceRepository. Prefer user query using the startDate/endDate fields.

Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #1844.
2017-05-23 18:48:29 +02:00
joewitt
762293c01c NIFI-3935 made the old persistent prov test more forgiving of timing issues and extended lucene test timeouts for slower environments. These should be integration tests really.
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #1823
2017-05-18 14:42:46 -04:00
Mark Payne
4fdea680ec NIFI-3905: This closes #1805. When a Provenance Query is submitted to WriteAheadProvenanceRepository, purge any obsolete queries from the internal state before rejecting the query due to 'too many outstanding queries'
Signed-off-by: joewitt <joewitt@apache.org>
2017-05-16 13:48:16 -04:00
Bryan Bende
3af53419af
NIFI-3770-RC2 prepare for next development iteration 2017-05-05 20:50:28 -04:00
Bryan Bende
3a605af8e0
NIFI-3770-RC2 prepare release nifi-1.2.0-RC2 2017-05-05 20:50:14 -04:00
Bryan Bende
1e75f33677
NIFI-3789 Removing unnecessary intermittent test failure as described in JIRA
This closes #1743.
2017-05-03 14:07:18 -04:00