1296 Commits

Author SHA1 Message Date
Mark Payne
2e1005e884 NIFI-5640: Improved efficiency of Avro Reader and some methods of AvroTypeUtil. Also switched ServiceStateTransition to using read/write locks instead of synchronized blocks because profiling showed that significant time was spent in determining state of a Controller Service when attempting to use it. Switching to a ReadLock should provide better performance there.
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3036
2018-09-27 15:38:47 -04:00
Mark Payne
ad4c886fbf
NIFI-5634: When merging RPG entities, ensure that we only send back the ports that are common to all nodes - even if that means sending back no ports
This closes #3030
2018-09-27 10:11:15 -04:00
Mark Payne
030129c7ce NIFI-5618: Avoid NPE when viewing Provenance Event details on a disconnected node
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3027.
2018-09-25 12:28:58 +02:00
Mark Payne
4b4c9e14cb NIFI-5630: Ensure that we include counters in Status History when present
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #3026.
2018-09-25 12:24:17 +02:00
Andrew I. Christianson
be2ed060a2
NIFI-5542 Added support for node groups to FileAccessPolicyProvider
This closes #2970.

Signed-off-by: Kevin Doran <kdoran@apache.org>
2018-09-20 10:27:28 -04:00
Mark Payne
3dd548e807 NIFI-5609: Fixed NullPointer when attempting to view status history for a component that has not yet run
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #3012
2018-09-19 16:56:42 -04:00
Koji Kawamura
f570cb980d
NIFI-375: Added operation policy
The operation policy allows that a user to operate components even if they does not have direct READ/WRITE
permission of the component.

Following operations are controlled by the new operate policy:
- Start/stop/enable/disable Processors, ControllerServices,
ReportingTasks, Input/OuputPorts
- Enable/disable transmission of RemoteInput/OutputPorts and
RemoteProcessGroups
- Terminate Processor threads

Refactored what API exposes

The previous commit let API exposes few fields in DTO. But we should
avoid returning partial DTO as it complicates authorization logic.

Instead, this commit adds StatusDTO for ReportingTaskEntity and
ControllerServiceEntity, so that it can be returned regardless of having
READ permission. Component DTO can only be returned with a READ
permission.

Refactor RPG same as ControllerService.

WIP incorporating review comments.

Incorporated review comments

- Cleaned up merger classes
- Recreate DTO instance at each function during two phase commmit

Restrict enabling ControllerService without read permission

Revert the last commit.

Fix review comments.

- Renamed confusing static method names and its parameters
- Removed unnecessary permission checks from UI condition

Fixed delete action display condition.

Fixed NPE at Summary.

Apply operation policy to activateControllerServices.

Removed OperationPermissible from ComponentEntity.

This closes #2990
2018-09-19 15:28:41 -04:00
Bryan Bende
14729be837 NIFI-5577 Fixing update method in NaiveRevisionManager so that failed updates don't change the revision
This closes #2995.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-09-14 11:33:53 -04:00
Mark Payne
87cf474e54
NIFI-5581: Disable connection pooling for OkHttpReplicationClient. We should revisit this in the future, but for now, it appears that Jetty is having problems with the connections if they are reused. By disabling the Connection Pooling, we address the concern, but for secure connections this means that every request results in a TLS handshake - and for a mutable request, both the verification and the 'performance' stages require the TLS handshake. But it's better than timing out, which is the currently observed behavior
This closes #2996
2018-09-10 09:43:53 -04:00
thenatog
fc1461298a
NIFI-5366 - Added ContentSecurityPolicyFilter which stops framing of NiFi resources. It applies the Content-Security-Policy header. This protects against clickjacking.
NIFI-5366 - Added unit test. Added single quotes around 'self' for frame-ancestors CSP header.
NIFI-5366 - Fixed dependencies.

This closes #2989.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-09-05 17:38:53 -07:00
Andy LoPresto
e83ea1f9ae
NIFI-5558 Fixed unit test to avoid contamination from System property "nifi.properties.file.path" when creating test NiFiProperties instance.
This closes #2972.

Co-authored-by: thenatog <thenatog@gmail.com>
Signed-off-by: Marc Parisi <phrocker@apache.org>
2018-08-29 14:52:07 -07:00
Matt Gilman
f04cd8681d NIFI-5479: Using the SUN provider when the keystore type is JKS.
This closes #2961.

Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:30 +09:00
joewitt
a27ccd8a56 NIFI-5479 Upgraded Jetty. Moved where we unpack bundled deps to so we can avoid a new jetty bug with META-INF loading logic.
Signed-off-by: Koji Kawamura <ijokarumawak@apache.org>
2018-08-28 15:28:11 +09:00
Mark Bean
58cb900b90
NIFI-5543: move bendPointDrag below mousedown.selection in nf-connection.js
This closes #2957
2018-08-27 11:04:07 -04:00
Mark Payne
aac2c6a60e
NIFI-5482: Made WriteAheadProvenanceRepository the default implementation
This closes #2960.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-08-23 16:49:55 -07:00
Andy LoPresto
744b15b4a7
NIFI-5540 Added unit test to demonstrate missing default sensitive properties key in flow election encryptor creation.
NIFI-5540 Added failing unit test and ignored regression test to demonstrate missing default sensitive properties key in flow election encryptor creation.

NIFI-5540 Added equality logic to StringEncryptor and utility equality methods to CryptoUtils.

NIFI-5540 Added default sensitive properties key population logic and log warning to StringEncryptor.

NIFI-5540 Cleaned up formatting.

NIFI-5540 Cleaned up boolean logic.

NIFI-5540 Added Javadoc to StringEncryptor.

NIFI-5540 Added unit test for StringEncryptor#equals().

NIFI-5540 Added performance benchmarking unit tests for constantTimeEquals methods for String, byte[], and char[].

NIFI-5540 Fixed checkstyle issue.

NIFI-5540 Fixed unit tests for default key population.

This closes #2959.

Signed-off-by: Bryan Bende <bbende@apache.org>
2018-08-22 13:56:16 -04:00
Mark Payne
5b57935bae NIFI-5527: Refactored unit test to make member variables for 'delay tolerances' so that they are easier to update in the future. Increased delay tolerances so that tests pass on heavily loaded system.
This closes #2955.
2018-08-22 10:51:47 -04:00
Mark Payne
7bbb5a823a NIFI-5466: Keep a running total of stats for each component. Refactored FlowFileEvent and repository in order to provide more efficient storage of objects on Java heap by allowing the same 'EMPTY' object to be reused
- Refactored VolatileComponentStatusRepository to avoid holding on to ProcessorStatus objects, etc, and only keep what they need
 - Updated VolatileComponentStatusRepository to ensure that we are efficiently storing metrics for processors, etc. that are not running

This closes #2939

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-08-21 11:44:25 -04:00
patricker
5106dc0af9
NIFI-4535 Only update Page Title to root flow name when user has permission.
This closes #2899
2018-08-13 12:46:16 -04:00
Scott Aslan
59a79c134f
[NIFI-5499] upgrade AngularJS to v1.7.2
[NIFI-5499] update package-lock.json

This closes #2941
2018-08-13 11:40:59 -04:00
Lars Francke
a19134f325 NIFI-5350 Add a way to provide arbitrary Java options in shell scripts
Signed-off-by: Mike Moser <mosermw@apache.org>

This closes #2823
2018-08-06 21:02:29 +00:00
Mark Payne
4cca9bef7c NIFI-5480: Use FlowController's maps of components in order to look up component by ID rather than iterating recursively through all Process Groups to find the component
This closes #2932

Signed-off-by: Jeremy Dyer <jeremydyer@apache.org>
2018-08-01 19:33:56 -04:00
Andy LoPresto
e62aa0252d
NIFI-5442 Get X-ProxyContextPath value from request attributes rather than directly from headers.
NIFI-5442 Populate request contextPath attribute during AccessResource before displaying on message-page.jsp.
Refactored shared code from CatchAllFilter to WebUtils.
NIFI-5442 Refactored filter and context path code to shared parent filter and subclass.
NIFI-5442 Removed unnecessary initParams from nifi-web-ui web.xml.
NIFI-5442 Added explicit dispatchers to nifi-web-ui web.xml and removed unnecessary code from AccessResource.

This closes #2908
2018-08-01 12:10:14 -04:00
Matthew Burgess
473221368c NIFI-5420: Allow StandardProcessSession to calculate duration for provenance events
This closes #2886.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-07-27 09:13:50 -04:00
Jan Hentschel
610cbb66b2 NIFI-5418 Removed duplicated jackson-databind dependency from nifi-framework-cluster
This closes #2885

Signed-off-by: zenfenan <zenfenan@apache.org>
2018-07-26 23:20:13 +05:30
Andy LoPresto
0ad30e188f NIFI-5451 Added test resources for 128 bit encryption.
Fixed unit test to perform properly without JCE unlimited strength policy installed.

This closes #2916.

Signed-off-by: Mike Moser <mosermw@apache.org>
2018-07-25 15:21:13 +00:00
Jeff Storck
46ce7aaa32
NIFI-5341 Enabled groovy tests in nifi-runtime
Fixed tests in NiFiGroovyTest in the nifi-runtime module
Updated NiFi.createBootstrapClassLoader to log a warning if lib/bootstrap does not exist rather than throwing a FileNotFoundException, since it already catches MalformedUrlException if there's an issue adding one of the bootstrap JARs to the bootstrap classpath
Explicitly handling InvocationTargetException in NiFi.initializeProperties to unwrap the cause and rewrap as an IllegalArgumentException to propogate the real cause of the underlying exception thrown by NiFiPropertiesLoader

This closes #2821.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-07-18 11:35:55 -07:00
Damian Czaja
ec7f131602
NIFI-4889: Logout not working properly with OIDC
- Redirect user agent to end session endpoint on the OIDC provider
- This closes #2830
2018-07-16 10:14:31 -04:00
Koji Kawamura
e94f0757db
NIFI-4654: Support reporting RAS S2S lineage to Atlas
- Added 's2s.port.id' FlowFile attribute to track target remote Port id
- Use 's2s.port.id' to analyze RAW S2S provenance events
- This closes #2863
2018-07-10 11:44:00 -04:00
Mark Bean
b279624398 NIFI-5368 controller services validated prior to enabling; referenced controller services must be enabled for referencing component to be valid (mock framework)
This closes #2873.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-07-10 11:23:40 -04:00
Mark Payne
35bfc93901
NIFI-5377: Addressed issue of infinite recursion when enabling/disabling controller services if there is a recursive loop (i.e., Service A references Service B references Service A). This closes #2847
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2018-07-10 09:57:13 -04:00
Mark Bean
0d07bc4951
NIFI-5377 prevent infinite loop if a controller service circular reference exists 2018-07-10 09:57:13 -04:00
Andy LoPresto
3ef8b4ab8d NIFI-5370 removed custom hostname verifier implementation from OkHttpReplicationClient (default handles wildcard certs).
This closes #2869.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-07-10 08:45:33 -04:00
Andy LoPresto
275789f8ca NIFI-5401 Improved logging for node identity loading.
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2871.
2018-07-10 09:51:52 +02:00
Andy LoPresto
f60585a9b6
NIFI-5376 Removed deprecation warnings.
Updated Javadoc for SiteToSiteClient#createTransaction() and HttpClient implementation.
Reverted exception listing in method contract for SiteToSiteClient#createTransaction and HttpClient tion of same.
Reverted import ordering in TestSiteToSiteClient.
Reverted exception listing in TestGetHDFSFileInfo, TestListHDFS, and StandardHttpFlowFileServerProtocol.
Restored @SuppressWarnings annotation and removed unnecessary "public static" keywords from inner classes in SiteToSiteClient.

This closes #2841.

Signed-off-by: Joe Witt <joewitt@apache.org>
2018-07-09 20:45:34 -07:00
Mark Payne
1e75f8c789 NIFI-5394: Ensure that we wait for service to be fully enabled when enabling a group, before moving on to the next in the list
Signed-off-by: Matthew Burgess <mattyb149@apache.org>

This closes #2867
2018-07-09 12:15:23 -04:00
patricker
d50e3f1747 NIFI-5384 FlowFile's queued in batches should all have the same Queue time
This closes #2849

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-07-07 07:56:53 -04:00
thenatog
b77d66889a
NIFI-5374 - Added ExceptionFilter which catches RequestRejectedException thrown in the nifi-api Jersey code. These exceptions were not caught by the Jetty error-page configuration because they're thrown before the endpoint/Jetty routing is hit.
Added integration test for checking the ExceptionFilter catches malicious string exceptions.
Made minor changes to PR 2840 for code style.

This closes #2840.

Co-authored-by: Andy LoPresto <alopresto@apache.org>

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-07-03 18:17:22 -07:00
Mark Payne
d4d4ddadee
NIFI-5361: When submitting many processors to start, calculate the 'timeout timestamp' immediately before calling @OnScheduled method, after the task has been scheduled to run, instead of before the task has a chance to run.
This closes #2831
2018-07-02 16:36:07 -04:00
Mark Payne
b549fbb67b NIFI-5362: When a processor is terminated and has no more active threads, ensure that we set this.hasActiveThreads = false
Signed-off-by: Pierre Villard <pierre.villard.fr@gmail.com>

This closes #2832.
2018-07-02 16:20:20 +02:00
patricker
4ea6e6a40e NIFI-4535
This closes #2826

Signed-off-by: Mike Thomsen <mikerthomsen@gmail.com>
2018-07-01 21:25:21 -04:00
thenatog
a274918dc5
NIFI-5258 - Changed the way the servlets are created for the documentation webapp.
Removed some unnecessary code.
Fixed imports.

This closes #2812.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-06-25 18:22:55 -07:00
Andy LoPresto
d42a1e8bf4
NIFI-5323-RC1 prepare for next development iteration 2018-06-19 20:02:21 -07:00
Andy LoPresto
99bcd1f88d
NIFI-5323-RC1 prepare release nifi-1.7.0-RC1 2018-06-19 20:02:01 -07:00
Mark Payne
d0499eb867
NIFI-5311: When creating a processor, controller service, or reporting task, give the component up to 50 ms to complete validation before returning the DTO. This closes #2795 2018-06-14 13:24:27 -04:00
Matt Gilman
fe31a06fdc
NIFI-4907:
- Minor adjustments following PR.
- Avoiding additional find operation when authorizing components when populating component details.
- Requiring access to provenance events when downloading content or submitting a replay as they may provide events details.
- Updating the REST API docs detailing the required permissions.
- Updating the wording in the documentation regarding the provenance and data policies.
- Removed the event attributes from the authorization calls that were verifying access to provenance events.
- Only checking content availability when the user is authorized for the components data.
- Addressing typo in JavaDoc.

This closes #2703
2018-06-14 12:21:47 -04:00
Mark Bean
e27798797a
NIFI-4907: add 'view provenance' component policy
whitespace removed for checkstyle
2018-06-14 10:59:17 -04:00
Marco Gaido
d98d335497 [NIFI-3242] Avoid double scheduling of a task due to quartz imprecision
This closes #2789.

Signed-off-by: Mark Payne <markap14@hotmail.com>
2018-06-14 10:10:45 -04:00
thenatog
dbf259508c
NIFI-5258 - Changed addHeader to setHeader which stops X-Frame-Options being added twice to responses. Added unit test.
This closes #2791.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2018-06-13 16:13:53 -07:00
Mark Payne
275b8cbf23
NIFI-5308: Avoid holding Template DOM Nodes in heap. This closes #2790 2018-06-13 16:00:01 -04:00