Commit Graph

428 Commits

Author SHA1 Message Date
Mark Payne 307189d15b Merge branch 'NIFI-1103' of https://github.com/adamonduty/nifi 2015-12-01 15:45:51 -05:00
Matt Gilman 7726d069cd Merge branch 'NIFI-655' 2015-12-01 11:20:56 -05:00
Matt Gilman e22b51f3a7 NIFI-655:
- Renaming spring tokens to avoid confusion over authentication and authorization.
2015-12-01 10:08:36 -05:00
Matt Gilman 85eb8defdd NIFI-655:
- Changing default expiration time to 12 hours.
2015-12-01 09:36:33 -05:00
Matt Gilman c100052dac NIFI-655:
- Adding additional logging when proceeding as an anonymous user.
2015-12-01 08:51:45 -05:00
Mark Payne e5281f1fc1 NIFI-1221: Support batching of Syslog messages
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-11-30 17:32:49 -05:00
Matt Gilman 2b0819a5f2 NIFI-655:
- Removing unused imports.
2015-11-30 16:33:11 -05:00
Matt Gilman 014b2ac4e8 NIFI-655:
- Removing proxied user chain as user details are already serialized.
2015-11-30 16:30:12 -05:00
Mark Payne c59087bc3a Merge branch 'NIFI-1049' 2015-11-30 16:02:36 -05:00
Mark Payne 07c619cf40 NIFI-1049: Added unit test to verify that reported bug was fixed; corrected a typo in log message 2015-11-30 15:37:58 -05:00
Matt Gilman 774d626f88 NIFI-655:
- Adding documentation around the behavior of the authentication filters.
- Only passing along necessary parameters.
2015-11-30 15:07:40 -05:00
Matt Gilman c722b56335 NIFI-655:
- Ensuring the access token is not replicated when the user is already authenticated/authorized.
2015-11-30 14:57:38 -05:00
Matt Gilman a84e505bcd NIFI-655:
- Ensuring the access token is not replicated when the user is already authenticated/authorized.
2015-11-30 14:47:30 -05:00
Matt Gilman 99016a835e NIFI-655:
- Ensuring the protocol is specified.
2015-11-30 12:38:17 -05:00
Matt Gilman 64beeef593 NIFI-655:
- Ensuring anonymous user label and login links are rendered when appropriate.
- Ensuring responses are accurate when making requests with a token when user log in is not supported.
2015-11-27 14:13:40 -05:00
Matt Gilman c1cc165edb NIFI-655:
- Fixing issue with filter bean initialization when clustered.
2015-11-27 10:05:58 -05:00
Mark Payne 9aa9c27dbe NIFI-1216: Check if log level is enabled immediately in the SimpleProcessLogger before formatting log message
Signed-off-by: joewitt <joewitt@apache.org>
2015-11-26 00:41:15 -05:00
joewitt 67aed5eb92 NIFI-1224: made TestExecuteSQL about 25 seconds faster. Same logic just less to process
Reviewed and Amended (removed trailing whitespace) by Tony Kurc (tkurc@apache.org)
2015-11-26 00:33:54 -05:00
joewitt a3cb803147 NIFI-1224: reduced size of test set produced and removed ignored memory test
Reviewed and amended (removed unused imports for checkstyle) by Tony Kurc (tkurc@apache.org)
2015-11-25 23:09:49 -05:00
joewitt 6692645209 NIFI-631: addressed environment specific unit test problems
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-25 22:06:45 -05:00
Joe Skora 226ac64ef9 NIFI-631: Correct the WritesAttributes descriptions.
Reviewed and Ammended (based on comments from dev list) by Tony Kurc (tkurc@apache.org)
2015-11-25 18:03:19 -05:00
Mark Payne ba83bf9c05 NIFI-631: Initial implementation of FetchFile processor
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-25 18:03:19 -05:00
Joe Skora 4c4d62c61f NIFI-631: Added ListFile processor.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-25 18:03:19 -05:00
Matt Gilman 6bce858e4a NIFI-655:
- Updated user guide with screenshots of login process.
- Tweaked wording in admin guide.
- Triggering login on enter press in login form.
2015-11-25 16:42:22 -05:00
Mark Payne 5061e5fa0a NIFI-1222: Session.adjustCounter keeps track of local and global counters; it then call processContext.adjustCounter with each of them, but ProcessContext was changed a while back to automatically increment both 'lcoal' and 'global' counters each time, so our numbers are doubled; removed the 'localCounters' and 'globalCounters' from StandardProcessSession and replaced with just 'counters'
Signed-off-by: joewitt <joewitt@apache.org>
2015-11-25 16:26:59 -05:00
Mark Payne a29b7b3bf0 NIFI-1203: Do not count looping connections when determining validity based on incoming connections
Signed-off-by: joewitt <joewitt@apache.org>
2015-11-25 15:32:41 -05:00
Matt Gilman 0435911186 NIFI-1198: - Updating the connection source and destination when appropriate (deletion and (re)connection).
Signed-off-by: joewitt <joewitt@apache.org>
2015-11-25 14:50:56 -05:00
Matt Gilman c073253366 NIFI-655:
- Update admin guide with documentation for username/password authentication.
- Setting default anonymous roles to none.
- Making account status messages to users more clear.
- Deleting user keys when an admin revokes/deletes an account.
- Updating authentication filter to error back whenever authentication fails.
2015-11-25 14:17:23 -05:00
Mark Payne e1742c5a04 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-25 13:03:22 -05:00
Mark Payne 4fa2a713a2 NIFI-1192: Removed some additional white space 2015-11-25 12:57:06 -05:00
Mark Payne 657885e5ba Merge branch 'NIFI-1192B' of https://github.com/olegz/nifi into NIFI-1192 2015-11-25 12:30:15 -05:00
Oleg Zhurakousky d949ee1a1e NIFI-1192 added support for dynamic properties to GetKafka
Due to the fact that current component uses artificial names for properties set via UI and then maps those properties to the actual names used by Kafka, we can not rely on NiFi UI to display an error if user attempts to set a dynamic property which will eventually map to the same Kafka property. So, I’ve decided that any dynamic property will simply override an existing property with WARNING message displayed. It is actually consistent with how Kafka does it and displayed the overrides in the console. Updated the relevant annotation description.
It is also worth to mentioned that current code was using an old property from Kafka 0.7 (“zk.connectiontimeout.ms”) which is no longer present in Kafka 0.8 (WARN Timer-Driven Process Thread-7 utils.VerifiableProperties:83 - Property zk.connectiontimeout.ms is not valid). The add/override strategy would provide for more flexibility when dealing with Kafka volatile configuration until things will settle down and we can get some sensible defaults in place.

While doing it addressed the following issues that were discovered while making modification and testing:
ISSUE: When GetKafka started and there are no messages in Kafka topic the onTrigger(..) method would block due to the fact that Kafka’s ConsumerIterator.hasNext() blocks. When attempt was made to stop GetKafka would stops successfully due to the interrupt. However in UI it would appear as ERROR based on the fact that InterruptException was not handled.
RESOLUTION: After discussing it with @markap14 the the general desire is to let the task exit as quick as possible and that the whole thread maintenance logic was there initially due to the fact that there was no way to tell Kafka consumer to return immediately if there are no events. In this patch we are now using ‘consumer.timeout.ms’ property of Kafka and setting its value to 1 millisecond (default is -1 - always block infinitely). This ensures that tasks that attempted to read an empty topic will exit immediately just to be rescheduled by NiFi based on user configurations.

ISSUE:  Kafka would not release FlowFile with events if it didn’t have enough to complete the batch since it would block waiting for more messages (based on the blocking issue described above).
RESOLUTION: The invocation of hasNext() results in Kafka’s ConsumerTimeoutException which is handled in the catch block where the FlowFile with partial batch will be released to success. Not sure if we need to put a WARN message. In fact in my opinion we should not as it may create unnecessary confusion.

ISSUE: When configuring a consumer for topic and specifying multiple concurrent consumers in ‘topicCountMap’ based on 'context.getMaxConcurrentTasks()’ each consumer would bind to a topic partition. If you have less partitions then the value returned by 'context.getMaxConcurrentTasks()’ you would essentially allocate Kafka resources that would never get a chance to receive a single message  (see more here https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example).
RESOLUTION: Logic was added to determine the amount of partitions for a topic and in the event where 'context.getMaxConcurrentTasks()’ value is greater than the amount of partitions, the partition count will be used to when creating ‘topicCountMap’ and WARNING message will be displayed)see code). Unfortunately we can’t do anything with the actual tasks, but based on current state of the code they will exit immediately just to be rescheduled where the process will repeat. NOTE: That is not ideal as it will be rescheduling tasks that will never have a chance to do anything, but at least it could be fixed on the user side after reading the warning message.

NIFI-1192 added dynamic properties support for PutKafka

NIFI-1192 polishing

NIFI-1192 polished and addressed PR comments
2015-11-24 12:14:36 -05:00
Matt Gilman 1312bde498 NIFI-655:
- Updating available links during login, registration, and account status review.
2015-11-24 00:37:47 -05:00
Bryan Bende 4281a51c83 Merge branch 'NIFI-1208' 2015-11-23 21:35:57 -05:00
Matt Gilman f2d82ee140 NIFI-655:
- Updating the version of ldap provider nar.
2015-11-23 16:53:26 -05:00
Matt Gilman a5754986e2 NIFI-655:
- Fixing the configuration property name for Authentication Expiration in the provided example configuration.
2015-11-23 15:55:24 -05:00
Matt Gilman aaf14c45c9 NIFI-655:
- Refactoring web security to use Spring Security Java Configuration.
- Introducing security in Web UI in order to get JWT.

NIFI-655:
- Setting up the resources (js/css) for the login page.

NIFI-655:
- Adding support for configuring anonymous roles.
- Addressing checkstyle violations.

NIFI-655:
- Moving to token api to web-api.
- Creating an LoginProvider API for user/pass based authentication.
- Creating a module for funneling access to the authorized useres.

NIFI-655:
- Moving away from usage of DN to identity throughout the application (from the user db to the authorization provider).
- Updating the authorized users schema to support login users.
- Creating an extension point for authentication of users based on username/password.

NIFI-655:
- Creating an endpoint for returning the identity of the current user.
- Updating the LoginAuthenticationFilter.

NIFI-655:
- Moving NiFi registration to the login page.
- Running the authentication filters in a different order to ensure we can disambiguate each case.
- Starting to layout each case... Forbidden, Login, Create User, Create NiFi Account.

NIFI-655:
- Addressing checkstyle issues.

NIFI-655:
- Making nf-storage available in the login page.
- Requiring use of local storage.
- Ignoring security for GET requests when obtaining the login configuration.

NIFI-655:
- Adding a new endpoint to obtain the status of a user registration.
- Updated the login page loading to ensure all possible states work.

NIFI-655:
- Ensuring we know the necessary state before we attempt to render the login page.
- Building the proxy chain in the JWT authentication filter.
- Only rendering the login when appropriate.

NIFI-655:
- Starting to style the login page.
- Added simple 'login' support by identifying username/password. Issuing JWT token coming...
- Added logout support
- Rendering the username when appropriate.

NIFI-655:
- Extracting certificate validation into a utility class.
- Fixing checkstyle issues.
- Cleaning up the web security context.
- Removing proxy chain checking where possible.

NIFI-655:
- Starting to add support for registration.
- Creating registration form.

NIFI-655:
- Starting to implement the JWT service.
- Parsing JWT on client side in order to render who the user currently is when logged in.

NIFI-655:
- Allowing the user to link back to the log in page from the new account page.
- Renaming DN to identity where possible.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Adding more/better support for logging out.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Adding a few new exceptions for the login identity provider.

NIFI-655:
- Disabling log in by default initially.
- Restoring authorization service unit test.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Updating packages for log in filters.
- Handling new registration exceptions.
- Code clean up.

NIFI-655:
- Removing registration support.
- Removing file based implementation.

NIFI-655:
- Removing file based implementation.

NIFI-655:
- Removing unused spring configuration files.

NIFI-655:
- Making the auto wiring more explicit.

NIFI-655:
- Removing unused dependencies.

NIFI-655:
- Removing unused filter.

NIFI-655:
- Updating the login API authenticate method to use a richer set of exceptions.
- UI code clean.

NIFI-655:
- Ensuring the login identity provider is able to switch context classloaders via the standard NAR mechanisms.

NIFI-655:
- Initial commit of the LDAP based identity providers.
- Fixed issue when attempting to log into a NiFi that does not support new account requests.

NIFI-655:
- Allowing the ldap provider to specify if client authentication is required/desired.

NIFI-655:
- Persisting keys to sign user tokens.
- Allowing the identity provider to specify the token expiration.
- Code clean up.

NIFI-655:
- Ensuring identities are unique in the key table.

NIFI-655:
- Adding support for specifying the user search base and user search filter in the active directory provider.

NIFI-655:
- Fixing checkstyle issues.

NIFI-655:
- Adding automatic client side token renewal.

NIFI-655:
- Ensuring the logout link is rendered when appropriate.

NIFI-655:
- Adding configuration options for referrals and connect/read timeouts

NIFI-655:
- Added an endpoint for access details including configuration, creating tokens, and checking status.
- Updated DTOs and client side to utilize new endpoints.

NIFI-655:
- Refactoring certificate extraction and validation.
- Refactoring how expiration is specified in the login identity providers.
- Adding unit tests for the access endpoints.
- Code clean up.

NIFI-655:
- Keeping token expiration between 1 minute and 12 hours.

NIFI-655:
- Using the user identity provided by the login identity provider.

NIFI-655: - Fixed typo in error message for unrecognized authentication strategy.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655. - Added logback-test.xml configuration resource for nifi-web-security.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655. - Added issuer field to LoginAuthenticationToken. - Updated AccessResource to pass identity provider class name when creating LoginAuthenticationTokens. - Began refactoring JWT logic from request parsing logic in JwtService. - Added unit tests for JWT logic.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655. - Changed issuer field to use FQ class name because some classes return an empty string for getSimpleName(). - Finished refactoring JWT logic from request parsing logic in JwtService. - Updated AccessResource and JwtAuthenticationFilter to call new JwtService methods decoupled from request header parsing. - Added extensive unit tests for JWT logic.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655:
- Refactoring key service to expose the key id.
- Handling client side expiration better.
- Removing specialized active directory provider and abstract ldap provider.

NIFI-655. - Updated JwtService and JwtServiceTest to use Key POJO instead of raw String key from KeyService.

Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>

NIFI-655:
- Fixing typo when loading the ldap connect timeout.
- Providing a better experience for session expiration.
- Using ellipsis for lengthly user name.
- Adding an issuer to the authentication response so the LIP can specify the appropriate value.

NIFI-655:
- Showing a logging in notification during the log in process.

NIFI-655:
- Removing unnecessary class.

NIFI-655:
- Fixing checkstyle issues.
- Showing the progress spinner while submitting account justification.

NIFI-655:
- Removing deprecated authentication strategy.
- Renaming TLS to START_TLS.
- Allowing the protocol to be configured.

NIFI-655:
- Fixing issue detecting the presence of DN column

NIFI-655:
- Pre-populating the login-identity-providers.xml file with necessary properties and documentation.
- Renaming the Authentication Duration property name.

NIFI-655:
- Updating documentation for the failure response codes.

NIFI-655:
- Ensuring the user identity is not too long.

NIFI-655:
- Updating default authentication expiration to 12 hours.

NIFI-655:
- Remaining on the login form when there is any unsuccessful login attempt.
- Fixing checkstyle issues.
2015-11-23 14:50:13 -05:00
Mark Payne 4e2c94d659 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-23 14:18:50 -05:00
Mark Payne 2516b1dad2 NIFI-1171: Ensure that we pick up changes when files roll over and ensure that we don't pick up the rolled over file multiple times 2015-11-23 14:11:14 -05:00
Bryan Bende 3ffb455903 NIFI-1208 Adding context.yield() to Listen and Put syslog when there is nothing to do in onTrigger() 2015-11-23 10:30:33 -05:00
Joseph Percivall cdd2c4f22c NIFI-1086: Changed behavior on retrieval with no input file to RETRIEVE events, removed @TriggerWhenEmpty
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-21 19:20:33 -05:00
Bryan Bende f1f67f6395 Fixing one-character typo in syslog attribute name 2015-11-20 10:45:21 -05:00
Aldrin Piri 08d59e4374 NIFI-1196 Providing handling of FETCH provenance events for their "unique" property, transit URI, within the framework and UI.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-19 17:42:15 -05:00
Bryan Bende 40dd8a0a84 NIFI-1174 Refactoring the HBase client API and adding a PutHBaseJSON which can write a whole row from a single json document - Adding Complex Field Strategy to PutHBaseJSON to allow more control of complex fields - Improving error messages to indicate what the problem was with an invalid row
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-11-19 13:49:02 -05:00
Joseph Percivall 8c2323dc8d NIFI-1086 Provide refactoring of InvokeHTTP
NIFI-980 Add support for HTTP Digest authentication to InvokeHttp
NIFI-1080 Provide additional InvokeHttp unit tests
NIFI-1133 InvokeHTTP Processor does not save Location header for 3xx responses
NIFI-1009 InvokeHTTP should be able to be scheduled without any incoming connection for GET operations
NIFI-61 Multiple improvements for InvokeHTTP inclusive of providing unique tx.id across clusters, dynamic HTTP header properties

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-19 01:40:21 -05:00
Joe Percivall fb335ea282 NIFI-1165: Fix for tests TestRouteText and PutHDFS which did not succeed on Windows
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-19 01:01:34 -05:00
Mark Payne e862f7ff03 NIFI-1165: Use FileChannel instead of RandomAccessFile in order to avoid locking files in Windows
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-19 01:01:28 -05:00
Joseph Percivall 1e5cc070a3 NIFI-1081 Adding option to ExecuteStreamCommand to put output value to an attribute
Reviewed and amended (comments,whitespace,and some code readability (discussed in ticket)) by Tony Kurc (tkurc@apache.org)
2015-11-18 23:23:10 -05:00
Tony Kurc 9e2f6df205 NIFI-1123: Fixing a botched commit. 2015-11-18 21:59:42 -05:00
Joe Skora 52b24b93d9 NIFI-1123 Adds expression language support to DeleteAttributesExpression on UpdateAttributes Processor.
Reviewed by Tony Kurc (trkurc@gmail.com) after Aldrin Piri <aldrin@apache.org> did the initial review and actionable comments
2015-11-18 19:10:21 -05:00
Tony Kurc ab7940368a NIFI-1187: Fixing issue of possible assigment reordering causing uninitalized values to be possibly returned
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-18 17:37:47 -05:00
Aldrin Piri c541c82c35 NIFI-1191 Adding missing tags for ConvertAvroToJSON 2015-11-18 16:38:15 -05:00
Mark Payne 69bce2c2db NIFI-1168: Ensure that processors with only looping
connections are scheduled to run, even if the connections have no FlowFiles;
 expose these details to processor developers; update documentation

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-18 14:53:30 -05:00
Aldrin Piri 773576e041 NIFI-1108 Providing additional annotations on processors to ensure utilization of the InputRequirement annotation. 2015-11-18 13:56:21 -05:00
Mark Payne 911e1c5412 NIFI-1108: Updated processors to include the @InputRequirement annotation
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-18 13:47:59 -05:00
Mark Payne d88b6cb6bc NIFI-1173: Even if FlowFile Queue is empty, it needs to hold onto the Empty Queue request so that subsequent cancel/clear requests can reference it
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-18 10:56:38 -05:00
Mark Payne 180ea1ba22 NIFI-1176: Use a smaller internal blocking queue size of only 5000 messages; do not create a new queue every time the processor is scheduled to run
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-18 10:51:22 -05:00
Mark Payne 93be753301 NIFI-1181: Ensure that a FlowFile's uuid cannot be modified by processors
Signed-off-by: Aldrin Piri <aldrin@apache.org>
2015-11-18 10:35:37 -05:00
Bryan Bende 2b9b5e008f NIFI-1175 Exposing minimum properties required to create an HBase connection on the HBaseClientService as an optional alternative to the conf files 2015-11-17 12:01:46 -05:00
Mark Payne 453b140d6b Merge branch 'NIFI-748' of https://github.com/olegz/nifi into NIFI-748 2015-11-17 09:23:10 -05:00
Naveen Madhire 03a54bf2d5 Changes after review 2015-11-16 13:32:17 -06:00
Naveen Madhire b954ca620e NIFI-1146 Allow GetKafka to be configured with auto.offset.reset to largest or smallest 2015-11-16 11:59:52 -06:00
Oleg Zhurakousky 15880f9fcc NIFI-748 addressed PR comments
- made DocReader package private
- polished logic in read(..) method to avoid escaping the loop
- added call to sorting logic in LuceneUtil.groupDocsByStorageFileName(..) to ensure that previous behavior and assumptions in read(..) methodd are preserved
- other minor polishing
2015-11-16 08:39:23 -05:00
Matt Gilman 64369f67f9 NIFI-1160:
- Addressing possible NPE when removing a drop request.
- Only updating a connection status once it's been retrieved for the first time.
2015-11-16 08:34:04 -05:00
Mark Payne da9f40b3f0 NIFI-973: Updated main Help page to provide a link to the Getting Started Guide.
Reviewed by Aldrin Piri (aldrin@apache.org)
2015-11-15 08:38:51 -05:00
Tony Kurc 528dab78d6 NIFI-1073: Fixing coverity discovered errors. Resource leaks, and statics
Reviewed by Bryan Bende (bbende@apache.org)
2015-11-14 18:56:43 -05:00
Aldrin Piri 01539ed323 NIFI-1163: Providing handling of SSLContext creation in GetHTTP in case of only performing a one-way SSL request and accompanying test to verify the configuration/usage.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-14 18:47:50 -05:00
Oleg Zhurakousky a4d93c62c8 NIFI-748 Fixed logic around handling partial query results from provenance repository
- Ensured that failures derived form correlating Document to its actual provenance event do fail the entire query and produce partial results with warning messages
- Refactored DocsReader.read() operation.
- Added test to validate two conditions where the such failures could occur
2015-11-13 14:23:31 -05:00
Mark Payne 90f6830003 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-13 10:47:05 -05:00
Mark Payne 22de23baa6 NIFI-1097: Rewrite PutKafka to use the new producer api 2015-11-13 10:46:46 -05:00
joewitt 36d00a60f5 NIFI-1155 fixed contrib-check violation 2015-11-13 00:06:12 -05:00
Mark Payne 37d6b7350e NIFI-1155: Refactored StandardFlowFileQueue to update member variables more intelligently, using CAS operations instead of locks. This reduces code complexities because other optimizations that previously existed are no longer needed
Signed-off-by: joewitt <joewitt@apache.org>
2015-11-13 00:06:12 -05:00
Mark Payne 3ed0949c55 NIFI-1155: Ensure that when poll(FlowFileFilter, Set) is called, we properly update the indicator for whether or not queue is full
Signed-off-by: joewitt <joewitt@apache.org>
2015-11-13 00:06:12 -05:00
Adam Lamar 428b20fc25 NIFI-1103: Add support for long polling in GetSQS processor 2015-11-12 22:03:36 -07:00
Tony Kurc e6086420aa NIFI-1158 Default timestamp based on now 2015-11-12 23:11:43 -05:00
Mark Payne 0900fb80c9 NIFI-1153: If no incoming FlowFile, don't try to transfer null
Reviewed by Bryan Bende (bbende@apache.org)
Amended based on review (change to a log message) by Tony Kurc (tkurc@apache.org)
2015-11-12 22:41:57 -05:00
Bryan Bende 33ef59c5ba NIFI-1127 Adding Kerberos properties to FetchHDFS and ListHDFS.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-12 22:14:12 -05:00
Mark Payne 65bd8c0b1f NIFI-1109 Updated documentation and cleaned up code
Reviewed by Bryan Bende (bbende@apache.org).
Committed with amendments (for whitespace, a couple misspellings and to change a test method name based on review) by Tony Kurc (tkurc@apache.org)
2015-11-12 21:07:01 -05:00
joewitt 02102ea1c2 Revert "NIFI-1134". This was removed because they looked like floating classes. But in reality they are used to make jars which we use to test the build. However, we really need the test to work differently but until that is fixed these must stay.
This reverts commit 41f3875347.
2015-11-12 18:00:40 -05:00
Mark Payne 56ad22aea6 NIFI-900: Created Processors for interacting with Microsoft Azure EventHubs
Reviewed (with amendments needed for clean merge, whitespace and NOTICEs) by Tony Kurc (tkurc@apache.org)
2015-11-11 23:06:04 -05:00
Mark Payne 7a165b62cc Merge branch 'NIFI-994' 2015-11-11 16:05:40 -05:00
Mark Payne 854d203982 Merge branch 'NIFI-1143' of https://github.com/olegz/nifi into NIFI-1143 2015-11-11 14:14:54 -05:00
Oleg Zhurakousky 5baafa156a NIFI-1143 Fixed race condition which caused intermittent failures
Fixed the order of service state check in PropertyDescriptor
Encapsulated the check into private method for readability
Modified and documented test to validate correct behavior.
For more details please see comment in https://issues.apache.org/jira/browse/NIFI-1143
2015-11-11 14:06:08 -05:00
Mark Payne 11fcad90d1 Merge branch 'NIFI-1083' 2015-11-11 12:07:15 -05:00
Mark Payne 054774fe7b Merge branch 'NIFI-1061' of https://github.com/olegz/nifi into NIFI-1061 2015-11-11 11:17:25 -05:00
Mark Payne e7c6c7cae9 NIFI-1083: Added a Grouping Regular Expression property for grouping lines of text 2015-11-11 10:19:07 -05:00
Oleg Zhurakousky a65a29921a NIFI-1124, NIFI-1062 addressed more PR comments
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-11-11 10:12:30 -05:00
Oleg Zhurakousky 70cc61bb53 NIFI-1124, NIFI-1062 addressed PR comments
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-11-11 10:12:30 -05:00
Oleg Zhurakousky 62e3cfc629 NIFI-1024 NIFI-1062 Fixed PutHDFS processor to properly route failures. Ensured that during put failures the FlowFile is routed to 'failure' relationship. Added validation test Re-enabled previously ignored test.
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-11-11 10:12:30 -05:00
Joseph Percivall 3e538d9007 NIFI-1083 Created a processor that routes lines of text based on different matching and routing strategies
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-11 09:12:47 -05:00
Oleg Zhurakousky 76690a8ee9 NIFI-1061 fixed deadlock caused by DBCPConnectionPool.onConfigured()
Current implementation of DBCPConnectionPool was attempting to test if connection could be obtained via dataSource.getConnection().
Such call is naturally a blocking call and the duration of the block is dependent on driver implementation. Some drivers (e.g., Phoenix - https://phoenix.apache.org/installation.html)
attempts numerous retries before failing creating a deadlock when attempt was made to disable DBCPConnectionPool which was still being enabled.

This fix removes the connection test from DBCPConnectionPool.onConfigured() operation returning successfully upon creation of DataSource.
For more details see comments in https://issues.apache.org/jira/browse/NIFI-1061
2015-11-11 08:33:51 -05:00
Mark Payne 6c510fae80 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-11 08:16:38 -05:00
Joseph Percivall d887d8fdf0 NIFI-993 added expression language support to GetHTTP for the filename and URL properties
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-11 08:05:23 -05:00
Tony Kurc ba5b1d837c NIFI-1117 Changed line endings to unix line endings in java source 2015-11-10 18:50:57 -05:00
Tony Kurc 9756e9f126 NIFI-1117 Added unit tests and default behavior for avro files with no records. Also fixed a findbugs warning in SplitAvro (made inner classes static) 2015-11-10 18:46:17 -05:00
Mark Payne e6644b3747 Merge branch 'NIFI-1141' 2015-11-10 17:17:55 -05:00
Mark Payne 7b9c8df6c5 NIFI-994: Fixed issue that could result in data duplication if more than 1 rollover of tailed file has occurred on restart of Processor 2015-11-10 16:53:29 -05:00
Tony Kurc 6ed5f28f26 NIFI-965 penalized failed flowfiles in S3 Processors. Also discovered a typo in a static final that I corrected
Signed-off-by: Mike Moser <mosermw@apache.org>
2015-11-10 15:14:28 -05:00
Mark Payne 72008035b1 Merge branch 'NIFI-1000' 2015-11-10 15:02:02 -05:00
Mark Payne 8f942fcdcc NIFI-1000: Fixed checkstyle violation in TestGetJSMQueue use activemq-broker with test scope and activemq-client with default scope rather than pulling in activemq-all; this reduces size of standard nar from 30 MB to 18 MB 2015-11-10 14:31:20 -05:00
Mark Payne 53725b5c72 Merge branch 'NIFI-1000' of https://github.com/olegz/nifi into NIFI-1000 2015-11-10 13:34:56 -05:00
Oleg Zhurakousky 8699e35108 NIFI-1000 addressed PR comments 2015-11-10 12:13:16 -05:00
Mark Payne 11768cc388 NIFI-1141: Fixed the order of the arguments to the setCurrentContentClaim method 2015-11-10 11:35:14 -05:00
Bryan Bende e748fd5848 NIFI-817 Processors for interacting with HBase
- Refactoring PutHBaseCell to batch Puts by table
- Adding optional Columns property to GetHBase to return only selected column families or columns
- Making GetHBase cluster friendly by storing state in the distributed cache and a local file
- Adding Initial Time Range property to GetHBase
- Adding Filter Expression property and custom validate to prevent using columns and a filter at the same time
- Creating an HBaseClientService controller service to isolate the HBase client and support multiple versions
- Creating appropriate LICENSE/NOTICE files
- Adding @InputRequirement to processors
- Addressing comments from review, moving hbase client services under standard services
- Making sure result of session.penalize() is assinged to FlowFile variable before transferring
2015-11-10 11:13:51 -05:00
Tony Kurc 2a90bd501b NIFI-797 Corrected URL for PutS3 to include correct protocol and endpoint by region.
- corrected a missed 'final' on org.apache.nifi.processors.aws.AbstractAWSProcessor.relationships
 - added protected method org.apache.nifi.processors.aws.AbstractAWSProcessor.getRegion()
 - added protected method org.apache.nifi.processors.aws.s3.AbstractS3Processor.getUrlForObject(String, String)
 - explicitly set AWS client protocol to HTTPS, and created a static final field with comments if other protocols may be considered
 - added a static final field for the UserAgent

Reviewed by Aldrin Piri <aldrin@apache.org>
2015-11-10 06:37:56 -05:00
Oleg Zhurakousky ef0be5a5d6 NIFI-1000 Fixed JmsFactory to properly obtain destiniation name
Re-enabled JMS Tests that were annotated with @Ignore
2015-11-09 18:35:31 -05:00
Mike Moser b75af5b344 Merge branch 'NIFI-1085' 2015-11-09 16:13:12 -05:00
joewitt 41f3875347 NIFI-1134 2015-11-09 15:37:40 -05:00
Mark Payne 73c1671975 NIFI-1132: Limited number of Lineage Identifiers held to 100 and marked the getLineageIdentifiers() method as deprecated 2015-11-09 12:09:56 -05:00
Mark Payne 5f8fdae909 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-09 12:07:45 -05:00
joewitt 64b1fc983c NIFI-925 reviewed patch/ builds well/ fixed licensing gaps. 2015-11-08 22:15:11 -05:00
Mark Payne 59a49aea12 Revert "NIFI-1082: Ensure that events returned from the provenance repository are ordered such that newest events are provided first"
This reverts commit cf8ca3dc2c.
2015-11-08 21:46:06 -05:00
Tony Kurc 440e4c4940 NIFI-1106 fixed platform-specific object separator. 2015-11-08 12:27:41 -05:00
Venkatesh Sellappa 77de51df19 NIFI-1025 Updated aws-sdk and joda-time to resolve AWS issues with Java 1.8 update 60.
- Refactored tests - created AbstractS3Test for common utility methods
 - Corrected incorrect unit test in TestDeleteS3Object, and adjusted processor documentation to reflect behavior
 - moved aws dependency management to root pom

This closes 

Tested, Reviewed and Amended by Tony Kurc (<tkurc@apache.org>)
2015-11-07 21:09:18 -05:00
joewitt 99629646fe NIFI-1122 moved to 0.4.0-SNAPSHOT 2015-11-06 23:41:15 -05:00
Mark Payne 0d6e81b54f Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-06 09:16:12 -05:00
Bryan Bende ee87f33ce7 Fixing check-style violation in StandardFlowFileQueue 2015-11-06 09:10:10 -05:00
Mark Payne 1a388f957e NIFI-1111: We were subtracting values in the wrong order so that we always would 'yield' for a negative amount of time, which equated to not yielding at all. 2015-11-06 09:03:38 -05:00
Mark Payne 98f5a1ab77 NIFI-1110: Fixed bug that caused queue size to become negative when FlowFiles are expired
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-05 12:12:02 -05:00
Bryan Bende 4ac52bfc0a Removing nifi-azure-bundle that was incorrectly added to master 2015-11-05 10:42:10 -05:00
Andrew Grande f1bd3d471d NIFI-1112 Fixed APIOperation Annotation on ReportingTaskResource.
Reviewed by Tony Kurc (tkurc@apache.org)
2015-11-04 18:31:34 -05:00
Bryan Bende 618f22e110 NIFI-274 - Fixing TestListenSyslog, fixing default buffer size to be bytes, adding syslog.protocol to attributes
- Adding syslog.port to ListenSyslog attributes, logging at warn level when rejecting tcp connections
         - Adding @InputRequirement to processors and adding appropriate send and receive provenance events
2015-11-04 18:01:59 -05:00
Tony Kurc 5611dac3f8 NIFI-274 - added use of Selectors for TCP and UDP connections. Added a max connections to the TCP thread
- Added comments and code review changes
         - fixed fixbugs bug
2015-11-04 18:00:18 -05:00
Bryan Bende 9c542432da NIFI-274 Adding ListenSyslog and PutSyslog to standard processors.
- Refactoring connection handling on put side, removing number of buffers from properties and basing it off concurrent tasks for the processor.
- Refactoring some of the TCP handling so it keeps reading from a connection until the client closes it
- Adding an error queue
- Adding a sender field on the syslog event to record the system that sent the message
2015-11-04 17:56:13 -05:00
Mike Moser 6add372bc1 NIFI-1085 WebClusterManager fair locking and HeartbeatMonitoringTimerTask scheduling 2015-11-04 12:19:00 -05:00
Matt Gilman 201eac052b NIFI-1098 - Ensuring we pull the correct attribute from the servlet context. Reviewed by Tony Kurc (tkurc@apache.org) 2015-11-04 12:11:19 -05:00
Mark Payne fedbe40313 NIFI-1103: Added InputRequirement annotation 2015-11-04 11:11:33 -05:00
Mark Payne 2b1d093120 NIFI-1105: Only trigger a processor that requires input to run if data is available for it process 2015-11-04 08:41:10 -05:00
Mark Payne dbf0c7893f NIFI-730: Completed merging of master 2015-11-02 14:31:28 -05:00
Mark Payne e4cebba3c7 Merge branch 'master' into NIFI-730 2015-11-02 14:02:16 -05:00
Joseph Percivall a549621267 NIFI-1077 enabling expression language on ConvertCharacterSet input and output
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-02 11:46:17 -05:00
Mark Payne 814e8b212c Merge branch 'NIFI-1082' 2015-11-02 11:29:52 -05:00
Mark Payne abf2c62884 NIFI-1051: Fixed checkstyle violations 2015-11-02 11:01:50 -05:00
Mark Payne 7a73867c46 Merge branch 'NIFI-1051' of https://github.com/olegz/nifi into NIFI-1051 2015-11-02 10:51:43 -05:00
Joseph Percivall 2ae49026e8 NIFI-883 Fixing issue HandleHttpRequest had with PrimaryNodeOnly scheduling
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-11-02 10:32:14 -05:00
Matt Gilman 5a04021dd7 NIFI-730:
- Updating logic now that original is guaranteed to be non null.
- Always reporting 100% once the drop request has completed.
2015-11-02 08:59:11 -05:00
Mark Payne f5727cfb0f NIFI-730: Ensure that we always populate queue counts when initiating a Drop FlowFile request 2015-11-02 08:36:50 -05:00
Mark Payne ad849c77df NIFI-1090: Fixed log message that was at info level but should have been debug level 2015-11-01 14:37:01 -05:00
Mark Payne b729bf4c19 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-11-01 14:16:54 -05:00
Mark Payne 6e193dffa3 Merge branch 'NIFI-1088' 2015-11-01 14:16:35 -05:00
Joseph Percivall cef7b6c736 NIFI-1080 fleshed out InvokeHttp unit tests
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2015-10-31 00:19:36 -04:00
Mark Payne 9c148da1c2 NIFI-869: Fixed checkstyle violations 2015-10-30 16:09:50 -04:00
Mark Payne ba72452f66 Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/nifi 2015-10-30 16:02:15 -04:00
Oleg Zhurakousky a3d43d23dc NIFI-869 Fixed SimpleProcessLogger to log correct messages Ensured that SimpleProcessLogger correctly interprets Throwable as discussed in JIRA Added tests
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-10-30 15:52:33 -04:00
Oleg Zhurakousky 90aea01350 NIFI-869 Fixed formatting issue Fixed formatting issue with printed error message which only apears when NiFi is cnfigured using Logback. Please see NIFI-869 for more details
Signed-off-by: Mark Payne <markap14@hotmail.com>
2015-10-30 15:52:33 -04:00
Jeremy Dyer eb389cf84c NIFI-1079 Replacing Name for NULL_VALUE_FOR_EMPTY_STRING as I had accidentally removed it.
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-10-30 14:30:56 -04:00
Jeremy Dyer aef0d8fe9b NIFI-1079 Check style and documentation based updates recommended by user bbende on Github pull request
Signed-off-by: Bryan Bende <bbende@apache.org>
2015-10-30 14:30:42 -04:00
Jeremy Dyer 217b1049cf NIFI-1079 Added Destination Property to control if JSON goes to Attribute or
Content of FlowFile. Added Include Core Attributes Property to control
if FlowFile CoreAttributes are included in the JSON output or not.
Added Null value for Empty String flag to control if empty values in
the JSON are empty string or true NULL values. Added more tests and
minor text refactoring per Github comments

Signed-off-by: Bryan Bende <bbende@apache.org>
2015-10-30 14:30:25 -04:00
Jeremy Dyer 19b7a4cc7d NIFI-1079 Create a new Attribute from the existing FlowFile attributes by taking
either all of the existing attributes or a user defined list. The
existing Attributes are converted to JSON and placed in a new Attribute
on the existing FlowFile as Attribute “JSONAttributes”

Signed-off-by: Bryan Bende <bbende@apache.org>
2015-10-30 14:29:29 -04:00
Mark Payne 9515b74607 NIFI-1088: Ensure that FlowFile is penalized before routing to failure 2015-10-30 14:25:27 -04:00