Commit Graph

2378 Commits

Author SHA1 Message Date
Oleg Zhurakousky 2d03489ec5 NIFI-1296, NIFI-1680, NIFI-1764, NIFI-1837, NIFI-1827, NIFI-1699 implemented new Kafka processors that leverage Kafka 0.9 API
- Improved StreamScanner for better performance
- Renamed StreamScanner to StreamDemarcator as suggested by Joe
- Added failure handling logic to ensure both processors can be reset to their initial state (as if they were just started)
- Provided comprehensive test suite to validate various aspects of both Publish and Consume from Kafka
- Added relevant javadocs
- Added initial additionalDetails docs
- Addressed NPE reported by NIFI-1764
- Life-cycle refactoring for the existing PutKafka to ensure producer restart after errors
- Incorporated code changes contributed by Ralph Perko (see NIFI-1837)
- Addressed partition issue in RoundRobinPartitioner discussed in NIFI-1827
- Updated PropertyDescriptor descriptions to reflect their purpose

NIFI-1296 added @Ignore on some Kafka tests to improve test time

NIFI-1296 reworked tests to avoid dependency on embedded Kafka

NIFI-1296 fixed spelling error

NIFI-1296 fixed trailing whitespaces in non-java files

This closes #366
2016-05-19 07:33:03 -04:00
Matt Burgess c13fb3159f
NIFI-1822 Enabled pooled ExecuteScript engines to facilitate concurrent processing.
Removed unused variable in unit test. (+10 squashed commits)
Squashed commits:
[7c5acc1] NIFI-1822 Removed trailing whitespace to conform with checkstyle rules.
[cb108cd] NIFI-1822 Added ASF License to unit test.
[9264428] NIFI-1822 Removed debugging log statements for script engine queue size.
Added unit tests demonstrating pooled execution timing and thread usage.
[bdbc4ba] NIFI-1822 Renamed reference to MockProcessorContext#setNumThreads to setMaxConcurrentTasks after refactor.
[12bbe82] NIFI-1822 Moved unit test to correct directory.
Added test script resource which generates flowfile and updates attribute with current thread.
Added tests for single run, serial run, and pooled run (not complete).
[4c174c8] NIFI-1822 Added debugging messages to script execution.
[8ab0ce5] NIFI-1822 Added for loop to instantiate multiple script engines in queue.
[8c5ba51] NIFI-1822 Added variable max concurrent task field in MockProcessorContext because it was previously hardcoded to 1. Changed setNumThreads to setMaxConcurrentTasks to maintain naming convention.
[fd9120c] NIFI-1822 Added unit test skeleton for pooled script processor execution.
[23e4f68] NIFI-1822: Allow concurrent execution in ExecuteScript

This closes #443.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-05-17 14:58:00 -07:00
Mark Payne a81c204565 NIFI-1892: Marked method arguments as final so that they can be referenced in inner classes 2016-05-17 13:31:50 -04:00
Bryan Bende 17c5496c04 NIFI-1858 Adding site-to-site reporting bundle
This closes #436.
2016-05-17 10:38:47 -04:00
Matt Gilman 8bb56fca68 NIFI-1554:
- Introducing ControllerService and ReportingTask authorization and revision.
- Removing the revision where appropriate.
NIFI-1883
- Restoring referencing components automatically.
- This closes #448
2016-05-17 10:32:55 -04:00
rkarthik29 cfd36c5539 fix for 1864
Signed-off-by: Matt Gilman <matt.c.gilman@gmail.com>
2016-05-17 08:38:26 -04:00
Andy LoPresto 3070bd5e13
Added PGP key to KEYS file.
This closes #450.
2016-05-16 20:25:38 -07:00
Pierre Villard fa390112ce NIFI-1213 Added the possibility to register FlowFile assertions in mock framework
This closes #404
2016-05-16 18:58:15 -04:00
Mark Payne 270944ec69 NIFI-1801: Scope Templates to Process Groups. This closes #446. 2016-05-16 16:12:43 -04:00
James Wing 347b281b2d NIFI-1887 Updating default timeout in Admin Guide
This closes #447.

Signed-off-by: James Wing <jvwing@gmail.com>
2016-05-16 11:57:33 -07:00
Pierre Villard b2ce6f1f96 NIFI-1755 Fixed remote process group status counts by only considering connected remote ports
This closes #347
2016-05-16 13:37:51 -04:00
Mark Payne 87d96c0225 NIFI-1872: Allow sufficient time for embedded server to startup for unit tests 2016-05-13 13:28:38 -04:00
Mark Payne 707aeb4231 NIFI-1872: Ignored unit test that sporadically fails 2016-05-13 11:27:54 -04:00
Matt Gilman 3cc16d35ed NIFI-1742:
- Addressing issues when creating a new inline controller service.
- Ensuring controller service referencing components are updated.
- Including revisions and status with each component.
- Dynamically updating component and authorization states.
- This closes #435
2016-05-12 12:41:11 -04:00
Mark Payne 687a686b21 NIFI-1872: Ignore failing unit test for now until we can properly address; since it is on master it's best to ignore, since the problem appears to be the lifecycle of the unit test 2016-05-12 11:57:40 -04:00
Scott Aslan 3c3304aff4 [NIFI-1782] new search UX and add angular control of flow status values. This closes #433 2016-05-11 16:39:38 -04:00
Mark Payne 25e7f314b1 NIFI-1800: Tie Controller Services to Process Groups. This closes #431 2016-05-11 14:56:05 -04:00
Pierre Villard b7aa381ab4 NIFI-1826 Updated documentation
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #396
2016-05-11 10:02:11 -04:00
Pierre Villard 7910e98ba1 NIFI-1826 Expression Language: add function to check enumerator
Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-05-11 10:01:15 -04:00
Koji Kawamura 7bd2c64adb NIFI-1865 Close StreamThrottler when processor stops.
- Also, replaced copyrighted sample texts for tests

Signed-off-by: Matt Burgess <mattyb149@apache.org>
2016-05-10 22:02:15 -04:00
Yolanda M. Davis 1370eefdd7
NIFI-1812 initial commit for nifi-env script, includes updates to linux/windows scripts, logback changes, RunNifi/Shutdownhook changes
This closes #386.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-05-10 16:37:30 -07:00
Pierre Villard 887648c86f NIFI-1863 extended caught exception in cleaning task for HTTPContextMap
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #428
2016-05-10 13:46:40 -04:00
Scott Aslan cb67520129 [NIFI-1707] upgradeable angular components 2016-05-09 15:16:12 -04:00
Andrew Lim dc4f983c7a
NIFI-1862 User Guide corrections/improvements
Made multiple edits to the User Guide documentation for correcting errors (spelling/grammatical) and improving readability.

This closes #427.

Signed-off-by: Andy LoPresto <alopresto@apache.org>
2016-05-09 10:31:33 -07:00
ijokarumawak 5fb27e608f NIFI-1860 Added ContainerRequestFilter to redirect. This closes #422
- redirect /controller to /site-to-site
2016-05-09 08:17:46 -04:00
James Wing 257590dec3 NIFI-1711 Client-side JS for proxy-friendly URLs. This closes #316 2016-05-08 21:32:34 -04:00
Jeremy Dyer 7a5d53b8d3 NIFI-1747 add maven-jar-plugin to create jar as part of the existing nifi-web-api build process
This closes #340.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-05-08 14:31:25 -04:00
Aldrin Piri b4e4f3eed5 Removing erroneously included files. 2016-05-07 15:01:24 -04:00
Pierre Villard 9064b97631 NIFI-1816 Added provenance event SEND in HandleHttpResponse
This closes #389
2016-05-07 11:51:26 -04:00
Aldrin Piri 993d3cd78f NIFI-1859 Using .equals instead of == for default locale check. Adding some additional information for the input and output schema properties.
This closes #418.

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-05-06 23:04:11 -04:00
Pierre Villard 63a0d0c211 NIFI-1732 set jetty timeout to Long.MAX_VALUE
This closes #337
2016-05-06 19:38:52 -04:00
jpercivall e5925ff4b3 Fixing contrib check 2016-05-06 18:02:44 -04:00
Pierre Villard e2bbea1c53 NIFI-1814 Extended caught exceptions
This closes #382
2016-05-06 18:00:40 -04:00
Matt Gilman 9db1def6c6 NIFI-1781:
- Incorporating updated styles to reflect component level authorization.
- Updating canvas for new look and feel.
- This closes #417
2016-05-06 15:50:06 -04:00
Mark Payne 04c41c0654 NIFI-1678:
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Ensure that ACL is set properly when creating heartbeat node. Removed unused ControllerStartupFailureMessage.java
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator
- Refactored CuratorHeartbeatMonitor into an abstract heartbeat monitor that is responsible for processing heartbeats and CuratorHeartbeatMonitor that is responsible for retrieving heartbeat information
- Refactored so that heartbeats are sent to Cluster Coordinator directly instead of to ZooKeeper. ZooKeeper is used to know which node is the cluster coordinator but heartbeats to the Cluster Coordinator provide additional information about the nodes.
- Started refactoring heartbeating mechanism, using a new package: org.apache.nifi.cluster.coordination
- Added configuration options for ZooKeeper username & password for heartbeat management. Also addressed issue where nodes that were previously disconnected were asked to disconnect upon restart
- Changed ZooKeeper ACL's so that container nodes that would not be sensitive are wide open and removed the usage of username & password when communicating with ZooKeeper. This was done specifically because username/password combination is considered a 'testing' feature that should not be used in production and is not supported by Apache Curator

NIFI-1727:
- Refactored logic for merging HTTP Requests that are federated across cluster

NIFI-1745:
- Refactoring how HTTP Requests are replicated to nodes
- Bug fixes and continuing to work on replication refactoring. Still need to handle cluster locking and revisions
- Begin work on RevisionManager
- Resolved some issues that resulted from rebase
- Fixed URIs to align with new URI's that will be used in 1.0.0
- This closes #413
2016-05-06 15:23:12 -04:00
ijokarumawak 3db14f58fe NIFI-1773 Removed unused import statement. 2016-05-06 14:35:16 +09:00
Scott Aslan c527f64230 [NIFI-1762] Remove bower and npm from convenience binaries. This closes #412. 2016-05-04 15:23:11 -04:00
Pierre Villard 5999eab43f NIFI-1759 Upgrade of commons-compress
This closes #407
Checked:
- no regression
- no use of deprecated methods
2016-05-04 14:23:10 -04:00
Adam Lamar ddf8c6f8fa NIFI-840: Create ListS3 processor
This closes #238

Signed-off-by: Aldrin Piri <aldrin@apache.org>
2016-05-04 12:15:45 -04:00
Pierre Villard 46f8693bc6 NIFI-1773 NIFI-1774 fixed typo/annotation
NIFI-1774 undid the change from OnConfigured to OnEnabled
This closes #406
2016-05-04 11:44:09 -04:00
Scott Aslan 2824f5a23c NIFI-1782 cleanup whitespace and code styles
This closes #411
2016-05-03 15:32:49 -04:00
Pierre Villard 194fc048fb NIFI-1721 updated mock framework to return correct number of threads from mock context
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #408
2016-05-03 15:09:29 -04:00
Yolanda M. Davis defb6f5b61 NIFI-361 - Create Processors to mutate JSON data
Signed-off-by: Matt Burgess <mattyb149@apache.org>

This closes #354
2016-05-03 14:38:11 -04:00
Mark Payne e3bdee8b1e NIFI-1824: If attempting to archive content, and there are no claimant counts for it, ensure that the stream is closed. 2016-05-03 14:27:13 -04:00
Scott Aslan 1df8fe44c4 [NIFI-1782] update toolbox, new global menu, and new pallattes...also introduced frontend-maven-plugin to allow bower to manage pulling in 3rd party libs at build time and not shipped with the source. This closes #395 2016-05-03 14:20:46 -04:00
Aldrin Piri 50c02ee459 NIFI-1654 Creating an initial appveyor.yml file. Adding appveyor.yml to the list of excluded items for RAT check.
This closes #343.
2016-05-03 14:19:28 -04:00
jpercivall 0557095613 NIFI-1028 initial commit of NiFi In Depth documentation
NIFI-1028 addressing review comments
This closes #339
2016-05-03 14:01:45 -04:00
Matt Burgess 106b0fa0fc NIFI-981: Added SelectHiveQL and PutHiveQL processors
This closes #410.

Signed-off-by: Bryan Bende <bbende@apache.org>
2016-05-03 13:51:38 -04:00
Devin Fisher 8336014064 NIFI-1560:
- Fixing a copy and paste error
- Looks like when the original coder copied code from AuthenticationStrategy for the ReferralStrategy and did not change this reference for the error case.
- Also addressing the typo in the message.
- This closes #402
2016-05-03 08:00:34 -04:00
Mark Payne 45ca978498 NIFI-1830: Fixed problems in the merging logic for Drop FlowFile Requests. This closes #394 2016-05-02 08:46:50 -04:00