Data Explorer can run arbitrary SQL queries which can be costly for us if over-used. Because of that we want to add the ability to rate limit the query run endpoint, in particular when requested programmatically using API.
This commit introduces a rate limit to the `QueryController#run` endpoint. It heavily leans on the existing `RateLimiter` implementation, and the ability of `ApplicationController` to turn rate limit exceptions into nicely formatted JSON responses.
The rate limit (per 10 seconds) can be configured through the global setting `max_data_explorer_api_reqs_per_10_seconds`, and defaults to 2.
Handling can be configured through `max_data_explorer_api_req_mode`, and can be set to warn, block, or both warn and block. We will default to warn for now and monitor the logs for a while.
This feature enables admins to create reports automatically based on a recurring schedule.
It introduces a new automation script that includes the new email_group_user field added to discourse-automation, along with a query_id and query_params to pass in parameters to the existing data explorer query.
The output of the report will be sent via pm (as a markdown table) to the recipients entered within the automation script.
The automation (supports individual users, email addresses and groups).
This commit updates the plugin to the latest guidelines, as shown in
discourse-plugin-skeleton, which involves moving a lot of the code to
dedicated files, use proper namespaces, use the autoloader as much as
possible, etc.
- Require query name is present
- Ensure all routes are treated by default as .json, so errors flow correctly
- Remove superflous save/cancel controls from group settings
- Remove group control when item is destroyed
- Disable editing of query when it is deleted
Co-authored-by: Osama Sayegh <asooomaasoooma90@gmail.com>
Before this fix, the use of PG template patterns containing ":" or the
use of "?" in comments in the SQL will result in an error being raised
because `DB.param_encoder.encode` calls ActiveRecord's `sanitize_sql_array` which is
meant for SQL fragments and not an entire SQL string.
Instead we change data-explorer to use `MiniSql::InlineParamEncoder`
instead which takes into account of template patterns and does not trip
on `?` which is a special param encoding character used by ActiveRecord.
Previously the `QUERY_RESULT_DEFAULT_LIMIT` const was used
to limit the number of query results. This commit adds the
`data_explorer_query_result_limit` site setting which defaults
to 1000 and has a max of 10000 which matches the const
`QUERY_RESULT_MAX_LIMIT`.
1. It doesn't seem to be necessary?
2. It looks like it was triggering some warnings (`Creating scope :for_group. Overwriting existing method DataExplorer::Query.for_group.`)
Previously this was adding an extra AJAX request to check if the group had any queries available. Now a boolean is included in the group serializer, so there is no need for the extra request.
Removing this ajax request will also stop other plugin JS integration tests from failing when the data-explorer plugin is installed.
This commit also fixes the HTML markup of the tab, so that it doesn't have a <ul> nested inside the existing <ul>. Also adds an icon for good measure.
* FEATURE: Add hide button (toggleable) for all queries (frontend only)
* Switches between hide/unhide on click
* Works almost like the delete button, but toggles between the query's
hidden attribute instead
* So far this is only a frontend feature, the backend implementation
still needs work
* Revert "FEATURE: Add hide button (toggleable) for all queries (frontend only)"
This reverts commit a8771d2ad5.
REVERT: Remove button that hides queries (frontend)
* Prepare for migration of old frontend logic to backend
* We are going to reuse the existing delete button, but change its
backend logic to enable soft deletion. From the user's perspective
nothing will change, but any deletion mistakes can be reverted.
* DEV: Hide user queries upon deletion, but keep them in store
* Creating a new query will set its hidden attribute to false by
default
* Deleting a user-made query will not delete it from the store, but
set its hidden attribute to true
* User queries will not be indexed if they are hidden
* Undeleting a query will unhide it, and will be indexed
* Updating a hidden query will unhide it, and will be indexed
* SPEC: Add spec for hidden/deleted queries
* Hidden queries should not be shown
* FEATURE: Add ability to delete/hide system queries
* System queries are now able to be deleted from view, but will remain
in the backend for retrieval, if necessary
* FEATURE/DEV: Add rake commands for query soft deletion
* query:list_hidden - Shows a list of hidden queries
* query:hide_all[only_default] - Hides all queries, w/ boolean arg to
hide only default ones
* query:unhide[id] - Unhides a query by id
* query:unhide_all[exclude_default] - Unhides all hidden queries,
w/ boolean arg to exclude default ones
* Remove rails loggers
* UX/DEV: Update query rake tasks to be more user friendly
* Split query:hide_all[only_default] into two tasks:
* query:hide_all - Hides all queries
* query:hide_all:only_default - Hide only default queries
* Split query:unhide_all[exclude_default] into two tasks:
* query:unhide_all - Unhides all hidden queries
* query:unhide_all:exclude_default - Unhides all non-default
queries
* Rename file to match task name
* UX: query:unhide can accept multiple arguments
* Example: rake query:unhide[-5,-6,-7,3,5,6,-13,-14,-10]
* UX: Update query rake tasks to output cleaner messages
* Remove unneeded comment
* DEV: Keep only necessary rake tasks, use more specific naming
* UX/DEV: Add rake task for hard deletion, better console logs
* User is able to hard delete a query only if it is hidden, otherwise
output a message stating so
* Add commented examples above each task
* Add rainbow support for more readable console logs
* Successful messages will display green, failures display red,
additional info displays yellow
* Separate multiple queries with spaces instead of lines
* DEV: Remove rainbow colorizing in console logs
* Rainbow is a dependency of rubocop and it may go away in the future
* Rainbow is only used for dev and test environments
* DEV: Add Rails engine to enable rake tasks to be loaded at runtime
* DEV: Favor require - load files only if they are not already loaded
* SPEC: Add tests for data_explorer[id] rake command
* Test if a single query is hidden correctly
* Expect length of query list to not be modified
* Expect array of hidden queries to have exactly 1 element
* Expect that one element to have the same ID as the one invoked to
be hidden
* Test if multiple queries are hidden correctly
* Expect length of query list to not be modified
* Expect array of hidden queries to have the number of elements
equal to the number invoked to be hidden
* Expect the elements to have the same ID as the ones invoked to be
hidden
* Test if a query exists in PluginStore
* Expect query list to be empty
* DEV: Clear pre-existing tasks before redefining
* This prevents double invocation when user invokes the task once
* SPEC: Add tests for unhide_query rake task
* Test if a single query unhides correctly
* Expect length of query list to not be modified
* Expect array of hidden queries to have exactly 1 element after
unhiding 1 of 2 queries
* Expect remaining element to be hidden
* Test if multiple queries unhide correctly
* Expect length of query list to not be modified
* Expect array of hidden queries to have exactly 1 element after
unhiding 3 of 4 queries
* Expect remaining element to be hidden
* Test if a query exists in PluginStore
* Expect query list to not be modified
* SPEC: Add tests for hard_delete rake task
* Test if a single query hard deletes correctly
* Expect length of query list to be shorter by 1
* Expect array of hidden queries to have exactly 1 element after
hard deleting 1 of 2 queries
* Expect 1 remaining hidden element
* Test if multiple queries hard delete correctly
* Expect length of query list to be shorter by 3 after hard deleting
3 of 4 queries
* Expect array of hidden queries to have exactly 1 element after
hard deleting 3 of 4 queries
* Expect 1 remaining hidden element
* Test if a query exists in PluginStore
* Expect hidden query list to not be modified
* Test if a query is not hidden
* Expect query list to not be modified
* UX: Favor newline char in place of puts for logs
* Condensed console logs to output newline char instead of another puts
statement (reduces number of lines used significantly)
* [WIP] group ids saving on new reports
* Add groups to default queries, and added tab connector
* group_ids set to empty array for default queries
* group reports route (in & and) action
* [WIP] created group reports show route/controller
* Find correct query in show route
* Removed empty array for group_ids in query file
* Add report show view, where users can run queries
* Removed unneeded commas from queries.rb
* Allow non-admin group members to access reports
* query-result component dynamic download url based on location
* Removed accidental changes, and corrected tab size
* Group members can add params to queries
* Specs for new QueryController actions
* remove "Inlude query plan" from group reports
* Run prettier
* return and return -> return render
Co-Authored-By: Robin Ward <robin.ward@gmail.com>
* [WIP] changes from review
* Remove weird [-1] group_ids logic, for a simply check for [] in query update action
* Added integration tests for group report access
* Using guardian for securing endpoints, and much improved specs
* Update assets/javascripts/discourse/components/group-reports-nav-item.js.es6
Co-Authored-By: Robin Ward <robin.ward@gmail.com>
This new feature means that if you name a column `xyz_url` then the return
value will be treated as a URL.
If you would like to name the link use the format `name,https://cnn.com`
The extra data is needed to render avatars and user metadata when rendering results in the browser,
we can skip this data for JSON exports to reduce bloat
Fixes a bug that prevented returning unlimited rows.
Example request:
{{base_url}}/admin/plugins/explorer/queries/6/run?api_key={{api_key}}&api_username={{api_username}}&limit=ALL
From the PostgreSQL docs: "LIMIT ALL" is treated as no limit
https://www.postgresql.org/docs/current/static/sql-select.html#SQL-LIMIT
Ted Johansson
David Battersby
Bianca Nenciu
Martin Brennan
David Taylor
Sam
Alan Guo Xiang Tan
Frank
Jarek Radosz
Osama Sayegh
Martin Brennan
Andrew Prigorshnev
tshenry
Kane York
Krzysztof Kotlarek
Mark VanLandingham
Ricky Chon
Vinoth Kannan
Arpit Jalan
Penar Musaraj
Justin DiRose
Joffrey JAFFEUX
romanrizzi
Mark VanLandingham
Guo Xiang Tan
Robin Ward
Rishabh
Rishabh