Commit Graph

10939 Commits

Author SHA1 Message Date
Abhishek Agarwal 1a15987432
Supporting filters in the left base table for join datasources (#10697)
* where filter left first draft

* Revert changes in calcite test

* Refactor a bit

* Fixing the Tests

* Changes

* Adding tests

* Add tests for correlated queries

* Add comment

* Fix typos
2021-03-04 10:39:21 -08:00
Atul Mohan 6040c30fcd
Upgrade jetty to latest version (#10937)
* Upgrade jetty

* Fix license
2021-03-04 08:28:50 -06:00
Atul Mohan be2ac8d6ce
Document type inference issues with dynamic params in SQL (#10801)
* Clarify docs

* Apply suggestions from code review

Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>

Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>
2021-03-04 03:48:11 -08:00
Gian Merlino 87a2abff79
Fix runtime error when IndexedTableJoinMatcher matches long selector to unique string index. (#10942)
* Fix runtime error when IndexedTableJoinMatcher matches long selector to unique string index.

The issue arises when matching against a long selector on the left-hand side to a string
typed Index on the right-hand side, and when that Index also returns true from areKeysUnique.
In this case, IndexedTableJoinMatcher would generate a ConditionMatcher that implements
matchSingleRow by calling findUniqueLong on the Index. This is inappropriate because the Index
is actually string typed. The fix is to check the type of the Index before deciding how to
implement the ConditionMatcher.

The patch adds "testMatchSingleRowToUniqueStringIndex" to IndexedTableJoinMatcherTest, which
explores this case.

* Update tests.
2021-03-04 00:57:59 -08:00
Jihoon Son d6e591220b
Add instructions for updating licenses (#10894)
* Add instructions for updating licenses

* updating license

* pull request template
2021-03-03 00:57:09 -08:00
Jihoon Son 8831c0d057
Update PR template to include the contributing doc; suggestion to not use force push (#10769)
* Update PR template to include the contributing doc; suggestion to not use force push

* make it a comment

* moved it to top

* address review comment
2021-03-02 22:26:49 -08:00
Maytas Monsereenusorn 23333914c7
add javadoc and test (#10938) 2021-03-03 11:34:00 +08:00
Abhishek Agarwal 7d9a61cf7f
Suppress CVE-2017-15288 and upgrade bcprov-ext-jdk15o (#10933) 2021-03-02 16:18:27 -08:00
Maytas Monsereenusorn b7b0ee8362
Add query granularity to compaction task (#10900)
* add query granularity to compaction task

* fix checkstyle

* fix checkstyle

* fix test

* fix test

* add tests

* fix test

* fix test

* cleanup

* rename class

* fix test

* fix test

* add test

* fix test
2021-03-02 11:23:52 -08:00
Gian Merlino 05e8f8fe06
CsvInputFormat: Create a parser per InputEntityReader. (#10923)
RFC4180Parser is not thread safe and cannot be shared across readers.
2021-02-27 18:37:05 -08:00
spinatelli 99198c02af
Add config and header support for confluent schema registry. (#10314)
* Add config and header support for confluent schema registry. (porting code from https://github.com/apache/druid/pull/9096)

* Add Eclipse Public License 2.0 to license check

* Update licenses.yaml, revert changes to check-licenses.py and dependencies for integration-tests

* Add spelling exception and remove unused dependency

* Use non-deprecated getSchemaById() and remove duplicated license entry

* Update docs/ingestion/data-formats.md

Co-authored-by: Clint Wylie <cjwylie@gmail.com>

* Added check for schema being null, as per Confluent code

* Missing imports and whitespace

* Updated unit tests with AvroSchema

Co-authored-by: Sergio Spinatelli <sergio.spinatelli.extern@7-tv.de>
Co-authored-by: Sergio Spinatelli <sergio.spinatelli.extern@joyn.de>
Co-authored-by: Clint Wylie <cjwylie@gmail.com>
2021-02-27 14:25:35 -08:00
Charles Smith 573de3bc0d
clarify security requirements around HTTPInputSource (#10914)
* clarify security requirements around HTTPInputSource

* explicitly mention write/datasource in best practices. clarify that the ingestion task is the risk

* Update docs/operations/security-overview.md

Co-authored-by: Suneet Saldanha <suneet@apache.org>

Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-02-26 09:37:47 -08:00
Alexander Saydakov f930cf14d6
Use the latest Apache DataSketches release 2.0.0 (#10917)
* use the latest Apache DataSketches release 2.0.0

* updated datasketches version

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2021-02-26 07:52:00 -06:00
Gian Merlino 07902f607b
Granularity: Introduce primitive-typed bucketStart, increment methods. (#10904)
* Granularity: Introduce primitive-typed bucketStart, increment methods.

Saves creation of unnecessary DateTime objects in timestamp_floor and
timestamp_ceil expressions.

* Fix style.

* Amp up the test coverage.
2021-02-25 07:59:20 -08:00
zachjsh 67eff4110d
Improve Druid ldap auth documentation (#10915)
* Improve Druid ldap auth documentation

Improved the ldap auth docs by clarifying that the object classes and
attributes noted are specific to Microsoft Active Directory, and could
be different depending on the specific ldap server being used. Also
emphasized the importance of the memberOf field and noted that the
step about adding users to roles is only needed in certain circumstances.

* * add another note

* Apply suggestions from code review

Co-authored-by: sthetland <steve.hetland@imply.io>

* * simplify

* * Address review comments

Co-authored-by: sthetland <steve.hetland@imply.io>
2021-02-24 15:28:41 -08:00
Clint Wylie 0ecc90142e
basic security extension ignore permissions that use unknown ResourceType or Action (#10896)
* suppress unknown ResourceType and Action for basic-security authorizer stuff

* fix pom

* print failed role, test logs
2021-02-23 14:49:09 -08:00
zachjsh 553f5c8570
Ldap integration tests (#10901)
* Add integration tests for ldap extension

* * refactor

* * add ldap-security integration test to travis

* * fix license error

* * Fix failing other integration test

* * break up large tests
* refactor
* address review comments

* * fix intellij inspections failure

* * remove dead code
2021-02-23 13:29:57 -08:00
Clint Wylie f34c6eb3c0
add druid jdbc handler config for minimum number of rows per frame (#10880)
* add druid jdbc handler config for minimum number of rows per frame

* javadocs and docs adjustments

* spelling

* adjust docs per review with minor tweaks

* adjust more
2021-02-23 02:11:04 -08:00
Abhishek Agarwal 3a0a0c033f
Reload segment usage when starting the process (#10884)
* Reload segment usage when starting the process

* doc

* Add more tests

* remove forbidden method

* Add alert
2021-02-22 00:08:44 -08:00
Gian Merlino b7e9f5bc85
BoundDimFilter: Simplify the various DruidLongPredicates. (#10906)
They all use Long.compare, but they don't need to. Changing to
regular comparisons simplifies the code and also removes branches.
(Internally, Long.compare has two branches.)
2021-02-19 16:44:56 -08:00
Maytas Monsereenusorn f5bfccc720
Fix maxBytesInMemory for heap overhead of all sinks and hydrants check (#10891)
* fix maxBytesInMemory

* fix maxBytesInMemory check

* fix maxBytesInMemory check

* fix test
2021-02-18 21:48:57 -08:00
Clint Wylie cbbef80c7f
add SQL operators for bitwise expressions (#10823)
* add SQL operators for bitwise expressions

* more test

* fix spelling

* more tests
2021-02-18 20:56:33 -08:00
Jonathan Wei 84341737d5
Add property for binding view manager type (#10895)
* Add property for binding view manager type

* Checkstyle

* Fix constructor

* Add @Test
2021-02-18 15:57:45 -08:00
Agustin Gonzalez eabad0fb35
Keep query granularity of compacted segments after compaction (#10856)
* Keep query granularity of compacted segments after compaction

* Protect against null isRollup

* Fix bugspot check RC_REF_COMPARISON_BAD_PRACTICE_BOOLEAN & edit an existing comment

* Make sure that NONE is also included when comparing for the finer granularity

* Update integration test check for segment size due to query granularity propagation affecting size

* Minor code cleanup

* Added functional test to verify queryGranlarity after compaction

* Minor style fix

* Update unit tests
2021-02-18 01:35:10 -08:00
Vadim Ogievetsky 1a4c43f9fd
Web console: remove namespace prop that does not exist from JDBC lookup (#10888)
* remove namespace prop that does not exist from JDBC lookup

* remove namespace from tests
2021-02-17 17:07:32 -08:00
Benedict Jin 32e801ceab
Bump Apache Parquet from 1.11.0 to 1.11.1 (#10889) 2021-02-17 12:18:17 +08:00
Suneet Saldanha bc7004006f
Update dependency-check plugin (#10883)
* Use dependency-check aggregate

* oops
2021-02-16 19:22:04 -08:00
Jonathan Wei 8ad68135c8
Filter unauthorized views in InformationSchema (#10874)
* Filter unauthorized views in InformationSchema

* Use fixed name for view schema

* Remove unused string
2021-02-16 17:36:45 -08:00
sthetland 1e40f51e65
Fix example names of security artifacts in docs (#10882)
* replacing example names

* unrelated typos

* unintended changes

* a few more typo fixes
2021-02-16 14:58:50 -08:00
Will Xu c8d2654605
Use native git for git-commit-id-plugin to speed up build (#10881)
* Segment timeline doesn't show results older than 3 months

* Adoption testing patch for web segment timeline view and also refactoring default time config

* Changing git-commit-id-plugin to use native git, shaving off 15% off build time

Co-authored-by: dev <dev@dev.minitoken.com>
2021-02-12 09:31:07 -08:00
Maytas Monsereenusorn 6541178c21
Support segmentGranularity for auto-compaction (#10843)
* Support segmentGranularity for auto-compaction

* Support segmentGranularity for auto-compaction

* Support segmentGranularity for auto-compaction

* Support segmentGranularity for auto-compaction

* resolve conflict

* Support segmentGranularity for auto-compaction

* Support segmentGranularity for auto-compaction

* fix tests

* fix more tests

* fix checkstyle

* add unit tests

* fix checkstyle

* fix checkstyle

* fix checkstyle

* add unit tests

* add integration tests

* fix checkstyle

* fix checkstyle

* fix failing tests

* address comments

* address comments

* fix tests

* fix tests

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test

* fix test
2021-02-12 03:03:20 -08:00
misqos e684b83e29
Add the ability to supply client certificate to dsql comand line tool. (#10765) 2021-02-11 20:16:47 -08:00
zachjsh 64774037c1
Add config option to specify zk version in integration tests (#10870)
* Update integration-tests README

Updated the integration-tests README file to include instructions
for setting the `ZK_VERSION` property which is now required to be
set prior to executing any integration test. Also added a note
about the importance of setting the test group parameter when
running integration tests, even when running single tests.

* * revert change made to DOCKER_IP doc

* * Add default value for zk version

* * update travis config to use new zk.version property when running
  integration tests

* Remove doc about needing to set ZK_VERSION variable when running
  integration tests
2021-02-11 10:31:49 -08:00
Abhishek Agarwal 9526fd38db
Exclude redundant jars from integration-tests build (#10878)
* Exclude redundant jars from integration-tests build

* changes
2021-02-10 23:53:34 -08:00
Abhishek Agarwal 8718155f8f
Allow for empty keys in hash map (#10869)
* allow for empty keys in hash map

* fix serde test
2021-02-10 11:19:57 -08:00
Jihoon Son 1ec3f0bd73
Revert "Add support for Blacklisting some domains for HTTPInputSource (#10535)" (#10871)
This reverts commit 6b14bdb3a5.
2021-02-09 17:51:26 -08:00
Charles Smith 41a0c1d4b5
reword appendToExisting for clarity. add failure behavior (#10748) 2021-02-08 13:00:01 -08:00
Suneet Saldanha d9c8fa8f42
Better stages for travis (#10815) 2021-02-05 17:50:29 -08:00
Egor Riashin 5d8a69ecf9
fixed input source sampler buildReader exp (#10651)
* fixed input source sampler buildReader exp

* sampler exception unit test

* styling

Co-authored-by: egor-ryashin <egor.ryashin@rilldata.com>
2021-02-05 13:33:55 -08:00
Clint Wylie fe30f4b414
refactor sql lifecycle, druid planner, views, and view permissions (#10812)
* before i leaped i should've seen, the view from halfway down

* fixes

* fixes, more test

* rename

* fix style

* further refactoring

* review stuffs

* rename

* more javadoc and comments
2021-02-05 12:56:55 -08:00
Agustin Gonzalez 3785ad5812
Add log message when local input's filter does not match any files (#10837)
* Add log message when local input's filter does not match any files

* Re-use previously defined fileIterator
2021-02-05 11:35:19 -06:00
Jihoon Son ac41e41232
Update doc for query errors and add unit tests for JsonParserIterator (#10833)
* Update doc for query errors and add unit tests for JsonParserIterator

* static constructor for convenience

* rename method
2021-02-05 02:55:32 -08:00
Abhishek Agarwal 96d26e5338
Fix kinesis ingestion bugs (#10761)
* add offsetFetchPeriod to kinesis ingestion doc

* Remove jackson dependencies from extensions

* Use fixed delay for lag collection

* Metrics reset after finishing processing

* comments

* Broaden the list of exceptions to retry for

* Unit tests

* Add more tests

* Refactoring

* re-order metrics

* Doc suggestions

Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>

* Add tests

Co-authored-by: Charles Smith <38529548+techdocsmith@users.noreply.github.com>
2021-02-05 02:49:58 -08:00
Jihoon Son a3af2ad2bd
Clearer instructions to update website for release (#10819)
* Clearer instructions to update website for release

* more about branches

* Update distribution/asf-release-process-guide.md

Co-authored-by: Clint Wylie <cjwylie@gmail.com>

Co-authored-by: Clint Wylie <cjwylie@gmail.com>
2021-02-04 18:40:05 -08:00
Jihoon Son a2b5e01142
Bump DataSketches memory to 1.3.0 (#10789) 2021-02-04 18:39:52 -08:00
Jihoon Son 6c24b6f023
Build distribution separately (#10841) 2021-02-04 15:01:47 -08:00
Jihoon Son 3f8f00a231
Fix CVE-2021-25646 (#10818) 2021-02-04 11:21:43 -08:00
Jihoon Son 397e7455ba
Increase heap to 64m for custom node (#10846) 2021-02-03 16:23:19 -08:00
Jihoon Son ad946559bf
Suppress CVE-2020-9492 for hadoop-mapreduce-client-core (#10847) 2021-02-03 15:54:25 -08:00
Jonathan Wei a1a49811d9
Address CVE-2020-8570, suppress CVE-2020-8554 (#10826)
* Address CVE-2020-8570, suppress CVE-2020-8554

* Update licenses.yaml
2021-02-03 15:17:06 -08:00