Commit Graph

10065 Commits

Author SHA1 Message Date
mcbrewster 8924d285dc Web-Console: add more side column functions to query view (#8283)
* small fixes

* run jest -u

* add trim

* add to groupby

* save

* add functionality to column tree menu

* remove history dialog

* rename file

* fixes

* add new values to time-menu

* add query dialog

* add more functions to collumn-tree menu

* add menu functions

* remove test

* update tests, fix extra 0s, clear in between time filters

* fixes

* add deffered

* save

* add alias's

* change menu style

* fix substring

* small fixes

* add auto run to run button

* move divider to a reasonable position

* fix auto expland

* run jest -u

* fix auto run

* prevent aggregate cooumn changes

* fix group by bug

* add quertAst to state

* ui fixes

* prevent filter on non existant ast

* add auto run to shows

* use handle run

* handle *

* add alias and tslint fix
2019-08-15 01:22:25 -07:00
Clint Wylie ff26fcb159 add copyright info back to NOTICE and NOTICE.BINARY (#8298)
* add Apache copyrights back to NOTICE per https://lists.apache.org/thread.html/4e296f3ede34aa5aeb0b2a8ed0f3540b320bce07936aa3c8bfa135b3@%3Cgeneral.incubator.apache.org%3E

* fixup notice.binary

* now with 3x the hive notices

* less redundant
2019-08-14 19:42:47 -05:00
Sayat 1f3a99616d Upgrade Kafka library for kafka-lookup module (#8078)
* Upgrade Kafka library for kafka-lookup module

* Update licenes.yaml

* Adopt class workaround from KafkaRecordSupplier#getKafkaConsumer

* Update lisences for kafka clients
2019-08-14 13:46:25 -07:00
Clint Wylie 7fa0ff5e11 adjust azure storage retry conditions (#8296) 2019-08-13 21:48:45 -07:00
Chi Cao Minh c8d7e0a868 Add slack badge (#8295)
* Add slack badge

* Disable broken badges
2019-08-13 21:48:17 -07:00
Fokko Driesprong 1a3aa1cfc0 Bump commons-io from 2.5 to 2.6 (#8006)
* Bump commons-io from 2.5 to 2.6

* Update licenses.yaml

* Address comments
2019-08-13 17:10:37 -07:00
Himanshu 176da53996
make double sum/min/max agg work on string columns (#8243)
* make double sum/min/max agg work on string columns

* style and compilation fixes

* fix tests

* address review comments

* add comment on SimpleDoubleAggregatorFactory

* make checkstyle happy
2019-08-13 15:55:14 -07:00
Jihoon Son a5c9c2950f Add missing maxBytesInMemory in tuningConfig for auto compaction (#8274)
* Add missing tuningConfigs for auto compaciton

* Add doc

* add test
2019-08-13 14:10:26 -05:00
Xavier Léauté eaa4651fa4 docker build cleanup and speed improvements (#8288)
* switch builder image to smaller "slim" debian base
* use built-in pyyaml
* skip various build checks to speed up compilation
* create user/group in target image directly
2019-08-13 08:33:06 -07:00
Jihoon Son 312cdc2452 Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks (#8236)
* Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks

* kill runner when it's ready

* add comment

* kill run thread

* fix test

* Take closeable out of Appenderator

* add javadoc

* fix test

* fix test

* update javadoc

* add javadoc about killed task

* address comment

* handling missing exceptions

* more clear javadoc for stopGracefully

* update javadoc

* Add missing statement in javadoc

* typo
2019-08-12 19:42:06 -05:00
Chi Cao Minh ccd87d2667 Speedup package check job (#8282)
The package check job sometimes hits the 50 minute Travis CI job time
limit. Move license checking tasks in "package check" job to "license
check" job to rebalance the job runtime (the "license check" job
currently takes about 1 minute). Moving the logic from build.sh to
.travis.yml also gives more visibility into how long each step takes
(i.e., generate-license-dependency-reports.py vs generate-license.py).
2019-08-12 13:25:08 -07:00
Alexandre Yang 6b4d028b96 [statsd-emitter] Add config to send Druid process/service as tag (#8238)
* [statsd-emitter] Add serviceAsTag option

* [statsd-emitter] Refactor serviceAsTag option

* [statsd-emitter] Update statsd.md

* [statsd-emitter] add default prefix

* [statsd-emitter] update statsd.md

* [statsd-emitter] Remove extra spaces

* [statsd-emitter] Improve docs for config `dogstatsdServiceAsTag`

* [statsd-emitter] Simplify equals() for StatsDEmitterConfig.java

* [statsd-emitter] Add @Nullable for StatsDEmitterConfig.java
2019-08-12 13:18:44 -07:00
Clint Wylie 1054d85171
add mechanism to control filter optimization in historical query processing (#8209)
* add support for mechanism to control filter optimization in historical query processing

* oops

* adjust

* woo

* javadoc

* review comments

* fix

* default

* oops

* oof

* this will fix it

* more nullable, refactor DimFilter.getRequiredColumns to use Set, formatting

* extract class DimFilterToStringBuilder with common code from custom DimFilter toString implementations

* adjust variable naming

* missing nullable

* more nullable

* fix javadocs

* nullable

* address review comments

* javadocs, precondition

* nullable

* rename method to be consistent

* review comments

* remove tuning from ColumnComparisonFilter/ColumnComparisonDimFilter
2019-08-09 16:36:18 -07:00
Nathan b28e252d9a Minor Spelling Error (#8277)
* Minor Spelling Error

* Update mySQL password in docs

/extensions-core/mysql update druid.metadata.storage.connector.password
2019-08-09 16:06:02 -05:00
Jonathan Wei e8727dc98b
Fix DeterminePartitionsJob reducer when total rows < targetPartitionSize * SHARD_COMBINE_THRESHOLD (#8273)
* Fix DeterminePartitionsJob reducer when rows < targetPartitionSize

* use isEmpty()
2019-08-09 16:03:30 -05:00
Benedict Jin 170368999d Bump rhino from 1.7R5 to 1.7.11 (#8008)
* Bump rhino from 1.7R5 to 1.7.11

* Update the version of rhino in licenses.yaml
2019-08-09 13:10:54 -07:00
Benedict Jin f7cf2f7cad Bump httpcore from 4.4.4 to 4.4.11 (#7870)
* Bump httpcore from 4.4.4 to 4.4.11

* Update the version of httpcore in licenses.yaml
2019-08-09 19:53:20 +03:00
mcbrewster d1f286b7f4 fix no from clause error (#8275) 2019-08-08 21:42:52 -07:00
Chi Cao Minh b359c5b3d9 Fix SIGAR dependency connection timeout (#8258)
After enabling parallel builds for "mvn install", the sigar dependency
would sometimes resolve to the incorrect artifact repo for some of the
maven modules. This issue seems to be fixed by moving the definition of
the sigar dependency's artifact repo to the root POM.

Also, depending on network speeds, "mvn -q install" may take longer than
the default 10 minute timeout to print any output. Use travis_wait to
extend the timeout to 15 minutes.
2019-08-08 20:13:18 -05:00
Fangjin Yang 76675d6163
Update README.md (#8261)
remove IRC
2019-08-08 10:23:41 -07:00
ilhanadiyaman 00e948a489 add slack url to readme (#8259) 2019-08-08 09:15:27 -07:00
Jonathan Wei e88bbe71c0 Adjust default globalIngestionHeapLimitBytes for indexer, add more docs (#8255) 2019-08-07 23:04:07 -07:00
mcbrewster 03a37936ec Web-Console: Column tree button fixes (#8253)
* small fixes

* run jest -u

* add trim

* remove trim, update package

* fix package-lock
2019-08-07 16:48:17 -07:00
Jihoon Son 8fa114c349 Fix bugs in overshadowableManager and add unit tests (#8222)
* Fix bugs in overshadowableManager and add unit tests

* Fix SegmentManager

* add segment manager test

* Address comments

* Address comments
2019-08-07 15:51:21 -05:00
Clint Wylie 172ebba4b8 more adjustments to PR template (#8048)
* more adjustments

* friendly thanks, remove italics, change key changed/added to list
2019-08-07 13:17:47 -07:00
Chi Cao Minh 05b44e3467 Speedup Travis CI jobs (#8240)
Reorganize Travis CI jobs into smaller faster (and more) jobs. Add
various maven options to skip unnecessary work and refactored Travis CI
job definitions to follow DRY.

Detailed changes:

.travis.yml
- Refactor build logic to get rid of copy-and-paste logic
- Skip static checks and enable parallelism for maven install
- Split static analysis into different jobs to ease triage
- Use "name" attribute instead of NAME environment variable
- Split "indexing" and "web console" out of "other modules test"
- Split 2 integration test jobs into multiple smaller jobs

build.sh
- Enable parallelism
- Disable more static checks

travis_script_integration.sh
travis_script_integration_part2.sh
integration-tests/README.md
- Use TestNG groups instead of shell scripts and move definition of jobs
  into Travis CI yaml

integration-tests/pom.xml
- Show elapsed time of individual tests to aid in future rebalancing of
  Travis CI integration test jobs run time

TestNGGroup.java
- Use TestNG groups to make it easy to have multiple Travis CI
  integration test jobs. TestNG groups also make it easier to have an
  "other" integration test group and make it less likely a test will
  accidentally not be included in a CI job.

IT*Test.java
AbstractITBatchIndexTest.java
AbstractKafkaIndexerTest.java
- Add TestNG group
- Fix various IntelliJ inspection warnings
- Reduce scope of helper methods since the TestNG group annotation on
  the class makes TestNG consider all public methods as test methods

pom.xml
- Allow enforce plugin to be run from command-line
- Bump resources plugin version so that "[debug] execute contextualize"
  output is correctly suppressed by "mvn -q"
- Bump exec plugin version so that skip property is renamed from "skip"
  to "exec.skip"

web-console/pom.xml
- Add property to allow disabling javascript-related work. This property
  is overridden in Travis CI to speed up the jobs.
2019-08-07 09:52:42 -07:00
mcbrewster e2d1d00fb8 Web-Console: Add show and copy actions to column tree (#8251)
* save

* add copy and show popup to column tree menu

* fixes

* nest css, uset functions

* fix copy state

* update package

* update package-lock
2019-08-06 23:58:00 -07:00
Fokko Driesprong 7702005f8f Use Closer instead of List<Closeable> (#8235)
* Use Closer instead of List<Closeable>

* Process comments

* Catch an Exception instead

* Removed unused import
2019-08-07 14:29:03 +08:00
Jonathan Wei 5e57492298 Add docs for CliIndexer as an experimental feature (#8245)
* Experimental CliIndexer docs

* PR comments
2019-08-06 15:57:17 -07:00
Vadim Ogievetsky b9c68a5b7b Web console: refactor home view, add tests (#8247)
* refactor home view

* updated mode button placement
2019-08-06 12:41:07 -07:00
ilhanadiyaman 38b6047aa9 Update docker build (#8244) 2019-08-06 08:32:59 -07:00
Himanshu 4507a4f8f1 fix merging of groupBy subtotal spec results (#8109)
* fix merging of groupBy subtotal spec results

* add post agg to subtotals spec ut

* add comment

* remove unnecessary agg transformation

* fix build

* fix test

* ignore unknown columns in ordering spec

* change variable names based on comment for easy read

* formatting

* don't ignore unknown columns in DefaultLimitSpec to not change existing behavior

* handle limit spec columns correctly

* uncomment inadvertantly commented lines

* GroupByStrategyV2 changes

* test changes wip

* more fixes to handle merge buffer closing and limit spec

* uncomment line commented accidentally
2019-08-06 07:06:28 -07:00
Lucas Capistrant e252abedc5 Enable toggling request logging on/off for different query types (#7562)
* Enable ability to toggle SegmentMetadata request logging on/off

* Move SegmentMetadata query log filter to FilteredRequestLogger

* Update documentation to reflect the segment metadata flag moving to the filtered request logger

* Modify patch to allow blacklist of query types to not log to request logger

* Address styling and naming requests following latest code review

* Fix indentation on multiple locations per Druid style rules
2019-08-06 15:47:30 +03:00
mcbrewster 62da1efc0f Web-Console: add dimensions to datasources icon (#8232)
* add dimensions to datasources icon

* fix test

* update snapshots

* save

* use table

* update snapshots

* rename to columns

* run jest -u
2019-08-06 00:04:37 -07:00
Vadim Ogievetsky 0235b338fc Web console: celebrate array based groupBy by supporting resultAsArray in the console (#8231)
* teach table about resultAsArray

* use query result decoder

* fix snapshot
2019-08-05 18:54:39 -07:00
mcbrewster 5d0805dd48 Web-Console: change go to sql button to more button (#8227)
* change go to sql button

* rename to See in SQL view

* update snapshots
2019-08-05 14:12:16 -07:00
Jihoon Son ab5b3be6c6 Add shuffleSegmentPusher for data shuffle (#8115)
* Fix race between canHandle() and addSegment() in StorageLocation

* add comment

* Add shuffleSegmentPusher which is a dataSegmentPusher used for writing shuffle data in local storage.

* add comments

* unused import

* add comments

* fix test

* address comments

* remove <p> tag from javadoc

* address comments

* comparingLong

* Address comments

* fix test
2019-08-05 13:38:35 -07:00
Samarth Jain 93cf9d4ad4 SQL support for t-digest based sketch aggregators (#8100)
* SQL support for t-digest based sketch aggregators

* Fix teamcity errors

* Add missing dependencies

* Remove unused dependency

* Address code review comments

* Add checks for compression param
2019-08-05 12:01:42 -07:00
mcbrewster ac856fe4c1 Web-Console: Adds edit-context-dialog (#8228)
* add edit query context

* remove console log

* add icon

* change to submit button

* fix c

* add stricter checking

* update snapshots
2019-08-04 07:18:08 -07:00
Eugene Sevastianov 3f3162b85e Enum of ResponseContext keys (#8157)
* Refactored ResponseContext and aggregated its keys into Enum

* Added unit tests for ResponseContext and refactored the serialization

* Removed unused methods

* Fixed code style

* Fixed code style

* Fixed code style

* Made SerializationResult static

* Updated according to the PR discussion:

Renamed an argument

Updated comparator

Replaced Pair usage with Map.Entry

Added a comment about quadratic complexity

Removed boolean field with an expression

Renamed SerializationResult field

Renamed the method merge to add and renamed several context keys

Renamed field and method related to scanRowsLimit

Updated a comment

Simplified a block of code

Renamed a variable

* Added JsonProperty annotation to renamed ScanQuery field

* Extension-friendly context key implementation

* Refactored ResponseContext: updated delegate type, comments and exceptions

Reducing serialized context length by removing some of its'
collection elements

* Fixed tests

* Simplified response context truncation during serialization

* Extracted a method of removing elements from a response context and
added some comments

* Fixed typos and updated comments
2019-08-03 12:05:21 +03:00
Jihoon Son 1ee828ff49
Add a cluster-wide configuration to force timeChunk lock and add a doc for segment locking (#8173)
* Add a cluster-wide configuration to force timeChunk lock and add a doc for segment locking

* add more test

* javadoc for missingIntervalsInOverwriteMode

* Fix test

* Address comments

* avoid spotbugs
2019-08-02 20:30:05 -07:00
Jihoon Son 8a16a8e97f
Teach tasks what machine they are running on (#8190)
* Teach the middleManager port to tasks

* parent annotation

* Bind parent for indexer
2019-08-02 15:34:44 -07:00
Clint Wylie e7c6deac76 optimize single input column multi-value expressions (#8047)
* optimize single input column multi-value expressions

* javadocs

* merge fixup

* vectorization fixup

* more fixes

* more docs

* more links

* empty

* javadocs are hard

* suppress javadoc refs issue

* fix it
2019-08-02 13:21:25 -07:00
Fokko Driesprong 91743eeebe Spotbugs: NP_NONNULL_PARAM_VIOLATION (#8129) 2019-08-02 19:20:22 +03:00
Chi Cao Minh 4bd3bad8ba Add IPv4 SQL functions (#8223)
* Add IPv4 SQL functions

New SQL functions for filtering IPv4 addresses:
- IPV4_MATCH: Check if IP address belongs to a subnet
- IPV4_PARSE: Convert string IP address to integer
- IPV4_STRINGIFY: Convert integer IP address to string

These are the SQL analogs of the druid expressions with the same name.
Filtering is more efficient when operating on IP addresses as integers
instead of strings.

* Refactor operator conversions into named constants
2019-08-01 21:29:58 -07:00
mcbrewster 5528dec4a4 Web-Console: stop log from making request after dialog close, implement query manager (#8219)
* stop logs from runnign after component unmount, use query managers

* address comments

* check if defined
2019-08-01 18:17:09 -07:00
mcbrewster 02c3a785a2 Web-Console: add go to task for datasources (#8210)
* add go to task for datasources

* general fixes

* use find
2019-08-01 18:15:22 -07:00
Clint Wylie 01c8c82982 correct kerberos doc extension load list (#8224) 2019-08-01 17:03:25 -07:00
Chi Cao Minh 7783b31846 Add IPv4 druid expressions (#8197)
* Add IPv4 druid expressions

New druid expressions for filtering IPv4 addresses:
- ipv4address_match: Check if IP address belongs to a subnet
- ipv4address_parse: Convert string IP address to long
- ipv4address_stringify: Convert long IP address to string

These expressions operate on IP addresses represented as either strings
or longs, so that they can be applied to dimensions with mixed
representation of IP addresses. The filtering is more efficient when
operating on IP addresses as longs. In other words, the intended use
case is:

1) Use ipv4address_parse to convert to long at ingestion time
2) Use ipv4address_match to filter (on longs) at query time
3) Use ipv4adress_stringify to convert to (readable) string at query
time

* Fix licenses and null handling

* Simplify IPv4 expressions

* Fix tests

* Fix check for valid ipv4 address string
2019-08-01 11:45:04 -07:00
Artiom Darie bcaffe2bc0 Fixed a bug in HttpPostEmitter leading to ClassCastException (#8205)
* Issue 8206: Fixed class cast exception in case of batch recovery

* Issue 8206: Added HttpPostEmitterTest license header

* Issue 8206: Updated comments accordingly to code review.

* Issue 8206: Updated HttpPostEmitterTest accordingly to new modifications.
2019-08-01 20:36:23 +03:00