Commit Graph

9307 Commits

Author SHA1 Message Date
Clint Wylie 939b417379 Update tutorial-kafka.md (#7678) 2019-05-16 23:10:45 -07:00
David Lim d38457933f Data loader (sampler component) - Kafka/Kinesis samplers (#7566)
* implement Kafka/Kinesis sampler

* add KafkaSamplerSpecTest and KinesisSamplerSpecTest

* code review changes
2019-05-16 20:26:23 -07:00
Jenny Zhu ec0b7787cf Web console: fixed issue when grouping tasks by different attributes (#7657)
* deleted created_time in "status"

* added toString on the status object
2019-05-16 16:46:04 -07:00
Jonathan Wei d99f77a01b
Add option to use YARN RM as fallback for JobHistory failure (#7673)
* Add option to use YARN RM as fallback for job status

* PR comments
2019-05-16 13:59:10 -07:00
Fangjin Yang dc85a5309e
some more doc improvements (#7675) 2019-05-16 13:17:21 -07:00
Jonathan Wei d667655871 Add basic tuning guide, getting started page, updated clustering docs (#7629)
* Add basic tuning guide, getting started page, updated clustering docs

* Add note about caching, fix tutorial paths

* Adjust hadoop wording

* Add license

* Tweak

* Shrink overlord heaps, fix tutorial urls

* Tweak xlarge peon, update peon sizing

* Update Data peon buffer size

* Fix cluster start scripts

* Add upper level _common to classpath

* Fix cluster data/query confs

* Address PR comments

* Elaborate on connection pools

* PR comments

* Increase druid.broker.http.maxQueuedBytes

* Add guidelines for broker backpressure

* PR comments
2019-05-16 11:13:48 -07:00
Himanshu 8687f424f9
make ComplexColumn an interface and ExtensionPoint (#7633)
* make ComplexColumn an interface and ExtensionPoint

* incorporate review comments

* make ColumnValueSelector @ExtensionPoint

* more java docs

* add close() method to ComplexColumn interface
2019-05-15 20:59:55 -07:00
Benedict Jin 3df364c472 Fix broken links in api-reference.md (#7670) 2019-05-15 18:53:34 -07:00
Clint Wylie c2abbc24a7 minor web console doc fixes (#7668) 2019-05-15 18:52:51 -07:00
Jonathan Wei 7d63c295cc Fix compilation error in CoordinatorBasicAuthorizerResourceTest (#7667)
* Fix compilation error in CoordinatorBasicAuthorizerResourceTest

* Don't use simplifyPermissions
2019-05-15 17:47:38 -07:00
Surekha d3545f5086 Show all server types in sys.servers table (#7654)
* update sys.servers table to show all servers

* update docs

* Fix integration test

* modify test query for batch integration test

* fix case in test queries

* make the server_type lowercase

* Apply suggestions from code review

Co-Authored-By: Himanshu <g.himanshu@gmail.com>

* Fix compilation from git suggestion

* fix unit test
2019-05-15 16:54:02 -07:00
Denis Boulas 0b739a5787 Added more mappings for new metrics. (#7665)
Signed-off-by: Denis Boulas <dene14@gmail.com>
2019-05-15 15:04:46 -07:00
Jonathan Wei 6901123a53 Fix compareAndSwap() in SQLMetadataConnector (#7661)
* Fix compareAndSwap() in SQLMetadataConnector

* Catch serialization_failure and retry for Postgres
2019-05-15 14:53:04 -07:00
Gian Merlino 0352f450d7 Fix broken links in docs, add broken link checker. (#7658)
Also adds back insert-segment-to-db.md with some docs about why and
when it was removed (in #6911).
2019-05-15 14:49:50 -07:00
Surekha 917106985f Update tutorial to delete data (#7577)
* Update tutorial to delete data

* update tutorial, remove old ways to drop data

* PR comments
2019-05-15 14:40:06 -07:00
Jonathan Wei e874da7cea
Add simpler permissions option to BasicAuthorizer GET APIs (#7635)
* Add simpler permissions option to BasicAuthorizer GET APIs

* Adjust log message

Co-Authored-By: Himanshu <g.himanshu@gmail.com>

* Adjust log message

Co-Authored-By: Himanshu <g.himanshu@gmail.com>
2019-05-15 12:59:32 -07:00
Clint Wylie b87c8f0314 fix lookup editor to use lookup tiers instead of historical tiers (#7647)
* fix lookup editor to use lookup tiers instead of historical tiers

* use default tier if empty response, fix if configured lookups is null

* fixes

* fix typo
2019-05-14 13:30:51 -07:00
Fokko Driesprong 2aa9613bed Bump Checkstyle to 8.20 (#7651)
* Bump Checkstyle to 8.20

Moderate severity vulnerability that affects:
com.puppycrawl.tools:checkstyle

Checkstyle prior to 8.18 loads external DTDs by default,
which can potentially lead to denial of service attacks
or the leaking of confidential information.

Affected versions: < 8.18

* Oops, missed one

* Oops, missed a few
2019-05-14 11:53:37 -07:00
Fokko Driesprong e8a6575fb3 Remove Joda from indexing-hadoop (#7650) 2019-05-13 12:31:13 -07:00
Fokko Driesprong 4c709ddbc1 Bump Apache Parquet to 1.10.1 (#7645)
https://github.com/apache/parquet-mr/blob/master/CHANGES.md#version-1101
2019-05-12 14:38:33 -07:00
Xue Yu 35a1fbefea upgrade avatica to 1.12.0 (#7644) 2019-05-12 14:38:06 -07:00
Clint Wylie c7b482625e travis.yml remove duplicate language entry (#7626)
* travis.yml remove duplicate language entry

* openjdk8 instead of oraclejdk8?
2019-05-10 17:09:54 -07:00
Xue Yu f7b8b57c3b simpilfy DruidConvertletTable.java, remove STANDARD_CONVERTLET declare (#7632) 2019-05-10 14:08:32 -07:00
Alexander Saydakov ca1a6649f6 Datasketches quantiles more post-aggs (#7550)
* rank and CDF post-aggs

* added post-aggs to the module

* added new post-aggs

* moved post-agg IDs

* moved post-agg IDs
2019-05-10 11:46:54 -07:00
Xavier Léauté 1d49364d08 Set direct memory if unable to detect JVM config (#7606)
* Set direct memory if unable to detect JVM config

Java 9 and above prevents us from detecting the maximum available direct
memory.

This change adds a fallback method to use at most 25% of maximum heap
size, which should be a reasonable default.

Unless -XX:MaxDirectMemorySize is set, recent JVMs will default maximum
direct memory to match the maximum heap size, so this should work out of
the box in most cases. For completeness we print instructions in the log
to explain how to adjust settings if necessary.

* skip test rather than succeeding

* reword log message

Co-Authored-By: Himanshu <g.himanshu@gmail.com>
2019-05-09 22:30:42 -07:00
Clint Wylie 402d76a10f make-redirects.py requires python3, explicitly specify it (#7625) 2019-05-09 21:32:58 -07:00
Clint Wylie 76e31937cb fix license.binary for new datasketches version (#7627) 2019-05-09 21:32:44 -07:00
Clint Wylie 6a6c6d573d
Add plain text README.txt, use relative link from README.md to build.md (#7611)
* use relative link to build instructions from top level readme

* add textfile to readme

* formatting

* make README.BINARY plaintext, move LABELS.md to LABELS, README.txt to README

* exclude README.BINARY still

* remove jdk links/recommmendations

* add script to use DRUIDVERSION in textfile README instead of latest, add links to recommended jdk to build.md

* license

* better readme template, links to latest if does not detect an apache release version

* fix
2019-05-09 21:29:26 -07:00
Jihoon Son 18e0d6acb4 Fix resultLevelCache for timeseries with grandTotal (#7624)
* Fix resultLevelCache for timeseries with grandTotal

* Address comment

* fix test
2019-05-09 18:11:04 -07:00
Alexander Saydakov 59f9ff38c7 fix issue #7607 (#7619)
* fix issue #7607

* exclude com.google.code.findbugs:annotations
2019-05-09 17:33:29 -07:00
Samarth Jain b542bb9f34 TDigest backed sketch aggregators (#7331)
* First set of changes for tDigest histogram

* Add license

* Address code review comments

* Add a doc page for new T-Digest sketch aggregators. Minor code cleanup and comments.

* Remove synchronization from BufferAggregators. Address code review comments

* Fix typo
2019-05-09 17:22:55 -07:00
Jihoon Son b32c9abdb9
Run npm ci instead of npm install (#7613) 2019-05-09 15:42:16 -07:00
Jonathan Wei 1b577c9b1d
Fix exception when using complex aggs with result level caching (#7614)
* Fix exception when using complex aggs with result level caching

* Add test comments

* checkstyle

* Add helper function for getting aggs from cache

* Move method to CacheStrategy

* Revert QueryToolChest changes

* Update test comments
2019-05-09 13:49:11 -07:00
Magnus Henoch 2ac112151f Fix formatting in scan query documentation (#7622)
Escape underscores in `__time`, so they're not interpreted as bold
formatting.
2019-05-09 11:32:37 -07:00
Jinseon Lee 0ef435a16c add postgresql meta db table schema configuration property (#7137) (#7183)
* add postgresql meta db table schema configuration property (#7137)

If the postgresql db schema changes, you must set the configuration
values.
You do not need to set it if there is no change from the default schema
'public'.
druid.metadata.postgres.dbTableSchema=public

* create postgresql metadb table schema configuration property (#7137)
If the postgresql db schema changes, you must set the configuration
values.
You do not need to set it if there is no change from the default schema
'public'.
druid.metadata.postgres.dbTableSchema=public
check PostgreSQLTablesConfig.java

* modify postgresql readme file. - metadb table schema (#7137)
If the postgresql db schema changes, you must set the configuration
values.
You do not need to set it if there is no change from the default schema
'public'.
druid.metadata.postgres.dbTableSchema=public
check PostgreSQLTablesConfig.java
2019-05-08 12:56:30 -07:00
Vadim Ogievetsky 9b197b436b added scrolling to data loader panes (#7605) 2019-05-07 08:45:20 -07:00
Jonathan Wei dadf6a2f11
Add tool for migrating from local deep storage/Derby metadata (#7598)
* Add tool for migrating from local deep storage/Derby metadata

* Split deep storage and metadata migration docs

* Support import into Derby

* Fix create tables cmd

* Fix create tables cmd

* Fix commands

* PR comment

* Add -p
2019-05-06 23:39:40 -07:00
Xavier Léauté b3c7463059
fix test pattern matching for jdk11 (#7602)
This change makes the error pattern more generic to account for
differences between JDK versions.
2019-05-06 21:02:44 -07:00
Xavier Léauté c58aa2f2ab
Remove unnecessary cast to URLClassLoader (#7603)
Java 9 and above will fail trying to cast the system classloader
2019-05-06 20:17:22 -07:00
Xavier Léauté 751e1c9ba7
add javax.xml.bind dependencies removed in jdk11 (#7604)
We depend on javax.xml.bind at runtime. This change adds an explicit
dependency on the J2EE module that was removed in Java 11.
2019-05-06 19:30:14 -07:00
Jonathan Wei 7c2ca474da Add single-machine deployment example cfgs and scripts (#7590)
* Add single-machine deployment example cfgs and scripts

* Add (8u92+)

* Use combined coordinator-overlord for single machine confs

* RAT fix
2019-05-06 19:11:13 -07:00
Xavier Léauté 51a62cb31b Update dependencies for JDK11 support (#7601)
* update asm for jdk11 support

* update jvm-attach-api for jdk11 support
2019-05-06 14:07:44 -07:00
Gian Merlino 727b65c7e5 Remove SQL experimental banner and other doc adjustments. (#7591)
* Remove SQL experimental banner and other doc adjustments.

Also,

- Adjust the ToC and other docs a bit so SQL and native queries are
  presented on more equal footing.
- De-emphasize querying historicals and peons directly in the
  native query docs. This is a really niche thing and may have been
  confusing to include prominently in the very first paragraph.
- Remove DataSketches and Kafka indexing service from the experimental
  features ToC. They are not experimental any longer and were there in
  error.

* More notes.

* Slight tweak.

* Remove extra extra word.

* Remove RT node from ToC.
2019-05-06 12:31:51 -07:00
Xavier Léauté f7bfe8f269
Update mocking libraries for Java 11 support (#7596)
* update easymock / powermock for to 4.0.2 / 2.0.2 for JDK11 support
* update tests to use new easymock interfaces
* fix tests failing due to easymock fixes
* remove dependency on jmockit
* fix race condition in ResourcePoolTest
2019-05-06 12:28:56 -07:00
Roman Leventov a00d3e1931
Fix old-console/cluster.html; Remove redundant throws in ITBasicAuthConfigurationTest (#7589)
* Fix old-console/cluster.html; Remove redundant throws in ITBasicAuthConfigurationTest

* Remove comment in cluster.html
2019-05-06 18:08:49 +02:00
Qi Shu 83b38c0911 Improved UI for actions in task/supervisor table (#7528)
* Grouped actions in task/supervisor table to a dialog

* Design change

* fix divider

* Removed await for onclick event on buttons

* fix package.json

* Better error handling

* remove log

* name change; better code organization

* Code refactor; extrac table action dialog

* minor change

* tidy up actions

* drop reload actions
2019-05-06 03:20:12 -07:00
Vadim Ogievetsky baf54f373c Data loader (GUI component) (#7572)
* data loader init

* fix timecolumn text

* feedback changes

* fixing typos and improving error reporting

* added local firehose warning

* update warning copy

* refine copy

* better copy

* fix tests

* remove console log

* copy change

* add banner message
2019-05-03 17:14:57 -07:00
Samarth Jain afbcb9c07f Improve parallelism of zookeeper based segment change processing (#7088)
* V1 - improve parallelism of zookeeper based segment change processing

* Create zk nodes in batches. Address code review comments.
Introduce various configs.

* Add documentation for the newly added configs

* Fix test failures

* Fix more test failures

* Remove prinstacktrace statements

* Address code review comments

* Use a single queue

* Address code review comments

Since we have a separate load peon for every historical, just having a single SegmentChangeProcessor
task per historical is enough. This commit also gets rid of the associated config druid.coordinator.loadqueuepeon.curator.numCreateThreads

* Resolve merge conflict

* Fix compilation failure

* Remove batching since we already have a dynamic config maxSegmentsInNodeLoadingQueue that provides that control

* Fix NPE in test

* Remove documentation for configs that are no longer needed

* Address code review comments

* Address more code review comments

* Fix checkstyle issue

* Address code review comments

* Code review comments

* Add back monitor node remove executor

* Cleanup code to isolate null checks  and minor refactoring

* Change param name since it conflicts with member variable name
2019-05-03 15:58:42 +02:00
Jonathan Wei a013350018 Adjust required permissions for system schema (#7579)
* Adjust required permissions for system schema

* PR comments, fix current_size handling

* Checkstyle

* Set curr_size instead of current_size

* Adjust information schema docs

* Fix merge conflict

* Update tests
2019-05-02 07:18:02 -07:00
David Lim ec8562c885 Data loader (sampler component) (#7531)
* sampler initial check-in
fix checkstyle issues
add sampler fix to process CSV files from cache properly
change to composition and rename some classes
add tests and report num rows read and indexed
remove excludedByFilter flag and don't send filtered out data
fix tests to handle both settings for druid.generic.useDefaultValueForNull

* wrap sampler firehose in TimedShutoffFirehoseFactory to support timeouts

* code review changes - add additional comments, limit maxRows
2019-05-01 22:37:14 -07:00