Commit Graph

8741 Commits

Author SHA1 Message Date
Gian Merlino e096a8d6c5 Emitter: Clarify contract of "emit". (#5486)
* Emitter: Clarify contract of "emit".

* New wording
2018-03-14 22:07:08 +01:00
Gian Merlino fdd55538e1 SQL: Remove unused escalator, authConfig from various classes. (#5483)
DruidPlanner.plan is responsible for checking authorization, so these objects
weren't needed in as many places as they were injected.
2018-03-14 13:28:51 -07:00
Niketh Sabbineni 40cc2c8740 Query should not fail because emitter fails or throws Exception (#5484) 2018-03-13 19:57:05 -07:00
Jihoon Son 9b2a25bd84
Refactor supervisorReport to be type-safe (#5479)
* refactor supervisorReport

* use primitives
2018-03-13 09:28:44 -07:00
Christoph Hösler 34f655599d Let MySQLConnector accept all UTF charsets and recommend utf8mb4 (#5411)
* Let MySQLConnector accept all UTF charsets and recommend utf8mb4

* Fix regex and remove newline in log statement
2018-03-13 01:16:10 -07:00
Niraja Mishra 96cebfc222 As part of this feature, implemented a new endpoint to get running tasks by datasources (#5260)
and added datasource information as part of existing endpoint /druid/indexer/v1/runningTasks.

Added junit test cases for the newly implemented API and fixed existing junit test cases.

Fixed review comments - added new method getCreatedDateTimeAndDataSource into TaskStorageQueryAdapter class
and formatted changed files
2018-03-12 23:48:11 -07:00
Himanshu e968811583 HttpServerInventoryView: fixed startup wait time and more informative logging (#5336) 2018-03-12 22:13:51 -07:00
Roman Leventov 6b158abe3f Enforce optimal IndexedInts iteration (#5456)
* Enforce optimal IndexedInts iteration

* Fix remaining suboptimal usages
2018-03-09 09:42:40 -08:00
Clint Wylie d159a4fa01 better error messaging when parseSpec is missing timestampSpec or dimensionSpec (#5439) 2018-03-08 07:57:13 -08:00
bolkedebruin 8f07a39af7 Skip OS cache on Linux when pulling segments (#5421)
Druid relies on the page cache of Linux in order to have memory segments.
However when loading segments from deep storage or rebalancing the page
cache can get poisoned by segments that should not be in memory yet.
This can significantly slow down Druid in case rebalancing happens
as data that might not be queried often is suddenly in the page cache.

This PR implements the same logic as is in Apache Cassandra and Apache
Bookkeeper.

Closes #4746
2018-03-08 07:54:21 -08:00
Himanshu 8fae0edc95 allow arbitrary aggregators for reindexing with hadoop (#5294) 2018-03-07 17:13:56 -08:00
Hongze Zhang b084075279 Add http/https proxy options to PullDependencies.java (#5450) 2018-03-07 15:05:43 -08:00
Slim 593e87637d
Inline some backward incompatible Hadoop 3.0 method (#5396)
* Inline some backward incompatible hadoop 3.0 method

Change-Id: I49aeff5412d5cdea95e30feb031b2c036d036e9a

* fix build issue

Change-Id: I0a42fdb83ce970d6a2d3d45f150556e45442a0ac
2018-03-07 07:58:18 -08:00
Gian Merlino 0f03ab0c74 SQL: Fix precision of TIMESTAMP types. (#5464)
Druid stores timestamps down to the millisecond, so we should use
precision = 3. Setting this wrong sometimes caused milliseconds
to be ignored in timestamp literals.

Fixes #5337.
2018-03-05 18:56:52 -08:00
Gian Merlino ff0de21fc5 SQL: Fix assumption that AND, OR have two arguments. (#5470)
Calcite can deliver an AND or OR operator with > 2 arguments.
Fixes #5468.
2018-03-05 18:56:35 -08:00
Jihoon Son c9b12e7813 Fix JSON serde for taskStatusPlus (#5469)
* Fix JSON serde for taskStatusPlus

* add newline

* fix to statusCode

* fix fobidden api check

* remove debugging code
2018-03-05 18:49:59 -08:00
Alexander Korablev 8a51800693 fix PortFinder issue #5466 (#5467) 2018-03-05 16:58:49 -08:00
Niraja Mishra ba3dbf2a42 Fixed NPE when dimension is null or empty. https://github.com/druid-io/druid/issues/3007 (#5299) 2018-03-05 16:27:35 -08:00
Gian Merlino 7416d1d02d Add "joda" option to timeFormat extractionFn. (#5448) 2018-03-02 19:59:26 -08:00
Clint Wylie f948066710 KafkaIndexTask remove branch with unreachable code (#5434) 2018-03-02 17:26:12 -08:00
Jonathan Wei b63f1c0e45
Fix authorization check in supervisor history API (#5460) 2018-03-02 14:03:07 -08:00
Kevin Conaway 969a12f6ca #5425 Refactor to use map.get() when asserting the existence of published segments (#5426) 2018-03-02 19:42:39 +01:00
Jonathan Wei cf5f74b013 Fix GroupBy limit push down descending sorting on numeric columns (#5453) 2018-03-01 18:43:45 -08:00
Nishant Bangarwa e0d456b1ba Uniformly set Calcite systemProperties for All Unit tests (#5451)
Fixes test failures reported in -
https://github.com/druid-io/druid/issues/4909

Issue is that If some test skips setting up Calcite system properties
with proper encoding and loads calcite classes that use that property,
All subsequent tests in the same JVM fails.

To reproduce the issue - ExpressionsTest and CalciteQueryTest from IDE
in this order.

A better fix would be to not use System Properties in calcite, This
will work for now.

All new Calcite Unit tests that are added need to inherit
CalciteTestBase.
2018-03-01 12:56:32 -08:00
Vinesh Chemmala Paul fb493ae13a Add repository url (#5437)
snapshot build fails to find dependent artifacts.
Add repoistory URL to resolved dependencies
2018-03-01 07:38:24 -08:00
Jihoon Son 16e08c9adb add task priority for kafka indexing (#5444) 2018-02-28 22:29:23 -08:00
Gian Merlino e4eaee3806
Support for disabling bitmap indexes. (#5402)
* Support for disabling bitmap indexes.

Can save space for columns where bitmap indexes are pointless (like
free-form text).

* Remove import.

* Fix CompactionTaskTest.

* Update for review comments.

* Review comments, tests.

* Fix test.
2018-02-28 19:19:56 -08:00
Alexander Korablev 6a3a5350b8 Make memcached protocol and locator configurable. (#5438)
* Make memcached protocol and locator configurable.

* Style fix.

* Style fix.

* Style fix.
2018-02-28 17:16:43 -08:00
Jonathan Wei c23b723510 Skip normal authentication for JDBC requests in Router (#5435)
* Skip normal authentication for JDBC requests in Router

* Add integration test

* PR comments
2018-02-28 12:25:32 -08:00
Gian Merlino 27bf619d5d
Test flattenSpecs where the referred-to object is missing. (#5318)
It worked, but it's still good to have a test.
2018-02-27 15:10:30 -08:00
Niraja Mishra 0f009a41e1 Fixed PeriodGranularity for Asia pacific timezones (#5410) 2018-02-27 10:39:50 -08:00
Niketh Sabbineni ac5034e241 Improve cache cost to handle heterogenous historicals (#5416) 2018-02-23 13:17:31 -08:00
Jonathan Wei e9977ce4ef
Automatically adjust com.metamx.metrics Monitor class references (#5412)
* Automatically adjust com.metamx.metrics monitor class references

* Log warning for old class names
2018-02-22 12:03:07 -08:00
David Lim 5c56d01daa pass configuration from context into JobConf for determining DatasourceInputFormat splits (#5408) 2018-02-21 10:01:31 -08:00
Gian Merlino f3796bc81b SQL: Lower default JDBC frame size. (#5409)
The previous default of 100,000 was a bit excessive and could easily
lead to OOM errors on "select *" style queries.
2018-02-21 10:00:48 -08:00
Gian Merlino 3f72537787 Fix missing task type in task payload API. (#5399)
* Fix missing task type in task payload API.

Apparently embedding a polymorphic object inside a Map<String, Object> is
a bit too much for Jackson to serialize properly. Fix this by using
wrapper classes.

* Fix OverlordTest casts.

* Remove import.

* Remove unused imports.

* Clarify comments.
2018-02-21 10:00:13 -08:00
Gian Merlino 818ce51964 SQL: Fix selecting BOOLEAN type in JDBC. (#5401) 2018-02-21 09:59:56 -08:00
Nishant Bangarwa 219e77aeac SQL compatible Null Handling Part - Expressions and Storage Changes (#5278)
* SQL compatible Null Handling Part - Expressions, Storage and Dimension Selector Changes

fix travis strict compilation

* fix teamcity error - remove unused method

* review comments

* review comments

* more comments

* review comments

* review comments

* Optimize isNull method

* Optimize isNull in ColumnarFloats/Longs/Doubles

* review comment - separate classes for null and non-null columns

fix intellij inspection

* remove unused import

* More Review comments

* improve comment

* More review comments

* fix checkstyle

* more review comments

* review comments.

fix javadoc links

remove Nullable from ConstantColumnValueSelector

* review comments.

* satisfy teamcity inspections
2018-02-21 13:27:26 +01:00
vvc11 305ecc2a78 adding a properties endpoint in status resource (#5276)
* adding a properties endpoint in status resource

* checkstyle fixes

* more checkstyle corrections

* correcting the resource filter for properties endpoint

* adding feature of hiding sensitive properties

* checkstyle changes

* review changes for adding default hidden properties and using jackson for arrays value

* making review changes
2018-02-18 12:51:02 -08:00
Jihoon Son deeda0dff2 Fix DefaultLimitSpec to respect sortByDimsFirst (#5385)
* Fix DefaultLimitSpec to respect sortByDimsFirst

* fix bug

* address comment
2018-02-16 15:26:32 -08:00
Parag Jain fba13d8978 time based checkpointing for Kafka Indexing Service (#5255)
* time based checkpointing

* add test and fix issue

* fix comments

* fix formatting

* update docs
2018-02-15 20:57:02 -08:00
David Lim 20a3164180 Support for router forwarding requests to active coordinator/overlord (#5369)
* allow router to forward requests to coordinator and overlord

* fix forbidden API

* more forbidden api fixes

* code review changes
2018-02-15 14:38:58 -08:00
Javier Collado c45fe37611 Feature add coordinator servers endpoint documentation (#5392)
* Add new servers section to the coordinator endpoints documentation

* Remove trailing whitespace
2018-02-15 14:37:58 -08:00
Dan Suzuki 472ba14dfe Support Map type in ORC extension (#5363)
* Support map type in orc extension.
Added getMapObject in OrcHadoopInputRowParser
Updated parse tests to parse map-type field in OrcHadoopInputRowParserTest

* changed from for-loop to foreach

* added resolution of column names when map types are exploded to several
columns. updated the document as well -- orc.md.

* Update orc.md

change from review
2018-02-15 13:03:15 -08:00
Parag Jain b9b3be6965 fix segment info in Kafka indexing service docs (#5390)
* fix segment info in Kafka indexing service docs

* review updates
2018-02-15 09:57:30 -08:00
Spyros Kapnissis be38b18a85 Support Hadoop batch ingestion for druid-azure-extensions (#5221)
* Support Hadoop batch ingestion for druid-azure-extensions #5181

* Fix indentation issues

* Fix forbidden-apis violation

* Code & doc improvements for azure-extensions

* Rename version to binaryVersion where appropriate to avoid confusion
* Set default protocol to wasbs://, as recommended by the Azure docs
* Add link to Azure documentation for wasb(s):// path
* Remove any colons from the dataSegment.getVersion()
* Added test for dataSegment.getVersion colon is replaced
* Use StringUtils.format for String concatenation
* remove empty lines

* Remove unneeded StringUtils.format from log.info
2018-02-15 16:45:18 +01:00
Jihoon Son cd929000ca
Change early publishing to early pushing in indexTask & refactor AppenderatorDriver (#5297)
* Fix early publishing to early pushing in batch indexing & refactor appenderatorDriver

* fix compile

* rename and add more javadocs

* Fix conflicts

* address comments

* revert await executors

* fix test
2018-02-14 12:48:33 -08:00
Roman Leventov 65225bd6e7 Make HttpPostEmitter more robust in the face of serious errors (like OutOfMemoryError) during onSealExclusive() (#5386) 2018-02-13 17:00:58 -08:00
QiuMM aa7aee53ce Opentsdb emitter extension (#5380)
* opentsdb emitter extension

* doc for opentsdb emitter extension

* update opentsdb emitter doc

* add the ms unit to the constant name

* add a configurable event limit

* fix version to 0.13.0-SNAPSHOT

* using a thread to consume metric event

* rename method and parameter
2018-02-13 13:10:22 -08:00
Jonathan Wei b234a119ac Log exceptions thrown before persist() for indexing tasks (#5374)
* Log exceptions thrown before persist() for indexing tasks

* PR comment
2018-02-13 09:20:07 -08:00