Commit Graph

7812 Commits

Author SHA1 Message Date
kaijianding 772de66e79 add filenameBase to log when exceed file size limit to indicate which column it is (#3982) 2017-03-01 13:05:07 -08:00
Eugene Sevastyanov 16bf62bacc BACKEND-564: Emitter upgrade from 0.4.0 to 0.4.1 (#3977) 2017-03-01 13:03:01 -08:00
Akash Dwivedi 94da5e80f9 Namespace optimization for hdfs data segments. (#3877)
* NN optimization for hdfs data segments.

* HdfsDataSegmentKiller, HdfsDataSegment finder changes to use new storage
format.Docs update.

* Common utility function in DataSegmentPusherUtil.

* new static method `makeSegmentOutputPathUptoVersionForHdfs` in JobHelper

* reuse getHdfsStorageDirUptoVersion in
DataSegmentPusherUtil.getHdfsStorageDir()

* Addressed comments.

* Review comments.

* HdfsDataSegmentKiller requested changes.

* extra newline

* Add maprfs.
2017-03-01 09:51:20 -08:00
Gian Merlino cc20133e70 Checkstyle rule to outlaw tabs. (#3988)
Tabs are the worst.
2017-02-28 23:52:53 -08:00
hzy001 e982a23a48 Fix one typo in the comments (#3980)
Signed-off-by: Hao Ziyu <haoziyu@qiyi.com>
2017-02-28 23:21:56 -08:00
Akash Dwivedi 91344cbe57 Enable GenericIndexed V2 for built-in(druid-io managed) complex columns. (#3987)
* Enable GenericIndexed V2 for complex columns.

* SerializerBuilder to use  GenericColumnSerializer.
2017-02-28 22:06:54 -08:00
Jonathan Wei a08660a9ca Support ingestion of long/float dimensions (#3966)
* Support ingestion for long/float dimensions

* Allow non-arrays for key components in indexing type strategy interfaces

* Add numeric index merge test, fixes

* Docs for numeric dims at ingestion

* Remove unused import

* Adjust docs, add aggregate on numeric dims tests

* remove unused imports

* Throw exception for bitmap method on numerics

* Move typed selector creation to DimensionIndexer interface

* unused imports

* Fix

* Remove unused DimensionSpec from indexer methods, check for dims first in inc index storage adapter

* Remove spaces
2017-02-28 19:04:41 -08:00
praveev 5ccfdcc48b Fix testDeadlock timeout delay (#3979)
* No more singleton. Reduce iterations

* Granularities

* Fix the delay in the test

* Add license header

* Remove unused imports

* Lot more unused imports from all the rearranging

* CR feedback

* Move javadoc to constructor
2017-02-28 12:51:41 -06:00
kaijianding ef6a19c81b buildV9Directly in MergeTask and AppendTask (#3976)
* buildV9Directly in MergeTask and AppendTask

* add doc
2017-02-28 10:04:32 -08:00
Fokko Driesprong 78b8a57ad6 Update PullDependencies.java (#3970)
Hi guys,

I've stumbled on this incomplete info line while debugging my mvn packaging job. 

Cheers,
Fokko Driesprong
2017-02-25 15:00:30 +05:30
Parag Jain 469ae374a3 add kill task link on console (#3974)
* add kill task link on console

* refresh after kill
2017-02-25 14:58:16 +05:30
praveev c3bf40108d One granularity (#3850)
* Refactor Segment Granularity

* Beginning of one granularity

* Copy the fix for custom periods in segment-grunalrity over here.

* Remove the custom serialization for now.

* Compilation cleanup

* Reformat code

* Fixing unit tests

* Unify to use a single iterable

* Backward compatibility for rolling upgrade

* Minor check style. Cosmetic changes.

* Rename length and millis to duration

* CR feedback

* Minor changes.
2017-02-25 01:02:29 -06:00
Jonathan Wei 58b704c3b4 Don't allow '__time' as a GroupBy output field name (#3967)
* Don't allow '__time' as a GroupBy column field name

* Tweak exception message
2017-02-23 14:39:17 -08:00
Niketh Sabbineni 02fc625b5f Fix bug with SegmentLoaderLocalCacheManager (#3929)
* Fix bug with SegmentLoaderLocalCacheManager

* Use collections.sort
2017-02-23 12:31:38 -08:00
Himanshu 2ead572639 make QueryResource return ETag header if If-None-Match header is provided (#3955)
also do not process query and return HTTP 304 NOT MODIFIED if given If-None-Match value matches current ETag
2017-02-23 13:39:59 -06:00
Aseem Bansal 1098ba7a7f Update toc.md (#3703) 2017-02-23 09:39:06 -08:00
kaijianding 7ce05d58bc fix NPE in search query when dimension contains null value (#3968)
* fix NPE when dimension contains null value in search query

* add ut

* search with not existed dimension should always return empty result
2017-02-23 08:07:59 -08:00
Jihoon Son ebd100cbb0 Set default query granularity for null value (#3965) 2017-02-22 17:38:43 -08:00
Gian Merlino f21641f0dc Fix over-optimistic log message. (#3963)
"Wrote task log" could be logged before the output stream is flushed and
closed, which could generate an error and not actually write the log.
2017-02-22 15:02:53 -08:00
Parag Jain edb032b96d add datasource in intermediate segment path (#3961) 2017-02-22 16:31:00 -06:00
Gian Merlino 372b84991c Add virtual columns to timeseries, topN, and groupBy. (#3941)
* Add virtual columns to timeseries, topN, and groupBy.

* Fix GroupByTimeseriesQueryRunnerTest.

* Updates from review comments.
2017-02-22 13:16:48 -08:00
Jihoon Son 7200dce112 Atomic merge buffer acquisition for groupBys (#3939)
* Atomic merge buffer acquisition for groupBys

* documentation

* documentation

* address comments

* address comments

* fix test failure

* Addressed comments

- Add InsufficientResourcesException
- Renamed GroupByQueryBrokerResource to GroupByQueryResource

* addressed comments

* Add takeBatch() to BlockingPool
2017-02-22 14:49:37 -06:00
Gian Merlino e7d01b67b6 Move SQL configs to sql.md. (#3959)
This puts all the SQL stuff in one place. It also makes life easier by
pointing out that configs be made in either common.runtime.properties
or the broker runtime.properties.
2017-02-22 08:37:24 -08:00
Gian Merlino 985203b634 Finalize fields in postaggs (#3957)
* initial commits for finalizeFieldAccess #2433

* fix some bugs to run a query

* change name of method Queries.verifyAggregations to Queries.prepareAggregations

* add Uts

* fix Ut failures

* rebased to master

* address comments and add a Ut for arithmetic post aggregators

* rebased to the master

* address the comment of injection within arithmetic post aggregator

* address comments and introduce decorate() in the PostAggregator interface.

* Address comments. 1. Implements getComparator in FinalizingFieldAccessPostAggregator and add Uts for it 2. Some minor changes like renaming a method name.

* Fix a code style mismatch.

* Rebased to the master
2017-02-21 16:32:14 -08:00
Gian Merlino a47206eaf8 Ability to filter on virtual columns. (#3942)
This didn't need much other than having BitmapIndexSelector return null from
various methods to trigger cursor based filtering.
2017-02-21 16:03:31 -08:00
Gian Merlino f910c050fd CachingClusteredClientTest: Use groupBy v1. (#3958)
GroupBy v2 doesn't cache on the broker, so it isn't actually testing
what the test was supposed to be testing. Also, the test failed due
to mismatched expectations.
2017-02-21 16:53:34 -06:00
Jihoon Son 128274c6f0 Disable caching on brokers for groupBy v2 (#3950)
* Disable caching on brokers for groupBy v2

* Rename parameter

* address comments
2017-02-21 09:49:49 -08:00
Jonathan Wei bc33b68b51 Use GroupBy V2 as default (#3953)
* Use GroupBy V2 as default

* Remove unused line

* Change assert to exception propagation
2017-02-18 07:40:40 -08:00
kaijianding 361d9d9802 fix dynamic schema data can't rollup correctly (#3949)
* fix dynamic schema data can't rollup correctly

* add ut
2017-02-17 15:07:29 -06:00
kaijianding a029b33499 fix cache populate incorrect content when numBackgroundThreads>1 (#3943)
* fix cache populate incorrect content when numBackgroundThreads>1

* simplify code by using Futures.allAsList and use CountDownLatch in UT

* fix test code style and assert countDownLatch.await()
2017-02-17 20:47:19 +05:30
Akash Dwivedi 797488a677 Removing Integer.MAX column size limit. (#3743)
* Removing Integer.MAX column size limit.

* On demand creation of headerLong, use v2 instead of v3

* Avoid reusing the same object from a previous test.

* Avoid reusing the same object from a previous test part#2

* code formatting.

* GenericIndexed/Writer code review changes.

* GenericIndexed/writer code review requested changes.

* checkIndex() to static

* native endianess for genericIndexedV2, code  review requested changes.

* Formatting

* Hll fix.

* use native endianess during bag size calculation.

* Code review requested changes.

* IOPeon close() changes.

* use different tmp directory path for testing.

* Code review requested changes.
2017-02-16 20:09:43 -06:00
michaelschiff e5fb0e1ff5 New property for each metric that tells the StatsDEmitter to convert metric values from range 0-1 to 0-100. This (#3936)
prevents rates and percentages expressed as Doubles (0.xx) from being rounded down to 0.
2017-02-16 13:55:56 -08:00
Gian Merlino ca6053d045 SQL: Resolve column type conflicts in favor of newer segments. (#3930)
* SQL: Resolve column type conflicts in favor of newer segments.

Helps with schema evolution from e.g. long -> float, which is supported
on the query side.

* Take columns from highest timestamp instead of max segment id.

* Fixes and docs.
2017-02-15 17:48:49 -08:00
Gian Merlino 16ef513c7d SQL: Add context and contextual functions to planner. (#3919)
* SQL: Add context and contextual functions to planner.

Added support for context parameters specified as JDBC connection properties
or a JSON object for SQL-over-JSON-over-HTTP.

Also added features that depend on context functionality:

- Added CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP functions.
- Added support for time zones other than UTC via a "timeZone" context.
- Pass down query context to Druid queries too.

Also some bug fixes:

- Fix DATE handling, it was largely done incorrectly before.
- Fix CAST(__time TO DATE) which should do a floor-to-day.
- Fix non-equality comparisons to FLOOR(__time TO X).
- Fix maxQueryCount property.

* Pass down context to nested queries too.
2017-02-15 14:09:14 -08:00
David Lim 3c54fc912a fix numShards = -1 not being handled correctly (#3937) 2017-02-14 18:45:38 -08:00
Jihoon Son a459db68b6 Fine grained buffer management for groupby (#3863)
* Fine-grained buffer management for group by queries

* Remove maxQueryCount from GroupByRules

* Fix code style

* Merge master

* Fix compilation failure

* Address comments

* Address comments

- Revert Sequence
- Add isInitialized() to Grouper
- Initialize the grouper in RowBasedGrouperHelper.Accumulator
- Simple refactoring RowBasedGrouperHelper.Accumulator
- Add tests for checking the number of used merge buffers
- Improve docs

* Revert unnecessary changes

* change to visible to testing

* fix misspelling
2017-02-14 12:55:54 -08:00
Gian Merlino 78b0d134ae Require Java 8 and include some Java 8 dependencies. (#3914)
* Require Java 8 and include some Java 8 dependencies.

- Upgrade Jetty to 9.3.16.v20170120.
- Upgrade DataSketches to 0.8.4.
- Bundle caffeine-cache by default.
- Still target Java 7 when compiling base Druid classes.

* Update cluster, quickstart docs.

* Remove oraclejdk7 from travis.yml.
2017-02-14 12:51:51 -08:00
Jakub Kukul 28d85702ad Fix rolling of request log files. (#3916)
* Use common date format for request log files.

* Remove code duplication in creating logging FileWriter.
2017-02-14 09:33:43 -08:00
Gian Merlino af67e8904e PreComputedHyperUniquesSerde: Fix formatting. (#3932) 2017-02-14 09:32:29 -08:00
DaimonPl a2875a4d91 pre-computed HLL support for hyperUnique aggregator (#3909) 2017-02-13 15:26:20 -08:00
Akash Dwivedi 8854ce018e File.deleteOnExit() (#3923)
* Less use of File.deleteOnExit()
 * removed deleteOnExit from most of the tests/benchmarks/iopeon
 * Made IOpeon closable

* Formatting.

* Revert DeterminePartitionsJobTest, remove cleanup method from IOPeon
2017-02-13 15:12:14 -08:00
Himanshu 9dfcf0763a disable javascript execution by default (#3818) 2017-02-13 15:11:18 -08:00
Himanshu 8cf7ad1e3a druid.coordinator.asOverlord.enabled flag at coordinator to make it an overlord too (#3711) 2017-02-13 15:03:59 -08:00
Pierre 9ab9feced6 Close all aggregators when closing onHeapIncrementalIndex (#3926)
* Close all aggregators when closing onHeapIncrementalIndex

* Aggregators are now handled as Closeables, remove unnecessary mock in test

* Fix variable shadowing
2017-02-13 15:01:27 -08:00
Parag Jain 1f263fe50b alert when resetting offsets (#3931)
* alert when resetting offsets

* add more data to alerts
2017-02-13 13:49:24 -08:00
michaelschiff c1eee9bbf3 modified "end" column to `end` (#3903)
* modified "end" column to `end`.  "end" is interpretted as a string rather than dereferencing the column value

* SQLMetadataConnector.getQuoteString defines the string that should be used to quote string fields

* positional arguments for String.format

* for Connectors that use " need to include the \ escape as well
2017-02-13 12:36:27 -08:00
Jihoon Son 991e2852da Add PostAggregators to generator cache keys for top-n queries (#3899)
* Add PostAggregators to generator cache keys for top-n queries

* Add tests for strings

* Remove debug comments

* Add type keys and list sizes to cache key

* Make post aggregators used for sort are considered for cache key generation

* Use assertArrayEquals()

* Improve findPostAggregatorsForSort()

* Address comments

* fix test failure

* address comments
2017-02-13 12:23:44 -08:00
Parag Jain 8e31a465ad report hand off count finite appenderator driver (#3925) 2017-02-13 10:41:24 -08:00
baruchz b7a88706f3 Add maprfs scheme (#3920)
Add maprfs scheme to JobHelper to treated as HDFS deep storage
2017-02-12 18:37:58 -08:00
Roman Leventov 38000576ea Optimizations of union, intersection and iterators of concise bitsets (part of #3798) (#3883)
* Port of metamx/extendedset#10, metamx/extendedset#13, metamx/extendedset#14, metamx/extendedset#15, metamx/bytebuffer-collections@9b199e3349, metamx/bytebuffer-collections#38 to Druid, remove unused code from extendedset module

* Remove ConciseSet.modCount

* Replace comments with assertions in ImmutableConciseSet

* Fix comments

* Fix asssertions in ImmutableConciseSet

* Add tests

* Comment fix
2017-02-10 18:02:26 -08:00