Commit Graph

7726 Commits

Author SHA1 Message Date
Jisoo Kim 7c0f462fbc fix bug in StringDimensionHandler and add a cli tool for validating segments (#3666) 2016-11-11 18:46:25 -08:00
Roman Leventov 988d97b09c Unwrap exceptions from RuntimeException in URIExtractionNamespaceCacheFactory.populateCache() (part of #3667) (#3668)
* Unwrap exceptions from RuntimeException in URIExtractionNamespaceCacheFactory.populateCache()

* Fix tests
2016-11-11 17:25:41 -08:00
Roman Leventov fbbb55f867 Update emitter dependency to 0.4.0 and emit "version" dimension for all druid metrics (#3679)
* Update emitter dependency to 0.4.0 and emit "version" dimension for all druid metrics, not only query metrics

* Remove unused imports

* Use empty string instead of "testing-version" as a version placeholder
2016-11-11 17:17:27 -06:00
Akash Dwivedi 3e408497b3 Migrating bytebuffercollections from Metamarkets. (#3647)
* Migrating  bytebuffercollections from Metamarkets.

* resolving code conflicts and removing <p> from bytebuffer-collections.
2016-11-11 10:51:07 -08:00
Himanshu ddc078926b consolidate different theta sketch representations into SketchHolder (#3671) 2016-11-11 10:20:41 -08:00
praveev 52a74cf84f Use timestamp in millis as Map key instead of DateTime object (#3674)
* Use Long timestamp as key instead of DateTime.

DateTime representation is screwed up when you store with an obj
and read with a different DateTime obj.

For example: The code below fails when you use DateTime as key
```
        DateTime odt = DateTime.now(DateTimeUtils.getZone(DateTimeZone.forID("America/Los_Angeles")));
        HashMap<DateTime, String> map = new HashMap<>();
        map.put(odt, "abc");
        DateTime dt = new DateTime(odt.getMillis());
        System.out.println(map.get(dt));
```

* Respect timezone when creating the file.

* Update docs with timezone caveat in granularity spec

* Remove unused imports
2016-11-11 10:20:20 -08:00
Gian Merlino fd5451486c Short-circuiting AndFilter. (#3676)
If any of the bitmaps are empty, the result will be false.
2016-11-11 10:14:56 -08:00
Akash Dwivedi 1acc816196 FileSmoosher requested changes (#3673)
* FileSmoosher requested changes 

from https://github.com/metamx/java-util/pull/55

* Addressed code review requested changes.
2016-11-10 10:11:49 -08:00
Himanshu b76b3f8d85 reset-cluster command to clean up druid state stored on metadata and deep storage (#3670) 2016-11-09 11:07:01 -06:00
Mark 575aeb843a Metadata Storage extension for Microsoft SqlServer (sqlserver-metadata-storage) (#3421) 2016-11-08 14:56:52 -08:00
Nicolas Colomer 37ecffb648 Add support for Confluent Schema Registry in the avro extension (#3529) 2016-11-08 16:10:45 -06:00
Gian Merlino 657e4512d2 Checkstyle checks for AvoidStaticImport, UnusedImports. (#3660)
Excludes tests from AvoidStaticImport, since those are used often there and
I didn't want to make this changeset too large. Production code use was minimal
and I switched those to non-static imports.
2016-11-05 11:34:36 -07:00
Roman Leventov 22b57ddd60 Make ExtractionNamespaceCacheFactory to populate cache directly instead of returning callable (#3651)
* Rename ExtractionNamespaceCacheFactory.getCachePopulator() to populateCache() and make it to populate cache itself instead of returning a Callable which populates cache, because this "callback style" is not actually needed.

ExtractionNamespaceCacheFactory isn't a "factory" so it should be renamed, but renaming right in this commit would tear the git history for files, because ExtractionNamespaceCacheFactory implementations have too many changed lines. Going to rename ExtractionNamespaceCacheFactory to something like "CachePopulator" in one of subsequent PRs.

This commit is a part of a bigger refactoring of the lookup cache subsystem.

* Remove unused line and imports
2016-11-04 13:33:16 -07:00
Gian Merlino 4cbebd0931 SubstringDimExtractionFn, BoundDimFilter: Implement typical style toString. (#3658) 2016-11-04 13:31:47 -07:00
Gian Merlino 2bbbad9908 Add NOTICE for TestNG code. (#3661) 2016-11-04 12:54:52 -07:00
cheddar c49a9d5693 Call out semver expectations for modules (#3659)
* Call out semver expectations for modules

* Update modules.md

* Link to versioning
2016-11-04 12:52:05 -07:00
Gian Merlino 600bbd4a17 BucketExtractionFn: Implement hashCode, fix toString. (#3656) 2016-11-04 11:24:02 -07:00
Gian Merlino 8b3c86f41f Fix FilteredAggregatorFactory toString formatting. (#3657) 2016-11-04 11:23:55 -07:00
Gian Merlino 2c504b6258 Add "like" filter. (#3642)
* Add "like" filter.

* Addressed some PR comments.

* Slight simplifications to LikeFilter.

* Additional simplifications.

* Fix comment in LikeFilter.

* Clarify comment in LikeFilter.

* Simplify LikeMatcher a bit.

* No use going through the optimized path if prefix is empty.

* Add more tests.
2016-11-04 23:25:03 +05:30
Nishant b961b6a69f Add configs to enable running integration tests for cluster running behind proxy (#3646)
* Add configs to enable running integration tests for cluster running behind proxy

As part of https://issues.apache.org/jira/browse/KNOX-758
I am working on adding support for proxying druid queries & UIs using
Apache KNOX gateway.
This PR adds support for integration-tests to be run using the proxy
gateway.
Changes Include -
1) Instead of hostName and port, ability to specify url in the config
file.
2) tests now use HTTPClient defined in DruidTestModule that can pass in
the request.

Note - the config changes are backwards compatible and existing configs
should work fine.

* review comments

* review comments
2016-11-04 08:19:38 -07:00
Navis Ryu b99e14e732 Support configuration for handling multi-valued dimension (#2541)
* Support configuration for handling multi-valued dimension

* Addressed comments

* use MultiValueHandling.ofDefault() for missing policy
2016-11-03 22:38:54 -06:00
Gian Merlino 4203580290 URIExtractionNamespace: Treat null values in lookup maps as missing entries. (#3512)
* URIExtractionNamespace: Treat null values in lookup maps as missing entries.

This is useful when many logical lookups are derived from the same base JSON file,
and some lookups' values may be unknown sometimes.

* Add test, logging message, and address other comments.

* Update docs.
2016-11-03 13:53:04 -07:00
Navis Ryu e10def32f2 Support string type in math expression (#2836)
* Support string type in math expression

addressed comments

addressed comments

Addressed comments

* Updated math function document

* Addressed comments
2016-11-02 21:10:48 -06:00
Himanshu 2362effd8c use FileSystem.rename(from,to,Rename.NONE) so that tmp dirs from replicating tasks are not moved to the segment directory created by first task (#3650) 2016-11-02 15:58:55 -07:00
Roman Leventov 36a1543222 Lookup cache bug fixes (#3609)
* Return better lastVersion from JDBCExtractionNamespaceCacheFactory's cache populator callable

* Return the lastVersion if URI lookup last modified date is not later than the last cached, from URIExtractionNamespaceCacheFactory's cache populator callable

* Fix a race condition in NamespaceExtractionCacheManager.cancelFuture()

* Don't delete cache from NamespaceExtractionCacheManager if the ExtractionNamespaceCacheFactory returned the same version as the last; Better exception treatment in the scheduled cache updater runnable in NamespaceExtractionCacheManager (in particular, don't consume Errors); throw AssertionError in StaticMapExtractionNamespaceCacheFactory if the lastVersion != null)

* In NamespaceExtractionCacheManager, put NamespaceImplData.latestVersion update in the same synchronized() block with swapAndClearCache(id, cacheId); Turn getPostRunnable which returns a callback into a simple updateNamespace() method

* In StaticMapExtractionNamespaceCacheFactory.getCachePopulator(), check the input directly, not inside a callback

* In URIExtractionNamespaceCacheFactory, allow URI last modified time to go backwards

* Better logging in NamespaceExtractionCacheManager

* Add comment on lastVersion nullability in URIExtractionNamespaceCacheFactory
2016-11-02 09:40:19 -07:00
kaijianding 2961406b90 fix zero period in PeriodGranularity causing gran.iterable(start, end) infinite loop (#3644) 2016-11-02 15:40:07 +05:30
Roman Leventov 4b0d6cf789 Fix resource leaks (ComplexColumn and GenericColumn) (#3629)
* Remove unused ComplexColumnImpl class

* Remove throws IOException from close() in GenericColumn, ComplexColumn, IndexedFloats and IndexedLongs

* Use concise try-with-resources syntax in several places

* Fix resource leaks (ComplexColumn and GenericColumn) in SegmentAnalyzer, SearchQueryRunner, QueryableIndexIndexableAdapter and QueryableIndexStorageAdapter

* Use Closer in Iterable, returned from QueryableIndexIndexableAdapter.getRows(), in order to try to close everything even if closing some parts thew exceptions
2016-11-02 09:23:52 +05:30
Himanshu eb70a12e43 fix cleanup of tmp dir in HdfsDataSegmentPusher (#3636) 2016-11-01 12:45:38 -05:00
kaijianding f1dee037d6 fix 'No Such File' error when execute script out of druid installation directory (#3517) 2016-11-01 09:57:09 -07:00
Gian Merlino 45940d6e40 Math expressions support for missing columns. (#3630)
Also add SchemaEvolutionTest to help test this kind of thing.

Fixes #3627 and includes test for #3625.
2016-11-01 09:40:25 -07:00
Himanshu 0e269ce72a max row limit is necessary is connector is setup for streaming (#3635) 2016-11-01 09:39:55 -07:00
Gian Merlino 89d9c61894 Deprecate Aggregator.getName and AggregatorFactory.getAggregatorStartValue. (#3572) 2016-10-31 15:24:30 -07:00
Himanshu 32c5494e97 eagerly allocate the intermediate computation buffers (#3628) 2016-10-31 15:24:07 -07:00
Gian Merlino 9f5c895d5f Fix imports in DefaultOfflineAppenderatorFactoryTest. (#3624) 2016-10-31 12:23:19 -07:00
Slim f6995bc908 offline appenderator factory. (#3483)
* adding default offline appenderator

* adding test

* fix comments

* fix comments
2016-10-31 10:05:58 -07:00
Aveplatter 317d62e18c teeny tiny wording change (#3623) 2016-10-31 09:46:54 -07:00
Navis Ryu 3fca3be9ea SpecificSegmentQueryRunner misses missing segments from toYielder() (#3617) 2016-10-30 11:47:29 -07:00
Himanshu 23a8e22836 fix SketchMergeAggregatorFactory.finalizeResults, comparator and more UTs for timeseries, topN (#3613) 2016-10-28 15:48:33 -07:00
Akash Dwivedi 6a845e1f7b Adding getDelegate() to directly access delegate. (#3616)
👍
2016-10-27 15:57:36 -07:00
Charles Allen 78159d7ca4 Move off-heap QTL global cache delete lock outside of subclass lock (#3597)
* Move off-heap QTL global cache delete lock outside of subclass lock

* Make `delete` thread safe
2016-10-27 22:23:53 +05:30
Navis Ryu 0799640299 Faster interval comparator (#3605) 2016-10-26 14:20:27 +05:30
Parag Jain 98465f47b5 start stop metamx lifecycle annotated objects as well (#3610) 2016-10-25 15:56:16 -07:00
Navis Ryu 898c1c21af More best-effort parse long (#3603)
* More best-effort parse long

* addressed comments
2016-10-25 10:31:51 -07:00
David Lim 3c56cbdf82 fix timing issue with KafkaLookupExtractorFactoryTest (#3604) 2016-10-25 07:04:51 -07:00
Himanshu 641469fc38 manage overshadowing efficiently at coordinator (#3584)
* manage overshadowing efficiently at coordinator

* take readlock in VersionedIntervalTimeline.isOvershadowed()
2016-10-24 22:49:08 +05:30
Charles Allen 9bb735133f Move copyright to proper druid.io form (#3602) 2016-10-21 16:23:53 -07:00
Akash Dwivedi 4b3bd8bd63 Migrating java-util from Metamarkets. (#3585)
* Migrating java-util from Metamarkets.

* checkstyle and updated license on java-util files.

* Removed unused imports from whole project.

* cherry pick metamx/java-util@826021f.

* Copyright changes on java-util pom, address review comments.
2016-10-21 14:57:07 -07:00
Navis Ryu 8b7ff4409a Math expressional parameters for aggregator (#2783)
* Supports expression-paramed aggregator (squashed and rebased on master) also includes math post aggregator (was #2820)

* Addressed comments

* addressed comments
2016-10-19 13:58:35 -05:00
Roman Leventov b113a34355 In CPUTimeMetricQueryRunner, account CPU consumed in baseSequence.toYielder() (#3587) 2016-10-18 09:06:42 -05:00
Charles Allen 2c5c8198db Make query/cpu/time still report on error (#3535) 2016-10-18 08:26:21 -05:00