Commit Graph

362 Commits

Author SHA1 Message Date
Roman Leventov 60cdf94677 Add PMD and prohibit unnecessary fully qualified class names in code (#4350)
* Add PMD and prohibit unnecessary fully qualified class names in code

* Extra fixes

* Remove extra unnecessary fully-qualified names

* Remove qualifiers

* Remove qualifier
2017-07-17 22:22:29 +09:00
Chris Gavin 960cb07ea6 Fix some unnecessary use of boxed types and incorrect format strings spotted by lgtm. (#4474)
* Remove some unnecessary use of boxed types.

* Fix some incorrect format strings.

* Enable IDEA's MalformedFormatString inspection.

* Add a Checkstyle check for finding uses of incorrect logging packages.

* Fix some incorrect usages of the metamx logger.

* Bypass incorrect logger Checkstyle check where using the correct logger is not simple.

* Fix some more places where the wrong number of arguments are provided to format strings.

* Suppress `MalformedFormatString` inspection on legacy logging test.

* Use @SuppressWarnings rather than a noinspection suppression comment.

* Fix some more incorrect format strings.

* Suppress some more incorrect format string warnings where the incorrect string is intentional.

* Log the aggregator when closing it fails.

* Remove some unneeded log lines.
2017-07-13 12:15:32 -07:00
Jihoon Son cc20260078 Early publishing segments in the middle of data ingestion (#4238)
* Early publishing segments in the middle of data ingestion

* Remove unnecessary logs

* Address comments

* Refactoring the patch according to #4292 and address comments

* Set the total shard number of NumberedShardSpec to 0

* refactoring

* Address comments

* Fix tests

* Address comments

* Fix sync problem of committer and retry push only

* Fix doc

* Fix build failure

* Address comments

* Fix compilation failure

* Fix transient test failure
2017-07-10 22:35:36 -07:00
Roman Leventov 9ae457f7ad Avoid using the default system Locale and printing to System.out in production code (#4409)
* Avoid usages of Default system Locale and printing to System.out or System.err in production code

* Fix Charset in DruidKerberosUtil

* Remove redundant string format in GenericIndexed

* Rename StringUtils.safeFormat() to unimportantSafeFormat(); add StringUtils.format() which fails as well as String.format()

* Fix testSafeFormat()

* More fixes of redundant StringUtils.format() inside ISE

* Rename unimportantSafeFormat() to nonStrictFormat()
2017-06-29 14:06:19 -07:00
Roman Leventov ae900a4934 Update versions to 0.11.0-SNAPSHOT (#4483) 2017-06-28 17:05:58 -07:00
Roman Leventov 2fa4b10145 More fine-grained DI for management node types. Don't allocate processing resources on Router (#4429)
* Remove DruidProcessingModule, QueryableModule and QueryRunnerFactoryModule from DI for coordinator, overlord, middle-manager. Add RouterDruidProcessing not to allocate processing resources on router

* Fix examples

* Fixes

* Revert Peon configs and add comments

* Remove qualifier
2017-06-27 22:58:01 -07:00
Jihoon Son 82140c2f86 Reduce number of tasks in ITUnionQueryTest (#4476) 2017-06-27 22:28:55 -07:00
Roman Leventov 63a897c278 Enable most IntelliJ 'Probable bugs' inspections (#4353)
* Enable most IntelliJ 'Probable bugs' inspections

* Fix in RemoteTestNG

* Fix IndexSpec's equals() and hashCode() to include longEncoding

* Fix inspection errors

* Extract global isntance of natural().nullsFirst(); address comments

* Fix

* Use noinspection comments instead of SuppressWarnings on method for IntelliJ-specific inspections

* Prohibit Ordering.natural().nullsFirst() using Checkstyle
2017-06-07 09:54:25 -07:00
Roman Leventov 31d33b333e Make using implicit system Charset an error (#4326)
* Make using implicit system charset an error

* Use StringUtils.toUtf8() and fromUtf8() instead of String.getBytes() and new String()

* Use English locale in StringUtils.safeFormat()

* Restore comment
2017-06-05 23:57:25 -07:00
Jihoon Son ba816063cb Wait until all data sources are ready for querying in ITUnionQueryTest (#4362) 2017-06-03 17:02:12 -07:00
Jihoon Son 7889891bd3 Fix integration tests (#4337)
* Fix integration tests

1) Use the same version of kafka
2) Change ServiceEmitter from LazySingleton to ManageLifecycle

* Revert unnecessary change
2017-05-28 08:48:39 -07:00
Roman Leventov 1ebfa22955 Update Error prone configuration; Fix bugs (#4252)
* Make Errorprone the default compiler

* Address comments

* Make Error Prone's ClassCanBeStatic rule a error

* Preconditions allow only %s pattern

* Fix DruidCoordinatorBalancerTester

* Try to give the compiler more memory

* Remove distribution module activation on jdk 1.8 because only jdk 1.8 is used now

* Don't show compiler warnings

* Try different travis script

* Fix travis.yml

* Make Error Prone optional again

* For error-prone compiler

* Increase compiler's maxmem

* Don't run Error Prone for benchmarks because of OOM

* Skip install step in Travis

* Remove MetricHolder.writeToChannel()

* In travis.yml, check compilation before tests, because it may fail faster
2017-05-12 15:55:17 +09:00
Roman Leventov 8277284d67 Add Checkstyle rule to force comments to classes and methods to be Javadoc comments (#4239) 2017-05-04 11:14:41 -07:00
Roman Leventov b9fd30e90a Add Checkstyle check to prohibit IntelliJ-style commented code lines (#4220)
* Add Checkstyle check to prohibit IntelliJ-style commented code lines

* Address comment

* Restore issue link
2017-04-27 18:11:25 -07:00
Gian Merlino 2ca7b00346 Update versions to 0.10.1-SNAPSHOT. (#4191) 2017-04-20 18:12:28 -07:00
Gian Merlino cc20133e70 Checkstyle rule to outlaw tabs. (#3988)
Tabs are the worst.
2017-02-28 23:52:53 -08:00
Gian Merlino 12317fd001 Bump version to 0.10.0-SNAPSHOT. (#3913) 2017-02-06 17:54:35 -08:00
Nishant Bangarwa a457cded28 Druid Extension to enable Authentication using Kerberos. (#3853)
* Add extension for supporting kerberos security

- This PR adds an extension for supporting druid authentication via
Kerberos.
- Working on the docs.

* Add docs

* review comments

* more review comments

* Block all paths by default

* more review comments - use proper Oid

* Allow extensions to override httpclient for integration tests

* Add kerberos lock to prevent multithreaded issues.

* review comment - remove enabled flag and fix router injection

* Add Cookie Handling and more detailed docs

* review comment - rename DruidKerberosConfig -> AuthKerberosConfig

* review comments

* fix travis failure on jdk7
2017-02-02 14:55:21 -06:00
Himanshu 17c6512ad8 integration-test update - use KafkaProducer in ITKafkaTest and add retries to task submission (#3888)
* use KafkaProducer instead of old Producer in ITKafkaTest

* add retries to OverlordResourceTestClient.submitTask(..)
2017-01-27 07:38:17 -08:00
Parag Jain 515caa8a85 ITKafkaIndexingServiceTest fixes (#3872)
* remove wait between sends, wait for ingestion to complete before querying

send fixed number of events

more fixes

* handle interrupted exception

* remove while

* review comments
2017-01-22 14:37:35 -08:00
Himanshu 7004f5d499 make ITKafkaTest less non-deterministic (#3856) 2017-01-17 16:52:51 -06:00
Jihoon Son d80bec83cc Enable auto license checking (#3836)
* Enable license checking

* Clean duplicated license headers
2017-01-10 18:13:47 -08:00
Jonathan Wei 2bfcc8a592 First and Last Aggregator (#3566)
* add first and last aggregator

* add test and fix

* moving around

* separate aggregator valueType

* address PR comment

* add finalize inner query and adjust v1 inner indexing

* better test and fixes

* java-util import fixes

* PR comments

* Add first/last aggs to ITWikipediaQueryTest
2016-12-16 15:26:40 -08:00
Jihoon Son 5e39578eee Enable parallel test (#3774)
* Enable parallel test

* Remove unnecessary NotThreadSafe annocation

* Randomize the start port when finding available ports

* Fix test failure

* Change to handle all negatives
2016-12-14 21:05:56 -08:00
Himanshu ed322a4beb remove size from default analysisTypes list for segmentMetadata query (#3773) 2016-12-13 18:01:21 -08:00
Nishant 361af4c94f Wait for any pending realtime task to complete before disabling datasource (#3757)
Noticed this in our internal testing, sometimes realtime index tasks in
kafkaIndexing service can get stuck waiting for handoff if datasource
is disabled before there task completion.
This is a workaround to ensure integration tests do not hit this case
until https://github.com/druid-io/druid/issues/1729 is fixed.
2016-12-07 10:17:16 -08:00
Parag Jain 877992fe63 use sync kafka producer for deterministic test (#3721) 2016-11-29 10:04:51 -08:00
Gian Merlino 657e4512d2 Checkstyle checks for AvoidStaticImport, UnusedImports. (#3660)
Excludes tests from AvoidStaticImport, since those are used often there and
I didn't want to make this changeset too large. Production code use was minimal
and I switched those to non-static imports.
2016-11-05 11:34:36 -07:00
Nishant b961b6a69f Add configs to enable running integration tests for cluster running behind proxy (#3646)
* Add configs to enable running integration tests for cluster running behind proxy

As part of https://issues.apache.org/jira/browse/KNOX-758
I am working on adding support for proxying druid queries & UIs using
Apache KNOX gateway.
This PR adds support for integration-tests to be run using the proxy
gateway.
Changes Include -
1) Instead of hostName and port, ability to specify url in the config
file.
2) tests now use HTTPClient defined in DruidTestModule that can pass in
the request.

Note - the config changes are backwards compatible and existing configs
should work fine.

* review comments

* review comments
2016-11-04 08:19:38 -07:00
Akash Dwivedi 4b3bd8bd63 Migrating java-util from Metamarkets. (#3585)
* Migrating java-util from Metamarkets.

* checkstyle and updated license on java-util files.

* Removed unused imports from whole project.

* cherry pick metamx/java-util@826021f.

* Copyright changes on java-util pom, address review comments.
2016-10-21 14:57:07 -07:00
Nishant 8ea5f9324d Integration Tests - fix middlemanager property name in doc (#3586) 2016-10-18 08:23:34 -05:00
Gian Merlino 40f2fe7893 Bump versions to 0.9.3-SNAPSHOT (#3524) 2016-09-29 13:53:32 -07:00
David Lim 96fcca18ea update KafkaSupervisor to make HTTP requests to tasks in parallel where possible (#3452) 2016-09-20 22:51:15 +05:30
Himanshu 2235988069 update wikipedia search query in the integration tests as per the fix in commit 0076b5f (#3420) 2016-09-01 10:13:17 -07:00
Himanshu be79b095ba fixing expected result for segmentMetadata query in integration tests (#3318) 2016-08-03 12:13:27 -07:00
Himanshu b0fa274481 fix segmentMetadata query results in integration tests (#3288) 2016-07-26 14:05:14 -07:00
Charles Allen 8b21036833 Make integration test dependency copy run in batch mode (#3152) 2016-06-16 18:44:15 -07:00
Gian Merlino ebf890fe79 Update master version to 0.9.2-SNAPSHOT. (#3133) 2016-06-13 13:10:38 -07:00
Himanshu 8e2742b7e8 adding QueryGranularity to segment metadata and optionally expose same from segmentMetadata query (#2873) 2016-05-03 11:31:10 -07:00
Nishant 9440598685 fix integration-test (#2814) 2016-04-11 15:46:32 -05:00
Robin 10ec53dc1c fix calculation of expected timestamp for query in ITRealtimeIndexTaskTest 2016-04-01 10:47:17 -05:00
Himanshu 8a473111c5 Merge pull request #2650 from rasahner/IT_index_hadoop
add test for batch indexing from hadoop
2016-03-23 13:04:55 -05:00
Gian Merlino 738dcd8cd9 Update version to 0.9.1-SNAPSHOT.
Fixes #2462
2016-03-17 10:34:20 -07:00
Robin 1205c572bb add count field to expected returns for search queries 2016-03-16 09:45:38 -05:00
Robin e7a7ecd65d add test for batch indexing from hadoop 2016-03-14 20:02:30 -05:00
rasahner 2861e854f0 Merge pull request #2540 from pjain1/remove_kill
Remove extra parameter from deleteDataSourceSpecificInterval endpoint and correct exception message for invalid interval
2016-03-14 11:16:23 -05:00
Nishant 9f8faabddb Merge pull request #2469 from pdeva/patch-10
correct service names
2016-02-26 21:15:58 +05:30
Parag Jain b82b487f20 remove extra kill parameter 2016-02-24 17:16:18 -06:00
pdeva 4725522759 update middlemanager.conf 2016-02-18 21:47:50 -08:00
Bingkun Guo 72ab621c2a Integration tests: print expected results in json format 2016-02-18 00:14:49 -06:00
Robin 19774b5e9a integration-tests/README: fix link to docker-install 2016-02-17 11:21:14 -06:00
Robin a785f0e6ce query tests: add max and min to segment metadata query results 2016-02-16 13:23:46 -06:00
Robin 34b143ae10 integration-tests: stop using kill=false to disable segments 2016-02-14 12:54:43 -06:00
Fangjin Yang 1430bc2c88 Merge pull request #2276 from harshjain2/feature-2021
Fix for issue 2021.
2016-02-10 17:04:45 -08:00
Harsh Jain 41730b96d4 Fix for issue 2021 2016-01-25 02:48:22 +05:30
Himanshu Gupta dccf636676 adding null aggregators and false hasMultipleValues to expected results of segment metadata query 2016-01-22 13:28:25 -06:00
Gian Merlino 42ee473052 ZooKeeper 3.4.7 was unreleased, revert to 3.4.6. 2016-01-17 21:31:29 -08:00
Bingkun Guo 36ca5b1703 change log level to info for ITKafkaTest 2016-01-11 16:03:57 -06:00
Robin 33504d0190 log start/end of each integration test, make test method names unique 2016-01-07 13:15:17 -06:00
Robin e280ab5f07 update zookeeper version to 3.4.7 2016-01-04 11:47:02 -06:00
Himanshu Gupta 47910a8333 adding numRows attribute to segment metadata query expected results 2015-12-17 19:31:29 -06:00
Fangjin Yang 3667f283ce Merge pull request #2030 from rasahner/intTestRmAllSegments
int tests: in unloadAndKill, remove all segments if no date range given
2015-12-09 11:12:14 -08:00
sahner ae4357dfab integration tests: in unloadAndKill, remove all segments if no date range given 2015-12-08 09:16:29 -06:00
Himanshu Gupta 62ba9ade37 unifying license header in all java files 2015-12-05 22:16:23 -06:00
Xavier Léauté 350fdbeec1 switch to Java 8 + cleanup 2015-11-17 13:35:06 -08:00
Nishant 4486b3fe8b Fix ITUnionQueryTest failing on machines with 1 core
ITUnionQueryTest runs 4 realtime tasks and runs a unionQuery over them,
the worker is not able to run all the tasks concurrently on machines
with less cpu cores as it picks default worker capacity.
Explicitly specify worker capacity to avoid this.
2015-11-13 13:46:36 -08:00
Fangjin Yang 846bef3d7a Merge pull request #1964 from metamx/update-docker-stuff
update integration-test docker instructions for docker-machine
2015-11-12 16:52:57 -08:00
Xavier Léauté bd7d2c9b3a update integration-test docker instructions for docker-machine 2015-11-12 16:44:28 -08:00
Xavier Léauté fa6142e217 cleanup and remove unused imports 2015-11-11 12:25:21 -08:00
sahner 040db413c5 wikipediaQueryTest: use very big timeouts in hopes of preventing spurious test failures 2015-10-31 09:13:02 -05:00
sahner 6c9ba72dea some changes for support of our tests 2015-10-26 07:48:56 -05:00
Xavier Léauté e4ac78e43d bump next snapshot to 0.9.0 2015-10-20 13:46:13 -07:00
sahner f31ae89724 enhance test support and update realtime test to use serverTime 2015-10-15 13:43:42 -05:00
Xavier Léauté 4c2c7a2c37 update version to 0.8.3 2015-10-14 21:40:55 -07:00
Nishant 9c5166c89c fix broken integration-test 2015-09-15 23:20:22 +05:30
Gian Merlino 940e1aa3eb Replace funky imports with standard ones.
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
2015-08-28 18:02:05 -07:00
Nishant e6b20db62d Support for JSON Smile format for EventReceiverFirehoseFactory
Support for Smile format for EventReceiverFireshoseFactory.
would help in decreasing network load on middle managers ingesting high
data volumes.

review comments
2015-08-20 19:52:45 +05:30
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Charles Allen db19d2d547 Revert "Update to guice 4.0" 2015-08-14 09:26:07 -07:00
Charles Allen 7e61216287 Update to guice 4.0
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
2015-08-10 13:57:18 -07:00
Bingkun Guo f9a64f31f3 use IOUtils.toString in getTaskAsString 2015-07-27 13:53:30 -05:00
sahner df5bc8974f add ability to get configuration information from a file 2015-07-17 10:19:42 -05:00
Charles Allen 456ad9ffba Merge pull request #1529 from metamx/update-versions
inrement version
2015-07-15 13:25:31 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Charles Allen 80a18243dd Minor change to integration-tests README formatting 2015-07-14 17:01:06 -07:00
Xavier Léauté 0a5bb909a2 [maven-release-plugin] prepare for next development iteration 2015-06-18 17:35:19 -07:00
Xavier Léauté 59c6b2b279 [maven-release-plugin] prepare release druid-0.8.0-rc1 2015-06-18 17:35:14 -07:00
fjy 7a6acf5c1b update pom to 0.8 2015-05-11 19:41:58 -06:00
Bingkun Guo cf155e4eba Fix an issue that after broker forwards GroupByQuery to historical, havingSpec is still applied
on postAggregations which are removed in the forwarded query.

Add a unit test to replicate the issue.
Add a query that can replicate this issue into integration test.
2015-04-17 13:00:41 -05:00
fjy aea7f9d192 [maven-release-plugin] prepare for next development iteration 2015-03-30 16:35:24 -07:00
fjy 060d7aef03 [maven-release-plugin] prepare release druid-0.7.1 2015-03-30 16:35:20 -07:00
nishantmonu51 084c57bf0d fix mvn soft link for integration tests
add comment and remove mvn in default container

remove unused links
2015-03-24 10:16:05 +05:30
fjy b389cfe404 [maven-release-plugin] prepare for next development iteration 2015-03-19 12:38:17 -07:00
fjy 60e7d543cc [maven-release-plugin] prepare release druid-0.7.1-rc1 2015-03-19 12:38:13 -07:00
Xavier Léauté b167dcf82c [maven-release-plugin] prepare for next development iteration 2015-02-23 14:28:06 -08:00
Xavier Léauté e81ac2ba43 [maven-release-plugin] prepare release druid-0.7.0 2015-02-23 14:27:58 -08:00
Fangjin Yang 25db9abb7f Merge pull request #1138 from metamx/better-default-hostname
Better default hostname
2015-02-18 17:37:34 -08:00
Xavier Léauté 6b391e4055 expose additional ports in Docker to help debugging 2015-02-18 16:46:06 -08:00
Xavier Léauté 78df7f6165 Move Druid release artifacts to Sonatype
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
2015-02-13 14:26:31 -08:00
fjy d29740ed9f [maven-release-plugin] prepare for next development iteration 2015-02-12 16:16:00 -08:00
fjy 211fd15b7e [maven-release-plugin] prepare release druid-0.7.0-rc3 2015-02-12 16:15:56 -08:00
nishantmonu51 feebcd757f fix-tests
add columns to parseSpec
2015-02-13 02:19:27 +05:30
fjy 708759e1e0 Update http-client to 1.0.0 2015-02-10 13:36:47 -08:00
fjy 1f12c5b2f1 [maven-release-plugin] prepare for next development iteration 2015-02-03 12:06:49 -08:00
fjy e82d431be7 [maven-release-plugin] prepare release druid-0.7.0-rc2 2015-02-03 12:06:41 -08:00
Fangjin Yang 92e616de11 Merge pull request #1077 from metamx/remove-unused-imports
remove unused imports
2015-02-02 10:45:27 -08:00
nishantmonu51 ba932bb1f2 remove unused imports 2015-02-02 21:53:39 +05:30
fjy d05032b98a towards a community led druid 2015-01-31 20:57:36 -08:00
nishantmonu51 f2d66858c5 add test for IngestSegmentFirehose
fix failing integration tests
add test for ingestSegmentFirehose
2015-01-27 23:06:30 +05:30
fjy 1f94de22c6 [maven-release-plugin] prepare for next development iteration 2015-01-20 14:23:55 -08:00
fjy 17476edc31 [maven-release-plugin] prepare release druid-0.7.0-rc1 2015-01-20 14:23:51 -08:00
nishantmonu51 f69e0591b0 working integration tests 2015-01-07 14:23:38 -08:00