Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,739 Commits 113 Branches 504 Tags 726 MiB
Commit Graph

2594 Commits

Author SHA1 Message Date
Bingkun Guo 57c8e56451 Fix ClientInfoResource 
Fix ClientInfoResource so that it doesn't return empty list for overshadowed segments
 2015-10-02 14:12:55 -05:00
Himanshu 166c4fcf46 Merge pull request #1795 from metamx/announcerTestTransactions 
Try and make AnnouncerTest a bit more predictable
 2015-10-02 13:04:33 -05:00
Charles Allen 7d635a2ce2 Try and make AnnouncerTest a bit more predictable 2015-10-02 09:19:40 -07:00
Xavier Léauté 5db4c5a089 move cache bindings into a dedicated module 2015-10-02 00:29:05 -04:00
Himanshu Gupta b9740a727b unit test for IngestSegmentFirehose 2015-10-01 16:06:04 -05:00
Charles Allen 2d847ad654 Merge pull request #1730 from metamx/union-queries-fix 
fix #1727 - Union bySegment queries fix
 2015-09-29 12:23:25 -07:00
Nishant 573aa96bd6 fix #1727 - Union bySegment queries fix 
Fixes #1727.
revert to doing merging for results for union queries on broker.

revert unrelated changes

Add test for union query runner

Add test

remove unused imports

fix imports

fix renamed file

fix test

update docs.
 2015-09-29 23:32:36 +05:30
Gian Merlino 348172203f OverlordRedirectInfo: Fix ability to detect that there is no leader. 2015-09-25 09:30:09 -07:00
Gian Merlino 906e99d631 RedirectFilter: User agents are more likely to preserve method on 307. 2015-09-25 08:54:23 -07:00
Gian Merlino 3aba401ee0 SQLMetadataConnector: Retry table creation, in case something goes wrong. 
Also rejigger table creation methods to not take a DBI. It's already available
inside the connector, and everyone was just using that one anyway.
 2015-09-24 21:39:36 -07:00
Charles Allen 4308aa93e2 Add meter metrics for memcached 2015-09-22 12:50:54 -07:00
Charles Allen 46ba60f10f Enable logging for memcached in factory 2015-09-22 11:13:47 -07:00
Xavier Léauté 1c86eac9e1 Merge pull request #1760 from metamx/memcached082Updates 
Update memcached client for better concurrency in metrics.
 2015-09-22 10:59:43 -07:00
Xavier Léauté 35caa753aa Merge pull request #1761 from gianm/local-path-for-hadoop 
LocalDataSegmentPusher: Fix for Hadoop + relative paths.
 2015-09-22 08:57:44 -07:00
Gian Merlino 4efbe649a3 LocalDataSegmentPusher: Fix for Hadoop + relative paths. 2015-09-22 07:44:12 -07:00
Charles Allen e6f07a832d Update memcached client for better concurrency in metrics. Also fixes another injection problem 2015-09-22 07:40:24 -07:00
Himanshu 34a8fbcd68 Merge pull request #1755 from metamx/update-druid-api 
update druid-api for timestamp parsing speedup
 2015-09-21 19:43:13 -05:00
Gian Merlino 490b1aea2a Merge pull request #1758 from metamx/fixMemcachedInjection 
Fix memcached cache provider injection and add test
 2015-09-21 15:36:29 -07:00
Charles Allen 37eb83a1d0 Fix memcached cache provider injection and add test 2015-09-21 15:00:22 -07:00
Xavier Léauté 8f489c1f66 fix type cast warnings 2015-09-21 09:57:29 -07:00
Xavier Léauté af86c0e6ea update druid-api + java-util for timstamp parsing speedup 2015-09-21 09:57:29 -07:00
Himanshu Gupta 117a6754b5 improving msging when indexing service is not found 2015-09-18 23:07:44 -05:00
Fangjin Yang 6605a60ef0 Merge pull request #1747 from jon-wei/realtime_corrupt_sink_fix 
Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4)
 2015-09-18 15:22:50 -07:00
jon-wei 9f6bb03ef4 Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4) 2015-09-18 14:53:37 -07:00
Xavier Léauté df4c8a3aa5 balance memcached cache across multiple connections 2015-09-18 09:52:16 -07:00
Himanshu Gupta a7b1cacbbf make NumberedShardSpec "elastic" and UTs 2015-09-17 08:38:48 -05:00
Gian Merlino 9705c5139b Merge pull request #1732 from jon-wei/segmentmeta 
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
 2015-09-16 12:36:25 -07:00
Himanshu Gupta 74f4572bd4 Lazily deserialize "parser" to InputRowParser in DataSchema 
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
 2015-09-16 10:58:13 -05:00
Xavier Léauté b464da438c Merge pull request #1688 from metamx/moreMemcachedMetrics 
More memcached metrics
 2015-09-15 15:33:51 -07:00
jon-wei 193fb4fdfc Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups 2015-09-14 19:41:42 -07:00
Himanshu 5ff92664f8 Merge pull request #1696 from metamx/cpuTimeReporting 
Add CPU time to metrics for segment scanning.
 2015-09-14 10:53:55 -05:00
Himanshu Gupta ebdb612933 composing emitter module to use multiple emitters together 2015-09-09 16:45:50 -05:00
Charles Allen 5813856819 More memcached metrics 2015-09-08 13:34:58 -07:00
Charles Allen fcf5cae81d Add CPU time to metrics for segment scanning. 2015-09-08 13:34:19 -07:00
Gian Merlino 062a47fba4 Modify Plumbers in these ways, 
1) Persist using Committer instead of Runnable. (Although the metadata object
   is ignored in this patch)

2) Remove the getSink method.

3) Plumbers are now responsible for time-based and hydrant-full-based periodic
   committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
 2015-09-03 11:13:06 -07:00
Nishant e79572b4bc Merge pull request #1637 from metamx/peonPlaysNiceOnShutdown 
Make jetty scheduler threads daemon thread
 2015-09-03 00:03:20 +05:30
Nishant 0096e6a0a0 Merge pull request #1658 from metamx/cleanupJDBCExtractionNamespaceTest 
Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest
 2015-09-02 23:49:49 +05:30
nishant 0835e12f2a add endpoint to fetch rule history for all datasources. 
review comments

Review comment

fix compilation
 2015-08-31 14:05:52 +05:30
Fangjin Yang dfe3be0f6d Merge pull request #1689 from metamx/lz4TranscoderUpdate 
Update LZ4Transcoder to match Compressed strategy factory type.
 2015-08-29 12:31:59 -07:00
Charles Allen 413c0d1c9e Update LZ4Transcoder to match Compressed strategy factory type. 2015-08-28 19:19:48 -07:00
Gian Merlino 940e1aa3eb Replace funky imports with standard ones. 
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
 2015-08-28 18:02:05 -07:00
Himanshu ceaa49ec4f Merge pull request #1681 from gianm/ingest-segment-overlapping 
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat
 2015-08-28 11:12:51 -05:00
Gian Merlino e575ef2b20 Remove unused Indexer interface. 2015-08-28 08:40:47 -07:00
Gian Merlino 414a6fb477 Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat. 
Fixes #1678. IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
 2015-08-28 07:32:41 -07:00
Himanshu Gupta 2e0dd1d792 adding UTs and addressing review comments to 
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
 2015-08-27 20:50:46 -05:00
lvjq 2237a8cf0f kafka 8 simple consumer firehose 2015-08-27 20:50:46 -05:00
Charles Allen 10af233b37 Merge pull request #1622 from metamx/smile-eventreceiverfirehose 
Support for JSON Smile format for EventReceiverFirehoseFactory
 2015-08-27 10:42:25 -07:00
Charles Allen ac8e32b58e Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest 2015-08-26 23:05:51 -07:00
Gian Merlino 5b67ec6cda Merge pull request #1672 from metamx/derbyRule 
Migrate TestDerbyConnector to a JUnit @Rule
 2015-08-26 22:56:18 -07:00
Charles Allen e38cf54bc8 Migrate TestDerbyConnector to a JUnit @Rule 2015-08-26 21:47:40 -07:00
Xavier Léauté e9237611c5 Merge pull request #1668 from metamx/cachingClusteredClientCodeCleanup1 
Code cleanup for CachingClusteredClientTest
 2015-08-26 17:13:55 -07:00
Xavier Léauté c325b194c0 Merge pull request #1661 from pjain1/poll_rules 
Poll rules immediately after change
 2015-08-25 13:46:55 -07:00
Charles Allen 85aeda834c Code cleanup for CachingClusteredClientTest 2015-08-25 12:49:55 -07:00
Gian Merlino 2bf9a70bfa Consolidate SQL retrying by moving logic into the connectors. 
Also change boolean removeLock to void addLock in MetadataStorageActionHandler.
 2015-08-25 12:42:29 -07:00
Parag Jain 073c32f01b Poll rules immediately after change 2015-08-24 16:08:25 -05:00
Fangjin Yang e3964cd9f6 Merge pull request #1653 from metamx/fix-success-flag 
fix success flag emitted in router query metrics
 2015-08-21 07:56:57 -07:00
Nishant 6d85a62ed2 fix success flag emitted in router query metrics 2015-08-21 12:25:57 +05:30
Nishant e6b20db62d Support for JSON Smile format for EventReceiverFirehoseFactory 
Support for Smile format for EventReceiverFireshoseFactory.
would help in decreasing network load on middle managers ingesting high
data volumes.

review comments
 2015-08-20 19:52:45 +05:30
Fangjin Yang 6a0389297c Merge pull request #1634 from metamx/IrcFirehoseFixes 
Allow IrcFirehoseFactory to shutdown cleanly
 2015-08-18 21:06:34 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Charles Allen 13773865cf Make jetty scheduler threads daemon thread 2015-08-18 13:08:19 -07:00
Charles Allen cd2c3776f8 Allow IrcFirehoseFactory to shutdown cleanly 2015-08-17 13:24:41 -07:00
Himanshu Gupta 4d4aa8bfc6 refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable 
Conflicts:
	indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
 2015-08-14 14:44:22 -05:00
Charles Allen db19d2d547 Revert "Update to guice 4.0" 2015-08-14 09:26:07 -07:00
Charles Allen be89105621 Merge pull request #1602 from metamx/more-code-cleanup 
Some perf Improvements in Broker
 2015-08-11 13:51:49 -07:00
Charles Allen 7e61216287 Update to guice 4.0 
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
 2015-08-10 13:57:18 -07:00
Nishant b33c644da0 avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals() 
avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
profiling shows it took upto 6-12% of cpu time in
JodaUtils.condenseIntervals
 2015-08-06 13:51:17 +05:30
Charles Allen 9dd18de1a5 Merge pull request #1596 from metamx/console-artifact 
make console a maven dependency instead of filedump
 2015-08-04 15:28:45 -07:00
Xavier Léauté 07ee1acca2 make console a maven dependency instead of filedump 2015-08-04 09:49:34 -07:00
Fangjin Yang b05be0a104 Merge pull request #1593 from metamx/run-tests-offline 
Fix test so that it can run offline
 2015-08-04 09:23:40 -07:00
Nishant f490547faf Fix test so that it can run offline too 
Test was getting stuck for me when I tried to run it offline while
resolving the host. creating unresolved SocketAddress works around
this.
 2015-08-03 23:37:04 +05:30
Bingkun Guo 479b09c2dd fix transient error in BrokerServerViewTest 2015-07-31 13:33:55 -05:00
Fangjin Yang 22567946cf Merge pull request #1259 from metamx/queryTimeLookup 
Query Time Lookup
 2015-07-28 11:43:05 -10:00
Charles Allen ba59f8afc4 Merge pull request #1550 from himanshug/optionally_log_all_requests 
print *all* HTTP requests to log if configured
 2015-07-28 11:19:36 -07:00
Charles Allen 86ede702b1 Add namespaced lookups as extensions 
* Adds kafka, URI, and JDBC namespace defintions
* Add ability to explicitly rename using a "namespace" which is a particular data collection that is loaded on all realtime, historic nodes, and brokers. If any of these nodes has the namespace extension, ALL nodes have the namespace extension.
* Add namespace caching and populating (can be on heap or off heap)
* Add NamespaceExtractionCacheManager for handling caches
* Added ExtractionNamespace for handling metadata on the extraction namespaces
* Added ExtractionNamespaceUpdate for handling metadata related to updates
* Add extension which caches renames from a kafka stream (requires kafka8)
* Added README.md for the namespace kafka extension
* Added docs
* Added namespace/size, namespace/count, namespace/deltaTasksStarted metrics

Add static config for namespaces via `druid.query.extraction.namespace`
* This is a rebase of https://github.com/b-slim/druid/tree/static_config_only
 2015-07-28 11:14:14 -07:00
Himanshu Gupta 90b475975e Log all the requests to debug level in logger io.druid.jetty.RequestLog 2015-07-28 12:56:04 -05:00
nishant efb6e0649e simplify merging of results on broker. 
add benchmark

formatting and review comments

organize imports
 2015-07-27 11:46:11 -07:00
Xavier Léauté 6e8da5de86 Merge pull request #1441 from metamx/queryResourceFlusher 
Add flush prior to close on output stream in QueryResource
 2015-07-17 16:32:52 -07:00
Fangjin Yang e21195f987 Merge pull request #1469 from guobingkun/table_config 
Inconsistent property names for "druid.metadata.storage.tables.xxx"
 2015-07-17 07:43:19 -07:00
Charles Allen 456ad9ffba Merge pull request #1529 from metamx/update-versions 
inrement version
 2015-07-15 13:25:31 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Fangjin Yang a4cb0332fb Merge pull request #1524 from metamx/peonShutdown 
Make CliPeon run shutdown hooks properly
 2015-07-15 10:09:33 -07:00
Charles Allen 92d5c250ab Remove comment 2015-07-14 16:20:36 -07:00
Charles Allen 5eadd395e2 Move lots of executor service creation to Execs 2015-07-14 15:38:49 -07:00
Charles Allen 2b87ad307d Make CliPeon run shutdown hooks properly 
* Also make jetty threads daemon
 2015-07-14 15:15:26 -07:00
Fangjin Yang da8e86146e Merge pull request #1505 from metamx/improve-test 
Modify test to check for multiple shards for same interval
 2015-07-14 08:50:47 -07:00
Fangjin Yang 3f7ba58227 Merge pull request #1504 from metamx/fix-1447 
fix for #1447
 2015-07-14 08:50:08 -07:00
Himanshu 1affdf7bec Merge pull request #1502 from samjhecht/poll-alerting 
add alert on errors polling for rules
 2015-07-13 18:37:37 -05:00
samjhecht fc438e5a72 add alert on errors polling for rules 
reset retrystarttime after alerting
 2015-07-13 09:03:51 -07:00
Himanshu 725086cc89 Merge pull request #1506 from gianm/realtime-plumber-nulls 
Consider null inputRows and parse errors as unparseable during realtime ingestion.
 2015-07-13 10:12:12 -05:00
Gian Merlino 9068bcd062 Consider null inputRows and parse errors as unparseable during realtime ingestion. 
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
 2015-07-11 20:40:03 -07:00
Himanshu e9b627bf55 Merge pull request #1476 from pjain1/improved_tests 
all the review comments have been taken care of, so merging.
 2015-07-09 16:25:30 -05:00
Nishant 1766341648 Modify test to check for multiple shards for same interval 
Modify test to check for multiple shards for same interval.
 2015-07-09 20:35:31 +05:30
Parag Jain 58cc3954ff improved coordinator and datasourceresource tests 2015-07-09 09:46:54 -05:00
Nishant 5fe27fe4ad fix for #1447 
fixes #1447
 2015-07-09 19:05:48 +05:30
Nishant 184b12bee8 fix groupBy caching to work with renamed aggregators 
Issue - while storing results in cache we store the event map which
contains aggregator names mapped to values. Now when someone fire same
query after renaming aggs, the cache key will be same but the event
will contain metric values mapped to older names which leads to wrong
results.
Fix - modify cache to not store raw event but the actual list of values
only.

review comments + fix dimension renaming

review comment
 2015-07-09 11:48:26 +05:30
Nishant fe36f7f852 emit alert for DB polling error 2015-07-07 19:51:36 +05:30
Fangjin Yang d846f3ced6 Merge pull request #1498 from rasahner/doc_localFirehose 
say explicitly that local firehose searches dirs recursively for files
 2015-07-05 15:34:36 -07:00
sahner acd20e8c00 say explicitly that local firehose searches directories recursively for files 2015-07-05 14:46:44 -05:00
Charles Allen f2919b98b1 Merge pull request #1471 from metamx/direct-client-metrics 
add query/node/time metrics to DirectDruidClient
 2015-06-30 22:06:44 -07:00
Bingkun Guo 4a0ae7d8d5 Fix inconsistent druid property names for "druid.metadata.storage.tables.xxx" between document and code 2015-06-29 10:12:30 -05:00
Himanshu 8172bdff1c Merge pull request #1468 from guobingkun/fix_npe_for_storage_action_handler 
Fix npe thrown from SQLMetadataStorageActionHandler
 2015-06-29 09:57:51 -05:00
Xavier Léauté 28fa1642b9 add node time metrics to DirectDruidClient 2015-06-26 17:57:44 -07:00
Charles Allen fbcac10e00 Remove metrics emitting from caching clustered client 2015-06-26 10:49:13 -07:00
Bingkun Guo e4e18b17b5 fix npe thrown from getEntry() and getStatus() SQLMetadataStorageActionHandler due to a non-exist entryId 2015-06-26 10:40:56 -05:00
Xavier Léauté 0c85c8c60a Merge pull request #1462 from pjain1/validate_rules 
tieredReplicants can be null
 2015-06-24 13:29:27 -07:00
Parag Jain ff738ace05 converting null tieredeplicants to default and not allowing empty ones 2015-06-24 14:55:12 -05:00
Fangjin Yang 0c13fd8018 Merge pull request #1450 from metamx/cache-key-distribution 
Fix bad distribution of cache keys across cache nodes
 2015-06-22 17:58:15 -07:00
Himanshu 34c8f4c3da Merge pull request #1437 from metamx/limitUsedSegmentInterval 
Add start and end bounds to getUsedSegmentsForInterval
 2015-06-22 14:11:38 -05:00
Charles Allen 8d91c8d399 Add start and end bounds to getUsedSegmentsForInterval 
* Add unit tests
* Add javadoc for methods in IndexerMetadataStorageCoordinator
 2015-06-22 11:46:53 -07:00
Xavier Léauté f1951b253c Fix bad distribution of cache keys across nodes 
With the existing hash function some nodes could end up with 3 times the
number of keys as others. The following changes improve that to roughly
less than 5% differences across nodes.

- switch from fnv-1a to murmur3_128 hash
- increase repetitions for ketama algorithm
- test to analyze distribution

Also updates spymemcached for recent bugfixes
 2015-06-19 15:30:35 -07:00
Xavier Léauté 0a5bb909a2 [maven-release-plugin] prepare for next development iteration 2015-06-18 17:35:19 -07:00
Xavier Léauté 59c6b2b279 [maven-release-plugin] prepare release druid-0.8.0-rc1 2015-06-18 17:35:14 -07:00
Xavier Léauté d276d2c90b Merge pull request #1444 from druid-io/logging-improvement 
Separate bootstrap threads from loading threads on historical startup
 2015-06-17 15:24:07 -07:00
Eric Tschetter 06c97b6d7d Separate out the "bootstrapping" threads from the "loading" threads. 
It is highly recommended to keep loading threads single threaded.  There can be benefits to having multiple bootstrapping threads, though.  Those bootstrapping threads shouldn't be kept alive for the whole life of the process, however.

Also, adjust some logging and do some code cleanup.
 2015-06-17 14:58:28 -07:00
Fangjin Yang 0539e0073d Merge pull request #1442 from guobingkun/broker_server_view_test 
Unit test for BrokerServerView
 2015-06-16 19:45:20 -07:00
Eric Tschetter 1aad7ce521 Add a bit more information to the mapping logging logic. 2015-06-16 19:33:38 -07:00
Bingkun Guo 446d01cc53 Unit test for BrokerServerView 2015-06-16 15:20:32 -05:00
Charles Allen 69bbe21512 Add flush prior to close on output stream in QueryResource 2015-06-16 11:40:56 -07:00
Xavier Léauté 9cf8662aeb Merge pull request #1427 from guobingkun/fix_issue_1016 
Fix duplicate extension loading issue described in #1016
 2015-06-10 14:42:24 -07:00
Bingkun Guo bdf4d541e3 Fix issue #1016 that if user specifies extension coordinates, the local extensions will be loaded twice, 
which could cause Guice duplicate binding errors.
Add unit test to replicate duplicate extension issue. Update documents. Add an package accessible getter for loadersMap for testing only.
Notice that extensions explicitly specified in druid.extensions.coordinates have a higher priority than ones included in the classpath.
Extension modules that don't have a canonical class name will be ignored.
 2015-06-10 15:48:03 -05:00
Xavier Léauté e5937482bc Merge pull request #1431 from pjain1/coordinator_ut 
Druid Coordinator test
 2015-06-10 07:24:39 -07:00
Parag Jain b72e6fd51a Druid Coordinator Unit test 2015-06-09 13:51:01 -05:00
Bingkun Guo 22607d19be Make AbstractHttpClientProvider contents lazily instantiated 2015-06-08 09:53:26 -05:00
Bingkun Guo 1646f650b6 Unit tests for LoadQueuePeon and SegmentLoaderLocalCacheManager, JavaDoc for LoadPeonCallback 2015-06-04 17:28:22 -05:00
Himanshu f81a0d9210 Merge pull request #1425 from guobingkun/guice_fix 
Anyways, @guobingkun , if possible,  can you please test proposed change and send another PR to fix #1016 .  but that doesn't necessarily hold this PR, so I'm merging this.
 2015-06-04 14:43:26 -05:00
Himanshu f44f7f07de Merge pull request #1424 from pjain1/validate_rules 
Check lower bound of replicant values for rules
 2015-06-04 14:16:50 -05:00
Parag Jain 1030899cdf Check lower bound of replicant values for rules 2015-06-04 13:47:43 -05:00
Bingkun Guo dc26350480 Separate overwrite of Guice modules into 2 steps so that extension modules can overwrite cli modules 2015-06-04 09:57:56 -05:00
nishant 21864a9407 reduce test runtime 
fix check

fix test
 2015-06-04 08:35:12 +05:30
Charles Allen 1c4d42bc15 Replace "timeout" with QueryContextKeys.TIMEOUT 2015-06-02 14:49:21 -07:00
fjy 185c99deec Add schema check for overlap between dimension and metric names 2015-05-28 22:18:08 -07:00
fjy be2a35188e Additional schema validations and better logs for common extensions 2015-05-27 16:25:02 -07:00
Charles Allen 051c3ccede Add futures to SQLMetadataSegmentManager and SQLMetadataRuleManager 2015-05-15 16:08:22 -07:00
Xavier Léauté 3c3db7229c Merge pull request #1355 from himanshug/long_max_min_aggregators 
Long max/min aggregators
 2015-05-13 12:08:11 -07:00
Himanshu Gupta d0ec945129 adding aliases doubleMax and doubleMin for max and min respectively 
renamed all [Max/Min]*.java to [DoubleMax/DoubleMin]*.java and created [Max/Min]AggregatorFactory.java which can be removed when we dont need the min/max aggregator type backward compatibility
 2015-05-13 09:25:41 -05:00
fjy 7a6acf5c1b update pom to 0.8 2015-05-11 19:41:58 -06:00
Fangjin Yang a2dc58cd2d Merge pull request #1345 from pjain1/unit_test_warn_fix 
fix warn msg and some unit tests
 2015-05-08 08:06:20 -07:00
Parag Jain 01448d264c Fix warn msg and added some unit tests 2015-05-07 17:10:05 -05:00
Fangjin Yang 7608bf5799 Merge pull request #1342 from guobingkun/unit_test 
Fix a potential issue in DeterminePartitionsJob, and two unit tests, reformat code
 2015-05-05 09:24:23 -07:00
Bingkun Guo 1ee550dd91 Fix a potential issue in DeterminePartitionsJob by making HadoopDruidIndexerConfig non-static, and two unit tests for DeterminPartitionsJob and LocalDataSegmentKiller 2015-05-04 20:00:29 -07:00
Gian Merlino e69d82a2b4 Realtime: Delay firehose connection until job is started. 
Some firehoses (like the Kafka firehose) acquire input resources when they
connect, so it helps to delay this until after plumber.startJob() runs.
 2015-05-04 10:54:07 -07:00
Gian Merlino 8eb441ec08 RealtimeManager: Include partitionNum in chief thread name. 2015-05-04 10:53:48 -07:00
Charles Allen 27e25fe513 fix IRCFirehoseFactory json name/nick mismatch 
* Now it is nick as per wikipedia example
 2015-05-01 12:35:44 -07:00
Fangjin Yang 15ed308a4b Merge pull request #1322 from himanshug/fix_audit_tbl_idx 
fixing the typo for audit table (type, created_date) index creation
 2015-04-30 13:22:04 -07:00
fjy a7f9bb1a1e Additional coordinator logs to figure out when racey things might be happening 2015-04-30 08:26:52 -07:00
Himanshu Gupta 4aa9381f6a fixing the typo for audit table (type, created_date) index creation 2015-04-29 17:09:35 -05:00
nishant 6e84caf3d5 Proxy Requests from coordinator to overlord 
Add unit test and support https scheme

review comments
 2015-04-29 20:58:26 +05:30
Charles Allen 062a413ccd Fix DruidNodeTest for ipv6 only hosts 2015-04-28 13:11:47 -07:00
Xavier Léauté 721505c017 Merge pull request #1208 from druid-io/rework-metrics 
Schemaless metrics + additional metrics for things we care about
 2015-04-27 15:04:54 -07:00
fjy 963e5765bf Schemaless metrics + additional metrics for things we care about 2015-04-27 13:39:40 -07:00
nishant 958b34e0c2 fix exception during timeboundary 
Fix Exception during time boundary query where results can be
represented as integers
 2015-04-23 20:32:35 +05:30
Fangjin Yang 1addbe163a Merge pull request #1295 from anubhgup/master 
Fix type of values in sql update.
 2015-04-17 20:26:17 -07:00
Anubhav Gupta 4c240da4e9 Fix type of values in sql update. 2015-04-17 18:44:49 -07:00
fjy d260515a43 update druid-api version 2015-04-17 14:58:35 -07:00
Xavier Léauté ea5572d001 Merge pull request #1271 from metamx/strictErrorChecking 
Add stricter checking for potential coding errors
 2015-04-15 15:21:41 -07:00
Charles Allen abdeaa0746 Add stricter checking for potential coding errors 
Can use via `mvn clean compile test-compile -P strict'
 2015-04-15 14:52:25 -07:00
Xavier Léauté 3a3046ccf3 add support for dimension compression 
- compression for single-value dimensions using CompressedVSizeIntsIndexedSupplier
- makes dimension compression configurable via IndexSpec
- IndexSpec also enables configuring bitmap and metric compression
 2015-04-14 10:44:18 -07:00
Xavier Léauté 44e23a8267 Merge pull request #1224 from metamx/alert-improve 
skip raising false alert when the coordinator looses leadership
 2015-04-07 10:11:34 -07:00
Fangjin Yang 208e307915 Merge pull request #1251 from metamx/uriSegmentLoaders 
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
 2015-03-30 17:43:51 -07:00
fjy aea7f9d192 [maven-release-plugin] prepare for next development iteration 2015-03-30 16:35:24 -07:00
fjy 060d7aef03 [maven-release-plugin] prepare release druid-0.7.1 2015-03-30 16:35:20 -07:00
Charles Allen 1c6cbea89c Revert "Revert "Overhaul of SegmentPullers to add consistency and retries"" 
This reverts commit f904bc7858.
 2015-03-30 13:40:04 -07:00
Fangjin Yang f904bc7858 Revert "Overhaul of SegmentPullers to add consistency and retries" 2015-03-30 13:15:50 -07:00
Charles Allen 6d407e8677 Add URI handling to SegmentPullers 
* Requires https://github.com/druid-io/druid-api/pull/37
* Requires https://github.com/metamx/java-util/pull/22
* Moves the puller logic to use a more standard workflow going through java-util helpers instead of re-writing the handlers for each impl
  * General workflow goes like this: 1) LoadSpec makes sure the correct Puller is called with the correct parameters. 2) The Puller sets up general information like how to make an InputStream, how to find a file name (for .gz files for example), and when to retry. 3) CompressionUtils does most of the heavy lifting when it can
 2015-03-30 12:33:23 -07:00
Fangjin Yang e5653f0752 Merge pull request #1190 from vigiglobe/master 
Fix NPE when partionNumber 0 does not exist.
 2015-03-26 13:25:39 -07:00
Xavier Léauté 2c58e355fd Merge pull request #1239 from metamx/log4j2forRealz 
Exclude log4j from curator dependencies in favor of log4j-1.2-api
 2015-03-26 13:21:13 -07:00
Charles Allen 9cd6c08e96 Exclude log4j from curator dependencies in favor of log4j-1.2-api 2015-03-26 13:05:12 -07:00
Xavier Léauté 47cc11829d Merge pull request #1240 from himanshug/async_query_forwarding_servlet_router_fix 
adding router numMaxThread configuration for ProxyServlet HttpClient executor
 2015-03-26 10:15:44 -07:00
msprunck 942c17a2aa Remove timeline chunk count assumptions. 
* Replace with generic iterables
 2015-03-24 22:40:49 +01:00
Himanshu Gupta b131a4fd3b adding router numMaxThread configuration for ProxyServlet HttpClient executor 2015-03-23 21:58:01 -05:00
nishantmonu51 3b894f9fb9 add ut for TierSelectorStrategies 
fix test
 2015-03-24 08:18:28 +05:30
Xavier Léauté 23528299c9 Merge pull request #1235 from druid-io/comments-update 
Minor update to zk comments to make it more clear what is happening
 2015-03-23 14:02:08 -07:00
fjy a0ce1caaf8 minor update to zk comments to make it more clear what is happening 2015-03-23 11:47:43 -07:00
Himanshu Gupta b5b32dbcb4 in test explicitly setting maxThreads for ProxyServlet to work around https://tickets.puppetlabs.com/browse/TK-152 2015-03-23 12:27:19 -05:00
fjy b389cfe404 [maven-release-plugin] prepare for next development iteration 2015-03-19 12:38:17 -07:00
fjy 60e7d543cc [maven-release-plugin] prepare release druid-0.7.1-rc1 2015-03-19 12:38:13 -07:00
nishantmonu51 21cd75bcf6 improve alert 
skip doing false alert when the coordinator looses leadership
 2015-03-18 09:42:22 +05:30
Fangjin Yang 8d0dfd9af1 Merge pull request #1221 from metamx/AnnounceIdUUID 
Change announcement ID to a UUID instead of Timestamp
 2015-03-17 15:38:53 -07:00
Charles Allen 79b1443cc3 Change announcement ID to a UUID instead of Timestamp 
* Also add a UUIDUtils to the common package
 2015-03-17 15:34:04 -07:00
nishantmonu51 f9821d242f also log author ip address in audit log 2015-03-17 23:15:15 +05:30
cheddar 526a386f50 Merge pull request #1193 from metamx/reduce-test-verbosity 
move test output to file for cleaner build logs
 2015-03-12 15:42:34 -07:00
nishantmonu51 3ec4a30ab5 initial commit 
review comments

more refactoring and cleaning of redundant code

add UT + docs + more refactoring

fixes + review comments

more cleanup

end points to fetch history

review comments

remove unnecessary changes

review comments rename header name

review comments + add test for MetadataRulesManager

review comments docs
 2015-03-12 22:50:29 +05:30
Xavier Léauté fc613771d2 move test output to file for cleaner build logs 
- removes the need for special test log4j2.xml
 2015-03-11 17:56:19 -07:00
Xavier Léauté 9f242ed1ba Merge pull request #1182 from druid-io/better-docs 
Rework the druid docs and fix many mistakes
 2015-03-10 14:23:38 -07:00
fjy 064d65a009 rework the druid docs and fix many mistakes 2015-03-10 14:19:19 -07:00
Xavier Léauté ef842b2eae add test coverage 2015-03-09 14:32:13 -07:00
Xavier Léauté e161021902 Merge pull request #1171 from druid-io/fix-rt-task 
Add more metrics and fail when realtime tasks fail to persist and merge
 2015-03-06 16:42:23 -08:00
fjy 86a0a3cab8 add more metrics and fail when realtime tasks fail to persist and merge 2015-03-06 11:00:34 -08:00
Himanshu Gupta 956361dbbd Refactoring: Moved BaseJettyServerInitializer.java to JettyServerInitUtils.java and changed it from abstract class to a utils class containing static methods 
And, moved jetty stuff to its own package io.druid.server.initialization.jetty
 2015-03-05 19:33:27 -06:00
Himanshu Gupta a8648d8f3d feature to add servlet filters in a druid node via extension modules 2015-03-03 22:35:02 -06:00
Fangjin Yang d8e199a3f5 Merge pull request #1161 from himanshug/zk_ood_updates 
Fix to maintain correctness when out-of-order ZK updates are received
 2015-03-03 14:42:31 -08:00
Himanshu Gupta dda2a62ff5 on receiving ZK ADD/UPDATE events, get latest data from zookeeper instead of taking it from the event which might be stale due to event coming out of order etc 2015-03-03 16:21:56 -06:00
Fangjin Yang 9578304d6f Merge pull request #1130 from housejester/jde-building 
Various improvements to the tutorial and building experience.
 2015-03-02 14:48:32 -08:00
Fangjin Yang e8605c63a9 Merge pull request #1150 from himanshug/broker-parallel-chunk-process 
interval chunk query runner now processes individual chunk in a threadpool
 2015-03-02 13:50:23 -08:00
Himanshu Gupta 29039fd541 interval chunk query runner now processes individual chunk in a thread pool and prints metrics query/time per chunk 2015-03-02 15:45:09 -06:00
Fangjin Yang a76a24742b Merge pull request #1149 from himanshug/broker-get-datasource-update 
updating ClientInfoResource.getDatasource() with a full flag
 2015-03-02 13:28:27 -08:00
Himanshu Gupta a39f4f8bf4 updating ClientInfoResource.getDatasource() with a full flag to return all the served intervals for given data source along with the dimension and metric information 2015-03-02 14:18:52 -06:00
Xavier Léauté 17743b94f3 Cleanup router servlet + tests for gzip proxying 
- Uses method overrides instead of modified Jetty code, now that
  ProxyServlet provides enough method hooks for proper overrides.
  This means we may also benefit from any Jetty ProxyServlet fixes
- Adds test for async proxy servlet to make sure gzip encoding is
  properly proxied.
- Router now proxies POST requests for requests that are not Druid
  queries, by only treating /druid/v2/* endpoints as queries.
 2015-03-02 11:36:15 -08:00
Fangjin Yang c21086b6a8 Merge pull request #1113 from metamx/fix1111 
Make Realtime queries not submit to thread pools for the master merger
 2015-02-27 11:13:25 -08:00
James Estes 9cb5d9bd00 Various improvements to the tutorial and building experience. 
- Improve geo ip download stability
- Retry default geo ip download if corrupt
- Clarify docs on building & where to get artifact
 2015-02-25 18:49:16 -07:00
Xavier Léauté b633111c76 Merge pull request #1156 from druid-io/fix-log 
Print better messages when there are problems with ingestion
 2015-02-25 17:02:46 -08:00
fjy 9a3ed353dc Have a more meaningful message when events are dropped or unparseable 2015-02-25 16:59:46 -08:00
Himanshu Gupta 2192d0cb57 UTs for server 2015-02-25 15:45:58 -08:00
Xavier Léauté 69ba802abb Merge pull request #1109 from metamx/fixAnnouncerTest 
Make the Announcer closing and unannouncing atomic.
 2015-02-25 15:24:18 -08:00
Xavier Léauté aa9dbb773d Merge pull request #1153 from metamx/announcerTestForWaitingFix 
Use forWaiting() in AnnouncerTest
 2015-02-24 21:48:54 -08:00
Charles Allen 29a3522708 Use forWaiting() in AnnouncerTest 
* Helps reduce random failures in unit test on slow systems
 2015-02-24 08:35:07 -08:00
Xavier Léauté b167dcf82c [maven-release-plugin] prepare for next development iteration 2015-02-23 14:28:06 -08:00
Xavier Léauté e81ac2ba43 [maven-release-plugin] prepare release druid-0.7.0 2015-02-23 14:27:58 -08:00
Fangjin Yang 25db9abb7f Merge pull request #1138 from metamx/better-default-hostname 
Better default hostname
 2015-02-18 17:37:34 -08:00
Xavier Léauté 53d2b961c5 default to canonical hostname instead of localhost 2015-02-18 16:44:48 -08:00
fjy 3fab045b2d Go through and fix mistakes in tutorials and docs 2015-02-17 15:21:16 -08:00
cheddar 2a45501393 Merge pull request #1110 from metamx/move-to-sonatype 
Move Druid release artifacts to Sonatype
 2015-02-17 11:54:16 -08:00
extpvaan b42b41cca7 Bind value of column "partitioned" as boolean, as it is defined in the "create table" ddl. 2015-02-17 10:22:10 -08:00
Fangjin Yang 2593864c52 Merge pull request #1126 from gianm/md-insert-or-update-tx 
Use jdbi inTransaction for SQLMetadataConnector's insertOrUpdate.
 2015-02-16 08:23:05 -08:00
Gian Merlino e00de8c19f Use jdbi inTransaction for SQLMetadataConnector's insertOrUpdate. 
This fixes an issue where failed transactions were not properly rolled back.
 2015-02-16 07:42:43 -08:00
Xavier Léauté 78df7f6165 Move Druid release artifacts to Sonatype 
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
 2015-02-13 14:26:31 -08:00
fjy d29740ed9f [maven-release-plugin] prepare for next development iteration 2015-02-12 16:16:00 -08:00
fjy 211fd15b7e [maven-release-plugin] prepare release druid-0.7.0-rc3 2015-02-12 16:15:56 -08:00
Fangjin Yang 90bc62eb5c Merge pull request #1108 from metamx/improve-groupby-perf 
Improve groupby by removing conversion to case insensitive row
 2015-02-12 11:45:20 -08:00
Charles Allen bd500fcf4b Make the Announcer closing and unannouncing atomic. 2015-02-11 19:47:04 -08:00
Charles Allen e116600c7f Make Realtime queries not submit to thread pools for the master merger 
Fix https://github.com/druid-io/druid/issues/1111
 2015-02-11 15:47:40 -08:00
nishantmonu51 15cf432b74 remove conversion to case insensitive row 
this is not required after death to casing in 0.7
 2015-02-11 19:40:36 +05:30
Charles Allen d51b37cfd5 Remove QueryResource's ObjectMapper::copy() in favor of using the injected ObjectMapper directly 
* Workaround for https://github.com/FasterXML/jackson-databind/issues/696
 2015-02-10 16:50:23 -08:00
Xavier Léauté 31e248736d Merge pull request #1106 from druid-io/update-http-client 
Update http client to 1.0.0
 2015-02-10 13:37:27 -08:00
fjy 708759e1e0 Update http-client to 1.0.0 2015-02-10 13:36:47 -08:00
Fangjin Yang 1418cbb0e9 Merge pull request #1103 from metamx/fix-request-time 
fix change in request/time meaning introduced by d2c729a, fixes #1102
 2015-02-09 17:23:05 -08:00
Xavier Léauté 2db3144f04 fix change in request/time meaning introduced by d2c729a, fixed #1102 2015-02-09 16:41:30 -08:00
Fangjin Yang 040e771da2 Merge pull request #1099 from metamx/console_cleanup 
console cleanup
 2015-02-08 09:06:05 -08:00
Tarek Rached f47f0d51c7 removed unecessary logging 
fixed simplifyInterval logic
 2015-02-06 21:13:25 -08:00
Tarek Rached a0fc20fc09 better regex for durations 2015-02-06 17:02:31 -08:00
Tarek Rached dae91e5894 fixed links back to old console 2015-02-04 14:21:46 -08:00
fjy 1f12c5b2f1 [maven-release-plugin] prepare for next development iteration 2015-02-03 12:06:49 -08:00
fjy e82d431be7 [maven-release-plugin] prepare release druid-0.7.0-rc2 2015-02-03 12:06:41 -08:00
Xavier Léauté 4eff269536 Merge pull request #1079 from druid-io/cleanup-deps 
Remove non friendly dependencies from Druid
 2015-02-03 11:56:41 -08:00
Xavier Léauté ffb180e29d Merge pull request #1080 from metamx/fix-cost-balancer-strategy 
fix integer value overflow and add test
 2015-02-03 11:45:03 -08:00
nishantmonu51 6da6b9eeda fix integer value overflow and add test 2015-02-03 11:39:30 -08:00
fjy 3e5d338c8e Remove non friendly dependencies from Druid 2015-02-03 11:36:08 -08:00
Eric Tschetter 42eba986ce Towards consistent null handling 
This commit also includes
1) the addition of a context parameter on timeseries queries that allows it to ignore empty buckets instead of generating results for them
2) A cleanup of an unused method on an interface
 2015-02-02 12:53:07 -08:00
Fangjin Yang 92e616de11 Merge pull request #1077 from metamx/remove-unused-imports 
remove unused imports
 2015-02-02 10:45:27 -08:00
nishantmonu51 ba932bb1f2 remove unused imports 2015-02-02 21:53:39 +05:30
fjy d05032b98a towards a community led druid 2015-01-31 20:57:36 -08:00
Fangjin Yang b94b24c46e Merge pull request #1072 from metamx/common-aws-config 
Common AWS module + consistent credential chain for both S3 and auto-scaling
 2015-01-30 14:41:24 -08:00
Fangjin Yang 14837d7808 Merge pull request #1075 from metamx/fix-test-timeout 
increase test timeouts on DruidClusterBridgeTest
 2015-01-30 14:31:32 -08:00
Xavier Léauté cd7e768640 increase test timeouts on DruidClusterBridgeTest 2015-01-30 13:18:35 -08:00
fjy bc1405bee0 fix worker status path announcement with indexer zk config 2015-01-30 12:26:08 -08:00
Xavier Léauté f00872c41b move common AWS related classes into a separate module 2015-01-29 13:55:49 -08:00
Fangjin Yang 2d96b62e6c Merge pull request #1066 from metamx/fix-background-caching-ordering 
fix background caching not preserving result order
 2015-01-28 06:09:17 +08:00
Xavier Léauté 59c1907322 add test for out-of-order background-caching execution 2015-01-27 13:38:08 -08:00
fjy cd9ba1f852 fix serde bug 2015-01-27 18:47:36 +08:00