3aba401ee0
SQLMetadataConnector: Retry table creation, in case something goes wrong.
...
Also rejigger table creation methods to not take a DBI. It's already available
inside the connector, and everyone was just using that one anyway.
2015-09-24 21:39:36 -07:00
4308aa93e2
Add meter metrics for memcached
2015-09-22 12:50:54 -07:00
46ba60f10f
Enable logging for memcached in factory
2015-09-22 11:13:47 -07:00
1c86eac9e1
Merge pull request #1760 from metamx/memcached082Updates
...
Update memcached client for better concurrency in metrics.
2015-09-22 10:59:43 -07:00
35caa753aa
Merge pull request #1761 from gianm/local-path-for-hadoop
...
LocalDataSegmentPusher: Fix for Hadoop + relative paths.
2015-09-22 08:57:44 -07:00
4efbe649a3
LocalDataSegmentPusher: Fix for Hadoop + relative paths.
2015-09-22 07:44:12 -07:00
e6f07a832d
Update memcached client for better concurrency in metrics. Also fixes another injection problem
2015-09-22 07:40:24 -07:00
34a8fbcd68
Merge pull request #1755 from metamx/update-druid-api
...
update druid-api for timestamp parsing speedup
2015-09-21 19:43:13 -05:00
490b1aea2a
Merge pull request #1758 from metamx/fixMemcachedInjection
...
Fix memcached cache provider injection and add test
2015-09-21 15:36:29 -07:00
37eb83a1d0
Fix memcached cache provider injection and add test
2015-09-21 15:00:22 -07:00
8f489c1f66
fix type cast warnings
2015-09-21 09:57:29 -07:00
af86c0e6ea
update druid-api + java-util for timstamp parsing speedup
2015-09-21 09:57:29 -07:00
117a6754b5
improving msging when indexing service is not found
2015-09-18 23:07:44 -05:00
6605a60ef0
Merge pull request #1747 from jon-wei/realtime_corrupt_sink_fix
...
Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4)
2015-09-18 15:22:50 -07:00
9f6bb03ef4
Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4)
2015-09-18 14:53:37 -07:00
df4c8a3aa5
balance memcached cache across multiple connections
2015-09-18 09:52:16 -07:00
a7b1cacbbf
make NumberedShardSpec "elastic" and UTs
2015-09-17 08:38:48 -05:00
9705c5139b
Merge pull request #1732 from jon-wei/segmentmeta
...
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-16 12:36:25 -07:00
74f4572bd4
Lazily deserialize "parser" to InputRowParser in DataSchema
...
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
2015-09-16 10:58:13 -05:00
b464da438c
Merge pull request #1688 from metamx/moreMemcachedMetrics
...
More memcached metrics
2015-09-15 15:33:51 -07:00
193fb4fdfc
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-14 19:41:42 -07:00
5ff92664f8
Merge pull request #1696 from metamx/cpuTimeReporting
...
Add CPU time to metrics for segment scanning.
2015-09-14 10:53:55 -05:00
ebdb612933
composing emitter module to use multiple emitters together
2015-09-09 16:45:50 -05:00
5813856819
More memcached metrics
2015-09-08 13:34:58 -07:00
fcf5cae81d
Add CPU time to metrics for segment scanning.
2015-09-08 13:34:19 -07:00
062a47fba4
Modify Plumbers in these ways,
...
1) Persist using Committer instead of Runnable. (Although the metadata object
is ignored in this patch)
2) Remove the getSink method.
3) Plumbers are now responsible for time-based and hydrant-full-based periodic
committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
2015-09-03 11:13:06 -07:00
e79572b4bc
Merge pull request #1637 from metamx/peonPlaysNiceOnShutdown
...
Make jetty scheduler threads daemon thread
2015-09-03 00:03:20 +05:30
0096e6a0a0
Merge pull request #1658 from metamx/cleanupJDBCExtractionNamespaceTest
...
Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest
2015-09-02 23:49:49 +05:30
0835e12f2a
add endpoint to fetch rule history for all datasources.
...
review comments
Review comment
fix compilation
2015-08-31 14:05:52 +05:30
dfe3be0f6d
Merge pull request #1689 from metamx/lz4TranscoderUpdate
...
Update LZ4Transcoder to match Compressed strategy factory type.
2015-08-29 12:31:59 -07:00
413c0d1c9e
Update LZ4Transcoder to match Compressed strategy factory type.
2015-08-28 19:19:48 -07:00
940e1aa3eb
Replace funky imports with standard ones.
...
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
2015-08-28 18:02:05 -07:00
ceaa49ec4f
Merge pull request #1681 from gianm/ingest-segment-overlapping
...
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat
2015-08-28 11:12:51 -05:00
e575ef2b20
Remove unused Indexer interface.
2015-08-28 08:40:47 -07:00
414a6fb477
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat.
...
Fixes #1678 . IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
2015-08-28 07:32:41 -07:00
2e0dd1d792
adding UTs and addressing review comments to
...
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
2015-08-27 20:50:46 -05:00
2237a8cf0f
kafka 8 simple consumer firehose
2015-08-27 20:50:46 -05:00
10af233b37
Merge pull request #1622 from metamx/smile-eventreceiverfirehose
...
Support for JSON Smile format for EventReceiverFirehoseFactory
2015-08-27 10:42:25 -07:00
ac8e32b58e
Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest
2015-08-26 23:05:51 -07:00
5b67ec6cda
Merge pull request #1672 from metamx/derbyRule
...
Migrate TestDerbyConnector to a JUnit @Rule
2015-08-26 22:56:18 -07:00
e38cf54bc8
Migrate TestDerbyConnector to a JUnit @Rule
2015-08-26 21:47:40 -07:00
e9237611c5
Merge pull request #1668 from metamx/cachingClusteredClientCodeCleanup1
...
Code cleanup for CachingClusteredClientTest
2015-08-26 17:13:55 -07:00
c325b194c0
Merge pull request #1661 from pjain1/poll_rules
...
Poll rules immediately after change
2015-08-25 13:46:55 -07:00
85aeda834c
Code cleanup for CachingClusteredClientTest
2015-08-25 12:49:55 -07:00
2bf9a70bfa
Consolidate SQL retrying by moving logic into the connectors.
...
Also change boolean removeLock to void addLock in MetadataStorageActionHandler.
2015-08-25 12:42:29 -07:00
073c32f01b
Poll rules immediately after change
2015-08-24 16:08:25 -05:00
e3964cd9f6
Merge pull request #1653 from metamx/fix-success-flag
...
fix success flag emitted in router query metrics
2015-08-21 07:56:57 -07:00
6d85a62ed2
fix success flag emitted in router query metrics
2015-08-21 12:25:57 +05:30
e6b20db62d
Support for JSON Smile format for EventReceiverFirehoseFactory
...
Support for Smile format for EventReceiverFireshoseFactory.
would help in decreasing network load on middle managers ingesting high
data volumes.
review comments
2015-08-20 19:52:45 +05:30
6a0389297c
Merge pull request #1634 from metamx/IrcFirehoseFixes
...
Allow IrcFirehoseFactory to shutdown cleanly
2015-08-18 21:06:34 -07:00
13773865cf
Make jetty scheduler threads daemon thread
2015-08-18 13:08:19 -07:00
cd2c3776f8
Allow IrcFirehoseFactory to shutdown cleanly
2015-08-17 13:24:41 -07:00
4d4aa8bfc6
refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable
...
Conflicts:
indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
2015-08-14 14:44:22 -05:00
db19d2d547
Revert "Update to guice 4.0"
2015-08-14 09:26:07 -07:00
be89105621
Merge pull request #1602 from metamx/more-code-cleanup
...
Some perf Improvements in Broker
2015-08-11 13:51:49 -07:00
7e61216287
Update to guice 4.0
...
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
2015-08-10 13:57:18 -07:00
b33c644da0
avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
...
avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
profiling shows it took upto 6-12% of cpu time in
JodaUtils.condenseIntervals
2015-08-06 13:51:17 +05:30
9dd18de1a5
Merge pull request #1596 from metamx/console-artifact
...
make console a maven dependency instead of filedump
2015-08-04 15:28:45 -07:00
07ee1acca2
make console a maven dependency instead of filedump
2015-08-04 09:49:34 -07:00
b05be0a104
Merge pull request #1593 from metamx/run-tests-offline
...
Fix test so that it can run offline
2015-08-04 09:23:40 -07:00
f490547faf
Fix test so that it can run offline too
...
Test was getting stuck for me when I tried to run it offline while
resolving the host. creating unresolved SocketAddress works around
this.
2015-08-03 23:37:04 +05:30
479b09c2dd
fix transient error in BrokerServerViewTest
2015-07-31 13:33:55 -05:00
22567946cf
Merge pull request #1259 from metamx/queryTimeLookup
...
Query Time Lookup
2015-07-28 11:43:05 -10:00
ba59f8afc4
Merge pull request #1550 from himanshug/optionally_log_all_requests
...
print *all* HTTP requests to log if configured
2015-07-28 11:19:36 -07:00
86ede702b1
Add namespaced lookups as extensions
...
* Adds kafka, URI, and JDBC namespace defintions
* Add ability to explicitly rename using a "namespace" which is a particular data collection that is loaded on all realtime, historic nodes, and brokers. If any of these nodes has the namespace extension, ALL nodes have the namespace extension.
* Add namespace caching and populating (can be on heap or off heap)
* Add NamespaceExtractionCacheManager for handling caches
* Added ExtractionNamespace for handling metadata on the extraction namespaces
* Added ExtractionNamespaceUpdate for handling metadata related to updates
* Add extension which caches renames from a kafka stream (requires kafka8)
* Added README.md for the namespace kafka extension
* Added docs
* Added namespace/size, namespace/count, namespace/deltaTasksStarted metrics
Add static config for namespaces via `druid.query.extraction.namespace`
* This is a rebase of https://github.com/b-slim/druid/tree/static_config_only
2015-07-28 11:14:14 -07:00
90b475975e
Log all the requests to debug level in logger io.druid.jetty.RequestLog
2015-07-28 12:56:04 -05:00
efb6e0649e
simplify merging of results on broker.
...
add benchmark
formatting and review comments
organize imports
2015-07-27 11:46:11 -07:00
6e8da5de86
Merge pull request #1441 from metamx/queryResourceFlusher
...
Add flush prior to close on output stream in QueryResource
2015-07-17 16:32:52 -07:00
e21195f987
Merge pull request #1469 from guobingkun/table_config
...
Inconsistent property names for "druid.metadata.storage.tables.xxx"
2015-07-17 07:43:19 -07:00
a4cb0332fb
Merge pull request #1524 from metamx/peonShutdown
...
Make CliPeon run shutdown hooks properly
2015-07-15 10:09:33 -07:00
92d5c250ab
Remove comment
2015-07-14 16:20:36 -07:00
5eadd395e2
Move lots of executor service creation to Execs
2015-07-14 15:38:49 -07:00
2b87ad307d
Make CliPeon run shutdown hooks properly
...
* Also make jetty threads daemon
2015-07-14 15:15:26 -07:00
da8e86146e
Merge pull request #1505 from metamx/improve-test
...
Modify test to check for multiple shards for same interval
2015-07-14 08:50:47 -07:00
3f7ba58227
Merge pull request #1504 from metamx/fix-1447
...
fix for #1447
2015-07-14 08:50:08 -07:00
1affdf7bec
Merge pull request #1502 from samjhecht/poll-alerting
...
add alert on errors polling for rules
2015-07-13 18:37:37 -05:00
fc438e5a72
add alert on errors polling for rules
...
reset retrystarttime after alerting
2015-07-13 09:03:51 -07:00
725086cc89
Merge pull request #1506 from gianm/realtime-plumber-nulls
...
Consider null inputRows and parse errors as unparseable during realtime ingestion.
2015-07-13 10:12:12 -05:00
9068bcd062
Consider null inputRows and parse errors as unparseable during realtime ingestion.
...
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
2015-07-11 20:40:03 -07:00
e9b627bf55
Merge pull request #1476 from pjain1/improved_tests
...
all the review comments have been taken care of, so merging.
2015-07-09 16:25:30 -05:00
1766341648
Modify test to check for multiple shards for same interval
...
Modify test to check for multiple shards for same interval.
2015-07-09 20:35:31 +05:30
58cc3954ff
improved coordinator and datasourceresource tests
2015-07-09 09:46:54 -05:00
5fe27fe4ad
fix for #1447
...
fixes #1447
2015-07-09 19:05:48 +05:30
184b12bee8
fix groupBy caching to work with renamed aggregators
...
Issue - while storing results in cache we store the event map which
contains aggregator names mapped to values. Now when someone fire same
query after renaming aggs, the cache key will be same but the event
will contain metric values mapped to older names which leads to wrong
results.
Fix - modify cache to not store raw event but the actual list of values
only.
review comments + fix dimension renaming
review comment
2015-07-09 11:48:26 +05:30
fe36f7f852
emit alert for DB polling error
2015-07-07 19:51:36 +05:30
d846f3ced6
Merge pull request #1498 from rasahner/doc_localFirehose
...
say explicitly that local firehose searches dirs recursively for files
2015-07-05 15:34:36 -07:00
acd20e8c00
say explicitly that local firehose searches directories recursively for files
2015-07-05 14:46:44 -05:00
f2919b98b1
Merge pull request #1471 from metamx/direct-client-metrics
...
add query/node/time metrics to DirectDruidClient
2015-06-30 22:06:44 -07:00
4a0ae7d8d5
Fix inconsistent druid property names for "druid.metadata.storage.tables.xxx" between document and code
2015-06-29 10:12:30 -05:00
8172bdff1c
Merge pull request #1468 from guobingkun/fix_npe_for_storage_action_handler
...
Fix npe thrown from SQLMetadataStorageActionHandler
2015-06-29 09:57:51 -05:00
28fa1642b9
add node time metrics to DirectDruidClient
2015-06-26 17:57:44 -07:00
fbcac10e00
Remove metrics emitting from caching clustered client
2015-06-26 10:49:13 -07:00
e4e18b17b5
fix npe thrown from getEntry() and getStatus() SQLMetadataStorageActionHandler due to a non-exist entryId
2015-06-26 10:40:56 -05:00
0c85c8c60a
Merge pull request #1462 from pjain1/validate_rules
...
tieredReplicants can be null
2015-06-24 13:29:27 -07:00
ff738ace05
converting null tieredeplicants to default and not allowing empty ones
2015-06-24 14:55:12 -05:00
0c13fd8018
Merge pull request #1450 from metamx/cache-key-distribution
...
Fix bad distribution of cache keys across cache nodes
2015-06-22 17:58:15 -07:00
34c8f4c3da
Merge pull request #1437 from metamx/limitUsedSegmentInterval
...
Add start and end bounds to getUsedSegmentsForInterval
2015-06-22 14:11:38 -05:00
8d91c8d399
Add start and end bounds to getUsedSegmentsForInterval
...
* Add unit tests
* Add javadoc for methods in IndexerMetadataStorageCoordinator
2015-06-22 11:46:53 -07:00
f1951b253c
Fix bad distribution of cache keys across nodes
...
With the existing hash function some nodes could end up with 3 times the
number of keys as others. The following changes improve that to roughly
less than 5% differences across nodes.
- switch from fnv-1a to murmur3_128 hash
- increase repetitions for ketama algorithm
- test to analyze distribution
Also updates spymemcached for recent bugfixes
2015-06-19 15:30:35 -07:00
d276d2c90b
Merge pull request #1444 from druid-io/logging-improvement
...
Separate bootstrap threads from loading threads on historical startup
2015-06-17 15:24:07 -07:00
Gian Merlino
Charles Allen
Xavier Léauté
Himanshu
Xavier Léauté
Fangjin Yang
jon-wei
Nishant
lvjq
Parag Jain
Bingkun Guo
samjhecht
sahner