Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Cleanup the base lookup cluster wide config docs (#3061) 

* Cleanup the base lookup cluster wide config docs

* Add better examples in lookups-cached-global.md

* Use actual valid stock lookups

* Fixed maps with :

* Add mix of lookups

* Better examples in extension

* Remove unneeded namespace requirement

* Add extra line space

* Add link to lookup tiers

* Renamed header
This commit is contained in:
Charles Allen 2016-06-07 10:42:41 -07:00 committed by Fangjin Yang
parent e6b32c24ae
commit fa41a6466a
2 changed files with 108 additions and 64 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

56
docs/content/development/extensions-core/lookups-cached-global.md
View File

@ -28,7 +28,6 @@ Cached namespace lookups can be specified as part of the [cluster wide config fo
"type": "cachedNamespace", "type": "cachedNamespace",
"extractionNamespace": { "extractionNamespace": {
"type": "uri", "type": "uri",
"namespace": "some_uri_lookup",
"uri": "file:/tmp/prefix/", "uri": "file:/tmp/prefix/",
"namespaceParseSpec": { "namespaceParseSpec": {
"format": "csv", "format": "csv",
@ -48,7 +47,6 @@ Cached namespace lookups can be specified as part of the [cluster wide config fo
"type": "cachedNamespace", "type": "cachedNamespace",
"extractionNamespace": { "extractionNamespace": {
"type": "jdbc", "type": "jdbc",
"namespace": "some_jdbc_lookup",
"connectorConfig": { "connectorConfig": {
"createTables": true, "createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid", "connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
@ -78,6 +76,58 @@ If `firstCacheTimeout` is set to a non-zero value, it should be less than `druid
Proper functionality of Namespaced lookups requires the following extension to be loaded on the broker, peon, and historical nodes: Proper functionality of Namespaced lookups requires the following extension to be loaded on the broker, peon, and historical nodes:
`druid-lookups-cached-global` `druid-lookups-cached-global`
## Example configuration
In a simple case where only one [tier](../../querying/lookups.html#dynamic-configuration) exists (`realtime_customer2`) with one `cachedNamespace` lookup called `country_code`, the resulting configuration json looks similar to the following:
```json
{
"realtime_customer2": {
"country_code": {
"type": "cachedNamespace",
"extractionNamespace": {
"type": "jdbc",
"connectorConfig": {
"createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
"user": "druid",
"password": "diurd"
},
"table": "lookupTable",
"keyColumn": "country_id",
"valueColumn": "country_name",
"tsColumn": "timeColumn"
},
"firstCacheTimeout": 120000,
"injective":true
}
}
}
```
Where the coordinator endpoint `/druid/coordinator/v1/lookups/realtime_customer2/country_code` should return
```json
{
"type": "cachedNamespace",
"extractionNamespace": {
"type": "jdbc",
"connectorConfig": {
"createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
"user": "druid",
"password": "diurd"
},
"table": "lookupTable",
"keyColumn": "country_id",
"valueColumn": "country_name",
"tsColumn": "timeColumn"
},
"firstCacheTimeout": 120000,
"injective":true
}
```
## Cache Settings ## Cache Settings
Lookups are cached locally on historical nodes. The following are settings used by the nodes which service queries when Lookups are cached locally on historical nodes. The following are settings used by the nodes which service queries when
@ -126,9 +176,9 @@ The remapping values for each namespaced lookup can be specified by a json objec
"pollPeriod":"PT5M" "pollPeriod":"PT5M"
} }
``` ```
|Property|Description|Required|Default| |Property|Description|Required|Default|
|--------|-----------|--------|-------| |--------|-----------|--------|-------|
|`namespace`|The namespace to define|Yes||
|`pollPeriod`|Period between polling for updates|No|0 (only once)| |`pollPeriod`|Period between polling for updates|No|0 (only once)|
|`uri`|URI for the file of interest|No|Use `uriPrefix`| |`uri`|URI for the file of interest|No|Use `uriPrefix`|
|`uriPrefix`|A URI which specifies a directory (or other searchable resource) in which to search for files|No|Use `uri`| |`uriPrefix`|A URI which specifies a directory (or other searchable resource) in which to search for files|No|Use `uri`|

116
docs/content/querying/lookups.md
View File

@ -79,6 +79,7 @@ Lookups can be updated in bulk by posting a JSON object to `/druid/coordinator/v
{ {
"tierName": { "tierName": {
"lookupExtractorFactoryName": { "lookupExtractorFactoryName": {
"type": "someExtractorFactoryType",
"someExtractorField": "someExtractorValue" "someExtractorField": "someExtractorValue"
} }
} }
@ -91,55 +92,54 @@ So a config might look something like:
{ {
"__default": { "__default": {
"country_code": { "country_code": {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
}, },
"site_id": { "site_id": {
"type": "confidential_jdbc", "type": "cachedNamespace",
"auth": "/etc/jdbc.internal", "extractionNamespace": {
"table": "sites", "type": "jdbc",
"key": "site_id", "connectorConfig": {
"value": "site_name" "createTables": true,
"connectURI": "jdbc:mysql:\/\/localhost:3306\/druid",
"user": "druid",
"password": "diurd"
},
"table": "lookupTable",
"keyColumn": "country_id",
"valueColumn": "country_name",
"tsColumn": "timeColumn"
},
"firstCacheTimeout": 120000,
"injective":true
}, },
"site_id_customer1": { "site_id_customer1": {
"type": "confidential_jdbc", "type": "map",
"auth": "/etc/jdbc.customer1", "map": {"847632": "Internal Use Only"}
"table": "sites",
"key": "site_id",
"value": "site_name"
}, },
"site_id_customer2": { "site_id_customer2": {
"type": "confidential_jdbc", "type": "map",
"auth": "/etc/jdbc.customer2", "map": {"AHF77": "Home"}
"table": "sites",
"key": "site_id",
"value": "site_name"
} }
}, },
"realtime_customer1": { "realtime_customer1": {
"country_code": { "country_code": {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
}, },
"site_id_customer1": { "site_id_customer1": {
"type": "confidential_jdbc", "type": "map",
"auth": "/etc/jdbc.customer1", "map": {"847632": "Internal Use Only"}
"table": "sites",
"key": "site_id",
"value": "site_name"
} }
}, },
"realtime_customer2": { "realtime_customer2": {
"country_code": { "country_code": {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
}, },
"site_id_customer2": { "site_id_customer2": {
"type": "confidential_jdbc", "type": "map",
"auth": "/etc/jdbc.customer2", "map": {"AHF77": "Home"}
"table": "sites",
"key": "site_id",
"value": "site_name"
} }
} }
} }
@ -154,11 +154,8 @@ For example, a post to `/druid/coordinator/v1/lookups/realtime_customer1/site_id
```json ```json
{ {
"type": "confidential_jdbc", "type": "map",
"auth": "/etc/jdbc.customer1", "map": {"847632": "Internal Use Only"}
"table": "sites_updated",
"key": "site_id",
"value": "site_name"
} }
``` ```
@ -171,11 +168,8 @@ Using the prior example, a `GET` to `/druid/coordinator/v1/lookups/realtime_cust
```json ```json
{ {
"type": "confidential_jdbc", "type": "map",
"auth": "/etc/jdbc.customer2", "map": {"AHF77": "Home"}
"table": "sites",
"key": "site_id",
"value": "site_name"
} }
``` ```
@ -205,10 +199,10 @@ The return value will be a json map of the lookups to their extractor factories.
```json ```json
{ {
"some_lookup_name": { "some_lookup_name": {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
} }
} }
``` ```
@ -220,8 +214,8 @@ The return value will be the json representation of the factory.
```json ```json
{ {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483", "United States"}
} }
``` ```
@ -232,8 +226,8 @@ The return value will be a JSON map in the following format:
```json ```json
{ {
"status": "accepted", "status": "accepted",
"failedUpdates": {} "failedUpdates": {}
} }
``` ```
@ -242,13 +236,13 @@ If a lookup cannot be started, or is left in an undefined state, the lookup in e
```json ```json
{ {
"status": "accepted", "status": "accepted",
"failedUpdates": { "failedUpdates": {
"country_code": { "country_code": {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
}
} }
}
} }
``` ```
@ -263,8 +257,8 @@ If `some_lookup_name` is desired to have the LookupExtractorFactory definition o
```json ```json
{ {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
} }
``` ```
@ -273,10 +267,10 @@ Then a post to `/druid/listen/v1/lookups/some_lookup_name` will behave the same
```json ```json
{ {
"some_lookup_name": { "some_lookup_name": {
"type": "simple_json", "type": "map",
"uri": "http://some.host.com/codes.json" "map": {"77483": "United States"}
} }
} }
``` ```