honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add Circle Processor (#43851) (#46097) 

add circle-processor that translates circles to polygons
This commit is contained in:
Tal Levy 2019-08-28 14:44:08 -07:00 committed by GitHub
parent d94c4dcffb
commit a356bcff41
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
10 changed files with 899 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
docs/reference/images/spatial/error_distance.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 193 KiB

1
docs/reference/ingest/ingest-node.asciidoc
View File

@ -838,6 +838,7 @@ See {plugins}/ingest.html[Ingest plugins] for information about the available in
include::processors/append.asciidoc[]
include::processors/bytes.asciidoc[]
include::processors/circle.asciidoc[]
include::processors/convert.asciidoc[]
include::processors/date.asciidoc[]
include::processors/date-index-name.asciidoc[]

165
docs/reference/ingest/processors/circle.asciidoc Normal file
View File

@ -0,0 +1,165 @@
[role="xpack"]
[testenv="basic"]
[[ingest-circle-processor]]
=== Circle Processor
Converts circle definitions of shapes to regular polygons which approximate them.
[[circle-processor-options]]
.Circle Processor Options
[options="header"]
|======
| Name | Required | Default | Description
| `field` | yes | - | The string-valued field to trim whitespace from
| `target_field` | no | `field` | The field to assign the polygon shape to, by default `field` is updated in-place
| `ignore_missing` | no | `false` | If `true` and `field` does not exist, the processor quietly exits without modifying the document
| `error_distance` | yes | - | The difference between the resulting inscribed distance from center to side and the circle's radius (measured in meters for `geo_shape`, unit-less for `shape`)
| `shape_type` | yes | - | which field mapping type is to be used when processing the circle: `geo_shape` or `shape`
include::common-options.asciidoc[]
|======
image:images/spatial/error_distance.png[]
[source,js]
--------------------------------------------------
PUT circles
{
"mappings": {
"properties": {
"circle": {
"type": "geo_shape"
}
}
}
}
PUT _ingest/pipeline/polygonize_circles
{
"description": "translate circle to polygon",
"processors": [
{
"circle": {
"field": "circle",
"error_distance": 28.0,
"shape_type": "geo_shape"
}
}
]
}
--------------------------------------------------
// CONSOLE
Using the above pipeline, we can attempt to index a document into the `circles` index.
The circle can be represented as either a WKT circle or a GeoJSON circle. The resulting
polygon will be represented and indexed using the same format as the input circle. WKT will
be translated to a WKT polygon, and GeoJSON circles will be translated to GeoJSON polygons.
==== Example: Circle defined in Well Known Text
In this example a circle defined in WKT format is indexed
[source,js]
--------------------------------------------------
PUT circles/_doc/1?pipeline=polygonize_circles
{
"circle": "CIRCLE (30 10 40)"
}
GET circles/_doc/1
--------------------------------------------------
// CONSOLE
// TEST[continued]
The response from the above index request:
[source,js]
--------------------------------------------------
{
"found": true,
"_index": "circles",
"_type": "_doc",
"_id": "1",
"_version": 1,
"_seq_no": 22,
"_primary_term": 1,
"_source": {
"circle": "polygon ((30.000365257263184 10.0, 30.000111397193788 10.00034284530941, 29.999706043744222 10.000213571721195, 29.999706043744222 9.999786428278805, 30.000111397193788 9.99965715469059, 30.000365257263184 10.0))"
}
}
--------------------------------------------------
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/]
==== Example: Circle defined in GeoJSON
In this example a circle defined in GeoJSON format is indexed
[source,js]
--------------------------------------------------
PUT circles/_doc/2?pipeline=polygonize_circles
{
"circle": {
"type": "circle",
"radius": "40m",
"coordinates": [30, 10]
}
}
GET circles/_doc/2
--------------------------------------------------
// CONSOLE
// TEST[continued]
The response from the above index request:
[source,js]
--------------------------------------------------
{
"found": true,
"_index": "circles",
"_type": "_doc",
"_id": "2",
"_version": 1,
"_seq_no": 22,
"_primary_term": 1,
"_source": {
"circle": {
"coordinates": [
[
[30.000365257263184, 10.0],
[30.000111397193788, 10.00034284530941],
[29.999706043744222, 10.000213571721195],
[29.999706043744222, 9.999786428278805],
[30.000111397193788, 9.99965715469059],
[30.000365257263184, 10.0]
]
],
"type": "polygon"
}
}
}
--------------------------------------------------
// TESTRESPONSE[s/"_seq_no": \d+/"_seq_no" : $body._seq_no/ s/"_primary_term": 1/"_primary_term" : $body._primary_term/]
==== Notes on Accuracy
Accuracy of the polygon that represents the circle is defined as `error_distance`. The smaller this
difference is, the closer to a perfect circle the polygon is.
Below is a table that aims to help capture how the radius of the circle affects the resulting number of sides
of the polygon given different inputs.
The minimum number of sides is `4` and the maximum is `1000`.
[[circle-processor-accuracy]]
.Circle Processor Accuracy
[options="header"]
|======
| error_distance | radius in meters | number of sides of polygon
| 1.00 | 1.0 | 4
| 1.00 | 10.0 | 14
| 1.00 | 100.0 | 45
| 1.00 | 1000.0 | 141
| 1.00 | 10000.0 | 445
| 1.00 | 100000.0 | 1000
|======

20
server/src/main/java/org/elasticsearch/ingest/ConfigurationUtils.java
View File

@ -189,6 +189,26 @@ public final class ConfigurationUtils {
}
}
/**
* Returns and removes the specified property from the specified configuration map.
*
* If the property value isn't of type int a {@link ElasticsearchParseException} is thrown.
* If the property is missing an {@link ElasticsearchParseException} is thrown
*/
public static Double readDoubleProperty(String processorType, String processorTag, Map<String, Object> configuration,
String propertyName) {
Object value = configuration.remove(propertyName);
if (value == null) {
throw newConfigurationException(processorType, processorTag, propertyName, "required property is missing");
}
try {
return Double.parseDouble(value.toString());
} catch (Exception e) {
throw newConfigurationException(processorType, processorTag, propertyName,
"property cannot be converted to a double [" + value.toString() + "]");
}
}
/**
* Returns and removes the specified property of type list from the specified configuration map.
*

10
x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/SpatialPlugin.java
View File

@ -8,12 +8,15 @@ package org.elasticsearch.xpack.spatial;
import org.elasticsearch.common.inject.Module;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.mapper.Mapper;
import org.elasticsearch.ingest.Processor;
import org.elasticsearch.plugins.IngestPlugin;
import org.elasticsearch.plugins.MapperPlugin;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.xpack.core.XPackPlugin;
import org.elasticsearch.plugins.SearchPlugin;
import org.elasticsearch.xpack.spatial.index.mapper.ShapeFieldMapper;
import org.elasticsearch.xpack.spatial.index.query.ShapeQueryBuilder;
import org.elasticsearch.xpack.spatial.ingest.CircleProcessor;
import java.util.Collection;
import java.util.Collections;
@ -23,7 +26,7 @@ import java.util.Map;
import static java.util.Collections.singletonList;
public class SpatialPlugin extends Plugin implements MapperPlugin, SearchPlugin {
public class SpatialPlugin extends Plugin implements MapperPlugin, SearchPlugin, IngestPlugin {
public SpatialPlugin(Settings settings) {
}
@ -45,4 +48,9 @@ public class SpatialPlugin extends Plugin implements MapperPlugin, SearchPlugin
public List<QuerySpec<?>> getQueries() {
return singletonList(new QuerySpec<>(ShapeQueryBuilder.NAME, ShapeQueryBuilder::new, ShapeQueryBuilder::fromXContent));
}
@Override
public Map<String, Processor.Factory> getProcessors(Processor.Parameters parameters) {
return Collections.singletonMap(CircleProcessor.TYPE, new CircleProcessor.Factory());
}
}

101
x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/SpatialUtils.java Normal file
View File

@ -0,0 +1,101 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.spatial;
import org.apache.lucene.util.SloppyMath;
import org.elasticsearch.geometry.Circle;
import org.elasticsearch.geometry.LinearRing;
import org.elasticsearch.geometry.Polygon;
import org.elasticsearch.index.mapper.GeoShapeIndexer;
/**
* Utility class for storing different helpful re-usable spatial functions
*/
public class SpatialUtils {
private SpatialUtils() {}
/**
* Makes an n-gon, centered at the provided circle's center, and each vertex approximately
* {@link Circle#getRadiusMeters()} away from the center.
*
* This does not split the polygon across the date-line. Relies on {@link GeoShapeIndexer} to
* split prepare polygon for indexing.
*
* Adapted from from org.apache.lucene.geo.GeoTestUtil
* */
public static Polygon createRegularGeoShapePolygon(Circle circle, int gons) {
double[][] result = new double[2][];
result[0] = new double[gons+1];
result[1] = new double[gons+1];
for(int i=0; i<gons; i++) {
double angle = i * (360.0 / gons);
double x = Math.cos(SloppyMath.toRadians(angle));
double y = Math.sin(SloppyMath.toRadians(angle));
double factor = 2.0;
double step = 1.0;
int last = 0;
// Iterate out along one spoke until we hone in on the point that's nearly exactly radiusMeters from the center:
while (true) {
double lat = circle.getLat() + y * factor;
double lon = circle.getLon() + x * factor;
double distanceMeters = SloppyMath.haversinMeters(circle.getLat(), circle.getLon(), lat, lon);
if (Math.abs(distanceMeters - circle.getRadiusMeters()) < 0.1) {
// Within 10 cm: close enough!
// lon/lat are left de-normalized so that indexing can properly detect dateline crossing.
result[0][i] = lon;
result[1][i] = lat;
break;
}
if (distanceMeters > circle.getRadiusMeters()) {
// too big
factor -= step;
if (last == 1) {
step /= 2.0;
}
last = -1;
} else if (distanceMeters < circle.getRadiusMeters()) {
// too small
factor += step;
if (last == -1) {
step /= 2.0;
}
last = 1;
}
}
}
// close poly
result[0][gons] = result[0][0];
result[1][gons] = result[1][0];
return new Polygon(new LinearRing(result[0], result[1]));
}
/**
* Makes an n-gon, centered at the provided circle's center. This assumes
* distance measured in cartesian geometry.
**/
public static Polygon createRegularShapePolygon(Circle circle, int gons) {
double[][] result = new double[2][];
result[0] = new double[gons+1];
result[1] = new double[gons+1];
for(int i=0; i<gons; i++) {
double angle = i * (360.0 / gons);
double x = circle.getRadiusMeters() * Math.cos(SloppyMath.toRadians(angle));
double y = circle.getRadiusMeters() * Math.sin(SloppyMath.toRadians(angle));
result[0][i] = x + circle.getX();
result[1][i] = y + circle.getY();
}
// close poly
result[0][gons] = result[0][0];
result[1][gons] = result[1][0];
return new Polygon(new LinearRing(result[0], result[1]));
}
}

169
x-pack/plugin/spatial/src/main/java/org/elasticsearch/xpack/spatial/ingest/CircleProcessor.java Normal file
View File

@ -0,0 +1,169 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.spatial.ingest;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.geo.GeometryFormat;
import org.elasticsearch.common.geo.GeometryParser;
import org.elasticsearch.common.xcontent.DeprecationHandler;
import org.elasticsearch.common.xcontent.NamedXContentRegistry;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.common.xcontent.support.MapXContentParser;
import org.elasticsearch.geometry.Circle;
import org.elasticsearch.geometry.Geometry;
import org.elasticsearch.geometry.ShapeType;
import org.elasticsearch.ingest.AbstractProcessor;
import org.elasticsearch.ingest.ConfigurationUtils;
import org.elasticsearch.ingest.IngestDocument;
import org.elasticsearch.ingest.Processor;
import org.elasticsearch.xpack.spatial.SpatialUtils;
import java.util.Arrays;
import java.util.EnumSet;
import java.util.HashMap;
import java.util.Locale;
import java.util.Map;
/**
* The circle-processor converts a circle shape definition into a valid regular polygon approximating the circle.
*/
public final class CircleProcessor extends AbstractProcessor {
public static final String TYPE = "circle";
static final GeometryParser PARSER = new GeometryParser(true, true, true);
static final int MINIMUM_NUMBER_OF_SIDES = 4;
static final int MAXIMUM_NUMBER_OF_SIDES = 1000;
private final String field;
private final String targetField;
private final boolean ignoreMissing;
private final double errorDistance;
private final CircleShapeFieldType circleShapeFieldType;
CircleProcessor(String tag, String field, String targetField, boolean ignoreMissing, double errorDistance,
CircleShapeFieldType circleShapeFieldType) {
super(tag);
this.field = field;
this.targetField = targetField;
this.ignoreMissing = ignoreMissing;
this.errorDistance = errorDistance;
this.circleShapeFieldType = circleShapeFieldType;
}
@Override
@SuppressWarnings("unchecked")
public IngestDocument execute(IngestDocument ingestDocument) {
Object obj = ingestDocument.getFieldValue(field, Object.class, ignoreMissing);
if (obj == null && ignoreMissing) {
return ingestDocument;
} else if (obj == null) {
throw new IllegalArgumentException("field [" + field + "] is null, cannot process it.");
}
final Map<String, Object> valueWrapper = new HashMap<>();
if (obj instanceof Map || obj instanceof String) {
valueWrapper.put("shape", obj);
} else {
throw new IllegalArgumentException("field [" + field + "] must be a WKT Circle or a GeoJSON Circle value");
}
MapXContentParser parser = new MapXContentParser(NamedXContentRegistry.EMPTY,
DeprecationHandler.THROW_UNSUPPORTED_OPERATION, valueWrapper, XContentType.JSON);
try {
parser.nextToken(); // START_OBJECT
parser.nextToken(); // "shape" field key
parser.nextToken(); // shape value
GeometryFormat geometryFormat = PARSER.geometryFormat(parser);
Geometry geometry = geometryFormat.fromXContent(parser);
if (ShapeType.CIRCLE.equals(geometry.type())) {
Circle circle = (Circle) geometry;
int numSides = numSides(circle.getRadiusMeters());
final Geometry polygonizedCircle;
switch (circleShapeFieldType) {
case GEO_SHAPE:
polygonizedCircle = SpatialUtils.createRegularGeoShapePolygon(circle, numSides);
break;
case SHAPE:
polygonizedCircle = SpatialUtils.createRegularShapePolygon(circle, numSides);
break;
default:
throw new IllegalStateException("invalid shape_type [" + circleShapeFieldType + "]");
}
XContentBuilder newValueBuilder = XContentFactory.jsonBuilder().startObject().field("val");
geometryFormat.toXContent(polygonizedCircle, newValueBuilder, ToXContent.EMPTY_PARAMS);
newValueBuilder.endObject();
Map<String, Object> newObj = XContentHelper.convertToMap(
BytesReference.bytes(newValueBuilder), true, XContentType.JSON).v2();
ingestDocument.setFieldValue(targetField, newObj.get("val"));
} else {
throw new IllegalArgumentException("found [" + geometry.type() + "] instead of circle");
}
} catch (Exception e) {
throw new IllegalArgumentException("invalid circle definition", e);
}
return ingestDocument;
}
@Override
public String getType() {
return TYPE;
}
String field() {
return field;
}
String targetField() {
return targetField;
}
double errorDistance() {
return errorDistance;
}
CircleShapeFieldType shapeType() {
return circleShapeFieldType;
}
int numSides(double radiusMeters) {
int val = (int) Math.ceil(2 * Math.PI / Math.acos(1 - errorDistance / radiusMeters));
return Math.min(MAXIMUM_NUMBER_OF_SIDES, Math.max(MINIMUM_NUMBER_OF_SIDES, val));
}
public static final class Factory implements Processor.Factory {
public CircleProcessor create(Map<String, Processor.Factory> registry, String processorTag, Map<String, Object> config) {
String field = ConfigurationUtils.readStringProperty(TYPE, processorTag, config, "field");
String targetField = ConfigurationUtils.readStringProperty(TYPE, processorTag, config, "target_field", field);
boolean ignoreMissing = ConfigurationUtils.readBooleanProperty(TYPE, processorTag, config, "ignore_missing", false);
double radiusDistance = Math.abs(ConfigurationUtils.readDoubleProperty(TYPE, processorTag, config, "error_distance"));
CircleShapeFieldType circleFieldType = CircleShapeFieldType.parse(
ConfigurationUtils.readStringProperty(TYPE, processorTag, config, "shape_type"));
return new CircleProcessor(processorTag, field, targetField, ignoreMissing, radiusDistance, circleFieldType);
}
}
enum CircleShapeFieldType {
SHAPE, GEO_SHAPE;
public static CircleShapeFieldType parse(String value) {
EnumSet<CircleShapeFieldType> validValues = EnumSet.allOf(CircleShapeFieldType.class);
try {
return valueOf(value.toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("illegal [shape_type] value [" + value + "]. valid values are " +
Arrays.toString(validValues.toArray()));
}
}
}
}

63
x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/SpatialUtilsTests.java Normal file
View File

@ -0,0 +1,63 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.spatial;
import org.apache.lucene.util.SloppyMath;
import org.elasticsearch.geometry.Circle;
import org.elasticsearch.geometry.LinearRing;
import org.elasticsearch.geometry.Polygon;
import org.elasticsearch.test.ESTestCase;
import static org.hamcrest.Matchers.closeTo;
import static org.hamcrest.Matchers.equalTo;
public class SpatialUtilsTests extends ESTestCase {
public void testCreateRegularGeoShapePolygon() {
double lon = randomDoubleBetween(-20, 20, true);
double lat = randomDoubleBetween(-20, 20, true);
double radiusMeters = randomDoubleBetween(10, 10000, true);
Circle circle = new Circle(lon, lat, radiusMeters);
int numSides = randomIntBetween(4, 1000);
Polygon polygon = SpatialUtils.createRegularGeoShapePolygon(circle, numSides);
LinearRing outerShell = polygon.getPolygon();
int numPoints = outerShell.length();
// check no holes created
assertThat(polygon.getNumberOfHoles(), equalTo(0));
// check there are numSides edges
assertThat(numPoints, equalTo(numSides + 1));
// check that all the points are about a radius away from the center
for (int i = 0; i < numPoints ; i++) {
double actualDistance = SloppyMath
.haversinMeters(circle.getY(), circle.getX(), outerShell.getY(i), outerShell.getX(i));
assertThat(actualDistance, closeTo(radiusMeters, 0.1));
}
}
public void testCreateRegularShapePolygon() {
double x = randomDoubleBetween(-20, 20, true);
double y = randomDoubleBetween(-20, 20, true);
double radius = randomDoubleBetween(10, 10000, true);
Circle circle = new Circle(x, y, radius);
int numSides = randomIntBetween(4, 1000);
Polygon polygon = SpatialUtils.createRegularShapePolygon(circle, numSides);
LinearRing outerShell = polygon.getPolygon();
int numPoints = outerShell.length();
// check no holes created
assertThat(polygon.getNumberOfHoles(), equalTo(0));
// check there are numSides edges
assertThat(numPoints, equalTo(numSides + 1));
// check that all the points are about a radius away from the center
for (int i = 0; i < numPoints ; i++) {
double deltaX = circle.getX() - outerShell.getX(i);
double deltaY = circle.getY() - outerShell.getY(i);
double distance = Math.sqrt(deltaX * deltaX + deltaY * deltaY);
assertThat(distance, closeTo(radius, 0.0001));
}
}
}

94
x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/ingest/CircleProcessorFactoryTests.java Normal file
View File

@ -0,0 +1,94 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.spatial.ingest;
import org.elasticsearch.ElasticsearchParseException;
import org.elasticsearch.test.ESTestCase;
import org.junit.Before;
import java.util.HashMap;
import java.util.Map;
import static org.hamcrest.Matchers.equalTo;
public class CircleProcessorFactoryTests extends ESTestCase {
private CircleProcessor.Factory factory;
@Before
public void init() {
factory = new CircleProcessor.Factory();
}
public void testCreateGeoShape() {
Map<String, Object> config = new HashMap<>();
config.put("field", "field1");
config.put("error_distance", 0.002);
config.put("shape_type", "geo_shape");
String processorTag = randomAlphaOfLength(10);
CircleProcessor processor = factory.create(null, processorTag, config);
assertThat(processor.getTag(), equalTo(processorTag));
assertThat(processor.field(), equalTo("field1"));
assertThat(processor.targetField(), equalTo("field1"));
assertThat(processor.errorDistance(), equalTo(0.002));
assertThat(processor.shapeType(), equalTo(CircleProcessor.CircleShapeFieldType.GEO_SHAPE));
}
public void testCreateShape() {
Map<String, Object> config = new HashMap<>();
config.put("field", "field1");
config.put("error_distance", 0.002);
config.put("shape_type", "shape");
String processorTag = randomAlphaOfLength(10);
CircleProcessor processor = factory.create(null, processorTag, config);
assertThat(processor.getTag(), equalTo(processorTag));
assertThat(processor.field(), equalTo("field1"));
assertThat(processor.targetField(), equalTo("field1"));
assertThat(processor.errorDistance(), equalTo(0.002));
assertThat(processor.shapeType(), equalTo(CircleProcessor.CircleShapeFieldType.SHAPE));
}
public void testCreateInvalidShapeType() {
Map<String, Object> config = new HashMap<>();
config.put("field", "field1");
config.put("error_distance", 0.002);
config.put("shape_type", "invalid");
String processorTag = randomAlphaOfLength(10);
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> factory.create(null, processorTag, config));
assertThat(e.getMessage(), equalTo("illegal [shape_type] value [invalid]. valid values are [SHAPE, GEO_SHAPE]"));
}
public void testCreateMissingField() {
Map<String, Object> config = new HashMap<>();
String processorTag = randomAlphaOfLength(10);
ElasticsearchParseException e = expectThrows(ElasticsearchParseException.class, () -> factory.create(null, processorTag, config));
assertThat(e.getMessage(), equalTo("[field] required property is missing"));
}
public void testCreateWithTargetField() {
Map<String, Object> config = new HashMap<>();
config.put("field", "field1");
config.put("target_field", "other");
config.put("error_distance", 0.002);
config.put("shape_type", "geo_shape");
String processorTag = randomAlphaOfLength(10);
CircleProcessor processor = factory.create(null, processorTag, config);
assertThat(processor.getTag(), equalTo(processorTag));
assertThat(processor.field(), equalTo("field1"));
assertThat(processor.targetField(), equalTo("other"));
assertThat(processor.errorDistance(), equalTo(0.002));
assertThat(processor.shapeType(), equalTo(CircleProcessor.CircleShapeFieldType.GEO_SHAPE));
}
public void testCreateWithNoErrorDistanceDefined() {
Map<String, Object> config = new HashMap<>();
config.put("field", "field1");
String processorTag = randomAlphaOfLength(10);
ElasticsearchParseException e = expectThrows(ElasticsearchParseException.class, () -> factory.create(null, processorTag, config));
assertThat(e.getMessage(), equalTo("[error_distance] required property is missing"));
}
}

277
x-pack/plugin/spatial/src/test/java/org/elasticsearch/xpack/spatial/ingest/CircleProcessorTests.java Normal file
View File

@ -0,0 +1,277 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.xpack.spatial.ingest;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexableField;
import org.apache.lucene.index.RandomIndexWriter;
import org.apache.lucene.search.IndexSearcher;
import org.apache.lucene.search.Query;
import org.apache.lucene.store.Directory;
import org.elasticsearch.common.bytes.BytesReference;
import org.elasticsearch.common.collect.Tuple;
import org.elasticsearch.common.geo.GeoJson;
import org.elasticsearch.common.geo.ShapeRelation;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.geometry.Circle;
import org.elasticsearch.geometry.Geometry;
import org.elasticsearch.geometry.Point;
import org.elasticsearch.geometry.Polygon;
import org.elasticsearch.geometry.utils.StandardValidator;
import org.elasticsearch.geometry.utils.WellKnownText;
import org.elasticsearch.index.mapper.GeoShapeFieldMapper;
import org.elasticsearch.index.mapper.GeoShapeIndexer;
import org.elasticsearch.index.mapper.MappedFieldType;
import org.elasticsearch.index.query.QueryShardContext;
import org.elasticsearch.index.query.VectorGeoShapeQueryProcessor;
import org.elasticsearch.ingest.IngestDocument;
import org.elasticsearch.ingest.RandomDocumentPicks;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.xpack.spatial.SpatialUtils;
import org.elasticsearch.xpack.spatial.index.mapper.ShapeFieldMapper;
import org.elasticsearch.xpack.spatial.index.mapper.ShapeIndexer;
import org.elasticsearch.xpack.spatial.index.query.ShapeQueryProcessor;
import java.io.IOException;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import static org.elasticsearch.ingest.IngestDocumentMatcher.assertIngestDocument;
import static org.elasticsearch.xpack.spatial.ingest.CircleProcessor.CircleShapeFieldType;
import static org.elasticsearch.xpack.spatial.ingest.CircleProcessor.CircleShapeFieldType.GEO_SHAPE;
import static org.elasticsearch.xpack.spatial.ingest.CircleProcessor.CircleShapeFieldType.SHAPE;
import static org.hamcrest.Matchers.containsString;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.instanceOf;
import static org.mockito.Matchers.any;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.when;
public class CircleProcessorTests extends ESTestCase {
private static final WellKnownText WKT = new WellKnownText(true, new StandardValidator(true));
public void testNumSides() {
double radiusDistanceMeters = randomDoubleBetween(0.01, 6371000, true);
CircleShapeFieldType shapeType = randomFrom(SHAPE, GEO_SHAPE);
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, radiusDistanceMeters, shapeType);
// radius is same as error distance
assertThat(processor.numSides(radiusDistanceMeters), equalTo(4));
// radius is much smaller than error distance
assertThat(processor.numSides(0), equalTo(4));
// radius is much larger than error distance
assertThat(processor.numSides(Math.pow(radiusDistanceMeters, 100)), equalTo(1000));
// radius is 5 times longer than error distance
assertThat(processor.numSides(5*radiusDistanceMeters), equalTo(10));
}
public void testFieldNotFound() throws Exception {
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
IngestDocument ingestDocument = RandomDocumentPicks.randomIngestDocument(random(), new HashMap<>());
Exception e = expectThrows(Exception.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), containsString("not present as part of path [field]"));
}
public void testFieldNotFoundWithIgnoreMissing() throws Exception {
CircleProcessor processor = new CircleProcessor("tag", "field", "field", true, 10, GEO_SHAPE);
IngestDocument originalIngestDocument = RandomDocumentPicks.randomIngestDocument(random(), new HashMap<>());
IngestDocument ingestDocument = new IngestDocument(originalIngestDocument);
processor.execute(ingestDocument);
assertIngestDocument(originalIngestDocument, ingestDocument);
}
public void testNullValue() throws Exception {
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
IngestDocument ingestDocument = RandomDocumentPicks.randomIngestDocument(random(), Collections.singletonMap("field", null));
Exception e = expectThrows(Exception.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("field [field] is null, cannot process it."));
}
public void testNullValueWithIgnoreMissing() throws Exception {
CircleProcessor processor = new CircleProcessor("tag", "field", "field", true, 10, GEO_SHAPE);
IngestDocument originalIngestDocument = RandomDocumentPicks.randomIngestDocument(random(), Collections.singletonMap("field", null));
IngestDocument ingestDocument = new IngestDocument(originalIngestDocument);
processor.execute(ingestDocument);
assertIngestDocument(originalIngestDocument, ingestDocument);
}
@SuppressWarnings("unchecked")
public void testJson() throws IOException {
Circle circle = new Circle(101.0, 1.0, 10);
HashMap<String, Object> map = new HashMap<>();
HashMap<String, Object> circleMap = new HashMap<>();
circleMap.put("type", "Circle");
circleMap.put("coordinates", Arrays.asList(circle.getLon(), circle.getLat()));
circleMap.put("radius", circle.getRadiusMeters() + "m");
map.put("field", circleMap);
Geometry expectedPoly = SpatialUtils.createRegularGeoShapePolygon(circle, 4);
assertThat(expectedPoly, instanceOf(Polygon.class));
IngestDocument ingestDocument = new IngestDocument(map, Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
processor.execute(ingestDocument);
Map<String, Object> polyMap = ingestDocument.getFieldValue("field", Map.class);
XContentBuilder builder = XContentFactory.jsonBuilder();
GeoJson.toXContent(expectedPoly, builder, ToXContent.EMPTY_PARAMS);
Tuple<XContentType, Map<String, Object>> expected = XContentHelper.convertToMap(BytesReference.bytes(builder),
true, XContentType.JSON);
assertThat(polyMap, equalTo(expected.v2()));
}
public void testWKT() {
Circle circle = new Circle(101.0, 0.0, 2);
HashMap<String, Object> map = new HashMap<>();
map.put("field", WKT.toWKT(circle));
Geometry expectedPoly = SpatialUtils.createRegularGeoShapePolygon(circle, 4);
IngestDocument ingestDocument = new IngestDocument(map, Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field",false, 2, GEO_SHAPE);
processor.execute(ingestDocument);
String polyString = ingestDocument.getFieldValue("field", String.class);
assertThat(polyString, equalTo(WKT.toWKT(expectedPoly)));
}
public void testInvalidWKT() {
HashMap<String, Object> map = new HashMap<>();
map.put("field", "invalid");
IngestDocument ingestDocument = new IngestDocument(map, Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("invalid circle definition"));
map.put("field", "POINT (30 10)");
e = expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("invalid circle definition"));
}
public void testMissingField() {
IngestDocument ingestDocument = new IngestDocument(new HashMap<>(), Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("field [field] not present as part of path [field]"));
}
public void testInvalidType() {
Map<String, Object> field = new HashMap<>();
field.put("coordinates", Arrays.asList(100, 100));
field.put("radius", "10m");
Map<String, Object> map = new HashMap<>();
map.put("field", field);
IngestDocument ingestDocument = new IngestDocument(map, Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
for (Object value : new Object[] { null, 4.0, "not_circle"}) {
field.put("type", value);
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("invalid circle definition"));
}
}
public void testInvalidCoordinates() {
Map<String, Object> field = new HashMap<>();
field.put("type", "circle");
field.put("radius", "10m");
Map<String, Object> map = new HashMap<>();
map.put("field", field);
IngestDocument ingestDocument = new IngestDocument(map, Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
for (Object value : new Object[] { null, "not_circle"}) {
field.put("coordinates", value);
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("invalid circle definition"));
}
}
public void testInvalidRadius() {
Map<String, Object> field = new HashMap<>();
field.put("type", "circle");
field.put("coordinates", Arrays.asList(100.0, 1.0));
Map<String, Object> map = new HashMap<>();
map.put("field", field);
IngestDocument ingestDocument = new IngestDocument(map, Collections.emptyMap());
CircleProcessor processor = new CircleProcessor("tag", "field", "field", false, 10, GEO_SHAPE);
for (Object value : new Object[] { null, "NotNumber", "10.0fs"}) {
field.put("radius", value);
IllegalArgumentException e = expectThrows(IllegalArgumentException.class, () -> processor.execute(ingestDocument));
assertThat(e.getMessage(), equalTo("invalid circle definition"));
}
}
public void testGeoShapeQueryAcrossDateline() throws IOException {
String fieldName = "circle";
Circle circle = new Circle(179.999746, 67.1726, randomDoubleBetween(1000, 300000, true));
int numSides = randomIntBetween(4, 1000);
Geometry geometry = SpatialUtils.createRegularGeoShapePolygon(circle, numSides);
MappedFieldType shapeType = new GeoShapeFieldMapper.GeoShapeFieldType();
shapeType.setHasDocValues(false);
shapeType.setName(fieldName);
VectorGeoShapeQueryProcessor processor = new VectorGeoShapeQueryProcessor();
QueryShardContext mockedContext = mock(QueryShardContext.class);
when(mockedContext.fieldMapper(any())).thenReturn(shapeType);
Query sameShapeQuery = processor.process(geometry, fieldName, ShapeRelation.INTERSECTS, mockedContext);
Query pointOnDatelineQuery = processor.process(new Point(180, circle.getLat()), fieldName,
ShapeRelation.INTERSECTS, mockedContext);
try (Directory dir = newDirectory(); RandomIndexWriter w = new RandomIndexWriter(random(), dir)) {
Document doc = new Document();
GeoShapeIndexer indexer = new GeoShapeIndexer(true, fieldName);
Geometry normalized = indexer.prepareForIndexing(geometry);
for (IndexableField field : indexer.indexShape(null, normalized)) {
doc.add(field);
}
w.addDocument(doc);
try (IndexReader reader = w.getReader()) {
IndexSearcher searcher = new IndexSearcher(reader);
assertThat(searcher.search(sameShapeQuery, 1).totalHits.value, equalTo(1L));
assertThat(searcher.search(pointOnDatelineQuery, 1).totalHits.value, equalTo(1L));
}
}
}
public void testShapeQuery() throws IOException {
String fieldName = "circle";
Circle circle = new Circle(0, 0, 10);
int numSides = randomIntBetween(4, 1000);
Geometry geometry = SpatialUtils.createRegularShapePolygon(circle, numSides);
MappedFieldType shapeType = new ShapeFieldMapper.ShapeFieldType();
shapeType.setHasDocValues(false);
shapeType.setName(fieldName);
ShapeQueryProcessor processor = new ShapeQueryProcessor();
QueryShardContext mockedContext = mock(QueryShardContext.class);
when(mockedContext.fieldMapper(any())).thenReturn(shapeType);
Query sameShapeQuery = processor.process(geometry, fieldName, ShapeRelation.INTERSECTS, mockedContext);
Query centerPointQuery = processor.process(new Point(circle.getLon(), circle.getLat()), fieldName,
ShapeRelation.INTERSECTS, mockedContext);
try (Directory dir = newDirectory(); RandomIndexWriter w = new RandomIndexWriter(random(), dir)) {
Document doc = new Document();
ShapeIndexer indexer = new ShapeIndexer(fieldName);
Geometry normalized = indexer.prepareForIndexing(geometry);
for (IndexableField field : indexer.indexShape(null, normalized)) {
doc.add(field);
}
w.addDocument(doc);
try (IndexReader reader = w.getReader()) {
IndexSearcher searcher = new IndexSearcher(reader);
assertThat(searcher.search(sameShapeQuery, 1).totalHits.value, equalTo(1L));
assertThat(searcher.search(centerPointQuery, 1).totalHits.value, equalTo(1L));
}
}
}
}