Commit Graph

215 Commits

Author SHA1 Message Date
David Pilato d08e9c7080 Test: add a standalone tool which process content
This tool is a simple main class which can be used to test what is extracted from a given binary file or from its base64 equivalent.

You can give as first argument the BASE64 content

Available options:

 -u file:/URL/TO/YOUR/DOC (in place of BASE64 content)
 -s set extracted size (default to mapper attachment size)

Examples:

```
StandaloneTest BASE64Text
StandaloneTest BASE64Text -s 1000000
StandaloneTest -u /tmp/mydoc.pdf
StandaloneTest -u /tmp/mydoc.pdf -s 1000000
```

Closes #89.
2014-10-27 22:01:22 +01:00
David Pilato c3bf3b1ce9 Tests: AnalysisService constructor signature change
Due to this [change](https://github.com/elasticsearch/elasticsearch/pull/8018), we need to fix our tests for elasticsearch 1.4.0 and above.

Closes #87.

(cherry picked from commit b3b0d34)
2014-10-15 13:05:41 +02:00
David Pilato 03b47d5a4c update documentation with release 2.4.0 2014-10-08 18:50:20 +02:00
mikemccand 2ff4eb58d6 Upgrade to Lucene 4.10.1 2014-09-28 17:57:06 -04:00
Michael McCandless 67a2548441 Upgrade to Lucene 4.10.1 snapshot 2014-09-24 17:10:08 -04:00
David Pilato eef6b61806 Create branch es-1.4 for elasticsearch 1.4.0 2014-09-12 16:08:59 +02:00
David Pilato ba74fc2b5e Remove netcdf support
Sadly netcdf library is not Apache2 License compatible so we should not package it anymore.

For users who wants to use it, they can add manually [netcdf librairies](http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/) in `plugins/mapper-attachments` dir and they will get the support back.

Closes #84.
2014-09-08 23:51:01 +02:00
David Pilato 888d79075e Update to Lucene 4.10.0
Closes #85.
2014-09-08 23:47:15 +02:00
David Pilato 20ee711436 parseMultiField() method signature change in es 1.4 and master
As seen with https://github.com/elasticsearch/elasticsearch/pull/7474, we need to update mapper attachment plugin with this new signature.

 Closes #83.
2014-09-04 11:23:09 +02:00
David Pilato c0d053d283 Update to elasticsearch 1.4
Related to #77

(cherry picked from commit ad1742a)
2014-09-01 10:26:38 +02:00
David Pilato 34fe111a2b update documentation with release 2.3.2 2014-09-01 09:53:26 +02:00
David Pilato 87b38c54eb Unable to extract text from Word documents
With issue #80 we explicitly removed appache POI dependency provided by Tika and replaced with a more recent one.
Sadly we forgot to add this new dependency to the assembly so the final ZIP file does not contain POI related jars.

Closes #82.

(cherry picked from commit 49793d5)
2014-09-01 09:41:57 +02:00
David Pilato cc1a43b5c3 update documentation with release 2.3.1 2014-08-18 21:52:53 +02:00
David Pilato 08454d72f6 update documentation with release 2.2.1 2014-08-18 21:39:31 +02:00
David Pilato 2b172f8ff6 Update a few dependencies
Related to #80.
2014-08-18 17:49:36 +02:00
David Pilato 5cf20331a8 Update to elasticsearch 1.4.0
Related to #77.

(cherry picked from commit 7e65cfb)
2014-08-18 15:39:19 +02:00
David Pilato 75d03621aa Update a few dependencies
Related to #80.

(cherry picked from commit 89d5460)
2014-08-18 15:37:03 +02:00
David Pilato 587e6d3da2 Docs: make the welcome page more obvious
Closes #79.
2014-08-18 12:38:03 +02:00
David Pilato f8d2975946 Update a few dependencies
Closes #80.

(cherry picked from commit 930c8be)
2014-08-18 12:27:23 +02:00
David Pilato 6edf3447b1 Remove old `content` deprecated field
In #73, we deprecated `content` field in favor of `_content` field.

In plugin version 2.4.0, we can now remove the old field name.

Closes #75.

(cherry picked from commit 7a0f838)
2014-07-26 00:33:50 +02:00
David Pilato e704f68525 Log tika exceptions
Currently tika exceptions are swallowed with no log message.
We'd like to be able to know when/if this occurs and for what reason.

Closes #78.

(cherry picked from commit 36b0117)
2014-07-26 00:27:49 +02:00
David Pilato ad986eb2fc Add support for multi-fields
Now https://github.com/elasticsearch/elasticsearch/pull/6867 is merged in elasticsearch core code (branch 1.x - es 1.4),
we can support multi fields in mapper attachment plugin.

```
DELETE /test
PUT /test
{
  "settings": {
    "number_of_shards": 1
  }
}
PUT /test/person/_mapping
{
  "person": {
    "properties": {
      "file": {
        "type": "attachment",
        "path": "full",
        "fields": {
          "file": {
            "type": "string",
            "fields": {
              "store": {
                "type": "string",
                "store": true
              }
            }
          },
          "content_type": {
            "type": "string",
            "fields": {
              "store": {
                "type": "string",
                "store": true
              },
              "untouched": {
                "type": "string",
                "index": "not_analyzed",
                "store": true
              }
            }
          }
        }
      }
    }
  }
}

PUT /test/person/1?refresh=true
{
  "file": "IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg=="
}

GET /test/person/_search
{
  "fields": [
    "file.store",
    "file.content_type.store"
  ],
  "aggs": {
    "store": {
      "terms": {
        "field": "file.content_type.store"
      }
    },
    "untouched": {
      "terms": {
        "field": "file.content_type.untouched"
      }
    }
  }
}
```

It gives:

```js
{
   "took": 3,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 1,
      "hits": [
         {
            "_index": "test",
            "_type": "person",
            "_id": "1",
            "_score": 1,
            "fields": {
               "file.store": [
                  "\"God Save the Queen\" (alternatively \"God Save the King\"\n"
               ],
               "file.content_type.store": [
                  "text/plain; charset=ISO-8859-1"
               ]
            }
         }
      ]
   },
   "aggregations": {
      "store": {
         "doc_count_error_upper_bound": 0,
         "buckets": [
            {
               "key": "1",
               "doc_count": 1
            },
            {
               "key": "8859",
               "doc_count": 1
            },
            {
               "key": "charset",
               "doc_count": 1
            },
            {
               "key": "iso",
               "doc_count": 1
            },
            {
               "key": "plain",
               "doc_count": 1
            },
            {
               "key": "text",
               "doc_count": 1
            }
         ]
      },
      "untouched": {
         "doc_count_error_upper_bound": 0,
         "buckets": [
            {
               "key": "text/plain; charset=ISO-8859-1",
               "doc_count": 1
            }
         ]
      }
   }
}
```

Note that using shorter definition works as well:

```
DELETE /test
PUT /test
{
  "settings": {
    "number_of_shards": 1
  }
}
PUT /test/person/_mapping
{
  "person": {
    "properties": {
      "file": {
        "type": "attachment"
      }
    }
  }
}
PUT /test/person/1?refresh=true
{
  "file": "IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg=="
}

GET /test/person/_search
{
  "query": {
    "match": {
      "file": "king"
    }
  }
}
```

gives:

```js
{
   "took": 53,
   "timed_out": false,
   "_shards": {
      "total": 1,
      "successful": 1,
      "failed": 0
   },
   "hits": {
      "total": 1,
      "max_score": 0.095891505,
      "hits": [
         {
            "_index": "test",
            "_type": "person",
            "_id": "1",
            "_score": 0.095891505,
            "_source": {
               "file": "IkdvZCBTYXZlIHRoZSBRdWVlbiIgKGFsdGVybmF0aXZlbHkgIkdvZCBTYXZlIHRoZSBLaW5nIg=="
            }
         }
      ]
   }
}
```

Closes #57.

(cherry picked from commit 432d7c0)
2014-07-26 00:27:28 +02:00
David Pilato 663d4eaddb Update to elasticsearch 1.4.0
Closes #77.

(cherry picked from commit c58516f)
2014-07-26 00:26:41 +02:00
David Pilato eaccd4383d Deprecate `content` by `_content`
When we want to force some values, we need to set those using `_field` where `field` is the field name we want to force:

```
{
  "file": {
    "_name": "myfilename.txt"
  }
}
```

But to set the content itself, we use `content` field name.

```
{
  "file": {
    "content": "VGhpcyBpcyBhbiBlbGFzdGljc2VhcmNoIG1hcHBlciBhdHRhY2htZW50IHRlc3Qu",
    "_name": "myfilename.txt"
  }
}
```

For consistency, we set `_content` instead:

```
{
  "file": {
    "_content": "VGhpcyBpcyBhbiBlbGFzdGljc2VhcmNoIG1hcHBlciBhdHRhY2htZW50IHRlc3Qu",
    "_name": "myfilename.txt"
  }
}
```

Closes #73.

(cherry picked from commit 2e6be20)
2014-07-25 18:15:37 +02:00
David Pilato 1d1225b87c Update to Lucene 4.9.0
Update to elasticsearch 1.3.0
Move to java 1.7

Related to #67.
Closed #76.

(cherry picked from commit 2303932)
2014-07-25 18:15:28 +02:00
David Pilato 310df36bfa SL4FJ dependency version problem
This is due to `edu.ucar:netcdf` lib which comes from `tika-parsers` dependency.

```
[INFO] +- org.apache.tika:tika-parsers:jar:1.5:compile
[INFO] |  +- edu.ucar:netcdf:jar:4.2-min:compile
[INFO] |  |  \- org.slf4j:slf4j-api:jar:1.5.6:compile
```

We can exclude this library from the generated ZIP artifact.

Closes #41.
2014-06-14 18:56:14 +02:00
David Pilato 51a8f6f1a0 Fix doc typo
(cherry picked from commit f70eb1d)
2014-06-03 10:13:12 +02:00
David Pilato a3bb103297 Remove deprecated `language` forced field
With #68 we replaced `language`field with `_language`.

We can now remove the old deprecated name.

Closes #69.
(cherry picked from commit e39f144)
2014-06-03 10:11:13 +02:00
David Pilato 94cf141108 Use` _language` field instead of `language`
When we want to force a language instead of using Tika language detection, we set `language` field in documents.

 To be consistent with other forced fields, `_content_type` and `_name`, we should prefix `language` field by an underscore `_`.

 So `language` become `_language`.

 We first deprecate `language` in version 2.1.0 and we remove it in 2.3.0.

 Closes #68.

(cherry picked from commit 2f46343)
2014-06-03 10:10:49 +02:00
David Pilato 7c1c2011bc Update to elasticsearch 1.3.0
Closes #67.
(cherry picked from commit d3eaac9)
2014-06-03 09:49:41 +02:00
David Pilato c0e7795f1f Update to elasticsearch 1.2.0
Closes #66.
(cherry picked from commit fb3b288)
2014-06-03 09:49:13 +02:00
David Pilato 4b35501cf3 Setting "_content_type" in indexing request has no effect
Example below. I set the type as text/plain but it is identified as text/html.

```sh
#!/bin/sh

echo "\n\n Delete testidx \n"
curl -XDELETE "http://localhost:9200/testidx"

echo "\n\n Create index and mapping \n"
curl -XPUT "http://localhost:9200/testidx" -d'
{
  "mappings": {
    "session": {
      "properties": {
        "Content": {
          "properties": {
            "content": {
              "type": "attachment",
              "path": "full",
              "store": "yes",
              "fields": {
                "content": {
                  "type": "string",
                  "store": "yes"
                },
                "author": {
                  "type": "string",
                  "store": "yes"
                },
                "title": {
                  "type": "string",
                  "store": "yes"
                },
                "name": {
                  "type": "string",
                  "store": "yes"
                },
                "date": {
                  "type": "date",
                  "format": "dateOptionalTime",
                  "store": "yes"
                },
                "keywords": {
                  "type": "string",
                  "store": "yes"
                },
                "content_type": {
                  "type": "string",
                  "store": "yes"
                },
                "content_length": {
                  "type": "integer",
                  "store": "yes"
                }
              }
            }
          }
        }
      }
    }
  }
}'

echo "\n\n Index document \n"
curl -XPOST "http://localhost:9200/_bulk" -d'
  {"index":{"_index":"testidx","_type":"session"}}
  {"Content":[{"_content_type":"text/plain","content":"BASE64ENCODED_CONTENT"}]}
'

echo "\n\n Refresh \n"
curl -XPOST "http://localhost:9200/testidx/_refresh"

echo "\n\n Get doc type \n"
curl -XPOST "http://localhost:9200/testidx/_search?pretty" -d'
{
  "fields": ["Content.content.content_type","Content.content.content_length","Content.content"]
}'
```

Closes #65.
(cherry picked from commit 38075dc)
2014-06-03 09:36:10 +02:00
David Pilato 7f8143ff12 Add highlighting documentation
Closes #54.
(cherry picked from commit efdf8ef)
2014-06-03 09:35:05 +02:00
David Pilato 8855bd7ddc Fix typo for JSON fields
(cherry picked from commit 63c60b8)
2014-06-03 09:34:51 +02:00
David Pilato 4d63130a23 Update to elasticsearch 2.0.0 / Lucene 4.8.1 2014-06-03 09:34:31 +02:00
David Pilato e95bb18edb Create branches according to elasticsearch versions
We create branches:

* es-0.90 for elasticsearch 0.90
* es-1.0 for elasticsearch 1.0
* es-1.1 for elasticsearch 1.1
* master for elasticsearch master

We also check that before releasing we don't have a dependency to an elasticsearch SNAPSHOT version.

Add links to each version in documentation
2014-03-28 17:47:38 +01:00
David Pilato 839c4dab16 prepare for next development iteration 2014-03-25 19:02:16 +01:00
David Pilato 74d882110d prepare release elasticsearch-mapper-attachments-2.0.0 2014-03-25 18:47:56 +01:00
Richard Louapre 3d15cb0484 Add language detection option
Based on PR #45, we add a new language detection option using Language detection feature available in Tika:
https://tika.apache.org/1.4/detection.html#Language_Detection

By default, language detection is disabled (`false`) as it could come with a cost.
This default value can be changed by setting the `index.mapping.attachment.detect_language` setting.
It can also be provided on a per document indexed using the `_detect_language` parameter.

Closes #45.
Closes #44.
2014-03-25 18:26:09 +01:00
David Pilato 621995d0b4 Upgrade to Tika 1.5
Closes #56.
2014-03-19 23:20:29 +01:00
David Pilato b8d7f17951 Update to elasticsearch 1.0.0
Closes #60.
2014-03-19 23:14:39 +01:00
David Pilato 1b7daafeac Add plugin version in es-plugin.properties
Closes #59.
2014-03-19 23:09:37 +01:00
David Pilato 9d0b700b05 Add plugin release semi-automatic script
Closes #58.
2014-03-19 23:04:09 +01:00
David Pilato c9d749b52d prepare for next development iteration 2014-01-15 23:46:26 +01:00
David Pilato 7fc31c89f7 prepare release elasticsearch-mapper-attachments-2.0.0.RC1 2014-01-15 23:37:44 +01:00
David Pilato 054d1acf3a Update to elasticsearch 1.0.0.RC1
Related to #48.
2014-01-15 23:37:43 +01:00
David Pilato a85c3becae Move tests to elasticsearch test framework
Related to #49.
2014-01-15 15:38:39 +01:00
David Pilato b877f1bd4f Update to elasticsearch 1.0.0.RC1
Closes #48.
2014-01-14 14:51:32 +01:00
David Pilato 2b4f875731 Move tests to elasticsearch test framework
Closes #49.
2014-01-13 23:18:04 +01:00
David Pilato f8f647dea9 update headers 2014-01-13 22:31:14 +01:00