* Upgrading Tika from 1.24.1 to 2.1.0 and bumping xmlbeans version
This major version upgrade requires an explicit dependency on tika-parsers-standard-package to import the parser implementations, and an update to the namespace of RTFParser. Also, LanguageIdentifier has been deprecated and replaced by LanguageDetector.
This change includes a bump in xmlbeans version from 3.0.1 to 3.1.0
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrade Tika libraries from 2.1.0 to 2.2.0
This also requires a update of Apache Commons-IO from 2.7 to 2.11.0
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrade Tika libraries from 2.2.0 to 2.2.1
Also update PDFBox to 2.0.25 as per Tika release notes
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgraded Tika and xmlbeans libraries
Tika libraries have been upgraded from 2.2.1 to 2.3.0. xmlbeans is now a subproject of POI, so POI was upgraded from 4.1.2 to 5.2.2. With POI 5.x the ooxml-schemas library has been moved to ooxml-lite/ooxml-full. Since ooxml-schemas no longer exists, the LICENSE and NOTICE files in the licenses/ directory have been removed. Finally, xmlbeans has been updated from 3.1.0 to 5.0.2
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* (In progress) Added tika-langdetect
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrading tika libraries to 2.4.0
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Switched from tika-langdetect to tika-langdetect-optimaize
To fix the license check, the mapping regex was expanded to tika-.*
This now means the tika-core LICENSE and NOTICE files are no longer needed.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* (Work in progress) Switching AttachmentProcessor to use OptimaizeLangDetector
This is a concrete implementation of LanguageDetector. Using this requires bringing in the optimaize dependency.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Manually added LICENSE and NOTICE files for Optimaize language-detector
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Move Optimaize dependency to runtimeOnly
Also bring in transitive Guava dependency. This requires manual addition of LICENSE and NOTICE files as with other plugins.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fix Optimaize langDetector to load models first before detecting
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fallback logic, and test updates
Following the Tika library upgrade, some fallback logic is necessary:
1. "Author" is deprecated for MSOffice document parsing. It is recommended to use CREATOR from Tika Core Properties instead.
2. EPUB parsing no longer automatically extracts keywords. The convention to fall back to SUBJECT is now manually implemented in AttachmentProcessor
Finally, unit tests have been upgraded to account for non-deterministic language results across library upgrades.
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Drop Guava version from 31.1 to 18.0
This is the version that Optimaize 0.6 depends on, and it allows for a smaller ignoreViolations list
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Fix ingest-attachment integration test to assert correct language
Signed-off-by: Kartik Ganesh <gkart@amazon.com>
* Upgrade to Apache Commons Logging 1.2
Signed-off-by: Yoann Rodière <yoann@hibernate.org>
* Clarify that Apache HTTP/commons-* dependencies are not just for tests
Signed-off-by: Yoann Rodière <yoann@hibernate.org>
The apache commons http client implementations recently released
versions that solve TLS compatibility issues with the new TLS engine
that supports TLSv1.3 with JDK 11. This change updates our code to
use these versions since JDK 11 is a supported JDK and we should
allow the use of TLSv1.3.
* Upgrade bouncycastle
Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11
* Downgrade bouncycastle to avoid invalid manifest
* Add checksum for new jars
* Update tika permissions for jdk 11
* Mute test failing on jdk 11
* Add JDK11 to CI
* Thread#stop(Throwable) was removed
http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html
* Disable failing tests #31456
* Temprorarily disable doc tests
To see if there are other failures on JDK11
* Only blacklist specific doc tests
* Disable only failing tests in ingest attachment plugin
* Mute failing HDFS tests #31498
* Mute failing lang-painless tests #31500
* Fix backwards compatability builds
Fix JAVA version to 10 for ES 6.3
* Add 6.x to bwx -> java10
* Prefix out and err from buildBwcVersion for readability
```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
[bwc] :buildSrc:compileJava
[bwc] WARNING: An illegal reflective access operation has occurred
[bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
[bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
[bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
[bwc] WARNING: All illegal access operations will be denied in a future release
[bwc] :buildSrc:compileGroovy
[bwc] :buildSrc:writeVersionProperties
[bwc] :buildSrc:processResources
[bwc] :buildSrc:classes
[bwc] :buildSrc:jar
```
* Also set RUNTIME_JAVA_HOME for bwcBuild
So that we can make sure it's not too new for the build to understand.
* Align bouncycastle dependency
* fix painles array tets
closes#31500
* Update jar checksums
* Keep 8/10 runtime/compile untill consensus builds on 11
* Only skip failing tests if running on Java 11
* Failures are dependent of compile java version not runtime
* Condition doc test exceptions on compiler java version as well
* Disable hdfs tests based on runtime java
* Set runtime java to minimum supported for bwc
* PR review
* Add comment with ticket for forbidden apis
In some cases (apparently with outlook files), mime4j library is needed.
We removed it in the past which can cause elasticsearch to crash when you are using ingest-attachment (and probably mapper-attachments as well in 2.x series) with a file which requires this library.
Similar problem as the one reported at #22077.
This change makes ES compile with java9 again, build 118.
* There are a handful of changes due to failure to determine types during compile.
* The attachment plugins which use tika needed to have tika upgraded in order to pickup fixes there for java 9.
* azure discovery and s3 repository indirectly depend on jaxb, which is no longer in the default modules. They now add a jaxb dependency externally, and make JarHell allow for this package.
This is a simple port of the mapper attachment plugin to the ingest
functionality, no new features. The only option is to limit
the number of chars to prevent indexing of huge documents.
Fields can be selected in the processor as well.
Close#16303
dependabot[bot]
Kartik Ganesh
Yoann Rodière
Sarat Vemulapalli
Abbas Hussain
Tianli Feng
Rabi Panda
Ioannis Kakavas
Igor Motov
Jason Tedor
Jay Modi
Ioannis Kakavas
Alpar Torok
Jack Conradson
Tal Levy
Guillaume Le Floch
David Pilato
Alexander Reelsen
Ryan Ernst