Merge branch 'master' into ccr

* master:
  HLRC: Move commercial clients from XPackClient (#32596)
  Add cluster UUID to Cluster Stats API response (#32206)
  Security: move User to protocol project (#32367)
  [TEST] Test for shard failures, add debug to testProfileMatchesRegular
  Minor fix for javadoc (applicable for java 11). (#32573)
  Painless: Move Some Lookup Logic to PainlessLookup (#32565)
  TEST: Avoid merges in testSeqNoAndCheckpoints
  [Rollup] Remove builders from HistoGroupConfig (#32533)
  Mutes failing SQL string function tests due to #32589
  fixed elements in array of produced terms (#32519)
  INGEST: Enable default pipelines (#32286)
  Remove cluster state initial customs (#32501)
  Mutes LicensingDocumentationIT due to #32580
  [ML] Remove multiple_bucket_spans (#32496)
  [ML] Rename JobProvider to JobResultsProvider (#32551)
  Correct minor typo in explain.asciidoc for HLRC
  Build: Add elastic maven to repos used by BuildPlugin (#32549)
  Clarify the error message when a pipeline agg is used in the 'order' parameter. (#32522)
  Revert "[test] turn on host io cache for opensuse (#32053)"
  Enable packaging tests on suse boxes
  [ML] Improve error when no available field exists for rule scope (#32550)
  [ML] Improve error for functions with limited rule condition support (#32548)
  Painless: Clean Up PainlessField (#32525)
  Add @AwaitsFix for #32554
  Remove broken @link in Javadoc
  Scripting: Conditionally use java time api in scripting (#31441)
  [ML] Fix thread leak when waiting for job flush (#32196) (#32541)
  Add AwaitsFix to failing test - see #32546
  Core: Minor size reduction for AbstractComponent (#32509)
  SQL: Added support for string manipulating functions with more than one parameter (#32356)
  [DOCS] Reloadable Secure Settings (#31713)
  Watcher: Reenable HttpSecretsIntegrationTests#testWebhookAction test (#32456)
  [Rollup] Remove builders from TermsGroupConfig (#32507)
  Use hostname instead of IP with SPNEGO test (#32514)
  Switch x-pack rolling restart to new style Requests (#32339)
  NETWORKING: Fix Netty Leaks by upgrading to 4.1.28 (#32511)
  [DOCS] Small fixes in rule configuration page (#32516)
  Painless: Clean up PainlessMethod (#32476)
  Build: Remove shadowing from benchmarks (#32475)
  Docs: Add all JDKs to CONTRIBUTING.md
  Add licensing enforcement for FIPS mode (#32437)
  SQL: Add test for handling of partial results (#32474)
  Mute testFilterCacheStats
  [ML][DOCS] Fix typo applied_to => applies_to
  Scripting: Fix painless compiler loader to know about context classes (#32385)

CONTRIBUTING.md

@@ -100,6 +100,12 @@ JDK 10 and testing on a JDK 8 runtime; to do this, set `RUNTIME_JAVA_HOME`
 pointing to the Java home of a JDK 8 installation. Note that this mechanism can
 be used to test against other JDKs as well, this is not only limited to JDK 8.
 
+> Note: It is also required to have `JAVA7_HOME`, `JAVA8_HOME` and
+`JAVA10_HOME` available so that the tests can pass.
+
+> Warning: do not use `sdkman` for Java installations which do not have proper
+`jrunscript` for jdk distributions.
+
 Elasticsearch uses the Gradle wrapper for its build. You can execute Gradle
 using the wrapper via the `gradlew` script in the root of the repository.
 

Vagrantfile

@@ -115,11 +115,6 @@ Vagrant.configure(2) do |config|
   'opensuse-42'.tap do |box|
     config.vm.define box, define_opts do |config|
       config.vm.box = 'elastic/opensuse-42-x86_64'
-
-      # https://github.com/elastic/elasticsearch/issues/30295
-      config.vm.provider 'virtualbox' do |vbox|
-        vbox.customize ['storagectl', :id, '--name', 'SATA Controller', '--hostiocache', 'on']
-      end
       suse_common config, box
     end
   end

benchmarks/README.md

@@ -4,36 +4,39 @@ This directory contains the microbenchmark suite of Elasticsearch. It relies on
 
 ## Purpose
 
-We do not want to microbenchmark everything but the kitchen sink and should typically rely on our 
+We do not want to microbenchmark everything but the kitchen sink and should typically rely on our
-[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with 
+[macrobenchmarks](https://elasticsearch-benchmarks.elastic.co/app/kibana#/dashboard/Nightly-Benchmark-Overview) with
-[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components. 
+[Rally](http://github.com/elastic/rally). Microbenchmarks are intended to spot performance regressions in performance-critical components.
 The microbenchmark suite is also handy for ad-hoc microbenchmarks but please remove them again before merging your PR.
 
 ## Getting Started
 
-Just run `gradle :benchmarks:jmh` from the project root directory. It will build all microbenchmarks, execute them and print the result.
+Just run `gradlew -p benchmarks run` from the project root
+directory. It will build all microbenchmarks, execute them and print
+the result.
 
 ## Running Microbenchmarks
 
-Benchmarks are always run via Gradle with `gradle :benchmarks:jmh`.
+Running via an IDE is not supported as the results are meaningless
- 
+because we have no control over the JVM running the benchmarks.
-Running via an IDE is not supported as the results are meaningless (we have no control over the JVM running the benchmarks).
 
-If you want to run a specific benchmark class, e.g. `org.elasticsearch.benchmark.MySampleBenchmark` or have special requirements 
+If you want to run a specific benchmark class like, say,
-generate the uberjar with `gradle :benchmarks:jmhJar` and run it directly with:
+`MemoryStatsBenchmark`, you can use `--args`:
 
 ```
-java -jar benchmarks/build/distributions/elasticsearch-benchmarks-*.jar
+gradlew -p benchmarks run --args ' MemoryStatsBenchmark'
 ```
 
-JMH supports lots of command line parameters. Add `-h` to the command above to see the available command line options.
+Everything in the `'` gets sent on the command line to JMH. The leading ` `
+inside the `'`s is important. Without it parameters are sometimes sent to
+gradle.
 
 ## Adding Microbenchmarks
 
-Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the 
+Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the
 [JMH samples](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/).
 
-In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and 
+In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and
 end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`.
 
 ## Tips and Best Practices
@@ -42,15 +45,15 @@ To get realistic results, you should exercise care when running benchmarks. Here
 
 ### Do
 
-* Ensure that the system executing your microbenchmarks has as little load as possible. Shutdown every process that can cause unnecessary 
+* Ensure that the system executing your microbenchmarks has as little load as possible. Shutdown every process that can cause unnecessary
   runtime jitter. Watch the `Error` column in the benchmark results to see the run-to-run variance.
 * Ensure to run enough warmup iterations to get the benchmark into a stable state. If you are unsure, don't change the defaults.
 * Avoid CPU migrations by pinning your benchmarks to specific CPU cores. On Linux you can use `taskset`.
-* Fix the CPU frequency to avoid Turbo Boost from kicking in and skewing your results. On Linux you can use `cpufreq-set` and the 
+* Fix the CPU frequency to avoid Turbo Boost from kicking in and skewing your results. On Linux you can use `cpufreq-set` and the
   `performance` CPU governor.
 * Vary the problem input size with `@Param`.
 * Use the integrated profilers in JMH to dig deeper if benchmark results to not match your hypotheses:
-    * Run the generated uberjar directly and use `-prof gc` to check whether the garbage collector runs during a microbenchmarks and skews 
+    * Run the generated uberjar directly and use `-prof gc` to check whether the garbage collector runs during a microbenchmarks and skews
    your results. If so, try to force a GC between runs (`-gc true`) but watch out for the caveats.
     * Use `-prof perf` or `-prof perfasm` (both only available on Linux) to see hotspots.
 * Have your benchmarks peer-reviewed.
@@ -59,4 +62,4 @@ To get realistic results, you should exercise care when running benchmarks. Here
 
 * Blindly believe the numbers that your microbenchmark produces but verify them by measuring e.g. with `-prof perfasm`.
 * Run more threads than your number of CPU cores (in case you run multi-threaded microbenchmarks).
-* Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.
+* Look only at the `Score` column and ignore `Error`. Instead take countermeasures to keep `Error` low / variance explainable.

benchmarks/build.gradle

@@ -18,11 +18,8 @@
  */
 
 apply plugin: 'elasticsearch.build'
-
+apply plugin: 'application'
-// order of this section matters, see: https://github.com/johnrengelman/shadow/issues/336
-apply plugin: 'application' // have the shadow plugin provide the runShadow task
 mainClassName = 'org.openjdk.jmh.Main'
-apply plugin: 'com.github.johnrengelman.shadow' // build an uberjar with all benchmarks
 
 // Not published so no need to assemble
 tasks.remove(assemble)
@@ -50,10 +47,8 @@ compileJava.options.compilerArgs << "-Xlint:-cast,-deprecation,-rawtypes,-try,-u
 // needs to be added separately otherwise Gradle will quote it and javac will fail
 compileJava.options.compilerArgs.addAll(["-processor", "org.openjdk.jmh.generators.BenchmarkProcessor"])
 
-forbiddenApis {
+// classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
-    // classes generated by JMH can use all sorts of forbidden APIs but we have no influence at all and cannot exclude these classes
+forbiddenApisMain.enabled = false
-    ignoreFailures = true
-}
 
 // No licenses for our benchmark deps (we don't ship benchmarks)
 dependencyLicenses.enabled = false
@@ -69,20 +64,3 @@ thirdPartyAudit.excludes = [
         'org.openjdk.jmh.profile.HotspotRuntimeProfiler',
         'org.openjdk.jmh.util.Utils'
 ]
-
-runShadow {
-    executable = new File(project.runtimeJavaHome, 'bin/java')
-}
-
-// alias the shadowJar and runShadow tasks to abstract from the concrete plugin that we are using and provide a more consistent interface
-task jmhJar(
-        dependsOn: shadowJar,
-        description: 'Generates an uberjar with the microbenchmarks and all dependencies',
-        group: 'Benchmark'
-)
-
-task jmh(
-        dependsOn: runShadow,
-        description: 'Runs all microbenchmarks',
-        group: 'Benchmark'
-)

buildSrc/src/main/groovy/org/elasticsearch/gradle/BuildPlugin.groovy

@@ -405,6 +405,10 @@ class BuildPlugin implements Plugin<Project> {
             repos.mavenLocal()
         }
         repos.mavenCentral()
+        repos.maven {
+            name "elastic"
+            url "https://artifacts.elastic.co/maven"
+        }
         String luceneVersion = VersionProperties.lucene
         if (luceneVersion.contains('-snapshot')) {
             // extract the revision number from the version with a regex matcher
@@ -777,11 +781,16 @@ class BuildPlugin implements Plugin<Project> {
                     systemProperty property.getKey(), property.getValue()
                 }
             }
+
+            // TODO: remove this once joda time is removed from scriptin in 7.0
+            systemProperty 'es.scripting.use_java_time', 'true'
+
             // Set the system keystore/truststore password if we're running tests in a FIPS-140 JVM
             if (project.inFipsJvm) {
                 systemProperty 'javax.net.ssl.trustStorePassword', 'password'
                 systemProperty 'javax.net.ssl.keyStorePassword', 'password'
             }
+
             boolean assertionsEnabled = Boolean.parseBoolean(System.getProperty('tests.asserts', 'true'))
             enableSystemAssertions assertionsEnabled
             enableAssertions assertionsEnabled

buildSrc/src/main/groovy/org/elasticsearch/gradle/vagrant/VagrantTestPlugin.groovy

@@ -526,11 +526,7 @@ class VagrantTestPlugin implements Plugin<Project> {
                     project.gradle.removeListener(batsPackagingReproListener)
                 }
                 if (project.extensions.esvagrant.boxes.contains(box)) {
-                    // these tests are temporarily disabled for suse boxes while we debug an issue
+                    packagingTest.dependsOn(batsPackagingTest)
-                    // https://github.com/elastic/elasticsearch/issues/30295
-                    if (box.equals("opensuse-42") == false && box.equals("sles-12") == false) {
-                        packagingTest.dependsOn(batsPackagingTest)
-                    }
                 }
             }
 
@@ -569,11 +565,7 @@ class VagrantTestPlugin implements Plugin<Project> {
                 project.gradle.removeListener(javaPackagingReproListener)
             }
             if (project.extensions.esvagrant.boxes.contains(box)) {
-                // these tests are temporarily disabled for suse boxes while we debug an issue
+                packagingTest.dependsOn(javaPackagingTest)
-                // https://github.com/elastic/elasticsearch/issues/30295
-                if (box.equals("opensuse-42") == false && box.equals("sles-12") == false) {
-                    packagingTest.dependsOn(javaPackagingTest)
-                }
             }
 
             /*

client/benchmark/README.md

@@ -2,10 +2,18 @@
 
 1. Build `client-benchmark-noop-api-plugin` with `gradle :client:client-benchmark-noop-api-plugin:assemble`
 2. Install it on the target host with `bin/elasticsearch-plugin install file:///full/path/to/client-benchmark-noop-api-plugin.zip`
-3. Start Elasticsearch on the target host (ideally *not* on the same machine)
+3. Start Elasticsearch on the target host (ideally *not* on the machine
-4. Build an uberjar with `gradle :client:benchmark:shadowJar` and execute it.
+that runs the benchmarks)
+4. Run the benchmark with
+```
+./gradlew -p client/benchmark run --args ' params go here'
+```
 
-Repeat all steps above for the other benchmark candidate.
+Everything in the `'` gets sent on the command line to JMH. The leading ` `
+inside the `'`s is important. Without it parameters are sometimes sent to
+gradle.
+
+See below for some example invocations.
 
 ### Example benchmark
 
@@ -13,32 +21,35 @@ In general, you should define a few GC-related settings `-Xms8192M -Xmx8192M -XX
 
 #### Bulk indexing
 
-Download benchmark data from http://benchmarks.elastic.co/corpora/geonames/documents.json.bz2 and decompress them.
+Download benchmark data from http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames and decompress them.
 
-Example command line parameters:
+Example invocation:
 
 ```
-rest bulk 192.168.2.2 ./documents.json geonames type 8647880 5000
+wget http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents-2.json.bz2
+bzip2 -d documents-2.json.bz2
+mv documents-2.json client/benchmark/build
+gradlew -p client/benchmark run --args ' rest bulk localhost build/documents-2.json geonames type 8647880 5000'
 ```
 
-The parameters are in order:
+The parameters are all in the `'`s and are in order:
 
 * Client type: Use either "rest" or "transport"
 * Benchmark type: Use either "bulk" or "search"
 * Benchmark target host IP (the host where Elasticsearch is running)
 * full path to the file that should be bulk indexed
 * name of the index
-* name of the (sole) type in the index 
+* name of the (sole) type in the index
 * number of documents in the file
 * bulk size
 
 
-#### Bulk indexing
+#### Search
 
-Example command line parameters:
+Example invocation:
 
 ```
-rest search 192.168.2.2 geonames "{ \"query\": { \"match_phrase\": { \"name\": \"Sankt Georgen\" } } }\"" 500,1000,1100,1200
+gradlew -p client/benchmark run --args ' rest search localhost geonames {"query":{"match_phrase":{"name":"Sankt Georgen"}}} 500,1000,1100,1200'
 ```
 
 The parameters are in order:
@@ -49,5 +60,3 @@ The parameters are in order:
 * name of the index
 * a search request body (remember to escape double quotes). The `TransportClientBenchmark` uses `QueryBuilders.wrapperQuery()` internally which automatically adds a root key `query`, so it must not be present in the command line parameter.
 * A comma-separated list of target throughput rates
-
-

client/benchmark/build.gradle

@@ -18,9 +18,6 @@
  */
 
 apply plugin: 'elasticsearch.build'
-// build an uberjar with all benchmarks
-apply plugin: 'com.github.johnrengelman.shadow'
-// have the shadow plugin provide the runShadow task
 apply plugin: 'application'
 
 group = 'org.elasticsearch.client'
@@ -32,7 +29,6 @@ build.dependsOn.remove('assemble')
 archivesBaseName = 'client-benchmarks'
 mainClassName = 'org.elasticsearch.client.benchmark.BenchmarkMain'
 
-
 // never try to invoke tests on the benchmark project - there aren't any
 test.enabled = false
 

client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java

@@ -205,6 +205,8 @@ public class RestHighLevelClient implements Closeable {
     private final SnapshotClient snapshotClient = new SnapshotClient(this);
     private final TasksClient tasksClient = new TasksClient(this);
     private final XPackClient xPackClient = new XPackClient(this);
+    private final WatcherClient watcherClient = new WatcherClient(this);
+    private final LicenseClient licenseClient = new LicenseClient(this);
 
     /**
      * Creates a {@link RestHighLevelClient} given the low level {@link RestClientBuilder} that allows to build the
@@ -296,18 +298,38 @@ public class RestHighLevelClient implements Closeable {
     }
 
     /**
-     * A wrapper for the {@link RestHighLevelClient} that provides methods for
+     * Provides methods for accessing the Elastic Licensed X-Pack Info
-     * accessing the Elastic Licensed X-Pack APIs that are shipped with the
+     * and Usage APIs that are shipped with the default distribution of
-     * default distribution of Elasticsearch. All of these APIs will 404 if run
+     * Elasticsearch. All of these APIs will 404 if run against the OSS
-     * against the OSS distribution of Elasticsearch.
+     * distribution of Elasticsearch.
      * <p>
-     * See the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/xpack-api.html">
+     * See the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/info-api.html">
-     * X-Pack APIs on elastic.co</a> for more information.
+     * Info APIs on elastic.co</a> for more information.
      */
     public final XPackClient xpack() {
         return xPackClient;
     }
 
+    /**
+     * Provides methods for accessing the Elastic Licensed Watcher APIs that
+     * are shipped with the default distribution of Elasticsearch. All of
+     * these APIs will 404 if run against the OSS distribution of Elasticsearch.
+     * <p>
+     * See the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api.html">
+     * Watcher APIs on elastic.co</a> for more information.
+     */
+    public WatcherClient watcher() { return watcherClient; }
+
+    /**
+     * Provides methods for accessing the Elastic Licensed Licensing APIs that
+     * are shipped with the default distribution of Elasticsearch. All of
+     * these APIs will 404 if run against the OSS distribution of Elasticsearch.
+     * <p>
+     * See the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/licensing-apis.html">
+     * Licensing APIs on elastic.co</a> for more information.
+     */
+    public LicenseClient license() { return licenseClient; }
+
     /**
      * Executes a bulk request using the Bulk API.
      * See <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html">Bulk API on elastic.co</a>

client/rest-high-level/src/main/java/org/elasticsearch/client/XPackClient.java

@@ -41,17 +41,9 @@ import static java.util.Collections.emptySet;
 public final class XPackClient {
 
     private final RestHighLevelClient restHighLevelClient;
-    private final WatcherClient watcherClient;
-    private final LicenseClient licenseClient;
 
     XPackClient(RestHighLevelClient restHighLevelClient) {
         this.restHighLevelClient = restHighLevelClient;
-        this.watcherClient = new WatcherClient(restHighLevelClient);
-        this.licenseClient = new LicenseClient(restHighLevelClient);
-    }
-
-    public WatcherClient watcher() {
-        return watcherClient;
     }
 
     /**
@@ -102,15 +94,4 @@ public final class XPackClient {
         restHighLevelClient.performRequestAsyncAndParseEntity(request, RequestConverters::xpackUsage, options,
             XPackUsageResponse::fromXContent, listener, emptySet());
     }
-
-    /**
-     * A wrapper for the {@link RestHighLevelClient} that provides methods for
-     * accessing the Elastic Licensing APIs.
-     * <p>
-     * See the <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/licensing-apis.html">
-     * X-Pack APIs on elastic.co</a> for more information.
-     */
-    public LicenseClient license() {
-        return licenseClient;
-    }
 }

client/rest-high-level/src/test/java/org/elasticsearch/client/RestHighLevelClientTests.java

@@ -755,7 +755,9 @@ public class RestHighLevelClientTests extends ESTestCase {
                             method.isAnnotationPresent(Deprecated.class));
                     } else {
                         //TODO xpack api are currently ignored, we need to load xpack yaml spec too
-                        if (apiName.startsWith("xpack.") == false) {
+                        if (apiName.startsWith("xpack.") == false &&
+                            apiName.startsWith("license.") == false &&
+                            apiName.startsWith("watcher.") == false) {
                             apiNotFound.add(apiName);
                         }
                     }

client/rest-high-level/src/test/java/org/elasticsearch/client/WatcherIT.java

@@ -46,7 +46,7 @@ public class WatcherIT extends ESRestHighLevelClientTestCase {
             "}";
         BytesReference bytesReference = new BytesArray(json);
         PutWatchRequest putWatchRequest = new PutWatchRequest(watchId, bytesReference, XContentType.JSON);
-        return highLevelClient().xpack().watcher().putWatch(putWatchRequest, RequestOptions.DEFAULT);
+        return highLevelClient().watcher().putWatch(putWatchRequest, RequestOptions.DEFAULT);
     }
 
     public void testDeleteWatch() throws Exception {
@@ -54,7 +54,7 @@ public class WatcherIT extends ESRestHighLevelClientTestCase {
         {
             String watchId = randomAlphaOfLength(10);
             createWatch(watchId);
-            DeleteWatchResponse deleteWatchResponse = highLevelClient().xpack().watcher().deleteWatch(new DeleteWatchRequest(watchId),
+            DeleteWatchResponse deleteWatchResponse = highLevelClient().watcher().deleteWatch(new DeleteWatchRequest(watchId),
                 RequestOptions.DEFAULT);
             assertThat(deleteWatchResponse.getId(), is(watchId));
             assertThat(deleteWatchResponse.getVersion(), is(2L));
@@ -64,7 +64,7 @@ public class WatcherIT extends ESRestHighLevelClientTestCase {
         // delete watch that does not exist
         {
             String watchId = randomAlphaOfLength(10);
-            DeleteWatchResponse deleteWatchResponse = highLevelClient().xpack().watcher().deleteWatch(new DeleteWatchRequest(watchId),
+            DeleteWatchResponse deleteWatchResponse = highLevelClient().watcher().deleteWatch(new DeleteWatchRequest(watchId),
                 RequestOptions.DEFAULT);
             assertThat(deleteWatchResponse.getId(), is(watchId));
             assertThat(deleteWatchResponse.getVersion(), is(1L));

client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/LicensingDocumentationIT.java

@@ -19,6 +19,7 @@
 
 package org.elasticsearch.client.documentation;
 
+import org.apache.lucene.util.LuceneTestCase.AwaitsFix;
 import org.elasticsearch.action.ActionListener;
 import org.elasticsearch.action.LatchedActionListener;
 import org.elasticsearch.client.ESRestHighLevelClientTestCase;
@@ -40,6 +41,7 @@ import static org.hamcrest.Matchers.startsWith;
  * Documentation for Licensing APIs in the high level java client.
  * Code wrapped in {@code tag} and {@code end} tags is included in the docs.
  */
+@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/32580")
 public class LicensingDocumentationIT extends ESRestHighLevelClientTestCase {
 
     public void testPutLicense() throws Exception {
@@ -60,7 +62,7 @@ public class LicensingDocumentationIT extends ESRestHighLevelClientTestCase {
             request.setLicenseDefinition(license);  // <1>
             request.setAcknowledge(false);          // <2>
 
-            PutLicenseResponse response = client.xpack().license().putLicense(request, RequestOptions.DEFAULT);
+            PutLicenseResponse response = client.license().putLicense(request, RequestOptions.DEFAULT);
             //end::put-license-execute
 
             //tag::put-license-response
@@ -96,7 +98,7 @@ public class LicensingDocumentationIT extends ESRestHighLevelClientTestCase {
             listener = new LatchedActionListener<>(listener, latch);
 
             // tag::put-license-execute-async
-            client.xpack().license().putLicenseAsync(
+            client.license().putLicenseAsync(
                     request, RequestOptions.DEFAULT, listener); // <1>
             // end::put-license-execute-async
 

client/rest-high-level/src/test/java/org/elasticsearch/client/documentation/WatcherDocumentationIT.java

@@ -49,7 +49,7 @@ public class WatcherDocumentationIT extends ESRestHighLevelClientTestCase {
                 "}");
             PutWatchRequest request = new PutWatchRequest("my_watch_id", watch, XContentType.JSON);
             request.setActive(false); // <1>
-            PutWatchResponse response = client.xpack().watcher().putWatch(request, RequestOptions.DEFAULT);
+            PutWatchResponse response = client.watcher().putWatch(request, RequestOptions.DEFAULT);
             //end::x-pack-put-watch-execute
 
             //tag::x-pack-put-watch-response
@@ -85,7 +85,7 @@ public class WatcherDocumentationIT extends ESRestHighLevelClientTestCase {
             listener = new LatchedActionListener<>(listener, latch);
 
             // tag::x-pack-put-watch-execute-async
-            client.xpack().watcher().putWatchAsync(request, RequestOptions.DEFAULT, listener); // <1>
+            client.watcher().putWatchAsync(request, RequestOptions.DEFAULT, listener); // <1>
             // end::x-pack-put-watch-execute-async
 
             assertTrue(latch.await(30L, TimeUnit.SECONDS));
@@ -94,7 +94,7 @@ public class WatcherDocumentationIT extends ESRestHighLevelClientTestCase {
         {
             //tag::x-pack-delete-watch-execute
             DeleteWatchRequest request = new DeleteWatchRequest("my_watch_id");
-            DeleteWatchResponse response = client.xpack().watcher().deleteWatch(request, RequestOptions.DEFAULT);
+            DeleteWatchResponse response = client.watcher().deleteWatch(request, RequestOptions.DEFAULT);
             //end::x-pack-delete-watch-execute
 
             //tag::x-pack-delete-watch-response
@@ -125,7 +125,7 @@ public class WatcherDocumentationIT extends ESRestHighLevelClientTestCase {
             listener = new LatchedActionListener<>(listener, latch);
 
             // tag::x-pack-delete-watch-execute-async
-            client.xpack().watcher().deleteWatchAsync(request, RequestOptions.DEFAULT, listener); // <1>
+            client.watcher().deleteWatchAsync(request, RequestOptions.DEFAULT, listener); // <1>
             // end::x-pack-delete-watch-execute-async
 
             assertTrue(latch.await(30L, TimeUnit.SECONDS));

docs/build.gradle

@@ -37,6 +37,9 @@ integTestCluster {
   extraConfigFile 'hunspell/en_US/en_US.dic', '../server/src/test/resources/indices/analyze/conf_dir/hunspell/en_US/en_US.dic'
   // Whitelist reindexing from the local node so we can test it.
   setting 'reindex.remote.whitelist', '127.0.0.1:*'
+
+  // TODO: remove this for 7.0, this exists to allow the doc examples in 6.x to continue using the defaults
+  systemProperty 'es.scripting.use_java_time', 'false'
 }
 
 // remove when https://github.com/elastic/elasticsearch/issues/31305 is fixed

docs/java-rest/high-level/search/explain.asciidoc

@@ -80,7 +80,7 @@ A typical listener for `ExplainResponse` is constructed as follows:
 include-tagged::{doc-tests}/SearchDocumentationIT.java[explain-execute-listener]
 --------------------------------------------------
 <1> Called when the execution is successfully completed.
-<2> Called when the whole `FieldCapabilitiesRequest` fails.
+<2> Called when the whole `ExplainRequest` fails.
 
 [[java-rest-high-explain-response]]
 ==== ExplainResponse

docs/painless/painless-getting-started.asciidoc

@@ -198,7 +198,7 @@ POST hockey/player/1/_update
 ==== Dates
 
 Date fields are exposed as
-`ReadableDateTime`
+`ReadableDateTime` or
 so they support methods like
 `getYear`,
 and `getDayOfWeek`.
@@ -220,6 +220,11 @@ GET hockey/_search
 }
 ----------------------------------------------------------------
 // CONSOLE
+// TEST[warning:The joda time api for doc values is deprecated. Use -Des.scripting.use_java_time=true to use the java time api for date field doc values]
+
+NOTE: Date fields are changing in 7.0 to be exposed as `ZonedDateTime`
+from Java 8's time API. To switch to this functionality early, 
+add `-Des.scripting.use_java_time=true` to `jvm.options`.
 
 [float]
 [[modules-scripting-painless-regex]]

docs/plugins/discovery-ec2.asciidoc

@@ -37,7 +37,6 @@ bin/elasticsearch-keystore add discovery.ec2.secret_key
 The following are the available discovery settings. All should be prefixed with `discovery.ec2.`.
 Those that must be stored in the keystore are marked as `Secure`.
 
-
 `access_key`::
 
     An ec2 access key. The `secret_key` setting must also be specified. (Secure)
@@ -122,6 +121,10 @@ Defaults to `private_ip`.
     How long the list of hosts is cached to prevent further requests to the AWS API.
     Defaults to `10s`.
 
+*All* secure settings of this plugin are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
+After you reload the settings, an aws sdk client with the latest settings
+from the keystore will be used.
+
 [IMPORTANT]
 .Binding the network host
 ==============================================

docs/plugins/repository-azure.asciidoc

@@ -11,7 +11,7 @@ include::install_remove.asciidoc[]
 ==== Azure Repository
 
 To enable Azure repositories, you have first to define your azure storage settings as
-{ref}/secure-settings.html[secured settings]:
+{ref}/secure-settings.html[secure settings], before starting up the node:
 
 [source,sh]
 ----------------------------------------------------------------
@@ -20,6 +20,7 @@ bin/elasticsearch-keystore add azure.client.default.key
 ----------------------------------------------------------------
 
 Where `account` is the azure account name and `key` the azure secret key.
+These settings are used by the repository's internal azure client.
 
 Note that you can also define more than one account:
 
@@ -31,7 +32,18 @@ bin/elasticsearch-keystore add azure.client.secondary.account
 bin/elasticsearch-keystore add azure.client.secondary.key
 ----------------------------------------------------------------
 
-`default` is the default account name which will be used by a repository unless you set an explicit one.
+`default` is the default account name which will be used by a repository,
+unless you set an explicit one in the
+<<repository-azure-repository-settings, repository settings>>.
+
+Both `account` and `key` storage settings are
+{ref}/secure-settings.html#reloadable-secure-settings[reloadable]. After you
+reload the settings, the internal azure clients, which are used to transfer the
+snapshot, will utilize the latest settings from the keystore.
+
+NOTE: In progress snapshot/restore jobs will not be preempted by a *reload*
+of the storage secure settings. They will complete using the client as it was built
+when the operation started.
 
 You can set the client side timeout to use when making any single request. It can be defined globally, per account or both.
 It's not set by default which means that Elasticsearch is using the

docs/plugins/repository-gcs.asciidoc

@@ -112,6 +112,15 @@ PUT _snapshot/my_gcs_repository
 // CONSOLE
 // TEST[skip:we don't have gcs setup while testing this]
 
+The `credentials_file` settings are {ref}/secure-settings.html#reloadable-secure-settings[reloadable].
+After you reload the settings, the internal `gcs` clients, used to transfer the
+snapshot contents, will utilize the latest settings from the keystore.
+
+
+NOTE: In progress snapshot/restore jobs will not be preempted by a *reload*
+of the client's `credentials_file` settings. They will complete using the client
+as it was built when the operation started.
+
 [[repository-gcs-client]]
 ==== Client Settings
 

docs/plugins/repository-s3.asciidoc

@@ -35,9 +35,9 @@ PUT _snapshot/my_s3_repository
 ==== Client Settings
 
 The client used to connect to S3 has a number of settings available. Client setting names are of
-the form `s3.client.CLIENT_NAME.SETTING_NAME` and specified inside `elasticsearch.yml`.  The
+the form `s3.client.CLIENT_NAME.SETTING_NAME`. The default client name, which is looked up by
-default client name looked up by a `s3` repository is called `default`, but can be customized
+an `s3` repository, is called `default`. It can be modified using the
-with the repository setting `client`. For example:
+<<repository-s3-repository, repository setting>> `client`. For example:
 
 [source,js]
 ----
@@ -53,8 +53,11 @@ PUT _snapshot/my_s3_repository
 // CONSOLE
 // TEST[skip:we don't have s3 setup while testing this]
 
-Some settings are sensitive and must be stored in the {ref}/secure-settings.html[elasticsearch keystore].
+Most client settings are specified inside `elasticsearch.yml`, but some are
-For example, to use explicit AWS access keys:
+sensitive and must be stored in the {ref}/secure-settings.html[elasticsearch keystore].
+
+For example, before you start the node, run these commands to add AWS access
+key settings to the keystore:
 
 [source,sh]
 ----
@@ -62,8 +65,19 @@ bin/elasticsearch-keystore add s3.client.default.access_key
 bin/elasticsearch-keystore add s3.client.default.secret_key
 ----
 
-The following are the available client settings. Those that must be stored in the keystore
+*All* client secure settings of this plugin are
-are marked as `Secure`.
+{ref}/secure-settings.html#reloadable-secure-settings[reloadable]. After you
+reload the settings, the internal `s3` clients, used to transfer the snapshot
+contents, will utilize the latest settings from the keystore. Any existing `s3`
+repositories, as well as any newly created ones, will pick up the new values
+stored in the keystore.
+
+NOTE: In progress snapshot/restore tasks will not be preempted by a *reload*
+of the client's secure settings. The task will complete using the client as it
+was built when the operation started.
+
+The following is the list of all the available client settings.
+Those that must be stored in the keystore are marked as `Secure` and are *reloadable*.
 
 `access_key`::
 

docs/reference/aggregations/bucket/datehistogram-aggregation.asciidoc

@@ -425,6 +425,7 @@ POST /sales/_search?size=0
 --------------------------------------------------
 // CONSOLE
 // TEST[setup:sales]
+// TEST[warning:The joda time api for doc values is deprecated. Use -Des.scripting.use_java_time=true to use the java time api for date field doc values]
 
 Response:
 

docs/reference/analysis/charfilters/pattern-replace-charfilter.asciidoc

@@ -87,7 +87,7 @@ The above example produces the following term:
 
 [source,text]
 ---------------------------
-[ My, credit, card, is 123_456_789 ]
+[ My, credit, card, is, 123_456_789 ]
 ---------------------------
 
 WARNING: Using a replacement string that changes the length of the original

docs/reference/cluster/stats.asciidoc

@@ -22,6 +22,7 @@ Will return, for example:
       "successful" : 1,
       "failed" : 0
    },
+   "cluster_uuid": "YjAvIhsCQ9CbjWZb2qJw3Q",
    "cluster_name": "elasticsearch",
    "timestamp": 1459427693515,
    "status": "green",

docs/reference/setup/secure-settings.asciidoc

@@ -75,3 +75,34 @@ To remove a setting from the keystore, use the `remove` command:
 bin/elasticsearch-keystore remove the.setting.name.to.remove
 ----------------------------------------------------------------
 
+[float]
+[[reloadable-secure-settings]]
+=== Reloadable secure settings
+
+Just like the settings values in `elasticsearch.yml`, changes to the
+keystore contents are not automatically applied to the running
+elasticsearch node. Re-reading settings requires a node restart.
+However, certain secure settings are marked as *reloadable*. Such settings
+can be re-read and applied on a running node.
+
+The values of all secure settings, *reloadable* or not, must be identical
+across all cluster nodes. After making the desired secure settings changes,
+using the `bin/elasticsearch-keystore add` command, call:
+[source,js]
+----
+POST _nodes/reload_secure_settings
+{
+  "secure_settings_password": ""
+}
+----
+// CONSOLE
+This API will decrypt and re-read the entire keystore, on every cluster node,
+but only the *reloadable* secure settings will be applied. Changes to other
+settings will not go into effect until the next restart. Once the call returns,
+the reload has been completed, meaning that all internal datastructures dependent
+on these settings have been changed. Everything should look as if the settings
+had the new value from the start.
+
+When changing multiple *reloadable* secure settings, modify all of them, on
+each cluster node, and then issue a `reload_secure_settings` call, instead
+of reloading after each modification.

modules/ingest-common/src/test/resources/rest-api-spec/test/ingest/200_default_pipeline.yml

@@ -0,0 +1,73 @@
+---
+teardown:
+  - do:
+      ingest.delete_pipeline:
+        id: "my_pipeline"
+        ignore: 404
+
+---
+"Test index with default pipeline":
+  - do:
+      ingest.put_pipeline:
+        id: "my_pipeline"
+        body:  >
+          {
+            "description": "_description",
+            "processors": [
+              {
+                "bytes" : {
+                  "field" : "bytes_source_field",
+                  "target_field" : "bytes_target_field"
+                }
+              }
+            ]
+          }
+  - match: { acknowledged: true }
+
+  - do:
+      indices.create:
+        index: test
+        body:
+          settings:
+            index:
+              default_pipeline: "my_pipeline"
+
+  - do:
+      index:
+        index: test
+        type: test
+        id: 1
+        body: {bytes_source_field: "1kb"}
+
+  - do:
+      get:
+        index: test
+        type: test
+        id: 1
+  - match: { _source.bytes_source_field: "1kb" }
+  - match: { _source.bytes_target_field: 1024 }
+
+  - do:
+      index:
+        index: test
+        type: test
+        id: 2
+        pipeline: "_none"
+        body: {bytes_source_field: "1kb"}
+        
+  - do:
+      get:
+        index: test
+        type: test
+        id: 2
+  - match: { _source.bytes_source_field: "1kb" }
+  - is_false: _source.bytes_target_field
+  
+  - do:
+      catch:  bad_request
+      index:
+        index: test
+        type: test
+        id: 3
+        pipeline: ""
+        body: {bytes_source_field: "1kb"}

modules/lang-painless/build.gradle

@@ -17,8 +17,6 @@
  * under the License.
  */
 
-
-
 esplugin {
   description 'An easy, safe and fast scripting language for Elasticsearch'
   classname 'org.elasticsearch.painless.PainlessPlugin'
@@ -26,6 +24,7 @@ esplugin {
 
 integTestCluster {
   module project.project(':modules:mapper-extras')
+  systemProperty 'es.scripting.use_java_time', 'true'
 }
 
 dependencies {

modules/lang-painless/src/main/java/org/elasticsearch/painless/Compiler.java

@@ -69,17 +69,14 @@ final class Compiler {
     /**
      * A secure class loader used to define Painless scripts.
      */
-    static final class Loader extends SecureClassLoader {
+    final class Loader extends SecureClassLoader {
         private final AtomicInteger lambdaCounter = new AtomicInteger(0);
-        private final PainlessLookup painlessLookup;
 
         /**
          * @param parent The parent ClassLoader.
          */
-        Loader(ClassLoader parent, PainlessLookup painlessLookup) {
+        Loader(ClassLoader parent) {
             super(parent);
-
-            this.painlessLookup = painlessLookup;
         }
 
         /**
@@ -90,7 +87,16 @@ final class Compiler {
          */
         @Override
         public Class<?> findClass(String name) throws ClassNotFoundException {
-            Class<?> found = painlessLookup.getClassFromBinaryName(name);
+            if (scriptClass.getName().equals(name)) {
+                return scriptClass;
+            }
+            if (factoryClass != null && factoryClass.getName().equals(name)) {
+                return factoryClass;
+            }
+            if (statefulFactoryClass != null && statefulFactoryClass.getName().equals(name)) {
+                return statefulFactoryClass;
+            }
+            Class<?> found = painlessLookup.canonicalTypeNameToType(name.replace('$', '.'));
 
             return found != null ? found : super.findClass(name);
         }
@@ -139,13 +145,23 @@ final class Compiler {
      * {@link Compiler}'s specified {@link PainlessLookup}.
      */
     public Loader createLoader(ClassLoader parent) {
-        return new Loader(parent, painlessLookup);
+        return new Loader(parent);
     }
 
     /**
-     * The class/interface the script is guaranteed to derive/implement.
+     * The class/interface the script will implement.
      */
-    private final Class<?> base;
+    private final Class<?> scriptClass;
+
+    /**
+     * The class/interface to create the {@code scriptClass} instance.
+     */
+    private final Class<?> factoryClass;
+
+    /**
+     * An optional class/interface to create the {@code factoryClass} instance.
+     */
+    private final Class<?> statefulFactoryClass;
 
     /**
      * The whitelist the script will use.
@@ -154,11 +170,15 @@ final class Compiler {
 
     /**
      * Standard constructor.
-     * @param base The class/interface the script is guaranteed to derive/implement.
+     * @param scriptClass The class/interface the script will implement.
+     * @param factoryClass An optional class/interface to create the {@code scriptClass} instance.
+     * @param statefulFactoryClass An optional class/interface to create the {@code factoryClass} instance.
      * @param painlessLookup The whitelist the script will use.
      */
-    Compiler(Class<?> base, PainlessLookup painlessLookup) {
+    Compiler(Class<?> scriptClass, Class<?> factoryClass, Class<?> statefulFactoryClass, PainlessLookup painlessLookup) {
-        this.base = base;
+        this.scriptClass = scriptClass;
+        this.factoryClass = factoryClass;
+        this.statefulFactoryClass = statefulFactoryClass;
         this.painlessLookup = painlessLookup;
     }
 
@@ -177,7 +197,7 @@ final class Compiler {
                 " plugin if a script longer than this length is a requirement.");
         }
 
-        ScriptClassInfo scriptClassInfo = new ScriptClassInfo(painlessLookup, base);
+        ScriptClassInfo scriptClassInfo = new ScriptClassInfo(painlessLookup, scriptClass);
         SSource root = Walker.buildPainlessTree(scriptClassInfo, reserved, name, source, settings, painlessLookup,
                 null);
         root.analyze(painlessLookup);
@@ -209,7 +229,7 @@ final class Compiler {
                 " plugin if a script longer than this length is a requirement.");
         }
 
-        ScriptClassInfo scriptClassInfo = new ScriptClassInfo(painlessLookup, base);
+        ScriptClassInfo scriptClassInfo = new ScriptClassInfo(painlessLookup, scriptClass);
         SSource root = Walker.buildPainlessTree(scriptClassInfo, new MainMethodReserved(), name, source, settings, painlessLookup,
                 debugStream);
         root.analyze(painlessLookup);

modules/lang-painless/src/main/java/org/elasticsearch/painless/Def.java

@@ -187,7 +187,7 @@ public final class Def {
         String key = PainlessLookupUtility.buildPainlessMethodKey(name, arity);
         // check whitelist for matching method
         for (Class<?> clazz = receiverClass; clazz != null; clazz = clazz.getSuperclass()) {
-            PainlessClass struct = painlessLookup.getPainlessStructFromJavaClass(clazz);
+            PainlessClass struct = painlessLookup.lookupPainlessClass(clazz);
 
             if (struct != null) {
                 PainlessMethod method = struct.methods.get(key);
@@ -197,7 +197,7 @@ public final class Def {
             }
 
             for (Class<?> iface : clazz.getInterfaces()) {
-                struct = painlessLookup.getPainlessStructFromJavaClass(iface);
+                struct = painlessLookup.lookupPainlessClass(iface);
 
                 if (struct != null) {
                     PainlessMethod method = struct.methods.get(key);
@@ -238,7 +238,7 @@ public final class Def {
          int numArguments = callSiteType.parameterCount();
          // simple case: no lambdas
          if (recipeString.isEmpty()) {
-             return lookupMethodInternal(painlessLookup, receiverClass, name, numArguments - 1).handle;
+             return lookupMethodInternal(painlessLookup, receiverClass, name, numArguments - 1).methodHandle;
          }
 
          // convert recipe string to a bitset for convenience (the code below should be refactored...)
@@ -262,7 +262,7 @@ public final class Def {
          // lookup the method with the proper arity, then we know everything (e.g. interface types of parameters).
          // based on these we can finally link any remaining lambdas that were deferred.
          PainlessMethod method = lookupMethodInternal(painlessLookup, receiverClass, name, arity);
-         MethodHandle handle = method.handle;
+         MethodHandle handle = method.methodHandle;
 
          int replaced = 0;
          upTo = 1;
@@ -281,7 +281,7 @@ public final class Def {
                      captures[capture] = callSiteType.parameterType(i + 1 + capture);
                  }
                  MethodHandle filter;
-                 Class<?> interfaceType = method.arguments.get(i - 1 - replaced);
+                 Class<?> interfaceType = method.typeParameters.get(i - 1 - replaced);
                  if (signature.charAt(0) == 'S') {
                      // the implementation is strongly typed, now that we know the interface type,
                      // we have everything.
@@ -326,15 +326,16 @@ public final class Def {
       */
     static MethodHandle lookupReference(PainlessLookup painlessLookup, MethodHandles.Lookup methodHandlesLookup, String interfaceClass,
                                         Class<?> receiverClass, String name) throws Throwable {
-         Class<?> interfaceType = painlessLookup.getJavaClassFromPainlessType(interfaceClass);
+         Class<?> interfaceType = painlessLookup.canonicalTypeNameToType(interfaceClass);
-         PainlessMethod interfaceMethod = painlessLookup.getPainlessStructFromJavaClass(interfaceType).functionalMethod;
+         PainlessMethod interfaceMethod = painlessLookup.lookupPainlessClass(interfaceType).functionalMethod;
          if (interfaceMethod == null) {
              throw new IllegalArgumentException("Class [" + interfaceClass + "] is not a functional interface");
          }
-         int arity = interfaceMethod.arguments.size();
+         int arity = interfaceMethod.typeParameters.size();
          PainlessMethod implMethod = lookupMethodInternal(painlessLookup, receiverClass, name, arity);
         return lookupReferenceInternal(painlessLookup, methodHandlesLookup, interfaceType,
-                PainlessLookupUtility.typeToCanonicalTypeName(implMethod.target), implMethod.name, receiverClass);
+                PainlessLookupUtility.typeToCanonicalTypeName(implMethod.targetClass),
+                implMethod.javaMethod.getName(), receiverClass);
      }
 
      /** Returns a method handle to an implementation of clazz, given method reference signature. */
@@ -344,12 +345,12 @@ public final class Def {
          final FunctionRef ref;
          if ("this".equals(type)) {
              // user written method
-             PainlessMethod interfaceMethod = painlessLookup.getPainlessStructFromJavaClass(clazz).functionalMethod;
+             PainlessMethod interfaceMethod = painlessLookup.lookupPainlessClass(clazz).functionalMethod;
              if (interfaceMethod == null) {
                  throw new IllegalArgumentException("Cannot convert function reference [" + type + "::" + call + "] " +
                          "to [" + PainlessLookupUtility.typeToCanonicalTypeName(clazz) + "], not a functional interface");
              }
-             int arity = interfaceMethod.arguments.size() + captures.length;
+             int arity = interfaceMethod.typeParameters.size() + captures.length;
              final MethodHandle handle;
              try {
                  MethodHandle accessor = methodHandlesLookup.findStaticGetter(methodHandlesLookup.lookupClass(),
@@ -360,7 +361,7 @@ public final class Def {
                  // is it a synthetic method? If we generated the method ourselves, be more helpful. It can only fail
                  // because the arity does not match the expected interface type.
                  if (call.contains("$")) {
-                     throw new IllegalArgumentException("Incorrect number of parameters for [" + interfaceMethod.name +
+                     throw new IllegalArgumentException("Incorrect number of parameters for [" + interfaceMethod.javaMethod.getName() +
                                                         "] in [" + clazz + "]");
                  }
                  throw new IllegalArgumentException("Unknown call [" + call + "] with [" + arity + "] arguments.");
@@ -418,7 +419,7 @@ public final class Def {
     static MethodHandle lookupGetter(PainlessLookup painlessLookup, Class<?> receiverClass, String name) {
         // first try whitelist
         for (Class<?> clazz = receiverClass; clazz != null; clazz = clazz.getSuperclass()) {
-            PainlessClass struct = painlessLookup.getPainlessStructFromJavaClass(clazz);
+            PainlessClass struct = painlessLookup.lookupPainlessClass(clazz);
 
             if (struct != null) {
                 MethodHandle handle = struct.getterMethodHandles.get(name);
@@ -428,7 +429,7 @@ public final class Def {
             }
 
             for (final Class<?> iface : clazz.getInterfaces()) {
-                struct = painlessLookup.getPainlessStructFromJavaClass(iface);
+                struct = painlessLookup.lookupPainlessClass(iface);
 
                 if (struct != null) {
                     MethodHandle handle = struct.getterMethodHandles.get(name);
@@ -489,7 +490,7 @@ public final class Def {
     static MethodHandle lookupSetter(PainlessLookup painlessLookup, Class<?> receiverClass, String name) {
         // first try whitelist
         for (Class<?> clazz = receiverClass; clazz != null; clazz = clazz.getSuperclass()) {
-            PainlessClass struct = painlessLookup.getPainlessStructFromJavaClass(clazz);
+            PainlessClass struct = painlessLookup.lookupPainlessClass(clazz);
 
             if (struct != null) {
                 MethodHandle handle = struct.setterMethodHandles.get(name);
@@ -499,7 +500,7 @@ public final class Def {
             }
 
             for (final Class<?> iface : clazz.getInterfaces()) {
-                struct = painlessLookup.getPainlessStructFromJavaClass(iface);
+                struct = painlessLookup.lookupPainlessClass(iface);
 
                 if (struct != null) {
                     MethodHandle handle = struct.setterMethodHandles.get(name);

modules/lang-painless/src/main/java/org/elasticsearch/painless/FunctionRef.java

@@ -19,6 +19,7 @@
 
 package org.elasticsearch.painless;
 
+import org.elasticsearch.painless.Locals.LocalMethod;
 import org.elasticsearch.painless.lookup.PainlessClass;
 import org.elasticsearch.painless.lookup.PainlessConstructor;
 import org.elasticsearch.painless.lookup.PainlessLookup;
@@ -89,10 +90,10 @@ public class FunctionRef {
             PainlessLookup painlessLookup, Class<?> expected, String type, String call, int numCaptures) {
 
         if ("new".equals(call)) {
-            return new FunctionRef(expected, painlessLookup.getPainlessStructFromJavaClass(expected).functionalMethod,
+            return new FunctionRef(expected, painlessLookup.lookupPainlessClass(expected).functionalMethod,
                     lookup(painlessLookup, expected, type), numCaptures);
         } else {
-            return new FunctionRef(expected, painlessLookup.getPainlessStructFromJavaClass(expected).functionalMethod,
+            return new FunctionRef(expected, painlessLookup.lookupPainlessClass(expected).functionalMethod,
                     lookup(painlessLookup, expected, type, call, numCaptures > 0), numCaptures);
         }
     }
@@ -108,24 +109,24 @@ public class FunctionRef {
         Constructor<?> javaConstructor = delegateConstructor.javaConstructor;
         MethodType delegateMethodType = delegateConstructor.methodType;
 
-        interfaceMethodName = interfaceMethod.name;
+        this.interfaceMethodName = interfaceMethod.javaMethod.getName();
-        factoryMethodType = MethodType.methodType(expected,
+        this.factoryMethodType = MethodType.methodType(expected,
                 delegateMethodType.dropParameterTypes(numCaptures, delegateMethodType.parameterCount()));
-        interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
+        this.interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
 
-        delegateClassName = javaConstructor.getDeclaringClass().getName();
+        this.delegateClassName = javaConstructor.getDeclaringClass().getName();
-        isDelegateInterface = false;
+        this.isDelegateInterface = false;
-        delegateInvokeType = H_NEWINVOKESPECIAL;
+        this.delegateInvokeType = H_NEWINVOKESPECIAL;
-        delegateMethodName = PainlessLookupUtility.CONSTRUCTOR_NAME;
+        this.delegateMethodName = PainlessLookupUtility.CONSTRUCTOR_NAME;
         this.delegateMethodType = delegateMethodType.dropParameterTypes(0, numCaptures);
 
         this.interfaceMethod = interfaceMethod;
-        delegateTypeParameters = delegateConstructor.typeParameters;
+        this.delegateTypeParameters = delegateConstructor.typeParameters;
-        delegateReturnType = void.class;
+        this.delegateReturnType = void.class;
 
-        factoryDescriptor = factoryMethodType.toMethodDescriptorString();
+        this.factoryDescriptor = factoryMethodType.toMethodDescriptorString();
-        interfaceType = Type.getMethodType(interfaceMethodType.toMethodDescriptorString());
+        this.interfaceType = Type.getMethodType(interfaceMethodType.toMethodDescriptorString());
-        delegateType = Type.getMethodType(this.delegateMethodType.toMethodDescriptorString());
+        this.delegateType = Type.getMethodType(this.delegateMethodType.toMethodDescriptorString());
     }
 
     /**
@@ -138,41 +139,63 @@ public class FunctionRef {
     public FunctionRef(Class<?> expected, PainlessMethod interfaceMethod, PainlessMethod delegateMethod, int numCaptures) {
         MethodType delegateMethodType = delegateMethod.methodType;
 
-        interfaceMethodName = interfaceMethod.name;
+        this.interfaceMethodName = interfaceMethod.javaMethod.getName();
-        factoryMethodType = MethodType.methodType(expected,
+        this.factoryMethodType = MethodType.methodType(expected,
                 delegateMethodType.dropParameterTypes(numCaptures, delegateMethodType.parameterCount()));
-        interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
+        this.interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
 
-        // the Painless$Script class can be inferred if owner is null
+        this.delegateClassName = delegateMethod.javaMethod.getDeclaringClass().getName();
-        if (delegateMethod.target == null) {
+        this.isDelegateInterface = delegateMethod.javaMethod.getDeclaringClass().isInterface();
-            delegateClassName = CLASS_NAME;
+
-            isDelegateInterface = false;
+        if (Modifier.isStatic(delegateMethod.javaMethod.getModifiers())) {
-        } else if (delegateMethod.augmentation != null) {
+            this.delegateInvokeType = H_INVOKESTATIC;
-            delegateClassName = delegateMethod.augmentation.getName();
+        } else if (delegateMethod.javaMethod.getDeclaringClass().isInterface()) {
-            isDelegateInterface = delegateMethod.augmentation.isInterface();
+            this.delegateInvokeType = H_INVOKEINTERFACE;
         } else {
-            delegateClassName = delegateMethod.target.getName();
+            this.delegateInvokeType = H_INVOKEVIRTUAL;
-            isDelegateInterface = delegateMethod.target.isInterface();
         }
 
-        if (Modifier.isStatic(delegateMethod.modifiers)) {
+        this.delegateMethodName = delegateMethod.javaMethod.getName();
-            delegateInvokeType = H_INVOKESTATIC;
-        } else if (delegateMethod.target.isInterface()) {
-            delegateInvokeType = H_INVOKEINTERFACE;
-        } else {
-            delegateInvokeType = H_INVOKEVIRTUAL;
-        }
-
-        delegateMethodName = delegateMethod.name;
         this.delegateMethodType = delegateMethodType.dropParameterTypes(0, numCaptures);
 
         this.interfaceMethod = interfaceMethod;
-        delegateTypeParameters = delegateMethod.arguments;
+        this.delegateTypeParameters = delegateMethod.typeParameters;
-        delegateReturnType = delegateMethod.rtn;
+        this.delegateReturnType = delegateMethod.returnType;
 
-        factoryDescriptor = factoryMethodType.toMethodDescriptorString();
+        this.factoryDescriptor = factoryMethodType.toMethodDescriptorString();
-        interfaceType = Type.getMethodType(interfaceMethodType.toMethodDescriptorString());
+        this.interfaceType = Type.getMethodType(interfaceMethodType.toMethodDescriptorString());
-        delegateType = Type.getMethodType(this.delegateMethodType.toMethodDescriptorString());
+        this.delegateType = Type.getMethodType(this.delegateMethodType.toMethodDescriptorString());
+    }
+
+    /**
+     * Creates a new FunctionRef (already resolved)
+     * @param expected functional interface type to implement
+     * @param interfaceMethod functional interface method
+     * @param delegateMethod implementation method
+     * @param numCaptures number of captured arguments
+     */
+    public FunctionRef(Class<?> expected, PainlessMethod interfaceMethod, LocalMethod delegateMethod, int numCaptures) {
+        MethodType delegateMethodType = delegateMethod.methodType;
+
+        this.interfaceMethodName = interfaceMethod.javaMethod.getName();
+        this.factoryMethodType = MethodType.methodType(expected,
+                delegateMethodType.dropParameterTypes(numCaptures, delegateMethodType.parameterCount()));
+        this.interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
+
+        this.delegateClassName = CLASS_NAME;
+        this.isDelegateInterface = false;
+        this.delegateInvokeType = H_INVOKESTATIC;
+
+        this.delegateMethodName = delegateMethod.name;
+        this.delegateMethodType = delegateMethodType.dropParameterTypes(0, numCaptures);
+
+        this.interfaceMethod = interfaceMethod;
+        this.delegateTypeParameters = delegateMethod.typeParameters;
+        this.delegateReturnType = delegateMethod.returnType;
+
+        this.factoryDescriptor = factoryMethodType.toMethodDescriptorString();
+        this.interfaceType = Type.getMethodType(interfaceMethodType.toMethodDescriptorString());
+        this.delegateType = Type.getMethodType(this.delegateMethodType.toMethodDescriptorString());
     }
 
     /**
@@ -181,24 +204,24 @@ public class FunctionRef {
      */
     public FunctionRef(Class<?> expected,
                        PainlessMethod interfaceMethod, String delegateMethodName, MethodType delegateMethodType, int numCaptures) {
-        interfaceMethodName = interfaceMethod.name;
+        this.interfaceMethodName = interfaceMethod.javaMethod.getName();
-        factoryMethodType = MethodType.methodType(expected,
+        this.factoryMethodType = MethodType.methodType(expected,
             delegateMethodType.dropParameterTypes(numCaptures, delegateMethodType.parameterCount()));
-        interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
+        this.interfaceMethodType = interfaceMethod.methodType.dropParameterTypes(0, 1);
 
-        delegateClassName = CLASS_NAME;
+        this.delegateClassName = CLASS_NAME;
-        delegateInvokeType = H_INVOKESTATIC;
+        this.delegateInvokeType = H_INVOKESTATIC;
         this.delegateMethodName = delegateMethodName;
         this.delegateMethodType = delegateMethodType.dropParameterTypes(0, numCaptures);
-        isDelegateInterface = false;
+        this.isDelegateInterface = false;
 
         this.interfaceMethod = null;
-        delegateTypeParameters = null;
+        this.delegateTypeParameters = null;
-        delegateReturnType = null;
+        this.delegateReturnType = null;
 
-        factoryDescriptor = null;
+        this.factoryDescriptor = null;
-        interfaceType = null;
+        this.interfaceType = null;
-        delegateType = null;
+        this.delegateType = null;
     }
 
     /**
@@ -207,15 +230,15 @@ public class FunctionRef {
     private static PainlessConstructor lookup(PainlessLookup painlessLookup, Class<?> expected, String type) {
         // check its really a functional interface
         // for e.g. Comparable
-        PainlessMethod method = painlessLookup.getPainlessStructFromJavaClass(expected).functionalMethod;
+        PainlessMethod method = painlessLookup.lookupPainlessClass(expected).functionalMethod;
         if (method == null) {
             throw new IllegalArgumentException("Cannot convert function reference [" + type + "::new] " +
                     "to [" + PainlessLookupUtility.typeToCanonicalTypeName(expected) + "], not a functional interface");
         }
 
         // lookup requested constructor
-        PainlessClass struct = painlessLookup.getPainlessStructFromJavaClass(painlessLookup.getJavaClassFromPainlessType(type));
+        PainlessClass struct = painlessLookup.lookupPainlessClass(painlessLookup.canonicalTypeNameToType(type));
-        PainlessConstructor impl = struct.constructors.get(PainlessLookupUtility.buildPainlessConstructorKey(method.arguments.size()));
+        PainlessConstructor impl = struct.constructors.get(PainlessLookupUtility.buildPainlessConstructorKey(method.typeParameters.size()));
 
         if (impl == null) {
             throw new IllegalArgumentException("Unknown reference [" + type + "::new] matching [" + expected + "]");
@@ -231,27 +254,27 @@ public class FunctionRef {
                                          String type, String call, boolean receiverCaptured) {
         // check its really a functional interface
         // for e.g. Comparable
-        PainlessMethod method = painlessLookup.getPainlessStructFromJavaClass(expected).functionalMethod;
+        PainlessMethod method = painlessLookup.lookupPainlessClass(expected).functionalMethod;
         if (method == null) {
             throw new IllegalArgumentException("Cannot convert function reference [" + type + "::" + call + "] " +
                     "to [" + PainlessLookupUtility.typeToCanonicalTypeName(expected) + "], not a functional interface");
         }
 
         // lookup requested method
-        PainlessClass struct = painlessLookup.getPainlessStructFromJavaClass(painlessLookup.getJavaClassFromPainlessType(type));
+        PainlessClass struct = painlessLookup.lookupPainlessClass(painlessLookup.canonicalTypeNameToType(type));
         final PainlessMethod impl;
         // look for a static impl first
         PainlessMethod staticImpl =
-                struct.staticMethods.get(PainlessLookupUtility.buildPainlessMethodKey(call, method.arguments.size()));
+                struct.staticMethods.get(PainlessLookupUtility.buildPainlessMethodKey(call, method.typeParameters.size()));
         if (staticImpl == null) {
             // otherwise a virtual impl
             final int arity;
             if (receiverCaptured) {
                 // receiver captured
-                arity = method.arguments.size();
+                arity = method.typeParameters.size();
             } else {
                 // receiver passed
-                arity = method.arguments.size() - 1;
+                arity = method.typeParameters.size() - 1;
             }
             impl = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey(call, arity));
         } else {

modules/lang-painless/src/main/java/org/elasticsearch/painless/LambdaBootstrap.java

@@ -36,7 +36,6 @@ import java.security.AccessController;
 import java.security.PrivilegedAction;
 
 import static java.lang.invoke.MethodHandles.Lookup;
-import static org.elasticsearch.painless.Compiler.Loader;
 import static org.elasticsearch.painless.WriterConstants.CLASS_VERSION;
 import static org.elasticsearch.painless.WriterConstants.CTOR_METHOD_NAME;
 import static org.elasticsearch.painless.WriterConstants.DELEGATE_BOOTSTRAP_HANDLE;
@@ -207,7 +206,7 @@ public final class LambdaBootstrap {
             MethodType delegateMethodType,
             int isDelegateInterface)
             throws LambdaConversionException {
-        Loader loader = (Loader)lookup.lookupClass().getClassLoader();
+        Compiler.Loader loader = (Compiler.Loader)lookup.lookupClass().getClassLoader();
         String lambdaClassName = Type.getInternalName(lookup.lookupClass()) + "$$Lambda" + loader.newLambdaIdentifier();
         Type lambdaClassType = Type.getObjectType(lambdaClassName);
         Type delegateClassType = Type.getObjectType(delegateClassName.replace('.', '/'));
@@ -457,11 +456,11 @@ public final class LambdaBootstrap {
     }
 
     /**
-     * Defines the {@link Class} for the lambda class using the same {@link Loader}
+     * Defines the {@link Class} for the lambda class using the same {@link Compiler.Loader}
      * that originally defined the class for the Painless script.
      */
     private static Class<?> createLambdaClass(
-            Loader loader,
+            Compiler.Loader loader,
             ClassWriter cw,
             Type lambdaClassType) {
 

modules/lang-painless/src/main/java/org/elasticsearch/painless/Locals.java

@@ -22,8 +22,8 @@ package org.elasticsearch.painless;
 import org.elasticsearch.painless.ScriptClassInfo.MethodArgument;
 import org.elasticsearch.painless.lookup.PainlessLookup;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
-import org.elasticsearch.painless.lookup.PainlessMethod;
 
+import java.lang.invoke.MethodType;
 import java.util.Arrays;
 import java.util.Collection;
 import java.util.Collections;
@@ -38,6 +38,30 @@ import java.util.Set;
  */
 public final class Locals {
 
+    /**
+     * Constructs a local method key used to lookup local methods from a painless class.
+     */
+    public static String buildLocalMethodKey(String methodName, int methodArity) {
+        return methodName + "/" + methodArity;
+    }
+
+    /**
+     * Stores information about methods directly callable on the generated script class.
+     */
+    public static class LocalMethod {
+        public final String name;
+        public final Class<?> returnType;
+        public final List<Class<?>> typeParameters;
+        public final MethodType methodType;
+
+        public LocalMethod(String name, Class<?> returnType, List<Class<?>> typeParameters, MethodType methodType) {
+            this.name = name;
+            this.returnType = returnType;
+            this.typeParameters = typeParameters;
+            this.methodType = methodType;
+        }
+    }
+
     /** Reserved word: loop counter */
     public static final String LOOP   = "#loop";
     /** Reserved word: unused */
@@ -110,9 +134,9 @@ public final class Locals {
     }
 
     /** Creates a new program scope: the list of methods. It is the parent for all methods */
-    public static Locals newProgramScope(PainlessLookup painlessLookup, Collection<PainlessMethod> methods) {
+    public static Locals newProgramScope(PainlessLookup painlessLookup, Collection<LocalMethod> methods) {
         Locals locals = new Locals(null, painlessLookup, null, null);
-        for (PainlessMethod method : methods) {
+        for (LocalMethod method : methods) {
             locals.addMethod(method);
         }
         return locals;
@@ -143,8 +167,8 @@ public final class Locals {
     }
 
     /** Looks up a method. Returns null if the method does not exist. */
-    public PainlessMethod getMethod(String key) {
+    public LocalMethod getMethod(String key) {
-        PainlessMethod method = lookupMethod(key);
+        LocalMethod method = lookupMethod(key);
         if (method != null) {
             return method;
         }
@@ -199,7 +223,7 @@ public final class Locals {
     // variable name -> variable
     private Map<String,Variable> variables;
     // method name+arity -> methods
-    private Map<String,PainlessMethod> methods;
+    private Map<String,LocalMethod> methods;
 
     /**
      * Create a new Locals
@@ -237,7 +261,7 @@ public final class Locals {
     }
 
     /** Looks up a method at this scope only. Returns null if the method does not exist. */
-    private PainlessMethod lookupMethod(String key) {
+    private LocalMethod lookupMethod(String key) {
         if (methods == null) {
             return null;
         }
@@ -256,11 +280,11 @@ public final class Locals {
         return variable;
     }
 
-    private void addMethod(PainlessMethod method) {
+    private void addMethod(LocalMethod method) {
         if (methods == null) {
             methods = new HashMap<>();
         }
-        methods.put(PainlessLookupUtility.buildPainlessMethodKey(method.name, method.arguments.size()), method);
+        methods.put(buildLocalMethodKey(method.name, method.typeParameters.size()), method);
         // TODO: check result
     }
 

modules/lang-painless/src/main/java/org/elasticsearch/painless/MethodWriter.java

@@ -20,6 +20,7 @@
 package org.elasticsearch.painless;
 
 import org.elasticsearch.painless.lookup.PainlessCast;
+import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.lookup.def;
 import org.objectweb.asm.ClassVisitor;
 import org.objectweb.asm.Label;
@@ -28,6 +29,7 @@ import org.objectweb.asm.Type;
 import org.objectweb.asm.commons.GeneratorAdapter;
 import org.objectweb.asm.commons.Method;
 
+import java.lang.reflect.Modifier;
 import java.util.ArrayDeque;
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -415,4 +417,26 @@ public final class MethodWriter extends GeneratorAdapter {
         System.arraycopy(params, 0, args, 2, params.length);
         invokeDynamic(name, methodType.getDescriptor(), DEF_BOOTSTRAP_HANDLE, args);
     }
+
+    public void invokeMethodCall(PainlessMethod painlessMethod) {
+        Type type = Type.getType(painlessMethod.javaMethod.getDeclaringClass());
+        Method method = Method.getMethod(painlessMethod.javaMethod);
+
+        if (Modifier.isStatic(painlessMethod.javaMethod.getModifiers())) {
+            // invokeStatic assumes that the owner class is not an interface, so this is a
+            // special case for interfaces where the interface method boolean needs to be set to
+            // true to reference the appropriate class constant when calling a static interface
+            // method since java 8 did not check, but java 9 and 10 do
+            if (painlessMethod.javaMethod.getDeclaringClass().isInterface()) {
+                visitMethodInsn(Opcodes.INVOKESTATIC, type.getInternalName(),
+                        painlessMethod.javaMethod.getName(), painlessMethod.methodType.toMethodDescriptorString(), true);
+            } else {
+                invokeStatic(type, method);
+            }
+        } else if (painlessMethod.javaMethod.getDeclaringClass().isInterface()) {
+            invokeInterface(type, method);
+        } else {
+            invokeVirtual(type, method);
+        }
+    }
 }

modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessExplainError.java

@@ -57,7 +57,7 @@ public class PainlessExplainError extends Error {
         if (objectToExplain != null) {
             toString = objectToExplain.toString();
             javaClassName = objectToExplain.getClass().getName();
-            PainlessClass struct = painlessLookup.getPainlessStructFromJavaClass(objectToExplain.getClass());
+            PainlessClass struct = painlessLookup.lookupPainlessClass(objectToExplain.getClass());
             if (struct != null) {
                 painlessClassName = PainlessLookupUtility.typeToCanonicalTypeName(objectToExplain.getClass());
             }

modules/lang-painless/src/main/java/org/elasticsearch/painless/PainlessScriptEngine.java

@@ -102,10 +102,10 @@ public final class PainlessScriptEngine extends AbstractComponent implements Scr
         for (Map.Entry<ScriptContext<?>, List<Whitelist>> entry : contexts.entrySet()) {
             ScriptContext<?> context = entry.getKey();
             if (context.instanceClazz.equals(SearchScript.class) || context.instanceClazz.equals(ExecutableScript.class)) {
-                contextsToCompilers.put(context, new Compiler(GenericElasticsearchScript.class,
+                contextsToCompilers.put(context, new Compiler(GenericElasticsearchScript.class, null, null,
                         PainlessLookupBuilder.buildFromWhitelists(entry.getValue())));
             } else {
-                contextsToCompilers.put(context, new Compiler(context.instanceClazz,
+                contextsToCompilers.put(context, new Compiler(context.instanceClazz, context.factoryClazz, context.statefulFactoryClazz,
                         PainlessLookupBuilder.buildFromWhitelists(entry.getValue())));
             }
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/ScriptClassInfo.java

@@ -190,7 +190,7 @@ public class ScriptClassInfo {
             componentType = componentType.getComponentType();
         }
 
-        if (painlessLookup.getPainlessStructFromJavaClass(componentType) == null) {
+        if (painlessLookup.lookupPainlessClass(componentType) == null) {
             throw new IllegalArgumentException(unknownErrorMessageSource.apply(componentType));
         }
 

modules/lang-painless/src/main/java/org/elasticsearch/painless/antlr/EnhancedPainlessLexer.java

@@ -75,7 +75,7 @@ final class EnhancedPainlessLexer extends PainlessLexer {
 
     @Override
     protected boolean isType(String name) {
-        return painlessLookup.isSimplePainlessType(name);
+        return painlessLookup.isValidCanonicalClassName(name);
     }
 
     @Override

modules/lang-painless/src/main/java/org/elasticsearch/painless/lookup/PainlessField.java

@@ -20,24 +20,20 @@
 package org.elasticsearch.painless.lookup;
 
 import java.lang.invoke.MethodHandle;
+import java.lang.reflect.Field;
 
 public final class PainlessField {
-    public final String name;
+    public final Field javaField;
-    public final Class<?> target;
+    public final Class<?> typeParameter;
-    public final Class<?> clazz;
-    public final String javaName;
-    public final int modifiers;
-    public final MethodHandle getter;
-    public final MethodHandle setter;
 
-    PainlessField(String name, String javaName, Class<?> target, Class<?> clazz, int modifiers,
+    public final MethodHandle getterMethodHandle;
-                  MethodHandle getter, MethodHandle setter) {
+    public final MethodHandle setterMethodHandle;
-        this.name = name;
+
-        this.javaName = javaName;
+    PainlessField(Field javaField, Class<?> typeParameter, MethodHandle getterMethodHandle, MethodHandle setterMethodHandle) {
-        this.target = target;
+        this.javaField = javaField;
-        this.clazz = clazz;
+        this.typeParameter = typeParameter;
-        this.modifiers = modifiers;
+
-        this.getter = getter;
+        this.getterMethodHandle = getterMethodHandle;
-        this.setter = setter;
+        this.setterMethodHandle = setterMethodHandle;
     }
 }

modules/lang-painless/src/main/java/org/elasticsearch/painless/lookup/PainlessLookup.java

@@ -19,41 +19,119 @@
 
 package org.elasticsearch.painless.lookup;
 
-import java.util.Collection;
 import java.util.Collections;
 import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+import static org.elasticsearch.painless.lookup.PainlessLookupUtility.buildPainlessConstructorKey;
+import static org.elasticsearch.painless.lookup.PainlessLookupUtility.buildPainlessFieldKey;
+import static org.elasticsearch.painless.lookup.PainlessLookupUtility.buildPainlessMethodKey;
+import static org.elasticsearch.painless.lookup.PainlessLookupUtility.typeToCanonicalTypeName;
 
-/**
- * The entire API for Painless.  Also used as a whitelist for checking for legal
- * methods and fields during at both compile-time and runtime.
- */
 public final class PainlessLookup {
 
-    public Collection<Class<?>> getStructs() {
-        return classesToPainlessClasses.keySet();
-    }
-
     private final Map<String, Class<?>> canonicalClassNamesToClasses;
     private final Map<Class<?>, PainlessClass> classesToPainlessClasses;
 
     PainlessLookup(Map<String, Class<?>> canonicalClassNamesToClasses, Map<Class<?>, PainlessClass> classesToPainlessClasses) {
+        Objects.requireNonNull(canonicalClassNamesToClasses);
+        Objects.requireNonNull(classesToPainlessClasses);
+
         this.canonicalClassNamesToClasses = Collections.unmodifiableMap(canonicalClassNamesToClasses);
         this.classesToPainlessClasses = Collections.unmodifiableMap(classesToPainlessClasses);
     }
 
-    public Class<?> getClassFromBinaryName(String painlessType) {
+    public boolean isValidCanonicalClassName(String canonicalClassName) {
-        return canonicalClassNamesToClasses.get(painlessType.replace('$', '.'));
+        Objects.requireNonNull(canonicalClassName);
+
+        return canonicalClassNamesToClasses.containsKey(canonicalClassName);
     }
 
-    public boolean isSimplePainlessType(String painlessType) {
+    public Class<?> canonicalTypeNameToType(String painlessType) {
-        return canonicalClassNamesToClasses.containsKey(painlessType);
+        Objects.requireNonNull(painlessType);
-    }
 
-    public PainlessClass getPainlessStructFromJavaClass(Class<?> clazz) {
-        return classesToPainlessClasses.get(clazz);
-    }
-
-    public Class<?> getJavaClassFromPainlessType(String painlessType) {
         return PainlessLookupUtility.canonicalTypeNameToType(painlessType, canonicalClassNamesToClasses);
     }
+
+    public Set<Class<?>> getClasses() {
+        return classesToPainlessClasses.keySet();
+    }
+
+    public PainlessClass lookupPainlessClass(Class<?> targetClass) {
+        return classesToPainlessClasses.get(targetClass);
+    }
+
+    public PainlessConstructor lookupPainlessConstructor(Class<?> targetClass, int constructorArity) {
+        Objects.requireNonNull(targetClass);
+
+        PainlessClass targetPainlessClass = classesToPainlessClasses.get(targetClass);
+        String painlessConstructorKey = buildPainlessConstructorKey(constructorArity);
+
+        if (targetPainlessClass == null) {
+            throw new IllegalArgumentException("target class [" + typeToCanonicalTypeName(targetClass) + "] " +
+                    "not found for constructor [" + painlessConstructorKey + "]");
+        }
+
+        PainlessConstructor painlessConstructor = targetPainlessClass.constructors.get(painlessConstructorKey);
+
+        if (painlessConstructor == null) {
+            throw new IllegalArgumentException(
+                    "constructor [" + typeToCanonicalTypeName(targetClass) + ", " + painlessConstructorKey + "] not found");
+        }
+
+        return painlessConstructor;
+    }
+
+    public PainlessMethod lookupPainlessMethod(Class<?> targetClass, boolean isStatic, String methodName, int methodArity) {
+        Objects.requireNonNull(targetClass);
+        Objects.requireNonNull(methodName);
+
+        if (targetClass.isPrimitive()) {
+            targetClass = PainlessLookupUtility.typeToBoxedType(targetClass);
+        }
+
+        PainlessClass targetPainlessClass = classesToPainlessClasses.get(targetClass);
+        String painlessMethodKey = buildPainlessMethodKey(methodName, methodArity);
+
+        if (targetPainlessClass == null) {
+            throw new IllegalArgumentException(
+                    "target class [" + typeToCanonicalTypeName(targetClass) + "] not found for method [" + painlessMethodKey + "]");
+        }
+
+        PainlessMethod painlessMethod = isStatic ?
+                targetPainlessClass.staticMethods.get(painlessMethodKey) :
+                targetPainlessClass.methods.get(painlessMethodKey);
+
+        if (painlessMethod == null) {
+            throw new IllegalArgumentException(
+                    "method [" + typeToCanonicalTypeName(targetClass) + ", " + painlessMethodKey + "] not found");
+        }
+
+        return painlessMethod;
+    }
+
+    public PainlessField lookupPainlessField(Class<?> targetClass, boolean isStatic, String fieldName) {
+        Objects.requireNonNull(targetClass);
+        Objects.requireNonNull(fieldName);
+
+        PainlessClass targetPainlessClass = classesToPainlessClasses.get(targetClass);
+        String painlessFieldKey = buildPainlessFieldKey(fieldName);
+
+        if (targetPainlessClass == null) {
+            throw new IllegalArgumentException(
+                    "target class [" + typeToCanonicalTypeName(targetClass) + "] not found for field [" + painlessFieldKey + "]");
+        }
+
+        PainlessField painlessField = isStatic ?
+                targetPainlessClass.staticFields.get(painlessFieldKey) :
+                targetPainlessClass.fields.get(painlessFieldKey);
+
+        if (painlessField == null) {
+            throw new IllegalArgumentException(
+                    "field [" + typeToCanonicalTypeName(targetClass) + ", " + painlessFieldKey + "] not found");
+        }
+
+        return painlessField;
+    }
 }

modules/lang-painless/src/main/java/org/elasticsearch/painless/lookup/PainlessLookupBuilder.java

@@ -540,7 +540,6 @@ public final class PainlessLookupBuilder {
             PainlessMethod painlessMethod = painlessClassBuilder.staticMethods.get(painlessMethodKey);
 
             if (painlessMethod == null) {
-                org.objectweb.asm.commons.Method asmMethod = org.objectweb.asm.commons.Method.getMethod(javaMethod);
                 MethodHandle methodHandle;
 
                 try {
@@ -554,19 +553,17 @@ public final class PainlessLookupBuilder {
 
                 painlessMethod = painlessMethodCache.computeIfAbsent(
                         new PainlessMethodCacheKey(targetClass, methodName, typeParameters),
-                        key -> new PainlessMethod(methodName, targetClass, null, returnType,
+                        key -> new PainlessMethod(javaMethod, targetClass, returnType, typeParameters, methodHandle, methodType));
-                                typeParameters, asmMethod, javaMethod.getModifiers(), methodHandle, methodType));
 
                 painlessClassBuilder.staticMethods.put(painlessMethodKey, painlessMethod);
-            } else if ((painlessMethod.name.equals(methodName) && painlessMethod.rtn == returnType &&
+            } else if (painlessMethod.returnType == returnType && painlessMethod.typeParameters.equals(typeParameters) == false) {
-                    painlessMethod.arguments.equals(typeParameters)) == false) {
                 throw new IllegalArgumentException("cannot have static methods " +
                         "[[" + targetCanonicalClassName + "], [" + methodName + "], " +
                         "[" + typeToCanonicalTypeName(returnType) + "], " +
                         typesToCanonicalTypeNames(typeParameters) + "] and " +
                         "[[" + targetCanonicalClassName + "], [" + methodName + "], " +
-                        "[" + typeToCanonicalTypeName(painlessMethod.rtn) + "], " +
+                        "[" + typeToCanonicalTypeName(painlessMethod.returnType) + "], " +
-                        typesToCanonicalTypeNames(painlessMethod.arguments) + "] " +
+                        typesToCanonicalTypeNames(painlessMethod.typeParameters) + "] " +
                         "with the same arity and different return type or type parameters");
             }
         } else {
@@ -597,19 +594,17 @@ public final class PainlessLookupBuilder {
 
                 painlessMethod = painlessMethodCache.computeIfAbsent(
                         new PainlessMethodCacheKey(targetClass, methodName, typeParameters),
-                        key -> new PainlessMethod(methodName, targetClass, augmentedClass, returnType,
+                        key -> new PainlessMethod(javaMethod, targetClass, returnType, typeParameters, methodHandle, methodType));
-                                typeParameters, asmMethod, javaMethod.getModifiers(), methodHandle, methodType));
 
                 painlessClassBuilder.methods.put(painlessMethodKey, painlessMethod);
-            } else if ((painlessMethod.name.equals(methodName) && painlessMethod.rtn == returnType &&
+            } else if (painlessMethod.returnType == returnType && painlessMethod.typeParameters.equals(typeParameters) == false) {
-                    painlessMethod.arguments.equals(typeParameters)) == false) {
                 throw new IllegalArgumentException("cannot have methods " +
                         "[[" + targetCanonicalClassName + "], [" + methodName + "], " +
                         "[" + typeToCanonicalTypeName(returnType) + "], " +
                         typesToCanonicalTypeNames(typeParameters) + "] and " +
                         "[[" + targetCanonicalClassName + "], [" + methodName + "], " +
-                        "[" + typeToCanonicalTypeName(painlessMethod.rtn) + "], " +
+                        "[" + typeToCanonicalTypeName(painlessMethod.returnType) + "], " +
-                        typesToCanonicalTypeNames(painlessMethod.arguments) + "] " +
+                        typesToCanonicalTypeNames(painlessMethod.typeParameters) + "] " +
                         "with the same arity and different return type or type parameters");
             }
         }
@@ -684,11 +679,20 @@ public final class PainlessLookupBuilder {
                     "for field [[" + targetCanonicalClassName + "], [" + fieldName + "]");
         }
 
+        MethodHandle methodHandleGetter;
+
+        try {
+            methodHandleGetter = MethodHandles.publicLookup().unreflectGetter(javaField);
+        } catch (IllegalAccessException iae) {
+            throw new IllegalArgumentException(
+                    "getter method handle not found for field [[" + targetCanonicalClassName + "], [" + fieldName + "]]");
+        }
+
         String painlessFieldKey = buildPainlessFieldKey(fieldName);
 
         if (Modifier.isStatic(javaField.getModifiers())) {
             if (Modifier.isFinal(javaField.getModifiers()) == false) {
-                throw new IllegalArgumentException("static field [[" + targetCanonicalClassName + "]. [" + fieldName + "]] must be final");
+                throw new IllegalArgumentException("static field [[" + targetCanonicalClassName + "], [" + fieldName + "]] must be final");
             }
 
             PainlessField painlessField = painlessClassBuilder.staticFields.get(painlessFieldKey);
@@ -696,28 +700,18 @@ public final class PainlessLookupBuilder {
             if (painlessField == null) {
                 painlessField = painlessFieldCache.computeIfAbsent(
                         new PainlessFieldCacheKey(targetClass, fieldName, typeParameter),
-                        key -> new PainlessField(fieldName, javaField.getName(), targetClass,
+                        key -> new PainlessField(javaField, typeParameter, methodHandleGetter, null));
-                                typeParameter, javaField.getModifiers(), null, null));
 
                 painlessClassBuilder.staticFields.put(painlessFieldKey, painlessField);
-            } else if (painlessField.clazz != typeParameter) {
+            } else if (painlessField.typeParameter != typeParameter) {
                 throw new IllegalArgumentException("cannot have static fields " +
                         "[[" + targetCanonicalClassName + "], [" + fieldName + "], [" +
                         typeToCanonicalTypeName(typeParameter) + "] and " +
-                        "[[" + targetCanonicalClassName + "], [" + painlessField.name + "], " +
+                        "[[" + targetCanonicalClassName + "], [" + painlessField.javaField.getName() + "], " +
-                        typeToCanonicalTypeName(painlessField.clazz) + "] " +
+                        typeToCanonicalTypeName(painlessField.typeParameter) + "] " +
-                        "with the same and different type parameters");
+                        "with the same name and different type parameters");
             }
         } else {
-            MethodHandle methodHandleGetter;
-
-            try {
-                methodHandleGetter = MethodHandles.publicLookup().unreflectGetter(javaField);
-            } catch (IllegalAccessException iae) {
-                throw new IllegalArgumentException(
-                        "getter method handle not found for field [[" + targetCanonicalClassName + "], [" + fieldName + "]]");
-            }
-
             MethodHandle methodHandleSetter;
 
             try {
@@ -732,17 +726,16 @@ public final class PainlessLookupBuilder {
             if (painlessField == null) {
                 painlessField = painlessFieldCache.computeIfAbsent(
                         new PainlessFieldCacheKey(targetClass, painlessFieldKey, typeParameter),
-                        key -> new PainlessField(fieldName, javaField.getName(), targetClass,
+                        key -> new PainlessField(javaField, typeParameter, methodHandleGetter, methodHandleSetter));
-                                typeParameter, javaField.getModifiers(), methodHandleGetter, methodHandleSetter));
 
                 painlessClassBuilder.fields.put(fieldName, painlessField);
-            } else if (painlessField.clazz != typeParameter) {
+            } else if (painlessField.typeParameter != typeParameter) {
                 throw new IllegalArgumentException("cannot have fields " +
                         "[[" + targetCanonicalClassName + "], [" + fieldName + "], [" +
                         typeToCanonicalTypeName(typeParameter) + "] and " +
-                        "[[" + targetCanonicalClassName + "], [" + painlessField.name + "], " +
+                        "[[" + targetCanonicalClassName + "], [" + painlessField.javaField.getName() + "], " +
-                        typeToCanonicalTypeName(painlessField.clazz) + "] " +
+                        typeToCanonicalTypeName(painlessField.typeParameter) + "] " +
-                        "with the same and different type parameters");
+                        "with the same name and different type parameters");
             }
         }
     }
@@ -806,8 +799,8 @@ public final class PainlessLookupBuilder {
             PainlessMethod newPainlessMethod = painlessMethodEntry.getValue();
             PainlessMethod existingPainlessMethod = targetPainlessClassBuilder.methods.get(painlessMethodKey);
 
-            if (existingPainlessMethod == null || existingPainlessMethod.target != newPainlessMethod.target &&
+            if (existingPainlessMethod == null || existingPainlessMethod.targetClass != newPainlessMethod.targetClass &&
-                    existingPainlessMethod.target.isAssignableFrom(newPainlessMethod.target)) {
+                    existingPainlessMethod.targetClass.isAssignableFrom(newPainlessMethod.targetClass)) {
                 targetPainlessClassBuilder.methods.put(painlessMethodKey, newPainlessMethod);
             }
         }
@@ -817,8 +810,9 @@ public final class PainlessLookupBuilder {
             PainlessField newPainlessField = painlessFieldEntry.getValue();
             PainlessField existingPainlessField = targetPainlessClassBuilder.fields.get(painlessFieldKey);
 
-            if (existingPainlessField == null || existingPainlessField.target != newPainlessField.target &&
+            if (existingPainlessField == null ||
-                    existingPainlessField.target.isAssignableFrom(newPainlessField.target)) {
+                    existingPainlessField.javaField.getDeclaringClass() != newPainlessField.javaField.getDeclaringClass() &&
+                    existingPainlessField.javaField.getDeclaringClass().isAssignableFrom(newPainlessField.javaField.getDeclaringClass())) {
                 targetPainlessClassBuilder.fields.put(painlessFieldKey, newPainlessField);
             }
         }
@@ -832,27 +826,27 @@ public final class PainlessLookupBuilder {
 
     private void cacheRuntimeHandles(PainlessClassBuilder painlessClassBuilder) {
         for (PainlessMethod painlessMethod : painlessClassBuilder.methods.values()) {
-            String methodName = painlessMethod.name;
+            String methodName = painlessMethod.javaMethod.getName();
-            int typeParametersSize = painlessMethod.arguments.size();
+            int typeParametersSize = painlessMethod.typeParameters.size();
 
             if (typeParametersSize == 0 && methodName.startsWith("get") && methodName.length() > 3 &&
                     Character.isUpperCase(methodName.charAt(3))) {
                 painlessClassBuilder.getterMethodHandles.putIfAbsent(
-                        Character.toLowerCase(methodName.charAt(3)) + methodName.substring(4), painlessMethod.handle);
+                        Character.toLowerCase(methodName.charAt(3)) + methodName.substring(4), painlessMethod.methodHandle);
             } else if (typeParametersSize == 0 && methodName.startsWith("is") && methodName.length() > 2 &&
                     Character.isUpperCase(methodName.charAt(2))) {
                 painlessClassBuilder.getterMethodHandles.putIfAbsent(
-                        Character.toLowerCase(methodName.charAt(2)) + methodName.substring(3), painlessMethod.handle);
+                        Character.toLowerCase(methodName.charAt(2)) + methodName.substring(3), painlessMethod.methodHandle);
             } else if (typeParametersSize == 1 && methodName.startsWith("set") && methodName.length() > 3 &&
                     Character.isUpperCase(methodName.charAt(3))) {
                 painlessClassBuilder.setterMethodHandles.putIfAbsent(
-                        Character.toLowerCase(methodName.charAt(3)) + methodName.substring(4), painlessMethod.handle);
+                        Character.toLowerCase(methodName.charAt(3)) + methodName.substring(4), painlessMethod.methodHandle);
             }
         }
 
         for (PainlessField painlessField : painlessClassBuilder.fields.values()) {
-            painlessClassBuilder.getterMethodHandles.put(painlessField.name, painlessField.getter);
+            painlessClassBuilder.getterMethodHandles.put(painlessField.javaField.getName(), painlessField.getterMethodHandle);
-            painlessClassBuilder.setterMethodHandles.put(painlessField.name, painlessField.setter);
+            painlessClassBuilder.setterMethodHandles.put(painlessField.javaField.getName(), painlessField.setterMethodHandle);
         }
     }
 

modules/lang-painless/src/main/java/org/elasticsearch/painless/lookup/PainlessMethod.java

@@ -19,67 +19,28 @@
 
 package org.elasticsearch.painless.lookup;
 
-import org.elasticsearch.painless.MethodWriter;
-import org.objectweb.asm.Opcodes;
-import org.objectweb.asm.Type;
-
 import java.lang.invoke.MethodHandle;
 import java.lang.invoke.MethodType;
-import java.lang.reflect.Modifier;
+import java.lang.reflect.Method;
 import java.util.Collections;
 import java.util.List;
 
 public class PainlessMethod {
-    public final String name;
+    public final Method javaMethod;
-    public final Class<?> target;
+    public final Class<?> targetClass;
-    public final Class<?> augmentation;
+    public final Class<?> returnType;
-    public final Class<?> rtn;
+    public final List<Class<?>> typeParameters;
-    public final List<Class<?>> arguments;
+    public final MethodHandle methodHandle;
-    public final org.objectweb.asm.commons.Method method;
-    public final int modifiers;
-    public final MethodHandle handle;
     public final MethodType methodType;
 
-    public PainlessMethod(String name, Class<?> target, Class<?> augmentation, Class<?> rtn, List<Class<?>> arguments,
+    public PainlessMethod(Method javaMethod, Class<?> targetClass, Class<?> returnType, List<Class<?>> typeParameters,
-                          org.objectweb.asm.commons.Method method, int modifiers, MethodHandle handle, MethodType methodType) {
+            MethodHandle methodHandle, MethodType methodType) {
-        this.name = name;
+
-        this.augmentation = augmentation;
+        this.javaMethod = javaMethod;
-        this.target = target;
+        this.targetClass = targetClass;
-        this.rtn = rtn;
+        this.returnType = returnType;
-        this.arguments = Collections.unmodifiableList(arguments);
+        this.typeParameters = Collections.unmodifiableList(typeParameters);
-        this.method = method;
+        this.methodHandle = methodHandle;
-        this.modifiers = modifiers;
-        this.handle = handle;
         this.methodType = methodType;
     }
-
-    public void write(MethodWriter writer) {
-        final org.objectweb.asm.Type type;
-        final Class<?> clazz;
-        if (augmentation != null) {
-            assert Modifier.isStatic(modifiers);
-            clazz = augmentation;
-            type = org.objectweb.asm.Type.getType(augmentation);
-        } else {
-            clazz = target;
-            type = Type.getType(target);
-        }
-
-        if (Modifier.isStatic(modifiers)) {
-            // invokeStatic assumes that the owner class is not an interface, so this is a
-            // special case for interfaces where the interface method boolean needs to be set to
-            // true to reference the appropriate class constant when calling a static interface
-            // method since java 8 did not check, but java 9 and 10 do
-            if (Modifier.isInterface(clazz.getModifiers())) {
-                writer.visitMethodInsn(Opcodes.INVOKESTATIC,
-                        type.getInternalName(), name, methodType.toMethodDescriptorString(), true);
-            } else {
-                writer.invokeStatic(type, method);
-            }
-        } else if (Modifier.isInterface(clazz.getModifiers())) {
-            writer.invokeInterface(type, method);
-        } else {
-            writer.invokeVirtual(type, method);
-        }
-    }
 }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ECallLocal.java

@@ -21,10 +21,11 @@ package org.elasticsearch.painless.node;
 
 import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
+import org.elasticsearch.painless.Locals.LocalMethod;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
-import org.elasticsearch.painless.lookup.PainlessMethod;
+import org.objectweb.asm.commons.Method;
 
 import java.util.List;
 import java.util.Objects;
@@ -40,7 +41,7 @@ public final class ECallLocal extends AExpression {
     private final String name;
     private final List<AExpression> arguments;
 
-    private PainlessMethod method = null;
+    private LocalMethod method = null;
 
     public ECallLocal(Location location, String name, List<AExpression> arguments) {
         super(location);
@@ -68,14 +69,14 @@ public final class ECallLocal extends AExpression {
         for (int argument = 0; argument < arguments.size(); ++argument) {
             AExpression expression = arguments.get(argument);
 
-            expression.expected = method.arguments.get(argument);
+            expression.expected = method.typeParameters.get(argument);
             expression.internal = true;
             expression.analyze(locals);
             arguments.set(argument, expression.cast(locals));
         }
 
         statement = true;
-        actual = method.rtn;
+        actual = method.returnType;
     }
 
     @Override
@@ -86,7 +87,7 @@ public final class ECallLocal extends AExpression {
             argument.write(writer, globals);
         }
 
-        writer.invokeStatic(CLASS_TYPE, method.method);
+        writer.invokeStatic(CLASS_TYPE, new Method(method.name, method.methodType.toMethodDescriptorString()));
     }
 
     @Override

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ECapturingFunctionRef.java

@@ -81,14 +81,14 @@ public final class ECapturingFunctionRef extends AExpression implements ILambda
                             PainlessLookupUtility.typeToCanonicalTypeName(captured.clazz), call, 1);
 
                     // check casts between the interface method and the delegate method are legal
-                    for (int i = 0; i < ref.interfaceMethod.arguments.size(); ++i) {
+                    for (int i = 0; i < ref.interfaceMethod.typeParameters.size(); ++i) {
-                        Class<?> from = ref.interfaceMethod.arguments.get(i);
+                        Class<?> from = ref.interfaceMethod.typeParameters.get(i);
                         Class<?> to = ref.delegateTypeParameters.get(i);
                         AnalyzerCaster.getLegalCast(location, from, to, false, true);
                     }
 
-                    if (ref.interfaceMethod.rtn != void.class) {
+                    if (ref.interfaceMethod.returnType != void.class) {
-                        AnalyzerCaster.getLegalCast(location, ref.delegateReturnType, ref.interfaceMethod.rtn, false, true);
+                        AnalyzerCaster.getLegalCast(location, ref.delegateReturnType, ref.interfaceMethod.returnType, false, true);
                     }
                 } catch (IllegalArgumentException e) {
                     throw createError(e);

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EExplicit.java

@@ -50,7 +50,7 @@ public final class EExplicit extends AExpression {
     @Override
     void analyze(Locals locals) {
         try {
-            actual = locals.getPainlessLookup().getJavaClassFromPainlessType(type);
+            actual = locals.getPainlessLookup().canonicalTypeNameToType(type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EFunctionRef.java

@@ -23,6 +23,7 @@ import org.elasticsearch.painless.AnalyzerCaster;
 import org.elasticsearch.painless.FunctionRef;
 import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
+import org.elasticsearch.painless.Locals.LocalMethod;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
@@ -65,13 +66,12 @@ public final class EFunctionRef extends AExpression implements ILambda {
             try {
                 if ("this".equals(type)) {
                     // user's own function
-                    PainlessMethod interfaceMethod = locals.getPainlessLookup().getPainlessStructFromJavaClass(expected).functionalMethod;
+                    PainlessMethod interfaceMethod = locals.getPainlessLookup().lookupPainlessClass(expected).functionalMethod;
                     if (interfaceMethod == null) {
                         throw new IllegalArgumentException("Cannot convert function reference [" + type + "::" + call + "] " +
                                 "to [" + PainlessLookupUtility.typeToCanonicalTypeName(expected) + "], not a functional interface");
                     }
-                    PainlessMethod delegateMethod =
+                    LocalMethod delegateMethod = locals.getMethod(Locals.buildLocalMethodKey(call, interfaceMethod.typeParameters.size()));
-                            locals.getMethod(PainlessLookupUtility.buildPainlessMethodKey(call, interfaceMethod.arguments.size()));
                     if (delegateMethod == null) {
                         throw new IllegalArgumentException("Cannot convert function reference [" + type + "::" + call + "] " +
                                 "to [" + PainlessLookupUtility.typeToCanonicalTypeName(expected) + "], function not found");
@@ -79,14 +79,14 @@ public final class EFunctionRef extends AExpression implements ILambda {
                     ref = new FunctionRef(expected, interfaceMethod, delegateMethod, 0);
 
                     // check casts between the interface method and the delegate method are legal
-                    for (int i = 0; i < interfaceMethod.arguments.size(); ++i) {
+                    for (int i = 0; i < interfaceMethod.typeParameters.size(); ++i) {
-                        Class<?> from = interfaceMethod.arguments.get(i);
+                        Class<?> from = interfaceMethod.typeParameters.get(i);
-                        Class<?> to = delegateMethod.arguments.get(i);
+                        Class<?> to = delegateMethod.typeParameters.get(i);
                         AnalyzerCaster.getLegalCast(location, from, to, false, true);
                     }
 
-                    if (interfaceMethod.rtn != void.class) {
+                    if (interfaceMethod.returnType != void.class) {
-                        AnalyzerCaster.getLegalCast(location, delegateMethod.rtn, interfaceMethod.rtn, false, true);
+                        AnalyzerCaster.getLegalCast(location, delegateMethod.returnType, interfaceMethod.returnType, false, true);
                     }
                 } else {
                     // whitelist lookup

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EInstanceof.java

@@ -58,7 +58,7 @@ public final class EInstanceof extends AExpression {
 
         // ensure the specified type is part of the definition
         try {
-            clazz = locals.getPainlessLookup().getJavaClassFromPainlessType(this.type);
+            clazz = locals.getPainlessLookup().canonicalTypeNameToType(this.type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + this.type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ELambda.java

@@ -23,6 +23,7 @@ import org.elasticsearch.painless.AnalyzerCaster;
 import org.elasticsearch.painless.FunctionRef;
 import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
+import org.elasticsearch.painless.Locals.LocalMethod;
 import org.elasticsearch.painless.Locals.Variable;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
@@ -120,27 +121,27 @@ public final class ELambda extends AExpression implements ILambda {
             }
         } else {
             // we know the method statically, infer return type and any unknown/def types
-            interfaceMethod = locals.getPainlessLookup().getPainlessStructFromJavaClass(expected).functionalMethod;
+            interfaceMethod = locals.getPainlessLookup().lookupPainlessClass(expected).functionalMethod;
             if (interfaceMethod == null) {
                 throw createError(new IllegalArgumentException("Cannot pass lambda to " +
                         "[" + PainlessLookupUtility.typeToCanonicalTypeName(expected) + "], not a functional interface"));
             }
             // check arity before we manipulate parameters
-            if (interfaceMethod.arguments.size() != paramTypeStrs.size())
+            if (interfaceMethod.typeParameters.size() != paramTypeStrs.size())
-                throw new IllegalArgumentException("Incorrect number of parameters for [" + interfaceMethod.name +
+                throw new IllegalArgumentException("Incorrect number of parameters for [" + interfaceMethod.javaMethod.getName() +
                         "] in [" + PainlessLookupUtility.typeToCanonicalTypeName(expected) + "]");
             // for method invocation, its allowed to ignore the return value
-            if (interfaceMethod.rtn == void.class) {
+            if (interfaceMethod.returnType == void.class) {
                 returnType = def.class;
             } else {
-                returnType = interfaceMethod.rtn;
+                returnType = interfaceMethod.returnType;
             }
             // replace any null types with the actual type
             actualParamTypeStrs = new ArrayList<>(paramTypeStrs.size());
             for (int i = 0; i < paramTypeStrs.size(); i++) {
                 String paramType = paramTypeStrs.get(i);
                 if (paramType == null) {
-                    actualParamTypeStrs.add(PainlessLookupUtility.typeToCanonicalTypeName(interfaceMethod.arguments.get(i)));
+                    actualParamTypeStrs.add(PainlessLookupUtility.typeToCanonicalTypeName(interfaceMethod.typeParameters.get(i)));
                 } else {
                     actualParamTypeStrs.add(paramType);
                 }
@@ -183,20 +184,22 @@ public final class ELambda extends AExpression implements ILambda {
         } else {
             defPointer = null;
             try {
-                ref = new FunctionRef(expected, interfaceMethod, desugared.method, captures.size());
+                LocalMethod localMethod =
+                        new LocalMethod(desugared.name, desugared.returnType, desugared.typeParameters, desugared.methodType);
+                ref = new FunctionRef(expected, interfaceMethod, localMethod, captures.size());
             } catch (IllegalArgumentException e) {
                 throw createError(e);
             }
 
             // check casts between the interface method and the delegate method are legal
-            for (int i = 0; i < interfaceMethod.arguments.size(); ++i) {
+            for (int i = 0; i < interfaceMethod.typeParameters.size(); ++i) {
-                Class<?> from = interfaceMethod.arguments.get(i);
+                Class<?> from = interfaceMethod.typeParameters.get(i);
                 Class<?> to = desugared.parameters.get(i + captures.size()).clazz;
                 AnalyzerCaster.getLegalCast(location, from, to, false, true);
             }
 
-            if (interfaceMethod.rtn != void.class) {
+            if (interfaceMethod.returnType != void.class) {
-                AnalyzerCaster.getLegalCast(location, desugared.rtnType, interfaceMethod.rtn, false, true);
+                AnalyzerCaster.getLegalCast(location, desugared.returnType, interfaceMethod.returnType, false, true);
             }
 
             actual = expected;

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EListInit.java

@@ -24,7 +24,6 @@ import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
 import org.elasticsearch.painless.lookup.PainlessConstructor;
-import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.lookup.def;
 import org.objectweb.asm.Type;
@@ -64,18 +63,16 @@ public final class EListInit extends AExpression {
 
         actual = ArrayList.class;
 
-        constructor = locals.getPainlessLookup().getPainlessStructFromJavaClass(actual).constructors.get(
+        try {
-                PainlessLookupUtility.buildPainlessConstructorKey(0));
+            constructor = locals.getPainlessLookup().lookupPainlessConstructor(actual, 0);
-
+        } catch (IllegalArgumentException iae) {
-        if (constructor == null) {
+            throw createError(iae);
-            throw createError(new IllegalStateException("Illegal tree structure."));
         }
 
-        method = locals.getPainlessLookup().getPainlessStructFromJavaClass(actual).methods
+        try {
-                .get(PainlessLookupUtility.buildPainlessMethodKey("add", 1));
+            method = locals.getPainlessLookup().lookupPainlessMethod(actual, false, "add", 1);
-
+        } catch (IllegalArgumentException iae) {
-        if (method == null) {
+            throw createError(iae);
-            throw createError(new IllegalStateException("Illegal tree structure."));
         }
 
         for (int index = 0; index < values.size(); ++index) {
@@ -100,7 +97,7 @@ public final class EListInit extends AExpression {
         for (AExpression value : values) {
             writer.dup();
             value.write(writer, globals);
-            method.write(writer);
+            writer.invokeMethodCall(method);
             writer.pop();
         }
     }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EMapInit.java

@@ -24,7 +24,6 @@ import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
 import org.elasticsearch.painless.lookup.PainlessConstructor;
-import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.lookup.def;
 import org.objectweb.asm.Type;
@@ -70,18 +69,16 @@ public final class EMapInit extends AExpression {
 
         actual = HashMap.class;
 
-        constructor = locals.getPainlessLookup().getPainlessStructFromJavaClass(actual).constructors.get(
+        try {
-                PainlessLookupUtility.buildPainlessConstructorKey(0));
+            constructor = locals.getPainlessLookup().lookupPainlessConstructor(actual, 0);
-
+        } catch (IllegalArgumentException iae) {
-        if (constructor == null) {
+            throw createError(iae);
-            throw createError(new IllegalStateException("Illegal tree structure."));
         }
 
-        method = locals.getPainlessLookup().getPainlessStructFromJavaClass(actual).methods
+        try {
-                .get(PainlessLookupUtility.buildPainlessMethodKey("put", 2));
+            method = locals.getPainlessLookup().lookupPainlessMethod(actual, false, "put", 2);
-
+        } catch (IllegalArgumentException iae) {
-        if (method == null) {
+            throw createError(iae);
-            throw createError(new IllegalStateException("Illegal tree structure."));
         }
 
         if (keys.size() != values.size()) {
@@ -123,7 +120,7 @@ public final class EMapInit extends AExpression {
             writer.dup();
             key.write(writer, globals);
             value.write(writer, globals);
-            method.write(writer);
+            writer.invokeMethodCall(method);
             writer.pop();
         }
     }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ENewArray.java

@@ -61,7 +61,7 @@ public final class ENewArray extends AExpression {
         Class<?> clazz;
 
         try {
-            clazz = locals.getPainlessLookup().getJavaClassFromPainlessType(this.type);
+            clazz = locals.getPainlessLookup().canonicalTypeNameToType(this.type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + this.type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/ENewObj.java

@@ -23,7 +23,6 @@ import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
-import org.elasticsearch.painless.lookup.PainlessClass;
 import org.elasticsearch.painless.lookup.PainlessConstructor;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.objectweb.asm.Type;
@@ -60,38 +59,36 @@ public final class ENewObj extends AExpression {
     @Override
     void analyze(Locals locals) {
         try {
-            actual = locals.getPainlessLookup().getJavaClassFromPainlessType(this.type);
+            actual = locals.getPainlessLookup().canonicalTypeNameToType(this.type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + this.type + "]."));
         }
 
-        PainlessClass struct = locals.getPainlessLookup().getPainlessStructFromJavaClass(actual);
+        try {
-        constructor = struct.constructors.get(PainlessLookupUtility.buildPainlessConstructorKey(arguments.size()));
+            constructor = locals.getPainlessLookup().lookupPainlessConstructor(actual, arguments.size());
-
+        } catch (IllegalArgumentException iae) {
-        if (constructor != null) {
+            throw createError(iae);
-            Class<?>[] types = new Class<?>[constructor.typeParameters.size()];
-            constructor.typeParameters.toArray(types);
-
-            if (constructor.typeParameters.size() != arguments.size()) {
-                throw createError(new IllegalArgumentException(
-                        "When calling constructor on type [" + PainlessLookupUtility.typeToCanonicalTypeName(actual) + "] " +
-                        "expected [" + constructor.typeParameters.size() + "] arguments, but found [" + arguments.size() + "]."));
-            }
-
-            for (int argument = 0; argument < arguments.size(); ++argument) {
-                AExpression expression = arguments.get(argument);
-
-                expression.expected = types[argument];
-                expression.internal = true;
-                expression.analyze(locals);
-                arguments.set(argument, expression.cast(locals));
-            }
-
-            statement = true;
-        } else {
-            throw createError(new IllegalArgumentException(
-                    "Unknown new call on type [" + PainlessLookupUtility.typeToCanonicalTypeName(actual) + "]."));
         }
+
+        Class<?>[] types = new Class<?>[constructor.typeParameters.size()];
+        constructor.typeParameters.toArray(types);
+
+        if (constructor.typeParameters.size() != arguments.size()) {
+            throw createError(new IllegalArgumentException(
+                    "When calling constructor on type [" + PainlessLookupUtility.typeToCanonicalTypeName(actual) + "] " +
+                    "expected [" + constructor.typeParameters.size() + "] arguments, but found [" + arguments.size() + "]."));
+        }
+
+        for (int argument = 0; argument < arguments.size(); ++argument) {
+            AExpression expression = arguments.get(argument);
+
+            expression.expected = types[argument];
+            expression.internal = true;
+            expression.analyze(locals);
+            arguments.set(argument, expression.cast(locals));
+        }
+
+        statement = true;
     }
 
     @Override

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/EStatic.java

@@ -48,7 +48,7 @@ public final class EStatic extends AExpression {
     @Override
     void analyze(Locals locals) {
         try {
-            actual = locals.getPainlessLookup().getJavaClassFromPainlessType(type);
+            actual = locals.getPainlessLookup().canonicalTypeNameToType(type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PCallInvoke.java

@@ -23,8 +23,6 @@ import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
-import org.elasticsearch.painless.lookup.PainlessClass;
-import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.lookup.def;
 
@@ -66,26 +64,16 @@ public final class PCallInvoke extends AExpression {
         prefix.expected = prefix.actual;
         prefix = prefix.cast(locals);
 
-        if (prefix.actual.isArray()) {
+        if (prefix.actual == def.class) {
-            throw createError(new IllegalArgumentException("Illegal call [" + name + "] on array type."));
-        }
-
-        PainlessClass struct = locals.getPainlessLookup().getPainlessStructFromJavaClass(prefix.actual);
-
-        if (prefix.actual.isPrimitive()) {
-            struct = locals.getPainlessLookup().getPainlessStructFromJavaClass(PainlessLookupUtility.typeToBoxedType(prefix.actual));
-        }
-
-        String methodKey = PainlessLookupUtility.buildPainlessMethodKey(name, arguments.size());
-        PainlessMethod method = prefix instanceof EStatic ? struct.staticMethods.get(methodKey) : struct.methods.get(methodKey);
-
-        if (method != null) {
-            sub = new PSubCallInvoke(location, method, prefix.actual, arguments);
-        } else if (prefix.actual == def.class) {
             sub = new PSubDefCall(location, name, arguments);
         } else {
-            throw createError(new IllegalArgumentException("Unknown call [" + name + "] with [" + arguments.size() + "] arguments " +
+            try {
-                    "on type [" + PainlessLookupUtility.typeToCanonicalTypeName(prefix.actual) + "]."));
+                PainlessMethod method =
+                        locals.getPainlessLookup().lookupPainlessMethod(prefix.actual, prefix instanceof EStatic, name, arguments.size());
+                sub = new PSubCallInvoke(location, method, prefix.actual, arguments);
+            } catch (IllegalArgumentException iae) {
+                throw createError(iae);
+            }
         }
 
         if (nullSafe) {

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PField.java

@@ -23,8 +23,6 @@ import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
-import org.elasticsearch.painless.lookup.PainlessClass;
-import org.elasticsearch.painless.lookup.PainlessField;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.lookup.def;
@@ -67,26 +65,34 @@ public final class PField extends AStoreable {
         } else if (prefix.actual == def.class) {
             sub = new PSubDefField(location, value);
         } else {
-            PainlessClass struct = locals.getPainlessLookup().getPainlessStructFromJavaClass(prefix.actual);
+            try {
-            PainlessField field = prefix instanceof EStatic ? struct.staticFields.get(value) : struct.fields.get(value);
+                sub = new PSubField(location,
+                        locals.getPainlessLookup().lookupPainlessField(prefix.actual, prefix instanceof EStatic, value));
+            } catch (IllegalArgumentException fieldIAE) {
+                PainlessMethod getter;
+                PainlessMethod setter;
 
-            if (field != null) {
+                try {
-                sub = new PSubField(location, field);
+                    getter = locals.getPainlessLookup().lookupPainlessMethod(prefix.actual, false,
-            } else {
+                            "get" + Character.toUpperCase(value.charAt(0)) + value.substring(1), 0);
-                PainlessMethod getter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey(
+                } catch (IllegalArgumentException getIAE) {
-                        "get" + Character.toUpperCase(value.charAt(0)) + value.substring(1), 0));
+                    try {
-
+                        getter = locals.getPainlessLookup().lookupPainlessMethod(prefix.actual, false,
-                if (getter == null) {
+                                "is" + Character.toUpperCase(value.charAt(0)) + value.substring(1), 0);
-                    getter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey(
+                    } catch (IllegalArgumentException isIAE) {
-                            "is" + Character.toUpperCase(value.charAt(0)) + value.substring(1), 0));
+                        getter = null;
+                    }
                 }
 
-                PainlessMethod setter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey(
+                try {
-                        "set" + Character.toUpperCase(value.charAt(0)) + value.substring(1), 1));
+                    setter = locals.getPainlessLookup().lookupPainlessMethod(prefix.actual, false,
+                            "set" + Character.toUpperCase(value.charAt(0)) + value.substring(1), 0);
+                } catch (IllegalArgumentException setIAE) {
+                    setter = null;
+                }
 
                 if (getter != null || setter != null) {
-                    sub = new PSubShortcut(
+                    sub = new PSubShortcut(location, value, PainlessLookupUtility.typeToCanonicalTypeName(prefix.actual), getter, setter);
-                            location, value, PainlessLookupUtility.typeToCanonicalTypeName(prefix.actual), getter, setter);
                 } else {
                     EConstant index = new EConstant(location, value);
                     index.analyze(locals);
@@ -99,12 +105,11 @@ public final class PField extends AStoreable {
                         sub = new PSubListShortcut(location, prefix.actual, index);
                     }
                 }
-            }
-        }
 
-        if (sub == null) {
+                if (sub == null) {
-            throw createError(new IllegalArgumentException(
+                    throw createError(fieldIAE);
-                "Unknown field [" + value + "] for type [" + PainlessLookupUtility.typeToCanonicalTypeName(prefix.actual) + "]."));
+                }
+            }
         }
 
         if (nullSafe) {

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSubCallInvoke.java

@@ -56,14 +56,14 @@ final class PSubCallInvoke extends AExpression {
         for (int argument = 0; argument < arguments.size(); ++argument) {
             AExpression expression = arguments.get(argument);
 
-            expression.expected = method.arguments.get(argument);
+            expression.expected = method.typeParameters.get(argument);
             expression.internal = true;
             expression.analyze(locals);
             arguments.set(argument, expression.cast(locals));
         }
 
         statement = true;
-        actual = method.rtn;
+        actual = method.returnType;
     }
 
     @Override
@@ -78,11 +78,11 @@ final class PSubCallInvoke extends AExpression {
             argument.write(writer, globals);
         }
 
-        method.write(writer);
+        writer.invokeMethodCall(method);
     }
 
     @Override
     public String toString() {
-        return singleLineToStringWithOptionalArgs(arguments, prefix, method.name);
+        return singleLineToStringWithOptionalArgs(arguments, prefix, method.javaMethod.getName());
     }
 }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSubField.java

@@ -51,22 +51,24 @@ final class PSubField extends AStoreable {
 
     @Override
     void analyze(Locals locals) {
-         if (write && Modifier.isFinal(field.modifiers)) {
+         if (write && Modifier.isFinal(field.javaField.getModifiers())) {
-             throw createError(new IllegalArgumentException("Cannot write to read-only field [" + field.name + "] for type " +
+             throw createError(new IllegalArgumentException("Cannot write to read-only field [" + field.javaField.getName() + "] " +
-                     "[" + PainlessLookupUtility.typeToCanonicalTypeName(field.clazz) + "]."));
+                     "for type [" + PainlessLookupUtility.typeToCanonicalTypeName(field.javaField.getDeclaringClass()) + "]."));
          }
 
-        actual = field.clazz;
+        actual = field.typeParameter;
     }
 
     @Override
     void write(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
 
-        if (java.lang.reflect.Modifier.isStatic(field.modifiers)) {
+        if (java.lang.reflect.Modifier.isStatic(field.javaField.getModifiers())) {
-            writer.getStatic(Type.getType(field.target), field.javaName, MethodWriter.getType(field.clazz));
+            writer.getStatic(Type.getType(
+                    field.javaField.getDeclaringClass()), field.javaField.getName(), MethodWriter.getType(field.typeParameter));
         } else {
-            writer.getField(Type.getType(field.target), field.javaName, MethodWriter.getType(field.clazz));
+            writer.getField(Type.getType(
+                    field.javaField.getDeclaringClass()), field.javaField.getName(), MethodWriter.getType(field.typeParameter));
         }
     }
 
@@ -94,10 +96,12 @@ final class PSubField extends AStoreable {
     void load(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
 
-        if (java.lang.reflect.Modifier.isStatic(field.modifiers)) {
+        if (java.lang.reflect.Modifier.isStatic(field.javaField.getModifiers())) {
-            writer.getStatic(Type.getType(field.target), field.javaName, MethodWriter.getType(field.clazz));
+            writer.getStatic(Type.getType(
+                    field.javaField.getDeclaringClass()), field.javaField.getName(), MethodWriter.getType(field.typeParameter));
         } else {
-            writer.getField(Type.getType(field.target), field.javaName, MethodWriter.getType(field.clazz));
+            writer.getField(Type.getType(
+                    field.javaField.getDeclaringClass()), field.javaField.getName(), MethodWriter.getType(field.typeParameter));
         }
     }
 
@@ -105,15 +109,17 @@ final class PSubField extends AStoreable {
     void store(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
 
-        if (java.lang.reflect.Modifier.isStatic(field.modifiers)) {
+        if (java.lang.reflect.Modifier.isStatic(field.javaField.getModifiers())) {
-            writer.putStatic(Type.getType(field.target), field.javaName, MethodWriter.getType(field.clazz));
+            writer.putStatic(Type.getType(
+                    field.javaField.getDeclaringClass()), field.javaField.getName(), MethodWriter.getType(field.typeParameter));
         } else {
-            writer.putField(Type.getType(field.target), field.javaName, MethodWriter.getType(field.clazz));
+            writer.putField(Type.getType(
+                    field.javaField.getDeclaringClass()), field.javaField.getName(), MethodWriter.getType(field.typeParameter));
         }
     }
 
     @Override
     public String toString() {
-        return singleLineToString(prefix, field.name);
+        return singleLineToString(prefix, field.javaField.getName());
     }
 }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSubListShortcut.java

@@ -24,7 +24,6 @@ import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
 import org.elasticsearch.painless.WriterConstants;
-import org.elasticsearch.painless.lookup.PainlessClass;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.elasticsearch.painless.lookup.PainlessMethod;
 
@@ -56,23 +55,26 @@ final class PSubListShortcut extends AStoreable {
 
     @Override
     void analyze(Locals locals) {
-        PainlessClass struct = locals.getPainlessLookup().getPainlessStructFromJavaClass(targetClass);
         String canonicalClassName = PainlessLookupUtility.typeToCanonicalTypeName(targetClass);
 
-        getter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey("get", 1));
+        try {
-        setter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey("set", 2));
+            getter = locals.getPainlessLookup().lookupPainlessMethod(targetClass, false, "get", 1);
+            setter = locals.getPainlessLookup().lookupPainlessMethod(targetClass, false, "set", 2);
+        } catch (IllegalArgumentException iae) {
+            throw createError(iae);
+        }
 
-        if (getter != null && (getter.rtn == void.class || getter.arguments.size() != 1 ||
+        if (getter != null && (getter.returnType == void.class || getter.typeParameters.size() != 1 ||
-            getter.arguments.get(0) != int.class)) {
+            getter.typeParameters.get(0) != int.class)) {
             throw createError(new IllegalArgumentException("Illegal list get shortcut for type [" + canonicalClassName + "]."));
         }
 
-        if (setter != null && (setter.arguments.size() != 2 || setter.arguments.get(0) != int.class)) {
+        if (setter != null && (setter.typeParameters.size() != 2 || setter.typeParameters.get(0) != int.class)) {
             throw createError(new IllegalArgumentException("Illegal list set shortcut for type [" + canonicalClassName + "]."));
         }
 
-        if (getter != null && setter != null && (!getter.arguments.get(0).equals(setter.arguments.get(0))
+        if (getter != null && setter != null && (!getter.typeParameters.get(0).equals(setter.typeParameters.get(0))
-            || !getter.rtn.equals(setter.arguments.get(1)))) {
+            || !getter.returnType.equals(setter.typeParameters.get(1)))) {
             throw createError(new IllegalArgumentException("Shortcut argument types must match."));
         }
 
@@ -81,7 +83,7 @@ final class PSubListShortcut extends AStoreable {
             index.analyze(locals);
             index = index.cast(locals);
 
-            actual = setter != null ? setter.arguments.get(1) : getter.rtn;
+            actual = setter != null ? setter.typeParameters.get(1) : getter.returnType;
         } else {
             throw createError(new IllegalArgumentException("Illegal list shortcut for type [" + canonicalClassName + "]."));
         }
@@ -119,21 +121,18 @@ final class PSubListShortcut extends AStoreable {
     @Override
     void load(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
+        writer.invokeMethodCall(getter);
 
-        getter.write(writer);
+        if (getter.returnType == getter.javaMethod.getReturnType()) {
-
+            writer.checkCast(MethodWriter.getType(getter.returnType));
-        if (getter.rtn == getter.handle.type().returnType()) {
-            writer.checkCast(MethodWriter.getType(getter.rtn));
         }
     }
 
     @Override
     void store(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
-
+        writer.invokeMethodCall(setter);
-        setter.write(writer);
+        writer.writePop(MethodWriter.getType(setter.returnType).getSize());
-
-        writer.writePop(MethodWriter.getType(setter.rtn).getSize());
     }
 
     @Override

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSubMapShortcut.java

@@ -23,7 +23,6 @@ import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
-import org.elasticsearch.painless.lookup.PainlessClass;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
 import org.elasticsearch.painless.lookup.PainlessMethod;
 
@@ -55,31 +54,34 @@ final class PSubMapShortcut extends AStoreable {
 
     @Override
     void analyze(Locals locals) {
-        PainlessClass struct = locals.getPainlessLookup().getPainlessStructFromJavaClass(targetClass);
         String canonicalClassName = PainlessLookupUtility.typeToCanonicalTypeName(targetClass);
 
-        getter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey("get", 1));
+        try {
-        setter = struct.methods.get(PainlessLookupUtility.buildPainlessMethodKey("put", 2));
+            getter = locals.getPainlessLookup().lookupPainlessMethod(targetClass, false, "get", 1);
+            setter = locals.getPainlessLookup().lookupPainlessMethod(targetClass, false, "put", 2);
+        } catch (IllegalArgumentException iae) {
+            throw createError(iae);
+        }
 
-        if (getter != null && (getter.rtn == void.class || getter.arguments.size() != 1)) {
+        if (getter != null && (getter.returnType == void.class || getter.typeParameters.size() != 1)) {
             throw createError(new IllegalArgumentException("Illegal map get shortcut for type [" + canonicalClassName + "]."));
         }
 
-        if (setter != null && setter.arguments.size() != 2) {
+        if (setter != null && setter.typeParameters.size() != 2) {
             throw createError(new IllegalArgumentException("Illegal map set shortcut for type [" + canonicalClassName + "]."));
         }
 
-        if (getter != null && setter != null &&
+        if (getter != null && setter != null && (!getter.typeParameters.get(0).equals(setter.typeParameters.get(0)) ||
-            (!getter.arguments.get(0).equals(setter.arguments.get(0)) || !getter.rtn.equals(setter.arguments.get(1)))) {
+                !getter.returnType.equals(setter.typeParameters.get(1)))) {
             throw createError(new IllegalArgumentException("Shortcut argument types must match."));
         }
 
         if ((read || write) && (!read || getter != null) && (!write || setter != null)) {
-            index.expected = setter != null ? setter.arguments.get(0) : getter.arguments.get(0);
+            index.expected = setter != null ? setter.typeParameters.get(0) : getter.typeParameters.get(0);
             index.analyze(locals);
             index = index.cast(locals);
 
-            actual = setter != null ? setter.arguments.get(1) : getter.rtn;
+            actual = setter != null ? setter.typeParameters.get(1) : getter.returnType;
         } else {
             throw createError(new IllegalArgumentException("Illegal map shortcut for type [" + canonicalClassName + "]."));
         }
@@ -90,11 +92,10 @@ final class PSubMapShortcut extends AStoreable {
         index.write(writer, globals);
 
         writer.writeDebugInfo(location);
+        writer.invokeMethodCall(getter);
 
-        getter.write(writer);
+        if (getter.returnType != getter.javaMethod.getReturnType()) {
-
+            writer.checkCast(MethodWriter.getType(getter.returnType));
-        if (getter.rtn != getter.handle.type().returnType()) {
-            writer.checkCast(MethodWriter.getType(getter.rtn));
         }
     }
 
@@ -121,21 +122,18 @@ final class PSubMapShortcut extends AStoreable {
     @Override
     void load(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
+        writer.invokeMethodCall(getter);
 
-        getter.write(writer);
+        if (getter.returnType != getter.javaMethod.getReturnType()) {
-
+            writer.checkCast(MethodWriter.getType(getter.returnType));
-        if (getter.rtn != getter.handle.type().returnType()) {
-            writer.checkCast(MethodWriter.getType(getter.rtn));
         }
     }
 
     @Override
     void store(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
-
+        writer.invokeMethodCall(setter);
-        setter.write(writer);
+        writer.writePop(MethodWriter.getType(setter.returnType).getSize());
-
-        writer.writePop(MethodWriter.getType(setter.rtn).getSize());
     }
 
     @Override

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/PSubShortcut.java

@@ -53,22 +53,22 @@ final class PSubShortcut extends AStoreable {
 
     @Override
     void analyze(Locals locals) {
-        if (getter != null && (getter.rtn == void.class || !getter.arguments.isEmpty())) {
+        if (getter != null && (getter.returnType == void.class || !getter.typeParameters.isEmpty())) {
             throw createError(new IllegalArgumentException(
                 "Illegal get shortcut on field [" + value + "] for type [" + type + "]."));
         }
 
-        if (setter != null && (setter.rtn != void.class || setter.arguments.size() != 1)) {
+        if (setter != null && (setter.returnType != void.class || setter.typeParameters.size() != 1)) {
             throw createError(new IllegalArgumentException(
                 "Illegal set shortcut on field [" + value + "] for type [" + type + "]."));
         }
 
-        if (getter != null && setter != null && setter.arguments.get(0) != getter.rtn) {
+        if (getter != null && setter != null && setter.typeParameters.get(0) != getter.returnType) {
             throw createError(new IllegalArgumentException("Shortcut argument types must match."));
         }
 
         if ((getter != null || setter != null) && (!read || getter != null) && (!write || setter != null)) {
-            actual = setter != null ? setter.arguments.get(0) : getter.rtn;
+            actual = setter != null ? setter.typeParameters.get(0) : getter.returnType;
         } else {
             throw createError(new IllegalArgumentException("Illegal shortcut on field [" + value + "] for type [" + type + "]."));
         }
@@ -78,10 +78,10 @@ final class PSubShortcut extends AStoreable {
     void write(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
 
-        getter.write(writer);
+        writer.invokeMethodCall(getter);
 
-        if (!getter.rtn.equals(getter.handle.type().returnType())) {
+        if (!getter.returnType.equals(getter.javaMethod.getReturnType())) {
-            writer.checkCast(MethodWriter.getType(getter.rtn));
+            writer.checkCast(MethodWriter.getType(getter.returnType));
         }
     }
 
@@ -109,10 +109,10 @@ final class PSubShortcut extends AStoreable {
     void load(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
 
-        getter.write(writer);
+        writer.invokeMethodCall(getter);
 
-        if (getter.rtn != getter.handle.type().returnType()) {
+        if (getter.returnType != getter.javaMethod.getReturnType()) {
-            writer.checkCast(MethodWriter.getType(getter.rtn));
+            writer.checkCast(MethodWriter.getType(getter.returnType));
         }
     }
 
@@ -120,9 +120,9 @@ final class PSubShortcut extends AStoreable {
     void store(MethodWriter writer, Globals globals) {
         writer.writeDebugInfo(location);
 
-        setter.write(writer);
+        writer.invokeMethodCall(setter);
 
-        writer.writePop(MethodWriter.getType(setter.rtn).getSize());
+        writer.writePop(MethodWriter.getType(setter.returnType).getSize());
     }
 
     @Override

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SCatch.java

@@ -67,7 +67,7 @@ public final class SCatch extends AStatement {
         Class<?> clazz;
 
         try {
-            clazz = locals.getPainlessLookup().getJavaClassFromPainlessType(this.type);
+            clazz = locals.getPainlessLookup().canonicalTypeNameToType(this.type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + this.type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SDeclaration.java

@@ -62,7 +62,7 @@ public final class SDeclaration extends AStatement {
         Class<?> clazz;
 
         try {
-            clazz = locals.getPainlessLookup().getJavaClassFromPainlessType(this.type);
+            clazz = locals.getPainlessLookup().canonicalTypeNameToType(this.type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + this.type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SEach.java

@@ -71,7 +71,7 @@ public class SEach extends AStatement {
         Class<?> clazz;
 
         try {
-            clazz = locals.getPainlessLookup().getJavaClassFromPainlessType(this.type);
+            clazz = locals.getPainlessLookup().canonicalTypeNameToType(this.type);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Not a type [" + this.type + "]."));
         }

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SFunction.java

@@ -31,14 +31,12 @@ import org.elasticsearch.painless.MethodWriter;
 import org.elasticsearch.painless.WriterConstants;
 import org.elasticsearch.painless.lookup.PainlessLookup;
 import org.elasticsearch.painless.lookup.PainlessLookupUtility;
-import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.node.SSource.Reserved;
 import org.objectweb.asm.ClassVisitor;
 import org.objectweb.asm.Handle;
 import org.objectweb.asm.Opcodes;
 
 import java.lang.invoke.MethodType;
-import java.lang.reflect.Modifier;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.HashSet;
@@ -92,9 +90,12 @@ public final class SFunction extends AStatement {
     private final List<AStatement> statements;
     public final boolean synthetic;
 
-    Class<?> rtnType = null;
+    Class<?> returnType;
+    List<Class<?>> typeParameters;
+    MethodType methodType;
+
+    org.objectweb.asm.commons.Method method;
     List<Parameter> parameters = new ArrayList<>();
-    PainlessMethod method = null;
 
     private Variable loop = null;
 
@@ -120,7 +121,7 @@ public final class SFunction extends AStatement {
 
     void generateSignature(PainlessLookup painlessLookup) {
         try {
-            rtnType = painlessLookup.getJavaClassFromPainlessType(rtnTypeStr);
+            returnType = painlessLookup.canonicalTypeNameToType(rtnTypeStr);
         } catch (IllegalArgumentException exception) {
             throw createError(new IllegalArgumentException("Illegal return type [" + rtnTypeStr + "] for function [" + name + "]."));
         }
@@ -134,7 +135,7 @@ public final class SFunction extends AStatement {
 
         for (int param = 0; param < this.paramTypeStrs.size(); ++param) {
             try {
-                Class<?> paramType = painlessLookup.getJavaClassFromPainlessType(this.paramTypeStrs.get(param));
+                Class<?> paramType = painlessLookup.canonicalTypeNameToType(this.paramTypeStrs.get(param));
 
                 paramClasses[param] = PainlessLookupUtility.typeToJavaType(paramType);
                 paramTypes.add(paramType);
@@ -145,11 +146,10 @@ public final class SFunction extends AStatement {
             }
         }
 
-        int modifiers = Modifier.STATIC | Modifier.PRIVATE;
+        typeParameters = paramTypes;
-        org.objectweb.asm.commons.Method method = new org.objectweb.asm.commons.Method(name, MethodType.methodType(
+        methodType = MethodType.methodType(PainlessLookupUtility.typeToJavaType(returnType), paramClasses);
-                PainlessLookupUtility.typeToJavaType(rtnType), paramClasses).toMethodDescriptorString());
+        method = new org.objectweb.asm.commons.Method(name, MethodType.methodType(
-        MethodType methodType = MethodType.methodType(PainlessLookupUtility.typeToJavaType(rtnType), paramClasses);
+                PainlessLookupUtility.typeToJavaType(returnType), paramClasses).toMethodDescriptorString());
-        this.method = new PainlessMethod(name, null, null, rtnType, paramTypes, method, modifiers, null, methodType);
     }
 
     @Override
@@ -177,7 +177,7 @@ public final class SFunction extends AStatement {
             allEscape = statement.allEscape;
         }
 
-        if (!methodEscape && rtnType != void.class) {
+        if (!methodEscape && returnType != void.class) {
             throw createError(new IllegalArgumentException("Not all paths provide a return value for method [" + name + "]."));
         }
 
@@ -192,7 +192,7 @@ public final class SFunction extends AStatement {
         if (synthetic) {
             access |= Opcodes.ACC_SYNTHETIC;
         }
-        final MethodWriter function = new MethodWriter(access, method.method, writer, globals.getStatements(), settings);
+        final MethodWriter function = new MethodWriter(access, method, writer, globals.getStatements(), settings);
         function.visitCode();
         write(function, globals);
         function.endMethod();
@@ -212,7 +212,7 @@ public final class SFunction extends AStatement {
         }
 
         if (!methodEscape) {
-            if (rtnType == void.class) {
+            if (returnType == void.class) {
                 function.returnValue();
             } else {
                 throw createError(new IllegalStateException("Illegal tree structure."));
@@ -225,11 +225,7 @@ public final class SFunction extends AStatement {
     }
 
     private void initializeConstant(MethodWriter writer) {
-        final Handle handle = new Handle(Opcodes.H_INVOKESTATIC,
+        final Handle handle = new Handle(Opcodes.H_INVOKESTATIC, CLASS_TYPE.getInternalName(), name, method.getDescriptor(), false);
-                CLASS_TYPE.getInternalName(),
-                name,
-                method.method.getDescriptor(),
-                false);
         writer.push(handle);
     }
 

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SSource.java

@@ -23,6 +23,7 @@ import org.elasticsearch.painless.CompilerSettings;
 import org.elasticsearch.painless.Constant;
 import org.elasticsearch.painless.Globals;
 import org.elasticsearch.painless.Locals;
+import org.elasticsearch.painless.Locals.LocalMethod;
 import org.elasticsearch.painless.Locals.Variable;
 import org.elasticsearch.painless.Location;
 import org.elasticsearch.painless.MethodWriter;
@@ -30,8 +31,6 @@ import org.elasticsearch.painless.ScriptClassInfo;
 import org.elasticsearch.painless.SimpleChecksAdapter;
 import org.elasticsearch.painless.WriterConstants;
 import org.elasticsearch.painless.lookup.PainlessLookup;
-import org.elasticsearch.painless.lookup.PainlessLookupUtility;
-import org.elasticsearch.painless.lookup.PainlessMethod;
 import org.elasticsearch.painless.node.SFunction.FunctionReserved;
 import org.objectweb.asm.ClassVisitor;
 import org.objectweb.asm.ClassWriter;
@@ -165,14 +164,15 @@ public final class SSource extends AStatement {
     }
 
     public void analyze(PainlessLookup painlessLookup) {
-        Map<String, PainlessMethod> methods = new HashMap<>();
+        Map<String, LocalMethod> methods = new HashMap<>();
 
         for (SFunction function : functions) {
             function.generateSignature(painlessLookup);
 
-            String key = PainlessLookupUtility.buildPainlessMethodKey(function.name, function.parameters.size());
+            String key = Locals.buildLocalMethodKey(function.name, function.parameters.size());
 
-            if (methods.put(key, function.method) != null) {
+            if (methods.put(key,
+                    new LocalMethod(function.name, function.returnType, function.typeParameters, function.methodType)) != null) {
                 throw createError(new IllegalArgumentException("Duplicate functions with name [" + function.name + "]."));
             }
         }
@@ -184,7 +184,7 @@ public final class SSource extends AStatement {
     void analyze(Locals program) {
         for (SFunction function : functions) {
             Locals functionLocals =
-                Locals.newFunctionScope(program, function.rtnType, function.parameters, function.reserved.getMaxLoopCounter());
+                Locals.newFunctionScope(program, function.returnType, function.parameters, function.reserved.getMaxLoopCounter());
             function.analyze(functionLocals);
         }
 

modules/lang-painless/src/main/java/org/elasticsearch/painless/node/SSubEachIterable.java

@@ -76,12 +76,10 @@ final class SSubEachIterable extends AStatement {
         if (expression.actual == def.class) {
             method = null;
         } else {
-            method = locals.getPainlessLookup().getPainlessStructFromJavaClass(expression.actual).methods
+            try {
-                    .get(PainlessLookupUtility.buildPainlessMethodKey("iterator", 0));
+                method = locals.getPainlessLookup().lookupPainlessMethod(expression.actual, false, "iterator", 0);
-
+            } catch (IllegalArgumentException iae) {
-            if (method == null) {
+                throw createError(iae);
-                throw createError(new IllegalArgumentException("Unable to create iterator for the type " +
-                        "[" + PainlessLookupUtility.typeToCanonicalTypeName(expression.actual) + "]."));
             }
         }
 
@@ -99,7 +97,7 @@ final class SSubEachIterable extends AStatement {
                     .getMethodType(org.objectweb.asm.Type.getType(Iterator.class), org.objectweb.asm.Type.getType(Object.class));
             writer.invokeDefCall("iterator", methodType, DefBootstrap.ITERATOR);
         } else {
-            method.write(writer);
+            writer.invokeMethodCall(method);
         }
 
         writer.visitVarInsn(MethodWriter.getType(iterator.clazz).getOpcode(Opcodes.ISTORE), iterator.getSlot());

modules/lang-painless/src/main/resources/org/elasticsearch/painless/spi/org.elasticsearch.txt

@@ -77,8 +77,8 @@ class org.elasticsearch.index.fielddata.ScriptDocValues$Longs {
 }
 
 class org.elasticsearch.index.fielddata.ScriptDocValues$Dates {
-  org.joda.time.ReadableDateTime get(int)
+  Object get(int)
-  org.joda.time.ReadableDateTime getValue()
+  Object getValue()
   List getValues()
 }
 

modules/lang-painless/src/test/java/org/elasticsearch/painless/BaseClassTests.java

@@ -69,7 +69,7 @@ public class BaseClassTests extends ScriptTestCase {
     }
 
     public void testGets() {
-        Compiler compiler = new Compiler(Gets.class, painlessLookup);
+        Compiler compiler = new Compiler(Gets.class, null, null, painlessLookup);
         Map<String, Object> map = new HashMap<>();
         map.put("s", 1);
 
@@ -87,7 +87,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute();
     }
     public void testNoArgs() {
-        Compiler compiler = new Compiler(NoArgs.class, painlessLookup);
+        Compiler compiler = new Compiler(NoArgs.class, null, null, painlessLookup);
         assertEquals(1, ((NoArgs)scriptEngine.compile(compiler, null, "1", emptyMap())).execute());
         assertEquals("foo", ((NoArgs)scriptEngine.compile(compiler, null, "'foo'", emptyMap())).execute());
 
@@ -111,13 +111,13 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(Object arg);
     }
     public void testOneArg() {
-        Compiler compiler = new Compiler(OneArg.class, painlessLookup);
+        Compiler compiler = new Compiler(OneArg.class, null, null, painlessLookup);
         Object rando = randomInt();
         assertEquals(rando, ((OneArg)scriptEngine.compile(compiler, null, "arg", emptyMap())).execute(rando));
         rando = randomAlphaOfLength(5);
         assertEquals(rando, ((OneArg)scriptEngine.compile(compiler, null, "arg", emptyMap())).execute(rando));
 
-        Compiler noargs = new Compiler(NoArgs.class, painlessLookup);
+        Compiler noargs = new Compiler(NoArgs.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, () ->
                 scriptEngine.compile(noargs, null, "doc", emptyMap()));
         assertEquals("Variable [doc] is not defined.", e.getMessage());
@@ -132,7 +132,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(String[] arg);
     }
     public void testArrayArg() {
-        Compiler compiler = new Compiler(ArrayArg.class, painlessLookup);
+        Compiler compiler = new Compiler(ArrayArg.class, null, null, painlessLookup);
         String rando = randomAlphaOfLength(5);
         assertEquals(rando, ((ArrayArg)scriptEngine.compile(compiler, null, "arg[0]", emptyMap())).execute(new String[] {rando, "foo"}));
     }
@@ -142,7 +142,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(int[] arg);
     }
     public void testPrimitiveArrayArg() {
-        Compiler compiler = new Compiler(PrimitiveArrayArg.class, painlessLookup);
+        Compiler compiler = new Compiler(PrimitiveArrayArg.class, null, null, painlessLookup);
         int rando = randomInt();
         assertEquals(rando, ((PrimitiveArrayArg)scriptEngine.compile(compiler, null, "arg[0]", emptyMap())).execute(new int[] {rando, 10}));
     }
@@ -152,7 +152,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(Object[] arg);
     }
     public void testDefArrayArg() {
-        Compiler compiler = new Compiler(DefArrayArg.class, painlessLookup);
+        Compiler compiler = new Compiler(DefArrayArg.class, null, null, painlessLookup);
         Object rando = randomInt();
         assertEquals(rando, ((DefArrayArg)scriptEngine.compile(compiler, null, "arg[0]", emptyMap())).execute(new Object[] {rando, 10}));
         rando = randomAlphaOfLength(5);
@@ -170,7 +170,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract boolean needsD();
     }
     public void testManyArgs() {
-        Compiler compiler = new Compiler(ManyArgs.class, painlessLookup);
+        Compiler compiler = new Compiler(ManyArgs.class, null, null, painlessLookup);
         int rando = randomInt();
         assertEquals(rando, ((ManyArgs)scriptEngine.compile(compiler, null, "a", emptyMap())).execute(rando, 0, 0, 0));
         assertEquals(10, ((ManyArgs)scriptEngine.compile(compiler, null, "a + b + c + d", emptyMap())).execute(1, 2, 3, 4));
@@ -198,7 +198,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(String... arg);
     }
     public void testVararg() {
-        Compiler compiler = new Compiler(VarargTest.class, painlessLookup);
+        Compiler compiler = new Compiler(VarargTest.class, null, null, painlessLookup);
         assertEquals("foo bar baz", ((VarargTest)scriptEngine.compile(compiler, null, "String.join(' ', Arrays.asList(arg))", emptyMap()))
                     .execute("foo", "bar", "baz"));
     }
@@ -214,7 +214,7 @@ public class BaseClassTests extends ScriptTestCase {
         }
     }
     public void testDefaultMethods() {
-        Compiler compiler = new Compiler(DefaultMethods.class, painlessLookup);
+        Compiler compiler = new Compiler(DefaultMethods.class, null, null, painlessLookup);
         int rando = randomInt();
         assertEquals(rando, ((DefaultMethods)scriptEngine.compile(compiler, null, "a", emptyMap())).execute(rando, 0, 0, 0));
         assertEquals(rando, ((DefaultMethods)scriptEngine.compile(compiler, null, "a", emptyMap())).executeWithASingleOne(rando, 0, 0));
@@ -228,7 +228,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract void execute(Map<String, Object> map);
     }
     public void testReturnsVoid() {
-        Compiler compiler = new Compiler(ReturnsVoid.class, painlessLookup);
+        Compiler compiler = new Compiler(ReturnsVoid.class, null, null, painlessLookup);
         Map<String, Object> map = new HashMap<>();
         ((ReturnsVoid)scriptEngine.compile(compiler, null, "map.a = 'foo'", emptyMap())).execute(map);
         assertEquals(singletonMap("a", "foo"), map);
@@ -247,7 +247,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract boolean execute();
     }
     public void testReturnsPrimitiveBoolean() {
-        Compiler compiler = new Compiler(ReturnsPrimitiveBoolean.class, painlessLookup);
+        Compiler compiler = new Compiler(ReturnsPrimitiveBoolean.class, null, null, painlessLookup);
 
         assertEquals(true, ((ReturnsPrimitiveBoolean)scriptEngine.compile(compiler, null, "true", emptyMap())).execute());
         assertEquals(false, ((ReturnsPrimitiveBoolean)scriptEngine.compile(compiler, null, "false", emptyMap())).execute());
@@ -289,7 +289,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract int execute();
     }
     public void testReturnsPrimitiveInt() {
-        Compiler compiler = new Compiler(ReturnsPrimitiveInt.class, painlessLookup);
+        Compiler compiler = new Compiler(ReturnsPrimitiveInt.class, null, null, painlessLookup);
 
         assertEquals(1, ((ReturnsPrimitiveInt)scriptEngine.compile(compiler, null, "1", emptyMap())).execute());
         assertEquals(1, ((ReturnsPrimitiveInt)scriptEngine.compile(compiler, null, "(int) 1L", emptyMap())).execute());
@@ -331,7 +331,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract float execute();
     }
     public void testReturnsPrimitiveFloat() {
-        Compiler compiler = new Compiler(ReturnsPrimitiveFloat.class, painlessLookup);
+        Compiler compiler = new Compiler(ReturnsPrimitiveFloat.class, null, null, painlessLookup);
 
         assertEquals(1.1f, ((ReturnsPrimitiveFloat)scriptEngine.compile(compiler, null, "1.1f", emptyMap())).execute(), 0);
         assertEquals(1.1f, ((ReturnsPrimitiveFloat)scriptEngine.compile(compiler, null, "(float) 1.1d", emptyMap())).execute(), 0);
@@ -362,7 +362,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract double execute();
     }
     public void testReturnsPrimitiveDouble() {
-        Compiler compiler = new Compiler(ReturnsPrimitiveDouble.class, painlessLookup);
+        Compiler compiler = new Compiler(ReturnsPrimitiveDouble.class, null, null, painlessLookup);
 
         assertEquals(1.0, ((ReturnsPrimitiveDouble)scriptEngine.compile(compiler, null, "1", emptyMap())).execute(), 0);
         assertEquals(1.0, ((ReturnsPrimitiveDouble)scriptEngine.compile(compiler, null, "1L", emptyMap())).execute(), 0);
@@ -396,7 +396,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(String foo);
     }
     public void testNoArgumentsConstant() {
-        Compiler compiler = new Compiler(NoArgumentsConstant.class, painlessLookup);
+        Compiler compiler = new Compiler(NoArgumentsConstant.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "1", emptyMap()));
         assertThat(e.getMessage(), startsWith(
@@ -409,7 +409,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(String foo);
     }
     public void testWrongArgumentsConstant() {
-        Compiler compiler = new Compiler(WrongArgumentsConstant.class, painlessLookup);
+        Compiler compiler = new Compiler(WrongArgumentsConstant.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "1", emptyMap()));
         assertThat(e.getMessage(), startsWith(
@@ -422,7 +422,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(String foo);
     }
     public void testWrongLengthOfArgumentConstant() {
-        Compiler compiler = new Compiler(WrongLengthOfArgumentConstant.class, painlessLookup);
+        Compiler compiler = new Compiler(WrongLengthOfArgumentConstant.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "1", emptyMap()));
         assertThat(e.getMessage(), startsWith("[" + WrongLengthOfArgumentConstant.class.getName() + "#ARGUMENTS] has length [2] but ["
@@ -434,7 +434,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(UnknownArgType foo);
     }
     public void testUnknownArgType() {
-        Compiler compiler = new Compiler(UnknownArgType.class, painlessLookup);
+        Compiler compiler = new Compiler(UnknownArgType.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "1", emptyMap()));
         assertEquals("[foo] is of unknown type [" + UnknownArgType.class.getName() + ". Painless interfaces can only accept arguments "
@@ -446,7 +446,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract UnknownReturnType execute(String foo);
     }
     public void testUnknownReturnType() {
-        Compiler compiler = new Compiler(UnknownReturnType.class, painlessLookup);
+        Compiler compiler = new Compiler(UnknownReturnType.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "1", emptyMap()));
         assertEquals("Painless can only implement execute methods returning a whitelisted type but [" + UnknownReturnType.class.getName()
@@ -458,7 +458,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(UnknownArgTypeInArray[] foo);
     }
     public void testUnknownArgTypeInArray() {
-        Compiler compiler = new Compiler(UnknownArgTypeInArray.class, painlessLookup);
+        Compiler compiler = new Compiler(UnknownArgTypeInArray.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "1", emptyMap()));
         assertEquals("[foo] is of unknown type [" + UnknownArgTypeInArray.class.getName() + ". Painless interfaces can only accept "
@@ -470,7 +470,7 @@ public class BaseClassTests extends ScriptTestCase {
         public abstract Object execute(boolean foo);
     }
     public void testTwoExecuteMethods() {
-        Compiler compiler = new Compiler(TwoExecuteMethods.class, painlessLookup);
+        Compiler compiler = new Compiler(TwoExecuteMethods.class, null, null, painlessLookup);
         Exception e = expectScriptThrows(IllegalArgumentException.class, false, () ->
             scriptEngine.compile(compiler, null, "null", emptyMap()));
         assertEquals("Painless can only implement interfaces that have a single method named [execute] but ["

modules/lang-painless/src/test/java/org/elasticsearch/painless/Debugger.java

@@ -40,7 +40,7 @@ final class Debugger {
         PrintWriter outputWriter = new PrintWriter(output);
         Textifier textifier = new Textifier();
         try {
-            new Compiler(iface, PainlessLookupBuilder.buildFromWhitelists(Whitelist.BASE_WHITELISTS))
+            new Compiler(iface, null, null, PainlessLookupBuilder.buildFromWhitelists(Whitelist.BASE_WHITELISTS))
                     .compile("<debugging>", source, settings, textifier);
         } catch (RuntimeException e) {
             textifier.print(outputWriter);

modules/lang-painless/src/test/java/org/elasticsearch/painless/OverloadTests.java

@@ -23,12 +23,12 @@ package org.elasticsearch.painless;
 public class OverloadTests extends ScriptTestCase {
 
     public void testMethod() {
-        assertEquals(2, exec("return 'abc123abc'.indexOf('c');"));
+        //assertEquals(2, exec("return 'abc123abc'.indexOf('c');"));
-        assertEquals(8, exec("return 'abc123abc'.indexOf('c', 3);"));
+        //assertEquals(8, exec("return 'abc123abc'.indexOf('c', 3);"));
         IllegalArgumentException expected = expectScriptThrows(IllegalArgumentException.class, () -> {
             exec("return 'abc123abc'.indexOf('c', 3, 'bogus');");
         });
-        assertTrue(expected.getMessage().contains("[indexOf] with [3] arguments"));
+        assertTrue(expected.getMessage().contains("[java.lang.String, indexOf/3]"));
     }
     
     public void testMethodDynamic() {

modules/lang-painless/src/test/java/org/elasticsearch/painless/PainlessDocGenerator.java

@@ -45,9 +45,9 @@ import java.util.List;
 import java.util.Map;
 import java.util.TreeMap;
 import java.util.function.Consumer;
+import java.util.stream.Collectors;
 
 import static java.util.Comparator.comparing;
-import static java.util.stream.Collectors.toList;
 
 /**
  * Generates an API reference from the method and type whitelists in {@link PainlessLookup}.
@@ -56,9 +56,9 @@ public class PainlessDocGenerator {
 
     private static final PainlessLookup PAINLESS_LOOKUP = PainlessLookupBuilder.buildFromWhitelists(Whitelist.BASE_WHITELISTS);
     private static final Logger logger = ESLoggerFactory.getLogger(PainlessDocGenerator.class);
-    private static final Comparator<PainlessField> FIELD_NAME = comparing(f -> f.name);
+    private static final Comparator<PainlessField> FIELD_NAME = comparing(f -> f.javaField.getName());
-    private static final Comparator<PainlessMethod> METHOD_NAME = comparing(m -> m.name);
+    private static final Comparator<PainlessMethod> METHOD_NAME = comparing(m -> m.javaMethod.getName());
-    private static final Comparator<PainlessMethod> METHOD_NUMBER_OF_PARAMS = comparing(m -> m.arguments.size());
+    private static final Comparator<PainlessMethod> METHOD_NUMBER_OF_PARAMS = comparing(m -> m.typeParameters.size());
     private static final Comparator<PainlessConstructor> CONSTRUCTOR_NUMBER_OF_PARAMS = comparing(m -> m.typeParameters.size());
 
     public static void main(String[] args) throws IOException {
@@ -74,9 +74,10 @@ public class PainlessDocGenerator {
             Files.newOutputStream(indexPath, StandardOpenOption.CREATE_NEW, StandardOpenOption.WRITE),
             false, StandardCharsets.UTF_8.name())) {
             emitGeneratedWarning(indexStream);
-            List<Class<?>> classes = PAINLESS_LOOKUP.getStructs().stream().sorted(comparing(Class::getCanonicalName)).collect(toList());
+            List<Class<?>> classes = PAINLESS_LOOKUP.getClasses().stream().sorted(
+                    Comparator.comparing(Class::getCanonicalName)).collect(Collectors.toList());
             for (Class<?> clazz : classes) {
-                PainlessClass struct = PAINLESS_LOOKUP.getPainlessStructFromJavaClass(clazz);
+                PainlessClass struct = PAINLESS_LOOKUP.lookupPainlessClass(clazz);
                 String canonicalClassName = PainlessLookupUtility.typeToCanonicalTypeName(clazz);
 
                 if (clazz.isPrimitive()) {
@@ -114,10 +115,10 @@ public class PainlessDocGenerator {
                     struct.constructors.values().stream().sorted(CONSTRUCTOR_NUMBER_OF_PARAMS).forEach(documentConstructor);
                     Map<String, Class<?>> inherited = new TreeMap<>();
                     struct.methods.values().stream().sorted(METHOD_NAME.thenComparing(METHOD_NUMBER_OF_PARAMS)).forEach(method -> {
-                        if (method.target == clazz) {
+                        if (method.targetClass == clazz) {
                             documentMethod(typeStream, method);
                         } else {
-                            inherited.put(canonicalClassName, method.target);
+                            inherited.put(canonicalClassName, method.targetClass);
                         }
                     });
 
@@ -147,17 +148,17 @@ public class PainlessDocGenerator {
         emitAnchor(stream, field);
         stream.print("]]");
 
-        if (Modifier.isStatic(field.modifiers)) {
+        if (Modifier.isStatic(field.javaField.getModifiers())) {
             stream.print("static ");
         }
 
-        emitType(stream, field.clazz);
+        emitType(stream, field.typeParameter);
         stream.print(' ');
 
         String javadocRoot = javadocRoot(field);
         emitJavadocLink(stream, javadocRoot, field);
         stream.print('[');
-        stream.print(field.name);
+        stream.print(field.javaField.getName());
         stream.print(']');
 
         if (javadocRoot.equals("java8")) {
@@ -212,11 +213,11 @@ public class PainlessDocGenerator {
         emitAnchor(stream, method);
         stream.print("]]");
 
-        if (null == method.augmentation && Modifier.isStatic(method.modifiers)) {
+        if (method.targetClass == method.javaMethod.getDeclaringClass() && Modifier.isStatic(method.javaMethod.getModifiers())) {
             stream.print("static ");
         }
 
-        emitType(stream, method.rtn);
+        emitType(stream, method.returnType);
         stream.print(' ');
 
         String javadocRoot = javadocRoot(method);
@@ -227,7 +228,7 @@ public class PainlessDocGenerator {
 
         stream.print("](");
         boolean first = true;
-        for (Class<?> arg : method.arguments) {
+        for (Class<?> arg : method.typeParameters) {
             if (first) {
                 first = false;
             } else {
@@ -269,20 +270,20 @@ public class PainlessDocGenerator {
      * Anchor text for a {@link PainlessMethod}.
      */
     private static void emitAnchor(PrintStream stream, PainlessMethod method) {
-        emitAnchor(stream, method.target);
+        emitAnchor(stream, method.targetClass);
         stream.print('-');
         stream.print(methodName(method));
         stream.print('-');
-        stream.print(method.arguments.size());
+        stream.print(method.typeParameters.size());
     }
 
     /**
      * Anchor text for a {@link PainlessField}.
      */
     private static void emitAnchor(PrintStream stream, PainlessField field) {
-        emitAnchor(stream, field.target);
+        emitAnchor(stream, field.javaField.getDeclaringClass());
         stream.print('-');
-        stream.print(field.name);
+        stream.print(field.javaField.getName());
     }
 
     private static String constructorName(PainlessConstructor constructor) {
@@ -290,7 +291,7 @@ public class PainlessDocGenerator {
     }
 
     private static String methodName(PainlessMethod method) {
-        return PainlessLookupUtility.typeToCanonicalTypeName(method.target);
+        return PainlessLookupUtility.typeToCanonicalTypeName(method.targetClass);
     }
 
     /**
@@ -359,16 +360,16 @@ public class PainlessDocGenerator {
         stream.print("link:{");
         stream.print(root);
         stream.print("-javadoc}/");
-        stream.print(classUrlPath(method.augmentation != null ? method.augmentation : method.target));
+        stream.print(classUrlPath(method.javaMethod.getDeclaringClass()));
         stream.print(".html#");
         stream.print(methodName(method));
         stream.print("%2D");
         boolean first = true;
-        if (method.augmentation != null) {
+        if (method.targetClass != method.javaMethod.getDeclaringClass()) {
             first = false;
-            stream.print(method.target.getName());
+            stream.print(method.javaMethod.getDeclaringClass().getName());
         }
-        for (Class<?> clazz: method.arguments) {
+        for (Class<?> clazz: method.typeParameters) {
             if (first) {
                 first = false;
             } else {
@@ -391,26 +392,26 @@ public class PainlessDocGenerator {
         stream.print("link:{");
         stream.print(root);
         stream.print("-javadoc}/");
-        stream.print(classUrlPath(field.target));
+        stream.print(classUrlPath(field.javaField.getDeclaringClass()));
         stream.print(".html#");
-        stream.print(field.javaName);
+        stream.print(field.javaField.getName());
     }
 
     /**
      * Pick the javadoc root for a {@link PainlessMethod}.
      */
     private static String javadocRoot(PainlessMethod method) {
-        if (method.augmentation != null) {
+        if (method.targetClass != method.javaMethod.getDeclaringClass()) {
             return "painless";
         }
-        return javadocRoot(method.target);
+        return javadocRoot(method.targetClass);
     }
 
     /**
      * Pick the javadoc root for a {@link PainlessField}.
      */
     private static String javadocRoot(PainlessField field) {
-        return javadocRoot(field.target);
+        return javadocRoot(field.javaField.getDeclaringClass());
     }
 
     /**

modules/lang-painless/src/test/java/org/elasticsearch/painless/RegexTests.java

@@ -252,7 +252,7 @@ public class RegexTests extends ScriptTestCase {
         IllegalArgumentException e = expectScriptThrows(IllegalArgumentException.class, () -> {
             exec("Pattern.compile('aa')");
         });
-        assertEquals("Unknown call [compile] with [1] arguments on type [java.util.regex.Pattern].", e.getMessage());
+        assertTrue(e.getMessage().contains("[java.util.regex.Pattern, compile/1]"));
     }
 
     public void testBadRegexPattern() {

modules/lang-painless/src/test/java/org/elasticsearch/painless/node/NodeToStringTests.java

@@ -404,7 +404,7 @@ public class NodeToStringTests extends ESTestCase {
 
     public void testPSubCallInvoke() {
         Location l = new Location(getTestName(), 0);
-        PainlessClass c = painlessLookup.getPainlessStructFromJavaClass(Integer.class);
+        PainlessClass c = painlessLookup.lookupPainlessClass(Integer.class);
         PainlessMethod m = c.methods.get(PainlessLookupUtility.buildPainlessMethodKey("toString", 0));
         PSubCallInvoke node = new PSubCallInvoke(l, m, null, emptyList());
         node.prefix = new EVariable(l, "a");
@@ -459,7 +459,7 @@ public class NodeToStringTests extends ESTestCase {
 
     public void testPSubField() {
         Location l = new Location(getTestName(), 0);
-        PainlessClass s = painlessLookup.getPainlessStructFromJavaClass(Boolean.class);
+        PainlessClass s = painlessLookup.lookupPainlessClass(Boolean.class);
         PainlessField f = s.staticFields.get("TRUE");
         PSubField node = new PSubField(l, f);
         node.prefix = new EStatic(l, "Boolean");
@@ -497,7 +497,7 @@ public class NodeToStringTests extends ESTestCase {
 
     public void testPSubShortcut() {
         Location l = new Location(getTestName(), 0);
-        PainlessClass s = painlessLookup.getPainlessStructFromJavaClass(FeatureTest.class);
+        PainlessClass s = painlessLookup.lookupPainlessClass(FeatureTest.class);
         PainlessMethod getter = s.methods.get(PainlessLookupUtility.buildPainlessMethodKey("getX", 0));
         PainlessMethod setter = s.methods.get(PainlessLookupUtility.buildPainlessMethodKey("setX", 1));
         PSubShortcut node = new PSubShortcut(l, "x", FeatureTest.class.getName(), getter, setter);

modules/lang-painless/src/test/resources/rest-api-spec/test/painless/20_scriptfield.yml

@@ -108,7 +108,7 @@ setup:
                 script_fields:
                     bar:
                         script:
-                            source: "doc.date.value.dayOfWeek"
+                            source: "doc.date.value.dayOfWeek.value"
 
     - match: { hits.hits.0.fields.bar.0: 7}
 
@@ -123,7 +123,7 @@ setup:
                             source: >
                               StringBuilder b = new StringBuilder();
                               for (def date : doc.dates) {
-                                b.append(" ").append(date.getDayOfWeek());
+                                b.append(" ").append(date.getDayOfWeek().value);
                               }
                               return b.toString().trim()
 

modules/lang-painless/src/test/resources/rest-api-spec/test/painless/50_script_doc_values.yml

@@ -95,7 +95,7 @@ setup:
                     field:
                         script:
                             source: "doc.date.get(0)"
-    - match: { hits.hits.0.fields.field.0: '2017-01-01T12:11:12.000Z' }
+    - match: { hits.hits.0.fields.field.0: '2017-01-01T12:11:12Z' }
 
     - do:
         search:
@@ -104,7 +104,7 @@ setup:
                     field:
                         script:
                             source: "doc.date.value"
-    - match: { hits.hits.0.fields.field.0: '2017-01-01T12:11:12.000Z' }
+    - match: { hits.hits.0.fields.field.0: '2017-01-01T12:11:12Z' }
 
 ---
 "geo_point":

modules/transport-netty4/build.gradle

@@ -34,13 +34,13 @@ compileTestJava.options.compilerArgs << "-Xlint:-cast,-deprecation,-rawtypes,-tr
 
 dependencies {
     // network stack
-    compile "io.netty:netty-buffer:4.1.16.Final"
+    compile "io.netty:netty-buffer:4.1.28.Final"
-    compile "io.netty:netty-codec:4.1.16.Final"
+    compile "io.netty:netty-codec:4.1.28.Final"
-    compile "io.netty:netty-codec-http:4.1.16.Final"
+    compile "io.netty:netty-codec-http:4.1.28.Final"
-    compile "io.netty:netty-common:4.1.16.Final"
+    compile "io.netty:netty-common:4.1.28.Final"
-    compile "io.netty:netty-handler:4.1.16.Final"
+    compile "io.netty:netty-handler:4.1.28.Final"
-    compile "io.netty:netty-resolver:4.1.16.Final"
+    compile "io.netty:netty-resolver:4.1.28.Final"
-    compile "io.netty:netty-transport:4.1.16.Final"
+    compile "io.netty:netty-transport:4.1.28.Final"
 }
 
 dependencyLicenses {
@@ -134,7 +134,6 @@ thirdPartyAudit.excludes = [
         'net.jpountz.xxhash.StreamingXXHash32',
         'net.jpountz.xxhash.XXHashFactory',
         'io.netty.internal.tcnative.CertificateRequestedCallback',
-        'io.netty.internal.tcnative.CertificateRequestedCallback$KeyMaterial',
         'io.netty.internal.tcnative.CertificateVerifier',
         'io.netty.internal.tcnative.SessionTicketKey',
         'io.netty.internal.tcnative.SniHostNameMatcher',
@@ -161,6 +160,6 @@ thirdPartyAudit.excludes = [
 
         'org.conscrypt.AllocatedBuffer',
         'org.conscrypt.BufferAllocator',
-        'org.conscrypt.Conscrypt$Engines',
+        'org.conscrypt.Conscrypt',
         'org.conscrypt.HandshakeListener'
 ]

modules/transport-netty4/licenses/netty-buffer-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-63b5fa95c74785e16f2c30ce268bc222e35c8cb5

modules/transport-netty4/licenses/netty-buffer-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+d6c2d13492778009d33f60e05ed90bcb535d1fd1

modules/transport-netty4/licenses/netty-codec-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-d84a1f21768b7309c2954521cf5a1f46c2309eb1

modules/transport-netty4/licenses/netty-codec-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+a38361d893900947524f8a9da980555950e73d6a

modules/transport-netty4/licenses/netty-codec-http-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-d64312378b438dfdad84267c599a053327c6f02a

modules/transport-netty4/licenses/netty-codec-http-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+897100c1022c780b0a436b9349e507e8fa9800dc

modules/transport-netty4/licenses/netty-common-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-177a6b30cca92f6f5f9873c9befd681377a4c328

modules/transport-netty4/licenses/netty-common-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+df69ce8bb9b544a71e7bbee290253cf7c93e6bad

modules/transport-netty4/licenses/netty-handler-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-fec0e63e7dd7f4eeef7ea8dc47a1ff32dfc7ebc2

modules/transport-netty4/licenses/netty-handler-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+a035784682da0126bc25f10713dac732b5082a6d

modules/transport-netty4/licenses/netty-resolver-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-f6eb553b53fb3a90a8ac1170697093fed82eae28

modules/transport-netty4/licenses/netty-resolver-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+f33557dcb31fa20da075ac05e4808115e32ef9b7

modules/transport-netty4/licenses/netty-transport-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-3c8ee2c4d4a1cbb947a5c184c7aeb2204260958b

modules/transport-netty4/licenses/netty-transport-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+d2ef28f49d726737f0ffe84bf66529b3bf6e0c0d

modules/transport-netty4/src/main/plugin-metadata/plugin-security.policy

@@ -23,6 +23,9 @@ grant codeBase "${codebase.netty-common}" {
 
    // netty makes and accepts socket connections
    permission java.net.SocketPermission "*", "accept,connect";
+
+   // Netty sets custom classloader for some of its internal threads
+   permission java.lang.RuntimePermission "*", "setContextClassLoader";
 };
 
 grant codeBase "${codebase.netty-transport}" {

modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Netty4BadRequestTests.java

@@ -20,6 +20,7 @@
 package org.elasticsearch.http.netty4;
 
 import io.netty.handler.codec.http.FullHttpResponse;
+import io.netty.util.ReferenceCounted;
 import org.elasticsearch.ElasticsearchException;
 import org.elasticsearch.common.network.NetworkService;
 import org.elasticsearch.common.settings.Settings;
@@ -92,15 +93,19 @@ public class Netty4BadRequestTests extends ESTestCase {
             try (Netty4HttpClient nettyHttpClient = new Netty4HttpClient()) {
                 final Collection<FullHttpResponse> responses =
                         nettyHttpClient.get(transportAddress.address(), "/_cluster/settings?pretty=%");
-                assertThat(responses, hasSize(1));
+                try {
-                assertThat(responses.iterator().next().status().code(), equalTo(400));
+                    assertThat(responses, hasSize(1));
-                final Collection<String> responseBodies = Netty4HttpClient.returnHttpResponseBodies(responses);
+                    assertThat(responses.iterator().next().status().code(), equalTo(400));
-                assertThat(responseBodies, hasSize(1));
+                    final Collection<String> responseBodies = Netty4HttpClient.returnHttpResponseBodies(responses);
-                assertThat(responseBodies.iterator().next(), containsString("\"type\":\"bad_parameter_exception\""));
+                    assertThat(responseBodies, hasSize(1));
-                assertThat(
+                    assertThat(responseBodies.iterator().next(), containsString("\"type\":\"bad_parameter_exception\""));
+                    assertThat(
                         responseBodies.iterator().next(),
                         containsString(
-                                "\"reason\":\"java.lang.IllegalArgumentException: unterminated escape sequence at end of string: %\""));
+                            "\"reason\":\"java.lang.IllegalArgumentException: unterminated escape sequence at end of string: %\""));
+                } finally {
+                    responses.forEach(ReferenceCounted::release);
+                }
             }
         }
     }

modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Netty4HttpRequestSizeLimitIT.java

@@ -20,6 +20,7 @@
 package org.elasticsearch.http.netty4;
 
 import io.netty.handler.codec.http.FullHttpResponse;
+import io.netty.util.ReferenceCounted;
 import org.elasticsearch.ESNetty4IntegTestCase;
 import org.elasticsearch.common.collect.Tuple;
 import org.elasticsearch.common.settings.Settings;
@@ -88,12 +89,20 @@ public class Netty4HttpRequestSizeLimitIT extends ESNetty4IntegTestCase {
 
         try (Netty4HttpClient nettyHttpClient = new Netty4HttpClient()) {
             Collection<FullHttpResponse> singleResponse = nettyHttpClient.post(transportAddress.address(), requests[0]);
-            assertThat(singleResponse, hasSize(1));
+            try {
-            assertAtLeastOnceExpectedStatus(singleResponse, HttpResponseStatus.OK);
+                assertThat(singleResponse, hasSize(1));
+                assertAtLeastOnceExpectedStatus(singleResponse, HttpResponseStatus.OK);
 
-            Collection<FullHttpResponse> multipleResponses = nettyHttpClient.post(transportAddress.address(), requests);
+                Collection<FullHttpResponse> multipleResponses = nettyHttpClient.post(transportAddress.address(), requests);
-            assertThat(multipleResponses, hasSize(requests.length));
+                try {
-            assertAtLeastOnceExpectedStatus(multipleResponses, HttpResponseStatus.SERVICE_UNAVAILABLE);
+                    assertThat(multipleResponses, hasSize(requests.length));
+                    assertAtLeastOnceExpectedStatus(multipleResponses, HttpResponseStatus.SERVICE_UNAVAILABLE);
+                } finally {
+                    multipleResponses.forEach(ReferenceCounted::release);
+                }
+            } finally {
+                singleResponse.forEach(ReferenceCounted::release);
+            }
         }
     }
 
@@ -113,8 +122,12 @@ public class Netty4HttpRequestSizeLimitIT extends ESNetty4IntegTestCase {
 
         try (Netty4HttpClient nettyHttpClient = new Netty4HttpClient()) {
             Collection<FullHttpResponse> responses = nettyHttpClient.put(transportAddress.address(), requestUris);
-            assertThat(responses, hasSize(requestUris.length));
+            try {
-            assertAllInExpectedStatus(responses, HttpResponseStatus.OK);
+                assertThat(responses, hasSize(requestUris.length));
+                assertAllInExpectedStatus(responses, HttpResponseStatus.OK);
+            } finally {
+                responses.forEach(ReferenceCounted::release);
+            }
         }
     }
 

modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Netty4HttpServerPipeliningTests.java

@@ -29,6 +29,7 @@ import io.netty.channel.SimpleChannelInboundHandler;
 import io.netty.handler.codec.http.FullHttpRequest;
 import io.netty.handler.codec.http.FullHttpResponse;
 import io.netty.handler.codec.http.HttpHeaderNames;
+import io.netty.util.ReferenceCounted;
 import org.elasticsearch.common.bytes.BytesArray;
 import org.elasticsearch.common.network.NetworkService;
 import org.elasticsearch.common.settings.Settings;
@@ -98,8 +99,12 @@ public class Netty4HttpServerPipeliningTests extends ESTestCase {
 
             try (Netty4HttpClient nettyHttpClient = new Netty4HttpClient()) {
                 Collection<FullHttpResponse> responses = nettyHttpClient.get(transportAddress.address(), requests.toArray(new String[]{}));
-                Collection<String> responseBodies = Netty4HttpClient.returnHttpResponseBodies(responses);
+                try {
-                assertThat(responseBodies, contains(requests.toArray()));
+                    Collection<String> responseBodies = Netty4HttpClient.returnHttpResponseBodies(responses);
+                    assertThat(responseBodies, contains(requests.toArray()));
+                } finally {
+                    responses.forEach(ReferenceCounted::release);
+                }
             }
         }
     }

modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Netty4HttpServerTransportTests.java

@@ -207,14 +207,23 @@ public class Netty4HttpServerTransportTests extends ESTestCase {
                 HttpUtil.setContentLength(request, contentLength);
 
                 final FullHttpResponse response = client.post(remoteAddress.address(), request);
-                assertThat(response.status(), equalTo(expectedStatus));
+                try {
-                if (expectedStatus.equals(HttpResponseStatus.CONTINUE)) {
+                    assertThat(response.status(), equalTo(expectedStatus));
-                    final FullHttpRequest continuationRequest =
+                    if (expectedStatus.equals(HttpResponseStatus.CONTINUE)) {
-                        new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/", Unpooled.EMPTY_BUFFER);
+                        final FullHttpRequest continuationRequest =
-                    final FullHttpResponse continuationResponse = client.post(remoteAddress.address(), continuationRequest);
+                            new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.POST, "/", Unpooled.EMPTY_BUFFER);
-
+                        final FullHttpResponse continuationResponse = client.post(remoteAddress.address(), continuationRequest);
-                    assertThat(continuationResponse.status(), is(HttpResponseStatus.OK));
+                        try {
-                    assertThat(new String(ByteBufUtil.getBytes(continuationResponse.content()), StandardCharsets.UTF_8), is("done"));
+                            assertThat(continuationResponse.status(), is(HttpResponseStatus.OK));
+                            assertThat(
+                                new String(ByteBufUtil.getBytes(continuationResponse.content()), StandardCharsets.UTF_8), is("done")
+                            );
+                        } finally {
+                            continuationResponse.release();
+                        }
+                    }
+                } finally {
+                    response.release();
                 }
             }
         }
@@ -280,10 +289,14 @@ public class Netty4HttpServerTransportTests extends ESTestCase {
                 final FullHttpRequest request = new DefaultFullHttpRequest(HttpVersion.HTTP_1_1, HttpMethod.GET, url);
 
                 final FullHttpResponse response = client.post(remoteAddress.address(), request);
-                assertThat(response.status(), equalTo(HttpResponseStatus.BAD_REQUEST));
+                try {
-                assertThat(
+                    assertThat(response.status(), equalTo(HttpResponseStatus.BAD_REQUEST));
+                    assertThat(
                         new String(response.content().array(), Charset.forName("UTF-8")),
                         containsString("you sent a bad request and you should feel bad"));
+                } finally {
+                    response.release();
+                }
             }
         }
 

modules/transport-netty4/src/test/java/org/elasticsearch/http/netty4/Netty4PipeliningIT.java

@@ -20,6 +20,7 @@
 package org.elasticsearch.http.netty4;
 
 import io.netty.handler.codec.http.FullHttpResponse;
+import io.netty.util.ReferenceCounted;
 import org.elasticsearch.ESNetty4IntegTestCase;
 import org.elasticsearch.common.transport.TransportAddress;
 import org.elasticsearch.http.HttpServerTransport;
@@ -45,14 +46,18 @@ public class Netty4PipeliningIT extends ESNetty4IntegTestCase {
 
         HttpServerTransport httpServerTransport = internalCluster().getInstance(HttpServerTransport.class);
         TransportAddress[] boundAddresses = httpServerTransport.boundAddress().boundAddresses();
-        TransportAddress transportAddress = (TransportAddress) randomFrom(boundAddresses);
+        TransportAddress transportAddress = randomFrom(boundAddresses);
 
         try (Netty4HttpClient nettyHttpClient = new Netty4HttpClient()) {
             Collection<FullHttpResponse> responses = nettyHttpClient.get(transportAddress.address(), requests);
-            assertThat(responses, hasSize(5));
+            try {
+                assertThat(responses, hasSize(5));
 
-            Collection<String> opaqueIds = Netty4HttpClient.returnOpaqueIds(responses);
+                Collection<String> opaqueIds = Netty4HttpClient.returnOpaqueIds(responses);
-            assertOpaqueIdsInOrder(opaqueIds);
+                assertOpaqueIdsInOrder(opaqueIds);
+            } finally {
+                responses.forEach(ReferenceCounted::release);
+            }
         }
     }
 

plugins/transport-nio/build.gradle

@@ -29,13 +29,13 @@ dependencies {
     compile "org.elasticsearch:elasticsearch-nio:${version}"
 
     // network stack
-    compile "io.netty:netty-buffer:4.1.16.Final"
+    compile "io.netty:netty-buffer:4.1.28.Final"
-    compile "io.netty:netty-codec:4.1.16.Final"
+    compile "io.netty:netty-codec:4.1.28.Final"
-    compile "io.netty:netty-codec-http:4.1.16.Final"
+    compile "io.netty:netty-codec-http:4.1.28.Final"
-    compile "io.netty:netty-common:4.1.16.Final"
+    compile "io.netty:netty-common:4.1.28.Final"
-    compile "io.netty:netty-handler:4.1.16.Final"
+    compile "io.netty:netty-handler:4.1.28.Final"
-    compile "io.netty:netty-resolver:4.1.16.Final"
+    compile "io.netty:netty-resolver:4.1.28.Final"
-    compile "io.netty:netty-transport:4.1.16.Final"
+    compile "io.netty:netty-transport:4.1.28.Final"
 }
 
 dependencyLicenses {
@@ -113,7 +113,6 @@ thirdPartyAudit.excludes = [
         'net.jpountz.xxhash.StreamingXXHash32',
         'net.jpountz.xxhash.XXHashFactory',
         'io.netty.internal.tcnative.CertificateRequestedCallback',
-        'io.netty.internal.tcnative.CertificateRequestedCallback$KeyMaterial',
         'io.netty.internal.tcnative.CertificateVerifier',
         'io.netty.internal.tcnative.SessionTicketKey',
         'io.netty.internal.tcnative.SniHostNameMatcher',
@@ -140,6 +139,6 @@ thirdPartyAudit.excludes = [
 
         'org.conscrypt.AllocatedBuffer',
         'org.conscrypt.BufferAllocator',
-        'org.conscrypt.Conscrypt$Engines',
+        'org.conscrypt.Conscrypt',
         'org.conscrypt.HandshakeListener'
 ]

plugins/transport-nio/licenses/netty-buffer-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-63b5fa95c74785e16f2c30ce268bc222e35c8cb5

plugins/transport-nio/licenses/netty-buffer-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+d6c2d13492778009d33f60e05ed90bcb535d1fd1

plugins/transport-nio/licenses/netty-codec-4.1.16.Final.jar.sha1

@@ -1 +0,0 @@
-d84a1f21768b7309c2954521cf5a1f46c2309eb1

plugins/transport-nio/licenses/netty-codec-4.1.28.Final.jar.sha1

@@ -0,0 +1 @@
+a38361d893900947524f8a9da980555950e73d6a