honeymoose/OpenSearch
1
0
Fork 0
mirror of https://github.com/honeymoose/OpenSearch.git synced 2025-03-09 14:34:43 +00:00
Code Issues Packages Projects Releases Wiki Activity

Set target and source compatibility to 11, required by Lucene 9. (#2407)

Browse Source 
* Set target and source compatibility to 11, required by Lucene 9.

Signed-off-by: dblock <dblock@dblock.org>

* Uncomment commented code in #2321 for killing child processes that uses JDK9+ ProcessInfo.

Signed-off-by: dblock <dblock@dblock.org>

* Set distribution checker target JDK compatibility to 11.

Signed-off-by: dblock <dblock@dblock.org>

* Supress processing warnings.

Signed-off-by: dblock <dblock@dblock.org>
This commit is contained in:
Daniel Doubrovkine (dB.) 2022-03-08 14:48:51 -05:00 committed by GitHub
parent c3712a51b3
commit 044f53630a
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 82 additions and 82 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
DEVELOPER_GUIDE.md
View File

@ -61,7 +61,19 @@ Fork [opensearch-project/OpenSearch](https://github.com/opensearch-project/OpenS
#### JDK 11 #### JDK 11
OpenSearch builds using Java 11 at a minimum. This means you must have a JDK 11 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 11 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-11`. OpenSearch builds using Java 11 at a minimum, using the Adoptium distribution. This means you must have a JDK 11 installed with the environment variable `JAVA_HOME` referencing the path to Java home for your JDK 11 installation, e.g. `JAVA_HOME=/usr/lib/jvm/jdk-11`. This is configured in [buildSrc/build.gradle](buildSrc/build.gradle) and [distribution/tools/java-version-checker/build.gradle](distribution/tools/java-version-checker/build.gradle).
```
allprojects {
targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_11
}
```
```
sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_11
```
Download Java 11 from [here](https://adoptium.net/releases.html?variant=openjdk11). Download Java 11 from [here](https://adoptium.net/releases.html?variant=openjdk11).
@ -69,9 +81,18 @@ Download Java 11 from [here](https://adoptium.net/releases.html?variant=openjdk1
To run the full suite of tests, download and install [JDK 14](https://jdk.java.net/archive/) and set `JAVA11_HOME`, and `JAVA14_HOME`. They are required by the [backwards compatibility test](./TESTING.md#testing-backwards-compatibility). To run the full suite of tests, download and install [JDK 14](https://jdk.java.net/archive/) and set `JAVA11_HOME`, and `JAVA14_HOME`. They are required by the [backwards compatibility test](./TESTING.md#testing-backwards-compatibility).
#### Runtime JDK #### JDK 17
By default, the test tasks use bundled JDK runtime, configured in `buildSrc/version.properties` and set to JDK 17 (LTS). Other kind of test tasks (integration, cluster, ... ) use the same runtime as `JAVA_HOME`. However, the build supports compiling with JDK 11 and testing on a different version of JDK runtime. To do this, set `RUNTIME_JAVA_HOME` pointing to the Java home of another JDK installation, e.g. `RUNTIME_JAVA_HOME=/usr/lib/jvm/jdk-14`. Alternatively, the runtime JDK version could be provided as the command line argument, using combination of `runtime.java=<major JDK version>` property and `JAVA<major JDK version>_HOME` environment variable, for example `./gradlew -Druntime.java=17 ...` (in this case, the tooling expects `JAVA17_HOME` environment variable to be set). By default, the test tasks use bundled JDK runtime, configured in [buildSrc/version.properties](buildSrc/version.properties), and set to JDK 17 (LTS).
```
bundled_jdk_vendor = adoptium
bundled_jdk = 17.0.2+8
```
#### Custom Runtime JDK
Other kind of test tasks (integration, cluster, etc.) use the same runtime as `JAVA_HOME`. However, the build also supports compiling with one version of JDK, and testing on a different version. To do this, set `RUNTIME_JAVA_HOME` pointing to the Java home of another JDK installation, e.g. `RUNTIME_JAVA_HOME=/usr/lib/jvm/jdk-14`. Alternatively, the runtime JDK version could be provided as the command line argument, using combination of `runtime.java=<major JDK version>` property and `JAVA<major JDK version>_HOME` environment variable, for example `./gradlew -Druntime.java=17 ...` (in this case, the tooling expects `JAVA17_HOME` environment variable to be set).
#### Windows #### Windows

2
build.gradle
View File

@ -244,7 +244,7 @@ allprojects {
compile.options.compilerArgs << '-Xlint:opens' compile.options.compilerArgs << '-Xlint:opens'
compile.options.compilerArgs << '-Xlint:overloads' compile.options.compilerArgs << '-Xlint:overloads'
compile.options.compilerArgs << '-Xlint:overrides' compile.options.compilerArgs << '-Xlint:overrides'
compile.options.compilerArgs << '-Xlint:processing' compile.options.compilerArgs << '-Xlint:-processing'
compile.options.compilerArgs << '-Xlint:rawtypes' compile.options.compilerArgs << '-Xlint:rawtypes'
compile.options.compilerArgs << '-Xlint:removal' compile.options.compilerArgs << '-Xlint:removal'
compile.options.compilerArgs << '-Xlint:requires-automatic' compile.options.compilerArgs << '-Xlint:requires-automatic'

4
buildSrc/build.gradle
View File

@ -158,8 +158,8 @@ if (project != rootProject) {
apply plugin: 'opensearch.publish' apply plugin: 'opensearch.publish'
allprojects { allprojects {
targetCompatibility = 8 targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = 8 sourceCompatibility = JavaVersion.VERSION_11
} }
// groovydoc succeeds, but has some weird internal exception... // groovydoc succeeds, but has some weird internal exception...

29
buildSrc/src/main/java/org/opensearch/gradle/ReaperPlugin.java
View File

@ -36,7 +36,6 @@ import org.opensearch.gradle.info.GlobalBuildInfoPlugin;
import org.gradle.api.Plugin; import org.gradle.api.Plugin;
import org.gradle.api.Project; import org.gradle.api.Project;
import java.lang.management.ManagementFactory;
import java.nio.file.Path; import java.nio.file.Path;
/** /**
@ -52,31 +51,15 @@ public class ReaperPlugin implements Plugin<Project> {
project.getPlugins().apply(GlobalBuildInfoPlugin.class); project.getPlugins().apply(GlobalBuildInfoPlugin.class);
Path inputDir = project.getRootDir().toPath().resolve(".gradle").resolve("reaper").resolve("build-" + getProcessId("xx")); Path inputDir = project.getRootDir()
.toPath()
.resolve(".gradle")
.resolve("reaper")
.resolve("build-" + ProcessHandle.current().pid());
ReaperService service = project.getExtensions() ReaperService service = project.getExtensions()
.create("reaper", ReaperService.class, project, project.getBuildDir().toPath(), inputDir); .create("reaper", ReaperService.class, project, project.getBuildDir().toPath(), inputDir);
project.getGradle().buildFinished(result -> service.shutdown()); project.getGradle().buildFinished(result -> service.shutdown());
} }
private static String getProcessId(final String fallback) {
// Note: may fail in some JVM implementations
// therefore fallback has to be provided
// something like '<pid>@<hostname>', at least in SUN / Oracle JVMs
final String jvmName = ManagementFactory.getRuntimeMXBean().getName();
final int index = jvmName.indexOf('@');
if (index < 1) {
// part before '@' empty (index = 0) / '@' not found (index = -1)
return fallback;
}
try {
return Long.toString(Long.parseLong(jvmName.substring(0, index)));
} catch (NumberFormatException e) {
// ignore
}
return fallback;
}
} }

6
buildSrc/src/main/java/org/opensearch/gradle/ReaperService.java
View File

@ -179,11 +179,7 @@ public class ReaperService {
InputStream jarInput = this.getClass().getResourceAsStream("/META-INF/reaper.jar"); InputStream jarInput = this.getClass().getResourceAsStream("/META-INF/reaper.jar");
) { ) {
logger.info("Copying reaper.jar..."); logger.info("Copying reaper.jar...");
byte[] buffer = new byte[4096]; jarInput.transferTo(out);
int len;
while ((len = jarInput.read(buffer)) > 0) {
out.write(buffer, 0, len);
}
} catch (IOException e) { } catch (IOException e) {
throw new UncheckedIOException(e); throw new UncheckedIOException(e);
} }

56
buildSrc/src/main/java/org/opensearch/gradle/testclusters/OpenSearchNode.java
View File

@ -93,7 +93,9 @@ import java.util.List;
import java.util.Map; import java.util.Map;
import java.util.Objects; import java.util.Objects;
import java.util.Set; import java.util.Set;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeUnit;
import java.util.concurrent.TimeoutException;
import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicBoolean;
import java.util.function.BiConsumer; import java.util.function.BiConsumer;
import java.util.function.Function; import java.util.function.Function;
@ -908,7 +910,7 @@ public class OpenSearchNode implements TestClusterConfiguration {
} catch (IOException e) { } catch (IOException e) {
throw new TestClustersException("Failed to start " + currentConfig.command + " process for " + this, e); throw new TestClustersException("Failed to start " + currentConfig.command + " process for " + this, e);
} }
// reaper.registerPid(toString(), opensearchProcess.pid()); reaper.registerPid(toString(), opensearchProcess.pid());
} }
@Internal @Internal
@ -975,7 +977,7 @@ public class OpenSearchNode implements TestClusterConfiguration {
LOGGER.info("Stopping `{}`, tailLogs: {}", this, tailLogs); LOGGER.info("Stopping `{}`, tailLogs: {}", this, tailLogs);
requireNonNull(opensearchProcess, "Can't stop `" + this + "` as it was not started or already stopped."); requireNonNull(opensearchProcess, "Can't stop `" + this + "` as it was not started or already stopped.");
// Test clusters are not reused, don't spend time on a graceful shutdown // Test clusters are not reused, don't spend time on a graceful shutdown
stopHandle(opensearchProcess, true); stopProcess(opensearchProcess.toHandle(), true);
reaper.unregister(toString()); reaper.unregister(toString());
if (tailLogs) { if (tailLogs) {
logFileContents("Standard output of node", currentConfig.stdoutFile); logFileContents("Standard output of node", currentConfig.stdoutFile);
@ -1000,9 +1002,9 @@ public class OpenSearchNode implements TestClusterConfiguration {
this.nameCustomization = nameCustomizer; this.nameCustomization = nameCustomizer;
} }
private void stopHandle(Process process, boolean forcibly) { private void stopProcess(ProcessHandle processHandle, boolean forcibly) {
// No-op if the process has already exited by itself. // No-op if the process has already exited by itself.
if (process.isAlive() == false) { if (processHandle.isAlive() == false) {
LOGGER.info("Process was not running when we tried to terminate it."); LOGGER.info("Process was not running when we tried to terminate it.");
return; return;
} }
@ -1011,19 +1013,19 @@ public class OpenSearchNode implements TestClusterConfiguration {
// they'll be recorded as having failed and won't restart when the cluster restarts. // they'll be recorded as having failed and won't restart when the cluster restarts.
// ES could actually be a child when there's some wrapper process like on Windows, // ES could actually be a child when there's some wrapper process like on Windows,
// and in that case the ML processes will be grandchildren of the wrapper. // and in that case the ML processes will be grandchildren of the wrapper.
// List<Process> children = process.children().collect(Collectors.toList()); List<ProcessHandle> children = processHandle.children().collect(Collectors.toList());
try { try {
// logProcessInfo( logProcessInfo(
// "Terminating " + currentConfig.command + " process" + (forcibly ? " forcibly " : "gracefully") + ":", "Terminating " + currentConfig.command + " process" + (forcibly ? " forcibly " : "gracefully") + ":",
// process.info() processHandle.info()
// ); );
if (forcibly) { if (forcibly) {
process.destroyForcibly(); processHandle.destroyForcibly();
} else { } else {
process.destroy(); processHandle.destroy();
waitForProcessToExit(process); waitForProcessToExit(processHandle);
if (process.isAlive() == false) { if (processHandle.isAlive() == false) {
return; return;
} }
LOGGER.info( LOGGER.info(
@ -1031,25 +1033,24 @@ public class OpenSearchNode implements TestClusterConfiguration {
OPENSEARCH_DESTROY_TIMEOUT, OPENSEARCH_DESTROY_TIMEOUT,
OPENSEARCH_DESTROY_TIMEOUT_UNIT OPENSEARCH_DESTROY_TIMEOUT_UNIT
); );
process.destroyForcibly(); processHandle.destroyForcibly();
} }
waitForProcessToExit(process); waitForProcessToExit(processHandle);
if (process.isAlive()) { if (processHandle.isAlive()) {
throw new TestClustersException("Was not able to terminate " + currentConfig.command + " process for " + this); throw new TestClustersException("Was not able to terminate " + currentConfig.command + " process for " + this);
} }
} finally { } finally {
// children.forEach(each -> stopHandle(each, forcibly)); children.forEach(each -> stopProcess(each, forcibly));
} }
// waitForProcessToExit(process); waitForProcessToExit(processHandle);
// if (process.isAlive()) { if (processHandle.isAlive()) {
// throw new TestClustersException("Was not able to terminate " + currentConfig.command + " process for " + this); throw new TestClustersException("Was not able to terminate " + currentConfig.command + " process for " + this);
// } }
} }
/* private void logProcessInfo(String prefix, ProcessHandle.Info info) {
private void logProcessInfo(String prefix, Process info) {
LOGGER.info( LOGGER.info(
prefix + " commandLine:`{}` command:`{}` args:`{}`", prefix + " commandLine:`{}` command:`{}` args:`{}`",
info.commandLine().orElse("-"), info.commandLine().orElse("-"),
@ -1057,7 +1058,6 @@ public class OpenSearchNode implements TestClusterConfiguration {
Arrays.stream(info.arguments().orElse(new String[] {})).map(each -> "'" + each + "'").collect(Collectors.joining(" ")) Arrays.stream(info.arguments().orElse(new String[] {})).map(each -> "'" + each + "'").collect(Collectors.joining(" "))
); );
} }
*/
private void logFileContents(String description, Path from) { private void logFileContents(String description, Path from) {
final Map<String, Integer> errorsAndWarnings = new LinkedHashMap<>(); final Map<String, Integer> errorsAndWarnings = new LinkedHashMap<>();
@ -1126,14 +1126,16 @@ public class OpenSearchNode implements TestClusterConfiguration {
return line; return line;
} }
private void waitForProcessToExit(Process process) { private void waitForProcessToExit(ProcessHandle processHandle) {
try { try {
process.waitFor(OPENSEARCH_DESTROY_TIMEOUT, OPENSEARCH_DESTROY_TIMEOUT_UNIT); processHandle.onExit().get(OPENSEARCH_DESTROY_TIMEOUT, OPENSEARCH_DESTROY_TIMEOUT_UNIT);
} catch (InterruptedException e) { } catch (InterruptedException e) {
LOGGER.info("Interrupted while waiting for {} process", currentConfig.command, e); LOGGER.info("Interrupted while waiting for {} process", currentConfig.command, e);
Thread.currentThread().interrupt(); Thread.currentThread().interrupt();
} catch (NullPointerException e) { } catch (ExecutionException e) {
LOGGER.info("Failure while waiting for process to exist", e); LOGGER.info("Failure while waiting for process to exist", e);
} catch (TimeoutException e) {
LOGGER.info("Timed out waiting for process to exit", e);
} }
} }

6
buildSrc/src/main/java/org/opensearch/gradle/transform/SymbolicLinkPreservingUntarTransform.java
View File

@ -84,11 +84,7 @@ public abstract class SymbolicLinkPreservingUntarTransform implements UnpackTran
// copy the file from the archive using a small buffer to avoid heaping // copy the file from the archive using a small buffer to avoid heaping
Files.createFile(destination); Files.createFile(destination);
try (FileOutputStream fos = new FileOutputStream(destination.toFile())) { try (FileOutputStream fos = new FileOutputStream(destination.toFile())) {
byte[] buffer = new byte[4096]; tar.transferTo(fos);
int len;
while ((len = tar.read(buffer)) > 0) {
fos.write(buffer, 0, len);
}
} }
} }
if (entry.isSymbolicLink() == false) { if (entry.isSymbolicLink() == false) {

3
buildSrc/src/main/java/org/opensearch/gradle/vagrant/VagrantBasePlugin.java
View File

@ -42,6 +42,7 @@ import org.gradle.api.execution.TaskExecutionListener;
import org.gradle.api.tasks.TaskState; import org.gradle.api.tasks.TaskState;
import java.io.ByteArrayOutputStream; import java.io.ByteArrayOutputStream;
import java.nio.charset.StandardCharsets;
import java.util.List; import java.util.List;
import java.util.function.Consumer; import java.util.function.Consumer;
import java.util.regex.Matcher; import java.util.regex.Matcher;
@ -101,7 +102,7 @@ public class VagrantBasePlugin implements Plugin<Project> {
spec.setCommandLine(tool, "--version"); spec.setCommandLine(tool, "--version");
spec.setStandardOutput(pipe); spec.setStandardOutput(pipe);
}); });
String output = pipe.toString().trim(); String output = pipe.toString(StandardCharsets.UTF_8).trim();
Matcher matcher = versionRegex.matcher(output); Matcher matcher = versionRegex.matcher(output);
if (matcher.find() == false) { if (matcher.find() == false) {
throw new IllegalStateException( throw new IllegalStateException(

2
buildSrc/src/main/resources/minimumRuntimeVersion
View File

@ -1 +1 @@
1.8 11

4
client/rest/build.gradle
View File

@ -33,8 +33,8 @@ import de.thetaphi.forbiddenapis.gradle.CheckForbiddenApis
apply plugin: 'opensearch.build' apply plugin: 'opensearch.build'
apply plugin: 'opensearch.publish' apply plugin: 'opensearch.publish'
targetCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_11
group = 'org.opensearch.client' group = 'org.opensearch.client'
archivesBaseName = 'opensearch-rest-client' archivesBaseName = 'opensearch-rest-client'

4
client/sniffer/build.gradle
View File

@ -30,8 +30,8 @@
apply plugin: 'opensearch.build' apply plugin: 'opensearch.build'
apply plugin: 'opensearch.publish' apply plugin: 'opensearch.publish'
targetCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_11
group = 'org.opensearch.client' group = 'org.opensearch.client'
archivesBaseName = 'opensearch-rest-client-sniffer' archivesBaseName = 'opensearch-rest-client-sniffer'

4
client/test/build.gradle
View File

@ -29,8 +29,8 @@
*/ */
apply plugin: 'opensearch.build' apply plugin: 'opensearch.build'
targetCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_11
sourceCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_11
group = "${group}.client.test" group = "${group}.client.test"

5
distribution/tools/java-version-checker/build.gradle
View File

@ -11,8 +11,9 @@
apply plugin: 'opensearch.build' apply plugin: 'opensearch.build'
sourceCompatibility = JavaVersion.VERSION_1_8 sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = JavaVersion.VERSION_1_8 targetCompatibility = JavaVersion.VERSION_11
// targetting very old java versions enables a warning by default on newer JDK: disable it. // targetting very old java versions enables a warning by default on newer JDK: disable it.
compileJava.options.compilerArgs += '-Xlint:-options' compileJava.options.compilerArgs += '-Xlint:-options'

6
libs/core/build.gradle
View File

@ -54,13 +54,13 @@ if (!isEclipse) {
} }
compileJava11Java { compileJava11Java {
sourceCompatibility = 11 sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = 11 targetCompatibility = JavaVersion.VERSION_11
} }
forbiddenApisJava11 { forbiddenApisJava11 {
if (BuildParams.runtimeJavaVersion < JavaVersion.VERSION_11) { if (BuildParams.runtimeJavaVersion < JavaVersion.VERSION_11) {
targetCompatibility = JavaVersion.VERSION_11.getMajorVersion() targetCompatibility = JavaVersion.VERSION_11
} }
replaceSignatureFiles 'jdk-signatures' replaceSignatureFiles 'jdk-signatures'
} }

6
server/build.gradle
View File

@ -64,14 +64,14 @@ if (!isEclipse) {
} }
compileJava11Java { compileJava11Java {
sourceCompatibility = 11 sourceCompatibility = JavaVersion.VERSION_11
targetCompatibility = 11 targetCompatibility = JavaVersion.VERSION_11
} }
tasks.named('forbiddenApisJava11').configure { tasks.named('forbiddenApisJava11').configure {
doFirst { doFirst {
if (BuildParams.runtimeJavaVersion < JavaVersion.VERSION_11) { if (BuildParams.runtimeJavaVersion < JavaVersion.VERSION_11) {
targetCompatibility = JavaVersion.VERSION_11.getMajorVersion() targetCompatibility = JavaVersion.VERSION_11
} }
} }
} }