honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
OpenSearch/modules/transport-netty4/build.gradle

209 lines
8.0 KiB
Groovy
Raw Normal View History

Introduce Netty 4 This commit adds transport-netty4, a transport and HTTP implementation based on Netty 4. Relates #19526
2016-07-22 22:26:35 -04:00
/*
* Licensed to Elasticsearch under one or more contributor
* license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright
* ownership. Elasticsearch licenses this file to you under
* the Apache License, Version 2.0 (the "License"); you may
* not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
[7.x] Introduce type-safe and consistent pattern for handling build globals (#48818) This commit introduces a consistent, and type-safe manner for handling global build parameters through out our build logic. Primarily this replaces the existing usages of extra properties with static accessors. It also introduces and explicit API for initialization and mutation of any such parameters, as well as better error handling for uninitialized or eager access of parameter values. Closes #42042
2019-11-01 14:33:11 -04:00
import org.elasticsearch.gradle.info.BuildParams
Add test tasks for unpooled and direct buffer pooling to netty (#46049) Some netty behavior is controlled by system properties. While we want to test with the defaults for Elasticsearch for most tests, within netty we want to ensure these netty settings exhibit correct behavior. This commit adds variants of test and integTest tasks for netty which set the unpooled and direct buffer pooled allocators. relates #45881
2019-08-30 14:37:18 -04:00
import org.elasticsearch.gradle.test.RestIntegTestTask
Convert modules to use yamlRestTest (#59089) (#59446) This commit moves the modules REST tests to the newly introduced yamlRestTest source set. A few tests have also been re-named to include the correct IT suffix. Without changing the names, the testing conventions task would fail since now that the YAML tests are no longer present pacify the convention. These tests have moved to the internalClusterTest source set. related: #56841
2020-07-13 14:53:05 -04:00
apply plugin: 'elasticsearch.yaml-rest-test'
7.x - Create plugin for yamlTest task (#56841) (#59090) This commit creates a new Gradle plugin to provide a separate task name and source set for running YAML based REST tests. The only project converted to use the new plugin in this PR is distribution/archives/integ-test-zip. For which the testing has been moved to :rest-api-spec since it makes the most sense and it avoids a small but awkward change to the distribution plugin. The remaining cases in modules, plugins, and x-pack will be handled in followups. This plugin is distinctly different from the plugin introduced in #55896 since the YAML REST tests are intended to be black box tests over HTTP. As such they should not (by default) have access to the classpath for that which they are testing. The YAML based REST tests will be moved to separate source sets (yamlRestTest). The which source is the target for the test resources is dependent on if this new plugin is applied. If it is not applied, it will default to the test source set. Further, this introduces a breaking change for plugin developers that use the YAML testing framework. They will now need to either use the new source set and matching task, or configure the rest resources to use the old "test" source set that matches the old integTest task. (The former should be preferred). As part of this change (which is also breaking for plugin developers) the rest resources plugin has been removed from the build plugin and now requires either explicit application or application via the new YAML REST test plugin. Plugin developers should be able to fix the breaking changes to the YAML tests by adding apply plugin: 'elasticsearch.yaml-rest-test' and moving the YAML tests under a yamlRestTest folder (instead of test)
2020-07-06 15:16:26 -04:00
Introduce Netty 4 This commit adds transport-netty4, a transport and HTTP implementation based on Netty 4. Relates #19526
2016-07-22 22:26:35 -04:00
/*
TODOs:
* fix permissions such that only netty4 can open sockets etc?
* fix the hack in the build framework that copies transport-netty4 into the integ test cluster
* maybe figure out a way to run all tests from core with netty4/network?
*/
esplugin {
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
description 'Netty 4 based transport implementation'
classname 'org.elasticsearch.transport.Netty4Plugin'
hasClientJar = true
Introduce Netty 4 This commit adds transport-netty4, a transport and HTTP implementation based on Netty 4. Relates #19526
2016-07-22 22:26:35 -04:00
}
dependencies {
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
// network stack
Replace compile configuration usage with api (7.x backport) (#58721) * Replace compile configuration usage with api (#58451) - Use java-library instead of plugin to allow api configuration usage - Remove explicit references to runtime configurations in dependency declarations - Make test runtime classpath input for testing convention - required as java library will by default not have build jar file - jar file is now explicit input of the task and gradle will ensure its properly build * Fix compile usages in 7.x branch
2020-06-30 09:57:41 -04:00
api "io.netty:netty-buffer:${versions.netty}"
api "io.netty:netty-codec:${versions.netty}"
api "io.netty:netty-codec-http:${versions.netty}"
api "io.netty:netty-common:${versions.netty}"
api "io.netty:netty-handler:${versions.netty}"
api "io.netty:netty-resolver:${versions.netty}"
api "io.netty:netty-transport:${versions.netty}"
Introduce Netty 4 This commit adds transport-netty4, a transport and HTTP implementation based on Netty 4. Relates #19526
2016-07-22 22:26:35 -04:00
}
[7.x] Optimize which Rest resources are used by the Rest tests… (#53766) This should help with Gradle's incremental compile such that projects only depend upon the resources they use. related #52114
2020-03-19 13:28:59 -04:00
restResources {
restApi {
includeCore '_common', 'cluster', 'nodes'
}
}
Unify dependency licenses task configuration (#58116) (#58274) - Remove duplicate dependency configuration - Use task avoidance api accross the build - Remove redundant licensesCheck config
2020-06-18 02:15:50 -04:00
tasks.named("dependencyLicenses").configure {
Build: Remove extra copies of netty license (#23361) The dependencyLicenses check has the ability to map multiple jar files to the same license file. However, netty was not taking advantage of this, and had duplicate copies of its license/notice files for each jar. This commit reduces the copies to one and uses the mapping feature.
2017-02-24 17:40:07 -05:00
mapping from: /netty-.*/, to: 'netty'
}
[Backport] Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) (#40993) * Replace usages RandomizedTestingTask with built-in Gradle Test (#40978) This commit replaces the existing RandomizedTestingTask and supporting code with Gradle's built-in JUnit support via the Test task type. Additionally, the previous workaround to disable all tasks named "test" and create new unit testing tasks named "unitTest" has been removed such that the "test" task now runs unit tests as per the normal Gradle Java plugin conventions. (cherry picked from commit 323f312bbc829a63056a79ebe45adced5099f6e6) * Fix forking JVM runner * Don't bump shadow plugin version
2019-04-09 14:52:50 -04:00
test {
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
/*
* We have to disable setting the number of available processors as tests in the same JVM randomize processors and will step on each
* other if we allow them to set the number of available processors as it's set-once in Netty.
*/
systemProperty 'es.set.netty.runtime.available.processors', 'false'
Set available processors for Netty Netty uses the number of processors for sizing various resources (e.g., thread pools, buffer pools, etc.). However, it uses the runtime number of available processors which might not match the configured number of processors as set in Elasticsearch to limit the number of threads (for example, in Docker containers). A new feature was added to Netty that enables configuring the number of processors Netty should see for sizing this various resources. This commit takes advantage of this feature to set this number of available processors to be equal to the configured number of processors set in Elasticsearch. Relates #24420
2017-05-01 19:27:28 -04:00
}
integTestRunner {
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
/*
* We have to disable setting the number of available processors as tests in the same JVM randomize processors and will step on each
* other if we allow them to set the number of available processors as it's set-once in Netty.
*/
systemProperty 'es.set.netty.runtime.available.processors', 'false'
Set available processors for Netty Netty uses the number of processors for sizing various resources (e.g., thread pools, buffer pools, etc.). However, it uses the runtime number of available processors which might not match the configured number of processors as set in Elasticsearch to limit the number of threads (for example, in Docker containers). A new feature was added to Netty that enables configuring the number of processors Netty should see for sizing this various resources. This commit takes advantage of this feature to set this number of available processors to be equal to the configured number of processors set in Elasticsearch. Relates #24420
2017-05-01 19:27:28 -04:00
}
Add test tasks for unpooled and direct buffer pooling to netty (#46049) Some netty behavior is controlled by system properties. While we want to test with the defaults for Elasticsearch for most tests, within netty we want to ensure these netty settings exhibit correct behavior. This commit adds variants of test and integTest tasks for netty which set the unpooled and direct buffer pooled allocators. relates #45881
2019-08-30 14:37:18 -04:00
TaskProvider<Test> pooledTest = tasks.register("pooledTest", Test) {
include '**/*Tests.class'
systemProperty 'es.set.netty.runtime.available.processors', 'false'
Remove option to enable direct buffer pooling (#48310) This commit removes the option to change the netty system properties to reenable the direct buffer pooling. It also removes the need for us to disable the buffer pooling in the system properties file. Instead, we programmatically craete an allocator that is used by our networking layer. This commit does introduce an Elasticsearch property which allows the user to fallback on the netty default allocator. If they choose this option, they can configure the default allocator how they wish using the standard netty properties.
2019-10-21 21:15:50 -04:00
systemProperty 'es.use_unpooled_allocator', 'false'
Add test tasks for unpooled and direct buffer pooling to netty (#46049) Some netty behavior is controlled by system properties. While we want to test with the defaults for Elasticsearch for most tests, within netty we want to ensure these netty settings exhibit correct behavior. This commit adds variants of test and integTest tasks for netty which set the unpooled and direct buffer pooled allocators. relates #45881
2019-08-30 14:37:18 -04:00
}
Remove random when using HLRC sync and async calls (#48211) This commit removes the randomization used by every execute call in the high level rest tests. Previously every execute call, which can be many calls per single test, would rely on a random boolean to determine if they should use the sync or async methods provided to the execute method. This commit runs the tests twice, using two different clusters, both of them providing the value one time via a sysprop. This ensures that the whole suite of tests is run using the sync and async code paths. Closes #39667
2019-10-24 10:04:57 -04:00
Add test tasks for unpooled and direct buffer pooling to netty (#46049) Some netty behavior is controlled by system properties. While we want to test with the defaults for Elasticsearch for most tests, within netty we want to ensure these netty settings exhibit correct behavior. This commit adds variants of test and integTest tasks for netty which set the unpooled and direct buffer pooled allocators. relates #45881
2019-08-30 14:37:18 -04:00
RestIntegTestTask pooledIntegTest = tasks.create("pooledIntegTest", RestIntegTestTask) {
runner {
systemProperty 'es.set.netty.runtime.available.processors', 'false'
}
}
testClusters.pooledIntegTest {
Remove option to enable direct buffer pooling (#48310) This commit removes the option to change the netty system properties to reenable the direct buffer pooling. It also removes the need for us to disable the buffer pooling in the system properties file. Instead, we programmatically craete an allocator that is used by our networking layer. This commit does introduce an Elasticsearch property which allows the user to fallback on the netty default allocator. If they choose this option, they can configure the default allocator how they wish using the standard netty properties.
2019-10-21 21:15:50 -04:00
systemProperty 'es.use_unpooled_allocator', 'false'
Add test tasks for unpooled and direct buffer pooling to netty (#46049) Some netty behavior is controlled by system properties. While we want to test with the defaults for Elasticsearch for most tests, within netty we want to ensure these netty settings exhibit correct behavior. This commit adds variants of test and integTest tasks for netty which set the unpooled and direct buffer pooled allocators. relates #45881
2019-08-30 14:37:18 -04:00
}
check.dependsOn(pooledTest, pooledIntegTest)
Split third party audit exclusions by type (#36763)
2019-01-07 10:24:19 -05:00
thirdPartyAudit {
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
ignoreMissingClasses(
// classes are missing
// from io.netty.handler.codec.protobuf.ProtobufDecoder (netty)
'com.google.protobuf.ExtensionRegistry',
'com.google.protobuf.MessageLite$Builder',
'com.google.protobuf.MessageLite',
'com.google.protobuf.Parser',
// from io.netty.logging.CommonsLoggerFactory (netty)
'org.apache.commons.logging.Log',
'org.apache.commons.logging.LogFactory',
// from Log4j (deliberate, Netty will fallback to Log4j 2)
'org.apache.log4j.Level',
'org.apache.log4j.Logger',
// from io.netty.handler.ssl.OpenSslEngine (netty)
'io.netty.internal.tcnative.Buffer',
'io.netty.internal.tcnative.Library',
'io.netty.internal.tcnative.SSL',
'io.netty.internal.tcnative.SSLContext',
'io.netty.internal.tcnative.SSLPrivateKeyMethod',
// from io.netty.handler.ssl.util.BouncyCastleSelfSignedCertGenerator (netty)
'org.bouncycastle.cert.X509v3CertificateBuilder',
'org.bouncycastle.cert.jcajce.JcaX509CertificateConverter',
'org.bouncycastle.cert.jcajce.JcaX509v3CertificateBuilder',
'org.bouncycastle.jce.provider.BouncyCastleProvider',
'org.bouncycastle.operator.jcajce.JcaContentSignerBuilder',
Enable tests in FIPS 140 in JDK 11 (#49485) This change changes the way to run our test suites in JVMs configured in FIPS 140 approved mode. It does so by: - Configuring any given runtime Java in FIPS mode with the bundled policy and security properties files, setting the system properties java.security.properties and java.security.policy with the == operator that overrides the default JVM properties and policy. - When runtime java is 11 and higher, using BouncyCastle FIPS Cryptographic provider and BCJSSE in FIPS mode. These are used as testRuntime dependencies for unit tests and internal clusters, and copied (relevant jars) explicitly to the lib directory for testclusters used in REST tests - When runtime java is 8, using BouncyCastle FIPS Cryptographic provider and SunJSSE in FIPS mode. Running the tests in FIPS 140 approved mode doesn't require an additional configuration either in CI workers or locally and is controlled by specifying -Dtests.fips.enabled=true
2020-01-27 04:14:52 -05:00
'org.bouncycastle.asn1.x500.X500Name',
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
// from io.netty.handler.ssl.JettyNpnSslEngine (netty)
'org.eclipse.jetty.npn.NextProtoNego$ClientProvider',
'org.eclipse.jetty.npn.NextProtoNego$ServerProvider',
'org.eclipse.jetty.npn.NextProtoNego',
// from io.netty.handler.codec.marshalling.ChannelBufferByteInput (netty)
'org.jboss.marshalling.ByteInput',
// from io.netty.handler.codec.marshalling.ChannelBufferByteOutput (netty)
'org.jboss.marshalling.ByteOutput',
// from io.netty.handler.codec.marshalling.CompatibleMarshallingEncoder (netty)
'org.jboss.marshalling.Marshaller',
// from io.netty.handler.codec.marshalling.ContextBoundUnmarshallerProvider (netty)
'org.jboss.marshalling.MarshallerFactory',
'org.jboss.marshalling.MarshallingConfiguration',
'org.jboss.marshalling.Unmarshaller',
// from io.netty.util.internal.logging.InternalLoggerFactory (netty) - it's optional
'org.slf4j.helpers.FormattingTuple',
'org.slf4j.helpers.MessageFormatter',
'org.slf4j.Logger',
'org.slf4j.LoggerFactory',
'org.slf4j.spi.LocationAwareLogger',
'com.google.protobuf.ExtensionRegistryLite',
'com.google.protobuf.MessageLiteOrBuilder',
'com.google.protobuf.nano.CodedOutputByteBufferNano',
'com.google.protobuf.nano.MessageNano',
'com.jcraft.jzlib.Deflater',
'com.jcraft.jzlib.Inflater',
'com.jcraft.jzlib.JZlib$WrapperType',
'com.jcraft.jzlib.JZlib',
'com.ning.compress.BufferRecycler',
'com.ning.compress.lzf.ChunkDecoder',
'com.ning.compress.lzf.ChunkEncoder',
Upgrade netty to 4.1.49.Final (#56059)
2020-04-30 17:58:27 -04:00
'com.ning.compress.lzf.LZFChunk',
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
'com.ning.compress.lzf.LZFEncoder',
'com.ning.compress.lzf.util.ChunkDecoderFactory',
'com.ning.compress.lzf.util.ChunkEncoderFactory',
'lzma.sdk.lzma.Encoder',
'net.jpountz.lz4.LZ4Compressor',
'net.jpountz.lz4.LZ4Factory',
'net.jpountz.lz4.LZ4FastDecompressor',
'net.jpountz.xxhash.XXHash32',
'net.jpountz.xxhash.XXHashFactory',
'io.netty.internal.tcnative.CertificateCallback',
'io.netty.internal.tcnative.CertificateVerifier',
'io.netty.internal.tcnative.SessionTicketKey',
'io.netty.internal.tcnative.SniHostNameMatcher',
'io.netty.internal.tcnative.SSL',
'org.eclipse.jetty.alpn.ALPN$ClientProvider',
'org.eclipse.jetty.alpn.ALPN$ServerProvider',
'org.eclipse.jetty.alpn.ALPN',
'org.conscrypt.AllocatedBuffer',
'org.conscrypt.BufferAllocator',
'org.conscrypt.Conscrypt',
Upgrade netty to 4.1.45.Final (#51689) Upgrade netty.
2020-01-31 12:33:23 -05:00
'org.conscrypt.HandshakeListener',
'reactor.blockhound.BlockHound$Builder',
'reactor.blockhound.integration.BlockHoundIntegration'
Apply 2-space indent to all gradle scripts (#49071) Backport of #48849. Update `.editorconfig` to make the Java settings the default for all files, and then apply a 2-space indent to all `*.gradle` files. Then reformat all the files.
2019-11-14 06:01:23 -05:00
)
ignoreViolations(
'io.netty.util.internal.PlatformDependent0',
'io.netty.util.internal.PlatformDependent0$1',
'io.netty.util.internal.PlatformDependent0$2',
'io.netty.util.internal.PlatformDependent0$3',
'io.netty.util.internal.PlatformDependent0$5',
'io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueConsumerNodeRef',
'io.netty.util.internal.shaded.org.jctools.queues.BaseLinkedQueueProducerNodeRef',
'io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueColdProducerFields',
'io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueConsumerFields',
'io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueueProducerFields',
'io.netty.util.internal.shaded.org.jctools.queues.LinkedQueueNode',
'io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueConsumerIndexField',
'io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerIndexField',
'io.netty.util.internal.shaded.org.jctools.queues.MpscArrayQueueProducerLimitField',
'io.netty.util.internal.shaded.org.jctools.util.UnsafeAccess',
'io.netty.util.internal.shaded.org.jctools.util.UnsafeRefArrayAccess',
'io.netty.handler.ssl.util.OpenJdkSelfSignedCertGenerator'
)
Split third party audit exclusions by type (#36763)
2019-01-07 10:24:19 -05:00
}