Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 15:38:09 -04:00
|
|
|
/*
|
|
|
|
* Licensed to the Apache Software Foundation (ASF) under one
|
|
|
|
* or more contributor license agreements. See the NOTICE file
|
|
|
|
* distributed with this work for additional information
|
|
|
|
* regarding copyright ownership. The ASF licenses this file
|
|
|
|
* to you under the Apache License, Version 2.0 (the
|
|
|
|
* "License"); you may not use this file except in compliance
|
|
|
|
* with the License. You may obtain a copy of the License at
|
|
|
|
*
|
|
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
*
|
|
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
* See the License for the specific language governing permissions and
|
|
|
|
* limitations under the License.
|
|
|
|
*/
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
import * as playwright from 'playwright-core';
|
|
|
|
|
|
|
|
import { selectSuggestibleInput } from '../../../util/playwright';
|
2020-10-02 02:59:21 -04:00
|
|
|
import { getLabeledInput } from '../../../util/playwright';
|
2020-10-01 18:14:41 -04:00
|
|
|
import { setLabeledInput } from '../../../util/playwright';
|
|
|
|
|
|
|
|
/* tslint:disable max-classes-per-file */
|
|
|
|
|
Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 15:38:09 -04:00
|
|
|
/**
|
|
|
|
* Possible values for partition step segment granularity.
|
|
|
|
*/
|
|
|
|
export enum SegmentGranularity {
|
|
|
|
HOUR = 'HOUR',
|
|
|
|
DAY = 'DAY',
|
|
|
|
MONTH = 'MONTH',
|
|
|
|
YEAR = 'YEAR',
|
|
|
|
}
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
const PARTITIONING_TYPE = 'Partitioning type';
|
|
|
|
|
|
|
|
export interface PartitionsSpec {
|
|
|
|
readonly type: string;
|
|
|
|
apply(page: playwright.Page): Promise<void>;
|
|
|
|
}
|
|
|
|
|
2020-10-02 02:59:21 -04:00
|
|
|
export async function readPartitionSpec(page: playwright.Page): Promise<PartitionsSpec | null> {
|
|
|
|
const type = await getLabeledInput(page, PARTITIONING_TYPE);
|
|
|
|
switch (type) {
|
|
|
|
case HashedPartitionsSpec.TYPE:
|
|
|
|
return HashedPartitionsSpec.read(page);
|
|
|
|
case SingleDimPartitionsSpec.TYPE:
|
|
|
|
return SingleDimPartitionsSpec.read(page);
|
|
|
|
}
|
|
|
|
return null;
|
|
|
|
}
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
export class HashedPartitionsSpec implements PartitionsSpec {
|
2020-10-02 02:59:21 -04:00
|
|
|
public static TYPE = 'hashed';
|
|
|
|
private static NUM_SHARDS = 'Num shards';
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
readonly type: string;
|
|
|
|
|
2020-10-02 02:59:21 -04:00
|
|
|
static async read(page: playwright.Page): Promise<HashedPartitionsSpec> {
|
|
|
|
const numShards = await getLabeledInputAsNumber(page, HashedPartitionsSpec.NUM_SHARDS);
|
|
|
|
return new HashedPartitionsSpec({ numShards });
|
|
|
|
}
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
constructor(props: HashedPartitionsSpecProps) {
|
|
|
|
Object.assign(this, props);
|
2020-10-02 02:59:21 -04:00
|
|
|
this.type = HashedPartitionsSpec.TYPE;
|
2020-10-01 18:14:41 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
async apply(page: playwright.Page): Promise<void> {
|
|
|
|
await setLabeledInput(page, PARTITIONING_TYPE, this.type);
|
|
|
|
if (this.numShards != null) {
|
2020-10-02 02:59:21 -04:00
|
|
|
await setLabeledInput(page, HashedPartitionsSpec.NUM_SHARDS, String(this.numShards));
|
2020-10-01 18:14:41 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2020-10-02 02:59:21 -04:00
|
|
|
async function getLabeledInputAsNumber(
|
|
|
|
page: playwright.Page,
|
|
|
|
label: string,
|
|
|
|
): Promise<number | null> {
|
|
|
|
const valueString = await getLabeledInput(page, label);
|
|
|
|
return valueString === '' ? null : Number(valueString);
|
|
|
|
}
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
interface HashedPartitionsSpecProps {
|
|
|
|
readonly numShards: number | null;
|
|
|
|
}
|
|
|
|
|
|
|
|
export interface HashedPartitionsSpec extends HashedPartitionsSpecProps {}
|
|
|
|
|
|
|
|
export class SingleDimPartitionsSpec implements PartitionsSpec {
|
2020-10-02 02:59:21 -04:00
|
|
|
public static TYPE = 'single_dim';
|
|
|
|
private static PARTITION_DIMENSION = 'Partition dimension';
|
|
|
|
private static TARGET_ROWS_PER_SEGMENT = 'Target rows per segment';
|
|
|
|
private static MAX_ROWS_PER_SEGMENT = 'Max rows per segment';
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
readonly type: string;
|
|
|
|
|
2020-10-02 02:59:21 -04:00
|
|
|
static async read(page: playwright.Page): Promise<SingleDimPartitionsSpec> {
|
|
|
|
const partitionDimension = await getLabeledInput(
|
|
|
|
page,
|
|
|
|
SingleDimPartitionsSpec.PARTITION_DIMENSION,
|
|
|
|
);
|
|
|
|
const targetRowsPerSegment = await getLabeledInputAsNumber(
|
|
|
|
page,
|
|
|
|
SingleDimPartitionsSpec.TARGET_ROWS_PER_SEGMENT,
|
|
|
|
);
|
|
|
|
const maxRowsPerSegment = await getLabeledInputAsNumber(
|
|
|
|
page,
|
|
|
|
SingleDimPartitionsSpec.MAX_ROWS_PER_SEGMENT,
|
|
|
|
);
|
|
|
|
return new SingleDimPartitionsSpec({
|
|
|
|
partitionDimension,
|
|
|
|
targetRowsPerSegment,
|
|
|
|
maxRowsPerSegment,
|
|
|
|
});
|
|
|
|
}
|
|
|
|
|
2020-10-01 18:14:41 -04:00
|
|
|
constructor(props: SingleDimPartitionsSpecProps) {
|
|
|
|
Object.assign(this, props);
|
2020-10-02 02:59:21 -04:00
|
|
|
this.type = SingleDimPartitionsSpec.TYPE;
|
2020-10-01 18:14:41 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
async apply(page: playwright.Page): Promise<void> {
|
|
|
|
await selectSuggestibleInput(page, PARTITIONING_TYPE, this.type);
|
2020-10-02 02:59:21 -04:00
|
|
|
await setLabeledInput(
|
|
|
|
page,
|
|
|
|
SingleDimPartitionsSpec.PARTITION_DIMENSION,
|
|
|
|
this.partitionDimension,
|
|
|
|
);
|
2020-10-01 18:14:41 -04:00
|
|
|
if (this.targetRowsPerSegment) {
|
2020-10-02 02:59:21 -04:00
|
|
|
await setLabeledInput(
|
|
|
|
page,
|
|
|
|
SingleDimPartitionsSpec.TARGET_ROWS_PER_SEGMENT,
|
|
|
|
String(this.targetRowsPerSegment),
|
|
|
|
);
|
2020-10-01 18:14:41 -04:00
|
|
|
}
|
|
|
|
if (this.maxRowsPerSegment) {
|
2020-10-02 02:59:21 -04:00
|
|
|
await setLabeledInput(
|
|
|
|
page,
|
|
|
|
SingleDimPartitionsSpec.MAX_ROWS_PER_SEGMENT,
|
|
|
|
String(this.maxRowsPerSegment),
|
|
|
|
);
|
2020-10-01 18:14:41 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
interface SingleDimPartitionsSpecProps {
|
|
|
|
readonly partitionDimension: string;
|
|
|
|
readonly targetRowsPerSegment: number | null;
|
|
|
|
readonly maxRowsPerSegment: number | null;
|
|
|
|
}
|
|
|
|
|
|
|
|
export interface SingleDimPartitionsSpec extends SingleDimPartitionsSpecProps {}
|
|
|
|
|
Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 15:38:09 -04:00
|
|
|
/**
|
|
|
|
* Data loader partition step configuration.
|
|
|
|
*/
|
|
|
|
export class PartitionConfig {
|
2020-10-01 18:14:41 -04:00
|
|
|
readonly forceGuaranteedRollupText: string;
|
|
|
|
|
Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 15:38:09 -04:00
|
|
|
constructor(props: PartitionConfigProps) {
|
|
|
|
Object.assign(this, props);
|
2020-10-01 18:14:41 -04:00
|
|
|
this.forceGuaranteedRollupText = this.forceGuaranteedRollup ? 'True' : 'False';
|
Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 15:38:09 -04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
interface PartitionConfigProps {
|
|
|
|
readonly segmentGranularity: SegmentGranularity;
|
2020-10-01 18:14:41 -04:00
|
|
|
readonly timeIntervals: string | null;
|
|
|
|
readonly forceGuaranteedRollup: boolean | null;
|
|
|
|
readonly partitionsSpec: PartitionsSpec | null;
|
Web console basic end-to-end-test (#9595)
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
2020-04-09 15:38:09 -04:00
|
|
|
}
|
|
|
|
|
|
|
|
export interface PartitionConfig extends PartitionConfigProps {}
|