web-console/e2e-tests/tutorial-batch.spec.ts would occasionally timeout
between the transition from the data loader "configure schema" and
"partition" steps due to missing waits when toggling the rollup setting.
Also, fix shellcheck warnings for script/druid.
Load data and query (i.e., automate
https://druid.apache.org/docs/latest/tutorials/tutorial-batch.html) to
have some basic checks ensuring the web console is wired up to druid
correctly.
The new end-to-end tests (tutorial-batch.spec.ts) are added to
`web-console/e2e-tests`. Within that directory:
- `components` represent the various tabs of the web console. Currently,
abstractions for `load data`, `ingestion`, `datasources`, and `query`
are implemented.
- `components/load-data/data-connector` contains abstractions for the
different data source options available to the data loader's `Connect`
step. Currently, only the `Local file` data source connector is
implemented.
- `components/load-data/config` contains abstractions for the different
configuration options available for each step of the data loader flow.
Currently, the `Configure Schema`, `Partition`, and `Publish` steps
have initial implementation of their configuration options.
- `util` contains various helper methods for the tests and does not
contain abstractions of the web console.
Changes to add the new tests to CI:
- `.travis.yml`: New "web console end-to-end tests" job
- `web-console/jest.*.js`: Refactor jest configurations to have
different flavors for unit tests and for end-to-end tests. In
particular, the latter adds a jest setup configuration to wait for the
web console to be ready (`web-console/e2e-tests/util/setup.ts`).
- `web-console/package.json`: Refactor run scripts to add new script for
running end-to-end tests.
- `web-console/script/druid`: Utility scripts for building, starting,
and stopping druid.
Other changes:
- `pom.xml`: Refactor various settings disable java static checks and to
disable java tests into two new maven profiles. Since the same
settings are used in several places (e.g., .travis.yml, Dockerfiles,
etc.), having them in maven profiles makes it more maintainable.
- `web-console/src/console-application.tsx`: Fix typo ("the the").
* expose props for S3
* added env inputs
* add scarry warning
* use .password
* put the warning front and center
* Update web-console/src/views/load-data-view/load-data-view.tsx
Co-Authored-By: Suneet Saldanha <44787917+suneet-s@users.noreply.github.com>
* let prettier rewrap the text
Co-authored-by: Suneet Saldanha <44787917+suneet-s@users.noreply.github.com>
* add support for new version of DQT
* update druid-query-toolkit
* fix direction css
* fix remove
* update package
* remove useless conditional
* bump package
* jest -u
Co-authored-by: Maggie Brewster <maggiebrewster@implydata20sMBP.attlocal.net>
* Reconcile terminology and method naming to 'used/unused segments'; Don't use terms 'enable/disable data source'; Rename MetadataSegmentManager to MetadataSegments; Make REST API methods which mark segments as used/unused to return server error instead of an empty response in case of error
* Fix brace
* Import order
* Rename withKillDataSourceWhitelist to withSpecificDataSourcesToKill
* Fix tests
* Fix tests by adding proper methods without interval parameters to IndexerMetadataStorageCoordinator instead of hacking with Intervals.ETERNITY
* More aligned names of DruidCoordinatorHelpers, rename several CoordinatorDynamicConfig parameters
* Rename ClientCompactTaskQuery to ClientCompactionTaskQuery for consistency with CompactionTask; ClientCompactQueryTuningConfig to ClientCompactionTaskQueryTuningConfig
* More variable and method renames
* Rename MetadataSegments to SegmentsMetadata
* Javadoc update
* Simplify SegmentsMetadata.getUnusedSegmentIntervals(), more javadocs
* Update Javadoc of VersionedIntervalTimeline.iterateAllObjects()
* Reorder imports
* Rename SegmentsMetadata.tryMark... methods to mark... and make them to return boolean and the numbers of segments changed and relay exceptions to callers
* Complete merge
* Add CollectionUtils.newTreeSet(); Refactor DruidCoordinatorRuntimeParams creation in tests
* Remove MetadataSegmentManager
* Rename millisLagSinceCoordinatorBecomesLeaderBeforeCanMarkAsUnusedOvershadowedSegments to leadingTimeMillisBeforeCanMarkAsUnusedOvershadowedSegments
* Fix tests, refactor DruidCluster creation in tests into DruidClusterBuilder
* Fix inspections
* Fix SQLMetadataSegmentManagerEmptyTest and rename it to SqlSegmentsMetadataEmptyTest
* Rename SegmentsAndMetadata to SegmentsAndCommitMetadata to reduce the similarity with SegmentsMetadata; Rename some methods
* Rename DruidCoordinatorHelper to CoordinatorDuty, refactor DruidCoordinator
* Unused import
* Optimize imports
* Rename IndexerSQLMetadataStorageCoordinator.getDataSourceMetadata() to retrieveDataSourceMetadata()
* Unused import
* Update terminology in datasource-view.tsx
* Fix label in datasource-view.spec.tsx.snap
* Fix lint errors in datasource-view.tsx
* Doc improvements
* Another attempt to please TSLint
* Another attempt to please TSLint
* Style fixes
* Fix IndexerSQLMetadataStorageCoordinator.createUsedSegmentsSqlQueryForIntervals() (wrong merge)
* Try to fix docs build issue
* Javadoc and spelling fixes
* Rename SegmentsMetadata to SegmentsMetadataManager, address other comments
* Address more comments
* update web-console data loader to support unified s3 and google input source
* fixes
* add placeholder for objects
* only show objects if it already exists
* fix bug with sqlOuterLimit, use sqlOuterLimit instead of wrapping sql query for web console
* fixes, refactors, tests
* meh
* better name
* fix comment location
* fix copy and paste
* first steps
* clean licenses
* fix capabilities
* fix specs
* more tests
* new web console on coordinator and overlord, remove setup for old consoles, old configs
* better message
* update licenses
* sync license files
* more button
* fix tslint issue
* jetty-rewrite dependency to add redirects for old console paths
* put dependency in the right place
* fix overlord detection
* fix notices, dedupe licenses
* make segment timeline work in no SQL mode
* update license
* revert hard coded coordinator mode from testing
* update restricted mode copy
* Add missing button for include future, and handle logic for default true case
* Remove duplicate go to tasks button
* Fix lgtm issue
* Revert changes on old console
* Made changes based on PR comments
* Created temporary interval input component
* Make reusable interval component
* Fixed errors with typing invalid dates
* Fix interval input styling and place into autoform
* Fix styling of popover calendar that opens off the page
* Add snapshot test and change interval to required props
* Add functionality to enter hours minutes second
* Fix min date limit
* Remove console log
* Fix difference in timezone
* Update snapshot test
* Fixed snapshot test without changing min max date
* Made changes based on discussion before converting to hooks
* Rewrote using hooks and deleted duplicate states
* Remove unused states
* Change sql query view numbers to monospace
* Made changes based on discussion
* Removed duplicate state
* fine grained capabilities
* fix tests
* configure all cards
* better detection
* update tests
* rename server to service
* node -> service
* remove console log
* better loader in data loader
* Stateful auto compaction
* javaodc
* add removed test back
* fix test
* adding indexSpec to compactionState
* fix build
* add lastCompactionState
* address comments
* extract CompactionState
* fix doc
* fix build and test
* Add a task context to store compaction state; add javadoc
* fix it test
* IOConfig for compaction task
* add javadoc, doc, unit test
* fix webconsole test
* add spelling
* address comments
* fix build and test
* address comments
* Add frontend buttons to remove group by
* Change icon for remove group by
* Update web console to use latest toolkit
* Add test cases to verify that remove group by buttons are rendered
* Correct mistake of using incorrect components
* Update tests for two cases
* Put remove button after group by
* Retrieved data from endpoint and displayed on table
* Added view raw button, removed json, and fixed formatting for table
* Remove error var
* Fixed snapshot for updated status dialog
* Made changes based on PR review
* Added version tag
* Updated snapshot to match changes
* Made more changes based on review
* Fix filter and formatting
* Fix filter and add unit test
* fix styling of dialog
* Fix footer height
* Fixed testing of filtering
* fixing ingest spec
* rm console.log
* ingest segment spec
* make sure step query always being run
* better example interface
* add keywords
* placeholders
* do not overwrite datasource name in data loader spec
* fix comment typo
* Use Codecov
Upload coverage reports to Codecov. For now, having Codecov comment on
PRs or enforcing a minimum coverage threshold are both disabled until
the Codecov coverage reports look reliable:
https://codecov.io/gh/apache/incubator-druid
* Split bash and curl into separate lines
* tidy up menus
* fix query output API
* save context also
* pull out auto run into a switch
* better copy
* add group_id
* support FLOAT also
* use built in time logic
* fix trunc direction
* add skipCache
* add manifest url
* remove depricated props
* data loader shuffle UI
* better support for inline firehose
* feedback changes
* make numShards mandatory sometimes
* edit context dialog fixes
* make auto form less agro
* download full log
* improve auto column detection
* add sample for example
* small fixes
* run jest -u
* add trim
* add to groupby
* save
* add functionality to column tree menu
* remove history dialog
* rename file
* fixes
* add new values to time-menu
* add query dialog
* add more functions to collumn-tree menu
* add menu functions
* remove test
* update tests, fix extra 0s, clear in between time filters
* fixes
* add deffered
* save
* add alias's
* change menu style
* fix substring
* small fixes
* add auto run to run button
* move divider to a reasonable position
* fix auto expland
* run jest -u
* fix auto run
* prevent aggregate cooumn changes
* fix group by bug
* add quertAst to state
* ui fixes
* prevent filter on non existant ast
* add auto run to shows
* use handle run
* handle *
* add alias and tslint fix
Reorganize Travis CI jobs into smaller faster (and more) jobs. Add
various maven options to skip unnecessary work and refactored Travis CI
job definitions to follow DRY.
Detailed changes:
.travis.yml
- Refactor build logic to get rid of copy-and-paste logic
- Skip static checks and enable parallelism for maven install
- Split static analysis into different jobs to ease triage
- Use "name" attribute instead of NAME environment variable
- Split "indexing" and "web console" out of "other modules test"
- Split 2 integration test jobs into multiple smaller jobs
build.sh
- Enable parallelism
- Disable more static checks
travis_script_integration.sh
travis_script_integration_part2.sh
integration-tests/README.md
- Use TestNG groups instead of shell scripts and move definition of jobs
into Travis CI yaml
integration-tests/pom.xml
- Show elapsed time of individual tests to aid in future rebalancing of
Travis CI integration test jobs run time
TestNGGroup.java
- Use TestNG groups to make it easy to have multiple Travis CI
integration test jobs. TestNG groups also make it easier to have an
"other" integration test group and make it less likely a test will
accidentally not be included in a CI job.
IT*Test.java
AbstractITBatchIndexTest.java
AbstractKafkaIndexerTest.java
- Add TestNG group
- Fix various IntelliJ inspection warnings
- Reduce scope of helper methods since the TestNG group annotation on
the class makes TestNG consider all public methods as test methods
pom.xml
- Allow enforce plugin to be run from command-line
- Bump resources plugin version so that "[debug] execute contextualize"
output is correctly suppressed by "mvn -q"
- Bump exec plugin version so that skip property is renamed from "skip"
to "exec.skip"
web-console/pom.xml
- Add property to allow disabling javascript-related work. This property
is overridden in Travis CI to speed up the jobs.
* Add segment timeline chart
* fix file
* Fix bugs: no data handling & scaling problems
* resolve conflict
* changed package-lock
* do not show by default
* trust the interop
* stricter type fixes
* fix sasslint
* tidy up nulls
* standardize more on undefined
* updated licenses
* do not do heavy handed rendering
* reindex from druid
* tidy up
* add inline firehose
* add husky
* sass lint
* better suggestion
* fix script typo
* adjust time formats
* add missing time formats
* use term 'reindex'
* fix lodash.compact
* tidy up nulls
* standardize more on undefined
* updated licenses
* update snapshot
* do not do heavy handed rendering
* add placeholder to SQL view
* remove pointelss fragment
Clint Wylie
mcbrewster
Chi Cao Minh
Jihoon Son
Vadim Ogievetsky
Aditya
Roman Leventov
Caroline1000
Jonathan Wei
Vadim Ogievetsky
jon-wei
Evan Ren
Benedict Jin
Gian Merlino