Commit Graph

9091 Commits

Author SHA1 Message Date
James Agnew b833c13a3c
Improve batch2 error handling (#3482)
* Improve batch2 error handling

* Build fixes

* Work on reindexing

* Test fixes

* Account for gzipped contents

* License header updates

* Register VS operations on public server

* Test fix

* Build fix

* Remove animal sniffer plugin

* Compile fix
2022-03-26 17:40:12 -04:00
Olivia You 625ed936e9
3470 ne date search (#3471)
* fix mongo date search with the NOT_EQUALS prefix

* add test & changelog

* add some unit tests

* rename unit test names to be more clear

Co-authored-by: olivia-you <olivia.you@smilecdr.com>
2022-03-25 17:51:17 -04:00
michaelabuckley 1dcc64ee44
Fast-path for lucene queries that skip the database (#3468)
* Store resource data inline with lucene/es resourcetable index.
* Use fulltextsearch to load resources if resource inlined in ES/Lucene index
Co-authored-by: Jaison Baskaran <jaisonb@gmail.com>
2022-03-25 16:21:47 +00:00
James Agnew 23cb65af25
Address github security report (#3492) 2022-03-25 07:41:48 -04:00
jamesagnew 34eb47aba6 Add changelog 2022-03-25 05:53:05 -04:00
Thomas Papke 91799b7a8c
Resolve #3490 ApacheProxyAddressStrategy should support Forward header from (#3491)
* #3490 ApacheProxyAddressStrategy should support Forward header from
Rfc7239

* #3490 ApacheProxyAddressStrategy should support Forward header from
Rfc7239

Remove one copy & paste error

* #3490 ApacheProxyAddressStrategy should support Forward header from
Rfc7239

Fix failing test

* #3490 ApacheProxyAddressStrategy should support Forward header from
Rfc7239

Fix failing test
2022-03-25 05:50:59 -04:00
michaelabuckley 9e46fad1b9
Undo "helpful" IntelliJ edit (#3489)
IntelliJ added a compiler configuration which should not have been committed.
2022-03-24 18:43:47 +00:00
michaelabuckley 92db526786
Skip database query when hibernate search fully satisfies search (#3478)
If a query is completely satisfied by Hibernate Search, skip the database.

Co-authored-by: Jaison Baskaran <jaisonb@gmail.com>
2022-03-22 22:41:21 +00:00
Jaison Baskaran 3eb3014d25
Revert change to treat Canonical URLs as Locals (#3483)
* Revert change added in MR: https://github.com/hapifhir/hapi-fhir/pull/3050 to add support for treating Canonical Urls as locals. This is no longer needed as the issue reported in #2843 (_include not working for Canonicals) has been addressed in MR: https://github.com/hapifhir/hapi-fhir/pull/3440
2022-03-22 17:46:10 +00:00
Ken Stevens 50e6e94105 review feedback 2022-03-21 15:08:04 -04:00
James Agnew f82534ca51
Convert Reindex into Batch2 Job (#3458)
* Start working on reindex job

* Updates

* Ongoing work

* Ongoing work

* Work on reindex

* Reindex work

* Add logging

* Fix typo

* Test fix

* Test fix

* Address fixme

* Liocense header

* Resolve fixme

* Add logging

* Update logs

* Address review comments

* License header

* Test fixes

* Test fix

* Test fixes

* Test fix

* Version bump

* Version bump
2022-03-19 16:07:58 -04:00
jmarchionatto f518f30506
Issue 3450 there is no way to recreate freetext indexes for terminology entities (#3481)
* Add reindex-terminology batch command

* Handle number of thread throttling and concurrency with other terminology batch operations

* Add required dbcp2 dependency

* Fix test

* Improve ConnectionPoolInfoProvider setup. Handle maximum connections.

* Remove java higher version construction

* Remove unused config

* Add reindex terminology integration test.
Reset termConcept counters before pre-expanding, which otherwise accumulate if it was pre-expanded before.

* Address MR comments

* Adjust test to tested class change

* Adjust test to tested class change

Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
2022-03-18 15:09:25 -04:00
jmarchionatto e45f7ee9cb
Issue 3450 there is no way to recreate freetext indexes for terminology entities (#3463)
* Add reindex-terminology batch command

* Handle number of thread throttling and concurrency with other terminology batch operations

* Add required dbcp2 dependency

* Fix test

* Improve ConnectionPoolInfoProvider setup. Handle maximum connections.

* Remove java higher version construction

* Remove unused config

* Add reindex terminology integration test.
Reset termConcept counters before pre-expanding, which otherwise accumulate if it was pre-expanded before.

Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
2022-03-18 08:39:12 -04:00
Tadgh 17cf1ed9ce
Rel 5 7 1 mergeback (#3480)
* fix handling of common search parameters

* Revert "fix handling of common search parameters"

This reverts commit 89c45eebdc.

* Fix implementation, add test (#3378)

* Fix implementation, add test

* Tighten test

* Rip out dead modules

* Add changelog

* Jr 20220210 handle common search params in contained searches (#3377)

* fix handling of common search parameters

* add support for reference search parameters with multiple paths

* Issue 3357

* Version bump

* Fixed null pointer exception for re-loading subscription on cdr restart and when there's no partition id in the request, and added tests

* added changelogs for this fix

* Fix broken changelog file

* Can't specify specific resource type permissions for bulk export (#3376)

* deny user from exporting without perms

* add unit tests

* add changelog

Co-authored-by: olivia-you <olivia.you@smilecdr.com>

* Make migration donothing as it was added in error

* Remove ehcache

* Add version.yaml:

* Add to sources/javadocs for dist

* Fix typo

* fix up pom

* wip test removing checkstyle plugin from deployable pom

* remove test pom changes, instead just dont deploy to sonatype

* typo

* Version bump

* Add implementation, testing, and changelog

* Add backport info, make aware of 6_0_0

* add jira ref

* Fix test

* Add versionenum

* Remove dups

* Remove unnecessary stuff

Co-authored-by: Jason Roberts <jason.roberts@smilecdr.com>
Co-authored-by: JasonRoberts-smile <85363818+JasonRoberts-smile@users.noreply.github.com>
Co-authored-by: Mark Iantorno <markiantorno@gmail.com>
Co-authored-by: Steven Li <steven@smilecdr.com>
Co-authored-by: Olivia You <46392181+oliviayou@users.noreply.github.com>
Co-authored-by: olivia-you <olivia.you@smilecdr.com>
2022-03-17 22:31:40 -04:00
Mark Iantorno 49e19d9779
Issue 3441 reindex query being ignored (#3451)
https://github.com/hapifhir/hapi-fhir/issues/3441

Co-authored-by: Tadgh <garygrantgraham@gmail.com>
2022-03-14 15:18:43 -04:00
michaelabuckley 03b6c589c8
New indexing for number and quantity search parameters (#3464)
* New Add Index forced-INCLUDE columns, and Online option

* Remove forced-include.  Simpler just to add all columns to index.

* New date index definitions.

* Oops.  Remove test for deleted feature.

* Cleanup

* Drop IDX_SP_DATE_UPDATED

* Add online option to drop index

* Update annotations to match and redefine FK

* Push sp->resource FK down to control name

* remove dead end

* Cleanup date tests

* New token query test cases for sql extraction

* New token search param indexing.

* Continue to allow the legacy hibernate names while we update the indexing.

* Fix jpa annotations with overloaded field

* Decide on token sorting - we do it.

* review comments

* fixme

* disable dead data migration too

* review fixups

* Test cases for numeric search

* add ne and ap

* Test for Numeric SP

* start

* finish number

* new indexing for quantity

* Move config

* cleanup
2022-03-11 16:23:06 +00:00
michaelabuckley 80d1a5a6f8
Mb new token search index (#3386)
* New token search param indexing.
* New token query test cases for sql extraction
* Retire JPA fk ugly nams
2022-03-11 00:51:53 +00:00
Ken Stevens dfd99c5471
improve search parameter registry logging (#3452)
* Startup log cleanup.  Use cached fhir contexts.

* improve logging

* improve logging

* add full comparator

* wrap resource search param map

* rename wrapped map methods

* add fixmes

* further searchparam encapsulation

* fix regression found by test

* improve logging

* pre-review cleanup

* pre-review cleanup

* bump hapi version

* license

* since I'm bumping hapi-fhir now, improve the api dependency

* licenses
2022-03-09 19:57:38 -05:00
Tadgh 7eb1bd14d1
Do not validate auto-created placeholder resources (#3461)
* Add implementation, testing, and changelog

* add jira ref
2022-03-08 22:22:04 +00:00
samuelwlee2 cde39dcc6c
3423 include with client assigned id partition (#3448)
* changes to doResolvePersistentIds

* make changes to tests

* Add code changes after the review
2022-03-08 11:48:33 -05:00
James Agnew ac80c7ffaa
Search Narrowing by Code Tweaks (#3453)
* Search narrowing tweaks

* Search narrowing tweaks

* Test

* Test fixes

* Try to fix LGTM

* Add license headers

* Test fixes

* Test fixes
2022-03-07 20:27:21 -05:00
Jaison Baskaran 0e9372775b
Fix to address lastN search limiting to only 100 hits. (#3456) 2022-03-07 22:04:54 +00:00
Jaison Baskaran abea0389be
2843 questionnaire include url fix (#3447)
* Added test to verify BUG: #2843 (_include doesn't return canonically referred resource)

* Fix '_include' query search by native query to support canonical url references

* Fixed code review comments

* Restored the res_type condition on join
2022-03-07 16:39:13 +00:00
Ken Stevens 62bcb80443 Fix sitemap.xml 2022-03-06 15:46:49 -05:00
Jaison Baskaran ecf782d59e
Fix '_include' to work for canonical urls (#3440)
* Added test to verify BUG: #2843 (_include doesn't return canonically referred resource)

* Fix '_include' query search by native query to support canonical url references

* Fixed code review comments
2022-03-03 21:28:19 -07:00
michaelabuckley 20e092ba6d
Better error messages for ValueSet $expand contextDirection=existing (#3446)
* Better error messages for ValueSet $expand contextDirection=existing
* Apply to $lastn too - it uses aggregations
2022-03-03 21:11:08 -07:00
Tadgh d11a312abf This bean should not be lazy 2022-03-03 18:59:14 -08:00
Tadgh a2133f6940
Break a couple Circular Bean Dependencies (#3437)
* Rely on context refresh event instead of postconstruct. Load bean lazily.

* Remove parent autowire

* Break termcodesystem circular dependency with new bean

* Break cycle between BaseTermReadSvcImpl and TermCodeSystemStorageSvcImpl

* move back into helper method

* Rip out scheduling and submission to spring batch away from BulkDataExportSvcImpl

* wip

* Fix bean naming

* Revert

* re-add autowiring

* Reformat

* docs and reformat

* Back out change

* feedback from review

* merge conflicts
2022-03-03 09:37:40 -05:00
Tadgh 016774d653
Bump mariadb (#3439) 2022-03-02 18:35:08 -05:00
jmarchionatto 4b5f75c069
Issue 3421 pre expansion fails for valuesets with more than 1024 concepts (#3434)
* Build search queries involving many predicates avoiding crashing when number is larger than configured BooleanQuery.maxClauseCount

* Add changelog

* Restore temp directory config

Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
2022-03-02 14:11:55 -05:00
James Agnew f614eaf6c8
Avoid error when unknown code system behaviour set to warning (#3436)
* Avoid error when unknown code system behaviour set to warning

* Add changelog

* Test fix

* Build fix

* Fixes

* Address review comment
2022-03-01 17:30:59 +00:00
James Agnew 2cba62b4e8
Enable search narrowing on large ValueSets (#3405)
* Enable search narrowing on large ValueSets

* ValueSet improvements

* Work on narrowing

* Work on narrowing

* Work on narrowing

* Work on narrowing

* Add test

* Work on narrowing interceptor

* Work on narrowing

* License headers

* Refactor code narrowing

* Add docs

* Version bump

* Test fix

* Test fixes

* Build fix

* Fixes

* Version bump

* Test fix

* License header updates

* Build fix

* Test fixes

* Test fix

* Test fix

* Docs fix

* Test fix

* Test fix

* Resolve fixme

* Bump

* Force a CI build

* Make CI happen again
2022-03-01 09:16:31 -05:00
Tadgh 68c1015552
Bump core to 5.6.36 (#3433)
* Bump version

* Add changelog, make use of theNoInactive flag

* add test

* remove magic strings
2022-03-01 08:23:48 -05:00
michaelabuckley 86f7373f16
Distinguish multi-coded items during token autocomplete (#3429)
Add nested indexing to support autocomplete of multi-coded items.
2022-03-01 00:59:09 +00:00
michaelabuckley b263c7a595
bump compiler settings. JDK 11 for prod, JDK 17 for tests (#3435) 2022-02-28 21:51:32 +00:00
kailiu-smile 1bc6e0e3ae
Smile 2775 package install logs ignore resource validation (#3381)
* Added log to the create() and valiForUpload() methods in the PackageInstallerSvcImpl class

* Added helper methods to extract URL from SearchParameter.Tested and fixed log entries.

* Changelog entry

* Update 3384-Package-Install-added-logs-for-resources-validation.yaml

Co-authored-by: Kai Liu <kliu@Kais-MacBook-Pro.local>
2022-02-28 19:52:43 +00:00
James Agnew bf04add6ef
Add missing fields to canonical subscription compare (#3431) 2022-02-28 06:48:59 -05:00
James Agnew f2afb0844d
Fix migration (#3428) 2022-02-27 21:25:14 -05:00
James Agnew b7d1ae217d
New batch framework / Bulk Import (#3387)
* Work on new batch framework

* Work on new batch framework

* Work on new batch framework

* Work on batch

* Compile working

* Work on bulk import

* Adjust import

* Work

* Work on batch

* Bump version

* WOrk on new batch processes

* Work on bath

* Bump to PRE4

* Build fixes

* CLeanup

* Small tweak

* Add exception code

* Test fixes

* Test fixes

* Test fix

* Additional synchronization

* Add license headers

* Test fixes

* Test fixes

* Add changelogs

* Address PG

* Test fix

* Test fix

* Test fixes

* Review notes

* Work on tests

* Test fixes

* Test fix

* Work on tests

* Tets fix

* Test fixes

* Test fixes

* Add missing exception codes

* Test fix

* Test fixes

* More test fixing

* License headers

* Test fix

* Add new test logging

* Work on tests

* Test fixes

* Test fix

* Resolve fixme

* Try to avoid test failure

* Add import command

* Work on storage

* Fix error codes

* Fixes

* License header

* Build fix

* Build fixes

* Fix dep
2022-02-27 16:04:49 -05:00
Ben Li-Sauerwine 65776bbab3
Adds a NDJSON-capable Bulk Data Import Provider (#3039)
* NDJsonParser with one example test.

* Add test for empty NDJson, and fix bug for empty NDJson.

* Adds multi-Patient test, and fixes bug whereby all multi-line NDJSON would be put into the same line.

* Adds test for NDJson with newlines in it.

* Adds test for converting non-Bundle types to NDJSON failing.

* Confirm that we can only extract to Bundle types in test.

* Update hapi-fhir-base/src/main/java/ca/uhn/fhir/context/FhirContext.java

Co-authored-by: James Agnew <jamesagnew@gmail.com>

* Documents behavior of the NDJsonParser in FhirContext.

* Attempt to fix failing build by using TestUtil to clear context in the manner of r4 parser tests instead of dstu

Also clean up indentation.

* Adds a BulkDataImportProvider, with a single test.  More tests are forthcoming.

* Adds an additional test.

* Enhance tests to include content of job files.  Fix a bug whereby the content was never actually being added.

* Adds several tests for BulkImportProvider's polling operation.

* Adds requred Msg.code to errors.

* Apparently I can't duplicate Msg.code even if it's the same issue.

Co-authored-by: James Agnew <jamesagnew@gmail.com>
2022-02-27 11:38:20 -05:00
Ken Stevens c7be674b84
small performance optimization (#3426) 2022-02-26 02:35:23 +00:00
katiesmilecdr c9b87d7c67
3418 get resource, response doesn't contain total if consent service enabled (#3420)
* throw an exception if parameter in use

* fix string

* add msg code

* fix msg code

* only check _total=accurate

* extract validateParameter()

* update validateParameter
2022-02-24 17:11:14 -05:00
jmarchionatto dbc9427a05
Issue 3391 2588 expand valueset regex include not working (#3410)
* Refactor ValueSet expansion code to simplify property filtering by using new Hibernate search V6 nested classes capabilities; use generic hibernate search queries where possible (mostly everywhere but regex queries) and add R4 integration tests  for both elastic and lucene.

* Refactor ValueSet expansion code to simplify property filtering by using new Hibernate search V6 nested classes capabilities; use generic hibernate search queries where possible (mostly everywhere but regex queries) and add R4 integration tests  for both elastic and lucene.

* Add comment for trick

* Add codes to exception messages

* Add test specific for the fix

* Move changelog to current release. Add upgrade notice.

* Improve fail-fast method

* Update upgrade instructions

* Try luck with new exception codes

* Update hapi-fhir-storage/src/test/java/ca/uhn/fhir/jpa/search/ElasticsearchNestedQueryBuilderUtilTest.java

Add missed quote

Co-authored-by: michaelabuckley <michael.buckley@smilecdr.com>

Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
Co-authored-by: michaelabuckley <michael.buckley@smilecdr.com>
2022-02-24 21:32:11 +00:00
Ken Stevens 6968cdfcfc
Subscription cleanup (#3422)
* fix log message

* improve unit test coverage

* make js subscription delivery handler a separate kind of handler so executor is automatically destroyed when handler is destroyed

* Significant Overhaul.  Return everything to the handler.

* test passes
cleaned up rest hook subscriber

* ensure deleting subscription destroys JavaScript engine

* add permission

* pre-review cleanup.  remove unneccesary api-breaking changes.

* pre-review cleanup.  remove unneccesary api-breaking changes.

* pre-review cleanup.

* review feedback

* review feedback
2022-02-24 18:55:16 +00:00
Tadgh ca6381a52a Build fix 2022-02-23 18:43:55 -08:00
jmarchionatto 53c5a9bbeb
Add FK index to avoid timeout when deleting large ValueSet (#3416)
* Add FK index to avoid timeout when deleting large ValueSet

* Add exception to allow same name for index and FK

Co-authored-by: juan.marchionatto <juan.marchionatto@smilecdr.com>
2022-02-23 13:54:30 -05:00
michaelabuckley 96cadbba3e
Run online index operations non-transactionally on Postgres (#3413)
* Run online index operations non-transactionally on Postgres

* narrow non-transactional
2022-02-23 02:49:53 +00:00
Tadgh fcada39044
add online deferred invalidation keyword for oracle (#3409)
* add online deferred invalidation keyword for oracle

* test

* remove online option from sql server drop index - it can only apply to clustered indices

Co-authored-by: Michael Buckley <michael.buckley@smilecdr.com>
2022-02-22 16:44:28 -05:00
James Agnew cc44deee1a
GraphQL Introspection Support (#3348)
* Start working on graphql Schema

* Start testing

* Work on introspection

* Work on introspection

* Use integers for date ordinals

* Add changelog

* GraphQL updates

* Ongoing wrk

* Cleanup

* Bump core lib

* Add changelog

* Clean up dependencies

* CLeanup

* Add missing message

* Test fix

* Change to force CI
2022-02-22 09:48:57 -05:00
James Agnew 1e2e17784a
Fix parameter count limitation on MSSQL/Oracle large transactions (#3075)
* Fix parameter count limitation on MSSQL/Oracle large transactions

* Changelog fix

* Move changelog

* Force a change to trigger CI

* Test refactoring
2022-02-20 11:52:42 -05:00