Commit Graph

333 Commits

Author SHA1 Message Date
James Agnew aae53d6ec7
Improve loading performance (#3197)
* Synthea perf work

* Cleanup

* Work on performance

* Ongoing perf work

* More perf work

* Perf work

* Test fixes

* Updates

* Test fix

* Test fixes

* Clean up tests

* Test cleanup

* Cleanup

* Add test

* One more perf tweak

* Fixes

* Add changelog

* Add changelog

* License header update

* License header updates
2021-11-28 10:47:30 -05:00
Tadgh 0a64294467
Release 5.6.0 (#3174)
* 3138 externalized binary packages (#3139)

* Add test and impl

* Add changelog

* Fix test

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/5_6_0/3138-support-externalized-binaries.yaml

* add beans to test configs

* Typo

* Update hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/binstore/IBinaryStorageSvc.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* Update hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/binstore/NullBinaryStorageSvcImpl.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

Co-authored-by: Kevin Dougan SmileCDR <72025369+KevinDougan-SmileCDR@users.noreply.github.com>
Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* 3131 - Added support for the lookup operation in the Remote Terminology code (#3134)

* Remove leading underscores from identifiers (#3146)

* Version bump

* License files

* version.yaml

Co-authored-by: Kevin Dougan SmileCDR <72025369+KevinDougan-SmileCDR@users.noreply.github.com>
Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>
2021-11-19 15:24:44 +00:00
TipzCM 25c025cdee
3153 update unknowncodesystemwarningvallidationsupport to have configurable severities (#3155)
* 3153 updating the unknowncodesystemwarningvalidationsupport class

* 3153 updating version for smile

* 3153 update broken tests

* 3153 adding log

* 3153 cleanup

* 3153 cleanup

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
2021-11-10 16:06:09 -05:00
jamesagnew 5de955c3c2 Licence header updates 2021-11-09 08:44:18 -05:00
TipzCM 59bf8b836f
Issue 3120 support adding retry extension (#3137)
* issue-1134 initial work on adding retry handler

* issue-1134 subdscription creation fixes

* issue-1134 update the test

* issue-1134 retry policy off canonicalsubscription

* issue 1134 base subscription changes

* add failing test

* issue 1134 retry configs on base parameters

* issue 1134 passing forward params

* issue-1134 added more tests for making sure subscriptions create their channels correctly

* issue-1134 updates to get the retry to channels

* issue-1134 updating channel factory

* issue-1134 remove the dlq since it's not defineable

* issue-3120 changelog added

* issue-3120 cleaning up

* issue-3120 test fix

* issue-3120 review fixes

* issue-3120 fixed bad master merge

* issue-3120 updates for new release

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Ken Stevens <khstevens@gmail.com>
2021-11-03 18:21:10 -04:00
Tadgh a10c58e839
Bump version, add version to versionenum (#3135) 2021-11-02 09:40:56 -04:00
Tadgh 7c63441663
Revert "Issue 1134 retry mechanic on subscriptions (#3121)" (#3128)
This reverts commit 9b23e9e825.
2021-10-30 19:46:42 +00:00
TipzCM 9b23e9e825
Issue 1134 retry mechanic on subscriptions (#3121)
* issue-1134 initial work on adding retry handler

* issue-1134 subdscription creation fixes

* issue-1134 update the test

* issue-1134 retry policy off canonicalsubscription

* issue 1134 base subscription changes

* add failing test

* issue 1134 retry configs on base parameters

* issue 1134 passing forward params

* issue-1134 added more tests for making sure subscriptions create their channels correctly

* issue-1134 updates to get the retry to channels

* issue-1134 updating channel factory

* issue-1134 remove the dlq since it's not defineable

* issue-3120 changelog added

* issue-3120 cleaning up

* issue-3120 test fix

* issue-3120 review fixes

* issue-3120 fixed bad master merge

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Ken Stevens <khstevens@gmail.com>
2021-10-29 11:21:43 -04:00
Tadgh 8a7b5c1b2f
Multiple improvements (#2997)
* Add failing test

* Tidy test for new tokenparam style

* Add removeByQualifier to SearchParameterMap

* Add new property binder for search terms

* Member move, add test for new parser

* whoops missed an add

* Passing test.

Introduced SearchParamTextWrapper to give a type for the binder.
Enhance FulltextSearchSvcImpl to understand TokenParams as well as StringParams.

* Add more to test, change analyzer for template

* Optimize imports

* Minor refactoring

* added eq and gt test cases

* added missing gt search param tests

* Search all CodeableConcepts

* Extract new LuceneRuntimeSearchParam to mediate indexing and searching.

* Fix up broken test with mock

* Add support for identifier type

* Bump for special maven package

* fix-me to todo

* Prevent version check

* FIx version for test

* DSTU2 does not support fluent path

* Don't accidentally strip SPs in DSTU2

* added lt tests

* Add some high-bound test cases

* More test cases and new compressed format

* More test cases

* More test cases

* Re-order columns for easier reading

* Comments

* Cleanup and comments

* Make support :text for all token SPs

* Make support :text for all token SPs

* Fixed incomplete date issue

* Set to the last millisecond 23:59:59.999

* Disabled 4 failed JVM/TZ related test cases

* add test for failing batch bug, remove bounding on queue

* remove parallelism from non-get batch operations

* Fix count for test

* New index layout

* Notes and cleanup

* Notes and cleanup

* remove the jetbrains

* Demote FIXME to WIP

* Change :text search to be lucene simple search and use standardAnalyzer.

* Change :text to require * for prefix match.

* add prefix negative test

* Bump version

* Added changelog

* Add more search parameters to lucene indexing (#3000)

* dirty the branch

* Add oversight to changelog

* Start using jpa indexing for es

* Correct negative search test

* Update templates, index tokens, write new test. TODO gotta modify SB

* Add test for code token search

* add comment

* Flesh out token search

* minor cleanup and comments

* Extract index handling from FulltextSearchSvcImpl

* D'oh.  Actually use elastic to search.

* Move supported query check closer to query builder

* String search test before activating lucene

* Add string:exact to hibernate search.

* Add string:contains search

* cleanup

* demote fixmes to allow build to run.

* Add unmodified string search.

* empty stubs for quantity and reference

* Ignore magic search params in lucene index

* Support full-text on all FHIR releases

* Include reference lookups in ES

* Fix and/or logic in _text

* Test for string with space

* Cherry-pick the NoFT fix

* Disable advanced indexing in a bunch of tests of emitted sql.

* Stub changelog

* Fix :missing flag

* Move DaoConfig up to BaseJpaTest to share teardown

* Disable new indexing for partition test

* Add a different analyzer and field for default string search vs text

* checkstyle for a pre-build

* Index full prefix for string search when using Lucene

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/5_6_0/3000-Extend-lucene-indexing.yaml

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Update hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/dao/HibernateSearchQueryBuilder.java

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Review feedback.

Co-authored-by: Tadgh <garygrantgraham@gmail.com>
Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Remove double-check

* Version bump

* Fix typo

Co-authored-by: Michael Buckley <michael.buckley@smilecdr.com>
Co-authored-by: Long Ma <longma@Longs-MacBook-Pro.local>
Co-authored-by: Frank Tao <frankjtao@gmail.com>
Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>
2021-10-28 15:13:16 -04:00
JasonRoberts-smile b267fdb752
Jr 20211018 chained references 2 (#3099)
* Create index entries for outbound references of contained resources

* build query for chained reference

* fix case where the contained reference is an explicit id rather than a continued chain

* fix contained index to use path names not search param names

* make qualified search work

* cleanup and changelog

* recurse while creating indexes on contained resources

* double link both contained

* longer contained subchains

* adding some failing test cases to illustrate the limitations of qualified searches

* clean up merge cruft

* changelog
2021-10-22 12:41:05 +00:00
michaelabuckley f98f3cdaa9
Extend ordinal date search to year and month precision (#3102)
* added eq and gt test cases

* added missing gt search param tests

* added lt tests

* Add some high-bound test cases

* More test cases and new compressed format

* More test cases

* More test cases

* Re-order columns for easier reading

* Comments

* Cleanup and comments

* Fixed incomplete date issue

* Set to the last millisecond 23:59:59.999

* Disabled 4 failed JVM/TZ related test cases

* Added changelog

Co-authored-by: Long Ma <longma@Longs-MacBook-Pro.local>
Co-authored-by: Frank Tao <frankjtao@gmail.com>
2021-10-21 15:32:26 -04:00
Tadgh b427135b04
Patient/$everything _id parameter (#3077)
* Add tests. Add implementation in R4. Add changelog

* Wip partial fixup of other STU versions

* Updated usages of new patient type everything api

* Partial commit of incomplete documentation

* Remove docs

* Cut to token instead of string

* Bump hapi version due to breaking api changes
2021-10-17 14:34:33 +00:00
jamesagnew a54ebd1d30 License header updates 2021-10-11 17:22:54 -04:00
James Agnew 310a4cdb58
Migrate CLOB columns to LOB (SMILE-3072) (#3016)
* Migration finished

* Work on LOBS

* Add to ClobMigrated annotation

* Deal with lobs

* Work on new approach

* HAPI FHIR version bump

* Add changelog

* Add license header

* Fix intermittent

* CLeanup

* Cleanup
2021-10-11 16:47:37 -04:00
James Agnew aed718bd6f
Avoid redundant PID lookups in transaction processing (#3061)
* Avoid redundant PID lookups in transaction processing

* Add changelog

* Test fix
2021-10-11 13:45:57 -04:00
Ken Stevens b1dd40fdba
Try commenting out clearAllStaticFieldsForUnitTest to see what happens (#3034)
* Try commenting out clearAllStaticFieldsForUnitTest to see what happens

* Rename clearAllStaticFieldsForUnitTest to randomizeLocaleAndTimezone and only do that
2021-09-24 16:01:28 -04:00
Ken Stevens 53c7476dae
hapi storage (#3024)
* first pass moving core storage classes to storage-api

* move term service apis

* nearly done

* rename hapi-fhir-storage-api to hapi-fhir-storage and move transaction processor

* rename hapi-fhir-storage-api to hapi-fhir-storage and move transaction processor

* create new SearchConstants class to store platform independent search constants

* move a couple of subscription services

* move transaction processor adapter to storage

* version bump

* move searchparam

* fix test

* fix compile includes

* fix text

* fix docs
2021-09-23 22:04:46 -04:00
Ken Stevens 060791aeb4
hapi-fhir-storage-api (#3006)
* rename hapi-fhir-jpaserver-migrate to hapi-fhir-sql-migrate and move hapi-tasks into jpaserver-persistence

* rename hapi-fhir-jpaserver-api to hapi-fhir-storage-api

* move bulk import apis

* create hapi-fhir-jpa

* create hapi-fhir-jpa

* move CircularQueueCaptureQueriesListener to japi-fhir-jpa

* move mdm logs to storage-api

* move gziputil to storage-api

* move quartz scheduling to hapi-fhir-jpa

* move default subscription channel factory to storage-api

* consolidated batch constants

* correct accidental Logs change

* remove dependency of cdr-api on cdr-persistence

* fix javadoc

* pom descriptions

* review feedback
2021-09-21 15:36:08 +00:00
James Agnew 5554431146
Avoid dead SearchParameter links in CapabilityStatement (#2790)
* Avoid dead SearchParameter links in CapabilityStatement

* Resolve fixmes

* Build fix

* Test fix

* Test fix

* Test fix

* Work on metadata

* Fixes

* Test fixes

* Test fix

* Test fix

* Test fixes

* Test fix

* Work on test fixes

* Test fixes

* Test fixes

* Test fixes

* Test fix

* Fix params

* Resolve fixme

* Adjust changelogs

* Version bump to 5.6.0-PRE7

* Add license header

* Bump version to 5.6.0-PRE5-SNAPSHOT
2021-09-11 18:27:18 -04:00
Tadgh 3ec47cb0ba Bump version as we break the localcontainerentitymanagerfactory bean constructors args 2021-09-07 13:58:22 -04:00
Ken Stevens 346e29920a
Issue 2927 convert mdm clear to spring batch (#2929)
* initial roughout of mdm clear batch job

* still roughing out classes

* finished first draft of reader

* most tests passing now

* all tests pass.  now FIXMEs

* FIXMEs done.  Time for regression.

* fix test

* changelog and docs

* fix test

* pre-review cleanup

* version bump

* move spring autowire deps for cdr

* move spring autowire deps for cdr

* finally got beans working phew

* rearrange method calls so persistence module can perform clear operation on its own

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa_mdm/mdm_operations.md

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* review feedback

* review feedback

* Remove inheritance

* fix beans

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>
Co-authored-by: Tadgh <garygrantgraham@gmail.com>
2021-09-03 14:58:19 -04:00
Tadgh 37ad73a422 Bump PRE version so we don't conflict downstream 2021-08-23 15:42:18 -04:00
Tadgh 4fadc62066 Bump Hapi Version 2021-08-06 13:44:04 -04:00
Ken Stevens f5dfc3a5ed
New reindex spring batch operation (#2845)
* first sweep to add new $reindex job

* first cut done.  next up integration test.

* start roughing out reindex everything

* Failing test

* test passes

* add BundleUtil helper method

* rough in processEverything

* rough in processEverything

* fix mock test

* merge master

* Make BaseJpaTest.mySrd lenient.

* fix test

* reindex everything works with FIXMEs

* reindex everything more tests

* moar tests

* moar tests

* fix test

* fix test

* fix regression and fix test

* fix test cycle

* fixme

* fix test

* test provider layer

* reindex everything partition aware

* yay last fixme!

* run reindex in partitioned threads

* add fixmes for areas that require extra testing

* added transaction and log message

* manual integration testing

* changelog

* reindex all partitions

* bump hapi version

* fix test

* moar test

* pre-review cleanup

* fix javadoc

* review feedback

* review feedback

* review feedback

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/5_5_0/2845-reindex-operation.yaml

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* review feedback

* review feedback

* Update hapi-fhir-jpaserver-base/src/test/java/ca/uhn/fhir/jpa/delete/job/ReindexJobTest.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* Update hapi-fhir-jpaserver-base/src/test/java/ca/uhn/fhir/jpa/delete/job/ReindexJobTest.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* Update hapi-fhir-jpaserver-base/src/test/java/ca/uhn/fhir/jpa/provider/r4/MultitenantBatchOperationR4Test.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* Update hapi-fhir-jpaserver-base/src/test/java/ca/uhn/fhir/jpa/provider/r4/MultitenantBatchOperationR4Test.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* Update hapi-fhir-jpaserver-searchparam/src/main/java/ca/uhn/fhir/jpa/searchparam/ResourceSearch.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

* Update hapi-fhir-server/src/main/java/ca/uhn/fhir/rest/api/server/storage/IDeleteExpungeJobSubmitter.java

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>

Co-authored-by: michaelabuckley <michaelabuckley@gmail.com>
2021-07-29 19:18:49 -04:00
James Agnew f957661dda
Patient ID Partition Mode should support conditional create (#2828)
* Fixes

* Test fix

* Add changelog

* Test fixes

* Test fixes

* Docs tweak

* Partitioning fixes

* Import fix

* Fix test data
2021-07-26 14:18:58 -04:00
jamesagnew a5e6674e84 Version bump to 5.5.0-PRE7-SNAPSHOT 2021-07-22 08:41:16 -04:00
James Agnew 6d37749be8
Add non unique combo search params (#2809)
* Start work on nonunique combo search params

* Work on SQL

* Version bump

* A fix

* Test fixes

* Fixes

* Undo version bump

* Test fixes

* Resolve fixme
2021-07-21 19:14:14 -04:00
Michael Buckley cbb9a4355a Add details when we fail to load an element of a package definition. 2021-07-19 17:59:54 -04:00
Tadgh a7e6813510 Update string length. Add migration 2021-07-13 15:46:47 -04:00
James Agnew d762a07817
Partition patient interceptor (#2766)
* Start work on interceptor

* Implemented

* Add interceptor and docs

* Add docs

* Compile fix

* Test fixes

* Test cleanup

* Test fixes

* Test fixes

* Improve error message

* Add broken tests

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/5_5_0/2766-add-patient-id-compartment-interceptor.yaml

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Update hapi-fhir-jpaserver-base/src/main/java/ca/uhn/fhir/jpa/interceptor/PatientIdPartitionInterceptor.java

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Address review comments

Co-authored-by: Tadgh <garygrantgraham@gmail.com>
Co-authored-by: Tadgh <tadgh@cs.toronto.edu>
2021-07-05 09:15:20 -04:00
James Agnew 4bb495e9d7
Fix NPE (#2774) 2021-07-02 05:29:49 -04:00
Tadgh f66f9b992e Bump pre-version as this will break downstream implementations 2021-06-30 15:37:19 -04:00
jamesagnew 9bc4859761 Add test 2021-06-23 20:55:22 -04:00
James Agnew 8379392e16
Add inline tag mode (#2748)
* Start work on tag mode 3

* Add changelog

* Test fix
2021-06-22 23:08:09 -04:00
James Agnew 8b205b23d0
Fix history paging (#2741)
* Fix history paging

* Add changelog

* Add license header
2021-06-18 17:47:51 -04:00
Ken Stevens 134631fdee
Convert delete expunge to use Spring Batch (#2697)
* prepare to add $delete-expunge operation that will create a spring batch job

* Add operation

* Wire up jpa provider.  Begin with failing test.

* Copy/paste bulk import job as a starting point.
FIXME with proposed design

* delete expunge job parameter validation with test

* implemented reader
stubbed processor, writer

* wip for master merge

* started implementing reader

* started implementing reader

* working with stubs

* happy path batch delete expunge is done

* Provider done but test not passing.  Guessing batch infrastructure not running in that test.

* IT test works now

* add reader test

* Converted delete _expunge=true to use new batch job

* DeleteExpungeDaoTest passes

* Fix test

* Change batch size to integer

* rename search count to batch size

* Make delete expunge partition aware

* updated docs

* pre-review cleanup

* change log

* add partition id to SystemRequestDetails

* Make RequestPartitionId serializable

* Change delete expunge provider to use partition id instead of tenant name

* fix tests

* test pointcut gets called

* assert on pointcut calls

* Add resource type to STORAGE_PARTITION_SELECTED pointcut

* bump hapi-fhir version
move expunge provider parameters from JpaConstants to ProviderConstants

* bump hapi-fhir version

* copyrights

* restore deleteexpungeservice for mdm

* restore deleteexpungeservice for mdm

* fix test

* public constants

* convert instant to date

* Moved expunge constants to ProviderConstants

* final review

* disabling InMemoryResourceMatcherR5Test.testNowNextMinute() to see if I can get a clean test run

* fix tests

* fix tests

* fix tests

* fix tests

* review feedback

* review feedback

* review feedback

* review feedback

* review feedback

* review feedback

* improve logging

* bump version

* version bump

* recovering from failed merge

* unzip RequestListJson per Gary's suggestion.  I didn't want to do it at first, but as usual Gary was right.

* fix serialization
2021-06-15 10:36:05 -04:00
jamesagnew 376a84d213 Version bump 2021-06-14 13:12:34 -04:00
James Agnew b934abb297
Impropve transaction Performance (#2717)
* Work on changes

* Work on perf

* Work on testing

* Work on perf

* Work on perf

* Work on fix

* Work on perf

* Ongoing work

* Add changelog

* Additional docs

* Test fixes

* Address review comments

* Test fix
2021-06-14 13:08:19 -04:00
James Agnew dc627dc019
Add bulk export authorization layer (#2712)
* Add bulk export authorization layer

* Add docs

* Version bump

* Authorize any

* Add bulk export - all

* Address lgtm issue
2021-06-06 20:47:43 -04:00
James Agnew 24b3f0f30d
Enable mass ingestion mode (#2681)
* Work on fixes

* Work on counts

* Enable mass ingestion mode

* Add changelog

* Test fix

* Test fix

* Test fixes

* Fixes

* Test fix

* Test fix
2021-05-27 18:43:51 -04:00
James Agnew 7a83c76e65
Add validation suppression mode (#2675)
* Add validation suppression mode

* Add changelog

* Compile fix
2021-05-26 17:41:48 -04:00
jamesagnew 926bf393d3 Merge branch 'rel_5_4' 2021-05-17 09:10:58 -04:00
James Agnew 40ecafcd31
Don't unescape plus in match URLs (#2658)
* Don't unescape plus in match URLs

* Perf improvements

* Add changelog

* Test fix

* Work on tests

* Fixes

* Test fix
2021-05-16 21:51:23 -04:00
Tadgh d6252526c3 Move member variable 2021-05-11 10:53:10 -04:00
jamesagnew 2068372998 Version bump 2021-05-07 17:43:54 -04:00
jamesagnew 63c4b22cd3 Undo bump 2021-05-07 17:42:50 -04:00
jamesagnew 17e41520a4 Version bump 2021-05-07 17:41:39 -04:00
Tadgh e15825b336 Increase length, add test 2021-05-05 15:29:02 -04:00
James Agnew 970a9884f0
Avoid tag definition constraint (#2620)
* Avoid tag definition constraint failure

* Version bumps

* Add changelog

* Avoid guava issue

* Test fix

* Test fix

* Bump pom
2021-05-01 17:16:15 -04:00
James Agnew 95d7ba6fa8
Streamline Search SQLs (#2576)
* Streamline SQL queries

* SQL query tidying

* Add changelog

* Test fixes

* Tests passing

* Ongoing work

* Test fixes

* Test fixes

* Build fix

* Cleanup

* Fixes
2021-04-28 10:29:09 -04:00