Commit Graph

109 Commits

Author SHA1 Message Date
James Agnew b7d1ae217d
New batch framework / Bulk Import (#3387)
* Work on new batch framework

* Work on new batch framework

* Work on new batch framework

* Work on batch

* Compile working

* Work on bulk import

* Adjust import

* Work

* Work on batch

* Bump version

* WOrk on new batch processes

* Work on bath

* Bump to PRE4

* Build fixes

* CLeanup

* Small tweak

* Add exception code

* Test fixes

* Test fixes

* Test fix

* Additional synchronization

* Add license headers

* Test fixes

* Test fixes

* Add changelogs

* Address PG

* Test fix

* Test fix

* Test fixes

* Review notes

* Work on tests

* Test fixes

* Test fix

* Work on tests

* Tets fix

* Test fixes

* Test fixes

* Add missing exception codes

* Test fix

* Test fixes

* More test fixing

* License headers

* Test fix

* Add new test logging

* Work on tests

* Test fixes

* Test fix

* Resolve fixme

* Try to avoid test failure

* Add import command

* Work on storage

* Fix error codes

* Fixes

* License header

* Build fix

* Build fixes

* Fix dep
2022-02-27 16:04:49 -05:00
Ken Stevens 631391819b
remove hapi jpa spring config hierarchy (#3390)
* initial refactor to remove hapi jpa spring config hierarchy

* fix poms

* fix batch config

* dependency rearrangement arising from further testing

* fix searchDao bean name

* fixed dstu2 tests

* consolidate validation config

* dstu2 validation regression (man dstu2 support is expensive)

* fix test (with WIP comment)

* restore poms

* fix demo projects

* fix build

* bumping version

* rename generated config class
2022-02-16 20:44:03 -05:00
Johnson Lu c42ecc5c08
Add error codes to all exceptions (#3278)
* Initial design of adding exception codes to exceptions

* Initial design of adding exception codes to exceptions

* change int module name to string

* add comment

* shorten message

* shorten message

* added checkstyle

* developing checker

* completed checkstyle plugin and unit test for it

* fix NPE

* fix checker so it ignores throws on exceptions that aren't new

* change parameter order in InvalidResponseException

* accept Msg.code() in any parameter since some exceptions like MessageException have the message string as the second parameter

* exclude generated files

* BIG COMMIT added error codes to all exception throws

* example test fix

* Modified tests such that the assert statements would better match the message responses.

* Modified additional tests such that the assert statements would better match the message responses.

* fix test

* merge master

* merge master

* Fourth commit of modified tests such that the assert statements would better match the message response

* Changes made to modify assert statements such that they match the new errors which now use error codes. Also changed ResourceNotFoundException to include error codes in its messages.

* Modified assert messages to better match new error codes

* Time Zone changes DRAFT. Certain tests pass in intellij and fail during mvn install

* Modified assert messages to better match new error codes

* Added comments for time zone fixes. Previous commit comment on time zone fixes was incorrect. By invaliding caches, all tests on both intellij and mvn install passed.

* Reverted changes on azure-pipelines.yml

* updated hapi-fhir version for checkstyle module

* changes to azure-pipelines.yml

* changes to azure-pipelines.yml

* change to pom.xml to include hapi-fhir-checkstyle in ALLMODULES

* changes to azure-piplines (these changes are to be reverted after testing)

* reverted changes to azure-piplines

* reverted change where hapi-fhir-checkstyle was added to pom.xml (hapi-fhir)

* Attempted fix on checkstyle build error

* modified azure-piplines.yml to change maven repository for checkstyle build

* Attempted fix on checkstyle build error

* checkstyle config location change for hapi-depoloyable-pom. (will be reverted if it does not work)

* attempted dependency add for hapi-depoloyable-pom. (will be reverted if it does not work)

* reverted changes for hapi-deployable-pom

* add javadoc

* update last code

* Attempted fix on checkstyle build error

* Finished adding in missing error codes to pass Checkstyle checks

* Test pipelining

* Fixing errors and bumping version (as suggested by Gary)

* Merge

* Fixing Checkstyle error

* Attempted fix for date error

* Add error code onto assert statement

* Add in a couple more error codes

* Fixing assert statements

* Bump to static version for dependency

* Adding error code to message

* Adding error codes to exceptions

* Adding error codes to exceptions

* Change to work with cdr branch

* Update to faulty error message

* Merge changes

* allow checkstyle pom to deploy

Co-authored-by: Ken Stevens <khstevens@gmail.com>
Co-authored-by: Simon Zuccherato <simon.zuccherato@smilecdr.com>
Co-authored-by: Predap <simon.zuccherato@gmail.com>
Co-authored-by: Predap <46201929+Predap@users.noreply.github.com>
Co-authored-by: Tadgh <garygrantgraham@gmail.com>
2022-01-31 13:10:15 -08:00
Tadgh 6506527dd6
Fix mdm config for single application context environment. (#3338)
* wip

* remove beans

* Remove javax.mail from the classpath as we use simplemail instead, which uses jakarta.
2022-01-27 06:18:44 -05:00
jamesagnew aafc42fdd8 License header updates 2022-01-04 13:37:10 -05:00
James Agnew d610d33ac3
Dependency bumps (#3264)
* Bump a number of versions

* Fixes

* Work on version bump

* Bump

* Test fix

* More version bumps

* Build fixes

* Test fixes

* Fixes

* Fixes

* Test fix

* Bump jacoco

* Test fix

* Test fix

* Test fix

* Build fix

* Build fix

* Remove versioned H2

* Build fix

* Avoid memory issue in validation tests

* JDK bump

* Restore compile with errorprone

* Force coloured output

* Force jansi

* Try again to force jansi

* Fix for hardcoded date that just passed

* Fixes to JPA

* Improve changelog

* Work on failing test

* Test fix

* Compile fix

* One more version bump

* Test fixes
2022-01-04 13:21:14 -05:00
Tadgh 700b614610 Allow system request details to go through MDM 2021-12-15 22:47:33 -08:00
Ken Stevens 9753b66610
2564 contained resource validation flag (#3225)
* First cut at multithreading bundle validation

* javadoc

* change test to a red-green test

* add daoconfig

* move bundle concurrency config from validationsupport to request validator

* Moved concurrent details from ValidationOptions to FhirValidator

* Remove bundleValidationThreadCount (it's set by creating an appropriate executor)

* validate threadpool prefixes

* FIXME cleanup

* reassign FIXMEs to JB

* only create the threadpool if it will be used

* Add the MdcTaskDectorator to all threadpools

* Add bundle path to concurrently validated bundle resource validation messages.

* Add test to check bundle entry path being added to validation messages

* Threaded validation of contained resources initial checkin

* FHIR validator refactored to maintain correct bundle path and async validation task

* Refactored to use validation complete invoke callback to support raw string resource or resource object

* switch from XML to JSON

* # Conflicts:
#	hapi-fhir-base/src/main/java/ca/uhn/fhir/validation/FhirValidator.java
#	hapi-fhir-validation/src/test/java/org/hl7/fhir/r4/validation/FhirInstanceValidatorR4Test.java

* Updated to use new HAPI-FHIR version

* merge 'validate_contains_change' into issue-3144-multi-threaded-bundle-validation

* Updated to use new HAPI-FHIR version

* Revert "Updated to use new HAPI-FHIR version"

This reverts commit 910b7f676b.

* Revert "merge 'validate_contains_change' into issue-3144-multi-threaded-bundle-validation"

This reverts commit c7d60fc3e6.

* Revert "Updated to use new HAPI-FHIR version"

This reverts commit aa8a7af455.

# Conflicts:
#	hapi-fhir-validation/src/test/java/org/hl7/fhir/r4/validation/FhirInstanceValidatorR4Test.java

* Fix Test dependencies after reverting changes.

* merge 'validate_contains_change' to 2564-contained-resource-validation-flag

* Disable threadpool prefix name validate check.

* merge master

* fix thread regexp

* pre-review cleanup

* change log

* fix test

* revert to XML as default validation serialization

* fix test

* fix test

Co-authored-by: Jaison B <jaisonb@gmail.com>
2021-12-08 15:18:54 -05:00
Ken Stevens 0656037c8d
Ks 20211203 mdm possible match (#3216)
* begin with failing test

* fix issue

* change log

* code review
2021-12-03 19:44:25 +00:00
James Agnew aae53d6ec7
Improve loading performance (#3197)
* Synthea perf work

* Cleanup

* Work on performance

* Ongoing perf work

* More perf work

* Perf work

* Test fixes

* Updates

* Test fix

* Test fixes

* Clean up tests

* Test cleanup

* Cleanup

* Add test

* One more perf tweak

* Fixes

* Add changelog

* Add changelog

* License header update

* License header updates
2021-11-28 10:47:30 -05:00
Tadgh 41ef7cf961
Support multiple EID systems for MDM. (#3178)
* Add implementation, changelog, docs, and tests

* Fix potential NPE

* Use constant instead of *
2021-11-24 06:05:13 +00:00
TipzCM 7e291bb469
3163 refactor some base mdm stuff (#3166)
Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
2021-11-16 10:47:44 -05:00
Tadgh e29ad5179a
Head off potential MDM bug (#3140)
* wip

* fix up error message

* wip

* Fix all tests

* Update old tests for failure messages

* Add another test case

* Remove dead file

* Fix log message
2021-11-04 22:09:46 +00:00
James Agnew 5f672bc9e8
Deprecate redundant BaseResourceModifiedMessage myId (#3116)
* Deprecate redundant BaseResourceModifiedMessage myId

* Add changelog

* Test fix

* Test fix
2021-10-28 07:55:46 -04:00
jamesagnew a54ebd1d30 License header updates 2021-10-11 17:22:54 -04:00
James Agnew 756b0fdc7b
Fix transaction scoping (#2799)
* Work on transaction boundaries

* Test fixes

* Test fixes

* Test fix

* Resolve FIXME

* Test fix

* Test fix

* Test fix

* Test fixes

* Test fix

* Adjust changelog

* Remove unneeded changelog

* Test fix

* Test fixes

* Test fixes

* Test fixes

* Test fixes

* Test fix

* Fixes

* Test fix
2021-10-11 16:45:22 -04:00
katiesmilecdr 3c80dba90e
[3040] finish implementation (#3049)
* [3040] finish implementation

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa_mdm/mdm_operations.md

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa_mdm/mdm_operations.md

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa_mdm/mdm_operations.md

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* [3040] throw exception if link exists

* [3040] add matchResult parameter/update documentation

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>
2021-10-07 17:42:21 +00:00
TipzCM a474e88ed4
issue-2397 implementing the mdm expansion on everything and searches (#3044)
* issue-2397 implementing the mdm expansion on everything and searches

* issue-2397 code cleanup and removal of test code

* issue-2397 some tests

* issue-2397 changelog added

* issue-2397 review fixes

* issue-2397 updated change log with example and fixed failing tests

* issue-2397 test fixing

* issue-2397 fixing test

* issue-2397 review fix

* issue 2397 fixing changelog file

Co-authored-by: leif stawnyczy <leifstawnyczy@leifs-MacBook-Pro.local>
Co-authored-by: Tadgh <tadgh@cs.toronto.edu>
2021-10-07 10:46:23 -04:00
Ken Stevens 060791aeb4
hapi-fhir-storage-api (#3006)
* rename hapi-fhir-jpaserver-migrate to hapi-fhir-sql-migrate and move hapi-tasks into jpaserver-persistence

* rename hapi-fhir-jpaserver-api to hapi-fhir-storage-api

* move bulk import apis

* create hapi-fhir-jpa

* create hapi-fhir-jpa

* move CircularQueueCaptureQueriesListener to japi-fhir-jpa

* move mdm logs to storage-api

* move gziputil to storage-api

* move quartz scheduling to hapi-fhir-jpa

* move default subscription channel factory to storage-api

* consolidated batch constants

* correct accidental Logs change

* remove dependency of cdr-api on cdr-persistence

* fix javadoc

* pom descriptions

* review feedback
2021-09-21 15:36:08 +00:00
Jaison B 8bb9aca8df Merge branch 'master' into 2849_add_new_mdm_param 2021-09-17 10:18:24 -06:00
Tadgh b4e2679872 Remove jetbrains 2021-09-13 21:15:23 -04:00
Nick Goupinets 35aaa98431 Merge branch 'master' into 2849_add_new_mdm_param 2021-09-07 11:27:49 -04:00
Nick Goupinets 010628fa86 Added asserts 2021-09-07 11:24:37 -04:00
Nick Goupinets c63a503841 Updated link processing for MDM events 2021-09-03 16:29:26 -04:00
Ken Stevens 346e29920a
Issue 2927 convert mdm clear to spring batch (#2929)
* initial roughout of mdm clear batch job

* still roughing out classes

* finished first draft of reader

* most tests passing now

* all tests pass.  now FIXMEs

* FIXMEs done.  Time for regression.

* fix test

* changelog and docs

* fix test

* pre-review cleanup

* version bump

* move spring autowire deps for cdr

* move spring autowire deps for cdr

* finally got beans working phew

* rearrange method calls so persistence module can perform clear operation on its own

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/docs/server_jpa_mdm/mdm_operations.md

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* review feedback

* review feedback

* Remove inheritance

* fix beans

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>
Co-authored-by: Tadgh <garygrantgraham@gmail.com>
2021-09-03 14:58:19 -04:00
Nick Goupinets 2a60e91713 WIP Dup Test 2021-09-02 17:41:57 -04:00
Nick Goupinets 4e012236c2 Updated comments and added misc refactorings 2021-08-27 16:31:23 -04:00
Nick Goupinets 8403b85429 Merge remote-tracking branch 'origin/master' into 2849_add_new_mdm_param 2021-08-27 09:42:08 -04:00
Nick Goupinets 6ea58f74d1 Code review 2021-08-23 10:55:09 -04:00
Tadgh 3135e82ea7 Strip jetbrains annotations 2021-08-20 18:48:34 -04:00
Nick Goupinets 1e41621eca Rolled back changes 2021-08-19 16:43:51 -04:00
Nick Goupinets 8670f107b5 Updated link expansion 2021-08-17 13:17:44 -04:00
Tadgh 0a303a0e11 Add MDM expansion by golden pid 2021-08-06 13:22:08 -04:00
Nick Goupinets 4be3334f6f WIP after persistence checked 2021-08-04 16:33:59 -04:00
Nick Goupinets 891a6304d0 WIP 2021-08-04 11:06:14 -04:00
Nick Goupinets 0d34fe61c8 Adding tests 2021-07-30 11:17:56 -04:00
Nick Goupinets a2aa2ca1c3 2849 Added new parameter to MDM processing 2021-07-29 15:52:27 -04:00
Ken Stevens 92ec113775
fixed bug (#2824) 2021-07-23 16:58:53 -04:00
Tadgh a4fb72975d
Merge pull request #2767 from hapifhir/2765-mdm-paging
Paging implementation for MDM query links operation
2021-07-01 23:16:47 -04:00
Tadgh cfcc939caa Fix broken tests, add license 2021-07-01 18:28:29 -04:00
Tadgh 1eb82d633d Rip out pagingprovider requirement 2021-06-30 18:41:18 -04:00
Michael Buckley fa750d209e $mdm-submit operation was only submitting 100 resources and then stopping.
It now correctly submits all requested resources.
2021-06-30 14:48:28 -04:00
Tadgh 8f3cbb0a7f Refactor docs 2021-06-30 14:48:13 -04:00
Tadgh 61260dd1b1 Slightly more portable solution for paging 2021-06-30 11:54:59 -04:00
Tadgh a770d577cb Initial quick implementation of paging with tests 2021-06-29 17:46:03 -04:00
Nick Goupinets 95012623f6 Added null checks 2021-06-22 15:12:10 -04:00
James Agnew 6ea1438f1d
Add transaction write semaphore interceptor (#2743)
* Add transaction write semaphore interceptor

* Add changelog

* Test fix
2021-06-20 12:55:49 -04:00
Ken Stevens 32ac16e2aa
add a limit to mdm candidate searches (#2742)
* done

* verbal review feedback

* Update hapi-fhir-docs/src/main/resources/ca/uhn/hapi/fhir/changelog/5_5_0/2739-max-mdm-matches.yaml

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>

* review feedback

Co-authored-by: Tadgh <tadgh@cs.toronto.edu>
2021-06-18 20:48:34 -04:00
Ken Stevens a4856bba17
fix iBundleProvider.getResources(from,to) throwing a NPE (#2632)
* fix iBundleProvider.getResources(from,to) throwing a NPE

* change log

* review feedback

* rename test to IT that has concurrency failures
2021-05-05 21:50:51 -04:00
Nick Goupinets 1f10a15c35 Merge branch 'master' into 2515_MDM_survivorship_rules 2021-04-19 17:33:53 -04:00